Running Wazuh fully offline: every phone-home, and how to close it

What we needed

Sentinel is the NOC and SOC appliance we’re building for networks that can’t call out. Wazuh does the endpoint work: log collection, file integrity, configuration checks and vulnerability detection.

Out of the box, Wazuh calls home. On a sealed network that call fails. Sometimes it fails quietly and a feature stops working without telling you. Sometimes it fails loudly and fills the logs with noise you learn to ignore. On a network that does have a way out, the same call tells someone you exist. All three are bad.

So the bar was simple. A running stack makes zero lookups of any name outside the site. Every feature that needs outside data gets it from a file we carry in.

Everything below was tested on Wazuh 4.14.8, the manager and agent containers from Docker Hub. Wazuh is open source: the manager and agent are GPLv2, and the indexer is an Apache-2.0 distribution of OpenSearch. Credit to the Wazuh team. Offline vulnerability detection has been supported since 4.8, and it works. The defaults just assume the internet.

How we listened

Reading the source tells you what the code can do. Listening tells you what it does. We did both, and the wire found one the source reading missed.

The rig is small. Put the manager on a Docker network with no route out. Run tcpdump in the same network namespace. Every name the manager tries to resolve shows up in the capture, even though none of them resolve.

# A network with no route out.
docker network create --internal wz-int

# A sidecar that owns the network namespace and logs every DNS query.
docker run -d --name dnswatch --network wz-int nicolaka/netshoot \
  tcpdump -n -l -vv -i any port 53

# The manager joins the sidecar's namespace, so every lookup it makes is captured.
docker run -d --name wazuh-manager --network container:dnswatch \
  wazuh/wazuh-manager:4.14.8

docker logs -f dnswatch

We ran the manager on its own for this. The indexer gets its own section further down.

What the stock image calls

Within a minute of start, the stock manager image looked up two names: cti.wazuh.com and cdn.amazonlinux.com. Then we closed them one at a time and ran it again.

Here are the manager settings that got us to zero, plus the two that close enrolment and pin upgrades. These are the elements to change in your ossec.conf, not a whole file.

<ossec_config>
  <global>
    <update_check>no</update_check>
  </global>

  <vulnerability-detection>
    <enabled>yes</enabled>
    <index-status>yes</index-status>
    <feed-update-interval>60m</feed-update-interval>
    <offline-url>file:///var/ossec/offline/vd-feed.zip</offline-url>
  </vulnerability-detection>

  <sca>
    <enabled>no</enabled>
  </sca>

  <auth>
    <remote_enrollment>no</remote_enrollment>
  </auth>

  <agent-upgrade>
    <wpk_repository>https://mirror.example/wazuh/wpk/</wpk_repository>
  </agent-upgrade>
</ossec_config>

Now each one, and why.

The update check

The Wazuh API checks for updates when it starts and every 24 hours after. It calls https://cti.wazuh.com/api/v1/ping and sends an installation ID.

<global>
  <update_check>no</update_check>
</global>

Vulnerability detection from a file

The vulnerability detector keeps its CVE data fresh with a content updater. It runs at start and then every feed-update-interval, on every manager node, workers included. By default it pulls from cti.wazuh.com.

Set offline-urland the content source switches to offline. The online API isn’t used at all. With the line set, the manager made zero lookups of cti.wazuh.com. Without it, it looked the name up within 60 seconds of start.

• offline-url takes file:// or http(s)://. The file can be zip, xz, gzip or raw JSON.
• The file must be owned root:wazuh with mode 750.
• The <indexer> block must be enabled. With index-status yes and the indexer off, the config is rejected.

Getting the feed

Do this on a machine that is allowed out. Never on the sealed side. The catalog is public and needs no account.

# Ask the catalog for the newest snapshot.
curl -fsS https://cti.wazuh.com/api/v1/catalog/contexts/vd_1.0.0/consumers/vd_4.8.0 \
  | jq -r '.data.last_snapshot_link'
# https://cti.wazuh.com/store/contexts/vd_1.0.0/consumers/vd_4.8.0/<offset>_<unixts>.zip

curl -fsS -o vd-feed.zip '<the link from above>'
sha256sum vd-feed.zip   # record it; check it again on the far side

What you get:

• A zip64 archive of about 320 MB, holding one NDJSON file of about 4.7 GB.
• One CVE record per line, with the fields name, offset, version, inserted_at and payload. The manager builds its own RocksDB from it.
• The feed is per consumer (vd_4.8.0), not per patch release. One feed serves 4.8 through 4.14.
• No deltas offline. The whole snapshot is re-ingested as upserts, and only when the file’s hash changes.

Check it before you carry it in

A check along these lines catches both. It needs only the Python standard library, which reads zip64 on its own.

import json
import sys
import zipfile

with zipfile.ZipFile(sys.argv[1]) as z:
    names = z.namelist()
    assert len(names) == 1, f"expected one member, got {names}"
    info = z.getinfo(names[0])
    # Over 4 GiB can only be recorded with zip64, so this proves both.
    assert info.file_size > 4 * 1024**3, f"member too small: {info.file_size}"
    with z.open(info) as f:
        first = json.loads(f.readline())
    missing = {"name", "offset", "version", "inserted_at", "payload"} - first.keys()
    assert not missing, f"first record lacks {missing}"

print("ok", names[0], info.file_size)

Then put it in place on the manager:

python3 check_feed.py vd-feed.zip
cp vd-feed.zip /var/ossec/offline/vd-feed.zip
chown root:wazuh /var/ossec/offline/vd-feed.zip
chmod 750 /var/ossec/offline/vd-feed.zip

After the next ingest, look at the vulnerability index. If it is empty after a rescan, something is wrong with the feed, not with your fleet.

Day zero, and what it costs

The manager image ships a preseeded database (vd_1.0.0_vd_4.13.0.tar.xz, 429 MB on Wazuh’s package server). It unpacks into queue/vd on first start. So a fresh install on a sealed network detects vulnerabilities from day zero, using data as old as the image.

Budget at least 20 GB on the manager volume for vulnerability detection. There are open disk-growth bugs (#33985, #34196). Our numbers come from a shared development VM, so treat them as a floor, not a sizing guide.

How fresh is fresh

One more reason to pin. Wazuh’s own planning issue (#34781) says offline mode is being removed in 5.0, with the indexer becoming the content source. The 5.x indexer plugins are also AGPL-3.0. We stay on 4.14.x until we’ve tested 5.x the same way.

The one we only found on the wire

With the update check off and the feed offline, one name was left: cdn.amazonlinux.com. Reading the source hadn’t turned it up.

The manager image is built on amazonlinux:2023. It runs Security Configuration Assessment (SCA) on the manager itself, with the policy cis_amazon_linux_2023.yml enabled. One check in that policy is:

c:dnf list installed | grep sudo

To answer it, dnf fetches its repository mirror list. Turning syscollector off made no difference. Turning SCA off on the manager did.

<sca>
  <enabled>no</enabled>
</sca>

This is the manager’s config, not the agents’. SCA on your endpoints keeps running. An endpoint check that calls dnf or yumuses that endpoint’s own repository config. That traffic is on your network, from your host, and it is a separate question.

Agent upgrades

PUT /agents/upgrade fetches the upgrade package (a WPK) from packages.wazuh.com. There is no switch to turn that off. In our test it looked the name up and failed with “The repository is not reachable”.

PUT /agents/upgrade_custom takes a file that is already on the manager. It made no lookups.

PUT /agents/upgrade_custom?agents_list=003&file_path=/var/ossec/var/upgrade/<package>.wpk

• Never call PUT /agents/upgrade. Also point <wpk_repository> at an internal mirror, as a backstop.
• Pick the WPK by the agent’s platform, architecture and package family (deb, rpm, msi or pkg). They are split all four ways.
• Leave the agents’ default <ca_store> alone. It is what accepts Wazuh-signed WPKs.
• The agent never downloads a WPK itself. The manager pushes it.

Mirror the packages on the connected side, and verify them there:

f=wazuh_agent_v4.14.8_linux_amd64.deb.wpk
curl -fsSO "https://packages.wazuh.com/4.x/wpk/linux/deb/amd64/$f"
curl -fsS "https://packages.wazuh.com/4.x/checksums/wazuh/4.14.8/$f.sha512" \
  | awk '{print $1}' > want.sha512
sha512sum "$f" | awk '{print $1}' | diff - want.sha512 && echo ok

Honest status: in our prototype the manager delivered and unpacked the WPK with no egress. The install step then failed inside the container agent. Upgrades on real machines are not proven yet.

Off by default. Keep them off.

These all reach out, and all of them ship disabled. The rule is to never write their blocks into the config at all.

• Cloud modules: office365, ms-graph, github, azure-logs, aws-s3, gcp.
• <integration>blocks such as VirusTotal and Slack, the API’s VirusTotal quota check, and maild.
• The shared-file download in remoted. Keep upload_configuration in api.yaml at its defaults.
• The indexer’s geospatial ip2geo datasource. It only runs if someone creates one.

The indexer

The Wazuh indexer is OpenSearch. It has no update check. Its telemetry is off by default (opensearch.experimental.feature.telemetry.enabledis false, and the otel plugin isn’t bundled). Performance Analyzer listens locally only.

The machine-learning plugin is the one to watch. ml-commons can download pretrained models and call remote connectors. The library underneath it, DJL, can download native libraries and send telemetry. It probes 169.254.169.254 first, then *.amazonaws.com.

plugins.ml_commons.remote_inference.enabled: false
plugins.ml_commons.local_model.enabled: false

OPT_OUT_TRACKING=true
DJL_OFFLINE=true

The indexer was not part of the first prototype, so these lines came from reading the OpenSearch and DJL source. They need the same capture as everything else. That is the point of the last section.

We don’t ship the Wazuh dashboard. If you run it: set wazuh.updates.disabled: true in wazuh.yml, and map.includeOpenSearchMapsService: false to stop map tiles.

Traps at deploy time

• Certificates. wazuh-certs-generator:0.0.4 downloads wazuh-certs-tool.sh from packages.wazuh.com when it runs. Generate the certificates yourself, from your own internal CA.
• Open enrolment by accident.The stock manager starts the enrolment service on port 1515 with “No password required”. If your rendered config goes missing and the container falls back to the image’s own, anyone who can reach 1515 can enrol. Fail the install if the rendered ossec.confisn’t there.
• The undocumented <cti-url>. Don’t use it to point at a mirror. The C and Python sides read it differently, and the update check pins the public CA bundle anyway.

Closing 1515

We don’t let hosts enrol themselves. Sentinel creates each agent through the Wazuh API and hands back only its key. So the enrolment port stays shut:

<auth>
  <remote_enrollment>no</remote_enrollment>
</auth>

That closes the TLS listener on 1515 and keeps the enrolment service’s local socket, which the API needs to create agents.

docker exec wazuh-manager grep "Port 1515 was set as disabled" /var/ossec/logs/ossec.log

Why the per-tenant design needed this is its own story. It’s in the tenant isolation paper.

Prove it stays quiet

Config is a claim. A capture is proof. And it has to keep being proof after every upgrade, because the next release can add a call.

In Sentinel, every internal container network is internal: true. The services that publish ports sit on bridges where a firewall chain logs and drops every new outbound connection. Our install test runs the full stack with that in place, captures DNS from the running services through an idle soak, and fails on any name outside the site. The two names from the stock image, cti.wazuh.com and cdn.amazonlinux.com, are the regression cases.

A minimal version for a lab box, with the stack on its own named bridge:

docker network create -o com.docker.network.bridge.name=br-stack stack

iptables -N STACK-EGRESS
iptables -A STACK-EGRESS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
iptables -A STACK-EGRESS -j LOG --log-prefix "STACK-EGRESS "
iptables -A STACK-EGRESS -j DROP
iptables --insert DOCKER-USER -i br-stack ! -o br-stack -j STACK-EGRESS

# Leave the stack idle for ten minutes, then:
journalctl -k | grep "STACK-EGRESS " || echo "quiet"

Do this yourself

Credits and sources

• Wazuh docs: offline vulnerability detection
• Wazuh docs: the global section and update_check
• Wazuh docs: agent-upgrade
• Wazuh source at v4.14.8
• OpenSearch ml-commons settings

Wazuh, OpenSearch and netshoot are their authors’ work. We just listened to them carefully.