Tenant isolation in a shared log pipeline: tag by peer address, not by what the sender claims

The problem

Sentinel runs one ingest pipeline for every tenant on the box. One Vectorreceives syslog, JSON, flow records and Wazuh alerts. One OpenSearch stores them, in a separate set of indices per tenant. It’s cheaper than a pipeline per tenant, and it’s how most shared setups end up.

The hard question is the first one. Whose event is this?Get it wrong and acme reads globex’s logs. Or worse, an attacker inside acme writes events into globex’s index, where globex’s analysts will trust them.

The answer is a rule. Decide the tenant from things the sender can’t write. Never from the event itself.

Who gets to say who sent it

We sorted every candidate field by who controls it.

We didn’t take that table on faith. We tested the ones that matter. An acme sender with {"tenant_id":"globex"} in its JSON body was still tagged acme. So was one with [meta tenant_id="globex"] in syslog structured data, and one whose syslog hostname was a globex address. A Wazuh agent in acme with a local label of tenant=globex produced alerts carrying that label, and they still landed in acme.

Tag network sources by peer address

For syslog, Vector’s syslog source puts the socket peer in .source_ip, for UDP and TCP. A structured-data element named source_ipdoes not override it. That’s the field to tag from.

sources:
  net_syslog_udp:
    type: syslog
    mode: udp
    address: 0.0.0.0:514
  net_syslog_tcp:
    type: syslog
    mode: tcp
    address: 0.0.0.0:601
  # JSON MUST be decoded as bytes. With codec json, a body {"host": ...}
  # replaces the peer address.
  net_json_tcp:
    type: socket
    mode: tcp
    address: 0.0.0.0:5140
    host_key: source_ip
    framing:
      method: newline_delimited
    decoding:
      codec: bytes

Longest prefix match with an exact-match table

Tenants own address ranges, not single addresses. So we need “which CIDR contains this IP”, and the most specific one wins. Vector’s CSV enrichment table can’t do that. It matches exactly, by wildcard, or by date range. No prefix match. ip_cidr_containstakes a list of CIDRs, but it only returns true or false, so it can’t say which tenant matched.

The trick: store each CIDR in canonical network form. Then, for each prefix length the table actually uses, longest first, cut the IP down to that length and do one exact lookup. The first hit is the longest match.

cidr,listener,tenant_slug,source_id,rate_eps
192.0.2.0/24,syslog,acme,acme-fw-01,5000
192.0.2.5/32,syslog,acme,acme-core-sw,500
192.0.2.0/24,json,acme,acme-app-01,5000
192.0.2.1/32,flow,acme,acme-edge-rtr,5000
198.51.100.0/24,syslog,globex,globex-dc-syslog,5000
198.51.100.0/24,json,globex,globex-json,5000
2001:db8:2::/48,syslog,globex,globex-v6-syslog,5000

tag = {"tenant": "quarantine", "kind": "syslog", "source_id": "unknown", "rate_eps": "q"}  # fail closed
ip = string(.source_ip) ?? ""
if is_ipv6(ip) { ip = ipv6_to_ipv4(ip) ?? ip }   # ::ffff:a.b.c.d from a [::] listener
tag.source_ip = ip
best = null
# The prefix lengths present in the table for this listener, longest first.
# Generate this list from the table. Don't hand-edit it.
lens = if is_ipv4(ip) { [32, 24] } else { [48] }
for_each(lens) -> |_i, p| {
  if best == null {
    net, err = ip_subnet(ip, "/" + to_string(p))
    if err == null {
      best = get_enrichment_table_record("net_sources",
        {"cidr": net + "/" + to_string(p), "listener": "syslog"}) ?? null
    }
  }
}
if best != null {
  tag.tenant = best.tenant_slug
  tag.source_id = best.source_id
  tag.rate_eps = best.rate_eps
}
claimed = del(.tenant_id)            # a claimed tenant is kept as evidence, never obeyed
if claimed != null { tag.claimed_tenant = claimed }
%sentinel = tag                      # the tag lives in metadata, set last

The cost grows with the number of distinct prefix lengths, not the number of rows. On one CPU with a 5,010-row table:

Both give the right answer. The loop is unusable past a few dozen rows. These numbers come from a shared development VM. Measure on your own hardware.

Keep the tag out of the body

The tag goes in event metadata, %sentinel, not in a document field. Route conditions, throttle keys and sink templates can all read it. It survives the disk buffer. And no field in the body can collide with it.

The index name comes straight from the tag. One shared sink writes every tenant:

sinks:
  opensearch:
    type: elasticsearch
    inputs: [seal_acme, seal_globex, seal_system, seal_quarantine, deadletter_acme]
    endpoints: [https://opensearch:9200]
    api_version: v8      # NOT auto, see below
    mode: bulk
    bulk:
      action: create
      index: 'sentinel-{{ %sentinel.tenant }}-{{ %sentinel.kind }}-%Y.%m.%d'
    request_retry_partial: true
    buffer:
      type: disk
      max_size: 1073741824
      when_full: block

Seal it after the tenant's own rules

Tenants want their own parsing rules. Rules are VRL, and any VRL can write %sentinel. So metadata alone isn’t tamper-proof. In our test, one hostile acme rule moved 55 of 55 acme events into globex.

The fix is the order of the graph:

tag -> throttle -> route by tenant -> that tenant's rules -> seal -> one shared sink

Routing happens beforethe tenant’s rules, so each rule only ever sees its own tenant’s events. The seal is a small remap the pipeline compiler owns, one per tenant branch. It writes the branch’s tenant back as a literal. With the seal in place, the same hostile rule left 55 of 55 events in acme.

%sentinel.tenant = "acme"
# kind becomes part of the index name, so only known values pass.
if !includes(["syslog", "cef", "json", "flow", "wazuh", "parsefail"], %sentinel.kind) {
  %sentinel.kind = "unknown"
}
# Clamp the index date to [now - 7d, now + 1d].
t, terr = timestamp(.timestamp)
age = if terr == null { to_unix_timestamp(now()) - to_unix_timestamp(t) } else { 0 }
if terr != null || age > 604800 || age < -86400 {
  if exists(.timestamp) { %sentinel.timestamp_raw = .timestamp }
  .timestamp = now()
}
del(.tenant_id)
.sentinel = compact({"tenant": "acme", "kind": %sentinel.kind,
                     "source_id": %sentinel.source_id, "source_ip": %sentinel.source_ip,
                     "claimed_tenant": %sentinel.claimed_tenant})

A rule that errors doesn’t lose the event. The rules remap uses drop_on_error, drop_on_abort and reroute_dropped, and the original, pre-rule event goes to that tenant’s parsefail index.

Fail closed

Every tag starts as quarantine. Only a match in a trusted table moves it. An unknown sender, an unmapped Wazuh agent or a line that won’t parse lands in sentinel-quarantine-*, which only operators can see. Nothing is dropped and nothing is guessed.

Unknown senders share one throttle bucket. Known sources get their own limits: one throttle transform per distinct limit value, keyed by tenant and source ID.

throttle_r500:
  type: throttle
  inputs: [by_limit.r500]
  threshold: 500
  window_secs: 1
  key_field: '{{ %sentinel.tenant }}/{{ %sentinel.source_id }}'
  internal_metrics:
    emit_events_discarded_per_key: true
throttle_quarantine:      # ONE shared bucket for every unknown sender
  type: throttle
  inputs: [by_limit._unmatched]
  threshold: 200
  window_secs: 1

In the same 10-second flood, the flooded acme firewall dropped 8,901 events and its neighbour on the same tenant, sending at its own limit, dropped none.

Wazuh: tag by agent ID

Wazuh alerts have no peer address worth using. They arrive from the manager’s alerts.json, and they carry no agent group. The group is where most people would look for the tenant. It isn’t there.

What is there is agent.id. The manager writes it, and it’s bound to the per-agent key. So the table is agent_id,tenant_slug:

agent_id,tenant_slug
001,acme
002,acme
003,globex

agent_id = to_string(body.agent.id) ?? ""   # written by the manager; never the labels
row = get_enrichment_table_record("wazuh_agents", {"agent_id": agent_id}) ?? null
if agent_id == "000" {
  tag.tenant = "system"                     # the manager itself: operator-only
} else if row != null {
  tag.tenant = row.tenant_slug
}

• Keep agent_id a string, so 001 keeps its leading zeros.
• Agent 000 is the manager itself. Its events go to an operator-only system index.
• If you ever run more than one manager, the key has to include the manager name.

Where the table comes from

The table is only as good as the enrolment behind it. Wazuh’s own enrolment service has one global password, and the enrolling host picks its own groups. In our test, a host holding the password enrolled into two tenants at once, just by asking.

So Sentinel owns enrolment. Its API checks a per-tenant token, creates the agent through the Wazuh API, puts it in that tenant’s group, writes its own enrolment record, and only then returns the agent key. The host never chooses its name, its ID or its group, and Wazuh’s enrolment port stays closed. The routing table is generated from those records, never from anything the agent reports. A background check quarantines any agent the manager knows about that has no matching record.

The Wazuh offline paper has the config that keeps port 1515 shut.

Change the table without losing events

Tenants come and go. Agents enrol all day. The table changes, and Vector has to pick it up without a restart. We measured reload loss over 40 instrumented runs.

So routine changes touch only tables and transforms. Sink and buffer changes are a maintenance job, done with OpenSearch healthy and the buffer empty.

# 1. Write the new table next to the live one, same filesystem.
# 2. Validate it with a small config that loads the new file as the table.
#    --skip-healthchecks loads the CSV; --no-environment does NOT.
vector validate --skip-healthchecks /etc/vector/check-table.yaml
# 3. Atomic rename onto the live path. Vector's --watch-config picks it up.
mv /var/lib/routing/.wazuh_agents.csv.tmp /var/lib/routing/wazuh_agents.csv

Don’t assume the reload happened. Check.

Vector 0.58 has no metric for an enrichment table reload. So the table carries a version row, and a heartbeat event every 10 seconds reports the version Vector has loaded, into an operator-only index. After each publish, the worker waits for a heartbeat that read the new version. If none comes within 60 seconds, that gets logged and audited. Deleting a tenant waits for that confirmation before it removes the tenant’s indices.

Test it like an attacker

vector test runs events through the transforms and asserts where they land. Every negative case belongs here: forged bodies, forged labels, rules that try to escape.

tests:
  - name: body tenant_id "globex" from an acme address stays acme
    inputs:
      - insert_at: tag_json
        type: log
        log_fields:
          source_ip: 192.0.2.20
          message: '{"tenant_id":"globex","sentinel":{"tenant":"globex"},"msg":"login ok"}'
    outputs:
      - extract_from: seal_acme
        conditions:
          - type: vrl
            source: |
              assert_eq!(.sentinel.tenant, "acme")
              assert_eq!(.sentinel.claimed_tenant, "globex")
              assert!(!exists(.tenant_id))
    no_outputs_from: [seal_globex, seal_quarantine]

  - name: wazuh agent 003 routes to globex whatever its labels say
    inputs:
      - insert_at: tag_wazuh
        type: log
        log_fields:
          message: '{"agent":{"id":"003","name":"web1","labels":{"tenant":"acme"}},"tenant_id":"acme"}'
    outputs:
      - extract_from: seal_globex
        conditions:
          - type: vrl
            source: assert_eq!(.sentinel.tenant, "globex")
    no_outputs_from: [seal_acme, seal_quarantine, seal_system]

vector test /etc/vector/vector.yaml /etc/vector/tests.yaml

The prototype’s suite covered 13 cases: acme to acme, longest prefix wins, unknown to quarantine only, forged body tenant, undecodable JSON, a rule that rewrites tenant, kind and date, a rule that wipes the event, a failing rule going to the dead letter path, Wazuh by agent ID, agent 000 and an unknown agent, a flow record, an IPv6 sender and an IPv4-mapped peer.

What this does not stop

• UDP source spoofing. Raw packets from an unknown host with a forged globex source address landed in globex, 5 of 5. Peer address is only as good as the transport. Use TCP or TLS syslog where the device supports it. We record this as an accepted risk, not a solved one. The sFlow agent address is the same story.
• A leaked enrolment token.Whoever holds it can create agents in that tenant and send forged content there. Routing can’t tell. Token lifetime, use limits, a per-tenant agent quota and an audit of every enrolment are the controls.
• Tenant rules that reach out. VRL is not side-effect free. http_request, dns_lookup, get_env_var and the metrics and table readers all pass vector validate. A live test rule fetched Vector’s own metrics, every tenant’s included. And SECRET[...] is expanded inside VRL strings: a test rule wrote the OpenSearch password into its own index. Our plan: an allowlist of VRL functions, a sandboxed test run with no network, and no secrets in the Vector config at all. Authenticate to OpenSearch with a client certificate. The prototype of the allowlist and sandbox worked. They aren’t built yet.
• A rule that eats the box.A 33-line rule gated on one magic word OOM-killed the whole Vector container the moment a device sent that word. Every tenant lost ingest. Nothing static catches that. Our plan for shared setups is that a tenant’s rules wait for an operator’s approval before they go live.
• Kernel UDP drops. Vector never sees them. Watch node_netstat_Udp_RcvbufErrors from node_exporter.

Where Sentinel is

Sentinel is in development and not released. Here’s what is built and what isn’t.

• Built:Wazuh events routed by agent ID from Sentinel’s own enrolment records, quarantine for unknown agents, the table publish with validation and heartbeat confirmation, and the pipeline compiler with per-tenant routes, seals and dead letters.
• Proven in the prototype, not built yet: peer-address tagging for syslog, JSON and flow sources, per-source rate limits, and the rule allowlist and sandbox.

The numbers in this paper come from a shared 4-vCPU development VM. Sizing on real hardware is still to come.

Do this yourself

Credits and sources

• Vector documentation. Vector is MPL-2.0. We used 0.58.0.
• VRL reference
• GoFlow2 for NetFlow, IPFIX and sFlow.
• Wazuh documentation and OpenSearch.

Tenant names in this paper are made up. Addresses are documentation ranges.