Health you can trust: nothing shows green unless a live check passed

The green lie

Every monitoring tool has a panel of green dots. The question nobody asks is what the green means.

Too often it means “configured”. Or “it was fine the last time we looked”. Or just “nothing has said otherwise”. A container that is running but refusing logins shows green. A database that went read-only an hour ago shows green until someone restarts the poller. An install wizard that ticked a box once shows that tick forever.

On a box that watches your network, a false green is worse than a red. Red gets fixed. False green gets trusted.

So Sentinel, the NOC and SOC appliance we’re building, has one rule for its stack health. Nothing shows up unless a live check just passed.

Three states, one rule

There is no fourth state. No “unknown”, no “pending”, no grey. If we can’t prove it’s up, it’s down, and we say why.

Real calls, short timeouts, fresh results

Each component gets a real call, not a port probe and not a container status. The rules:

• Every check is live. A real request to the component, every time the result is refreshed.
• Every check times out after 3 seconds.A component that takes longer is down for the purpose of this panel. It is not “still checking”.
• Results are at most 15 seconds old.Cache them so a busy panel can’t hammer the stack. Never serve one older than that.

These are the ten parts Sentinel checks today:

No stored “configured” flag

Sentinel stores no “setup complete” flag. The setup status is computed from live state every time it’s asked for. The setup wizard’s health step reads the same live results as the panel.

One set of results, three audiences

The same checks feed everything. What changes is how much each reader is allowed to see.

Feeding the results into VictoriaMetrics means the stack’s own health sits next to the network metrics it collects. The box watches itself with the same tools it uses to watch everything else.

A runbook per component

A red dot with no next step just makes people anxious. Every component in the panel links to its own fix steps. They all start the same way, on the host, as root, from the compose directory:

docker compose ps                  # is the container running at all?
docker compose logs <service>      # what did it say before it stopped?
sentinel check                     # host problems: free disk, vm.max_map_count
sentinel install <bundle>          # re-run the install to repair

Re-running the install with the same bundle is the repair. It recreates missing containers and rendered config, and restarts the local registry. It never removes data volumes, and it doesn’t regenerate secrets or the internal CA. On a healthy stack it changes nothing. Make your own repair path that boring and you’ll use it.

Check the dependency first

• Vector down or stuck?Check OpenSearch first. Vector can’t write while OpenSearch is down.
• vmalert down? Check VictoriaMetrics first. vmalert needs it to evaluate anything.
• GoFlow2 up, but no flow data? GoFlow2 writes to a file that Vector reads. Check Vector.
• Wazuh manager down with an authentication reason? The credentials Sentinel generated no longer match the manager.
• OpenSearch degraded? Usually a yellow cluster. Run sentinel check, which looks at free disk and vm.max_map_count.

Trust the checker too

A health check is only as honest as the thing answering it. We know where ours are weak, so here it is.

The same idea runs through the rest of the build. When Sentinel publishes a new routing table to Vector, it doesn’t assume Vector loaded it. It waits for a heartbeat that proves the new version is live. That’s in the tenant isolation paper.

Build it yourself

This is a minimal version of the pattern, written for this paper. Standard library only. Swap in your own components, and your CA and credentials where they’re needed.

import json
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor, wait

TIMEOUT = 3.0   # slower than this is down
MAX_AGE = 15.0  # never serve a result older than this


class Degraded(Exception):
    """The component answered and reported a partial problem itself."""


def get(url):
    with urllib.request.urlopen(url, timeout=TIMEOUT) as r:
        return r.read()


def opensearch():
    status = json.loads(get("https://opensearch.example:9200/_cluster/health"))["status"]
    if status == "yellow":
        raise Degraded("cluster status yellow")
    if status != "green":
        raise RuntimeError(f"cluster status {status}")


CHECKS = {
    "opensearch": opensearch,
    "victoriametrics": lambda: get("http://victoriametrics.example:8428/health"),
    "blackbox-exporter": lambda: get("http://blackbox.example:9115/-/healthy"),
}


def check(name):
    try:
        CHECKS[name]()
        return {"name": name, "status": "up", "reason": ""}
    except Degraded as e:
        return {"name": name, "status": "degraded", "reason": str(e)}
    except Exception as e:  # refused, timed out, 401: all down
        return {"name": name, "status": "down", "reason": str(e) or type(e).__name__}


_pool = ThreadPoolExecutor(max_workers=3 * len(CHECKS))
_cache = (float("-inf"), [])


def check_all():
    futures = {n: _pool.submit(check, n) for n in CHECKS}
    wait(futures.values(), timeout=TIMEOUT)
    late = {"status": "down", "reason": f"timed out after {TIMEOUT:g} s"}
    return [f.result() if f.done() else {"name": n, **late} for n, f in futures.items()]


def health():
    """Every component, from checks that started at most MAX_AGE seconds ago."""
    global _cache
    started, result = _cache
    if time.monotonic() - started >= MAX_AGE:
        started = time.monotonic()
        result = check_all()
        _cache = (started, result)
    return result


def overall(result):
    return "down" if any(c["status"] == "down" for c in result) else "up"


if __name__ == "__main__":
    r = health()
    for c in r:
        print(f'{c["name"]:20} {c["status"]:9} {c["reason"]}')
    print("overall:", overall(r))

Run it as written, against placeholder hosts, and every line comes back down with the error as its reason. That’s the right answer. Nothing answered, so nothing is up.

Do this yourself

Sentinel is in development and not released. The health checks described here are built and running in the development build. Alerting on these results is not built yet.