I JUST WANTED TO… GRAFUNA RED TEAM

Observability is about visibility.

Visibility works both ways. If you can see it, someone else can too.

This post is the polite version of a talk I gave. The impolite version is the repo.

https://github.com/scadastrangelove/zeronights2025-GRAFUNA 

Cloud observability is everywhere: dashboards, metrics, traces, logs — the modern comfort blanket. It’s powerful, it’s convenient, and it tends to ship with defaults that assume your network is a cozy village with a single locksmith and no wolves.

The paradox is simple: we built monitoring to reduce incidents, and accidentally created a new attack surface to route them. Not because observability is evil. Because it’s useful. And “useful” is a feature that spreads.

You don’t need a villain arc for this. You just need:

• one “temporary” exception,

• one “internal-only” endpoint,

• one token that lives a little too long,

• and a belief that “nobody will look there”.

Someone will look there. That’s the whole point of observability: looking.

Grafana (and friends) are not just pretty charts. They are a control plane: people, roles, data sources, plugins, and a long tail of “helper” features. Most of the time they make your life better. Sometimes they make someone else’s life easier.

And if you observe long enough, observability will gaze back into you.
(And it will file a ticket.)