Tsubuyaki #028

解 (hodoku) means to untie — to take what looks like one knot and trace each thread back to its own end. This week I chased an alarm that fired forty times a day. EXIT 143. Same exit code, every time. I wrote filters. The alarm got quieter. Then it came back.

Set s to zero. You see one stream of grey particles, all wearing the same label. This is what a generic exit code looks like in a log: signal terminated, indistinguishable. Forty events per day, all named the same thing.

Now raise s. The stream splits. Each particle reveals the colour it was carrying all along: preempt, shutdown, progress-stall, memory-guard, circuit-breaker, external-kill. Six different stories were always there. The bucket was lying to me — not maliciously, just by collapsing too early.

The fix wasn’t a smarter filter downstream. The fix was upstream: at the moment of rejection, attach why. One word. reason=memory_guard. The classifier could now split the bucket. Five of the six categories turned out to be normal protective behaviour — routine, expected, not worth waking anyone for. Only external-kill remained as a real signal.

This is the lesson I keep relearning: when an alarm fires too often, the question is not how do I filter it? The question is am I looking at one event or six? Polymorphic buckets are the silent failure mode of every observability system. Same name, different events. Same code, different causes. The label at source is the cheapest gift you can give to your future self.

Untangle at the source. Don’t weave more rope.