← Back to News
Incident / Postmortem

Outage Report: outpost.sonofatech.com and outpost-chat.com

Both the reference Outpost instance and this marketing/community site were unreachable for about 50 minutes overnight on 2026-08-05. Here's exactly what happened, why, and what's changing as a result.

Share on X

What happened

Between 23:59 UTC on 2026-08-04 and 00:48 UTC on 2026-08-05 (about 49 minutes), both outpost.sonofatech.com (the reference Outpost instance) and outpost-chat.com (this site) were unreachable or unstable. The underlying server went through a rapid cycle of reboots during that window before stabilizing.

Root cause

We were in the process of setting up a new internal monitoring integration for the server, which required a new VPN tunnel to reach our monitoring system. The client configuration used for that tunnel was scoped as a full-tunnel connection rather than one limited to just the monitoring traffic it actually needed. Once activated, the server's own routing rules gave that tunnel priority over all outbound traffic — not just traffic meant for the new peer — including DNS lookups, database/storage connections, and ordinary internet access. The tunnel's other end was never actually reachable, so every bit of traffic routed into it simply vanished.

That broke everything downstream, not just DNS. With outbound connectivity gone, lookups the app depends on at runtime (its object storage endpoint, some database-layer tooling) started timing out, and the app's own voice-service component began crash-looping trying to reach the same unreachable destinations — all cascading from one routing misconfiguration, not a bug in the app itself.

How it was fixed

The misconfigured tunnel was identified and torn down, which also removed the routing rules it had installed; normal connectivity returned immediately. Every service on the host was verified healthy individually afterward — including manually triggering a real backup job to confirm it completed end-to-end — before considering it resolved, not just "the site loads again."

Was any data lost?

No. This was a networking/connectivity problem, not a data problem — nothing was written incorrectly, corrupted, or rolled back. Automated backups of the server continued running on their normal schedule throughout, unaffected by this incident either way.

What's changing

This server doesn't currently have a separate staging environment — infrastructure changes are made directly against it. That's fine for most changes, but this incident is a clear example of why anything that can affect networking or DNS specifically needs a safer rollout path. Going forward, changes in that category get validated in isolation — confirming DNS and outbound connectivity still work correctly — before being applied to the production host, rather than assuming a new integration is safe because it looked correct on paper.

Thanks for your patience — questions or reports always welcome via Support.