Discord outage causing instability
Incident Report for Fire
Resolved
An outage on Discord's end resulted in many servers becoming unavailable for long periods of time and even coming back & going again multiple times. Based on internal monitoring, the incident started affecting Fire at around 00:03 UTC and resolved by 01:00 with a full reboot taking place at 01:07-01:08

Fire experienced some instability due to the sudden surge of GUILD_DELETE (i.e. a server going unavailable) & GUILD_CREATE (i.e. a server becoming available again) events. These events almost never come en masse and functions I've designed around these were not intended to handle so much back and forth happening at once.

The bot (both beta & production) appear to have recovered automatically as intended once the issues on Discord's end subsided but I performed a full reboot of Aether & Fire just to ensure their internal states were in order along with clearing their logs as they got spammed a lot during the outage
Posted Dec 16, 2023 - 00:00 UTC