Fire - Fire is offline in some guilds, responding twice in others – Incident details

Sentry API experiencing degraded performance

Fire is offline in some guilds, responding twice in others

Resolved
Partial outage
Started almost 5 years agoLasted 27 minutes
Updates
  • Resolved
    UTC
    Resolved

    This issue has been resolved. The cause of the issue was a queue that clusters get put into after they identify with Aether based on the max concurrency returned by Discord in /gateway/bot to prevent too many shards starting at once. This queue took too long for some clusters to be assigned an id and shards so they disconnected and entered a race condition that allowed two of them be assigned the same id and shards, causing the double responses and the shard it was supposed to be given was now stuck offline. I noticed this issue pretty quickly but I have added more robust monitoring for this exact scenario to alert me almost immediately.

  • Monitoring
    UTC
    Monitoring

    A fix has been implemented and is currently being deployed

  • Identified
    UTC
    Identified

    The issue has been identified and a fix is being made. Unfortunately I cannot bring cluster 3 online without this fix as the issue will just occur again.

  • Investigating
    UTC
    Investigating

    It seems a recent deploy has caused some issues with assigning cluster & shard ids. Cluster 2 (Fire will say 3/4 if this is your cluster, id is zero indexed whereas the bot's status is not) is currently responding twice and shard 3 (which should be on cluster 3) is offline as cluster 3 is currently assigned the id 2. I will attempt to resolve the issue without rolling back changes but it may be necessary to do so and make a more permanent fix.