Fire - Moderation/Action/Member logs not being sent – Incident details

All systems operational

Moderation/Action/Member logs not being sent

Resolved
Partial outage
Started about 4 years agoLasted 42 minutes
Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Fix has been deployed to both clusters and the log queue should've been recovered for guilds on Cluster 1 though it may have needed to attempt another log to actually send the queue. I'm monitoring logging in one of the larger guilds Fire is in to ensure there's no lingering issues.

  • Update
    Update

    After deploying the fix and seeing it not work, I noticed I had forgot to add some changes to the commit so it was only half fixed. Full fix being pushed and deployed now 😅

  • Update
    Update

    The fix has been pushed and is ready to deploy. Due to logging being broken on Cluster 0 after the failed fix, I will not be able to recover items currently in the queue. As for Cluster 1, I should be able to recover the queue once the fix has deployed to Cluster 0

  • Identified
    Identified

    The issue has been identified and a fix is being worked on

  • Investigating
    Investigating

    Seems the fix just broke it even more, back to the drawing board I go...

  • Identified
    Identified

    A potential fix has been made and pushed. I will now attempt to send any messages currently in the queue for all guilds on Cluster 0 and then deploy the update.

  • Investigating
    Investigating

    It seems the most recent deployment to try improve Fire's log queue system has caused logs to get stuck in the queue and not sending. This queue system is to prevent Fire from getting ratelimited by sending too many webhook messages, so it bundles logs when they're going too fast but it seems logs are not getting past the queue. I will be attempting to resolve the issue and get a fix pushed. I should be able to recover logs that are currently stuck in the queue to prevent missed logs