Fire - Potential disruption to Fire (Automatic) – Incident details

All systems operational

Potential disruption to Fire (Automatic)

Resolved
Major outage
Started 1 day agoLasted about 2 hours

Affected

Bot

Operational from 2:55 AM to 2:55 AM, Partial outage from 2:55 AM to 2:57 AM, Major outage from 2:57 AM to 3:51 AM, Operational from 3:51 AM to 4:53 AM

Website

Operational from 2:55 AM to 2:57 AM, Partial outage from 2:57 AM to 3:51 AM, Operational from 3:51 AM to 4:53 AM

Backend

Operational from 2:55 AM to 2:57 AM, Partial outage from 2:57 AM to 3:51 AM, Operational from 3:51 AM to 4:53 AM

Aether

Operational from 2:55 AM to 2:57 AM, Partial outage from 2:57 AM to 3:51 AM, Operational from 3:51 AM to 4:53 AM

Updates
  • Resolved
    Resolved

    The issue seems to have stemmed from Fire being stuck in a loop, constantly making requests to edit a channel, which resulted in it getting ratelimited. As far as I can tell, the edits were part of the permission role feature, so I've made a few changes to it which should a) help identify it and b) help resolve the issue

    This feature is pretty old so it likely needs a rewrite to ensure things work smoothly, which has been added to my lengthy to-do list for Fire

    For now, Fire is operational and should stay that way. If you encounter any issues, please report them in the Fire Discord (discord.gg/firebot)

  • Monitoring
    Monitoring

    Fire is now back online and looks to be operating just fine. I've made some changes that may help prevent the issue re-occurring but I'll be looking further into what caused this issue so that I can properly resolve it

  • Update
    Update

    Fire's VPS appears to have been blocked from accessing Discord's API due to exceeding ratelimits. It's unknown how this occurred at this time but for now, the bot has been turned off to avoid making requests while ratelimited.

    It looks to be only an hour long block with about 50 minutes remaining so it shouldn't be down for long but unfortunately there's nothing that can be done other than waiting

  • Investigating
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable