Fire - Notice history

All systems operational

Bot - Operational

100% - uptime
Oct 2020 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2020
Nov 2020
Dec 2020

Website - Operational

100% - uptime
Oct 2020 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2020
Nov 2020
Dec 2020

Notice history

Dec 2020

Potential API Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    It seems the error rate has decreased to normal levels. Keeping this open a bit longer to be sure

  • Investigating
    Investigating

    It seems Fire is experiencing an increased level of 500 errors from Discord. I will keep this incident updated with any updates if issues continue

Google Outages
  • Resolved
    Resolved

    Discord have marked the issue as resolved and Fire has not seen any issues recently.

  • Update
    Update

    Related Incident: https://status.cloud.google.com/incident/zall/20013

  • Update
    Update

    Both Fire & Fire TS are online but may have limited functionality due to ongoing issues with Discord

  • Update
    Update

    Fire & Fire TS are still unable to connect to Discord. Google have marked all of their components as operational and Discord has marked the API as operational (when it's clearly not) so I'm unsure how much longer this downtime will last

  • Update
    Update

    Discord have opened an incident on their status page. You can find it @ https://discordstatus.com/incidents/9hcyj1v9qft9 You can also check out Google's status page @ https://www.google.com/appsstatus#hl=en&v=status

  • Update
    Update

    It seems Google's services are starting to recover. I am still attempting to get both bots to connect but it may take a while...

  • Identified
    Identified

    Fire TS has unfortunately been disconnected and is currently struggling to reconnect it's shards. I've also updated this to identified since everyone and their mother knows Google is having issues...

  • Update
    Update

    Fire has been disconnected from Discord and is attempting but struggling to reconnect.

  • Investigating
    Investigating

    It seems Google is having a major outage with a ton of it's services down which Fire directly & indirectly rely on (YouTube for filters, Google Assistant for the google command and most importantly, Discord is hosted on GCP) meaning Fire may experience issues.

Potential DDoS Attack
  • Resolved
    Resolved

    I have now disabled under attack mode and deployed an update to Vanity URLs to try prevent this from happening in the future. Traffic levels have returned to normal so I'm marking this as resolved.

  • Monitoring
    Monitoring

    Services have started recovering. Under attack mode will stay enabled until I am sure the attack has stopped.

  • Investigating
    Investigating

    It seems Fire's services are getting hit with a DDoS attack (3m+ requests in the last hour) and it seems to be affecting some services. I've activated Cloudflare's under attack mode on the domains that are being hit with the most requests so you may see Cloudflare checking your browser while attempting to visit some pages. I will attempt to restore services that have been affected but it may require a reboot (VPS seems to have run out of memory)

Leave messages & logging not working
  • Resolved
    Resolved

    The fix, along with a fix for the fix (forgot to put await) has been deployed and leave messages & logging appear to be working. I've had to remove stayed for, nickname and roles from leave logging due to not having access to that data. I'm unsure on whether or not these will ever return since I don't want Fire caching a ton of members.

  • Update
    Update

    The fix is being deployed now. Fire will go offline for a few seconds while it restarts

  • Identified
    Identified

    It seems that the lack of cache has caused leave messages & logging to stop working. I have identified a possible fix and am working on implementing it now.

Elevated latency & error rates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Discord have implemented a fix and the latency has decreased to normal levels. I am monitoring the situation to ensure everything is resolved

  • Identified
    Identified

    It seems Fire is experiencing increased latency and error rates. This is due to issues with Discord which you can find more information on at https://discordstatus.com/incidents/h22zw3cshbz7

Nov 2020

Elevated error rates
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    The amount of errors seems to have calmed down and Fire is once again able to send messages. Will keep this open until I get confirmation the issues are resolved

  • Identified
    Identified

    It seems Discord is dying and Fire has started crying about it so I'm gonna open an incident here so I can easily leave updates here. It doesn't seem like Discord's status page has been updated to reflect the seemingly major outage (I'm getting 500s on almost every request, https://i.imgur.com/q012H27.png)

Services Unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Identified
    Identified

    Services are slowly but surely automatically recovering after the reboot. If you encounter any errors, please keep this in mind. Components will be set to operational once I am sure they're fully recovered

  • Update
    Update

    The issues seem to have gone away but I am going to do a full reboot of the vps to be safe.

  • Investigating
    Investigating

    It seems my VPS is currently having some issues causing all services to appear slow or offline. I am working on getting everything restored.

Oct 2020

Fire leaving guilds for no apparent reason
  • Resolved
    Resolved

    This incident has been open for a few days, marking as resolved

  • Identified
    Identified

    so uh, turns out I am a dumbass and due to my lack of braincells I am the reason why Fire left all of those guilds basically, I ran the TypeScript rewrite of Fire on the main token to check resource usage with all 631 guilds and since the hosted instance of the rewrite is for premium guilds only, I had "premiumOnly" set to true meaning it left guilds that weren't premium. It thankfully didn't get down to just premium guilds but boy am I pissed that I didn't realize what happened sooner. I sincerely apologize for my stupidity. To make it up to y'all, I will make Fire TS available for non-premium guilds too as soon as I can get this update deployed correctly. This incident will stay open for another little while to allow people to see the update.

  • Investigating
    Investigating

    I am opening an incident for this so that people can hopefully find it if Fire has left their guild and reinvite it. Updates will be posted here and in Fire's Discord ( https://inv.wtf/fire ); ----- So uh, not the kind of announcement I'm wanting to make but it seems somehow Fire has left most of it's guilds. It went from 639 to 151. Around the time it happened, Fire got disconnected from the gateway so I'm hoping that it's just a bug and the guilds it lost will return but if Fire has left your guild I would strongly recommend just reinviting it from https://inv.wtf/bot just in case it doesn't magically return. I have also reset Fire's token in case it somehow got leaked, which is very unlikely as Fire has never had a token leak. I apologize for any inconvenience this may have caused. I know a lot of you rely on Fire for managing your server e.g. logging or moderation. I will be investigating the cause for this and will provide new information as soon as I find anything. Once again I am deeply sorry for any inconvenience caused.

Members not being cached
  • Update
    Update

    The root cause behind the issue was the deployment of Fire’s TypeScript rewrite for premium guilds. The way the rewrite had been saving configs added extra quotes around the config causing normal Fire to be unable to parse the JSON. While trying to reload a config for a premium guild, it panicked due to being unable to parse it and rebooted and because it rebooted, it had to load all the configs again bringing everything back to the issue causing it to hit an error and stop. The fix for the issue was to make the TypeScript rewrite INSERT/UPDATE configs as is, no JSON.stringify and it seems to have worked but there’s still gonna be _some issues_ with consistency due to both Python & TypeScript bots running on the same database causing configs to become desynced but that’s only for the few premium guilds and it’s just meant to be a beta test anyways so any inconsistencies are fine.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been deployed and I am now monitoring the situation

  • Identified
    Identified

    The issue has been identified and a fix is being made

  • Update
    Update

    A temporary patch has been put in place to allow configs to load. It is still unknown why they are failing to be parsed when loading.

  • Update
    Update

    It seems configs failed to be parsed and were left as strings causing a bunch of other tasks to fail. I am investigating the cause.

  • Investigating
    Investigating

    It seems Fire ran into an issue with configs, rebooted and because of said issue, members are not being cached in guilds where they should be. This will affect logging, autodecancer/autodehoist and more

Oct 2020 to Dec 2020

Next