Fire - Notice history

All systems operational

Bot - Operational

100% - uptime
Sep 2020 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2020
Oct 2020
Nov 2020

Website - Operational

100% - uptime
Sep 2020 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2020
Oct 2020
Nov 2020

Notice history

Nov 2020

Elevated error rates
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    The amount of errors seems to have calmed down and Fire is once again able to send messages. Will keep this open until I get confirmation the issues are resolved

  • Identified
    Identified

    It seems Discord is dying and Fire has started crying about it so I'm gonna open an incident here so I can easily leave updates here. It doesn't seem like Discord's status page has been updated to reflect the seemingly major outage (I'm getting 500s on almost every request, https://i.imgur.com/q012H27.png)

Services Unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Identified
    Identified

    Services are slowly but surely automatically recovering after the reboot. If you encounter any errors, please keep this in mind. Components will be set to operational once I am sure they're fully recovered

  • Update
    Update

    The issues seem to have gone away but I am going to do a full reboot of the vps to be safe.

  • Investigating
    Investigating

    It seems my VPS is currently having some issues causing all services to appear slow or offline. I am working on getting everything restored.

Oct 2020

Fire leaving guilds for no apparent reason
  • Resolved
    Resolved

    This incident has been open for a few days, marking as resolved

  • Identified
    Identified

    so uh, turns out I am a dumbass and due to my lack of braincells I am the reason why Fire left all of those guilds basically, I ran the TypeScript rewrite of Fire on the main token to check resource usage with all 631 guilds and since the hosted instance of the rewrite is for premium guilds only, I had "premiumOnly" set to true meaning it left guilds that weren't premium. It thankfully didn't get down to just premium guilds but boy am I pissed that I didn't realize what happened sooner. I sincerely apologize for my stupidity. To make it up to y'all, I will make Fire TS available for non-premium guilds too as soon as I can get this update deployed correctly. This incident will stay open for another little while to allow people to see the update.

  • Investigating
    Investigating

    I am opening an incident for this so that people can hopefully find it if Fire has left their guild and reinvite it. Updates will be posted here and in Fire's Discord ( https://inv.wtf/fire ); ----- So uh, not the kind of announcement I'm wanting to make but it seems somehow Fire has left most of it's guilds. It went from 639 to 151. Around the time it happened, Fire got disconnected from the gateway so I'm hoping that it's just a bug and the guilds it lost will return but if Fire has left your guild I would strongly recommend just reinviting it from https://inv.wtf/bot just in case it doesn't magically return. I have also reset Fire's token in case it somehow got leaked, which is very unlikely as Fire has never had a token leak. I apologize for any inconvenience this may have caused. I know a lot of you rely on Fire for managing your server e.g. logging or moderation. I will be investigating the cause for this and will provide new information as soon as I find anything. Once again I am deeply sorry for any inconvenience caused.

Members not being cached
  • Update
    Update

    The root cause behind the issue was the deployment of Fire’s TypeScript rewrite for premium guilds. The way the rewrite had been saving configs added extra quotes around the config causing normal Fire to be unable to parse the JSON. While trying to reload a config for a premium guild, it panicked due to being unable to parse it and rebooted and because it rebooted, it had to load all the configs again bringing everything back to the issue causing it to hit an error and stop. The fix for the issue was to make the TypeScript rewrite INSERT/UPDATE configs as is, no JSON.stringify and it seems to have worked but there’s still gonna be _some issues_ with consistency due to both Python & TypeScript bots running on the same database causing configs to become desynced but that’s only for the few premium guilds and it’s just meant to be a beta test anyways so any inconsistencies are fine.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been deployed and I am now monitoring the situation

  • Identified
    Identified

    The issue has been identified and a fix is being made

  • Update
    Update

    A temporary patch has been put in place to allow configs to load. It is still unknown why they are failing to be parsed when loading.

  • Update
    Update

    It seems configs failed to be parsed and were left as strings causing a bunch of other tasks to fail. I am investigating the cause.

  • Investigating
    Investigating

    It seems Fire ran into an issue with configs, rebooted and because of said issue, members are not being cached in guilds where they should be. This will affect logging, autodecancer/autodehoist and more

Sep 2020

GalaxyGate Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    It seems GalaxyGate (a hosting provider) is having issues right now and some services Fire uses are hosted there meaning it impacts Fire too. KSoft.Si features (memes, lyrics, global bans, chatwatch) are unavailable at this current point in time. There may be other features impacted but these are the only ones I am aware of.

KSoft.Si Maintenance
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    KSoft.Si are currently undergoing maintenance therefore commands/features that rely on KSoft (lyrics, meme, global bans) will fail. They expect maintenance to last 15 minutes.

Discord API Issues
  • Resolved
    Resolved

    There has been no further errors since so I am marking this as resolved

  • Monitoring
    Monitoring

    It seems Fire is no longer getting 502's when sending messages meaning the issue was most likely linked to Discord's ongoing incident as the error rate calmed down after the most recent update. I am continuing to monitor for any more errors

  • Investigating
    Investigating

    It seems Fire is having trouble sending some messages, particularly but not exclusively those with images. This may or may not be related to Discord's ongoing incident but the API is still marked as operational.

Sep 2020 to Nov 2020

Next