Fire - Notice history

All systems operational

Bot - Operational

100% - uptime
Dec 2020 · 100.0%Jan 2021 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2020
Jan 2021
Feb 2021

Website - Operational

100% - uptime
Dec 2020 · 100.0%Jan 2021 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2020
Jan 2021
Feb 2021

Notice history

Feb 2021

Mute/Ban/Reminder times not being parsed correctly
  • Resolved
    Resolved

    I had just recently been made aware of an issue with commands that use Fire's parseTime function (reminders & temp mutes/bans) not parsing times correctly if the time was at the start of the reason. This would result in the time being parsed as 0 minutes which resulted in errors while creating reminders (saying you can't set reminders for less than 2 minutes) and permanent mutes/bans rather than temporary. This issue has been resolved as of 13:32 UTC and Fire should start parsing times correctly. Note: If the time was NOT at the start of the reason, they were unaffected by this bug and therefore were being parsed correctly

Some Web Services Unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    The issue has been identified (it was a comma in the config that shouldn't be there but somehow worked in the old NGINX version!?) and has been fixed. I am monitoring to ensure this issue does not occur on other pages (it seems Aether's API was the only one affected) and will update this incident as I check all pages.

  • Investigating
    Investigating

    While work was being done on improving Fire's rate limiting, an NGINX upgrade was required so that I could make use of new features to improve it. This upgrade did not go as planned and some pages may display an NGINX error or "Welcome to nginx" The cause of these issues is unknown as all configs are still intact and most pages seem to be working fine. Known affected pages are: - Aether Rest API - Fire Website (relies on above)

Discord CDN Issues
  • Resolved
    Resolved

    Seems to be fully resolved. If any more issues occur, another incident will be opened (provided I am awake)

  • Monitoring
    Monitoring

    Discord has claimed the issue is resolved and their engineering team are monitoring the situation.

  • Identified
    Identified

    Discord has opened an incident, https://discordstatus.com/incidents/4ylv6l1g10v6 500s are occurring on almost all requests, no matter the endpoint. https://i.imgur.com/vUaYxpQ.png

  • Update
    Update

    API is kicking the bucket real hard... https://media.discordapp.net/attachments/804820966545948702/805967529415409686/graph.png Changing to major outage

  • Investigating
    Investigating

    Seems Discord's CDN has died completely, meaning any commands that upload files will fail. Discord have not yet said anything about the issue but if/when they open an incident, I'll post it here.

Jan 2021

Sporadic Restarts
  • Resolved
    Resolved

    Everything seems fine, just deployed an update so it's a good time to mark this as resolved.

  • Monitoring
    Monitoring

    Of course right as I go to investigate, the issue stops occurring. If it happens again, I will move this back to investigating but for now everything seems fine. I'll be rebooting cluster 1 to remove some debug logging I added which should only take a few seconds

  • Investigating
    Investigating

    Cluster 1 is currently having issues where seemingly out of nowhere, it will crash and sometimes fail to recover. I am currently investigating the cause for these crashes and will update this incident with information as I find it (because right now I am clueless). If you want to find out what cluster your server is on, you can enter your server id at https://fire.gaminggeek.dev/stats and your cluster will start flashing.

Services unavailable
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    I am monitoring to ensure everything has fully recovered.

  • Update
    Update

    Fire's VPS is no longer ratelimted, working on restoring services to full capacity now

  • Update
    Update

    Cluster 1 is online, meaning the bot should basically be operational apart from db changes not saving once it goes back to running on the VPS. Due to issues connecting to the remote database, I will not be able to bring up a local instance of inv.wtf meaning Vanity URLs will still be broken.

  • Update
    Update

    Cluster 0 is running locally with seemingly no issues. Beware, any changes that involve the database will NOT BE SAVED due to it being a local copy of the database. I will work on getting cluster 1 online now.

  • Update
    Update

    Attempts to bring Fire online locally have not worked due to issues with my local database and connecting to the prod database remotely. I will continue to try bring Fire back online locally but it may come down to just waiting out the ratelimit (there's about 25 minutes left)

  • Identified
    Identified

    Upon further investigation, it seems Fire's VPS has gotten hit with a 1 hour ban from the API, which means this affects more than just the bot. I am working on attempting to bring up instances of affected services locally to minimise the downtime.

  • Investigating
    Investigating

    Seems Fire has gone unresponsive. I am investigating the cause as we speak. Cluster 0 seems to be down completely but cluster 1 seems to at least be receiving gateway events.

New Guilds Not Being Cached
  • Resolved
    Resolved

    Since around 1:00 UTC, new guilds were not being cached by Fire, causing a number of issues with the biggest being the bot not working in new guilds until a restart. This was due to me adding a listener for GUILD_CREATE on Fire's websocket manager without knowing it prevented the normal listener from running. This issue was not apparent when testing as I have no need to invite my dev bot to a new guild. If you recently invited Fire and it appeared to not work, this would be why and has since been fixed. I apologise for the inconvenience.

Fire Offline
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    The commits have been reverted and Fire is back online in it's normal state. I am monitoring to ensure that no leftover yarn v2 issues occur

  • Update
    Update

    Fire is online though I'm not sure how? Seems pm2 is really not a fan of yarn v2 so I am attempting to revert

  • Identified
    Identified

    Fire has gone offline due to some issues when deploying the latest commit, which upgraded to yarn v2. It seems pm2 doesn't like attempting to run "yarn node" causing it to fail to start. I will attempt to fix the issue as quickly as possible but I may have to revert.

Elevated latency
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    Fire is experiencing elevated latency. This is due to a Discord issue which you can find updates for at https://stspg.io/h7b0gxgsgsmd?u=wsnwrnyfg96g

Dec 2020

Potential API Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    It seems the error rate has decreased to normal levels. Keeping this open a bit longer to be sure

  • Investigating
    Investigating

    It seems Fire is experiencing an increased level of 500 errors from Discord. I will keep this incident updated with any updates if issues continue

Google Outages
  • Resolved
    Resolved

    Discord have marked the issue as resolved and Fire has not seen any issues recently.

  • Update
    Update

    Related Incident: https://status.cloud.google.com/incident/zall/20013

  • Update
    Update

    Both Fire & Fire TS are online but may have limited functionality due to ongoing issues with Discord

  • Update
    Update

    Fire & Fire TS are still unable to connect to Discord. Google have marked all of their components as operational and Discord has marked the API as operational (when it's clearly not) so I'm unsure how much longer this downtime will last

  • Update
    Update

    Discord have opened an incident on their status page. You can find it @ https://discordstatus.com/incidents/9hcyj1v9qft9 You can also check out Google's status page @ https://www.google.com/appsstatus#hl=en&v=status

  • Update
    Update

    It seems Google's services are starting to recover. I am still attempting to get both bots to connect but it may take a while...

  • Identified
    Identified

    Fire TS has unfortunately been disconnected and is currently struggling to reconnect it's shards. I've also updated this to identified since everyone and their mother knows Google is having issues...

  • Update
    Update

    Fire has been disconnected from Discord and is attempting but struggling to reconnect.

  • Investigating
    Investigating

    It seems Google is having a major outage with a ton of it's services down which Fire directly & indirectly rely on (YouTube for filters, Google Assistant for the google command and most importantly, Discord is hosted on GCP) meaning Fire may experience issues.

Potential DDoS Attack
  • Resolved
    Resolved

    I have now disabled under attack mode and deployed an update to Vanity URLs to try prevent this from happening in the future. Traffic levels have returned to normal so I'm marking this as resolved.

  • Monitoring
    Monitoring

    Services have started recovering. Under attack mode will stay enabled until I am sure the attack has stopped.

  • Investigating
    Investigating

    It seems Fire's services are getting hit with a DDoS attack (3m+ requests in the last hour) and it seems to be affecting some services. I've activated Cloudflare's under attack mode on the domains that are being hit with the most requests so you may see Cloudflare checking your browser while attempting to visit some pages. I will attempt to restore services that have been affected but it may require a reboot (VPS seems to have run out of memory)

Leave messages & logging not working
  • Resolved
    Resolved

    The fix, along with a fix for the fix (forgot to put await) has been deployed and leave messages & logging appear to be working. I've had to remove stayed for, nickname and roles from leave logging due to not having access to that data. I'm unsure on whether or not these will ever return since I don't want Fire caching a ton of members.

  • Update
    Update

    The fix is being deployed now. Fire will go offline for a few seconds while it restarts

  • Identified
    Identified

    It seems that the lack of cache has caused leave messages & logging to stop working. I have identified a possible fix and am working on implementing it now.

Elevated latency & error rates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Discord have implemented a fix and the latency has decreased to normal levels. I am monitoring the situation to ensure everything is resolved

  • Identified
    Identified

    It seems Fire is experiencing increased latency and error rates. This is due to issues with Discord which you can find more information on at https://discordstatus.com/incidents/h22zw3cshbz7

Dec 2020 to Feb 2021

Next