Fire - Notice history

Beta Bot experiencing major outage

Bot - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Beta Bot - Major outage

Website - Operational

100.0% uptime
Feb 2021 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2021
Mar 2021
Apr 2021

Notice history

View current status

Apr 2021

Potential Discord Issues
ResolvedDegraded performance1 hour 4 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Fire does not seem to be experiencing high latency anymore. I will keep this open for a little while longer to be sure

  • Investigating
    UTC
    Investigating

    Fire is currently experiencing high latency on some requests, indicating there may be some issues with Discord (or Cloudflare, since they currently have an open incident) Updates will follow once I get more details of any potential issues

Services Offline
ResolvedMajor outage1 hour 7 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Everything is coming back online. I'll be monitoring requests to ensure this does not happen again.

  • Update
    UTC
    Update

    Fire & other services should be back online in ~20 minutes

  • Identified
    UTC
    Identified

    It seems Fire's VPS has gotten a temp Cloudflare ban and I'm pretty sure Donald Duck is the cause (if you know, you know) meaning any service that uses Discord will be offline (likely for about an hour)

  • Investigating
    UTC
    Investigating

    It seems Fire has gone offline, I am currently investigating the cause

Mar 2021

Action Log Delays
ResolvedDegraded performance11 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    It seems Fire has restarted and everything is back to realtime now but log queues were lost as they're not persisted across restarts. I am monitoring to ensure the issue does not reoccur.

  • Investigating
    UTC
    Investigating

    It seems action logs are currently quite delayed. I am investigating the cause of the delays and will attempt to get the backlog processed and sent as soon as possible

Potential Discord Outage
ResolvedDegraded performance28 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Fire is no longer seeing high response times. I am monitoring to ensure the issue is fully resolved.

  • Update
    UTC
    Update

    It seems Cloudflare may be having some issues in Europe and while they haven't explicitly listed the nearest data center to Fire (FRA) as an affected region, it may still be impacted which may impact Fire's ability to send requests to sites behind Cloudflare. You can monitor their incident at https://stspg.io/rmtjpnkbcmrz?u=fw0b7s4cfm0p

  • Investigating
    UTC
    Investigating

    It seems Discord may be having some issues as Fire is seeing some high response times. You may notice delays in responses from Fire.

Moderation/Action/Member logs not being sent
ResolvedPartial outage42 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Fix has been deployed to both clusters and the log queue should've been recovered for guilds on Cluster 1 though it may have needed to attempt another log to actually send the queue. I'm monitoring logging in one of the larger guilds Fire is in to ensure there's no lingering issues.

  • Update
    UTC
    Update

    After deploying the fix and seeing it not work, I noticed I had forgot to add some changes to the commit so it was only half fixed. Full fix being pushed and deployed now 😅

  • Update
    UTC
    Update

    The fix has been pushed and is ready to deploy. Due to logging being broken on Cluster 0 after the failed fix, I will not be able to recover items currently in the queue. As for Cluster 1, I should be able to recover the queue once the fix has deployed to Cluster 0

  • Identified
    UTC
    Identified

    The issue has been identified and a fix is being worked on

  • Investigating
    UTC
    Investigating

    Seems the fix just broke it even more, back to the drawing board I go...

  • Identified
    UTC
    Identified

    A potential fix has been made and pushed. I will now attempt to send any messages currently in the queue for all guilds on Cluster 0 and then deploy the update.

  • Investigating
    UTC
    Investigating

    It seems the most recent deployment to try improve Fire's log queue system has caused logs to get stuck in the queue and not sending. This queue system is to prevent Fire from getting ratelimited by sending too many webhook messages, so it bundles logs when they're going too fast but it seems logs are not getting past the queue. I will be attempting to resolve the issue and get a fix pushed. I should be able to recover logs that are currently stuck in the queue to prevent missed logs

Elevated Error Rates & Missed Logs
ResolvedPartial outage41 minutes
  • Resolved
    UTC
    Resolved

    Discord has marked the issue as resolved and I have seen error rates return to normal.

  • Monitoring
    UTC
    Monitoring

    Discord have seemingly mitigated the issues and are monitoring recovery. I am doing the same on my end

  • Identified
    UTC
    Identified

    Fire is currently experiencing high error rates and missing logs due to an ongoing Discord outage. You can find more information @ https://stspg.io/02g53tzl95nc?u=pqtdnzffzgbb

Config migrations
Resolved43 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Update
    UTC
    Update

    Fire has restarted and I am monitoring to ensure there's no lingering issues

  • Update
    UTC
    Update

    Fire is now restarting. This should only take around a minute!

  • Update
    UTC
    Update

    Migration has fully completed and was a success! Fire will soon reboot fully to remove migration & legacy code.

  • Update
    UTC
    Update

    Migration has completed on Cluster 1! I will now migrate configs for guilds Fire is no longer in. No impact on the bot is to be expected during this final stage

  • Update
    UTC
    Update

    Migration has completed on Cluster 0! Cluster 1 will now reboot to update and start migration...

  • Update
    UTC
    Update

    Migration has technically completed but I am going to rerun migration to make it delete default prefixes since there's no need to store default values. This migration will happen in a few minutes. Migration has still not happened on Cluster 1!

  • Update
    UTC
    Update

    Uh oh, seems like I made an oopsie and forgot to release the lock, causing the migration to get stuck after two guilds have been migrated. Working on getting a fix deployed!

  • Update
    UTC
    Update

    Cluster 0 is about to restart. Once again, some features may be unavailable until migration has completed for your guild. If migration has not completed, you may receive an error message when attempting to complete some actions.

  • Monitoring
    UTC
    Monitoring

    To allow for a new feature, all guild configs with the "config.prefix" or "main.prefix" setting must be migrated to a new format. This update brings support for multiple prefixes per guild meaning the prefixes must be an array, which is what this migration will change existing configs to. Migration will be done on cluster 0 first (you can check which cluster your guild is on at https://fire.gaminggeek.dev/stats by entering your guild id) and some features may be unavailable until migration has completed for your guild. A backup has been made if major issues arise but it should be smooth sailing. Cluster 0 will reboot in about 10 minutes to start migration. This incident will be updated to reflect this when it happens.

Feb 2021

Mute/Ban/Reminder times not being parsed correctly
ResolvedDegraded performance
  • Resolved
    UTC
    Resolved

    I had just recently been made aware of an issue with commands that use Fire's parseTime function (reminders & temp mutes/bans) not parsing times correctly if the time was at the start of the reason. This would result in the time being parsed as 0 minutes which resulted in errors while creating reminders (saying you can't set reminders for less than 2 minutes) and permanent mutes/bans rather than temporary. This issue has been resolved as of 13:32 UTC and Fire should start parsing times correctly. Note: If the time was NOT at the start of the reason, they were unaffected by this bug and therefore were being parsed correctly

Some Web Services Unavailable
ResolvedPartial outage16 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    The issue has been identified (it was a comma in the config that shouldn't be there but somehow worked in the old NGINX version!?) and has been fixed. I am monitoring to ensure this issue does not occur on other pages (it seems Aether's API was the only one affected) and will update this incident as I check all pages.

  • Investigating
    UTC
    Investigating

    While work was being done on improving Fire's rate limiting, an NGINX upgrade was required so that I could make use of new features to improve it. This upgrade did not go as planned and some pages may display an NGINX error or "Welcome to nginx" The cause of these issues is unknown as all configs are still intact and most pages seem to be working fine. Known affected pages are: - Aether Rest API - Fire Website (relies on above)

Discord CDN Issues
ResolvedMajor outage41 minutes
  • Resolved
    UTC
    Resolved

    Seems to be fully resolved. If any more issues occur, another incident will be opened (provided I am awake)

  • Monitoring
    UTC
    Monitoring

    Discord has claimed the issue is resolved and their engineering team are monitoring the situation.

  • Identified
    UTC
    Identified

    Discord has opened an incident, https://discordstatus.com/incidents/4ylv6l1g10v6 500s are occurring on almost all requests, no matter the endpoint. https://i.imgur.com/vUaYxpQ.png

  • Update
    UTC
    Update

    API is kicking the bucket real hard... https://media.discordapp.net/attachments/804820966545948702/805967529415409686/graph.png Changing to major outage

  • Investigating
    UTC
    Investigating

    Seems Discord's CDN has died completely, meaning any commands that upload files will fail. Discord have not yet said anything about the issue but if/when they open an incident, I'll post it here.

Previous

Feb 2021 to Apr 2021

Next