Fire - Notice history

All systems operational

Bot - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Website - Operational

100% - uptime
Mar 2021 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2021
Apr 2021
May 2021

Notice history

May 2021

Elevated latency
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We're experiencing elevated latency on Fire. This is most likely an issue on Discord's end rather than ours. We're currently investigating the issue

Major Discord Outage
  • Resolved
    Resolved

    Welp, right before I posted the previous update, Discord updated their incident stating the issue is resolved. So I guess this is resolved too now...

  • Identified
    Identified

    From about 1:13 UTC to 1:21 UTC, Fire experienced elevated latency and failed requests. These issues seem to have calmed down for now but it's unknown whether Discord has fully resolved the issue yet. For now, Fire seems to be having minimal issues, so I will change the status to "Degraded Performance" until I have confirmation the issues are resolved

  • Investigating
    Investigating

    Discord's API is currently having major issues. As I am currently on my phone I am unable to monitor the impact on Fire and other services but I will update this incident with more info as soon as I can

Apr 2021

Potential Discord Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Fire does not seem to be experiencing high latency anymore. I will keep this open for a little while longer to be sure

  • Investigating
    Investigating

    Fire is currently experiencing high latency on some requests, indicating there may be some issues with Discord (or Cloudflare, since they currently have an open incident) Updates will follow once I get more details of any potential issues

Services Offline
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Everything is coming back online. I'll be monitoring requests to ensure this does not happen again.

  • Update
    Update

    Fire & other services should be back online in ~20 minutes

  • Identified
    Identified

    It seems Fire's VPS has gotten a temp Cloudflare ban and I'm pretty sure Donald Duck is the cause (if you know, you know) meaning any service that uses Discord will be offline (likely for about an hour)

  • Investigating
    Investigating

    It seems Fire has gone offline, I am currently investigating the cause

Mar 2021

Action Log Delays
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    It seems Fire has restarted and everything is back to realtime now but log queues were lost as they're not persisted across restarts. I am monitoring to ensure the issue does not reoccur.

  • Investigating
    Investigating

    It seems action logs are currently quite delayed. I am investigating the cause of the delays and will attempt to get the backlog processed and sent as soon as possible

Potential Discord Outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Fire is no longer seeing high response times. I am monitoring to ensure the issue is fully resolved.

  • Update
    Update

    It seems Cloudflare may be having some issues in Europe and while they haven't explicitly listed the nearest data center to Fire (FRA) as an affected region, it may still be impacted which may impact Fire's ability to send requests to sites behind Cloudflare. You can monitor their incident at https://stspg.io/rmtjpnkbcmrz?u=fw0b7s4cfm0p

  • Investigating
    Investigating

    It seems Discord may be having some issues as Fire is seeing some high response times. You may notice delays in responses from Fire.

Moderation/Action/Member logs not being sent
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Fix has been deployed to both clusters and the log queue should've been recovered for guilds on Cluster 1 though it may have needed to attempt another log to actually send the queue. I'm monitoring logging in one of the larger guilds Fire is in to ensure there's no lingering issues.

  • Update
    Update

    After deploying the fix and seeing it not work, I noticed I had forgot to add some changes to the commit so it was only half fixed. Full fix being pushed and deployed now 😅

  • Update
    Update

    The fix has been pushed and is ready to deploy. Due to logging being broken on Cluster 0 after the failed fix, I will not be able to recover items currently in the queue. As for Cluster 1, I should be able to recover the queue once the fix has deployed to Cluster 0

  • Identified
    Identified

    The issue has been identified and a fix is being worked on

  • Investigating
    Investigating

    Seems the fix just broke it even more, back to the drawing board I go...

  • Identified
    Identified

    A potential fix has been made and pushed. I will now attempt to send any messages currently in the queue for all guilds on Cluster 0 and then deploy the update.

  • Investigating
    Investigating

    It seems the most recent deployment to try improve Fire's log queue system has caused logs to get stuck in the queue and not sending. This queue system is to prevent Fire from getting ratelimited by sending too many webhook messages, so it bundles logs when they're going too fast but it seems logs are not getting past the queue. I will be attempting to resolve the issue and get a fix pushed. I should be able to recover logs that are currently stuck in the queue to prevent missed logs

Elevated Error Rates & Missed Logs
  • Resolved
    Resolved

    Discord has marked the issue as resolved and I have seen error rates return to normal.

  • Monitoring
    Monitoring

    Discord have seemingly mitigated the issues and are monitoring recovery. I am doing the same on my end

  • Identified
    Identified

    Fire is currently experiencing high error rates and missing logs due to an ongoing Discord outage. You can find more information @ https://stspg.io/02g53tzl95nc?u=pqtdnzffzgbb

Config migrations
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    Fire has restarted and I am monitoring to ensure there's no lingering issues

  • Update
    Update

    Fire is now restarting. This should only take around a minute!

  • Update
    Update

    Migration has fully completed and was a success! Fire will soon reboot fully to remove migration & legacy code.

  • Update
    Update

    Migration has completed on Cluster 1! I will now migrate configs for guilds Fire is no longer in. No impact on the bot is to be expected during this final stage

  • Update
    Update

    Migration has completed on Cluster 0! Cluster 1 will now reboot to update and start migration...

  • Update
    Update

    Migration has technically completed but I am going to rerun migration to make it delete default prefixes since there's no need to store default values. This migration will happen in a few minutes. Migration has still not happened on Cluster 1!

  • Update
    Update

    Uh oh, seems like I made an oopsie and forgot to release the lock, causing the migration to get stuck after two guilds have been migrated. Working on getting a fix deployed!

  • Update
    Update

    Cluster 0 is about to restart. Once again, some features may be unavailable until migration has completed for your guild. If migration has not completed, you may receive an error message when attempting to complete some actions.

  • Monitoring
    Monitoring

    To allow for a new feature, all guild configs with the "config.prefix" or "main.prefix" setting must be migrated to a new format. This update brings support for multiple prefixes per guild meaning the prefixes must be an array, which is what this migration will change existing configs to. Migration will be done on cluster 0 first (you can check which cluster your guild is on at https://fire.gaminggeek.dev/stats by entering your guild id) and some features may be unavailable until migration has completed for your guild. A backup has been made if major issues arise but it should be smooth sailing. Cluster 0 will reboot in about 10 minutes to start migration. This incident will be updated to reflect this when it happens.

Mar 2021 to May 2021

Next