Bot - Operational
Bot
Previous page
Next page
Beta Bot - Operational
Beta Bot
Website - Operational
Website
Expand group
Backend
External
Extras
This incident has been resolved.
We're experiencing elevated latency on Fire. This is most likely an issue on Discord's end rather than ours. We're currently investigating the issue
Welp, right before I posted the previous update, Discord updated their incident stating the issue is resolved. So I guess this is resolved too now...
From about 1:13 UTC to 1:21 UTC, Fire experienced elevated latency and failed requests. These issues seem to have calmed down for now but it's unknown whether Discord has fully resolved the issue yet. For now, Fire seems to be having minimal issues, so I will change the status to "Degraded Performance" until I have confirmation the issues are resolved
Discord's API is currently having major issues. As I am currently on my phone I am unable to monitor the impact on Fire and other services but I will update this incident with more info as soon as I can
Fire does not seem to be experiencing high latency anymore. I will keep this open for a little while longer to be sure
Fire is currently experiencing high latency on some requests, indicating there may be some issues with Discord (or Cloudflare, since they currently have an open incident) Updates will follow once I get more details of any potential issues
Everything is coming back online. I'll be monitoring requests to ensure this does not happen again.
Fire & other services should be back online in ~20 minutes
It seems Fire's VPS has gotten a temp Cloudflare ban and I'm pretty sure Donald Duck is the cause (if you know, you know) meaning any service that uses Discord will be offline (likely for about an hour)
It seems Fire has gone offline, I am currently investigating the cause
It seems Fire has restarted and everything is back to realtime now but log queues were lost as they're not persisted across restarts. I am monitoring to ensure the issue does not reoccur.
It seems action logs are currently quite delayed. I am investigating the cause of the delays and will attempt to get the backlog processed and sent as soon as possible
Fire is no longer seeing high response times. I am monitoring to ensure the issue is fully resolved.
It seems Cloudflare may be having some issues in Europe and while they haven't explicitly listed the nearest data center to Fire (FRA) as an affected region, it may still be impacted which may impact Fire's ability to send requests to sites behind Cloudflare. You can monitor their incident at https://stspg.io/rmtjpnkbcmrz?u=fw0b7s4cfm0p
It seems Discord may be having some issues as Fire is seeing some high response times. You may notice delays in responses from Fire.
Fix has been deployed to both clusters and the log queue should've been recovered for guilds on Cluster 1 though it may have needed to attempt another log to actually send the queue. I'm monitoring logging in one of the larger guilds Fire is in to ensure there's no lingering issues.
After deploying the fix and seeing it not work, I noticed I had forgot to add some changes to the commit so it was only half fixed. Full fix being pushed and deployed now 😅
The fix has been pushed and is ready to deploy. Due to logging being broken on Cluster 0 after the failed fix, I will not be able to recover items currently in the queue. As for Cluster 1, I should be able to recover the queue once the fix has deployed to Cluster 0
The issue has been identified and a fix is being worked on
Seems the fix just broke it even more, back to the drawing board I go...
A potential fix has been made and pushed. I will now attempt to send any messages currently in the queue for all guilds on Cluster 0 and then deploy the update.
It seems the most recent deployment to try improve Fire's log queue system has caused logs to get stuck in the queue and not sending. This queue system is to prevent Fire from getting ratelimited by sending too many webhook messages, so it bundles logs when they're going too fast but it seems logs are not getting past the queue. I will be attempting to resolve the issue and get a fix pushed. I should be able to recover logs that are currently stuck in the queue to prevent missed logs
Discord has marked the issue as resolved and I have seen error rates return to normal.
Discord have seemingly mitigated the issues and are monitoring recovery. I am doing the same on my end
Fire is currently experiencing high error rates and missing logs due to an ongoing Discord outage. You can find more information @ https://stspg.io/02g53tzl95nc?u=pqtdnzffzgbb
Fire has restarted and I am monitoring to ensure there's no lingering issues
Fire is now restarting. This should only take around a minute!
Migration has fully completed and was a success! Fire will soon reboot fully to remove migration & legacy code.
Migration has completed on Cluster 1! I will now migrate configs for guilds Fire is no longer in. No impact on the bot is to be expected during this final stage
Migration has completed on Cluster 0! Cluster 1 will now reboot to update and start migration...
Migration has technically completed but I am going to rerun migration to make it delete default prefixes since there's no need to store default values. This migration will happen in a few minutes. Migration has still not happened on Cluster 1!
Uh oh, seems like I made an oopsie and forgot to release the lock, causing the migration to get stuck after two guilds have been migrated. Working on getting a fix deployed!
Cluster 0 is about to restart. Once again, some features may be unavailable until migration has completed for your guild. If migration has not completed, you may receive an error message when attempting to complete some actions.
To allow for a new feature, all guild configs with the "config.prefix" or "main.prefix" setting must be migrated to a new format. This update brings support for multiple prefixes per guild meaning the prefixes must be an array, which is what this migration will change existing configs to. Migration will be done on cluster 0 first (you can check which cluster your guild is on at https://fire.gaminggeek.dev/stats by entering your guild id) and some features may be unavailable until migration has completed for your guild. A backup has been made if major issues arise but it should be smooth sailing. Cluster 0 will reboot in about 10 minutes to start migration. This incident will be updated to reflect this when it happens.
Mar 2021 to May 2021
Next
