Fire - Notice history

This incident has been resolved.

The fix has been implemented and is currently being deployed to both Fire & Fire Beta. The bot will go offline for a moment in your server as it deploys

The issue has been identified and a fix will be deployed soon. The cause turned out to not be discord.js but some code for an experimental Discord feature that will be releasing later this year.

This incident has been resolved.

After a quick investigation, Fire's VPS did indeed reboot but PM2 successfully restarted all processes from the dump. It seems something got updated sometime around the reboot and it caused issues with native modules for some services. I've gone ahead and rebuilt any native modules and restarted any errored processes and it seems everything is back online. I will keep monitoring to ensure everything is running smoothly

This issue has been resolved. The cause of the issue was a queue that clusters get put into after they identify with Aether based on the max concurrency returned by Discord in /gateway/bot to prevent too many shards starting at once. This queue took too long for some clusters to be assigned an id and shards so they disconnected and entered a race condition that allowed two of them be assigned the same id and shards, causing the double responses and the shard it was supposed to be given was now stuck offline. I noticed this issue pretty quickly but I have added more robust monitoring for this exact scenario to alert me almost immediately.

A fix has been implemented and is currently being deployed

The issue has been identified and a fix is being made. Unfortunately I cannot bring cluster 3 online without this fix as the issue will just occur again.

This incident has been resolved.

Everything that was running is once again running (after pain with poetry), I am now going to check to make sure nothing has broken in the process

All services listed on the statuspage are up and running! Working on internal services now. These services being down may affect some features in external services so you may encounter some errors.

Some services are back online, continuing to work on the rest!

Restoring from the PM2 dump was not successful so I will be manually bringing all processes back. This may take a while but I will try and start important processes first. I apologise for the inconvenience

While working on restoring, I had noticed PM2 was still trying to use the old Node version. It's been a couple minutes and I have figured out why so restoring should hopefully not take too much longer!

Everything seems to be running smoothly now. It is still unknown when the issues started so Fire may have been offline for quite a while. I have systems in place to alert me if a cluster dies but due to a misconfiguration, it didn't see the one offline cluster as it was only set to recognize 2 of the 4 clusters meaning 1 out of 4 crashing didn't trigger the alert. This issue has since been resolved and any future issues like this should be resolved much sooner. I apologise for the inconvenience. I strive to have near perfect uptime for Fire as poor uptime is one of the issues I've had with other bots but while this issue was caused by something not 100% in my control I do consider it unacceptable that it wasn't able to automatically recover and will be working to improve detection of issues like this which alongside the fixed alerting should ensure this doesn't happen again. An interesting note on this incident: The ongoing rewrite of the Fire website was able to correctly identify and display the outage status for cluster 3 which means the issue was detected therefore it's possible it was actually an issue on Discord's end that lead it to not reconnect.

Fire should now be online in all servers. If you still see Fire as offline, first try restarting Discord and if that does not resolve the issue, let me know in the #fire-help channel in Fire's Discord server ( https://inv.wtf/fire )

While attempting to get cluster 3 back online, it got assigned a different cluster id and shard which indicates another cluster may be having issues. I will instead perform a full clean restart of Fire (taking all clusters offline and then bringing them up one by one to ensure each process gets the correct id) so it will go offline temporarily in all servers.

It seems that Fire's VPS had some intermittent network issues and that specific cluster did not recover (the process is still alive but not connected to Discord) It should come back online in ~30 seconds

Marking this incident as resolved since there's been no issues on Fire's end for a while now

Discord have opened a new incident mentioning issues upstream with their provider (likely talking about Google Cloud) You can keep up to date with this incident at https://stspg.io/skqfyslpp9h0?u=1mx6b389pzpg or in the Status Discord server at https://inv.wtf/statuspages (messages load now!) Fire does not seem to have any issues at the moment but this incident will stay open until Discord's is resolved

Discord has reported that the service has returned to normal conditions. I will be monitoring this from Fire's end and will provide updates as necessary

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Latency has returned to acceptable levels but is unfortunately not back to what it was. I am monitoring request latency and will close this incident if it stays at acceptable levels for a decent amount of time

Error rates have returned to normal and Fire is fully operational. I will be deploying some updates soon that updates Fire to Discord.JS v13, fixes some bugs & logs requests to Influx so I can monitor them in Grafana so you may notice Fire restart within the hour

I kept Fire offline for a minute or so and started it up again and it seems to be able to make requests again. I am monitoring requests to ensure no further issues arise

This incident has been resolved.

The 503s seem to have gone away both for Fire and my client. Will keep this open in case any further issues arise.