Fire - Notice history

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Slash commands are now functioning correctly. Fire should be fully operational once again. If you encounter any issues, please let me know in the support Discord ( https://inv.wtf/fire )

Discord seems to be available again and Fire has been brought back online. I will be monitoring things closely to ensure everything is working smoothly

This incident has been resolved.

A temporary fix has been implemented and commands should no longer return errors. A permanent fix is being made and will be deployed before this issue can reoccur

This incident has been resolved.

The fix has been implemented and is currently being deployed to both Fire & Fire Beta. The bot will go offline for a moment in your server as it deploys

The issue has been identified and a fix will be deployed soon. The cause turned out to not be discord.js but some code for an experimental Discord feature that will be releasing later this year.

This incident has been resolved.

After a quick investigation, Fire's VPS did indeed reboot but PM2 successfully restarted all processes from the dump. It seems something got updated sometime around the reboot and it caused issues with native modules for some services. I've gone ahead and rebuilt any native modules and restarted any errored processes and it seems everything is back online. I will keep monitoring to ensure everything is running smoothly

This issue has been resolved. The cause of the issue was a queue that clusters get put into after they identify with Aether based on the max concurrency returned by Discord in /gateway/bot to prevent too many shards starting at once. This queue took too long for some clusters to be assigned an id and shards so they disconnected and entered a race condition that allowed two of them be assigned the same id and shards, causing the double responses and the shard it was supposed to be given was now stuck offline. I noticed this issue pretty quickly but I have added more robust monitoring for this exact scenario to alert me almost immediately.

A fix has been implemented and is currently being deployed

The issue has been identified and a fix is being made. Unfortunately I cannot bring cluster 3 online without this fix as the issue will just occur again.

This incident has been resolved.

Everything that was running is once again running (after pain with poetry), I am now going to check to make sure nothing has broken in the process

All services listed on the statuspage are up and running! Working on internal services now. These services being down may affect some features in external services so you may encounter some errors.

Some services are back online, continuing to work on the rest!

Restoring from the PM2 dump was not successful so I will be manually bringing all processes back. This may take a while but I will try and start important processes first. I apologise for the inconvenience

While working on restoring, I had noticed PM2 was still trying to use the old Node version. It's been a couple minutes and I have figured out why so restoring should hopefully not take too much longer!