Fire - Notice history

Everything seems fine, just deployed an update so it's a good time to mark this as resolved.

Of course right as I go to investigate, the issue stops occurring. If it happens again, I will move this back to investigating but for now everything seems fine. I'll be rebooting cluster 1 to remove some debug logging I added which should only take a few seconds

This incident has been resolved.

I am monitoring to ensure everything has fully recovered.

Fire's VPS is no longer ratelimted, working on restoring services to full capacity now

Cluster 1 is online, meaning the bot should basically be operational apart from db changes not saving once it goes back to running on the VPS. Due to issues connecting to the remote database, I will not be able to bring up a local instance of inv.wtf meaning Vanity URLs will still be broken.

Cluster 0 is running locally with seemingly no issues. Beware, any changes that involve the database will NOT BE SAVED due to it being a local copy of the database. I will work on getting cluster 1 online now.

Attempts to bring Fire online locally have not worked due to issues with my local database and connecting to the prod database remotely. I will continue to try bring Fire back online locally but it may come down to just waiting out the ratelimit (there's about 25 minutes left)

Upon further investigation, it seems Fire's VPS has gotten hit with a 1 hour ban from the API, which means this affects more than just the bot. I am working on attempting to bring up instances of affected services locally to minimise the downtime.

This incident has been resolved.

The commits have been reverted and Fire is back online in it's normal state. I am monitoring to ensure that no leftover yarn v2 issues occur

Fire is online though I'm not sure how? Seems pm2 is really not a fan of yarn v2 so I am attempting to revert

This incident has been resolved.

This incident has been resolved.

It seems the error rate has decreased to normal levels. Keeping this open a bit longer to be sure

Discord have marked the issue as resolved and Fire has not seen any issues recently.

Related Incident: https://status.cloud.google.com/incident/zall/20013

Both Fire & Fire TS are online but may have limited functionality due to ongoing issues with Discord

Fire & Fire TS are still unable to connect to Discord. Google have marked all of their components as operational and Discord has marked the API as operational (when it's clearly not) so I'm unsure how much longer this downtime will last

Discord have opened an incident on their status page. You can find it @ https://discordstatus.com/incidents/9hcyj1v9qft9 You can also check out Google's status page @ https://www.google.com/appsstatus#hl=en&v=status

It seems Google's services are starting to recover. I am still attempting to get both bots to connect but it may take a while...

Fire TS has unfortunately been disconnected and is currently struggling to reconnect it's shards. I've also updated this to identified since everyone and their mother knows Google is having issues...

Fire has been disconnected from Discord and is attempting but struggling to reconnect.

I have now disabled under attack mode and deployed an update to Vanity URLs to try prevent this from happening in the future. Traffic levels have returned to normal so I'm marking this as resolved.

Services have started recovering. Under attack mode will stay enabled until I am sure the attack has stopped.

The fix, along with a fix for the fix (forgot to put await) has been deployed and leave messages & logging appear to be working. I've had to remove stayed for, nickname and roles from leave logging due to not having access to that data. I'm unsure on whether or not these will ever return since I don't want Fire caching a ton of members.

The fix is being deployed now. Fire will go offline for a few seconds while it restarts

This incident has been resolved.

Discord have implemented a fix and the latency has decreased to normal levels. I am monitoring the situation to ensure everything is resolved

This incident has been resolved.

The amount of errors seems to have calmed down and Fire is once again able to send messages. Will keep this open until I get confirmation the issues are resolved

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

Services are slowly but surely automatically recovering after the reboot. If you encounter any errors, please keep this in mind. Components will be set to operational once I am sure they're fully recovered

The issues seem to have gone away but I am going to do a full reboot of the vps to be safe.