Fire - Notice history

This incident has been resolved.

Services should be back online after a small little data issue that required a restart. As per usual, services may show degraded performance for a bit when first coming back online while everything warms back up (hypothetically that is, not actual temperature) If you experience any issues, report them in the Fire Discord server: discord.gg/firebot

Hetzner has resolved the issues on their end and I will be working on getting things back smoothly

Hetzner has finally posted on their own status page about this issue. You can monitor their progress at the link below https://status.hetzner.com/incident/78b912a9-6409-4007-bdd1-02d86af2a7b7

Everything is back to normal!

Services have been restored and I am monitoring to ensure they perform as expected!

With the data from just Aether removed, influxdb booted up without any issues, so my assumption was indeed correct. Now it's time to figure out *why* that data caused this issue and figure out if the data can be restored and preventative measures put in place to ensure this does not happen again Services will begin to come back online soon.

I've managed to download the data from influx and am now going to try and get it working again. With influx currently stopped & the machine no longer compressing/uploading all the data, it is running as smoothly as it normally would so I think my assumption is correct. I've taken Aether & Fire offline again temporarily while I work on this

The issue with Fire & Aether using the wrong Node version has been resolved and they should come back online shortly. The backup of influxdb data is still ongoing so unfortunately no progress can be made on getting everything back to normal just yet

After getting some things back online, I noticed that they're using the wrong version of Node so I'm working on rectifying that and will bring them back up as soon as I can

I've updated Fire/Aether to add a toggle for using Influx, disabled that and have got them back up and running. Performance will be degraded though

The investigation is still ongoing. My current suspect is influxdb as it is failing to start after I did a reboot of the system so I am currently trying to download all the data from it so that I can clear it out and see if that changes anything. I have a lot of data stored in influx and have not cleared it out recently so this process will likely take some time. Services are currently offline as a lot of them make use of influx (especially Fire/Aether) and I don't want anything impacting this process so unfortunately, they will be staying offline until I can get everything downloaded.

Affected users have been sent details about this issue and have been linked to this incident. I will be monitoring tickets for a little while longer tonight (currently 00:20 here) and will continue during the day.

I have restored as much as I can and Fire is now back up and running. Here's the details for those who still have missing reminders and would like some help restoring as many as I possibly can. I've created a small little eval script that will fetch the messages in your DMs with the bot, find the reminder messages sent after the broken update and try rerun the command using the message link found in the DM. This only works for reminders set using a message command or the "--remind" flag, not slash/context commands. If you wish to have this script ran for you, join the Fire Discord server (discord.gg/firebot) and open a ticket using the /new command in either the #testing or #bot-commands channels. As mentioned earlier, I will be working on improving backup processes to ensure a situation like this does not occur again.

A fix has been implemented and we are monitoring the results.

The *intended* bug fix along with some additional tracking for errors is being deployed now and Fire will become fully operational again. Once again, this may result in old reminders being sent due to the restored state from the backup. The reminders from the command logs have not yet been restored as they require the bot to be fully operational. Details about aiding in restoring reminders will come later.

I have restored the reminders from the backup and am now making my way through the command logs to recover as much as I can. I did notice one unfortunate side effect of the latest reminders update though. Due to changes in how the arguments are passed to the command, the actual data is no longer stored in the command logs as it doesn't log full objects inside the arguments object I will update this page & make an announcement in the Fire Discord server (discord.gg/firebot) once I'm finished restoring things about how I will attempt to aid in recovering reminders that did not get restored by using the DMs that were mistakenly sent, since they should link to the original command used to set the reminder. The restoration process from the command logs will take some time as I need to manually sort them and filter any that would've already been sent.

Unfortunately, there may be some loss of data here as I don't have a super recent backup. Fire's DB backups are completed manually and it would appear I haven't done so in a while. I will first attempt to restore all the reminders from the latest backup I have Thankfully, I have command logs that should allow me to restore most of the missing reminders since the backup. I say most as due to how these logs work, it isn't guaranteed that they will all contain enough data for the reminder to be restored and any reminders created with the "--remind" flag are not included I really do apologise for any inconvenience caused by my own negligence. Once all this is finished, I will be working on improving my backup process to prevent issues like this in the future.