Reminders being sent before the intended date
Incident
Report for Fire
Resolved
Affected users have been sent details about this issue and have been linked to this incident.
I will be monitoring tickets for a little while longer tonight (currently 00:20 here) and will continue during the day.
I will be monitoring tickets for a little while longer tonight (currently 00:20 here) and will continue during the day.
Update
I have restored as much as I can and Fire is now back up and running. Here's the details for those who still have missing reminders and would like some help restoring as many as I possibly can.
I've created a small little eval script that will fetch the messages in your DMs with the bot, find the reminder messages sent after the broken update and try rerun the command using the message link found in the DM. This only works for reminders set using a message command or the "--remind" flag, not slash/context commands.
If you wish to have this script ran for you, join the Fire Discord server (discord.gg/firebot) and open a ticket using the /new command in either the #testing or #bot-commands channels.
As mentioned earlier, I will be working on improving backup processes to ensure a situation like this does not occur again.
I've created a small little eval script that will fetch the messages in your DMs with the bot, find the reminder messages sent after the broken update and try rerun the command using the message link found in the DM. This only works for reminders set using a message command or the "--remind" flag, not slash/context commands.
If you wish to have this script ran for you, join the Fire Discord server (discord.gg/firebot) and open a ticket using the /new command in either the #testing or #bot-commands channels.
As mentioned earlier, I will be working on improving backup processes to ensure a situation like this does not occur again.
Monitoring
A fix has been implemented and we are monitoring the results.
Update
The *intended* bug fix along with some additional tracking for errors is being deployed now and Fire will become fully operational again.
Once again, this may result in old reminders being sent due to the restored state from the backup. The reminders from the command logs have not yet been restored as they require the bot to be fully operational.
Details about aiding in restoring reminders will come later.
Once again, this may result in old reminders being sent due to the restored state from the backup. The reminders from the command logs have not yet been restored as they require the bot to be fully operational.
Details about aiding in restoring reminders will come later.
Update
I have restored the reminders from the backup and am now making my way through the command logs to recover as much as I can.
I did notice one unfortunate side effect of the latest reminders update though. Due to changes in how the arguments are passed to the command, the actual data is no longer stored in the command logs as it doesn't log full objects inside the arguments object
I will update this page & make an announcement in the Fire Discord server (discord.gg/firebot) once I'm finished restoring things about how I will attempt to aid in recovering reminders that did not get restored by using the DMs that were mistakenly sent, since they should link to the original command used to set the reminder.
The restoration process from the command logs will take some time as I need to manually sort them and filter any that would've already been sent.
I did notice one unfortunate side effect of the latest reminders update though. Due to changes in how the arguments are passed to the command, the actual data is no longer stored in the command logs as it doesn't log full objects inside the arguments object
I will update this page & make an announcement in the Fire Discord server (discord.gg/firebot) once I'm finished restoring things about how I will attempt to aid in recovering reminders that did not get restored by using the DMs that were mistakenly sent, since they should link to the original command used to set the reminder.
The restoration process from the command logs will take some time as I need to manually sort them and filter any that would've already been sent.
Update
Unfortunately, there may be some loss of data here as I don't have a super recent backup. Fire's DB backups are completed manually and it would appear I haven't done so in a while. I will first attempt to restore all the reminders from the latest backup I have
Thankfully, I have command logs that should allow me to restore most of the missing reminders since the backup. I say most as due to how these logs work, it isn't guaranteed that they will all contain enough data for the reminder to be restored and any reminders created with the "--remind" flag are not included
I really do apologise for any inconvenience caused by my own negligence. Once all this is finished, I will be working on improving my backup process to prevent issues like this in the future.
Thankfully, I have command logs that should allow me to restore most of the missing reminders since the backup. I say most as due to how these logs work, it isn't guaranteed that they will all contain enough data for the reminder to be restored and any reminders created with the "--remind" flag are not included
I really do apologise for any inconvenience caused by my own negligence. Once all this is finished, I will be working on improving my backup process to prevent issues like this in the future.
Identified
Due to a mistake in what should've been a bug fix, Fire has unfortunately started sending out all reminders.
I am working on getting reminders restored before I turn Aether (Fire's cluster manager which also handles reminders) back online
I am working on getting reminders restored before I turn Aether (Fire's cluster manager which also handles reminders) back online
This incident affected: Main (Bot) and Backend (Aether).