Fire Status - Incident history

Beta Bot outage

Sun, 15 Mar 2026 16:41:11 +0000

Type: Incident

Duration: 15 minutes

Affected Components:

Mar 15, 16:41:11 GMT+0
Investigating - Beta Bot cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 15, 16:56:12 GMT+0
Resolved - Beta Bot is now operational! This update was created by an automated monitoring service..

Discord API Issues (dsv750sj44fn)

Thu, 12 Mar 2026 20:30:00 +0000

Type: Incident

Duration: 41 minutes

Affected Components: ,

Mar 12, 20:30:00 GMT+0
Identified - Fire is currently experiencing elevated latency & error rates due to issues with Discord's API. Discord has posted about the issue on their own status page, [https://discordstatus.com/incidents/](https://discordstatus.com/incidents/dsv750sj44fn).

Mar 12, 20:59:52 GMT+0
Monitoring - Discord has implemented a fix and both error rates & latency have returned closer to expected levels. I will continue monitoring for some time to ensure things are fully resolved.

Mar 12, 21:10:56 GMT+0
Resolved - This incident has been resolved..

Networking Issues

Wed, 25 Feb 2026 21:08:32 +0000

Type: Incident

Duration: 29 minutes

Affected Components: , , , ,

Feb 25, 21:08:32 GMT+0
Investigating - It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console.

Feb 25, 21:13:12 GMT+0
Monitoring - Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning.

Feb 25, 21:37:39 GMT+0
Resolved - This incident has been resolved..

Infrastructure Upgrades

Wed, 18 Feb 2026 11:00:00 +0000

Type: Maintenance

Duration: 2 hours and 26 minutes

Affected Components: , , ,

Feb 18, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 18, 11:28:56 GMT+0
Identified - Fire is now back online and now running in Kubernetes! Next up will be reconfiguring DNS records which may take some time due to SSL certificate provisioning. This means that web services will still be unavailable for a while.

Feb 18, 11:00:00 GMT+0
Identified - During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like environment by moving Fire Beta to a separate machine for testing & learning. The main upgrade should realistically only take 15-20 minutes with most of the time being spent replacing updating configurations for services exposed to the web as they can no longer be simply pointed to `localhost` & some port and even then, I don't expect this to take the entire duration of the maintenance but I'd rather schedule a larger window just in case For anyone interested, I am moving Fire away from running directly on the machine and being managed via PM2 to using Docker containers & Kubernetes. This move will make it much easier to keep Fire up to date as upgrading the OS or Node version is as simple as changing the image used. It also comes with a more maintainable solution for offsite backups as I've configured the Postgres cluster with automatic backups to CloudFlare R2 rather than moving them to my Mac every so often.

Feb 18, 12:05:31 GMT+0
Identified - Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventually resolve themselves Extensive review & monitoring will be taking place for the next hour or so to ensure nothing has been missed and to do some cleanup of older resources .

Feb 18, 13:25:40 GMT+0
Completed - Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days.

Potential disruption to Fire (Automatic)

Tue, 3 Feb 2026 15:35:46 +0000

Type: Incident

Duration: 1 hour and 40 minutes

Affected Components:

Feb 3, 15:35:46 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Feb 3, 17:15:46 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Server Upgrades

Mon, 19 Jan 2026 10:00:00 +0000

Type: Maintenance

Duration: 6 days, 20 hours and 32 minutes

Affected Components: , ,

Jan 19, 10:00:00 GMT+0
Identified - During this time, several upgrades (both hardware & software) will be performed on Fire's VPS which will result in some lengthy downtime of ALL SERVICES. This downtime is unfortunate but necessary to ensure stability, security and open the door for new features that require newer software versions During this time, I will be keeping an eye on the Fire Discord (discord.gg/firebot) to answer any questions and let people know the current status of the upgrades. I'll also be around for the next while to answer questions about this maintenance if you have any so feel free to join the Discord server and ask in the #fire-help channel.

Jan 19, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 19, 10:17:01 GMT+0
Identified - The hardware of the server has been upgraded slightly (more vCPUs, more RAM & more storage) and I am now taking a snapshot of the server that I hopefully won't have to use but is there just in case anything goes wrong.

Jan 19, 10:49:08 GMT+0
Identified - The snapshot has been taken and the software side of the upgrades is now in progress.

Jan 19, 10:07:09 GMT+0
Identified - Redirects have now been put in place to redirect any traffic to Fire's web services towards a dedicated maintenance page. I will now be turning off all services to start the upgrades.

Jan 19, 12:44:31 GMT+0
Completed - Maintenance has completed successfully.... almost. During the upgrades, Fire's postgres installation got completely removed along with all the data. Fortunately, I have backups, but only for the main bot. Fire Beta has had its database recreated from scratch so no previous data will be present there so if you were relying on that, you shouldn't have! Aside from that, everything looks to be in working order. NodeJS has been updated to the latest LTS release, nginx has been updated to the latest release, postgres has been updated to the latest release and internal tooling (e.g. Grafana, Telegraf) has also been updated I'll be keeping a close eye on things throughout today to ensure things continue running smoothly. If you encounter any issues, let me know in the Fire Discord (discord.gg/firebot) .

Potential disruption to Fire (Automatic)

Thu, 1 Jan 2026 00:05:45 +0000

Type: Incident

Duration: 1 minute

Affected Components:

Jan 1, 00:05:45 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Jan 1, 00:07:07 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Internal Restructuring

Wed, 31 Dec 2025 08:24:21 +0000

Type: Incident

Duration: 1 hour and 1 minute

Affected Components: , ,

Dec 31, 08:24:21 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Dec 31, 08:29:07 GMT+0
Investigating - As part of some internal restructuring, it seems that the websocket connection to Aether is not working correctly, preventing the bot from launching correctly. I'm currently investigating this and hope to get this resolved as soon as possible.

Dec 31, 08:36:52 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Dec 31, 09:25:19 GMT+0
Resolved - This incident has been resolved..

Potential disruption to Fire (Automatic)

Wed, 31 Dec 2025 05:47:23 +0000

Type: Incident

Duration: 6 minutes

Affected Components:

Dec 31, 05:47:23 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Dec 31, 05:53:07 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Potential disruption to Fire (Automatic)

Wed, 31 Dec 2025 04:57:11 +0000

Type: Incident

Duration: 8 minutes

Affected Components:

Dec 31, 04:57:11 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Dec 31, 05:05:06 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Potential disruption to Fire (Automatic)

Mon, 29 Dec 2025 18:37:53 +0000

Type: Incident

Duration: 46 minutes

Affected Components:

Dec 29, 18:37:53 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Dec 29, 19:24:11 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Potential disruption to Fire (Automatic)

Sun, 28 Dec 2025 02:55:47 +0000

Type: Incident

Duration: 1 hour and 58 minutes

Affected Components: , ,

Dec 28, 02:55:47 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Dec 28, 02:57:52 GMT+0
Investigating - Fire's VPS appears to have been blocked from accessing Discord's API due to exceeding ratelimits. It's unknown how this occurred at this time but for now, the bot has been turned off to avoid making requests while ratelimited. It looks to be only an hour long block with about 50 minutes remaining so it shouldn't be down for long but unfortunately there's nothing that can be done other than waiting.

Dec 28, 03:51:00 GMT+0
Monitoring - Fire is now back online and looks to be operating just fine. I've made some changes that may help prevent the issue re-occurring but I'll be looking further into what caused this issue so that I can properly resolve it.

Dec 28, 04:53:41 GMT+0
Resolved - The issue seems to have stemmed from Fire being stuck in a loop, constantly making requests to edit a channel, which resulted in it getting ratelimited. As far as I can tell, the edits were part of the permission role feature, so I've made a few changes to it which should a) help identify it and b) help resolve the issue This feature is pretty old so it likely needs a rewrite to ensure things work smoothly, which has been added to my lengthy to-do list for Fire For now, Fire is operational and should stay that way. If you encounter any issues, please report them in the Fire Discord (discord.gg/firebot) .

Discord Outage - Slow and failing requests (xt705bsck7sk)

Fri, 19 Dec 2025 23:00:38 +0000

Type: Incident

Duration: 39 minutes

Affected Components:

Dec 19, 23:00:38 GMT+0
Investigating - Many of Fire's API requests to Discord are currently failing indicating a potential outage. I have had to shut down one of Fire's clusters as it got stuck in a loop retrying requests due to this issue and am working on a solution so that it can be restored.

Dec 19, 23:22:38 GMT+0
Monitoring - Things look to have improved, requests aren't failing nearly as often and latency has returned to normal levels. I will continue to monitor requests for some time just to be sure it's fully resolved.

Dec 19, 23:39:55 GMT+0
Resolved - This incident has been resolved..

Cloudflare Outage

Tue, 18 Nov 2025 11:52:30 +0000

Type: Incident

Duration: 7 hours

Affected Components: , , ,

Nov 18, 18:52:18 GMT+0
Resolved - Cloudflare have stated that things are now operating normally.

Nov 18, 11:52:30 GMT+0
Identified - Cloudflare's network is currently experiencing issues which will affect Fire due to both it and Discord making use of Cloudflare's services You may experience slow responses or failed interactions when attempting to use commands in the bot and web services may return 5XX errors.

New Website

Mon, 10 Nov 2025 15:00:00 +0000

Type: Maintenance

Duration: 44 minutes

Affected Components: ,

Nov 10, 15:00:00 GMT+0
Identified - During this time, the new website will be deployed alongside some updates to the bot. This is not expected to take long nor should there be any problems as both the website & bot changes have been tested locally & in staging The website will be unavailable for a brief period of time while a production build is deployed and configured while the bot should only be unavailable for a few seconds while it restarts for a normal deploy.

Nov 10, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 10, 15:07:14 GMT+0
Identified - The bot has been updated so now the website is going to be changed over.

Nov 10, 15:23:21 GMT+0
Identified - The new website has been successfully deployed and final checks are being ran to ensure everything is working as intended.

Nov 10, 15:43:42 GMT+0
Completed - Maintenance has completed successfully..

Discord Outage - Guilds Unavailability (9bm86bnmfhfr)

Mon, 8 Sep 2025 22:21:47 +0000

Type: Incident

Duration: 1 hour and 34 minutes

Affected Components:

Sep 8, 23:56:00 GMT+0
Resolved - This incident has been resolved..

Sep 8, 22:21:47 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Sep 8, 22:41:36 GMT+0
Identified - Upon investigating, it appears that the issues Discord are having at the moment have resulted in the gateway connections for Fire's clusters getting stuck in a form of limbo state where it appears to be connected but nothing is being received (and I have no way to test if anything can be sent as I'd normally do that via a command) I will attempt to restart clusters, starting with the cluster that has the least unavailable servers (cluster 3 with only a single unavailable server), since servers being unavailable on connect can slow down startup and might result in further issues.

Sep 8, 22:44:11 GMT+0
Identified - The restart was successful meaning cluster 3 is now online and functioning. I will continue going in order of least unavailable servers as that seems like the best thing to do right now.

Sep 8, 22:46:37 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Sep 8, 22:47:43 GMT+0
Monitoring - All clusters are back online, however, they may have slow responses or other issues related to the Discord outage. Monitoring will continue until Discord has marked their incident as resolved.

Quotes Disabled

Sun, 22 Jun 2025 12:04:55 +0000

Type: Incident

Duration: 16 minutes

Affected Components:

Jun 22, 12:04:55 GMT+0
Investigating - Quotes are currently disabled while an issue is being investigated.

Jun 22, 12:17:18 GMT+0
Identified - The issue has been identified and a fix is being made.

Jun 22, 12:21:22 GMT+0
Resolved - The fix has been deployed and quotes have been re-enabled.

Slow responses and failed interactions

Wed, 14 May 2025 00:49:50 +0000

Type: Incident

Duration: 14 minutes

Affected Components: , ,

May 14, 00:49:50 GMT+0
Identified - Discord turned 10 years old and is now old and slow. Rest latency started to increase at 00:20 UTC but not enough to trigger alerts so I only discovered things are broken a few minutes ago and seems like Discord did too as they only posted about it now too.

May 14, 00:58:19 GMT+0
Monitoring - Discord claims that things are fixed but I don't trust them so this will stay open a little longer while I keep an eye on boring graphs to see if it goes stonks (but in a bad way) again.

May 14, 01:04:04 GMT+0
Resolved - Some requests (primarily adding roles to members) are still showing somewhat higher latency than normal but things are generally fine and Fire should be functioning normally.

Discord Outage

Thu, 10 Apr 2025 19:59:48 +0000

Type: Incident

Duration: 28 minutes

Affected Components:

Apr 10, 19:59:48 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Apr 10, 20:02:12 GMT+0
Investigating - It seems that Discord may be having issues with the gateway at the moment. All of Fire's websocket connections were dropped simultaneously and are unable to reconnect along with my own websocket connection in the app experiencing the same issue There are also many reports flooding in on Twitter about issues connecting so it doesn't seem like I or Fire are alone in this..

Apr 10, 20:13:04 GMT+0
Investigating - Shard 3 has been able to successfully reconnect so the rest should hopefully follow suit but unfortunately I am still unable to get in myself and therefore unable to verify if that shard is functioning correctly.

Apr 10, 20:18:51 GMT+0
Investigating - Shard 2 has now been able to reconnect. I tried restarting cluster 0 (shard 0) to see if that would help but it is still stuck in the same loop as shard 1.

Apr 10, 20:27:14 GMT+0
Investigating - Shard 0 has risen from the dead, now just waiting on shard 1....

Apr 10, 20:27:44 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Vanity URL & Redirects Unavailable

Sun, 23 Feb 2025 14:39:00 +0000

Type: Incident

Affected Components:

Feb 23, 14:39:00 GMT+0
Resolved - An issue has just been resolved where the [inv.wtf](http://inv.wtf) domain was returning 404s for all requests due to a mistake in an earlier deploy. Existing monitoring was unfortunately unable to detect the issue and fire alerts which meant it went unnoticed for longer than it should have. Apologies for any inconvenience caused. I will be looking into improving monitoring to detect these kinds of issues and ensure they do not happen (or at least not for as long) in the future.

Potential disruption to Fire (Automatic)

Fri, 21 Feb 2025 02:04:58 +0000

Type: Incident

Affected Components:

Feb 21, 02:04:58 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Feb 21, 02:05:10 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Services Unavailable

Sat, 1 Feb 2025 22:43:31 +0000

Type: Incident

Duration: 48 minutes

Affected Components: , , ,

Feb 1, 22:43:31 GMT+0
Identified - It looks like Fire's VPS has ran out of available space. I knew this was approaching but I thought I had more time than I did. I have shut it down for now and will attempt to increase the partition size (as the disk itself had already been increased) and then bring it back online.

Feb 1, 23:07:33 GMT+0
Monitoring - Fire now has about double the storage it used to and after a slight hiccup with InfluxDB not wanting to start, everything should be back online. I will be monitoring the situation alongside trying to figure out what caused the sudden increase in disk space (my current suspicion is InfluxDB considering the trouble I had getting it to come back).

Feb 1, 23:31:38 GMT+0
Resolved - This incident has been resolved..

Migration to rewrite of Vanity URLs & Redirects

Thu, 30 Jan 2025 16:00:00 +0000

Type: Maintenance

Duration: 44 minutes

Affected Components: ,

Jan 30, 16:28:14 GMT+0
Identified - The issue regarding redirects has been fixed and they are now fully operational. I'll be monitoring it for the next 10 or so minutes to ensure no other issues appear.

Jan 30, 16:02:33 GMT+0
Identified - I have restarted Aether before moving the domains over to ensure it has the latest data (since the bot doesn't currently send the request to refresh to it) The Vanity URL & Redirect modules have been disabled on the bot now and the switchover will occur within the next minute or so.

Jan 30, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 30, 16:00:00 GMT+0
Identified - The current service handling Vanity URLs & Redirects was the first project I worked on in TypeScript which means it's very old, horribly written and a pain to work with, which is why I spent the last few days rewriting parts of it and merging them into Aether since the service already relied on it for some features so why not just combine 'em! The switch from the separate service to Aether should be mostly instant & seamless. Everything should pretty much be the same except for response times which will be quicker due to changes in caching & click vs. link behavior. The only thing that will _not_ be instant & seamless is updating the bot. Fire currently sends a HTTP request to the original service whenever a vanity/redirect is created, updated or deleted and this results in all data being reloaded, rather than just the specific item that changed. For the new merged version, it'll send a request over the existing websocket connection and that will either reload the server's vanities or reload the specific redirect, not everything. Because it is moving away from HTTP, it can't be as seamless as everything else since there is no endpoint to reload data in Aether for it to use once the switch has taken place. This will however only impact creating/updating/deleting vanities & redirects for a few minutes while the bot updates and these features will be disabled shortly before the switch takes place and will return as soon as the bot has restarted.

Jan 30, 16:07:23 GMT+0
Identified - After a slight mishap with the switchover (I forgot to add a trailing slash in the configurations), [inv.wtf](http://inv.wtf) is now pointing to Aether instead of the standalone service. The old service has now been shut down so if you encounter any issues with 502 errors, please let me know in the Fire Discord server () Next up is updating the bot. First, the update will be deployed to Fire Beta just for final testing and then it will be deployed to the main bot.

Jan 30, 16:43:32 GMT+0
Completed - Everything is working as expected. If you encounter any issues, please report them in the Fire Discord (https://discord.gg/firebot).

Jan 30, 16:25:30 GMT+0
Identified - Fire has now been updated, however, an issue has been found regarding redirects. At the moment, it seems all redirects are leading to an error page, rather than the intended destination This issue is currently being investigated and a fix will be made and deployed as soon as the cause is found.

Deploying various updates

Wed, 22 Jan 2025 23:16:00 +0000

Type: Maintenance

Duration: 53 minutes

Affected Components: ,

Jan 22, 23:41:46 GMT+0
Identified - After some small reminder related bugs (mainly related to edits) discovered after the staging deploy, it is now ready to go to production. The main Fire bot & Aether instance will now go offline.

Jan 22, 23:45:44 GMT+0
Identified - Fire & Aether are now back online with everything deployed. I'll be keeping a close eye on things for roughly 15 minutes before I mark this as complete.

Jan 23, 00:10:15 GMT+0
Completed - 👍.

Jan 22, 23:16:00 GMT+0
Identified - At this time, I will be deploying changes to both Fire & Aether which need to be done simultaneously, meaning both need to be taken offline temporarily These changes are primarily related to new reminders code but also contain some changes which should help with internal monitoring falsely claiming that a cluster has been disconnected, resulting in alerts being sent to me and this page updated. These changes were not intended to go out just yet but with the changes to fix monitoring issues, it has moved all changes up. The reminder changes have been tested thoroughly but there may be some edge cases that I missed in testing so if you encounter any issues related to reminders, please let me know in the Fire Discord (discord.gg/firebot).

Jan 22, 23:17:19 GMT+0
Identified - Staging instances of Fire & Aether will soon go offline to deploy the updates there for a final check before being deployed to production.

Config handling update

Sun, 24 Nov 2024 14:00:00 +0000

Type: Maintenance

Duration: 1 hour and 2 minutes

Affected Components: , ,

Nov 24, 14:29:53 GMT+0
Identified - After a slight hiccup with one of the new additions to Aether, the deploy to the beta services has been successful. Fire & Aether will now be turned off to facilitate the deploy .

Nov 24, 14:45:15 GMT+0
Identified - All clusters are now back online with the update deployed. I will be actively monitoring for the next 15 or so minutes to ensure everything is working as it should.

Nov 24, 15:02:20 GMT+0
Completed - Everything looks to be running perfectly and my small bit of testing on the production services has been successful so I consider this maintenance complete!.

Nov 24, 14:00:00 GMT+0
Identified - An update will be releasing to Fire & Aether (the service which manages Fire's clusters) to change how configs are handled to make reading them from external services (such as the website) easier and allowing better syncing across clusters and better handling of failures to update the config These updates are closely linked so they cannot be updated seamlessly like most incremental changes and will require both services to be offline to update them. I do not anticipate this taking a long time but I've left in room for error (which hopefully will not happen 🤞) just in case.

Nov 24, 14:01:57 GMT+0
Identified - Fire Beta & the matching beta instance of Aether will now be turned off to deploy the update to those two first The main bot will not currently be affected..

Nov 24, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 24, 14:36:57 GMT+0
Identified - The deploy to production Aether was successful. Fire's clusters will start coming back online now though there will be some spacing in-between the launch of each cluster just to ensure things are going smoothly.

Potential disruption to Fire (Automatic)

Wed, 13 Nov 2024 14:02:16 +0000

Type: Incident

Duration: 2 hours and 55 minutes

Affected Components:

Nov 13, 14:02:16 GMT+0
Investigating - Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable.

Nov 13, 14:05:11 GMT+0
Resolved - Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues.

Fire offline

Fri, 5 Jul 2024 00:24:11 +0000

Type: Incident

Jul 5, 00:32:35 GMT+0
Resolved - This incident has been resolved..

Jul 5, 00:27:51 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results..

Jul 5, 00:24:11 GMT+0
Investigating - An issue that is impacting the availability of both Fire & Fire Beta is currently being investigated.

Services unavailable

Mon, 1 Jul 2024 21:57:07 +0000

Type: Incident

Jul 1, 22:55:04 GMT+0
Resolved - The block has been removed and services are back online. Permission roles will remain disabled for the time being.

Jul 1, 22:41:11 GMT+0
Identified - The cause of the ratelimit appears to be recent changes to permission roles. This feature will be disabled temporarily once the block is removed for further investigation. There is about 10 minutes left on the block so Fire should be back soon:tm:.

Jul 1, 21:57:08 GMT+0
Investigating - It looks like Fire has been temporarily blocked from Discord's API. I'm unsure of why it got blocked at the moment but for now, all services that interact with Discord are unavailable.

Services Slow / Not Responding

Fri, 7 Jun 2024 10:40:48 +0000

Type: Incident

Jun 7, 11:21:22 GMT+0
Resolved - This incident has been resolved..

Jun 7, 11:16:49 GMT+0
Monitoring - Network activity has started to return to normal levels and services should be operating normally now..

Jun 7, 11:03:28 GMT+0
Identified - It appears this was caused by a router fault. Hetzner has identified the affected router and isolated it so things should start to return to normal soon..

Jun 7, 10:44:35 GMT+0
Investigating - It appears that Fire's network activity dropped significantly at 11:32, both in and out, leading me to believe it may be an issue with Hetzner's networking. I am continuing to investigate to find more concrete details about what exactly happened at this time.

Jun 7, 10:40:48 GMT+0
Investigating - I have received alerts about increased latency on Fire services and am investigating the issue. This appears to be impacting the bot as well as web services.

Node Version Upgrade

Sun, 26 May 2024 15:00:00 +0000

Type: Maintenance

Duration: 2 hours

May 26, 14:00:02 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

May 24, 19:06:39 GMT+0
Identified - During this time, the Node version that runs Fire and related services will be updated to the latest LTS version (20.13.1) after previously being stuck on a much older version due to the Google command requiring libraries that could not be used on newer versions This process will require some downtime of all services to perform the upgrade. I have allocated more time than I believe it will take just in case things go wrong as I have encountered issues before when attempting a Node upgrade Local tests have already been completed to ensure all services work correctly with the latest version and most were immediately functional whereas some required updates to packages or changes to code..

May 26, 15:00:02 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

May 26, 15:04:25 GMT+0
Completed - Due to unforeseen issues when attempting this upgrade, this maintenance is being postponed until further notice. Fire & all related services have remained online and will continue to do so as there will no longer be any downtime.

Google Command Unavailable

Sat, 13 Apr 2024 00:33:57 +0000

Type: Incident

May 24, 19:01:17 GMT+0
Resolved - Google has approved Fire's application and after resolving an issue that occurred once approved & monitoring to ensure that it continued to work, it is now ready for release! It will be released next week after an upcoming maintenance to upgrade things that were previously blocked by the Google command.

Apr 14, 01:23:40 GMT+0
Monitoring - The command is now available again but without the full update as that will remain locked away until Google approves it You may notice some smaller changes to the responses though and there's a hidden trick it can do :).

Apr 13, 00:33:57 GMT+0
Identified - As part of ongoing work to upgrade the /google command, it seems that the credentials got cycled which has broken the command in production I am working very hard on a big update to it which will resolve the issue but until this is released, the command will be unavailable.

Minecraft log scanning not working for Forge logs

Fri, 29 Mar 2024 16:32:48 +0000

Type: Incident

Mar 29, 18:07:49 GMT+0
Resolved - This incident has been resolved..

Mar 29, 17:20:00 GMT+0
Monitoring - The website appears to be back up and running so FIre should respond almost instantly to uploaded Forge logs. I'll keep this incident open for a while just in case it goes down again.

Mar 29, 16:44:36 GMT+0
Identified - An update is currently being deployed that should resolve it not responding at all, though it'll still be slow while it waits for the requests to time out.

Mar 29, 16:32:48 GMT+0
Identified - Fire's log scanning feature is currently unable to provide solutions or recommendations for Forge logs. It appears that Forge's website is currently timing out which results in the bot not being able to fetch version data.

Server Upgrades

Sat, 30 Dec 2023 14:00:00 +0000

Type: Maintenance

Duration: 4 hours

Dec 30, 16:21:29 GMT+0
Completed - The scheduled maintenance has been completed..

Dec 30, 14:01:05 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Dec 30, 14:09:39 GMT+0
Identified - Services are starting to be turned off before a full shutdown of the server to facilitate upgrading it.

Dec 30, 15:23:23 GMT+0
Identified - All services should be back up and running on the upgraded server. If you experience any issues, please let me know in the Fire Discord server, https://discord.gg/firebot.

Dec 30, 12:59:57 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Dec 30, 14:16:25 GMT+0
Identified - Fire's VPS is now being rescaled. This shouldn't take long, but services will not be immediately restored once it is complete.

Dec 30, 14:24:40 GMT+0
Identified - It has been rescaled and booted up. I'm now going to perform some tweaks & updates before restoring services.

Dec 25, 18:40:10 GMT+0
Identified - Happy holidays everyone! Hope you all have/had a wonderful day and spent some quality time with the people you care about. Fire has been a very good bot this year and is going to be getting some upgrades from Santa, which will result in some downtime. This has been something I've been planning on doing for a while but recently, things have started to get quite slow so it may be time to actually do so This time will be spent on upgrading the specs of the server Fire runs on as well as (hopefully) doing some long overdue software updates. It likely won't last for the 4 hours I've set it for, but I'm not quite sure how long it'll take so I'd rather overestimate than underestimate. Also, might as well add this onto the end here, you can use code "XMAS2023" for 50% off your first month/year of Fire Premium up until December 31st at 23:59 UTC!.

Deploying discord.js update

Sat, 26 Nov 2022 19:00:00 +0000

Type: Maintenance

Duration: 45 minutes

Nov 26, 19:20:55 GMT+0
Completed - The scheduled maintenance has been completed..

Nov 26, 18:00:24 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Nov 26, 19:14:53 GMT+0
Identified - Fire is back online with the updates and everything looks to be working great! I'm now double checking that everything is, in fact, working great so I can mark this as complete..

Nov 26, 19:00:15 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Nov 23, 07:59:59 GMT+0
Identified - During this time, Fire will be taken offline to perform updates. Downtime will not last throughout the full maintenance period but you may experience degraded performance for some time after the update is deployed. The amount of downtime will be kept as minimal as possible to avoid major disruption. This update brings Fire's discord.js version from 13.4.0 to 13.12.0 with support for Forum channels included in the forked version it is using along with various other bug fixes and new features. Work on new features should speed up to what it originally was after updating..

Nov 26, 19:09:09 GMT+0
Identified - I am about to take Fire offline to make the necessary changes. This should not take long and any delays will be noted here.

Deploying new and improved link filter exclusion

Thu, 24 Feb 2022 16:00:00 +0000

Type: Maintenance

Duration: 45 minutes

Feb 23, 17:45:40 GMT+0
Identified - Maintenance has been delayed to 16:00 UTC tomorrow.

Feb 24, 16:13:47 GMT+0
Identified - Testing has completed and I am about to deploy the update. This will involve short downtime of Fire. While waiting for the update to be deployed, you can check the various links below to donate and help various charities supporting Ukraine with all the horrible things happening over there ❤🇺🇦 https://voices.org.ua/en/ https://savelife.in.ua/en/donate/ https://www.rsukraine.org/ https://armysos.com.ua/en/.

Feb 24, 16:00:41 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Feb 24, 16:21:20 GMT+0
Identified - The update has been deployed and migration has completed. The new commands (including 2 new context menu commands) may take up to an hour to appear in your server and you may experience errors when attempting to use the old /linkfilter command Another update will be deployed in a short while to remove the migration script.

Feb 24, 16:04:01 GMT+0
Identified - I am running some last minute tests on the config migration script before deploying. I estimate the testing will complete at 16:10 UTC when I proceed with deploying based on the results.

Feb 23, 15:14:11 GMT+0
Identified - During this time, I will be deploying the brand new link filter exclusion which replaces the old "/filterexclude" command. With this update, "/linkfilter" has been moved to "/linkfilter toggle" and two new commands have been added, "/linkfilter exclude" & "/linkfilter list-exclude" allowing you to exclude a user, role or channel from link filters and to list who and what has been excluded In addition to the new slash commands, two new context menu commands have been added for both users & messages, "Link Filter Exclude". When used on a user, it will exclude that user from link filters and when used on a message, it will exclude all mentioned users, roles, and channels. This change involves moving the old config format for filter exclusions to the new one meaning for any servers that have added exclusions in the past, some features may be unavailable at first while the config gets moved over and you may experience errors when attempting to run commands..

Feb 24, 15:00:29 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Feb 24, 16:31:38 GMT+0
Completed - The scheduled maintenance has been completed..

Deploying improved slash commands

Fri, 4 Feb 2022 21:00:00 +0000

Type: Maintenance

Duration: 1 hour and 20 minutes

Feb 4, 21:41:47 GMT+0
Identified - The improvements and a couple bug fixes have been deployed and commands should be syncing across guilds (if you see /guild, you can try run it to force sync the commands as this was removed).

Feb 4, 19:59:33 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Feb 4, 19:13:05 GMT+0
Identified - During this time, I will be deploying the improvements I've been making to slash commands for a while now (since September 2021), including grouping commands with sub-commands and moving moderation commands to only slash commands. When these changes are deployed, ALL MODERATION COMMANDS WILL NO LONGER RESPOND TO MESSAGE COMMANDS! You will be required to use /ban, /kick, /mute etc. but fortunately, they've been greatly improved through adding argument descriptions, separating reason & time where applicable and making them respond ephemerally. I've also rewritten the parsing of slash commands to make it cleaner and more efficient which should improve response times too If you want a more in-depth changelog, check https://h.inv.wtf/busujiruwe.diff The changes to commands will take up to an hour to sync to your server once deployed so you may encounter errors until they've synced globally.

Feb 4, 21:00:44 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Feb 4, 21:16:18 GMT+0
Identified - Fire will go offline temporarily as the update is deployed. Please be patient as commands may not function correctly for a short while.

Feb 4, 22:21:40 GMT+0
Completed - The scheduled maintenance has been completed..

Database Maintenance

Thu, 15 Jul 2021 15:30:00 +0000

Type: Maintenance

Duration: 30 minutes

Jul 15, 10:24:41 GMT+0
Identified - During this time, I will be performing some maintenance on the production database which will require Fire & Aether to be offline for a (hopefully) short period of time. I have created a script to make all the changes needed and have tested it locally with no issues and will be using it on the staging database soon. While this maintenance is set to last 30 minutes, I definitely do not expect it to. That is to allow some leeway for any hiccups that may (and hopefully will not) occur..

Jul 15, 15:50:09 GMT+0
Completed - The scheduled maintenance has been completed..

Jul 15, 15:30:33 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Jul 15, 15:33:12 GMT+0
Identified - Due to changes in scheduling for another project, the maintenance has been moved up an hour. I am about to shut down Fire & Aether so I can take a backup the database, run the script (which worked flawlessly locally and on staging) and tweak a few other things. The downtime should not last long..

Jul 15, 15:40:13 GMT+0
Identified - Fire & Aether are back online and everything seems to have gone smoothly. I will monitor logs for a couple minutes to make sure everything is working as it should be.

VPS Upgrades

Wed, 17 Mar 2021 13:30:00 +0000

Type: Maintenance

Duration: 30 minutes

Mar 17, 13:31:09 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Mar 17, 13:35:41 GMT+0
Identified - I am beginning to take services safely offline so I can power off the VPS without data loss. You will start to experience issues accessing services (502 bad gateway for web, silence for message commands on the bot and failed interactions for slash commands).

Mar 17, 13:42:31 GMT+0
Identified - The rescale has completed and services are coming back online. It may take a while for everything to be fully operational, so if you encounter any errors just wait a few minutes and try again..

Mar 17, 13:38:33 GMT+0
Identified - Everything has been shutdown and a rescale has been started. This should only take a few minutes.

Mar 17, 13:00:31 GMT+0
Identified - During this time, I will be upgrading Fire's VPS to Hetzner's CPX31 server type as it provides 2 more vCPUs and uses AMD Epyc 2nd Gen processors for better CPU performance rather than the current Xeon Gold processor. During this time, ALL services hosted on Fire's VPS (Fire itself, Aether, my haste instance and Status) will be offline as it must be powered off to rescale. This process should only take a few minutes and everything should automatically start back up again once it's done. The Fire website will remain online as it's hosted on Cloudflare Pages but will have some issues due to being unable to fetch data (e.g. for public servers, commands and stats).

Mar 17, 13:59:00 GMT+0
Completed - The scheduled maintenance has been completed..

Aether Update

Sat, 13 Feb 2021 19:00:00 +0000

Type: Maintenance

Duration: 1 hour

Feb 13, 19:13:54 GMT+0
Identified - Some issues have arisen while deploying the update to Fire. These should not affect functionality of the bot but if you do encounter any issues, it is likely these issues. I am working on resolving them now!.

Feb 13, 19:11:59 GMT+0
Identified - Aether is back online, deploying the Fire update now.

Feb 13, 19:10:38 GMT+0
Identified - I am about to deploy the update to Aether which will involve taking Fire offline as there are breaking changes to the identify process. Once Aether is back online, I will be able to deploy the Fire update, including the ecosystem file changes.

Feb 13, 19:07:27 GMT+0
Identified - Another change coming from this Aether update is the switch from gateway to rest for slash commands. Slash commands will stop working on Fire in a moment as I switch it over and will resume functioning once the update to Aether has been deployed.

Feb 13, 19:03:24 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Feb 13, 17:59:31 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Feb 13, 10:19:54 GMT+0
Identified - During this time, I will be deploying an update to Aether which will remove the reliance on pm2's instance id for determining which process is which cluster. Instead, it will just assign the first available cluster id. This has been tested locally and has worked flawlessly but as always issues may arise in production for the sole reason of it being in production so I will have an instance of Fire that doesn't connect to Aether ready to boot up if any issues occur to minimise downtime. I will also be deploying a small change to Fire's ecosystem file (https://pm2.keymetrics.io/docs/usage/application-declaration/) that adds an exponential backoff for restarts if an error occurs and adds a max memory restart of 1GB..

Feb 13, 20:08:12 GMT+0
Identified - I have fixed the issue with stats but I am going to need to take Aether offline for a minute or two as I'd like to clear influx to get rid of the mess that was written to it. Slash commands, reminders, carbon and potentially more features will be affected.

Feb 13, 20:59:06 GMT+0
Completed - Just deployed some replay socket & DM slash command fixes. Everything seems good 👍.

Feb 13, 20:14:34 GMT+0
Identified - I am once again verifying the status of the maintenance. I sincerely apologise for the mess this maintenance has been but there were a few oversights in the update that caused some issues in production (mainly the lack of influx as I do not run an instance of influx when running Fire locally). Stats are being reported correctly now and there does not seem to be any issues but I will leave this open for a while longer as the stats issue took a while to show itself..

Feb 13, 19:18:07 GMT+0
Identified - Verification is currently underway for the maintenance items..

Feb 13, 19:32:19 GMT+0
Identified - It seems that stats tracking was broken in the process, leading to some unintended side effects. A fix is being worked on..

Deploying rewrite of Fire 👀

Tue, 19 Jan 2021 16:00:00 +0000

Type: Maintenance

Duration: 2 hours

Jan 19, 16:46:46 GMT+0
Identified - The website has been taken offline so I can pull changes. You may see 502 errors when attempting to access fire.gaminggeek.dev & fire-ts.gaminggeek.dev.

Jan 19, 16:34:44 GMT+0
Identified - Fire is online and running the rewrite! 🎉 It seems there are some latency issues though so I am investigating those as we speak..

Jan 19, 17:24:30 GMT+0
Completed - It is done. Fire is now running the rewrite, the website has been updated to reflect the change to Aether and the bot invite (inv.wtf/bot) has been updated to request applications.commands :D Hope everyone enjoys the new Fire. If you find any issues, please let me know in the support server, https://inv.wtf/fire.

Jan 19, 16:29:41 GMT+0
Identified - I am about to take Fire offline to deploy the rewrite. It should hopefully only take less than a minute to switch over but if issues occur, they may prolong the deploy..

Jan 19, 16:15:19 GMT+0
Identified - Aether is back online, ready to be used with the main bot..

Jan 19, 15:00:34 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Jan 18, 16:07:59 GMT+0
Identified - During this time I will be deploying the TypeScript rewrite of Fire on the main bot. Started on the 2nd of September, it is finally finished (ok well technically it's like 99% finished but I just wanna get rid of Python Fire) I will do my best to make a full changelog on https://kb.gaminggeek.dev/ but there's a lot that has changed so it may take a few days for it to be posted. During this time, Fire Beta & Fire will go offline and only one will come back online. Fire Beta will eventually be made into, as the name implies, a beta bot. Running on a separate database (with a clean start though I will transfer data if requested but some data will not be transferrable), it will be a way for users to test upcoming features before they come to the main bot. I hope for this transition to be as seamless as possible but issues may arise so please beware that things might break for a short while afterwards. I will be fixing any bugs that may appear as soon as possible and getting fixes deployed once they're made. Can't wait for you guys to see what I've been working on for the past 4 months :D.

Jan 19, 16:51:38 GMT+0
Identified - fire.gaminggeek.dev has been restored. fire-ts.gaminggeek.dev will not be brought online as it was only for testing the website with the rewrite. It seems the deployment is complete and I am currently verifying that everything is working as it should be!.

Jan 19, 16:11:43 GMT+0
Identified - Fire Beta & Aether have been taken offline so I can make changes to Aether. Fire Beta WILL NOT come back online. It will be offline for a while after this until I am ready to deploy it as a beta version of Fire, including a separate database..

Jan 19, 16:00:43 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Changes to member caching

Wed, 2 Dec 2020 00:30:00 +0000

Type: Maintenance

Duration: 15 minutes

Dec 2, 00:33:43 GMT+0
Identified - Scheduled maintenance is still in progress. We will provide updates as necessary..

Dec 2, 00:46:09 GMT+0
Completed - The scheduled maintenance has been completed..

Dec 2, 00:13:39 GMT+0
Identified - During this time I will be deploying an update that affects how members are cached, causing other features to be affected too, some in a positive way and others negatively. Due to these changes, I have had to remove role update logging & nickname update logging. These relied on members being cached due to Discord not providing the previous state. They may return in the rewrite but it is unclear at the moment. A positive side affect to this change is that I've made autodecancer/autodehoist function similarly to the rewrite where members don't need to be cached for them to be decancered/dehoisted. It will also attempt to run the checks on each message too. If, after this update is deployed, you encounter issues related to autodecancer or autodehoist, please let me know. I have done some testing but my one-user test may not account for all scenarios..

Dec 2, 00:30:33 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

System Updates

Tue, 20 Oct 2020 04:30:00 +0000

Type: Maintenance

Duration: 45 minutes

Oct 20, 04:43:18 GMT+0
Identified - Services are coming back online, starting with Fire.

Oct 20, 04:38:47 GMT+0
Identified - I am rebooting again to try and resolve issues with pm2.

Oct 20, 05:38:52 GMT+0
Completed - The scheduled maintenance has been completed..

Oct 20, 04:11:56 GMT+0
Identified - During this time, I will be performing some much needed updates on Fire's VPS which will require a full system reboot, causing ALL SERVICES TO BE OFFLINE! This downtime shouldn't be long as it shouldn't take more than 5 minutes to reboot and get everything up and running (since 99.999999% of things will start automagically once the VPS is back online) but issues may arise. If any issues occur, I will be sure to update this incident (and I will try and self-host my Status bot so that the status updates in the Fire Discord continue to be updated).

Oct 20, 04:58:33 GMT+0
Identified - Verification is currently underway for the maintenance items..

Oct 20, 04:30:36 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Oct 20, 04:34:50 GMT+0
Identified - It seems pm2 didn't want to start up everything once the VPS rebooted. Manually starting everything now!.

Database Maintenance

Sun, 13 Sep 2020 13:10:00 +0000

Type: Maintenance

Duration: 1 hour

Sep 13, 13:10:35 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Sep 13, 14:31:02 GMT+0
Completed - The scheduled maintenance has been completed..

Sep 13, 13:38:32 GMT+0
Identified - Fire has converted all configs for current guilds. Everything should be working as normal now. I am about to convert configs for guilds that have removed Fire just in case they re-add it ;).

Sep 13, 14:27:24 GMT+0
Identified - Verification is currently underway for the maintenance items..

Sep 13, 13:15:16 GMT+0
Identified - I have altered as much as I can while keeping Fire online. For the rest, I'll need to take it offline. Hopefully this won't take long, I have the queries already, I just need to copy and paste them..

Sep 13, 13:24:59 GMT+0
Identified - Fire is back online but some features may be unavailable while configs are being converted. If your server's config hasn't been converted, Fire should let you know and link to this incident..

Sep 13, 12:08:39 GMT+0
Identified - Fire will be undergoing maintenance to convert data types in it's database from bigint to text for compatibility with the rewrite (as discord.js uses strings for ids) During this time, Fire may be offline while I work on the database so it doesn't write any data containing bigints while being converted. Configs will be converted automatically on startup once work on the database has been completed due to being handled differently. Precautionary measures have been taken to ensure no data is lost and the maintenance goes smoothly :).

Sep 13, 14:22:07 GMT+0
Identified - I will now be restarting the backend for inv.wtf to update types.

Sep 13, 14:15:53 GMT+0
Identified - While converting configs for guilds Fire is no longer in, I noticed an option that didn't get converted before so alongside guilds fire isn't in, if you have set excluded.filter ($filterexcl command) then your config will be converted again..

Deploying update to inv.wtf

Wed, 26 Aug 2020 01:00:00 +0000

Type: Maintenance

Duration: 25 minutes

Aug 26, 00:17:34 GMT+0
Identified - During this time I will be deploying an update to inv.wtf (Vanity URLs & Redirects) that adds some cool graphs for me to look at but will now track clicks & links for redirects instead of just vanity's and also track referrers. You will be able to view clicks & links for all your redirects in Fire soon, most likely by the end of the week. Viewing referrers for vanity's & redirects will come some time after that..

Aug 26, 01:13:27 GMT+0
Completed - The scheduled maintenance has been completed..

Aug 26, 01:00:47 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Aug 26, 01:02:43 GMT+0
Identified - Vanity URLs & Redirects may be unavailable for a moment while I deploy the update so you may receive a 502 Bad Gateway error..

Aug 26, 01:04:38 GMT+0
Identified - Everything should be accessible again. If you encounter any errors, please let me know in the Fire Discord, https://inv.wtf/fire.