Fire - Notice history

Sentry API experiencing degraded performance

Bot - Operational

100% - uptime
Dec 2025 · 99.80%Jan 2026 · 100.0%Feb · 99.91%

Previous page

Next page

Dec 2025
Jan 2026
Feb 2026

Beta Bot - Operational

100% - uptime
Dec 2025 · 100.0%Jan 2026 · 100.0%Feb · 99.99%

Previous page

Next page

Dec 2025
Jan 2026
Feb 2026

Website - Operational

100% - uptime
Dec 2025 · 99.97%Jan 2026 · 100.0%Feb · 99.99%

Previous page

Next page

Dec 2025
Jan 2026
Feb 2026

Notice history

Feb 2026

Networking Issues
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning

  • Investigating
    UTC
    Investigating

    It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost

    It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console

Infrastructure Upgrades
  • Completed
    February 18, 2026 at 1:25 PMUTC
    Completed
    February 18, 2026 at 1:25 PMUTC

    Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days

  • Update
    February 18, 2026 at 12:05 PMUTC
    Update
    February 18, 2026 at 12:05 PMUTC

    Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventually resolve themselves

    Extensive review & monitoring will be taking place for the next hour or so to ensure nothing has been missed and to do some cleanup of older resources

  • Update
    February 18, 2026 at 11:28 AMUTC
    Update
    February 18, 2026 at 11:28 AMUTC

    Fire is now back online and now running in Kubernetes!

    Next up will be reconfiguring DNS records which may take some time due to SSL certificate provisioning. This means that web services will still be unavailable for a while

  • In progress
    February 18, 2026 at 11:00 AMUTC
    In progress
    February 18, 2026 at 11:00 AMUTC
    Maintenance is now in progress
  • Planned
    February 18, 2026 at 11:00 AMUTC
    Planned
    February 18, 2026 at 11:00 AMUTC

    During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like environment by moving Fire Beta to a separate machine for testing & learning.

    The main upgrade should realistically only take 15-20 minutes with most of the time being spent replacing updating configurations for services exposed to the web as they can no longer be simply pointed to localhost & some port and even then, I don't expect this to take the entire duration of the maintenance but I'd rather schedule a larger window just in case

    For anyone interested, I am moving Fire away from running directly on the machine and being managed via PM2 to using Docker containers & Kubernetes. This move will make it much easier to keep Fire up to date as upgrading the OS or Node version is as simple as changing the image used. It also comes with a more maintainable solution for offsite backups as I've configured the Postgres cluster with automatic backups to CloudFlare R2 rather than moving them to my Mac every so often

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Jan 2026

Server Upgrades
  • Completed
    January 19, 2026 at 12:44 PMUTC
    Completed
    January 19, 2026 at 12:44 PMUTC

    Maintenance has completed successfully.... almost.

    During the upgrades, Fire's postgres installation got completely removed along with all the data. Fortunately, I have backups, but only for the main bot. Fire Beta has had its database recreated from scratch so no previous data will be present there so if you were relying on that, you shouldn't have!

    Aside from that, everything looks to be in working order. NodeJS has been updated to the latest LTS release, nginx has been updated to the latest release, postgres has been updated to the latest release and internal tooling (e.g. Grafana, Telegraf) has also been updated

    I'll be keeping a close eye on things throughout today to ensure things continue running smoothly. If you encounter any issues, let me know in the Fire Discord (discord.gg/firebot)

  • Update
    January 19, 2026 at 10:49 AMUTC
    Update
    January 19, 2026 at 10:49 AMUTC

    The snapshot has been taken and the software side of the upgrades is now in progress

  • Update
    January 19, 2026 at 10:17 AMUTC
    Update
    January 19, 2026 at 10:17 AMUTC

    The hardware of the server has been upgraded slightly (more vCPUs, more RAM & more storage) and I am now taking a snapshot of the server that I hopefully won't have to use but is there just in case anything goes wrong

  • Update
    January 19, 2026 at 10:07 AMUTC
    Update
    January 19, 2026 at 10:07 AMUTC

    Redirects have now been put in place to redirect any traffic to Fire's web services towards a dedicated maintenance page. I will now be turning off all services to start the upgrades

  • In progress
    January 19, 2026 at 10:00 AMUTC
    In progress
    January 19, 2026 at 10:00 AMUTC
    Maintenance is now in progress
  • Planned
    January 19, 2026 at 10:00 AMUTC
    Planned
    January 19, 2026 at 10:00 AMUTC

    During this time, several upgrades (both hardware & software) will be performed on Fire's VPS which will result in some lengthy downtime of ALL SERVICES.

    This downtime is unfortunate but necessary to ensure stability, security and open the door for new features that require newer software versions

    During this time, I will be keeping an eye on the Fire Discord (discord.gg/firebot) to answer any questions and let people know the current status of the upgrades.

    I'll also be around for the next while to answer questions about this maintenance if you have any so feel free to join the Discord server and ask in the #fire-help channel

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Dec 2025

Internal Restructuring
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Update
    UTC
    Update

    As part of some internal restructuring, it seems that the websocket connection to Aether is not working correctly, preventing the bot from launching correctly.

    I'm currently investigating this and hope to get this resolved as soon as possible

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
  • Resolved
    UTC
    Resolved

    The issue seems to have stemmed from Fire being stuck in a loop, constantly making requests to edit a channel, which resulted in it getting ratelimited. As far as I can tell, the edits were part of the permission role feature, so I've made a few changes to it which should a) help identify it and b) help resolve the issue

    This feature is pretty old so it likely needs a rewrite to ensure things work smoothly, which has been added to my lengthy to-do list for Fire

    For now, Fire is operational and should stay that way. If you encounter any issues, please report them in the Fire Discord (discord.gg/firebot)

  • Monitoring
    UTC
    Monitoring

    Fire is now back online and looks to be operating just fine. I've made some changes that may help prevent the issue re-occurring but I'll be looking further into what caused this issue so that I can properly resolve it

  • Update
    UTC
    Update

    Fire's VPS appears to have been blocked from accessing Discord's API due to exceeding ratelimits. It's unknown how this occurred at this time but for now, the bot has been turned off to avoid making requests while ratelimited.

    It looks to be only an hour long block with about 50 minutes remaining so it shouldn't be down for long but unfortunately there's nothing that can be done other than waiting

  • Investigating
    UTC
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Dec 2025 to Feb 2026

Next