Fire - Notice history

All systems operational

Bot - Operational

100% - uptime
Jan 2026 · 100.0%Feb · 99.91%Mar · 100.0%

Previous page

Next page

Jan 2026
Feb 2026
Mar 2026

Beta Bot - Operational

100% - uptime
Jan 2026 · 100.0%Feb · 99.99%Mar · 100.0%

Previous page

Next page

Jan 2026
Feb 2026
Mar 2026

Website - Operational

100% - uptime
Jan 2026 · 100.0%Feb · 99.99%Mar · 100.0%

Previous page

Next page

Jan 2026
Feb 2026
Mar 2026

Notice history

Mar 2026

No notices reported this month

Feb 2026

Networking Issues
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning

  • Investigating
    Investigating

    It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost

    It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console

Infrastructure Upgrades
  • Completed
    February 18, 2026 at 1:25 PM
    Completed
    February 18, 2026 at 1:25 PM

    Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days

  • Update
    February 18, 2026 at 12:05 PM
    Update
    February 18, 2026 at 12:05 PM

    Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventually resolve themselves

    Extensive review & monitoring will be taking place for the next hour or so to ensure nothing has been missed and to do some cleanup of older resources

  • Update
    February 18, 2026 at 11:28 AM
    Update
    February 18, 2026 at 11:28 AM

    Fire is now back online and now running in Kubernetes!

    Next up will be reconfiguring DNS records which may take some time due to SSL certificate provisioning. This means that web services will still be unavailable for a while

  • In progress
    February 18, 2026 at 11:00 AM
    In progress
    February 18, 2026 at 11:00 AM
    Maintenance is now in progress
  • Planned
    February 18, 2026 at 11:00 AM
    Planned
    February 18, 2026 at 11:00 AM

    During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like environment by moving Fire Beta to a separate machine for testing & learning.

    The main upgrade should realistically only take 15-20 minutes with most of the time being spent replacing updating configurations for services exposed to the web as they can no longer be simply pointed to localhost & some port and even then, I don't expect this to take the entire duration of the maintenance but I'd rather schedule a larger window just in case

    For anyone interested, I am moving Fire away from running directly on the machine and being managed via PM2 to using Docker containers & Kubernetes. This move will make it much easier to keep Fire up to date as upgrading the OS or Node version is as simple as changing the image used. It also comes with a more maintainable solution for offsite backups as I've configured the Postgres cluster with automatic backups to CloudFlare R2 rather than moving them to my Mac every so often

Potential disruption to Fire (Automatic)
  • Resolved
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Jan 2026

Server Upgrades
  • Completed
    January 19, 2026 at 12:44 PM
    Completed
    January 19, 2026 at 12:44 PM

    Maintenance has completed successfully.... almost.

    During the upgrades, Fire's postgres installation got completely removed along with all the data. Fortunately, I have backups, but only for the main bot. Fire Beta has had its database recreated from scratch so no previous data will be present there so if you were relying on that, you shouldn't have!

    Aside from that, everything looks to be in working order. NodeJS has been updated to the latest LTS release, nginx has been updated to the latest release, postgres has been updated to the latest release and internal tooling (e.g. Grafana, Telegraf) has also been updated

    I'll be keeping a close eye on things throughout today to ensure things continue running smoothly. If you encounter any issues, let me know in the Fire Discord (discord.gg/firebot)

  • Update
    January 19, 2026 at 10:49 AM
    Update
    January 19, 2026 at 10:49 AM

    The snapshot has been taken and the software side of the upgrades is now in progress

  • Update
    January 19, 2026 at 10:17 AM
    Update
    January 19, 2026 at 10:17 AM

    The hardware of the server has been upgraded slightly (more vCPUs, more RAM & more storage) and I am now taking a snapshot of the server that I hopefully won't have to use but is there just in case anything goes wrong

  • Update
    January 19, 2026 at 10:07 AM
    Update
    January 19, 2026 at 10:07 AM

    Redirects have now been put in place to redirect any traffic to Fire's web services towards a dedicated maintenance page. I will now be turning off all services to start the upgrades

  • In progress
    January 19, 2026 at 10:00 AM
    In progress
    January 19, 2026 at 10:00 AM
    Maintenance is now in progress
  • Planned
    January 19, 2026 at 10:00 AM
    Planned
    January 19, 2026 at 10:00 AM

    During this time, several upgrades (both hardware & software) will be performed on Fire's VPS which will result in some lengthy downtime of ALL SERVICES.

    This downtime is unfortunate but necessary to ensure stability, security and open the door for new features that require newer software versions

    During this time, I will be keeping an eye on the Fire Discord (discord.gg/firebot) to answer any questions and let people know the current status of the upgrades.

    I'll also be around for the next while to answer questions about this maintenance if you have any so feel free to join the Discord server and ask in the #fire-help channel

Potential disruption to Fire (Automatic)
  • Resolved
    Resolved

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • Investigating
    Investigating

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Jan 2026 to Mar 2026

Next