Fire - Notice history

Beta Bot experiencing major outage

Bot - Operational

99.90% uptime
Dec 2025 · 99.80%Jan 2026 · 100.0%Feb · 99.91%

Previous page

Next page

Dec 2025
Jan 2026
Feb 2026

Beta Bot - Major outage

Website - Operational

99.99% uptime
Dec 2025 · 99.97%Jan 2026 · 100.0%Feb · 99.99%

Previous page

Next page

Dec 2025
Jan 2026
Feb 2026
Degraded performance

Discord API - Operational

Discord Realtime - Operational

Discord Media Proxy - Operational

Cloudflare (FRA) - Operational

Cloudflare Websockets - Operational

Cloudflare DNS - Operational

Cloudflare Firewall - Operational

Cloudflare DNS Updates - Degraded performance

GitHub API - Operational

GitHub Webhooks - Operational

GitHub - Git operations - Operational

Sentry API - Operational

Stripe API - Operational

Notice history

View current status

Feb 2026

Networking Issues
ResolvedMajor outage29 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Connectivity appears to have returned and services are now reporting as healthy. Active monitoring will continue for a period of time in the event of issues returning

  • InvestigatingUTC

    It seems Fire and related services are currently unavailable due to possible networking issues. Services are running but it seems that all connectivity on both machines (production & staging) has been lost

    It's unknown what exactly is causing this issue but my guess is Hetzner due to how both machines lost connectivity at the same time and I'm unable to access the Hetzner console

FEB18
Infrastructure Upgrades
CompletedMaintenance2 hours 25 minutes
  • CompletedFebruary 18, 2026 at 1:25 PMUTC

    Maintenance has completed successfully. There are a few issues with certain features/tooling but they're minor enough that I'm willing to let them be for now and will work on them within the next few days

  • UpdateFebruary 18, 2026 at 12:05 PMUTC

    Everything should be up and running now with all DNS records recreated with their new values. Some users may experience issues due to delays in DNS propagation and SSL certificate provisioning but these should eventually resolve themselves

    Extensive review & monitoring will be taking place for the next hour or so to ensure nothing has been missed and to do some cleanup of older resources

  • UpdateFebruary 18, 2026 at 11:28 AMUTC

    Fire is now back online and now running in Kubernetes!

    Next up will be reconfiguring DNS records which may take some time due to SSL certificate provisioning. This means that web services will still be unavailable for a while

  • In progressFebruary 18, 2026 at 11:00 AMUTC
    Maintenance is now in progress
  • PlannedFebruary 18, 2026 at 11:00 AMUTC

    During this time, we will be performing upgrades to Fire's infrastructure, changing how the bot and related applications are deployed/managed on the machine. These upgrades have already been tested in a production-like environment by moving Fire Beta to a separate machine for testing & learning.

    The main upgrade should realistically only take 15-20 minutes with most of the time being spent replacing updating configurations for services exposed to the web as they can no longer be simply pointed to localhost & some port and even then, I don't expect this to take the entire duration of the maintenance but I'd rather schedule a larger window just in case

    For anyone interested, I am moving Fire away from running directly on the machine and being managed via PM2 to using Docker containers & Kubernetes. This move will make it much easier to keep Fire up to date as upgrading the OS or Node version is as simple as changing the image used. It also comes with a more maintainable solution for offsite backups as I've configured the Postgres cluster with automatic backups to CloudFlare R2 rather than moving them to my Mac every so often

Potential disruption to Fire (Automatic)
ResolvedPartial outage1 hour 40 minutes
  • ResolvedUTC

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Jan 2026

JAN19
Server Upgrades
CompletedMaintenance2 hours 44 minutes
  • CompletedJanuary 19, 2026 at 12:44 PMUTC

    Maintenance has completed successfully.... almost.

    During the upgrades, Fire's postgres installation got completely removed along with all the data. Fortunately, I have backups, but only for the main bot. Fire Beta has had its database recreated from scratch so no previous data will be present there so if you were relying on that, you shouldn't have!

    Aside from that, everything looks to be in working order. NodeJS has been updated to the latest LTS release, nginx has been updated to the latest release, postgres has been updated to the latest release and internal tooling (e.g. Grafana, Telegraf) has also been updated

    I'll be keeping a close eye on things throughout today to ensure things continue running smoothly. If you encounter any issues, let me know in the Fire Discord (discord.gg/firebot)

  • UpdateJanuary 19, 2026 at 10:49 AMUTC

    The snapshot has been taken and the software side of the upgrades is now in progress

  • UpdateJanuary 19, 2026 at 10:17 AMUTC

    The hardware of the server has been upgraded slightly (more vCPUs, more RAM & more storage) and I am now taking a snapshot of the server that I hopefully won't have to use but is there just in case anything goes wrong

  • UpdateJanuary 19, 2026 at 10:07 AMUTC

    Redirects have now been put in place to redirect any traffic to Fire's web services towards a dedicated maintenance page. I will now be turning off all services to start the upgrades

  • In progressJanuary 19, 2026 at 10:00 AMUTC
    Maintenance is now in progress
  • PlannedJanuary 19, 2026 at 10:00 AMUTC

    During this time, several upgrades (both hardware & software) will be performed on Fire's VPS which will result in some lengthy downtime of ALL SERVICES.

    This downtime is unfortunate but necessary to ensure stability, security and open the door for new features that require newer software versions

    During this time, I will be keeping an eye on the Fire Discord (discord.gg/firebot) to answer any questions and let people know the current status of the upgrades.

    I'll also be around for the next while to answer questions about this maintenance if you have any so feel free to join the Discord server and ask in the #fire-help channel

Potential disruption to Fire (Automatic)
ResolvedPartial outage1 minute
  • ResolvedUTC

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Dec 2025

Internal Restructuring
ResolvedMajor outage1 hour
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • UpdateUTC

    As part of some internal restructuring, it seems that the websocket connection to Aether is not working correctly, preventing the bot from launching correctly.

    I'm currently investigating this and hope to get this resolved as soon as possible

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
ResolvedPartial outage5 minutes
  • ResolvedUTC

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
ResolvedPartial outage7 minutes
  • ResolvedUTC

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
ResolvedPartial outage46 minutes
  • ResolvedUTC

    Automated systems have detected new connections from previously unavailable clusters and all clusters are now connected. Fire should no longer have any availability issues

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Potential disruption to Fire (Automatic)
ResolvedMajor outage1 hour 57 minutes
  • ResolvedUTC

    The issue seems to have stemmed from Fire being stuck in a loop, constantly making requests to edit a channel, which resulted in it getting ratelimited. As far as I can tell, the edits were part of the permission role feature, so I've made a few changes to it which should a) help identify it and b) help resolve the issue

    This feature is pretty old so it likely needs a rewrite to ensure things work smoothly, which has been added to my lengthy to-do list for Fire

    For now, Fire is operational and should stay that way. If you encounter any issues, please report them in the Fire Discord (discord.gg/firebot)

  • MonitoringUTC

    Fire is now back online and looks to be operating just fine. I've made some changes that may help prevent the issue re-occurring but I'll be looking further into what caused this issue so that I can properly resolve it

  • UpdateUTC

    Fire's VPS appears to have been blocked from accessing Discord's API due to exceeding ratelimits. It's unknown how this occurred at this time but for now, the bot has been turned off to avoid making requests while ratelimited.

    It looks to be only an hour long block with about 50 minutes remaining so it shouldn't be down for long but unfortunately there's nothing that can be done other than waiting

  • InvestigatingUTC

    Automated systems have detected a prolonged issue with the connection to one or more Fire clusters. This may not necessarily mean the bot is down but it will at minimum result in features being unavailable

Previous

Dec 2025 to Feb 2026

Next