Fire - Config handling update – Maintenance details

Sentry API experiencing degraded performance

Config handling update

Completed
Scheduled for November 24, 2024 at 2:00 PM – 3:02 PMUTC

Affects

Bot

Under maintenance from 2:00 PM to 3:02 PM

Website

Under maintenance from 2:00 PM to 3:02 PM

Backend

Under maintenance from 2:00 PM to 3:02 PM

Aether

Under maintenance from 2:00 PM to 3:02 PM

Updates
  • Completed
    November 24, 2024 at 3:02 PMUTC
    Completed
    November 24, 2024 at 3:02 PMUTC

    Everything looks to be running perfectly and my small bit of testing on the production services has been successful so I consider this maintenance complete!

  • Update
    November 24, 2024 at 2:45 PMUTC
    Update
    November 24, 2024 at 2:45 PMUTC

    All clusters are now back online with the update deployed. I will be actively monitoring for the next 15 or so minutes to ensure everything is working as it should

  • Update
    November 24, 2024 at 2:36 PMUTC
    Update
    November 24, 2024 at 2:36 PMUTC

    The deploy to production Aether was successful. Fire's clusters will start coming back online now though there will be some spacing in-between the launch of each cluster just to ensure things are going smoothly

  • Update
    November 24, 2024 at 2:29 PMUTC
    Update
    November 24, 2024 at 2:29 PMUTC

    After a slight hiccup with one of the new additions to Aether, the deploy to the beta services has been successful.

    Fire & Aether will now be turned off to facilitate the deploy

  • Update
    November 24, 2024 at 2:01 PMUTC
    Update
    November 24, 2024 at 2:01 PMUTC

    Fire Beta & the matching beta instance of Aether will now be turned off to deploy the update to those two first

    The main bot will not currently be affected.

  • In progress
    November 24, 2024 at 2:00 PMUTC
    In progress
    November 24, 2024 at 2:00 PMUTC
    Maintenance is now in progress
  • Planned
    November 24, 2024 at 2:00 PMUTC
    Planned
    November 24, 2024 at 2:00 PMUTC

    An update will be releasing to Fire & Aether (the service which manages Fire's clusters) to change how configs are handled to make reading them from external services (such as the website) easier and allowing better syncing across clusters and better handling of failures to update the config

    These updates are closely linked so they cannot be updated seamlessly like most incremental changes and will require both services to be offline to update them. I do not anticipate this taking a long time but I've left in room for error (which hopefully will not happen 🤞) just in case