Fire - Aether Update – Maintenance details

All systems operational

Aether Update

Completed
Scheduled for February 13, 2021 at 7:00 PM – 8:59 PM

Affects

Updates
  • Completed
    February 13, 2021 at 8:59 PM
    Completed
    February 13, 2021 at 8:59 PM

    Just deployed some replay socket & DM slash command fixes. Everything seems good 👍

  • Update
    February 13, 2021 at 8:14 PM
    In progress
    February 13, 2021 at 8:14 PM

    I am once again verifying the status of the maintenance. I sincerely apologise for the mess this maintenance has been but there were a few oversights in the update that caused some issues in production (mainly the lack of influx as I do not run an instance of influx when running Fire locally). Stats are being reported correctly now and there does not seem to be any issues but I will leave this open for a while longer as the stats issue took a while to show itself.

  • Update
    February 13, 2021 at 8:08 PM
    In progress
    February 13, 2021 at 8:08 PM

    I have fixed the issue with stats but I am going to need to take Aether offline for a minute or two as I'd like to clear influx to get rid of the mess that was written to it. Slash commands, reminders, carbon and potentially more features will be affected

  • Update
    February 13, 2021 at 7:32 PM
    In progress
    February 13, 2021 at 7:32 PM

    It seems that stats tracking was broken in the process, leading to some unintended side effects. A fix is being worked on.

  • Update
    February 13, 2021 at 7:18 PM
    In progress
    February 13, 2021 at 7:18 PM

    Verification is currently underway for the maintenance items.

  • Update
    February 13, 2021 at 7:13 PM
    In progress
    February 13, 2021 at 7:13 PM

    Some issues have arisen while deploying the update to Fire. These should not affect functionality of the bot but if you do encounter any issues, it is likely these issues. I am working on resolving them now!

  • Update
    February 13, 2021 at 7:11 PM
    In progress
    February 13, 2021 at 7:11 PM

    Aether is back online, deploying the Fire update now

  • Update
    February 13, 2021 at 7:10 PM
    In progress
    February 13, 2021 at 7:10 PM

    I am about to deploy the update to Aether which will involve taking Fire offline as there are breaking changes to the identify process. Once Aether is back online, I will be able to deploy the Fire update, including the ecosystem file changes

  • Update
    February 13, 2021 at 7:07 PM
    In progress
    February 13, 2021 at 7:07 PM

    Another change coming from this Aether update is the switch from gateway to rest for slash commands. Slash commands will stop working on Fire in a moment as I switch it over and will resume functioning once the update to Aether has been deployed

  • In progress
    February 13, 2021 at 7:03 PM
    In progress
    February 13, 2021 at 7:03 PM

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • Update
    February 13, 2021 at 5:59 PM
    Planned
    February 13, 2021 at 5:59 PM

    Maintenance will begin as scheduled in 60 minutes.

  • Planned
    February 13, 2021 at 10:19 AM
    Planned
    February 13, 2021 at 10:19 AM

    During this time, I will be deploying an update to Aether which will remove the reliance on pm2's instance id for determining which process is which cluster. Instead, it will just assign the first available cluster id. This has been tested locally and has worked flawlessly but as always issues may arise in production for the sole reason of it being in production so I will have an instance of Fire that doesn't connect to Aether ready to boot up if any issues occur to minimise downtime. I will also be deploying a small change to Fire's ecosystem file (https://pm2.keymetrics.io/docs/usage/application-declaration/) that adds an exponential backoff for restarts if an error occurs and adds a max memory restart of 1GB.