Fire - Most if not all services unavailable – Incident details

All systems operational

Most if not all services unavailable

Resolved
Major outage
Started over 3 years agoLasted about 1 hour
Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Everything that was running is once again running (after pain with poetry), I am now going to check to make sure nothing has broken in the process

  • Update
    Update

    All services listed on the statuspage are up and running! Working on internal services now. These services being down may affect some features in external services so you may encounter some errors.

  • Update
    Update

    Some services are back online, continuing to work on the rest!

  • Update
    Update

    Restoring from the PM2 dump was not successful so I will be manually bringing all processes back. This may take a while but I will try and start important processes first. I apologise for the inconvenience

  • Update
    Update

    While working on restoring, I had noticed PM2 was still trying to use the old Node version. It's been a couple minutes and I have figured out why so restoring should hopefully not take too much longer!

  • Identified
    Identified

    While switching from nodesource to nvm for managing Node versions, PM2 was killed and did not restore the process list when restarting. I am working on restoring everything now