Fire - Most if not all services unavailable – Incident details
All systems operational
Most if not all services unavailable
Resolved
Major outage
Started over 3 years agoLasted about 1 hour
Updates
Resolved
Resolved
This incident has been resolved.
Monitoring
Monitoring
Everything that was running is once again running (after pain with poetry), I am now going to check to make sure nothing has broken in the process
Update
Update
All services listed on the statuspage are up and running!
Working on internal services now. These services being down may affect some features in external services so you may encounter some errors.
Update
Update
Some services are back online, continuing to work on the rest!
Update
Update
Restoring from the PM2 dump was not successful so I will be manually bringing all processes back. This may take a while but I will try and start important processes first.
I apologise for the inconvenience
Update
Update
While working on restoring, I had noticed PM2 was still trying to use the old Node version. It's been a couple minutes and I have figured out why so restoring should hopefully not take too much longer!
Identified
Identified
While switching from nodesource to nvm for managing Node versions, PM2 was killed and did not restore the process list when restarting.
I am working on restoring everything now