Fire - Services unavailable – Incident details

All systems operational

Services unavailable

Resolved
Major outage
Started over 4 years agoLasted about 1 hour
Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    I am monitoring to ensure everything has fully recovered.

  • Update
    Update

    Fire's VPS is no longer ratelimted, working on restoring services to full capacity now

  • Update
    Update

    Cluster 1 is online, meaning the bot should basically be operational apart from db changes not saving once it goes back to running on the VPS. Due to issues connecting to the remote database, I will not be able to bring up a local instance of inv.wtf meaning Vanity URLs will still be broken.

  • Update
    Update

    Cluster 0 is running locally with seemingly no issues. Beware, any changes that involve the database will NOT BE SAVED due to it being a local copy of the database. I will work on getting cluster 1 online now.

  • Update
    Update

    Attempts to bring Fire online locally have not worked due to issues with my local database and connecting to the prod database remotely. I will continue to try bring Fire back online locally but it may come down to just waiting out the ratelimit (there's about 25 minutes left)

  • Identified
    Identified

    Upon further investigation, it seems Fire's VPS has gotten hit with a 1 hour ban from the API, which means this affects more than just the bot. I am working on attempting to bring up instances of affected services locally to minimise the downtime.

  • Investigating
    Investigating

    Seems Fire has gone unresponsive. I am investigating the cause as we speak. Cluster 0 seems to be down completely but cluster 1 seems to at least be receiving gateway events.