Fire - Services Offline / Unavailable – Incident details

All systems operational

Services Offline / Unavailable

Resolved
Partial outage
Started almost 4 years agoLasted 37 minutes
Updates
  • Resolved
    Resolved

    Fire & Aether are now operational.

  • Update
    Update

    Vanity URLs & Redirects are now operational

  • Update
    Update

    Services are being brought back online, starting with Status, which should be up and running to send this update.

  • Identified
    Identified

    It seems Fire's VPS has gotten a temp Cloudflare ban which caused Fire and other services to be offline from at least 3:15 UTC. I have partially recovered Fire & Fire Beta, they are currently running locally for the duration of the ban and will be moved back to the VPS once the ban is lifted (which should be ~4:05 UTC) and I will restore other affected services like Status & inv.wtf too The ban seems to have happened while trying to sync the mute role in a server. Somehow Fire got stuck in a request loop and kept sending requests to update the channel permissions of that server even though they had already been updated resulting in 429s and eventually this ban. It is unknown why Fire got stuck in this loop and I will be investigating tomorrow (as it's currently 4:42 in the morning for me)