Fire - Members not being cached – Incident details

All systems operational

Members not being cached

Resolved
Major outage
Started over 4 years agoLasted about 1 hour
Updates
  • Update
    Update

    The root cause behind the issue was the deployment of Fire’s TypeScript rewrite for premium guilds. The way the rewrite had been saving configs added extra quotes around the config causing normal Fire to be unable to parse the JSON. While trying to reload a config for a premium guild, it panicked due to being unable to parse it and rebooted and because it rebooted, it had to load all the configs again bringing everything back to the issue causing it to hit an error and stop. The fix for the issue was to make the TypeScript rewrite INSERT/UPDATE configs as is, no JSON.stringify and it seems to have worked but there’s still gonna be _some issues_ with consistency due to both Python & TypeScript bots running on the same database causing configs to become desynced but that’s only for the few premium guilds and it’s just meant to be a beta test anyways so any inconsistencies are fine.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been deployed and I am now monitoring the situation

  • Identified
    Identified

    The issue has been identified and a fix is being made

  • Update
    Update

    A temporary patch has been put in place to allow configs to load. It is still unknown why they are failing to be parsed when loading.

  • Update
    Update

    It seems configs failed to be parsed and were left as strings causing a bunch of other tasks to fail. I am investigating the cause.

  • Investigating
    Investigating

    It seems Fire ran into an issue with configs, rebooted and because of said issue, members are not being cached in guilds where they should be. This will affect logging, autodecancer/autodehoist and more