Fire - Notice history

This incident has been open for a few days, marking as resolved

so uh, turns out I am a dumbass and due to my lack of braincells I am the reason why Fire left all of those guilds basically, I ran the TypeScript rewrite of Fire on the main token to check resource usage with all 631 guilds and since the hosted instance of the rewrite is for premium guilds only, I had "premiumOnly" set to true meaning it left guilds that weren't premium. It thankfully didn't get down to just premium guilds but boy am I pissed that I didn't realize what happened sooner. I sincerely apologize for my stupidity. To make it up to y'all, I will make Fire TS available for non-premium guilds too as soon as I can get this update deployed correctly. This incident will stay open for another little while to allow people to see the update.

The root cause behind the issue was the deployment of Fire’s TypeScript rewrite for premium guilds. The way the rewrite had been saving configs added extra quotes around the config causing normal Fire to be unable to parse the JSON. While trying to reload a config for a premium guild, it panicked due to being unable to parse it and rebooted and because it rebooted, it had to load all the configs again bringing everything back to the issue causing it to hit an error and stop. The fix for the issue was to make the TypeScript rewrite INSERT/UPDATE configs as is, no JSON.stringify and it seems to have worked but there’s still gonna be _some issues_ with consistency due to both Python & TypeScript bots running on the same database causing configs to become desynced but that’s only for the few premium guilds and it’s just meant to be a beta test anyways so any inconsistencies are fine.

This incident has been resolved.

A fix has been deployed and I am now monitoring the situation

The issue has been identified and a fix is being made

A temporary patch has been put in place to allow configs to load. It is still unknown why they are failing to be parsed when loading.

It seems configs failed to be parsed and were left as strings causing a bunch of other tasks to fail. I am investigating the cause.

This incident has been resolved.

This incident has been resolved.

There has been no further errors since so I am marking this as resolved

It seems Fire is no longer getting 502's when sending messages meaning the issue was most likely linked to Discord's ongoing incident as the error rate calmed down after the most recent update. I am continuing to monitor for any more errors

This incident has been resolved.

From http://stspg.io/fbk53trwrfsz: "As the major provider that was experiencing issues has implemented a fix in their network, other global network providers have started to re-enable their peering sessions with it. We are continuing to see improved reachability and monitoring the situation."

Fire still seems to be unaffected by this incident. If you notice any issues feel free to drop a message in the Fire support server, https://inv.wtf/fire :) Cloudflare are continuing to engage with their partners and fixes are being implemented with other transit providers to resolve the ongoing issues. You can find more info about this incident at the following sites; https://www.cloudflarestatus.com/incidents/hptvkprkvp23 https://status.ctl.io/history/f19a0555-abbd-4038-91cb-b55a7645c1f5 https://discordstatus.com/incidents/dv69h597l1z2

This incident has been resolved.

It seems that the command is working once again (even though I changed nothing) so I have re-enabled it. I am monitoring for errors to ensure it's actually working correctly.

Seems everything is working fine now. If you're wondering what the issue was, Centra seems to send requests with an empty User Agent if you don't set one and I recently modified my firewall to block empty User Agents due to some attacks. Once that issue was resolved, another issue appeared, specifically involving permission checking. The API checks if the guild has enabled the public feature in Fire with the "$public" command or has the DISCOVERABLE feature but this seemed to be failing. Switching from bigints to strings seems to have resolved it though.

A fix has been implemented and we are monitoring the results.

While attempting to apply a patch, another issue has come up and is being investigated. I should have more details soon.

The issue has been identified and a temporary fix has been put in place. A full patch is being worked on.

This incident has been resolved.

I have located the cause of the issue and am working on a fix, this shouldn't take too long