Still seeing intermittent timeouts

Incident Report for Palmetto

Resolved

We think the impact from the issue is over.

Posted Jan 24, 2024 - 14:45 UTC

Investigating

Manually bumped the max allowable database connections from each pod. Hoping this will keep the API usable until the migration is finished.

Posted Jan 24, 2024 - 06:56 UTC

Monitoring

Error rate is declining. I have been able to test all data endpoints in the API without timeout errors. Seems the in progress migration has completed or is no longer consuming too many resources.

Posted Jan 24, 2024 - 06:21 UTC

Investigating

All GLIDE endpoints which query data from the database are generally timing out and are thus down. I have been able to get a few successful requests through. Currently investigating the cause of the timeouts.

Posted Jan 24, 2024 - 05:48 UTC