Post mortem completed
Actions identified for resolution:
Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment.
Resolve the issue with the alarms on the customer environment that delayed the response.
Resolved
Post mortem completed
Actions identified for resolution:
Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment.
Resolve the issue with the alarms on the customer environment that delayed the response.
Monitoring
Customer notified that issue has been resolved and logging in should be operational again.
Monitoring
Deployment successful, impacted environment fully operational again.
Investigating
Preparing a fix for the migration and re-deploying to the impacted environment.
Investigating
The issue reported was caused by a recent deployment. It was identified as a minor database migration that failed silently but didn't block the deployment. The successfully deployed code then caused issues preventing the user from logging in.
Only a single customer environment was impacted by the issue, the deployment across other environments, including the main app.humanloop.com, was successful with no issues.
Investigating
We provided first response to the user in Slack acknowledging the issue and starting the investigation.
Identified
User reported issue in Slack with failed SSO when attempting to log in.