Vector database backup issues on GCP

Incident Report for DataStax Astra

Resolved

This incident has been resolved.
Posted Dec 14, 2023 - 19:56 UTC

Monitoring

Backup functionality has been restored and we are monitoring the situation.
Posted Dec 14, 2023 - 14:38 UTC

Update

Commitlog replay functionality has been fully restored. No data has been lost and consistency should be restored for all affected databases. We are working on restoring the hourly backup service now.
Posted Dec 14, 2023 - 13:51 UTC

Update

The workaround has been applied successfully. We are working on restoring the normal operations now. This should be completed within the next hour.
Posted Dec 14, 2023 - 13:07 UTC

Update

We are continuing work on the resolution.
Posted Dec 13, 2023 - 19:09 UTC

Identified

We identified the root cause of the issues and are working on a resolution.
Posted Dec 13, 2023 - 14:31 UTC

Investigating

We are currently having issues with our snapshotting and replaying mechanisms on Vector databases in GCP. Consequently, some databases may currently face temporary data inconsistencies and may be without regular backups.
We are investigating this issue.
Posted Dec 13, 2023 - 13:02 UTC
This incident affected: Astra (Astra Serverless DB).