After investigating this issue further and the incident two days ago, we think we may have identified an issue that caused continuous retries to load data which eventually overwhelmed the storage. We have made changes that should prevent this issue happening again, as we are using our newer database and caching as well to handle many more requests.
Posted Feb 14, 2024 - 23:17 PST
Monitoring
We saw that there was heavy usage on some of our servers, and when this was reset the problem was solved. We are looking into why these servers had suddenly high usage.
Posted Feb 14, 2024 - 10:01 PST
Update
We are continuing to investigate this issue.
Posted Feb 14, 2024 - 09:55 PST
Investigating
Unfortunately, Kalix is experiencing time out error. We are currently investigating the problem. There will be updates ASAP. Telehealth is unaffected.
Posted Feb 14, 2024 - 09:53 PST
This incident affected: Kalix Platform, Online Schedulers, Messaging, and Notifications.