Online schedulers that use the old widget code have now been fixed. This resolves all known issues relating to the downtime today.
Posted Mar 26, 2020 - 20:19 PDT
Update
We have discovered that the issue affecting online schedulers does not affect all calendars, only calendars that are using the old widget url. You can work around this issue by updating your widget code from the 'Appointment' -> 'Online Scheduling' page. We are still working on a permanent fix for this issue.
Posted Mar 26, 2020 - 18:13 PDT
Update
We have identified an issue with online schedulers due to what we think is a related problem with the certificate issue. We are currently working on a fix for this.
All other parts of Kalix are unaffected.
Posted Mar 26, 2020 - 18:05 PDT
Update
There was a secondary issue with validating the certificate which caused authentication issues after logging on, but this issue is also fixed. Kalix should now be working again.
We are monitoring the situation and will notify here if anything comes up.
Posted Mar 26, 2020 - 16:21 PDT
Monitoring
The fix to make sure that we find the correct certificate has been implemented, and was successful on one server. We are waiting for the changes to propagate to our other servers. Kalix should hopefully be working again in 10-15 mins.
Posted Mar 26, 2020 - 16:04 PDT
Identified
The issue has been identified and a fix is being deployed. The issue was due to an expired certificate which meant that the Kalix server could not start up. We actually already knew this certificate was due to expire, and had already setup and deployed a new certificate, so we are not sure how the old certificate stayed around.
Because the issue happened before the servers have even started, our monitoring tools did not pick up the issue, because the tools had not even started.
Posted Mar 26, 2020 - 15:59 PDT
Investigating
Around an hour ago an error occurred with an internal certificate that resulted in server issues. Unfortunately, our automated notification system did not alert us of this issue. We are currently investigating the cause of this problem. Thank you for your patience.
Posted Mar 26, 2020 - 15:32 PDT
This incident affected: Kalix Platform, Telehealth, Online Schedulers, Messaging, and Notifications.