Extended Maintenance
Incident Report for Kalix EMR
Resolved
All message have finally been processed - all previously saved data should now be up to date. Kalix is functioning again at 100%.
Posted Oct 09, 2018 - 03:44 PDT
Update
We have updated the platform so that the big stream of data has been pushed off to a background queue. This means that Kalix will function as expected while the queue is finishing up.

However any data that was produced yesterday is also in this stream, so it may take a few hours for all the data to appear from yesterday.

We will give a final notice once the background queue is completely finished.
Posted Oct 09, 2018 - 01:10 PDT
Update
Unfortunately, in our attempts to improve the speed of the re-indexing, we inadvertently caused some secondary issues that increased the total time of the work.

The queue length is back on its way down now, and we will be measuring how it goes over the next hour to get an idea of what is left. We are just under a 1/4 of what we started with, to give an idea of how much is left to go. We are estimating a 10 hour completion time from now 12 midnight PST.
Posted Oct 08, 2018 - 13:11 PDT
Update
Unfortunately, the estimated completion time has been extended. We now calculate the maintenance will be completed by 2 pm PST Monday, October 8.

We are still investigating ways of making the maintenance run faster.

Kalix is functioning as normal with the exception of:

The indexing of new clients and contacts (new clients and contact appearing in searches)
The sending and receiving of messages e.g. appointment reminders, client responses
The sharing of documents e.g. online forms, faxes
The receiving of new notifications e.g. new appointment notifications, new message notifications
The creation of back-ups
The creation of insurance bill batches.

There is currently a two-hour delay in these tasks. The delay will continue to reduce over time as the maintenance proceeds.
Posted Oct 08, 2018 - 03:45 PDT
Update
Unfortunately the estimated timeframe for the completion of the maintenance was underestimated. We have made some improvements to the indexing so it should complete faster, but there is still a little over half of the remaining data to go.

After looking at how much data has been process, vs how much left to go, our updated maintenance time should be complete by 1am PST.
Posted Oct 07, 2018 - 13:07 PDT
Monitoring
We are migrating some data (reminders) and the move has taken longer than expected. Kalix is mostly usable but since the migration involves the indexing of records certain components are affected.

For example if you create a new client or contact, the new record will not be searchable until the maintenance is complete. Or if you send a message, it will be queued until maintenance is complete.

We expect that the maintenance will be complete by 1pm PST.
Posted Oct 07, 2018 - 05:42 PDT
This incident affected: Kalix Platform, Messaging, and Notifications.