Inconsistent platform functionality – Dec 28

[christina]

Between around 17:20-19:55 UTC on Mon, a scaling malfunction of the main database has led to repeated inadequacies in functionality of multiple services.

Among other symptoms, wordpress backends have been at times unable to register or start new events, certain ongoing live events have been turned off or discontinued, and streaming bandwidth consumption has not been accounted for. Overall this has affected an estimated 15-25% of customers.

Issue has been acknowledged immediately, however efforts to restore full functionality lasted more than an hour. During this time some services have been restarted or rerouted, leading to spotty downtime of the ‘my-account’ area and also the whole wpstream.net website in some regions.

Currently the scaling issue has been addressed by manually increasing capacity. Investigation into the main cause of the problem is still ongoing and we are manning the infrastructure until a definitive fix is in place.

We wish to apologize for the effects this has had on our customers. We realize how important the availability of the service is to you, we have learned important lessons from this and actively taking steps to prevent recurrence of similar issues.

[christina]

We have completely overhauled the database underlying layer, both hardware and scalability engine. Process is complete and not expecting any kind of similar issues. We thank you for your continued support and wish you a happy new year.

[Author]

Posts