Inconsistent platform functionality – June 10
Tagged: downtime malfunction
- June 10, 2021 at 9:11 am #128747BeatriceModerator
Unfortunately, it appears that we have been the target of a cybernetic attack between 00:20 and 08:20 GMT during which the website, along with all functionalities were down.
The good news is that we are now back up and running as usual. Our sincerest apologies for the disruption this may have caused and wish to thank you for your patience.June 15, 2021 at 2:22 pm #129521ChristinaModerator
We’ve been through a similar attack today, starting around 9:05 UTC
Following the previous occurrence, we upgraded the infrastructure in hope it would withstand such brute force and also set up various alarms so as to be alerted of such events.
We managed to mitigate the event today and restore all services in under 30 minutes, however it impacted a a large number of live streams and other functionality.
Steps we’re taking to avoid similar happenings in the future:
* Further upgrade infrastructure and enhance scalability to withstand initial attacks – ongoing
* Specific filtering and watchdogs to avoid similar attacks – coming days
* Pinpoint potential vulnerabilities that made the attack possible – ongoing
* Patch the platform to allow continued functionality of live events in case of downtime or malfunction – next week
* Further isolate live events functionality from platform and API – coming weeks
We understand how critical our services are to our customers and extend our sincerest apologies to those who have been impacted. We are taking this matter very seriously and are actively sorting out ways to prevent similar attacks. In the mean time, you can always get in contact with us at email@example.com and we will get back to you ASAP.June 17, 2021 at 8:05 am #129740ChristinaModerator
We have determined that the attack has been carried out by executing malformed API calls that triggered resource consuming searches against the past live events database.
In between the disruptive reported attacks there has been another one on Sunday afternoon, lasting less than 30 minutes, which has been ‘absorbed’ by the upgraded infrastructure and did not cause service interruptions.
On the latest (early Tue morning), we observed requests originating at over 100 distinct IP addresses at rates ranging from a few to a few dozen per second each.
The vulnerability that allowed this series of attacks to be effective has meanwhile been identified and addressed. Despite, we are on our toes for a follow up attack and hope to be able to respond in minutes may it be the case.
Furthermore, medium and long term goals to avoid and mitigate similar happenings have been discussed. To mention the most relevant:
* Allow ongoing live broadcasts to continue unhindered in the case of an outage/malfunction of the client/channel/event management layer – in place since yesterday
* Allow continued playback of ongoing live events by viewers already attending the event in the case of an outage/malfunction of the client/channel/event management layer – planned for the next (expedited) release of the plugin
* Allow full functionality of ongoing live events, by attending and new viewers, in the case of an outage/malfunction of the client/channel/event management layer – planned for a future release of the plugin
* A new API (the underlying communication between the WpStream plugin and the WpStream platform) version is in the works and we expect to release it in autumn; its core is built with improved security in mind and this event has further opened our eyes to the increasing possibility of exploits and malware, thus we’re bound to pay extra attention to details and safeguard every routine
We have been prioritizing this matter ever since the initial attack and will continue doing so until we are confident that a similar attack will not take place again. We understand your frustrations and possible reconsiderations but we assure you that we will do every thing we can to prevent any future attacks from happening. We value our customers and their own customers and apologize for this ordeal.
- You must be logged in to reply to this topic.