Visual Radio Assist Status - Incident history https://status.visualradio.cloud Visual Radio Assist Thu, 1 Oct 2026 23:00:00 +0000 Cloud Infrastructure Routine Maintenance Type: Maintenance Affected Components: api.visualradio.cloud, Core, live.visualradio.cloud Sep 29, 08:51:14 GMT+0 - Identified - We are planning for a scheduled maintenance during the night to make sure all our Cloud systems stay up to date with latest performance and security updates. There is **no downtime** expected for this timeframe (2 October 1:00 - 2:00 GMT+2), but affected components may be slower than expected due to Load Balancing and scaling maintenance. Type: Maintenance

Affected Components: , ,

<p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>08:51:14</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during the night to make sure all our Cloud systems stay up to date with latest performance and security updates. There is **no downtime** expected for this timeframe (2 October 1:00 - 2:00 GMT+2), but affected components may be slower than expected due to Load Balancing and scaling maintenance..</p> ]]> Thu, 1 Oct 2026 23:00:00 +0000 https://status.visualradio.cloud/maintenance/cmumfst8x037h13obq65omr77 https://status.visualradio.cloud/maintenance/cmumfst8x037h13obq65omr77 Cloud unavailable: Upstream provider load balancing issues Type: Incident Duration: 1 hour and 59 minutes Affected Components: live.visualradio.cloud, api.visualradio.cloud Dec 20, 13:41:07 GMT+0 - Identified - We've identified an issue with our upstream provider and the availability of a critical load balancer of our infrastructure. Due to the unavailable Load Balancer the Cloud API component is currently inaccessible. All running VRA Clients (Core, Output Player, Audio Manager) will keep running. We will be providing status updates every 30 minutes on the incident. Dec 20, 14:11:43 GMT+0 - Identified - We are continuing to work on a fix for this incident together with our upstream provider. Dec 20, 14:17:09 GMT+0 - Monitoring - We're activating a backup solution, which is impacted by DNS propagation delay but is verified to be working successfully. Dec 20, 14:22:00 GMT+0 - Resolved - We've successfully replaced the faulty components of the cloud infrastructure. The full Cloud service is fully recovered since 15:14 UTC+1 Type: Incident

Duration: 1 hour and 59 minutes

Affected Components: ,

<p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>13:41:07</var> GMT+0</small><br><strong>Identified</strong> - We've identified an issue with our upstream provider and the availability of a critical load balancer of our infrastructure. Due to the unavailable Load Balancer the Cloud API component is currently inaccessible. All running VRA Clients (Core, Output Player, Audio Manager) will keep running. We will be providing status updates every 30 minutes on the incident. .</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>14:11:43</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident together with our upstream provider. .</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>14:17:09</var> GMT+0</small><br><strong>Monitoring</strong> - We're activating a backup solution, which is impacted by DNS propagation delay but is verified to be working successfully..</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>14:22:00</var> GMT+0</small><br><strong>Resolved</strong> - We've successfully replaced the faulty components of the cloud infrastructure. The full Cloud service is fully recovered since 15:14 UTC+1.</p> ]]> Sat, 20 Dec 2025 13:41:07 +0000 https://status.visualradio.cloud/incident/cmjecjkaz0hglot82yiuau58u https://status.visualradio.cloud/incident/cmjecjkaz0hglot82yiuau58u Degraded availability of cloud.visualradioassist.live Type: Incident Duration: 3 hours and 25 minutes Affected Components: Third Party: Cloudflare → Cloudflare Sites and Services → Infrastructure Nov 18, 12:11:36 GMT+0 - Identified - The front of [cloud.visualradioassist.live](http://cloud.visualradioassist.live) is instable and sometimes inaccessible due to a major outage at [Cloudflare](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). We are monitoring the situation at Cloudflare closely and will communicate updates when the situation improves. Do not try to restart or reload Output Players when accessible due to the size of the network problems - this can cause graphic outputs to display a unwanted error page. Nov 18, 12:45:00 GMT+0 - Resolved - The front of the VRA Cloud is fully stable and operational again since 13:45 CEST. Due to an other Cloudflare issue this status update could not be sent out earlier. Type: Incident

Duration: 3 hours and 25 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>12:11:36</var> GMT+0</small><br><strong>Identified</strong> - The front of [cloud.visualradioassist.live](http://cloud.visualradioassist.live) is instable and sometimes inaccessible due to a major outage at [Cloudflare](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). We are monitoring the situation at Cloudflare closely and will communicate updates when the situation improves. Do not try to restart or reload Output Players when accessible due to the size of the network problems - this can cause graphic outputs to display a unwanted error page..</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>12:45:00</var> GMT+0</small><br><strong>Resolved</strong> - The front of the VRA Cloud is fully stable and operational again since 13:45 CEST. Due to an other Cloudflare issue this status update could not be sent out earlier. .</p> ]]> Tue, 18 Nov 2025 12:11:36 +0000 https://status.visualradio.cloud/incident/cmi4j95z60296dp44gtm8vnkd https://status.visualradio.cloud/incident/cmi4j95z60296dp44gtm8vnkd api.cloud.visualradioassist.live is down Type: Incident Affected Components: live.visualradio.cloud, api.visualradio.cloud Oct 30, 17:24:01 GMT+0 - Investigating - api.visualradio.cloud is down at the moment. This incident was automatically created by Instatus monitoring. Oct 30, 17:28:37 GMT+0 - Identified - We're seeing increased error rates on our Cloud API communication after a infrastructure upgrade of our upstream provider. We are currently working on rolling back the infra upgrade to restore full API connectivity. Oct 30, 17:50:40 GMT+0 - Monitoring - We implemented a the correct version upgrade and are currently monitoring the result. Cloud API performance may still be degraded. Oct 30, 17:55:55 GMT+0 - Resolved - This incident has been resolved. Oct 30, 17:56:37 GMT+0 - Resolved - api.visualradio.cloud is back up. This incident was automatically resolved by Instatus monitoring. Type: Incident

Affected Components: ,

<p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:24:01</var> GMT+0</small><br><strong>Investigating</strong> - api.visualradio.cloud is down at the moment. This incident was automatically created by Instatus monitoring..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:28:37</var> GMT+0</small><br><strong>Identified</strong> - We're seeing increased error rates on our Cloud API communication after a infrastructure upgrade of our upstream provider. We are currently working on rolling back the infra upgrade to restore full API connectivity..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:50:40</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a the correct version upgrade and are currently monitoring the result. Cloud API performance may still be degraded..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:55:55</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:56:37</var> GMT+0</small><br><strong>Resolved</strong> - api.visualradio.cloud is back up. This incident was automatically resolved by Instatus monitoring..</p> ]]> Thu, 30 Oct 2025 17:24:00 +0000 https://status.visualradio.cloud/incident/cmhdp1pjj004asua2yp4z56q7 https://status.visualradio.cloud/incident/cmhdp1pjj004asua2yp4z56q7 Degraded Cloud API due to worldwide AWS outage Type: Incident Duration: 8 minutes Affected Components: api.visualradio.cloud Oct 20, 09:03:52 GMT+0 - Identified - The Cloud API has degraded performance due to an external component relying on [AWS Services (down)](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE%5FSERVICES/AWS%5FMULTIPLE%5FSERVICES%5FOPERATIONAL%5FISSUE/AWS%5FMULTIPLE%5FSERVICES%5FOPERATIONAL%5FISSUE%5FBA540%5F514A652BE1A). We have implemented a workaround for the component to restore the full Cloud API operation. Oct 20, 09:11:24 GMT+0 - Resolved - The external component workaround fix has been implemented and validated operational, we'll continue to closely monitor the situation to identify any further anomalies. Type: Incident

Duration: 8 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>09:03:52</var> GMT+0</small><br><strong>Identified</strong> - The Cloud API has degraded performance due to an external component relying on [AWS Services (down)](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE%5FSERVICES/AWS%5FMULTIPLE%5FSERVICES%5FOPERATIONAL%5FISSUE/AWS%5FMULTIPLE%5FSERVICES%5FOPERATIONAL%5FISSUE%5FBA540%5F514A652BE1A). We have implemented a workaround for the component to restore the full Cloud API operation..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>09:11:24</var> GMT+0</small><br><strong>Resolved</strong> - The external component workaround fix has been implemented and validated operational, we'll continue to closely monitor the situation to identify any further anomalies..</p> ]]> Mon, 20 Oct 2025 09:03:52 +0000 https://status.visualradio.cloud/incident/cmgyws0yp03o4yldjux7q5xnm https://status.visualradio.cloud/incident/cmgyws0yp03o4yldjux7q5xnm Cloud inaccessible due to problems at infrastructure provider Type: Incident Duration: 7 hours and 16 minutes Affected Components: , Core, Media Server, Audio Manager, live.visualradio.cloud, Output Player, Server Applications → Jul 1, 15:08:38 GMT+0 - Identified - We are currently successfully seeing services recover in another infrastructure region. Our cloud provider is confirming instabilities due to the severe heat (weather related). The continuation of the recovery will take approx. 15 minutes. In parallel to internal region infrastructure migration, we are activating a secondary recovery procedure in another Europe based Infrastructure location, to prevent any delay due to a failing procedure 1\. Updates will follow every 30 minutes on [status.visualradio.cloud.](http://status.visualradio.cloud) Jul 1, 17:35:24 GMT+0 - Monitoring - We're still waiting on the green light of the full recovery of our restore procedure 2, which is taking longer than expected due to the size of the infrastructure. In the meantime our cloud provider is monitoring the temperature improvement and will be recovering services as soon as all indicators allow them to. Jul 1, 16:36:00 GMT+0 - Identified - Due to the severity of the situation in the Amsterdam region we're abandoning procedure 1 to quickly recover services ASAP. Our parallel efforts with procedure two are getting to a point of working recovery, we'll proceed with several tests to confirm full recovery of the VRA Cloud infrastructure. Next update will follow in 60 minutes on [status.visualradio.cloud.](http://status.visualradio.cloud) Jul 1, 20:53:31 GMT+0 - Monitoring - Due to the complexity of the outage we are still not able to provide a fully completed data recovery out of the temperature affected infrastructure region, this significantly slows down recovery in the alternative location. Our cloud provider just confirmed green light on the recovery of the affected region, we'll be monitoring the situation closely to bring back Cloud as soon as possible in a stable manner. Jul 1, 15:45:43 GMT+0 - Identified - Our cloud provider officially confirmed the abnormal weather problems and is preventively shutting down services. We're continuously monitoring the recovery of the infrastructure in a close backup region. Due to the stagnation of this process we've ramped up the recovery scenario 2, as mentioned earlier. Jul 1, 22:17:20 GMT+0 - Resolved - We've confirmed full recovery of VRA Cloud Infrastructure - all services are now operating normally. It is not required to restart any Server Applications but refreshing the Cloud in your browser ensures you have the latest state. We sincerely apologize for this severe outage and understand how it disrupted your Visual Radio production workflows. We're implementing additional monitoring and failover procedures within the next 24 hours to prevent similar incidents related to our cloud provider reliability. A detailed postmortem including root cause analysis and our prevention measures will be shared by Friday this week. Jul 1, 15:00:59 GMT+0 - Identified - We are currently activating P1 scenarios to recover critical parts of the infrastructure at another location as soon as possible. Jul 4, 07:54:32 GMT+0 - Postmortem - # Postmortem Cloud Incident - Tuesday, July 1, 2025 ## Summary On Tuesday, July 1, 2025, at 16:45 UTC+2, VRA Cloud experienced a major service disruption due to an infrastructure outage at our cloud provider, Scaleway. The incident was classified as a Major Outage due to complete loss of access to the Cloud dashboard and Live Playout control functionality. While **all running applications** (Core/Output Player/Audio Manager) continued to operate normally, and internal client communication remained uninterrupted, external control from the Cloud was completely unavailable. The root cause was a cooling system failure at our cloud provider's colocation datacenter, which resulted in the loss of an entire Availability Zone. Recovery was further complicated by an inaccessible backup component that extended the restoration timeline beyond our standard recovery procedures. Full service restoration was achieved at 00:17 UTC+2 on July 2, 2025, representing a total outage duration of approximately 7 hours and 32 minutes. ## Incident Timeline **16:45 UTC+2** \- Initial Detection * Monitoring systems detected elevated error rates across infrastructure components * Users began experiencing difficulties accessing the Cloud and controlling VRA applications * Investigation initiated **17:00 UTC+2** \- Escalation to P1 Incident * Activated Priority 1 incident response procedures * Cloud provider (Scaleway) communicated cooling system issues in Amsterdam datacenter * Initiated recovery procedures to alternate availability zone * Reference: <https://status.scaleway.com/incidents/1vz4xfgy2gcl> **17:08 UTC+2** \- Initial Recovery Attempts * Services began recovering in alternate infrastructure region * Cloud provider confirmed instabilities related to extreme weather conditions * Estimated recovery time: 15 minutes * Activated secondary recovery procedure (Procedure 2) in alternate European location as contingency **17:45 UTC+2** \- Recovery Complications * Cloud provider confirmed weather-related infrastructure problems and initiated preventive service shutdowns * Primary recovery procedure (Procedure 1) blocked due to outage characteristics * Proceeded with alternative recovery efforts (Procedure 2) - regularly tested bi-weekly process **18:36 UTC+2** \- Strategy Pivot * Abandoned Procedure 1 due to severity of Amsterdam region situation * Focused resources on Procedure 2 recovery efforts * Initiated comprehensive testing to verify full infrastructure recovery **19:35 UTC+2** \- Extended Recovery Phase * Procedure 2 restoration taking longer than expected due to infrastructure complexity * Cloud provider monitoring temperature improvements for service restoration **20:00 UTC+2** \- Data Integrity Issues Identified * Analysis revealed database inconsistencies in recovered infrastructure * Data integrity errors traced to initial outage preventing proper restoration **22:53 UTC+2** \- Continued Recovery Efforts * Unable to complete data recovery from temperature-affected infrastructure region * Cloud provider confirmed clearance for affected region recovery * Monitoring situation for stable service restoration **00:17 UTC+2** \- Full Service Restoration * Confirmed complete recovery of VRA Cloud infrastructure * All services operating normally * No server application restarts required * Users advised to refresh Cloud browser sessions for latest state ## Impact Assessment **Services Affected:** * VRA Cloud dashboard access * Live Playout control functionality * Visual Player content updates (potential impact) **Services Maintained:** * All running applications (Core/Output Player/Audio Manager) * Internal client communication * Audio triggers and Output Playout internal operations **Duration:** 7 hours 32 minutes (16:45 - 00:17 UTC+2) ## Root Cause Analysis The incident originated from a cooling system failure at our cloud provider's colocation datacenter, resulting in the complete loss of an Availability Zone. This infrastructure failure was beyond our direct control and affected multiple services within the facility. Recovery was significantly prolonged due to a critical gap in our backup procedures. Following infrastructure performance changes implemented in January 2025, our Procedure 2 recovery process encountered unexpected obstacles when attempting to access backup data from our High Availability storage location. The complete loss of storage access to the Amsterdam availability region prevented successful execution of our standard recovery protocols. ## Response and Recovery Actions Our incident response immediately implemented our established failover procedures: 1. **Parallel Recovery Approach:** Executed both Procedure 1 (10-minute recovery target) and Procedure 2 (60-80 minute recovery target) simultaneously 2. **Resource Allocation:** Dedicated efforts to each recovery track to maximize restoration speed 3. **Continuous Monitoring:** Maintained real-time assessment of both recovery efforts and cloud provider status 4. **Communication:** Provided regular updates on recovery progress and service status ## Immediate Remediation **Completed Actions:** * **July 2, 12:00 UTC+2:** Resolved critical backup procedure blocking issues * Enhanced High Availability backup data storage redundancy * Verified recovery procedures for complete datacenter loss scenarios ## Long-term Improvements **Process Enhancements:** * Strengthened backup redundancy across multiple geographic regions * Enhanced monitoring and alerting for backup system accessibility * Improved failover testing to include extreme failure scenarios **Testing Protocol (continuing existing efforts):** * Continued bi-weekly failover scenario testing * Extended test scenarios to include complete availability zone loss * Regular validation of backup data accessibility across all storage locations ## Lessons Learned 1. **Infrastructure Dependencies:** Single points of failure in cloud provider infrastructure can significantly impact service availability 2. **Backup Strategy:** Backup accessibility must be tested under various failure scenarios, not just standard conditions 3. **Recovery Procedures:** Multiple recovery strategies should be maintained and regularly tested for different failure modes ## Our Apology and Next Steps We know this 7+ hour outage seriously disrupted your work, and we're genuinely sorry about that. We recognize the significant impact this incident had on your operations and sincerely apologize for the disruption to your Visual Radio production workflows. We've already fixed the backup issue that made this take so much longer than it should have, and we're continuing our regular testing to catch problems before they become outages like this one. If you have any questions about what happened or concerns about your setup, just reach out to us directly. Type: Incident

Duration: 7 hours and 16 minutes

Affected Components: , , , , ,

<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:08:38</var> GMT+0</small><br><strong>Identified</strong> - We are currently successfully seeing services recover in another infrastructure region. Our cloud provider is confirming instabilities due to the severe heat (weather related). The continuation of the recovery will take approx. 15 minutes. In parallel to internal region infrastructure migration, we are activating a secondary recovery procedure in another Europe based Infrastructure location, to prevent any delay due to a failing procedure 1\. Updates will follow every 30 minutes on [status.visualradio.cloud.](http://status.visualradio.cloud) .</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>17:35:24</var> GMT+0</small><br><strong>Monitoring</strong> - We're still waiting on the green light of the full recovery of our restore procedure 2, which is taking longer than expected due to the size of the infrastructure. In the meantime our cloud provider is monitoring the temperature improvement and will be recovering services as soon as all indicators allow them to..</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>16:36:00</var> GMT+0</small><br><strong>Identified</strong> - Due to the severity of the situation in the Amsterdam region we're abandoning procedure 1 to quickly recover services ASAP. Our parallel efforts with procedure two are getting to a point of working recovery, we'll proceed with several tests to confirm full recovery of the VRA Cloud infrastructure. Next update will follow in 60 minutes on [status.visualradio.cloud.](http://status.visualradio.cloud).</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>20:53:31</var> GMT+0</small><br><strong>Monitoring</strong> - Due to the complexity of the outage we are still not able to provide a fully completed data recovery out of the temperature affected infrastructure region, this significantly slows down recovery in the alternative location. Our cloud provider just confirmed green light on the recovery of the affected region, we'll be monitoring the situation closely to bring back Cloud as soon as possible in a stable manner..</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:45:43</var> GMT+0</small><br><strong>Identified</strong> - Our cloud provider officially confirmed the abnormal weather problems and is preventively shutting down services. We're continuously monitoring the recovery of the infrastructure in a close backup region. Due to the stagnation of this process we've ramped up the recovery scenario 2, as mentioned earlier..</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>22:17:20</var> GMT+0</small><br><strong>Resolved</strong> - We've confirmed full recovery of VRA Cloud Infrastructure - all services are now operating normally. It is not required to restart any Server Applications but refreshing the Cloud in your browser ensures you have the latest state. We sincerely apologize for this severe outage and understand how it disrupted your Visual Radio production workflows. We're implementing additional monitoring and failover procedures within the next 24 hours to prevent similar incidents related to our cloud provider reliability. A detailed postmortem including root cause analysis and our prevention measures will be shared by Friday this week..</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:00:59</var> GMT+0</small><br><strong>Identified</strong> - We are currently activating P1 scenarios to recover critical parts of the infrastructure at another location as soon as possible..</p> <p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>07:54:32</var> GMT+0</small><br><strong>Postmortem</strong> - # Postmortem Cloud Incident - Tuesday, July 1, 2025 ## Summary On Tuesday, July 1, 2025, at 16:45 UTC+2, VRA Cloud experienced a major service disruption due to an infrastructure outage at our cloud provider, Scaleway. The incident was classified as a Major Outage due to complete loss of access to the Cloud dashboard and Live Playout control functionality. While **all running applications** (Core/Output Player/Audio Manager) continued to operate normally, and internal client communication remained uninterrupted, external control from the Cloud was completely unavailable. The root cause was a cooling system failure at our cloud provider's colocation datacenter, which resulted in the loss of an entire Availability Zone. Recovery was further complicated by an inaccessible backup component that extended the restoration timeline beyond our standard recovery procedures. Full service restoration was achieved at 00:17 UTC+2 on July 2, 2025, representing a total outage duration of approximately 7 hours and 32 minutes. ## Incident Timeline **16:45 UTC+2** \- Initial Detection * Monitoring systems detected elevated error rates across infrastructure components * Users began experiencing difficulties accessing the Cloud and controlling VRA applications * Investigation initiated **17:00 UTC+2** \- Escalation to P1 Incident * Activated Priority 1 incident response procedures * Cloud provider (Scaleway) communicated cooling system issues in Amsterdam datacenter * Initiated recovery procedures to alternate availability zone * Reference: <https://status.scaleway.com/incidents/1vz4xfgy2gcl> **17:08 UTC+2** \- Initial Recovery Attempts * Services began recovering in alternate infrastructure region * Cloud provider confirmed instabilities related to extreme weather conditions * Estimated recovery time: 15 minutes * Activated secondary recovery procedure (Procedure 2) in alternate European location as contingency **17:45 UTC+2** \- Recovery Complications * Cloud provider confirmed weather-related infrastructure problems and initiated preventive service shutdowns * Primary recovery procedure (Procedure 1) blocked due to outage characteristics * Proceeded with alternative recovery efforts (Procedure 2) - regularly tested bi-weekly process **18:36 UTC+2** \- Strategy Pivot * Abandoned Procedure 1 due to severity of Amsterdam region situation * Focused resources on Procedure 2 recovery efforts * Initiated comprehensive testing to verify full infrastructure recovery **19:35 UTC+2** \- Extended Recovery Phase * Procedure 2 restoration taking longer than expected due to infrastructure complexity * Cloud provider monitoring temperature improvements for service restoration **20:00 UTC+2** \- Data Integrity Issues Identified * Analysis revealed database inconsistencies in recovered infrastructure * Data integrity errors traced to initial outage preventing proper restoration **22:53 UTC+2** \- Continued Recovery Efforts * Unable to complete data recovery from temperature-affected infrastructure region * Cloud provider confirmed clearance for affected region recovery * Monitoring situation for stable service restoration **00:17 UTC+2** \- Full Service Restoration * Confirmed complete recovery of VRA Cloud infrastructure * All services operating normally * No server application restarts required * Users advised to refresh Cloud browser sessions for latest state ## Impact Assessment **Services Affected:** * VRA Cloud dashboard access * Live Playout control functionality * Visual Player content updates (potential impact) **Services Maintained:** * All running applications (Core/Output Player/Audio Manager) * Internal client communication * Audio triggers and Output Playout internal operations **Duration:** 7 hours 32 minutes (16:45 - 00:17 UTC+2) ## Root Cause Analysis The incident originated from a cooling system failure at our cloud provider's colocation datacenter, resulting in the complete loss of an Availability Zone. This infrastructure failure was beyond our direct control and affected multiple services within the facility. Recovery was significantly prolonged due to a critical gap in our backup procedures. Following infrastructure performance changes implemented in January 2025, our Procedure 2 recovery process encountered unexpected obstacles when attempting to access backup data from our High Availability storage location. The complete loss of storage access to the Amsterdam availability region prevented successful execution of our standard recovery protocols. ## Response and Recovery Actions Our incident response immediately implemented our established failover procedures: 1. **Parallel Recovery Approach:** Executed both Procedure 1 (10-minute recovery target) and Procedure 2 (60-80 minute recovery target) simultaneously 2. **Resource Allocation:** Dedicated efforts to each recovery track to maximize restoration speed 3. **Continuous Monitoring:** Maintained real-time assessment of both recovery efforts and cloud provider status 4. **Communication:** Provided regular updates on recovery progress and service status ## Immediate Remediation **Completed Actions:** * **July 2, 12:00 UTC+2:** Resolved critical backup procedure blocking issues * Enhanced High Availability backup data storage redundancy * Verified recovery procedures for complete datacenter loss scenarios ## Long-term Improvements **Process Enhancements:** * Strengthened backup redundancy across multiple geographic regions * Enhanced monitoring and alerting for backup system accessibility * Improved failover testing to include extreme failure scenarios **Testing Protocol (continuing existing efforts):** * Continued bi-weekly failover scenario testing * Extended test scenarios to include complete availability zone loss * Regular validation of backup data accessibility across all storage locations ## Lessons Learned 1. **Infrastructure Dependencies:** Single points of failure in cloud provider infrastructure can significantly impact service availability 2. **Backup Strategy:** Backup accessibility must be tested under various failure scenarios, not just standard conditions 3. **Recovery Procedures:** Multiple recovery strategies should be maintained and regularly tested for different failure modes ## Our Apology and Next Steps We know this 7+ hour outage seriously disrupted your work, and we're genuinely sorry about that. We recognize the significant impact this incident had on your operations and sincerely apologize for the disruption to your Visual Radio production workflows. We've already fixed the backup issue that made this take so much longer than it should have, and we're continuing our regular testing to catch problems before they become outages like this one. If you have any questions about what happened or concerns about your setup, just reach out to us directly..</p> ]]> Tue, 1 Jul 2025 15:00:59 +0000 https://status.visualradio.cloud/incident/cmcknnq7x0028w5x5006t9av6 https://status.visualradio.cloud/incident/cmcknnq7x0028w5x5006t9av6 Issues accessing VRA Cloud & video media Type: Incident Duration: 1 hour and 9 minutes Affected Components: Media Server Jun 12, 19:20:00 GMT+0 - Monitoring - We're notified about instabilities with our upstream provider for the VRA Cloud dashboard and video provider. The upstream provider identified the issue and is monitoring recovery of the situation. Access to video media and some pages of the [visualradio.cloud](http://visualradio.cloud) can be degraded or non available at the moment. **Running VRA installations are not affected** and we will continue monitoring the situation. Jun 12, 20:28:54 GMT+0 - Resolved - We have successfully verified the restoration of the upstream provider services. VRA services should be working as expected. If you still experience problems accessing VRA Cloud – don't hesitate to contact support@visualradioassist.live. Type: Incident

Duration: 1 hour and 9 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>19:20:00</var> GMT+0</small><br><strong>Monitoring</strong> - We're notified about instabilities with our upstream provider for the VRA Cloud dashboard and video provider. The upstream provider identified the issue and is monitoring recovery of the situation. Access to video media and some pages of the [visualradio.cloud](http://visualradio.cloud) can be degraded or non available at the moment. **Running VRA installations are not affected** and we will continue monitoring the situation..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>20:28:54</var> GMT+0</small><br><strong>Resolved</strong> - We have successfully verified the restoration of the upstream provider services. VRA services should be working as expected. If you still experience problems accessing VRA Cloud – don't hesitate to contact support@visualradioassist.live..</p> ]]> Thu, 12 Jun 2025 19:20:00 +0000 https://status.visualradio.cloud/incident/cmbtrjnn2006xjmdrvc5bvoeg https://status.visualradio.cloud/incident/cmbtrjnn2006xjmdrvc5bvoeg Cloud is not available for all users Type: Incident Feb 11, 10:32:13 GMT+0 - Resolved - The recovery of the services has been completed and all services are operational again. Type: Incident

<p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>10:32:13</var> GMT+0</small><br><strong>Resolved</strong> - The recovery of the services has been completed and all services are operational again. .</p> ]]> Tue, 11 Feb 2025 10:32:13 +0000 https://status.visualradio.cloud/incident/cm70ceu9h000xnf59ret92wn5 https://status.visualradio.cloud/incident/cm70ceu9h000xnf59ret92wn5 Unable to upload images & preview Output thumbnails due to upstream provider issues Type: Incident Duration: 40 minutes Affected Components: Cloudflare → R2, Cloudflare → Images, Cloudflare → Stream Feb 6, 09:37:36 GMT+0 - Resolved - The upstream provider has implemented a fix, and we are now seeing recovery of both the complete image and output thumbnail services in VRA Cloud. We apologize for any inconvenience this issue may have caused with VRA Cloud Media services. Feb 6, 08:57:29 GMT+0 - Identified - Due to an issue at an upstream provider (Cloudflare) we are unable to process requests regarding new image uploads and updated thumbnails for running Output Players. We continue to investigate the availability issues at the provider and commit to posting updates hourly or sooner if there are significant developments. Type: Incident

Duration: 40 minutes

Affected Components: , ,

<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>09:37:36</var> GMT+0</small><br><strong>Resolved</strong> - The upstream provider has implemented a fix, and we are now seeing recovery of both the complete image and output thumbnail services in VRA Cloud. We apologize for any inconvenience this issue may have caused with VRA Cloud Media services..</p> <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>08:57:29</var> GMT+0</small><br><strong>Identified</strong> - Due to an issue at an upstream provider (Cloudflare) we are unable to process requests regarding new image uploads and updated thumbnails for running Output Players. We continue to investigate the availability issues at the provider and commit to posting updates hourly or sooner if there are significant developments..</p> ]]> Thu, 6 Feb 2025 08:57:29 +0000 https://status.visualradio.cloud/incident/cm6t3tsa8001ns7aibd8wol7r https://status.visualradio.cloud/incident/cm6t3tsa8001ns7aibd8wol7r Cloud Infrastructure Routine Maintenance Type: Maintenance Duration: 1 hour Affected Components: live.visualradio.cloud, Third Party -> SCW -> Block Storage Jan 16, 00:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during the night to make sure all our Cloud systems stay up to date with latest performance and security updates. There is **no downtime** expected for this timeframe (16 January 1:00 - 2:00 UTC+1), but affected components may be slower than expected due to Load Balancing and scaling maintenance. Jan 16, 01:00:00 GMT+0 - Completed - Maintenance has completed successfully Jan 16, 00:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components: ,

<p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during the night to make sure all our Cloud systems stay up to date with latest performance and security updates. There is **no downtime** expected for this timeframe (16 January 1:00 - 2:00 UTC+1), but affected components may be slower than expected due to Load Balancing and scaling maintenance..</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>00:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Thu, 16 Jan 2025 00:00:00 +0000 https://status.visualradio.cloud/maintenance/cm5wek39a0036rbxz9vgsy2dz https://status.visualradio.cloud/maintenance/cm5wek39a0036rbxz9vgsy2dz Scheduling Program Media not working in v c1fd4a68 Type: Incident Duration: 18 minutes Affected Components: Output Player Oct 23, 07:11:11 GMT+0 - Identified - The Scheduling Program Media layer, connecting the media from the current Scheduling program to the layer in the Visual, is currently creating an error and not displaying the latest program media. We are currently investigating this error and will be providing a fix soon. Oct 23, 07:29:32 GMT+0 - Resolved - The Scheduling Program Media Layer bug has been resolved. It may be required to manually Reload the Visuals in your Output Player to quickly update to the latest version 15d5a1ff. **Reload all Visual Players on**[ https://cloud.visualradioassist.live/studio/active/advanced/output](https://cloud.visualradioassist.live/studio/active/advanced/output) Type: Incident

Duration: 18 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:11:11</var> GMT+0</small><br><strong>Identified</strong> - The Scheduling Program Media layer, connecting the media from the current Scheduling program to the layer in the Visual, is currently creating an error and not displaying the latest program media. We are currently investigating this error and will be providing a fix soon..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:29:32</var> GMT+0</small><br><strong>Resolved</strong> - The Scheduling Program Media Layer bug has been resolved. It may be required to manually Reload the Visuals in your Output Player to quickly update to the latest version 15d5a1ff. **Reload all Visual Players on**[ https://cloud.visualradioassist.live/studio/active/advanced/output](https://cloud.visualradioassist.live/studio/active/advanced/output).</p> ]]> Wed, 23 Oct 2024 07:11:11 +0000 https://status.visualradio.cloud/incident/cm2ljcql40030oplmr0ygfw0i https://status.visualradio.cloud/incident/cm2ljcql40030oplmr0ygfw0i Maintenance on Cloud Infrastructure Networking Type: Incident Duration: 2 hours and 35 minutes Affected Components: live.visualradio.cloud Oct 7, 09:38:15 GMT+0 - Monitoring - Our upstream cloudprovider is currently doing maintenance on the network of our infrastructure. Due to the redundant set-ups in place we do not expect any downtime, but Visual Radio Assist Clients could try to reconnect due to a interrupted Studio Bus connection. The maintenance is expected to be completed by 13:00 UTC. Oct 7, 12:13:38 GMT+0 - Resolved - Cloudprovider Maintenance has been completed successfully. No major downtime detected and no further action required. Type: Incident

Duration: 2 hours and 35 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>09:38:15</var> GMT+0</small><br><strong>Monitoring</strong> - Our upstream cloudprovider is currently doing maintenance on the network of our infrastructure. Due to the redundant set-ups in place we do not expect any downtime, but Visual Radio Assist Clients could try to reconnect due to a interrupted Studio Bus connection. The maintenance is expected to be completed by 13:00 UTC..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>12:13:38</var> GMT+0</small><br><strong>Resolved</strong> - Cloudprovider Maintenance has been completed successfully. No major downtime detected and no further action required..</p> ]]> Mon, 7 Oct 2024 09:38:15 +0000 https://status.visualradio.cloud/incident/cm1ytk8ri000cun0uxe4og64x https://status.visualradio.cloud/incident/cm1ytk8ri000cun0uxe4og64x Degraded performance on Cloud API Type: Incident Duration: 10 hours and 25 minutes Apr 9, 12:52:32 GMT+0 - Identified - Due to an unexpected and abnormal increase of the load on the Cloud API the Cloud App and App connectivity have degraded performance. We are currently scaling to handle the increased load and bring back operational stability. Our apologies for the inconvenience. Apr 9, 13:52:00 GMT+0 - Monitoring - Cloud API Availability levels are back to normal. We're currently monitoring the situation closely and scaling where possible. You can still experience problems when processing updates for Scheduling Programs and Visuals in the Core and Output Player apps. We are working on a final fix to prevent the excessive (data-update) traffic in the future. This fix will be available ASAP, at least before the update window tonight (3:00 - 3:15 CET). Apr 9, 23:17:42 GMT+0 - Resolved - All Cloud API systems are operational again. Data updates to Scheduling Programs and Visuals should also work as expected, if you still experience update issues contact [support@visualradioassist.live](mailto:support@visualradioassist.live) or restart your VRA Clients. **UPGRADE NOTICE: Make sure that all VRA Clients are updated to version >= 4.1.20 (or running a version lower than < 4.1.16).** If you're already running >= 4.1.21 no action is required. Type: Incident

Duration: 10 hours and 25 minutes

<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:52:32</var> GMT+0</small><br><strong>Identified</strong> - Due to an unexpected and abnormal increase of the load on the Cloud API the Cloud App and App connectivity have degraded performance. We are currently scaling to handle the increased load and bring back operational stability. Our apologies for the inconvenience..</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>13:52:00</var> GMT+0</small><br><strong>Monitoring</strong> - Cloud API Availability levels are back to normal. We're currently monitoring the situation closely and scaling where possible. You can still experience problems when processing updates for Scheduling Programs and Visuals in the Core and Output Player apps. We are working on a final fix to prevent the excessive (data-update) traffic in the future. This fix will be available ASAP, at least before the update window tonight (3:00 - 3:15 CET). .</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>23:17:42</var> GMT+0</small><br><strong>Resolved</strong> - All Cloud API systems are operational again. Data updates to Scheduling Programs and Visuals should also work as expected, if you still experience update issues contact [support@visualradioassist.live](mailto:support@visualradioassist.live) or restart your VRA Clients. **UPGRADE NOTICE: Make sure that all VRA Clients are updated to version >= 4.1.20 (or running a version lower than < 4.1.16).** If you're already running >= 4.1.21 no action is required..</p> ]]> Tue, 9 Apr 2024 12:52:32 +0000 https://status.visualradio.cloud/incident/clusdsx6w32268b1oexvto14dg https://status.visualradio.cloud/incident/clusdsx6w32268b1oexvto14dg Upcoming App Update to major version 4.1 Type: Maintenance Duration: 1 minute Affected Components: Core, Output Player, Audio Manager Jan 30, 02:11:00 GMT+0 - Completed - Maintenance has completed successfully Jan 30, 02:10:00 GMT+0 - Identified - **! AUTO UPDATE STUDIOS ONLY !** – No direct action required. The [major update to 4.1](https://visualradioassist.live/release/4.1) will be automatically rolled out to your Visual Radio Assist App Clients (Audio, Output and Core) the upcoming night of 30 January 2024. The update' installation does not differ to previous updates, which happens completely in the background and usually only causes a minor downtime (2-10 sec) to the Output Players. ### Release 4.1 Learn more about the new version in this 4.1 Release Week! https://visualradioassist.live/release/4.1 ### The update process Despite our major efforts to make this upgrade from 4.0.x to 4.1 as smooth as possible, there is a slight possibility that the clients do not work as expected after this update, due to the magnitude of this update (for all three clients). Please make sure to contact support@visualradioassist.live / https://cloud.visualradioassist.live/support a.s.a.p. when you experience anomalies so we can verify your 4.1 installation. More information about the update process is documented on: https://docs.visualradioassist.live/advanced-installation/manual-update-process In case you prefer to trigger the update manually: the update 4.1.1 will be available this afternoon at ~~17:00~~ CET *– due to a unexpected bug the release is postponed to 20:00 CET*. ### Auto updating By the default configuration, autoupdating is and should enabled, when no timeslot is specified: https://cloud.visualradioassist.live/studio/active/advanced/core Jan 30, 02:10:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 minute

Affected Components: , ,

<p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>02:11:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>02:10:00</var> GMT+0</small><br><strong>Identified</strong> - **! AUTO UPDATE STUDIOS ONLY !** – No direct action required. The [major update to 4.1](https://visualradioassist.live/release/4.1) will be automatically rolled out to your Visual Radio Assist App Clients (Audio, Output and Core) the upcoming night of 30 January 2024. The update' installation does not differ to previous updates, which happens completely in the background and usually only causes a minor downtime (2-10 sec) to the Output Players. ### Release 4.1 Learn more about the new version in this 4.1 Release Week! https://visualradioassist.live/release/4.1 ### The update process Despite our major efforts to make this upgrade from 4.0.x to 4.1 as smooth as possible, there is a slight possibility that the clients do not work as expected after this update, due to the magnitude of this update (for all three clients). Please make sure to contact support@visualradioassist.live / https://cloud.visualradioassist.live/support a.s.a.p. when you experience anomalies so we can verify your 4.1 installation. More information about the update process is documented on: https://docs.visualradioassist.live/advanced-installation/manual-update-process In case you prefer to trigger the update manually: the update 4.1.1 will be available this afternoon at ~~17:00~~ CET *– due to a unexpected bug the release is postponed to 20:00 CET*. ### Auto updating By the default configuration, autoupdating is and should enabled, when no timeslot is specified: https://cloud.visualradioassist.live/studio/active/advanced/core .</p> <p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>02:10:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Tue, 30 Jan 2024 02:10:00 +0000 https://status.visualradio.cloud/maintenance/clryzsh6r24273bgo9cm5j4xbh https://status.visualradio.cloud/maintenance/clryzsh6r24273bgo9cm5j4xbh Cloud update maintenance planned Type: Maintenance Duration: 29 minutes Jan 10, 22:15:00 GMT+0 - Identified - We are planning for a scheduled maintenance to the general infrastructure of the Visual Radio Cloud during a short period of time between 23:15 – 23:45 CET 10 January 2024 to improve the security and stability. The Cloud environment should not experience any degraded performance during that time. The live-studiobus connection could be interrupted for a short period of time (2-10 sec.), after which all (app) clients automatically reconnect. We'll keep the [status page](https://status.visualradio.cloud) up to date if any anomalies occur. Jan 10, 22:44:16 GMT+0 - Completed - Maintenance activities have been completed successfully with no monitored or reported downtime or issues. Jan 10, 22:15:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 29 minutes

<p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>22:15:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance to the general infrastructure of the Visual Radio Cloud during a short period of time between 23:15 – 23:45 CET 10 January 2024 to improve the security and stability. The Cloud environment should not experience any degraded performance during that time. The live-studiobus connection could be interrupted for a short period of time (2-10 sec.), after which all (app) clients automatically reconnect. We'll keep the [status page](https://status.visualradio.cloud) up to date if any anomalies occur..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>22:44:16</var> GMT+0</small><br><strong>Completed</strong> - Maintenance activities have been completed successfully with no monitored or reported downtime or issues..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>22:15:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Wed, 10 Jan 2024 22:15:00 +0000 https://status.visualradio.cloud/maintenance/clqzeupus80821b8ofqoc60xow https://status.visualradio.cloud/maintenance/clqzeupus80821b8ofqoc60xow Short Cloud Maintenance to increase High Availability Type: Maintenance Duration: 10 minutes Dec 23, 00:05:01 GMT+0 - Identified - Maintenance is now in progress Dec 23, 00:05:00 GMT+0 - Identified - We are planning for a scheduled maintenance to the caching and database storage layer of the Visual Radio Cloud during a short period of time between 1:05 – 1:15 CET 23 December 2023. The Cloud environment could experience degraded performance during that time. Locally installed clients and their communication layer is not impacted by this degraded performance. Dec 23, 00:15:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 10 minutes

<p><small>Dec <var data-var='date'> 23</var>, <var data-var='time'>00:05:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 23</var>, <var data-var='time'>00:05:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance to the caching and database storage layer of the Visual Radio Cloud during a short period of time between 1:05 – 1:15 CET 23 December 2023. The Cloud environment could experience degraded performance during that time. Locally installed clients and their communication layer is not impacted by this degraded performance..</p> <p><small>Dec <var data-var='date'> 23</var>, <var data-var='time'>00:15:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sat, 23 Dec 2023 00:05:00 +0000 https://status.visualradio.cloud/maintenance/clqgu9d0f14023b4nasxjw8vrq https://status.visualradio.cloud/maintenance/clqgu9d0f14023b4nasxjw8vrq Major Outage on Cloud Provider Storage Type: Incident Duration: 7 hours and 46 minutes Dec 8, 13:30:00 GMT+0 - Resolved - ### Postmortem 8 december Cloud Incident The morning of 8 dec. 2023 the VRA Cloud infrastructure experienced downtime due to our Cloud Provider having issues with the storage solution. At first this only impacted basic communication to the Cloud dashboard, but it later on could also have resulted in out-of-date Visual Player content in the Output Players. We scaled the incident to Major Outage because users were not able to access the Cloud at some moments during the incident. The incident **did not affect any running applications** (Core/Output Player/Audio Manager). Internal communication between clients (for Audio triggers, Output Playout etc.) was not interrupted, but external control from the Cloud had partial outage. Unfortunately the root cause of the issue was external at our cloud provider. Our apologies for the inconvenience of the resulted downtime of the VRA Cloud. We are working on a faster failover switch for these major outages, which will shorten the time the Cloud is inaccessible. ### Summary & Timeline Our monitoring began alerting about increasing API response times and decreasing success rate at 10:59 CET. While investigating the issues, the main Cloud API, responsible for the communication between the Cloud dashboard and local applications, started showing signs of degraded performance with as root cause an unstable database. During the degradation of the Cloud API, the database became completely unavailable at 11:24 CET, around the time that our cloud provider declared a [major incident on the storage side](https://status.scaleway.com/incidents/1p5k6r8mc4gh) of the cloud infrastructure in region AMS-1. This storage incident caused multiple components of the Cloud to fail with problems accessing storage. We immediately started escalating the incident, this means that our backup solution is being prepared for failover. At 12:30 CET our backup was ready for complete failover, because this is an very invasive measure we waited on the recent updates from the provider that was confirming recovery of the situation in ams-1. When we got confirmation (12:50) that the recovery of the outage in ams-1 would take extra time, we directly triggered the failover scenario, which made the backup scenario active. After an initial Load Balancing issue, the failover infrastructure became generally available at 13:20 CET. The main infrastructure became partially stable again at 14:00 CET. While we were monitoring the situation at our backup infra, we prepared to fail back to normal asap (to decrease the size of the sync back). All tests and monitoring at our regular NL-AMS-1 infrastructure went green again at 14:08 CET, which initiated a direct recovery of the complete Cloud service in the following minutes. Throughout the remainder of the day, we maintained vigilant monitoring of the infrastructure without any service interruptions following the complete recovery at 14:08. ### Response During these kinds of major incidents we immediately take action in activating and preparing our failover infrastructure. Due to the high probability of just a short outage and quick recovery at the cloud provider, the failover scenario was activated at a later point in time. When finally activating the scenario DNS settings issues, combined with an internal networking issue that prevented the Load Balancer to route traffic correctly, resulted in a slower failover. ### Recommendations We’ll continue to bi-monthly test our failover scenario’s and have already mitigated the issues with our Load Balancing set-up and DNS this afternoon. In the future we want to be able to recover service in case of a major outage even faster by creating more flexible ways to route traffic. We’ve also seen a slow recovery of Studio Bus connections with the Cloud which should be improved with resilliant connection recovery. [status.visualradio.cloud](https://status.visualradio.cloud) will be always up to date with current status and incident reports. Subscribe with your email to stay up to date with the current status of VRA. Dec 8, 09:06:00 GMT+0 - Investigating - We're currently experiencing issues with this service. We're working on it and will update you as soon as possible. Dec 8, 09:48:00 GMT+0 - Identified - Our Cloud Provider has issues with an increased latency on certain storage components. This is impacting all communication to VRA Cloud currently. We are activating our backup scenario ASAP and will be providing updates here. Dec 8, 10:29:00 GMT+0 - Monitoring - The backup scenario is in place, we are currently monitoring the stability of the backup. Dec 8, 12:19:49 GMT+0 - Monitoring - The backup infrastructure is completely rolled out and available. We'll keep on monitoring the infrastructure for and degradation. Dec 8, 12:31:12 GMT+0 - Monitoring - We currently see positive test results on our main infrastructure. We'll give an update when we start downscaling our backup infra. Dec 8, 12:12:59 GMT+0 - Monitoring - The backup infrastructure is in place and will be functional with the incremental DNS updates that are being processed. Dec 8, 13:09:17 GMT+0 - Resolved - We are currently scaling back to our original production infrastructure. Some connections will be discarded in this process and will be recovered shortly. Type: Incident

Duration: 7 hours and 46 minutes

<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>13:30:00</var> GMT+0</small><br><strong>Resolved</strong> - ### Postmortem 8 december Cloud Incident The morning of 8 dec. 2023 the VRA Cloud infrastructure experienced downtime due to our Cloud Provider having issues with the storage solution. At first this only impacted basic communication to the Cloud dashboard, but it later on could also have resulted in out-of-date Visual Player content in the Output Players. We scaled the incident to Major Outage because users were not able to access the Cloud at some moments during the incident. The incident **did not affect any running applications** (Core/Output Player/Audio Manager). Internal communication between clients (for Audio triggers, Output Playout etc.) was not interrupted, but external control from the Cloud had partial outage. Unfortunately the root cause of the issue was external at our cloud provider. Our apologies for the inconvenience of the resulted downtime of the VRA Cloud. We are working on a faster failover switch for these major outages, which will shorten the time the Cloud is inaccessible. ### Summary & Timeline Our monitoring began alerting about increasing API response times and decreasing success rate at 10:59 CET. While investigating the issues, the main Cloud API, responsible for the communication between the Cloud dashboard and local applications, started showing signs of degraded performance with as root cause an unstable database. During the degradation of the Cloud API, the database became completely unavailable at 11:24 CET, around the time that our cloud provider declared a [major incident on the storage side](https://status.scaleway.com/incidents/1p5k6r8mc4gh) of the cloud infrastructure in region AMS-1. This storage incident caused multiple components of the Cloud to fail with problems accessing storage. We immediately started escalating the incident, this means that our backup solution is being prepared for failover. At 12:30 CET our backup was ready for complete failover, because this is an very invasive measure we waited on the recent updates from the provider that was confirming recovery of the situation in ams-1. When we got confirmation (12:50) that the recovery of the outage in ams-1 would take extra time, we directly triggered the failover scenario, which made the backup scenario active. After an initial Load Balancing issue, the failover infrastructure became generally available at 13:20 CET. The main infrastructure became partially stable again at 14:00 CET. While we were monitoring the situation at our backup infra, we prepared to fail back to normal asap (to decrease the size of the sync back). All tests and monitoring at our regular NL-AMS-1 infrastructure went green again at 14:08 CET, which initiated a direct recovery of the complete Cloud service in the following minutes. Throughout the remainder of the day, we maintained vigilant monitoring of the infrastructure without any service interruptions following the complete recovery at 14:08. ### Response During these kinds of major incidents we immediately take action in activating and preparing our failover infrastructure. Due to the high probability of just a short outage and quick recovery at the cloud provider, the failover scenario was activated at a later point in time. When finally activating the scenario DNS settings issues, combined with an internal networking issue that prevented the Load Balancer to route traffic correctly, resulted in a slower failover. ### Recommendations We’ll continue to bi-monthly test our failover scenario’s and have already mitigated the issues with our Load Balancing set-up and DNS this afternoon. In the future we want to be able to recover service in case of a major outage even faster by creating more flexible ways to route traffic. We’ve also seen a slow recovery of Studio Bus connections with the Cloud which should be improved with resilliant connection recovery. [status.visualradio.cloud](https://status.visualradio.cloud) will be always up to date with current status and incident reports. Subscribe with your email to stay up to date with the current status of VRA. .</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>09:06:00</var> GMT+0</small><br><strong>Investigating</strong> - We're currently experiencing issues with this service. We're working on it and will update you as soon as possible..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>09:48:00</var> GMT+0</small><br><strong>Identified</strong> - Our Cloud Provider has issues with an increased latency on certain storage components. This is impacting all communication to VRA Cloud currently. We are activating our backup scenario ASAP and will be providing updates here..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>10:29:00</var> GMT+0</small><br><strong>Monitoring</strong> - The backup scenario is in place, we are currently monitoring the stability of the backup. .</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:19:49</var> GMT+0</small><br><strong>Monitoring</strong> - The backup infrastructure is completely rolled out and available. We'll keep on monitoring the infrastructure for and degradation..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:31:12</var> GMT+0</small><br><strong>Monitoring</strong> - We currently see positive test results on our main infrastructure. We'll give an update when we start downscaling our backup infra..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:12:59</var> GMT+0</small><br><strong>Monitoring</strong> - The backup infrastructure is in place and will be functional with the incremental DNS updates that are being processed..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>13:09:17</var> GMT+0</small><br><strong>Resolved</strong> - We are currently scaling back to our original production infrastructure. Some connections will be discarded in this process and will be recovered shortly..</p> ]]> Fri, 8 Dec 2023 09:11:00 +0000 https://status.visualradio.cloud/incident/clpwl757d114743b5omixerdgqn https://status.visualradio.cloud/incident/clpwl757d114743b5omixerdgqn AMS-IX Outage causing degraded Cloud performance Type: Incident Duration: 1 hour Affected Components: Cloudflare → Europe → Amsterdam, Netherlands - (AMS), Core Nov 22, 18:05:00 GMT+0 - Identified - The [currently ongoing AMS-IX outage](https://stats.ams-ix.net/index.html) is causing some connections with the VRA Cloud to drop, which may result in delayed connectivity from commando's executed from Cloud. This incident does not impact any Output or Audio app related connectivity. Nov 22, 19:05:00 GMT+0 - Resolved - Alternative AMS-IX route is working as expected, service connectivity is restored to operational state. Type: Incident

Duration: 1 hour

Affected Components: ,

<p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>18:05:00</var> GMT+0</small><br><strong>Identified</strong> - The [currently ongoing AMS-IX outage](https://stats.ams-ix.net/index.html) is causing some connections with the VRA Cloud to drop, which may result in delayed connectivity from commando's executed from Cloud. This incident does not impact any Output or Audio app related connectivity..</p> <p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>19:05:00</var> GMT+0</small><br><strong>Resolved</strong> - Alternative AMS-IX route is working as expected, service connectivity is restored to operational state..</p> ]]> Wed, 22 Nov 2023 18:05:00 +0000 https://status.visualradio.cloud/incident/clpa6qu0s91067biopox21dfd9 https://status.visualradio.cloud/incident/clpa6qu0s91067biopox21dfd9 Difficulties uploading new Output Media Type: Incident Duration: 15 days, 10 hours and 11 minutes Affected Components: Cloudflare → Images Oct 9, 14:22:10 GMT+0 - Resolved - The incident root cause at the Third Party has been fixed. We are still monitoring the situation. Oct 9, 08:12:00 GMT+0 - Identified - An currently ongoing incident at Cloudflare Images prevents users from uploading new images. We are monitoring the situation at Cloudflare closely to restore the service. Type: Incident

Duration: 15 days, 10 hours and 11 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>14:22:10</var> GMT+0</small><br><strong>Resolved</strong> - The incident root cause at the Third Party has been fixed. We are still monitoring the situation..</p> <p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>08:12:00</var> GMT+0</small><br><strong>Identified</strong> - An currently ongoing incident at Cloudflare Images prevents users from uploading new images. We are monitoring the situation at Cloudflare closely to restore the service..</p> ]]> Mon, 9 Oct 2023 08:12:00 +0000 https://status.visualradio.cloud/incident/clnisqcim119058bdmvgq35brcm https://status.visualradio.cloud/incident/clnisqcim119058bdmvgq35brcm