Affecting Other - SGP
Dear Valued Customers,
We want to inform you that our Singapore data center is currently unreachable, potentially due to a DDoS attack. Our team is actively investigating the issue and working to resolve it as quickly as possible.
We appreciate your patience and will provide updates as soon as we have more information.
Thank you for your understanding.
Update: issue has been resolved, rca to be updated soon.
Update 2: issue resurfaced, the team is working on mitigation
Update 3: issue has been resolved, the team is monitoring.
Affecting System - SPK
Affecting Other - SPK
The team is investigating an issue with IPv4 DNS being unreachable (outgoing) from our Spokane, WA location.
We will provide updates here regarding this issue.
Affecting Other - SPK
The network engineering team has identified the issue as a line card failure impacting the main upstream connection. They are actively working to resolve it, and we will provide an estimated time for repair (ETR) as soon as possible. If you wish to raise a request for compensation, please submit a support ticket, and compensation will be provided as per the SLA outlined in our Terms of Service, if applicable. Thank you for your patience.
Update 1
Our team has identified a solution for the issue, and we’re actively working to restore connectivity. Estimated time for repair (ETR) is approximately 1 hour. Thank you for your patience as we bring services back online. We’ll provide further updates if anything changes.
Update 2
The network is back online with limited capacity after the initial backup line didn’t fully resolve the issue. We've switched to a different line card and we are working on restoring the full capacity. Thank you for your patience. More updates to follow.
Resolved.
Affecting System - COVENTRY
Affecting System - SGP
An ongoing ddos attack is impacting the Singapore network. We are working on resolving this at the earliest.
Update: the issue is resolved, and we are closely monitoring the situation.
Updated: Closing the network issue, services are restored and operating normally.
Affecting System - SPK-10
SPK-10 is scheduled for a hardware upgrade at 22:00 GMT+4 20th August 2024.
Update: Hardware upgrade completed, raid resyncing.
Affecting Other - SPK-10
SPK-10 is scheduled for a hardware upgrade.
Update: SPK-10 is being shut down for hardware upgrades.
Update: SPK-10 hardware replacement completed. Raid resync in progress.
Affecting System - SPK-10
We are investigating an issue with SPK-10
Update: SPK-10 being rebooted, control panel actions will be unresponsive till the HN boots up.
Update: Issue has been resolved; monitoring.
Update: Issue has resurfaced, team is investigating.
Affecting Other - SPK-10
SPK-10 suffered a power outage and service has been restored. Currently monitoring the situation.
Affecting System - SGP-01
Inbound DDoS SGP-01
Update: Resolved. Monitoring the situation.
Affecting System - SGP-01
Inbound DDOS attack reported. The team is working on restoring access at the earliest.
Edit: Service has been restored.
Affecting System - SPK-10
Investigating RAID array degradation on SPK-10.
DR backup strategy has been activated.
SPK-10 Scheduled Activity - 29th Apr 8 PM GMT+4
Update: Server is brought down for a Scheduled Activity
Update 2: Server is back online, and VPSs are booting up.
Update 3: Performance impacted due to RAID rebuild, will update this notification once it is completed.
Update 4: RAID rebuild completed.
Affecting System - SPK-10
Investigating the outage.
Update: Service has been restored.
Affecting System - SPK-14
Server is rebooting for an urgent maintenance.
Update: server is back online.
Affecting System - SGP-01
We are aware of a system outage on SGP-01, and the team is investigating.
Issue has been resolved. Full RCA to be provided.
Affecting System - SPK-12
One of our NVMe drives has experienced a failure. Our technical team is actively investigating the issue to restore services as quickly as possible. This NVMe drive is non hot-swap and replacing the drive will result in downtime. As a precautionary measure, some VPSs will be migrated to other hypervisors to ensure minimal disruption. We apologize for any inconvenience this may cause and appreciate your patience and understanding during this time. Rest assured, we are working diligently to resolve the issue and restore normal operations.
Migrating the VPS to other nodes allows you to avoid the necessary downtime needed to replace the physical NVMe drive, and the risk of losing your data. If you are unhappy with this operational migration, please do open a ticket and we will move you back to SPK-12 once the service is restored.
Backups of non-migrated/remaining servers on SPK-12 is now completed. SPK-12 will be scheduled for drive replacement.
Update: SPK-12 is currently down under maintenance.
Update 2: SPK-12 is currently up, rebuilding RAID.
Update 3: SPK-12 services are back to normal.
Thank you for your continued support.
Affecting System - SPK-10
SPK-10 scheduled for urgent reboot - libvirtd bug
Affecting Other - SPK-14
We are investigating the reason of a recent downtime on SPK-14
Affecting Other - SPK-14
SPK-14 is scheduled for a maintenance for a kernel upgrade.
Update: maintenance is in progress.
Update: maintenance completed, server has been booted, and virtual machines are booting up.
Affecting Other - SPK-12
The server SPK-12 has been showing signs of instability.
The team has brought the server down to perform a chassis swap.
The team has brought the server back online.
The server is currently online, but still showing signs of instability - the team is investigating the issue with the datacenter.
Course of action, replacement of RAM and possible rollback of bios.
Planned activity: RAM replacement scheduled for 06:00 AM GMT+4, 1st December 2023 (30 minutes downtime) - impacted due to weather conditions.
Replanned for 2nd December 11 PM GMT+4.
Activity In Progress
RAM replacement has been completed.. The service is back online.
