Major connectivity outage in Johannesburg
Update - We've confirmed that the disk array is failed and we're waiting for delivery of new SSDs.
Nov 26, 08:20 UTC
Update - We're aware the node is down. Working on bringing it back again.
Nov 25, 23:54 UTC
Update - We've set up temporarily transit over IPv6 tunnel. Still waiting for further investigation from our upstream.
Nov 20, 21:54 UTC
Identified - We're aware that our only IP transit provider MFN (Metrofibre Networx) is down for 11 hours. Our NAPAfrica link is still up but our IPv4 network is only reachable via peering now.

They've acknowledged about this 6 hrs ago but no ETA is given.
Nov 19, 16:40 UTC
Public API Operational
Cloud Servers Operational
Object Storage Operational
Block Storage Operational
Package Mirrors Operational
Authoritative DNS Operational
Recursive DNS Operational
DNS Updates Operational
Analytics Operational
SDN Controller Operational
Mail Queue Operational
Legacy Services Operational
SolusVM Master Operational
Metadata Service (Legacy) Operational
WHMCS (ZeptoVM) Operational
External Services Operational
Cloudflare CDN Operational
Mailgun Operational
Stripe API Operational
Datacenters Major Outage
Ashburn, VA, United States - (IAD01) Operational
Reston, VA, United States - (IAD03) Operational
Kansas City, MO, United States - (MCI01) Operational
Tukwila, WA, United States - (SEA03) Operational
Fremont, CA, United States - (FMT02) Operational
Los Angeles, CA, United States - (LAX07) Operational
Montréal, QC, Canada - (MTL03) Operational
São Paulo, Brazil - (SAO03) Operational
Amsterdam, Netherlands - (AMS03) ? Operational
Amsterdam, Netherlands - (AMS04) ? Operational
Frankfurt, Germany - (FRA08) ? Operational
London, United Kingdom - (LHR06) ? Operational
London, United Kingdom - (LHR07) ? Operational
Madrid, Spain - (MAD02) Operational
Milan, Italy - (MIL01) Operational
Prague, Czech Republic - (PRG01) Operational
Sofia, Bulgaria - (SOF01) Operational
Stockholm, Sweden - (STO01) Operational
Oslo, Norway - (OSL01) Operational
Moscow, Russia - (MSK04) ? Operational
Saint Petersburg, Russia - (LED01) Operational
Khabarovsk, Russia - (KHV01) Operational
Tokyo, Japan - (NRT04) ? Operational
Tokyo, Japan - (NRT05) ? Operational
Hong Kong - (HKG12) ? Operational
Seoul, South Korea - (ICN05) Operational
Johannesburg, South Africa - (JNB01) Major Outage
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Nov 30, 2020

No incidents reported today.

Nov 29, 2020

No incidents reported.

Nov 28, 2020

No incidents reported.

Nov 27, 2020

No incidents reported.

Nov 26, 2020

Unresolved incident: Major connectivity outage in Johannesburg.

Nov 25, 2020
Nov 24, 2020

No incidents reported.

Nov 23, 2020

No incidents reported.

Nov 22, 2020

No incidents reported.

Nov 21, 2020

No incidents reported.

Nov 20, 2020
Nov 19, 2020
Scheduled Maintenance of Khabarovsk Region
Resolved - All done.
Nov 19, 22:32 UTC
Update - Migration is still in progress, remaining 23% and ETA 50 minutes.
Nov 19, 21:48 UTC
Update - Migration is still in progress, ETA 1.5 hrs
Nov 19, 19:48 UTC
Update - Server is back though IO performance is not fixed.

We'll continue the migration and let you know once it's done.
Nov 19, 16:09 UTC
Update - Updates from provider: We have checked and see that your issue is demanded more time to be resolved. Our engineers from HDWR are working on it!
Nov 19, 06:18 UTC
Update - Still waiting for our provider to change the RAID controller. Service is down for 30hrs+
Nov 18, 14:19 UTC
Update - Passing the RUKHV01C08 node to our provider's hardware team, this node will be temporarily offline for diagnosis.
Nov 16, 09:36 UTC
Update - All VMs on C08 are now migrated to new system.
Due to the I/O performance issue, we will pause migration of other nodes before it's solved.
Nov 14, 06:50 UTC
Update - Migration still in progress (18/50)
Nov 14, 05:03 UTC
Update - Data migration is still in progress, now 20%.
Also, We noticed that the RAID controller installed is not working well.
Unfortunately it's weekend and we'll ask our provider to replace it next week. Before it's replaced, we'll limit IOPS and throughput of each VM to reduce the load.
Nov 14, 03:58 UTC
Update - Data migration is still in progress.
We except that it can be done in 12 hours.
Nov 14, 00:50 UTC
Update - We're still waiting for the data migration. Current speed is only around 80MB/s.
We'll contact our provider to check the SSD speed issue next week.
Nov 13, 23:26 UTC
Identified - Maintenance took longer than excepted due to slow I/O performance.

Approximately 50 servers are still down.
Nov 13, 22:20 UTC
Nov 18, 2020
Nov 17, 2020

No incidents reported.

Nov 16, 2020
Network outage in Reston, Ashburn
Resolved - This incident has been resolved.
Nov 16, 04:26 UTC
Update - Network is back however there's a node failing due to NIC issue.
We've dispatched local technician to replace the faulty NIC.
Nov 16, 02:51 UTC
Investigating - One of our core network equipment is flapping, we're applying an emergency fix.
Nov 16, 02:01 UTC
← Incident History Powered by Statuspage