SOAR

At 2:15 PM EST today our network monitors informed us that all of our primary data servers were inaccessible.

We immediately placed a support call to our service provider www.cogecopeer1.com. They confirmed there was a network issue that affected their entire San Antonio, TX datacenter. That is something that frankly, shouldn’t happen. They are one of the largest hosting companies and redundancy should have prevented this.

We have 4 dedicated servers in that location. These are the servers that host the websites, database servers, file storage, and development environments for your unit websites.

When we have had outages in the past it is usually related to a single server. We can easily get access to backup data on one of the other servers. That allows us to fix/rebuild the affected server and communicate with you - our customers.

Without access to any of these servers we couldn't pull customer contacts and communicate with you to give you updates.

At 3:15 PM EST we received an email confirming the outage, providing no ETA for the issue.

At 4:25 PM EST we received a second email - they are still working on the issue and no ETA for restoration of network services.

At 6:15 PM EST we received a 3rd email that informed us they had found the problem – a fiber cut from their upstream connection.

Between 6:30-45 PM EST we emailed a single point of contact for each customer with the above information. The email went to the person that paid the renewal fee last. We had that information from PayPal, our payment processor.

The network issue was fixed at 7:00 PM EST and service to your website was restored. Our servers were patiently waiting for your return :)

You can see the communications we received from Cogeco Peer 1 at status.cogecopeer1.com.

While we have used this company for almost a decade, we are devastated that your SOAR website down for 5 hours, the issue was out of our control, and we do not have the ability to communicate any of that to you directly.

In the next couple days we will receive a detailed report from Cogeco Peer 1 on what happened. After that, we will be looking into how to better serve our customers to help insure this does not happen again.

SOAR Support