Between 4:40pm and 6:10pm PST on 4/5 StoreSecured experienced intermittent unexpected downtime for approximately 1/4 of the visitors to all sites. As of 6:10pm all sites were operating as normal and 100% of traffic has been restored. The downtime did not affect any other services such as email, ftp, etc which were fully operational throughout.
Technical explanation for those who are interested…
The StoreSecured system uses 4 servers to display all site traffic and a internal networking cable on one of the servers suddenly failed. Because we use 4 different servers this downtime was only experienced by approximately 1/4 of all of the total traffic during this time and it was intermittent, ie the hardware would work and then it would fail again and again. The problem was identified and the offending server was removed from the rotation so that all traffic was split between the remaining 3 servers. The bad network cable has now replaced and all traffic is again going to all 4 servers as normal.