Webhosting2024 offline and restore

Quags

Administrator
Staff member
The webhosting2024 server is currently down, and unable to boot. All data appears unaffacted, but after much work on the system with out a resolution we are beginning a restore to a rebuild server.
 

Quags

Administrator
Staff member
The new build is up with the backups copied to it, as restore is being started.

We will update when completed and then further analysis and a reason for outage will be written with further details.
 

Quags

Administrator
Staff member
Currently the backup has been restored. Accounts that were not included could have been over the standard backup size or newly created with in the last 6 days. For those accounts they will be restored beginning now. Once completed, accounts who need more recent data - such as changed in the last 48 hours - will be able to request it once the next backup phase is restored.

Here are the details so far which we know:

Around 7AM EST the server monitoring displayed the webserver as down. The system was responding to ping and ssh accepted an initial connection but it was not completed. No details were on console.

The system was rebooted but failed to boot, due to a boot loader/grub error. The system was brought into a rescue mode to check the grub config and repair it. This was completed, however the server failed to boot giving little information on an error.

The system was brought into a live cd. Some partitions could be seen, but it was returning an error on a file system check. At this point a new server build was began in case backups needed to be restored from by another team

After several hours, it appeared nothing further could be done. With the new server ready the backup drives were moved to the new server to begin a restore.


Currently we do not know why the system failed, as it didn't show any harddrive errors before hand and the system had no noticeable errors otherwise. The server was also a recent build, less than 2 months, and had been a build to replace a server reaching end of life. A weekly backup was running during the time. Once the second and third phase backups are completed further testing will be done to try to determine where the error may have come from. The data on the previous server is now readable by our techs.
 

Quags

Administrator
Staff member
All data has been restored, most by Monday with some that required more time. InterServer will make an announcement on further backup changes shortly.
 
Top