It Finally Happened..............

Well after two years of luck, it finally ran out. I had an automated message about 5pm UK Time to say that one of my web urls wasnt responding. On checking the VM it indeed wasnt responding..... then the fun began.........

Not only was the url down but hidden in an avalanche of alerts in Jira was that the entire hypervisor containing approx 25 VM's and 32tb of active data had fallen over, along with my Zabbix and SIEM instances.

Luckily I have an "every 24 hour" backup strategy, and, as of 5am this morning all the VMs on that Hypervisor had backed up. So for the moment I am restoring critical VM's to one of my other two servers, then on the weekend I will begin the painful task of a full rebuild of an 80tb Hypervisor Server.

So everyone, ensure you backup regular, you never know when its your turn.......

2 comments