Recovering from Disaster · Home Lab Explorers

Recovering from Disaster

As an avid user of used and aged gear, talking through disaster recovery might help others make the right decisions as they get started, upgrade or continue to expand their home labs.

Recent Experience, background: My homelab suffered a prolonged (~1 minute) brownout condition about 4 weeks ago which caused equipment damage despite having inline power conditioning (a UPS + surge protectors) This caused downstream power issues and completely disabled my NAS.

Root cause:

UPS Type Mismatch - If your equipment is using ActivePFC power supplied (i.e. they accept 100-240V) your UPS *must* have a Pure Sine Wave output. "Simulated Sine Wave" can cause erratic behavior in ActivePFC supplies, possibly damage them and damage downstream components.

Recovery:

The mainboard / CPU were toast... I had to lift and shift drives, cards and boot devices to a new mainboard / CPU.

Challenges:

Generational leap presented challenges during reinitialization. PCI Express lane reassignment, network changes and a lack of integrated GPU on the replacement made getting back online challenging.
New processor generation means you might need a different cooling system.
New processor generation means you might need new RAM.

Lessons learned:

Document which drives are what inside the system. I found (and assumed) my boot media was an onboard nVME... I had two random SATA drives installed as well. Turns out one of the SATA SSDs was the boot drive, the onboard nVME wasn't used at all and the other "random" drive was added for ISO storage for Proxmox.
Even if you keep the same network cards from build to build, the IDs can change, which makes determining if the system worked challenging. For Proxmox, you have to ensure your network bridge is referencing the correct NIC.
Avoid changing the SAS/SATA topology during the migration. Get everything back online in as close of a configuration as before as possible, _then_ make necessary changes.

8 comments