Rebuild Procedure (disaster recovery)

The straightforward, low‑risk process for rebuilding the RAID5 array and restoring the system exactly as it was. The workflow captures the important metadata first, rebuilds the array with the same parameters, recreates the aligned ext4 filesystem, and restores data using integrity‑preserving tools. Once mounted, services are brought back online and validated to ensure permissions, ACLs, and the media pipeline all function correctly. A clear backup strategy and verification steps keep the system recoverable and dependable long‑term.

Goal: Recreate the RAID5 array and restore data from backups with minimal risk and clear verification steps.

1. Document current state (before teardown)

Collect identifiers and configs:
- sudo mdadm --detail /dev/md0 — record UUID, devices, chunk size, bitmap.
- sudo blkid /dev/md0 — record filesystem UUID.
- cat /etc/fstab — save mount entry.
- sudo mdadm --examine /dev/sd* — save per‑disk superblocks.
- Export service configs and data: /var/lib/sonarr, /var/lib/radarr, /var/lib/prowlarr, /home/plutus/nzbget, systemd unit files.
Why: ensures you can recreate identical layout and restore services exactly.

2. Prepare replacement drives (wipe and partition)

Wipe partition table:

bash

sudo sgdisk -Z /dev/sdX

Create partition(s) matching original layout (example creates a single RAID partition):

bash

sudo sgdisk -n1:0:0 -t1:fd00 /dev/sdX

Verify with lsblk and sudo sgdisk -p /dev/sdX.

Why: identical partition type and alignment avoids stripe misalignment and performance loss.

3. Create mdadm array

Recreate array with same parameters (example):

bash

sudo mdadm --create /dev/md0 \
  --level=5 \
  --raid-devices=5 \
  --chunk=512 \
  --bitmap=internal \
  /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde

Save mdadm config:

bash

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u

Why: use the same chunk size, layout, and bitmap to match original performance and rebuild behavior.

4. Create filesystem with stripe alignment

Create ext4 tuned for RAID:

bash

sudo mkfs.ext4 -E stride=128,stripe-width=512 /dev/md0

Label or record UUID (use blkid to capture).

Why: stride/stripe‑width alignment reduces write amplification and improves throughput.

5. Mount and restore data

Mount:

bash

sudo mkdir -p /media/plutus
sudo mount /dev/md0 /media/plutus

Restore from backup (example using rsync):

bash

sudo rsync -aHAX --progress /path/to/backup/ /media/plutus/

Alternative tools: borg, rclone, or tape restore depending on backup method.

Why: restore in a controlled way; preserve permissions and ACLs.

6. Recreate fstab entry

Get UUID and add to /etc/fstab:

bash

sudo blkid /dev/md0
# then edit /etc/fstab and add:
# UUID=<uuid> /media/plutus ext4 defaults,noatime,nofail 0 2

Mount via fstab to verify:

bash

sudo umount /media/plutus
sudo mount -a
df -h /media/plutus

Why: ensures consistent mounts across reboots and avoids device node drift.

7. Restart services and validate imports

Start services:

bash

sudo systemctl start sonarr radarr prowlarr qbittorrent-nox nzbget

Verify imports and service health:

bash

systemctl status sonarr radarr prowlarr qbittorrent-nox nzbget
journalctl -u sonarr -n 200
# Test a sample import cycle in Sonarr/Radarr

Verify Samba and ACLs:

bash

sudo systemctl restart smbd
sudo testparm -s
getfacl /media/plutus | head -n 20

Why: confirm services can read/write and that the pipeline functions end‑to‑end.

Safety note

Test in a lab before performing on production hardware.
Double‑check device nodes before failing/removing drives.
Keep backups verified and accessible before any destructive operations.

Backups & Durability Strategy (Concise)

Metadata & configs: daily backups of /var/lib/sonarr, /var/lib/radarr, /var/lib/prowlarr, /home/plutus/nzbget, and systemd unit files. Use rsync or borg to an offsite or separate node.
Media: primary copy on RAID5; maintain at least one offsite copy for irreplaceable data (cloud via rclone or a second physical backup).
Automated verification: periodic checksum catalog (e.g., sha256sum) and scheduled restore drills to validate backups.
Retention & rotation: keep multiple restore points and test restores quarterly.

Appendix — Useful Commands (copy/paste)

bash

# RAID status
sudo mdadm --detail /dev/md0
cat /proc/mdstat

# SMART health (per drive)
sudo smartctl -a /dev/sda

# Filesystem metadata
sudo tune2fs -l /dev/md0
sudo dumpe2fs -h /dev/md0

# Services
systemctl status sonarr radarr prowlarr qbittorrent-nox nzbget
journalctl -u sonarr -f

# Samba
sudo testparm -s
smbclient -L localhost -N

# Rebuild watch
watch -n 10 cat