Many of my servers are configured with a software mirrored boot volume configured as per the FreeBSD handbook. Unfortunately the process outlined on those pages results in a mirror configuration that sometimes works in FreeBSD 9.0 and sometimes does not. Here is how I handle the situation where it does not.
Firstly, I assume you know how to run a source upgrade, and you already
have mirroring configured. The instructions are pretty clear in the
handbook and the
UPDATING file in the source tree.
The trouble will become apparent on the initial reboot into the new kernel. The tell-tale sign is that you get the following error:
GEOM_PART: integrity check failed (mirror/gm0, MBR)
and you end up with the
mountroot\> prompt. If you do not get this
problem, then your gmirror will work fine with FreeBSD 9.0, and you
don’t need to do anything otherwise special to survive this transition.
If you do get this error, the fix is to break the mirror, and re-create
it using the GPT partitions which FreeBSD 9 prefers. My examples assume
two SATA drives which in FreeBSD 8 and earlier show up as
Breaking the Mirror #
Reboot the system.
Escape to the boot loader prompt at the FreeBSD boot menu.
At the OK prompt, type
boot kernel.old -sand it will get you to the single-user FreeBSD 8 system again.
Clear the existing gmirror configuration:
gmirror remove gm0 ad6 gmirror remove gm0 ad4
This will destroy the mirror. Reboot again, into single user mode in FreeBSD 9 and resume the upgrade process. Return here to rebuild the mirrors using GPT partitions before the final reboot (or reboot back into single user).
Rebuilding GMIRROR #
The process is to clear the secondary drive, partition it using GPT, copy the data over to it, boot from this drive, re-partition the first drive similarly, and set up the mirrors for each pat. Be sure to keep the partitions similarly sized to existing ones in the old gmirror, and don’t forget any data partition we may have. This is done after booting into the system single-user. A good time is prior to the final reboot of the upgrade steps after the delete-old step or after mergemaster.
Set up the partitions #
We need to clear the drive, and partition it. In this example, I set up 512MB root file system, 2G each for swap and /usr, and the rest for /var.
dd if=/dev/zero of=/dev/ada1 count=2 gpart create -s gpt ada1 gpart add -t freebsd-boot -l boot-secondary -s 64 ada1 gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 gpart add -t freebsd-ufs -l rootfs-secondary -s 512M ada1 gpart add -t freebsd-swap -l swap-secondary -s 2G ada1 gpart add -t freebsd-ufs -l usr-secondary -s 2G ada1 gpart add -t freebsd-ufs -l var-secondary ada1 gpart show ada1
Create new file systems #
The root partition is not recommended to have soft-updates. Also, if you
have an SSD, the
-t flag is useless once the mirror is set up so there
is no point in using that. Since this is an SSD, I’m going to specify
-E flag to clear file system back to the hardware. I also like to
label my partitions so my /etc/fstab file is consistent across all
newfs -E -L root /dev/gpt/rootfs-secondary newfs -E -L usr -U /dev/gpt/usr-secondary newfs -E -L var -U /dev/gpt/var-secondary
Turn on soft-updates journaling #
Soft updates journaling is a new feature in FreeBSD 9. We enable it now on the new file systems, and should also do it for existing file systems which are not being re-mirrored.
tunefs -A -j enable /dev/gpt/var-secondary tunefs -A -j enable /dev/gpt/usr-secondary
Copy the data #
restore to copy the file systems. The
-L is probably not
necessary since they are not mounted (and in the case of root, mounted
(mount /dev/gpt/rootfs-secondary /mnt && cd /mnt && dump -C24 -0aL -f - / \| restore -rf - && cd && umount /mnt) (mount /dev/gpt/var-secondary /mnt && cd /mnt && dump -C24 -0aL -f - /var \| restore -rf - && cd && umount /mnt) (mount /dev/gpt/usr-secondary /mnt && cd /mnt && dump -C24 -0aL -f - /usr \| restore -rf - && cd && umount /mnt)
Reboot into new drive #
Break into the boot prompt and specify the alternate boot device:
set vfs.root.mountfrom=ufs:gpt/rootfs-secondary boot -s
This will now be running the system in single user mode from the second drive.
Partition the first drive #
We partition the first drive identically to the second drive we did above, just using different names:
dd if=/dev/zero of=/dev/ada0 count=2 gpart create -s gpt ada0 gpart add -t freebsd-boot -l boot-primary -s 64 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 gpart add -t freebsd-ufs -l rootfs-primary -s 512M ada0 gpart add -t freebsd-swap -l swap-primary -s 2G ada0 gpart add -t freebsd-ufs -l usr-primary -s 2G ada0 gpart add -t freebsd-ufs -l var-primary ada0 gpart show ada0
Set up the mirrors #
gmirror label -v gmroot /dev/gpt/rootfs-secondary gmirror label -v gmusr /dev/gpt/usr-secondary gmirror label -v gmvar /dev/gpt/var-secondary
And insert the partitions from the empty drive:
gmirror insert gmroot /dev/gpt/rootfs-primary gmirror insert gmusr /dev/gpt/usr-primary gmirror insert gmvar /dev/gpt/var-primary
Wait a while for it to finish sync;
gmirror status is your friend here.
Finishing Up #
The file system labels seem to get masked by the mirror labels in this configuration, so you need to update the /etc/fstab to use those.
mount -u / mount /dev/mirror/gmusr /usr mount /dev/mirror/gmvar /var
/etc/fstab to change the `/dev/ufs/XYZ to /dev/mirror/gmXYZ devices
for root, usr, and var partitions.