Upgrade Freebsd 8.2 to 9.0 With Gmirror Boot Disk

Many of my servers are configured with a software mirrored boot volume configured as per the FreeBSD handbook. Unfortunately the process outlined on those pages results in a mirror configuration that sometimes works in FreeBSD 9.0 and sometimes does not. Here is how I handle the situation where it does not.

Firstly, I assume you know how to run a source upgrade, and you already have mirroring configured. The instructions are pretty clear in the handbook and the UPDATING file in the source tree.

The trouble will become apparent on the initial reboot into the new kernel. The tell-tale sign is that you get the following error:

GEOM_PART: integrity check failed (mirror/gm0, MBR)

and you end up with the mountroot\> prompt. If you do not get this problem, then your gmirror will work fine with FreeBSD 9.0, and you don’t need to do anything otherwise special to survive this transition.

If you do get this error, the fix is to break the mirror, and re-create it using the GPT partitions which FreeBSD 9 prefers. My examples assume two SATA drives which in FreeBSD 8 and earlier show up as /dev/ad4 and /dev/ad6.

Breaking the Mirror #

Reboot the system.
Escape to the boot loader prompt at the FreeBSD boot menu.
At the OK prompt, type boot kernel.old -s and it will get you to the single-user FreeBSD 8 system again.
Clear the existing gmirror configuration:

gmirror remove gm0 ad6
gmirror remove gm0 ad4

This will destroy the mirror. Reboot again, into single user mode in FreeBSD 9 and resume the upgrade process. Return here to rebuild the mirrors using GPT partitions before the final reboot (or reboot back into single user).

Rebuilding `GMIRROR` #

The process is to clear the secondary drive, partition it using GPT, copy the data over to it, boot from this drive, re-partition the first drive similarly, and set up the mirrors for each pat. Be sure to keep the partitions similarly sized to existing ones in the old gmirror, and don’t forget any data partition we may have. This is done after booting into the system single-user. A good time is prior to the final reboot of the upgrade steps after the delete-old step or after mergemaster.

Set up the partitions #

We need to clear the drive, and partition it. In this example, I set up 512MB root file system, 2G each for swap and /usr, and the rest for /var.

dd if=/dev/zero of=/dev/ada1 count=2
gpart create -s gpt ada1
gpart add -t freebsd-boot -l boot-secondary -s 64 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1
gpart add -t freebsd-ufs -l rootfs-secondary -s 512M ada1
gpart add -t freebsd-swap -l swap-secondary -s 2G ada1
gpart add -t freebsd-ufs -l usr-secondary -s 2G ada1
gpart add -t freebsd-ufs -l var-secondary ada1
gpart show ada1

Create new file systems #

The root partition is not recommended to have soft-updates. Also, if you have an SSD, the -t flag is useless once the mirror is set up so there is no point in using that. Since this is an SSD, I’m going to specify the -E flag to clear file system back to the hardware. I also like to label my partitions so my /etc/fstab file is consistent across all machines.

newfs -E -L root /dev/gpt/rootfs-secondary
newfs -E -L usr -U /dev/gpt/usr-secondary
newfs -E -L var -U /dev/gpt/var-secondary

Turn on soft-updates journaling #

Soft updates journaling is a new feature in FreeBSD 9. We enable it now on the new file systems, and should also do it for existing file systems which are not being re-mirrored.

tunefs -A -j enable /dev/gpt/var-secondary
tunefs -A -j enable /dev/gpt/usr-secondary

Copy the data #

Use dump + restore to copy the file systems. The -L is probably not necessary since they are not mounted (and in the case of root, mounted read-only).

(mount /dev/gpt/rootfs-secondary /mnt && cd /mnt && dump -C24 -0aL -f -
/ \| restore -rf - && cd && umount /mnt)

(mount /dev/gpt/var-secondary /mnt && cd /mnt && dump -C24 -0aL -f -
/var \| restore -rf - && cd && umount /mnt)

(mount /dev/gpt/usr-secondary /mnt && cd /mnt && dump -C24 -0aL -f -
/usr \| restore -rf - && cd && umount /mnt)

Reboot into new drive #

reboot

Break into the boot prompt and specify the alternate boot device:

set vfs.root.mountfrom=ufs:gpt/rootfs-secondary

boot -s

This will now be running the system in single user mode from the second drive.

Partition the first drive #

We partition the first drive identically to the second drive we did above, just using different names:

dd if=/dev/zero of=/dev/ada0 count=2
gpart create -s gpt ada0
gpart add -t freebsd-boot -l boot-primary -s 64 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
gpart add -t freebsd-ufs -l rootfs-primary -s 512M ada0
gpart add -t freebsd-swap -l swap-primary -s 2G ada0
gpart add -t freebsd-ufs -l usr-primary -s 2G ada0
gpart add -t freebsd-ufs -l var-primary ada0
gpart show ada0

Set up the mirrors #

gmirror label -v gmroot /dev/gpt/rootfs-secondary
gmirror label -v gmusr /dev/gpt/usr-secondary
gmirror label -v gmvar /dev/gpt/var-secondary

And insert the partitions from the empty drive:

gmirror insert gmroot /dev/gpt/rootfs-primary
gmirror insert gmusr /dev/gpt/usr-primary
gmirror insert gmvar /dev/gpt/var-primary

Wait a while for it to finish sync; gmirror status is your friend here.

Finishing Up #

The file system labels seem to get masked by the mirror labels in this configuration, so you need to update the /etc/fstab to use those.

mount -u /
mount /dev/mirror/gmusr /usr
mount /dev/mirror/gmvar /var

Edit /etc/fstab to change the /dev/ufs/XYZ to /dev/mirror/gmXYZ devices for root, usr, and var partitions.

Reboot.

Profit.