Upgrade FreeBSD 8.2 to 9.0 with GMIRROR boot disk

posted Jan 31, 2012, 8:32 AM by Vick Khera [ updated Jan 31, 2012, 8:48 AM ]

Many of my servers are configured with a software mirrored boot volume configured as per the FreeBSD handbook. Unfortunately the process outlined on those pages results in a mirror configuration that sometimes works in FreeBSD 9.0 and sometimes does not. Here is how I handle the situation where it does not.

Firstly, I assume you know how to run a source upgrade, and you already have mirroring configured. The instructions are pretty clear in the handbook and the UPDATING file in the source tree.

The trouble will become apparent on the initial reboot into the new kernel. The tell-tale sign is that you get the following error:

GEOM_PART: integrity check failed (mirror/gm0, MBR)

and you end up with the mountroot> prompt. If you do not get this problem, then your gmirror will work fine with FreeBSD 9.0, and you don't need to do anything otherwise special to survive this transition.

If you do get this error, the fix is to break the mirror, and re-create it using the GPT partitions which FreeBSD 9 prefers. My examples assume two SATA drives which in FreeBSD 8 and earlier show up as /dev/ad4 and /dev/ad6.

Breaking the Mirror

Reboot the system.
Escape to the boot loader prompt at the FreeBSD boot menu.
At the OK prompt, type boot kernel.old -s and it will get you to the single-user FreeBSD 8 system again.
Clear the existing gmirror configuration:

gmirror remove gm0 ad6

gmirror remove gm0 ad4

This will destroy the mirror. Reboot again, into single user mode in FreeBSD 9 and resume the upgrade process. Return here to rebuild the mirrors using GPT partitions before the final reboot (or reboot back into single user).

Rebuilding GMIRROR

The process is to clear the secondary drive, partition it using GPT, copy the data over to it, boot from this drive, re-partition the first drive similarly, and set up the mirrors for each pat. Be sure to keep the partitions similarly sized to existing ones in the old gmirror, and don't forget any data partition we may have. This is done after booting into the system single-user. A good time is prior to the final reboot of the upgrade steps after the delete-old step or after mergemaster.

Set up the partitions

We need to clear the drive, and partition it. In this example, I set up 512MB root file system, 2G each for swap and /usr, and the rest for /var.

dd if=/dev/zero of=/dev/ada1 count=2

gpart create -s gpt ada1

gpart add -t freebsd-boot -l boot-secondary -s 64 ada1

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

gpart add -t freebsd-ufs -l rootfs-secondary -s 512M ada1

gpart add -t freebsd-swap -l swap-secondary -s 2G ada1

gpart add -t freebsd-ufs -l usr-secondary -s 2G ada1

gpart add -t freebsd-ufs -l var-secondary ada1

gpart show ada1

Create new file systems

The root partition is not recommended to have soft-updates. Also, if you have an SSD, the -t flag is useless once the mirror is set up so there is no point in using that. Since this is an SSD, I'm going to specify the -E flag to clear file system back to the hardware. I also like to label my partitions so my /etc/fstab file is consistent across all machines.

newfs -E -L root /dev/gpt/rootfs-secondary

newfs -E -L usr -U /dev/gpt/usr-secondary

newfs -E -L var -U /dev/gpt/var-secondary

Turn on soft-updates journaling

Soft updates journaling is a new feature in FreeBSD 9. We enable it now on the new file systems, and should also do it for existing file systems which are not being re-mirrored.

tunefs -A -j enable /dev/gpt/var-secondary

tunefs -A -j enable /dev/gpt/usr-secondary

Copy the data

Use dump + restore to copy the file systems. The -L is probably not necessary since they are not mounted (and in the case of root, mounted read-only).

(mount /dev/gpt/rootfs-secondary /mnt && cd /mnt && dump -C24 -0aL -f - / | restore -rf - && cd && umount /mnt)

(mount /dev/gpt/var-secondary /mnt && cd /mnt && dump -C24 -0aL -f - /var | restore -rf - && cd && umount /mnt)

(mount /dev/gpt/usr-secondary /mnt && cd /mnt && dump -C24 -0aL -f - /usr | restore -rf - && cd && umount /mnt)

Reboot into new drive

reboot

Break into the boot prompt and specify the alternate boot device:

set vfs.root.mountfrom=ufs:gpt/rootfs-secondary

boot -s

This will now be running the system in single user mode from the second drive.

Partition the first drive

We partition the first drive identically to the second drive we did above, just using different names:

dd if=/dev/zero of=/dev/ada0 count=2

gpart create -s gpt ada0

gpart add -t freebsd-boot -l boot-primary -s 64 ada0

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

gpart add -t freebsd-ufs -l rootfs-primary -s 512M ada0

gpart add -t freebsd-swap -l swap-primary -s 2G ada0

gpart add -t freebsd-ufs -l usr-primary -s 2G ada0

gpart add -t freebsd-ufs -l var-primary ada0

gpart show ada0

Set up the mirrors

gmirror label -v gmroot /dev/gpt/rootfs-secondary

gmirror label -v gmusr /dev/gpt/usr-secondary

gmirror label -v gmvar /dev/gpt/var-secondary

And insert the partitions from the empty drive:

gmirror insert gmroot /dev/gpt/rootfs-primary

gmirror insert gmusr /dev/gpt/usr-primary

gmirror insert gmvar /dev/gpt/var-primary

Wait a while for it to finish sync. gmirror status is your friend here.

Finishing Up

The file system labels seem to get masked by the mirror labels in this configuration, so you need to update the /etc/fstab to use those.

mount -u /

mount /dev/mirror/gmusr /usr

mount /dev/mirror/gmvar /var

Edit /etc/fstab to change the /dev/ufs/XYZ to /dev/mirror/gmXYZ devices for root, usr, and var partitions.

Reboot.

Profit.

Comments