Monday, November 5, 2007

Resizing a RAID1 system partition.

Scenario: I have a Linux server (running Gentoo) with 2x 80 GB drives in a software RAID1 configuration that is used as the system drives to boot the server. I wanted to upgrade the drives and swap in some bigger 160 GB drives I had laying around. So, how to do it?

PS! Command syntax is for example only and is not necessarily correct nor complete.
  1. Make sure grub is installed on both drives so the system will still boot if one drive is missing.
  2. Fail all partitions from one of the drives: mdadm --manage /dev/md3 --fail /dev/hdc3
  3. Remove the drive from the array: mdadm --manage /dev/md3 --remove /dev/hdc3
  4. Shutdown server, and replace the drive with a bigger one.
  5. After reboot, create the same number of partitions that was on the old drive. Each partition must be at least as big as the one it is replacing.
  6. Add the new partitions to the array: mdadm --manage /dev/md3 --add /dev/hdc3
  7. Let them fester.. i mean rebuild. You can check the status by: cat /proc/mdstat
  8. Once finished rebuilding, repeat step 2-7 for the other drive.
  9. I now had an identical array to the one on the old drives, except i have space to resize the filesystem to make use of any extra space on the new partition scheme.
  10. First we need to make the array partition make use of all the available storage on the harddrive partitions: mdadm --grow /dev/md3 --size=max
  11. Online resizing doesn't work for mounted filesystems, and the system disks can't be unmounted while in use. So, reboot into a rescue CD. I chose gParted.
  12. Then re-created the raid array with the second drive missing. mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/hdc3 missing
  13. Ran fsck -n /dev/md3
  14. Removed the journaling from my ext3 filesystem (making it into an ext2 fs basically): tune2fs -O ^has_journal /dev/md3
  15. Ran e2fsck -f /dev/md3
  16. Ran resize2fs /dev/md3 and waited a fair amount of time.
  17. Once more ran fsck -n /dev/md3
  18. Before re-enabling the journaling: tune2fs -j /dev/md3
  19. Just to be sure, i rebooted into gParted once more, and created the array again just as in step 12.
  20. I then added the second drive to the array: mdadm --manage /dev/md3 --add /dev/hda3
  21. Sat back and watched: watch -n 1 'cat /proc/mdstat' for a long time while the second drive got rebuildt with the new filesystem size.
As a last step, I edited /etc/mdadm.conf on the system disk before I rebooted. Not really sure if it was needed, but didn't hurt.
Before I edited the file it did not contain any settings. I basically just added definitions for the arrays i have on my system disks, and listed the partitions to use for each of them.

Update:
You might wonder why I did not add both drives to the array BEFORE I resized the file system. And yes, you can indeed do this. It would probably even be faster! But: REMEBER TO MAKE A BACKUP! With my approach I had the second drive as a backup in case something went boom on the first drive.

7 comments:

  1. Excellent post! But why do you have to do all those extra steps from #11 on?

    ReplyDelete
  2. Yes. Up until that point you have only re-sized the *device*.
    #11 and later resizes the file-system to utilize the available size of the device.

    ReplyDelete
  3. well, I actually just tried this on Ubuntu 7.10 and not only didn't I need to do those extra steps, I didn't need to do them offline either.

    Here's my post on the Ubuntu forums,
    http://ubuntuforums.org/showthread.php?t=756407

    Aside from the online/offline issue, why did you say you had to disable journaling and run all those fschk's?

    ReplyDelete
  4. I am not running ubuntu, but a stock gentoo installation. I know there are some mods to the kernel that you can apply that leets you resize things on the fly, but i didn't have/want those in my setup.

    The resize command i use is for ext2 so i basically turned my ext3 into an ext2 fs by turning off journaling temporarily.

    The checkfs is basically just to force everything to be in sync.. better safe than sorry etc.

    ReplyDelete
  5. Good directions.

    I attempted this myself, but momentarily forgot md 0.9 format superblocks are stored at the end of the device and resized partitions on both disks first. Had to revert one partition table and begin anew. (Useful to keep backups.)

    Probably should've just created a new md array and added it to my LVM instead. Quite a bit of work to keep the same number of partitions on my disk, eh?

    ReplyDelete
  6. No matter if this post is from 2007 it was exactly the cookbook i needed to switch from two 750GB to 2TB.
    Thank you very much! Excellent work!

    ReplyDelete
  7. Excellent guide, easy to follow. I would add the resize2fs /dev/md0 command. My new array was reporting the full size to parted / fdisk, but my df reported the old size. I was confused for a second until I examined the block size. resize2fs fixed the issue.

    ReplyDelete