07 Jan 2010

Building an iSCSI Openfiler SAN on a USB Stick

As I have now rebuilt my Openfiler 2.3 iSCSI box, I thought that it would be wise to document the procedure as I have installed Openfiler on a USB memory stick. This was something I’ve wanted to do this for a while now. Basically, I’m trying to cut back on the number hard disk drives in my environment. If therefore decided to install Openfiler on a USB memory stick instead of another hard drive. I could then run 4 750GB SATA drives in RAID10 and leave the Openfiler OS to run on the USB stick.

As most servers can boot from USB, I didn’t expect any issues with installing and booting Openfiler from USB. However, Openfiler doesn’t load the USB storage drivers when it boots by default. You’ll have to tweak the initrd image in order to boot from USB.

I recommend you use a 2GB or 4GB memory stick. Although Openfiler can install in less that 1GB, you still want to leave some free space for logs and swap space. I used the SanDisk Cruzer Micro 4GB as it's a small device (physically) and it's made out of metal and not plastic and should be a little more durable. However, this memory stick also has a plastic sleeve covering the memory stick, which I removed in order to keep the memory stick as cool as possible inside my server. The server I've used is the HP ProLiant ML115 G5. This server has a USB port located on the motherboard which is perfect! As the USB memory stick will be located inside the server, it won't be be dangling on the outside where it can be "removed" by "accident". You know what people are like ;-)

1.       Unplug all SATA drives from the server. This is so that Openfiler detects the USB flash memory as /dev/sda and not as “/dev/sde” or something. It just makes things a little neater and simpler later on.

2.       Power on the server and boot from the Openfiler 2.3 installation CD.

3.       At the Openfiler Boot Menu, type linux expert and press enter. You have to parse the “expert command” or else the Openfiler installer will not display the USB flash memory as available storage to install Openfiler to.

4.       When the installer has loaded, follow the screens until you get to decided whether to manually or automatically partition the drives. For normal installations automatic partitioning should be OK, however, as we will have to do some post install work on the kernel and the fact that we are only installing Openfiler on a 2GB or 4GB volume, we'll need more control over partition device numbers (/dev/sda1,2,3, etc) as well as partition sizes. I’ve therefore created my partitions manually as follows:

Device

Mount Point

File System

Size

Primary

/dev/sda1

/boot

ext2

100MB

Yes

/dev/sda2

swap

 

1024MB

Yes

/dev/sda3

/

ext3

Fill to max

Yes

/dev/sda5

/var/log

ext3

512MB

No

 Some notes on partitioning:

  • /dev/sda4 will automatically be created as an EXTENDED partition when you create the /var/log as a non-primary partition. 
  • The reason for making /var/log a separate partition is because the log files can grow to large sizes. Making the /var/log mount point its own partition, protects the / partition from running out of space by growing log files in /var/log

 

5.       Continue with installing Openfiler. As this is now installing on a USB 2.0 device, this may take a while longer than normal to complete. Mine took a good 50 minutes to install.

6.       Once the installation has completed, keep the installation CD in the CD drive and reboot the server, this time booting from the Openfiler CD-ROM again and not from the newly installed Openfiler on the USB stick. If at this point you boot from the USB stick, the Kernel will not load USB storage drivers and won’t therefore be able to mount the root partition. This will result in a kernel panic.

7.       When the server boots from the Openfiler CD, you will again be presented with the boot menu. This time, type: linux rescue

8.       When asked to start the network system, choose “No

9.       At the rescue menu, choose “Skip”

10.   At the prompt, create a new directory called system, and mount the root filesystem (/dev/sda3) to that directory

mkdir –p /mnt/system

mount /dev/sda3 /mnt/system

11.   Now, mount the boot partition to /mnt/system/boot

mount /dev/sda1 /mnt/system/boot

12.   Change the system root to /mnt/system. This allows you to work from /mnt/system as if it was the root filesystem

chroot /mnt/system

13.   Now, extract the initrd image in /tmp

cp /boot/initrd-2.6* /tmp/initrd.gz && gunzip /tmp/initrd.gz

 

14.   Create a new directory where we can extract the initrd files to and work on them

mkdir /tmp/tmp-initrd
cd /tmp/tmp-initrd

 

15.   Extract the initrd files to the /tmp/tmp-initrd directory

 

cpio –i < /tmp/initrd

 

16.   Now we need to edit the init file under /tmp/tmp-initrd/. I like to use vi as my text editor. So, here we go:

 

vi init

Press to enter Inert Mode

Find the line with insmod /lib/sd_mod.ko

 

Enter the following lines after it:

 

insmod /lib/rs_mod.ko

insmod /lib/ehci-hcd.ko

insmod /lib/uhci-hcd.ko

sleep 5

echo “Loading USB Storage Drivers”

insmod /lib/usb-storage.ko

sleep 10

 

To save the file and quit vi, press “:wq

17.   Now we have configured the init script to load some modules when the kernel boots, however, we now need to copy the actual module files to /lib/

 

cd /lib/modules/kernel/drivers

cp usb/storage/usb-storage.ko /tmp/tmp-initrd/lib/
cp usb/host/ehci-hcd.ko /tmp/tmp-initrd/lib/
cp usb/host/uhci-hcd.ko /tmp/tmp-initrd/lib/

cp scsi/sr_mod.ko /tmp/tmp-initrd/lib/

18.   Now, all we need to do is to pack it all in a .img file and place it in /boot/. We then need to tell grub (the boot loader) where to find it.

 

cd /tmp/tmp-initrd

find . |cpio –c –o | gzip -9 > /boot/usb-initrd.img

cd /boot/grub
vi grub.conf

 

Find the line that contains the old initrd file and change the file name to usb-initrd.img

 

Save the grub.conf file and exit.

 

To reboot, type "exit" twice. The server should reboot and should now be able to mount /dev/sdb3when it boots from the USB stick.

 

Comments (3)

  1. jlabs

i found this exactly what the doctor ordered and i can tell you this saved my day thank you. only feedback would be to include an example grub.conf as i had forgotten the syntax and ended up adding /boot/ to the beginning of the line and had to...

i found this exactly what the doctor ordered and i can tell you this saved my day thank you. only feedback would be to include an example grub.conf as i had forgotten the syntax and ended up adding /boot/ to the beginning of the line and had to go back in and fix it.

Read More
  Attachments
 
  1. Sergio Sa

Hello Rynardt Spies

Thank You for this post. it is helping me lot.
I am trying to install OpenFiler in a usb stick in order to be able to install 4 500Gb hd and creat Raid 5.
Unfortunately I am not good with Linux operating...

Hello Rynardt Spies

Thank You for this post. it is helping me lot.
I am trying to install OpenFiler in a usb stick in order to be able to install 4 500Gb hd and creat Raid 5.
Unfortunately I am not good with Linux operating systems and during your Step, precisely on 17th step, I as not able to find lib/modules/kernel/drivers . there is not kernel directory in my installation neither drivers...
How can I copy the necessary drivers to /tmp/tmp-initrd/lib/ ?
I am waiting your answer.

Thank you in advance.
Regards
Sergio Sa.

Read More
  Attachments
 
  1. Rynardt Spies    Sergio Sa

Hi Sergio,

What version of OpenFiler are you trying to build?

Also, can you provide the list of directories that you can see available under /lib/modules ?

Thanks,

Rynardt

  Attachments
 
There are no comments posted here yet

Leave your comments

Posting comment as a guest. Sign up or login to your account.
0 Characters
Attachments (0 / 3)
Share Your Location

@evankirstel I know what they are for. It just feels wrong to throw a good cable out, even if it seems obsolete
Follow Rynardt Spies on Twitter