Lustre – Cluster File System : Quick Setup

Luster File System Quick Setup

                        The Lustre file system is a distributed high performance cluster filesytem that redefines I/O performance and scalability for large and complex computing environments. This is ideally suited for data-intensive applications which requires the high IO performance.

Lustre components:
MDS – Metadata Server: The MDS server makes metadata stored in the metadata targets available to Lustre clients.

MDT – Metadata Target: This stores metadata, such as filenames, directories, permissions, and file layout, on the metadata server.

OSS – Object Storage Server: The OSS provides file I/O service, and network request handling for the OSTs. The MDT, OSTs and Lustre clients can run concurrently on a single node. However, a typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.

OST – Object Storage Target: The OST stores data as data objects on one or more OSSs. A single Lustre file system can have multiple OSTs, each serving a subset of file data.

Client: The systems that mount the Lustre filesystem.


Steps to create Lustre File System:
Configure Lustre Management Server (lustre-mgs.unixfoo.biz – Server 1)
   1. Add the disk to volume manager

      [root@lustre-mgs mnt]# pvcreate /dev/sdb
      Physical volume "/dev/sdb" successfully created

      [root@lustre-mgs mnt]# pvs
        PV         VG   Fmt  Attr PSize   Pfree
        /dev/sdb        lvm2 --   136.73G 136.73G

   2. Create lustre volume group

      [root@lustre-mgs mnt]# vgcreate lustre /dev/sdb
        Volume group "lustre" successfully created
      [root@lustre-mgs mnt]# vgs
        VG     #PV #LV #SN Attr   VSize   Vfree
        lustre   1   0   0 wz--n- 136.73G 136.73G

   3. Create logical volume "MGS" (the management server)

      [root@lustre-mgs ~]# lvcreate -L 25G -n MGS lustre

   4. Create Lustre Management filesystem.

      [root@lustre-mgs ~]# mkfs.lustre --mgs /dev/lustre/MGS

         Permanent disk data:
      Target:     MGS
      Index:      unassigned
      Lustre FS:  lustre
      Mount type: ldiskfs
      Flags:      0x74
                    (MGS needs_index first_time update )
      Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
      Parameters:

      checking for existing Lustre data: not found
      device size = 10240MB
      2 6 18
      formatting backing filesystem ldiskfs on /dev/lustre/MGS
              target name  MGS
              4k blocks     0
              options        -J size=400 -q -O dir_index,uninit_groups –F

      mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS  -J size=400 -q -O dir_index,uninit_groups -F /dev/lustre/MGS
      Writing CONFIGS/mountdata
      [root@lustre-mgs ~]#

   5. Activate the lustre management filesystem using mount command.

      [root@lustre-mgs ~]# mount -t lustre /dev/lustre/MGS /lustre/MGS/
      [root@lustre-mgs ~]# df
      Filesystem           1K-blocks      Used Available Use% Mounted on
      /dev/sda2             54558908   5276572  46466144  11% /
      /dev/sda1               497829     29006    443121   7% /boot
      tmpfs                  8216000         0   8216000   0% /dev/shm
      /dev/lustre/MGS       10321208    433052   9363868   5% /lustre/MGS
      [root@lustre-mgs ~]#

Configure Lustre Metadata Server (In this guide, both the management & metadata server runs on the same host)
   1. Create logical volume "MDT_unixfoo_cloud"

      [root@lustre-mgs ~]# lvcreate -L 25G -n MDT_unixfoo_cloud lustre

   2. Create Lustre Metdata filesystem for the filesystem “unixfoo_cloud”.

      [root@lustre-mgs ~]# mkfs.lustre --fsname=unixfoo_cloud --mdt  --reformat --mgsnode=lustre-mgs@tcp0 /dev/lustre/MDT_unixfoo_cloud
         Permanent disk data:
      Target:     unixfoo_cloud-MDTffff
      Index:      unassigned
      Lustre FS:  unixfoo_cloud
      Mount type: ldiskfs
      Flags:      0x71
                    (MDT needs_index first_time update )

      Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
      Parameters: mgsnode=10.217.0.237@tcp mdt.group_upcall=/usr/sbin/l_getgroups
      device size = 20480MB
      2 6 18

      formatting backing filesystem ldiskfs on /dev/lustre/MDT_unixfoo_cloud
              target name  unixfoo_cloud-MDTffff
              4k blocks     0
              options        -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups –F
      mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-MDTffff  -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/lustre/MDT_unixfoo_cloud
      Writing CONFIGS/mountdata
      [root@lustre-mgs ~]#

   3. Activate the metdata filesystem using mount command.

      [root@lustre-mgs ~]# mkdir /lustre/MDT_unixfoo_cloud
      [root@lustre-mgs ~]# mount -t lustre  /dev/lustre/MDT_unixfoo_cloud /lustre/MDT_unixfoo_cloud

Configure OSTs ( servers: oss1, oss2 .. )
   1. Add /dev/md0 to volume manager

      [root@oss1 ~]# pvcreate /dev/md0

   2. Create volume group "lustre"

      [root@oss1 ~]# vgcreate lustre /dev/md0

   3. Create logical volume (ost) for unixfoo_cloud

      [root@oss1 ~]# lvcreate -n OST_unixfoo_cloud_1 -L 100G lustre

   4. Create OST lustre filesystem

      [root@oss1 ~]# mkfs.lustre --fsname=unixfoo_cloud --ost --mgsnode=lustre-mgs@tcp0 /dev/lustre/OST_unixfoo_cloud_1

      mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-OSTffff  -J size=400 -i 16384 -I 256 -q -O dir_index,uninit_groups -F /dev/lustre/OST_unixfoo_cloud_1
      Writing CONFIGS/mountdata
      [root@oss1 ~]#

   5. Activate the OST by using mount command

      [root@oss1 ~]# mkdir -p /lustre/unixfoo_cloud_oss1
      [root@oss1 ~]# mount -t lustre /dev/lustre/OST_unixfoo_cloud_1 /lustre/unixfoo_cloud_oss1

Mount on the client:
   1. Mount the lustre filesystem unixfoo_cloud

      [root@lustreclient1 ~]# mount -t lustre lustre-mgs@tcp0:/unixfoo_cloud /mnt

      [root@lustreclient1 ~]# df –h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/sda2              52G  5.1G   44G  11% /
      /dev/sda1             487M   29M  433M   7% /boot
      tmpfs                 7.9G     0  7.9G   0% /dev/shm
      lustre-mgs@tcp0:/unixfoo_cloud
                             99G  461M   93G   1% /mnt
      [root@lustreclient1 ~]#

   2. Done.

Thus the Quick Setup of Luster File System with One MGS/MDS and two OSS and a Client.