Using native IB support for GPFS

資料來源： http://mytravelingfamily.com/2009/03/04/using-native-ib-support-for-gpfs/

March 4, 2009 in Computers | Tags: gpfs, Infiniband, verbsPorts, verbsRdma

Just a quick follow on to the last post. If you try to use native IB support with GPFS instead of the IP over IB you need to set the verbsRdma to enabled. You also need to set the rdma device to use with the verbsPort setting. If you get the following error in /var/mfms/gen/mmfslog then you didn’t properly set the verbsPort



VERBS RDMA starting.
VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized.
VERBS RDMA library libibverbs.so unloaded.
VERBS RDMA failed to start, no verbsPorts defined.

Here is what I used for our single port HBA Infiniband setup.

mmchconfig verbsPorts="ib0"
mmchconfig verbsRdma=enable

Then looking at my configuration I now show things correctly

[root@topaz-m1 ~]# mmlsconfig
Configuration data for cluster aaa.bbb.navy.mil:
—————————————————-
clusterName aaa.bbb.navy.mil
clusterId 72452797724XXXXXXXX
clusterType lc
autoload no
minReleaseLevel 3.2.1.5
dmapiFileHandleSize 32
dataStructureDump /scratch/root/GPFS_Dump
maxblocksize 512K
maxFilesToCache 7000
maxMBpS 2048
maxStatCache 28000
pagepool 512M
verbsPorts ib0/1
verbsRdma enable

File systems in cluster aaa.bbb.navy.mil:
———————————————
/dev/gpfs_a
/dev/gpfs_b
/dev/gpfs_c
/dev/gpfs_d

Success. I am now running gpfs natively across Infiniband with Linux. Much better performance this way.
Our basic configuration in case you are wondering.