GPFS 安裝

要安裝kernel source
/usr/src/kernels要有SMP kernel的目錄
2.6.9-42.0.3.EL-smp-x86_64

如果沒有可以執行
yum install kernel-smp kernel-smp-devel






gpfs-3.1.0-6.x86_64.update.tar.gz
放在/opt/pkg/tarball/gpfs/

解壓縮產生
changelog
README
gpfs.msg.en_US-3.1.0-6.noarch.rpm
gpfs.base-3.1.0-6.x86_64.update.rpm
gpfs.docs-3.1.0-6.noarch.rpm
gpfs.gpl-3.1.0-6.noarch.rpm

rpm -ivh gpfs.msg.en_US-3.1.0-6.noarch.rpm
rpm -ivh gpfs.docs-3.1.0-6.noarch.rpm
rpm -ivh gpfs.gpl-3.1.0-6.noarch.rpm
rpm -ivh --noscripts gpfs.base-3.1.0-6.x86_64.update.rpm

[root@cpa001 ~]# rpm -qa|grep gpfs
gpfs.base-3.1.0-6
gpfs.msg.en_US-3.1.0-6
gpfs.docs-3.1.0-6
gpfs.gpl-3.1.0-6


四台NSD(cpa001,cpa002,cpa003,cpa004)要能夠互相以root不輸入密碼來ssh
方法如下:
1.從cpa001 ssh 連線到"自己本身cpa001",cpa002,cpa003,cpa004各一次
從cpa001 ssh 連線到"自己本身iba001",iba002,iba003,iba004各一次
目的是為了讓 known_hosts 有8筆連線紀錄(4筆Ethernet ,4筆infinband)
2.於cpa001:ssh-genkey -t dsa
cat id_dsa.pub >> authorized_key
目的為了讓root可以不用輸入密碼進行連線
3.複製cpa001:/root/.ssh/這整個目錄內容到 cpa002,cpa003,cpa004:/root/.ssh
4.如此一來就可以root 互相進入四台連線





修改cpa001的site.mcr
1. cp site.mcr.proto site.mcr
vi site.mcr

#define GPFS_ARCH_X86_64

LINUX_DISTRIBUTION = REDHAT_LINUX 保持預設值
#define LINUX_DISTRIBUTION_LEVEL 90 保持預設值

#define LINUX_KERNEL_VERSION 2060942
KERNEL_HEADER_DIR = /lib/modules/`uname -r`/build/include

2.
cd /usr/lpp/mmfs/src
make Autoconfig
就可以自動產生 /usr/lpp/mmfs/src/config/site.mcr

檢查
/lib/modules/2.6.9-42.0.3.ELsmp/build/include/asm
應該要有ioctl32.h

cd /usr/lpp/mmfs/src/
make
export SHARKCLONEROOT=`pwd`
make World
cd ibm-kxi/
ln -s /usr/lpp/mmfs/include/gpfs_gpl.h .
cd ..
make World
make InstallImages


複製/usr/lpp/mmfs/bin目錄下面的執行檔
dumpconv
lxtrace
mmfs26
mmfslinux
tracedev
到其他NSD(cpa002,cpa003,cpa004)的相同目錄下
scp dumpconv lxtrace mmfs26 mmfslinux tracedev cpa002:/




準備/etc/gpfs.allnodes
[root@cpa001 bin]# cat /etc/gpfs.allnodes
iba001:quorum-manager
iba002:quorum-manager
iba003:quorum-manager
iba004:quorum-manager

準備/etc/gpfs.alldisks_emc
[root@cpa001 bin]# cat /etc/gpfs.alldisks_emc
/dev/sdb1:iba001::dataAndMetadata::
/dev/sdc1:iba002::dataOnly::
/dev/sdd1:iba003::dataAndMetadata::
/dev/sde1:iba004::dataOnly::

其中/dev/sdb1 ,sdc1 ,sdd1 ,sde1 需要先格式化成ext3
如果重新create nsd, 也還要重複格式化一次
mkfs.ext3 ,讓partition可以被使用



[root@cpa001 bin]# ./mmcrcluster -p iba001 -n /etc/gpfs.allnodes -r /usr/bin/ssh -R /usr/bin/scp

[root@cpa001 bin]# ./mmlscluster

GPFS cluster information
========================
GPFS cluster name: iba001
GPFS cluster id: 13882357951455911707
GPFS UID domain: iba001
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
Primary server: iba001
Secondary server: (none)

Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------------------------
1 iba001 192.168.11.1 iba001 quorum-manager
2 iba002 192.168.11.2 iba002 quorum-manager
3 iba003 192.168.11.3 iba003 quorum-manager
4 iba004 192.168.11.4 iba004 quorum-manager


要先mmcrcluster ,才能用mmstartup -a (第一台會帶動其他三台NSD啟動gpfs)

./mmstartup -a後會看到載入

lsmod查看
Module Size Used by
mmfs 1020736 2
mmfslinux 224360 7 mmfs
tracedev 15992 3 mmfs,mmfslinux

ps aux查看
root 7432 0.0 0.0 4984 2828 pts/0 S< 15:47 0:00 /bin/ksh /usr/lpp/mmfs/bin/runmmfs root 7564 0.0 0.9 1179032 74028 pts/0 S /etc/redhat-release
2. cd /usr/lpp/mmfs/src/
3. make Autoconfig

cd config; ./configure; exit $? || exit 1;

1. Make World <----Now it works!!]
2. Make InstallImages <----Also works fine


I know CentOS is probably not technically supported. But it works because all the libraries and binaries are identical to RHEL (which is the aim of CentOS), just the configure script seems to be hardwired to require RHEL designation in /etc/redhat-release.

Thanks for helping me troubleshoot this in any case!!

Cheers,
some_guy

Message was edited by: some_guy