Adding New Pools to dCache
This brief tutorial covers adding new pools to our Nebraska site.
dCache needs Sun's Java installed to work; consider this step #0. You can get the RPM from scientificlinux.org
After
installing the Java JDK, replace the link from the system installed
java to the new installed java. On recent installs this
entailed something like:
rm /usr/bin/java
ln -s /usr/java/jdk1.5.0_10/bin/java /usr/bin/java
We now deploy dcache pools using the vdt install package. Grab the VDT install package from another dcache pool node. The name is something like: vdt-dcache-SL4_64-1.0.7.tar.gz, this contains everything you need execpt the latest dcache RPMS. Copy those over as well. As of this writing it was: dcache-server-1.7.0-34.noarch.rpm.
Blow up the vdt tar ball and copy the rpms to the correct directory:tar -xzvf vdt-dcache-SL4_64-1.0.7.tar.gz
cd vdt-dcache-SL4_64-1.0.7/RPMS
mv ~/dcache-server-1.7.0-34.noarch.rpm ./
Then install the RPM:
rpm -i dcache-server-1.7.0-34.noarch.rpmNow for the VDT magic. Get the site-info.def file from a working dcache pool. (Try to get the file from the last dcache installed pool so it up to date, or pull the master off of red, it should be in red's root home area or the /opt/kickstart area.) This file contains the infomation for configuring the pool, so we'll have to edit one line.
Edit the site-info.def file to include the new pool. Append the pool to the following line:
DCACHE_POOLS="dcache01.unl.edu:all:/mnt/raid/ dcache03.unl.edu:all:/mnt/raid/ dcache04.unl.edu:all:/mnt/raid/ dcache-s01.unl.edu:all:/mnt/raid/ dcache-s02.unl.edu:all:/mnt/raid1/ dcache-s02.unl.edu:all:/mnt/raid2/ dcache-s03.unl.edu:all:/mnt/raid/pool dcache-s04.unl.edu:all:/mnt/raid dcache-s04.unl.edu:all:/opt thpc-1.unl.edu:all:/pool/pool dcache-fc01.unl.edu:all:/mnt/xz1 dcache-fc01.unl.edu:all:/mnt/xz2 dcache-s05.unl.edu:all:/mnt/raid dcache-s06.unl.edu:all:/mnt/raid dcache-s07.unl.edu:all:/mnt/raid dcache-s08.unl.edu:all:/mnt/raid dcache-s09.unl.edu:all:/mnt/raid new-pool.unl.edu:all:/mount/point_for/pool"Once the site-info.def file is up to date, run the vdt install script:
./install.sh
This will place all you need to get dCache up and running in /opt/d-cache.
If you aren't going to run a gridFTP door on the server you can skip down to "Start the server".
If we want to run a gridFTP door, there are some additional steps.
If we're running this node as a gridFTP door as well as a pool, then I'll assume you've installed the OSG:ce stack to manage the certificates. Then do the following:
a) sync the /etc/passwd and /etc/group files with a working gridFTP door.
b) create the mount point /pnfs/fs and edit the /etc/fstab to mount the /pnfs filesystem, the following line added to fstab should suffice:
dcache-pnfs.unl.edu:/pnfsdoors /pnfs/fs nfs hard,intr,ro,bg,noac,auto 0 0c) create two links in the /pnfs area:
ln -s /pnfs/fs /pnfs/unl.edud) edit the node config file in /opt/d-cache/etc to turn on the gridFTP service.
ln -s /pnfs/fs /pnfs/ftpBase
vi /opt/d-cache/etc/node_configFollow all the commands for start the server with the added steps of repeating the process for dcache-core.
<Go down to here>
# ---- Services to be started on this node
# The following services are only started on this node
# if the corresponding parameter is set to 'yes'.
# Exeption: The PnfsManager is started on the admin node
# if the parameter is not specified.
GSIDCAP=no
DCAP=no
GRIDFTP=yes <----Change this line from no to yes
SRM=no
XROOTD=no
Start the server:
[root@dcache01 ~]# /opt/d-cache/bin/dcache-pool start
Starting dcache pool: Starting dcache01Domain 6 5 4 3 2 1 0 Done (pid=23660)
Additionally, copy the pool scripts to the init directory and set it to start the service on boot.
[root@dcache01 ~]# cp /opt/d-cache/bin/dcache-pool /etc/init.d/dcache-poolTo improve network performance modify the tcp settings to the following in /etc/sysct.conf:
[root@dcache01 ~]# chkconfig --add dcache-pool
[root@dcache01 ~]# chkconfig dcache-pool on
# Kernel sysctl configuration file for Red Hat Linux
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.kernel.core_uses_pid = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.core.rmem_max = 1048576
net.core.rmem_default = 87380
net.core.wmem_max = 131072
net.core.wmem_default = 32768
net.ipv4.tcp_rmem = 4096 87380 2097152
net.ipv4.tcp_wmem = 4096 32768 262144
net.ipv4.tcp_mem = 65536 87380 196608
vm.min_free_kbytes = 65536
vm.overcommit_memory = 2
Don't forget to add the pool to the resilentPools group. Modify the /opt/d-cache/config/PoolManager.conf file on the dcache headnode. There are three places the pool needs listed, register pool, default group and the resilient group. Follow the examples already in that file.
Finally, check the usage info on the webpage: http:/dcache-head.unl.edu:2288/usageInfo