Pre-Install

This install process is based on pacman; to install Pacman, do the following:

mkdir /opt/pacman
cd /opt/pacman
wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.26.tar.gz
tar zxf pacman-3.26.tar.gz
cd pacman-3.26
source setup.sh

Installation

There is a special Pacman install for Nebraska; based on the hostname, it decides which software is needed.

Please help keep the installation method automatic!  If there are any failures, do not fix them - report them and make sure they are fixed in the pacman install!

All questions should be pre-answered; root install is required because FUSE modules must be installed.

pacman -get http://t2.unl.edu/store/cache:Hadoop-Nebraska

Post-install, all you should need to do is

vdt-control --on

Deployment Notes

After installing the software, there are some post-install activities based on what type of node you just added.

Adding a node which needs FUSE (worker nodes and SRM nodes)

1. Do a "ls" on "/mnt/hadoop"; it should not be empty.
1. If it is empty, make sure that vdt-control --on is set, and hadoop_fuse is set to enabled for vdt-control.
2. (Temporary; should disappear in next packaging attempt) If this is a CentOS 5 node, fix the link in $HADOOP_HOME/build/libhdfs/libhdfs.so to point at libhdfs.so.1.
   
3. Contact Brian if FUSE does not work.  Have the following info:
   
1. Does $VDT_LOCATION/fuse exist?
2. Is kernel-devel RPM installed?
3. What is the version printed out from "uname -r" compared to "rpm -q kernel-devel"?
   
2. To manually mount fuse on a worker node, do the following
   

   fuse_dfs -oserver=hadoop-name -oport=9000 /mnt/hadoop -oallow_other -ordbufffer=131072

Adding a data node

1. Make sure that /scratch, exists, and has plenty of space.
• If there is not a large amount of space in /scratch, edit hadoop-site.xml to reflect where the Hadoop data should go.
  
2. Ping hadoop-name; it should be resolvable.  Add it to /etc/hosts if it is not already there.
• Perform "hadoop ls /"; this command should succeed.
  
3. View http://dcache-head.unl.edu:8088; your new node should be listed.
4. Follow the post-install instructions below
   

Adding a gridftp node

• Copy $VDT_LOCATION/post-install/*.conf to /etc/grid-security
• Edit /etc/grid-security/prima-authz.conf so imsContact points to osg-test3.unl.edu
• Link /etc/grid-security/certificates to /opt/osg/osg-100/globus/TRUSTED_CA
• There are several nodes with memory problems, and the gridftp server uses lots of memory.
• Make sure that /proc/sys/vm/overcommit_memory is 0 and NOT 2
  
• Make sure that "java -version" doesn't return any memory errors
• The recommended amount of memory per gridftp server is 8GB; newer kernel versions also help significantly with network and memory management.
• Check the version of the GridFTP_HDFS package.
• If the installed version is 0.0.2, do the following:
• "svn up" (install the "subversion" rpm via yum if necessary)
• source the OSG environment
• type "make"
• Contact Brian if the make fails
• Add the gridftp server to the tests contained in /opt/kickstart/hadoop_gftp.sh.
  
• Add the GridFTP server to /opt/kickstart/hadoop_gftp.sh

Adding a SRM server

We currently use the BeStMan SRM server.  Source code can be downloaded here: http://datagrid.lbl.gov/bestman/.  The current version is 2.2.1.1-r6.  We do not use the VDT packaging because we require that version.  To install and configure:

• Install Hadoop-Nebraska on the server.  This will bring in Java and Hadoop.
• Source the environment
• Java 1.6 is required for BestMan
  
• Configure and mount Hadoop via FUSE at /mnt/hadoop.
• Unpack the Bestman code into /opt/osg/bestman
• cd to /opt/osg/bestman/setup
• Run the configure script with the following options:
• --enable-gums
• --enable-gateway-mode
• --with-gums-url=https://osg-test3.unl.edu:8443/gums/services/GUMSAuthorizationServicePort
• Edit conf/bestman.rc
  

  staticTokenList=DEFAULT[desc:DEFAULT][10000]
  supportedProtocolList=gsiftp://dcache-s10.unl.edu:5000;gsiftp://dcache06.unl.edu:5000;gsiftp://dcache-s08.unl.edu:8000;gsiftp://dcache-s01.unl.edu:5000;gsiftp://dcache-s10.unl.edu;gsiftp://dcache-s07.unl.edu:5000;gsiftp://dcache-s08.unl.edu:5000;gsiftp://dcache-s05.unl.edu:5000;gsiftp://dcache-s10.unl.edu:5000;gsiftp://dcache-s08.unl.edu:5000
  

  You may want to examine the bestman.rc on dcache07 or srm to see if there are any other differences.
  
• Start the server directly:
  

  /opt/bestman/sbin/bestman.server

  Test a few SRM operations (srm-ping, srmcp) from another server to make sure the deploy works.
• Start the BestMan service:
  

  /opt/bestman/sbin/SXXbestman start

  Again, test a few SRM operations
• Add the SRM server to the test commands in /opt/kickstart/hadoop_gftp.sh.