Daily Operations

All of the admin operations must be done as root on hadoop-name, unless otherwise noted.

DFS filesystem check

hadoop fsck / -blocks

A successful check will end with these words:

The filesystem under path '/' is HEALTHY

A unsuccessful check will end with the following:

The filesystem under path '/' is CORRUPTED

Follow up with more diagnostic in this case.

FUSE on worker nodes check

If you see the message "Transport endpoint is not connected" on worker nodes, this means that the FUSE mount has died.  Simply log into the worker nodes, unmount the file system, and remount it:

umount /mnt/hadoop
fuse_dfs -oserver=hadoop-name -oport=9000 /mnt/hadoop -oallow_other -ordbufffer=131072
ls /mnt/hadoop

You may want to check for this condition every morning with "part 'ls /mnt/hadoop > /dev/null'" and look for the "Transport endpoint..." message.

DFS overall report

hadoop dfsadmin -report

A similar amount of information can be found from

• http://dcache-head.unl.edu:8088/dfshealth.jsp
  

Get safemode status

hadoop dfsadmin -safemode get

Leave or enter safemode

hadoop dfsadmin -safemode leave

hadoop dfsadmin -safemode enter

Decommissioning a data node

1. Add the data node entry to the file /scratch/hadoop-root/hosts_exclude (one host per line).
2. Perform "hadoop dfsadmin -refreshNodes"
3. Watch the logs of the namenode to make sure all the appropriate actions are taken!
4. The decommissioning will end in the logfiles with this message: "Decommission complete for node 172.16.1.55:50010"
   

Note: The namenode must see the transition from normal host to excluded host in order for it to realize a node is being decommissioned.  If you stop the namenode, add the file to the hosts_exclude, then start the namenode again, the namenode will have the following compliants in the log: "ProcessReport from unregisterted node: node055:50010".  This is because the namenode thinks it is being contacted by a node which was never in the system at all, not by a node which should be decommissioned.

Cleaning up a CORRUPT filesystem (as reported by fsck)

When the namenode is in safemode, one cannot make any edits to the file system.  First, run fsck and determine the extent of the damage.  If it is acceptable to delete or otherwise move aside the damaged files, turn off safemode, and move the file using the following command

hadoop fsck -move

This moves any files with problematic blocks into /lost+found.

Restoring from a checkpoint

Shut down the namenode.  There are two checkpoint images being kept at any time:

/scratch/hadoop/dfs/namesecondary/current/
/scratch/hadoop/dfs/namesecondary/previous.checkpoint/

Copy all the files in one of these directories into the following directory:

/scratch/hadoop/dfs/name/current/

Start the namenode again, and watch the logs for activities.

Miscellaneous Activities

Fixing "stuck" under-replicated files

We have observed a few occaisions where a few blocks stay under-replicated; this seems to correspond with blocks written during or around the time of namenode crashes (last time, it was due to a kernel panic unrelated to Hadoop).  Assume file X has a block with only Y replicas, but the target number is Z.  After an appropriate amount of time (block replicas usually occur within a few seconds; wait no more than 10 minutes), do the following:

1. Confirm the file is having the problem using "hadoop fsck X"
   
2. Set the replicas of file X to the current number of replicas, Y: "hadoop fs -setrep Y X"
3. Use fsck to make sure the file now shows up fine; "hadoop fsck X"
4. Set the number of replicas back up to Z: "hadoop fs -setrep Z X"
5. Watch the file using fsck and make sure the replicas get created: "hadoop fsck X"

This has happened once, and we'll be on the lookout for this to reoccur.  If this occurs for more than 5 files, this can be done in bulk using xargs (use grep and tr to create a file containing all the filenames of the problematic files from "hadoop fsck"; use "xargs -t -i 'hadoop fs -setrep Y {}'", etc, to do the hadoop commands.

Here's the script using xargs:

hadoop fsck / | awk '{print $1}' | grep user | tr -d ':' | sort | uniq > /tmp/stuck_replicas
cat /tmp/stuck_replicas | xargs -t -i hadoop fs -setrep 2 {}
hadoop fsck / # Make sure everything is happy
cat /tmp/stuck_replicas | xargs -t -i hadoop fs -setrep 3 {}
hadoop fsck / # Watch and see if everything becomes happy

Port forwarding for the hadoop web interface

This only needs to be done once:

/sbin/iptables -t nat -A PREROUTING -p tcp --dport 8088 -i eth0 -j DNAT --to-destination 172.16.100.8:50070
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 8089 -i eth0 -j DNAT --to-destination 172.16.1.9:50030

IMPORTANT: make sure to not kill the normal nat behavior on dcache-head. In fact, just use the /root/restart_NAT.sh script which includes the above lines.

Starting and Stopping

Controlling via init scripts

Every node which is installed via VDT should have init scripts; if they aren't there, copy them from a node which has them.

• (If installed through VDT): vdt-control --on
• If not installed via VDT, use chkconfig to add the scripts
• To start/stop datanode: /etc/init.d/hadoop_datanode [start|stop]
• To start/stop namenode: /etc/init.d/hadoop_nadenode [start|stop]
• To start/stop FUSE mount: /etc/init.d/hadoop_fuse [start|stop]
• To start/stop the GridFTP xinetd install:
• vdt-control --enable gridftp-hdfs
• vdt-control [--on|--off]

Manually starting a Hadoop daemon

We recommend using the init scripts to start and stop daemons.  However, if you must do this manually:

cd $HADOOP_HOME/bin
source ./hadoop-config.sh
./hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode

The valid actions are start or stop; the valid daemons are datanode or namenode.

Manually remounting FUSE

To unmount,

umount /mnt/hadoop

To mount, after sourcing the Hadoop environment,

fuse_dfs -oserver=hadoop-name -oport=9000 /mnt/hadoop -oallow_other -ordbufffer=131072