dCache Space Monitoring
This page describes a simple method for monitoring the amount of free space in dCache
The best way to get free space information out of dCache is to utilize the "srm-storage-element-info" tool. However, the problem with this is that it requires a valid user proxy in order to get the information.
However, at most CMS sites, there is a very long lasting user proxy on the Phedex server, because that also requires a valid proxy. In this recipe, assume $X509_USER_PROXY points to a proxy which will be automatically renewed before it expires (again, the Phedex admin should have already set this up).
Test out the storage info tool:
For available space,
In order to get storage information about each pool cell, download the above script to a computer with the command line tool links installed. This is installed by default on most Redhat-based distros, and has been tested on ScientificLinux 3.
The script requires you to have the dCache web interface running and accessible. If you don't know what I'm talking about, it should look like this. In our case, the host is thpc-1.unl.edu and the port is 2288. When the script is invoked, it takes two parameters: the host:port of the dCache server and the host:port of the ApMon-enabled MonaLisa service. Test it out once, and view the results in your MonaLisa Java client. They should show up under cluster dCache.
However, at most CMS sites, there is a very long lasting user proxy on the Phedex server, because that also requires a valid proxy. In this recipe, assume $X509_USER_PROXY points to a proxy which will be automatically renewed before it expires (again, the Phedex admin should have already set this up).
Test out the storage info tool:
srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdlChange "thpc-1.unl.edu" to the appropriate server name. Stringing together a couple of command line tools, we can filter out the appropriate numbers:
For available space,
srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep availableSpace | tr '(' ' ' | awk '{print $3;}'
For total space,srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep total | tr '(' ' ' | awk '{print $3;}'
For used space,srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep used | tr '(' ' ' | awk '{print $3;}'We now take these three commands, and send the values to MonaLisa using mlmetric. Add the following to your Phedex user's crontab (using crontab -e):2,12,22,32,42,52 * * * * source /opt/osg/osg-030/setup.sh; /home/phedex/mlmetric.py -dest red.unl.edu:58884 -cluster "dCache" -node "Storage Info" -param "Available Space" -value `srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep availableSpace | tr '(' ' ' | awk '{print $3;}'`
3,13,23,33,43,53 * * * * source /opt/osg/osg-030/setup.sh; /home/phedex/mlmetric.py -dest red.unl.edu:58884 -cluster "dCache" -node "Storage Info" -param "Total Space" -value `srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep total | tr '(' ' ' | awk '{print $3;}'`
4,14,24,34,44,54 * * * * source /opt/osg/osg-030/setup.sh; /home/phedex/mlmetric.py -dest red.unl.edu:58884 -cluster "dCache" -node "Storage Info" -param "Used Space" -value `srm-storage-element-info -debug=false https://thpc-1.unl.edu:8443/srm/infoProvider1_0.wsdl 2>&1 | grep used | tr '(' ' ' | awk '{print $3;}'`
Again, change the lines to reflect your site's setup (you'll need to source a different script, mlmetric might be in a different place, send to a different destination, etc).Detailed dCache Usage
This section utilizes the dcache_usage.py script.In order to get storage information about each pool cell, download the above script to a computer with the command line tool links installed. This is installed by default on most Redhat-based distros, and has been tested on ScientificLinux 3.
The script requires you to have the dCache web interface running and accessible. If you don't know what I'm talking about, it should look like this. In our case, the host is thpc-1.unl.edu and the port is 2288. When the script is invoked, it takes two parameters: the host:port of the dCache server and the host:port of the ApMon-enabled MonaLisa service. Test it out once, and view the results in your MonaLisa Java client. They should show up under cluster dCache.
/home/brian/monitoring/dcache_usage.py --dest red.unl.edu:58884 --dcache thpc-1.unl.edu:2288Next, add the script to your crontab using crontab -e so the information gets gathered frequently. I have it updated once every 5 minutes:
0-59/5 * * * * /home/brian/monitoring/dcache_usage.py --dest red.unl.edu:58884 --dcache thpc-1.unl.edu:2288