Personal tools
You are here: Home Documentation Hadoop RSV Reporting for Hadoop
Document Actions

RSV Reporting for Hadoop

by admin last modified 2008-10-13 04:31

A list of changes we had to make in order for RSV to work with Hadoop

RSV Reporting Changes

All changes on this page are applied to the file
$VDT_LOCATION/osg-rsv/bin/probes/OSG_RSV_Probe_Base.pm


Port Changes

RSV hardcodes the port number for SRM, but we have SRM running for Hadoop on 8446, not 8443.
Change all instance of the string "8443" to "8446"

Timestamp changes

RSV uses timestamps of the form "12:12:12"; the ":" character is not supported under Hadoop.  Hence, we changed the Get_Timestamp function to the following:
sub Get_Timestamp {

    my %h_tmp        = %{$_[0]};
    my $type         = $_[1];  ## UTC for UTC, local for local time
    my $system_date_cmd_out    = undef;

    if ($type eq "LOCAL") {  ## Double escaping required to pass "\ "
        $system_date_cmd_out = `$h_tmp{'systemDateCmd'} +%s`;
    }
    else { $system_date_cmd_out = `$h_tmp{'systemDateCmd'} -u +%s`; }

    chomp ($system_date_cmd_out);
    return \$system_date_cmd_out;
}

Gratia Changes

We had to change Gratia.pm to support specifying timestamps as the time for the probe.
In the file
$VDT_LOCATION/gratia/probe/metric/Metric.py
Change the function "Timestamp" to have the following contents:
    def Timestamp(self,value):
" The time the metric was gathered "
" Expressed in number of second since epoch or a string formated using the format xsd:dateTime. "
try:
value = eval(value, {}, {})
except:
pass
if type(value)==str : realvalue = value
else : realvalue = TimeToString(time.gmtime(value))
# self.AppendToList(self.RecordData, "Timestamp", self.Type(timetype)+self.Description(description), realvalue)
self.AppendToList(self.RecordData, "Timestamp", "", realvalue)

Condor Job Submission

We changed the submit file in
$VDT_LOCATION/osg-rsv/submissions/probes/srm.unl.edu__srmcp-srm-probe\@org.osg.srm.srmcp-readwrite.sub
The Arguments line now reads:
Arguments = /opt-head/osg/osg-100/osg-rsv/bin/probes/srmcp-srm-probe  -m org.osg.srm.srmcp-readwrite --verbose  --srm-destination-dir /mnt/hadoop/user/uscms01/rsvtest/ --ggs --gsl /opt-head/osg/osg-100/osg-rsv/output/gratia --python-loc /usr/bin/python --uri srm.unl.edu --proxy /tmp/x509up_u1343 --srm-webservice-path=srm/v2/server
This probably can be done directly from the RSV configuration in config.ini.
After changing the RSV submit file, you need to restart the RSV service:
/etc/init.d/osg-rsv restart
as the user who runs RSV.



Powered by Plone, the Open Source Content Management System