RSV Reporting for Hadoop
A list of changes we had to make in order for RSV to work with Hadoop
RSV Reporting Changes
All changes on this page are applied to the file$VDT_LOCATION/osg-rsv/bin/probes/OSG_RSV_Probe_Base.pm
Port Changes
RSV hardcodes the port number for SRM, but we have SRM running for Hadoop on 8446, not 8443.Change all instance of the string "8443" to "8446"
Timestamp changes
RSV uses timestamps of the form "12:12:12"; the ":" character is not supported under Hadoop. Hence, we changed the Get_Timestamp function to the following:sub Get_Timestamp {
my %h_tmp = %{$_[0]};
my $type = $_[1]; ## UTC for UTC, local for local time
my $system_date_cmd_out = undef;
if ($type eq "LOCAL") { ## Double escaping required to pass "\ "
$system_date_cmd_out = `$h_tmp{'systemDateCmd'} +%s`;
}
else { $system_date_cmd_out = `$h_tmp{'systemDateCmd'} -u +%s`; }
chomp ($system_date_cmd_out);
return \$system_date_cmd_out;
}Gratia Changes
We had to change Gratia.pm to support specifying timestamps as the time for the probe.In the file
$VDT_LOCATION/gratia/probe/metric/Metric.pyChange the function "Timestamp" to have the following contents:
def Timestamp(self,value):
" The time the metric was gathered "
" Expressed in number of second since epoch or a string formated using the format xsd:dateTime. "
try:
value = eval(value, {}, {})
except:
pass
if type(value)==str : realvalue = value
else : realvalue = TimeToString(time.gmtime(value))
# self.AppendToList(self.RecordData, "Timestamp", self.Type(timetype)+self.Description(description), realvalue)
self.AppendToList(self.RecordData, "Timestamp", "", realvalue)
Condor Job Submission
We changed the submit file in$VDT_LOCATION/osg-rsv/submissions/probes/srm.unl.edu__srmcp-srm-probe\@org.osg.srm.srmcp-readwrite.subThe Arguments line now reads:
Arguments = /opt-head/osg/osg-100/osg-rsv/bin/probes/srmcp-srm-probe -m org.osg.srm.srmcp-readwrite --verbose --srm-destination-dir /mnt/hadoop/user/uscms01/rsvtest/ --ggs --gsl /opt-head/osg/osg-100/osg-rsv/output/gratia --python-loc /usr/bin/python --uri srm.unl.edu --proxy /tmp/x509up_u1343 --srm-webservice-path=srm/v2/serverThis probably can be done directly from the RSV configuration in config.ini.
After changing the RSV submit file, you need to restart the RSV service:
/etc/init.d/osg-rsv restartas the user who runs RSV.