The dCache PFM is a script which takes in information about the state of replicas in the dCache system and, for each file considered, issues a replication or deletion command based upon some policy.  It takes the following as axioms:

• Never keep centralized, stateful information about a decentralized system!
  
• If a pool doesn't respond to commands, it is not present, and its replicas don't count.
  
• Only cancel a transfer if it cannot maintain some base rate; no timeouts.
  
• Pool P2P queue limits are set appropriately by the system administrator.
• Pools are the only place where replica information is correct.
• Pools are named using the following convention: <hostname>_<number>.  There are a few hostname related policies:
• Two replicas on the same host are only counted as one replica.
• File X will not be replicated to a pool Y on host Z if host Z already has file X on it.
  
• Source pools are ranked by lowest CPU cost.
• Destination pools are ranked by most freespace.
• Note that we currently don't respect routing information!  The whole system is viewed as a flat set of pools.  We do wish to respect routing in the future.
• Policies should be set by file name, not by physical pool they are on.  Users don't care about the site setup, they care about their file's policy.

To check out the dcache_pfm.py script, use subversion (try yum install svn if the svn binary isn't present):

svn co svn://t2.unl.edu/brian/dCacheNebraska/scripts/ dCacheNebraska

This will copy a bunch of files to the dCacheNebraska/ directory.  Next, cd to that directory and examine pfm_config.  This config file controls the PFM policy.  Options include:

• base: The base directory.  Only examine / change files inside this directory.
• ignore:  A comma-separated list of regular expressions.  If a file matches any of these, it will be ignored by the PFM.
• repmin: The default minimum number of replicas.
• repmax: The default maximum number of replicas.
• replicate_X (where X is an integer): A comma-separated list of regular expressions.  If a file matches any of these, it will eventually be replicated X number of times.
• max_replications: The maximum number of replications which will occur per run.  Use this as a safeguard against pools which commonly don't respond, or just turn it off.
• logging_config: A configuration file for the logging.  Alter the file if you have a fancy log setup or syslog.
• stopfile: If the file corresponding to this value exists, the script will refuse to run.  This way, system administrators can easily "turn off" dcache_pfm.py if run via cron script.
• blacklist: Comma-separated list of pools.  Replicas of these pools are not counted to a file's total and the pool may not recieve any new P2P copies.  However, this pool may be used as a source for P2P transfers.
• timeout: Timeout to use when communicating with the pools.
  

These directives are also documented within the default pfm_config configuration file.

To run the script, simply start it from the directory containing pfm_config (or specify some other config file from the command line):

./dcache_pfm.py

This will log changes to the command line and pfm_logfile.  If you do not want any command line output, add the -q flag.  If you have multiple config files, you may specify them by adding the following flag:

-config config1,config2,config3

Note that the file names are comma-separated and there are no spaces between entries.

WARNINGS

The dCache PFM does a few things which I'm not entirely comfortable with, but I have come to accept as necessary.  (Of course, none of these warning have been known to adversely effect the site or cause data loss, but you still will want to read them).

• I specifically want all of the files on my site to be in the "precious" state.  It appears that after a P2P transfer is completed, the resulting file is left in the "cached" state, meaning the pool is free to delete it as needed.  So, after the file transfer is finished, an additional operation needs to be done; this goes against my "state-free" goals for dCache PFM.  Hence, at the beginning of the run, when the pool replicas are being counted, I move every cached file into the "precious" state.  I have observed no ill effects on my system by doing this.  However, THIS MEANS THAT DCACHE PFM SHOULD CURRENTLY NOT BE USED WITH SITES HAVING TAPE BACKENDS.  I plan for future versions to have a way to disable this action.
• Question for dCache experts:  if I set P2P file mode to precious, does this problem go away?
  
• When in-progress P2P transfers are being inventoried, I cancel any active servers with no active client and vice versa.  This state can occur when one of the two pools involved in the P2P copy has some sort of problem, but does not notify the other side.  Of course, this does not adversely effect PFM operation; the PFM will simply retry the transfer.  However, it could possibly interfere with other P2P activities.
• This is done because sometimes the P2P server or client queue might fill up completely with dead transfers, causing a backlog of copy requests.
• Again, future versions of the program will have a way to disable this.
• Hardlinks should be handled very carefully.  Make sure that any hardlinked file only has one policy active at a time, and any other links to that file are in the ignore list!
  

Explanation

The algorithm will make all correct decisions.  Like any good mathematician, I have made this true by taking a sufficiently long set of axioms.  However, it's worth reading how the PFM works in order to understand it's actions.  An explanation is here.

FAQ

• If there is one unique copy of file X, and I have "repmin" set to 2, how many new replicas will be created? 1 or 2?
• The repmin setting is the minimum number of replicas desired to be in the system.  So, one additional replicas will be created.
• Executing dcache_pfm.py from the scripts/ directory works, but it does not work from any other directory.  What's up?
• dcache_pfm.py looks for the default config file, pfm_config, in the current working directory.  The pfm_config config file also has several relative paths.  You will need to specify the absolute path of pfm_config to the -config flag and, within your config file, have absolute paths for the following:
• config_file.  Note that the default for config_file is DBParam; dcache_pfm.py looks for it first in $PWD, then $HOME, then /etc.
  
• logging_config.  This file is highly flexible, but most people will want to use the default one in the scripts/ directory.
• Too many of my pools go on the blacklist because of timeouts!
• Try adjusting the timeout parameter in your config file.  This value there is set in seconds.
• Can I use the dcache_pfm.py to replicate all unique files on a certain pool, in order to get it ready to be taken offline?
• Sure!  However, this is awkward to do.  We recommend using the retire_pool.py script instead; this is designed to do exactly what you are asking.
• I see messages like this one:
  

  10/18/07 12:20:32. PFM:INFO. Replicating 0001000000000000022EECE0; current replicas 0; target replicas 2; /pnfs/ultralight.org/data/store/mc/2007/6/15/Spring07-BB2MuMu-1947/0020/3807B16D-BB3F-DC11-B06C-0015C5E5B2EC.root

  How is it replicating a file with no replicas!
• If a replica exists on a pool on the blacklist (and that pool is responding), that replica does not count toward the total replica count.  However, if responding, that pool may be used as a source for the creation of a new replica