nodes

The nodes command provides users with information about the farm cluster. The short listing displays the status of each node, the process types associated with the node, and ids of job sections that are running on the node. The format of the command is:

  fbs nodes [-d][-l][-t '<class> ...'][<node> ...]

The following is an example of the output:

> fbs nodes 
  HOST               STATUS CLASS       PROCESSES
 fnpc112.fnal.gov    up     IO          ['step1.1547.1']
 fnpc5.fnal.gov      down   Worker      []

The number in parenthesis after the job number indicates the number of user processes associated with that job that are running on that node.

The CLASS column shows the node class.

The PROCESSES column shows what processes are currently running on that node. The processes are shown in the form:

  sectionid.logical_pid

The nodes also has a long format that displays load averages and the reason that a node is down if applicable. The following two commands illustrate this output:

[farms@fnpc112 tests]> fbs nodes -l
--------------------------------------------------------
HOST: yogi               CLASS: Worker           STATUS: up        
RESOURCES:  {'disk': (0, 18), 'cpu': (90, 300), 'blue': (0, 0)}
PROCESSES:  ['39.Start.1']

The resource values are represented by a pair of values (used,capacity). In the example above, the resource cpu has a total capacity of 300, 90 of which are allocated. The -d option displays only nodes which are "down".

A node may be in one of four states:
up - The node is up and running, jobs are allowed to be dispatched to this node;
down - FBSNG detected a node failure and disabled the node until the failure condition is cleared. Status of batch processes running on the node is unknown.
held - This indicates that jobs are not allowed to be dispatched to this node.
released - This indicates that jobs are allowed to be dispatched to this node. The -c <class>... flag causes the command to only print information about nodes that belong to a particular class.