Installing FBSNG

FBSNG is distributed as a Fermilab Unix Environment (FUE) product. FUE simplifies the installation of FBSNG. However, FBSNG is fully capable to operate in non-FUE environemt. This document covers installation procedures for both FUE and non-FUE initial FBSNG installation. Refer to FBSNG Release Notes document if you are upgrading FBSNG to a newer version.

Requirements

FBSNG requires the following products to be installed:

Python 1.5.2. Versions 1.6 and higher are not currently supported;
FCSLIB package available from Fermitools must be installed. FCSLIB installation procedure is described below;
FBSNG GUI requires Python to be built with Tcl/Tk support module Tkinter, and Tcl/Tk must be installed. However, FBSNG GUI functionality is available from command line interface;
Node resource utilization monitoring tools sysmon and xsysmon included in FBSNG distribution package require rstatd to be installed and running on all nodes to be monitored. Note that FBSNG itself does not depend on sysmon/xsysmon or rstatd;

FUE Installation

Te following is set of steps the administrator has to perform in order to install FBSNG and products it depends on a FUE farm.

1. Using UPD/UPS install and declare FCSLIB product. It is FUE-compliant product distributed through KITS. It is portable across different supported UNIX versions, so it should be declared for NULL UPS flavor. FCSLIB comes with fcslib.table UPS action table file. It must be specified in "ups declare" command:

ups declare -0 -c -m fcslib.table ... fcslib <version>

2. Using UPS/UPD install and declare fbsng product. This will unwind the product into a directory referred to as $FBSNG_DIR. Currently, fbsng is portable across different known versions of Linux and IRIX, so only OS flavor, not version has to be included in UPS declaration.

FBSNG comes with UPS action table file fbsng.table, which must be mentioned in "ups declare" command.

ups declare -1 -m fbsng.table ... fbsng <version>

ups declare -f Linux -m fbsng.table ... fbsng <version>

ups declare -f IRIX -m fbsng.table ... fbsng <version>

3. Create a directory, preferrably in NFS-shared area, for FBSNG configuration files, databases and utility scripts. This directory should be shared by nodes running different flavors of UNIX. This directory will be referred to as <FBSNG_ROOT> or $FBSNG_ROOT.

4. Using UPS, "tailor" the product:

ups tailor -O <FBSNG_ROOT> fbsng <version>

ups tailor -O <FBSNG_ROOT> -c fbsng

This step must be performed from the same account as used on step 1.

5. Copy directory tree from <FBSNG_DIR>/templates to <FBSNG_ROOT> using:

cp -r <FBSNG_DIR>/templates/* <FBSNG_ROOT>

This should create directories:

<FBSNG_ROOT>/cfg - for configuration

/bin - for utility scripts

/history - for historical DB

/logs - for internal FBSNG logs

/slog - for section log files

/archive - for compressed log archives

/jdb - job database

6. Configure FBSNG (see FBSNG Configuration section below). Create FBSNG configuration files fbs.cfg and farm.cfg in <FBSNG_ROOT>/cfg. You can use fbs.cfg.example file in the same directory as an example. Note that interactive FBSNG configuration tools can not be used until BMGR process is running. Therefore, it is recommended to either

· Create initial configuration editing farm.cfg file with a text editor, or

· Rename farm.cfg.template file into farm.cfg, start BMGR (see Starting FBSNG section below), and then use FBSNG Configuration Utility or FBSNG GUI to create desired configuration.

7. Follow instructions in scripts in <FBSNG_ROOT>/bin to adjust them

to your installation, in particular, location of setups.(sh|csh) scripts.

Non-FUE Installation

The following procedure should be used to initially install FBSNG on a non-FUE farm.

1. Install FCSLIB product:

Download fcslib_<version>_<flavor>.tar file.

Unwind the tar file into, preferably, NFS-shared directory. This

directory will be referred to as <FCSLIB_DIR>. FCSLIB contains

some Python modules used by FBSNG. It is portable across different

flavors of UNIX, and therefore <FCSLIB_DIR> can be shared by

all nodes of the farm.

2. Designate a directory for FBSNG configuration files, databases and utility scripts. This directory should be shared by nodes running different flavors of UNIX, therefore, it is recommended to have it in NFS-shared space. This directory will be referred to as <FBSNG_ROOT> or $FBSNG_ROOT.

3. Designate a directory for FBSNG scripts, executables and libraries. This directory will be referred to as <FBSNG_DIR> or $FBSNG_DIR. Each different flavor of UNIX (not each separate version of the same OS) must have its own <FBSNG_DIR>.

4. Download and unwind fbsng_<version>_<flavor>.tar file(s) into

corresponding <FBSNG_DIR>s. This will create:

<FBSNG_DIR>/bin - with FBSNG binaries and Python sources

<FBSNG_DIR>/lib - with FBSNG API and miscellaneous modules

<FBSNG_DIR>/templates - with configuration and script templates

5. Copy directory tree from <FBSNG_DIR>/templates to <FBSNG_ROOT> using:

cp -r <FBSNG_DIR>/templates/* <FBSNG_ROOT>

This should create directories:

<FBSNG_ROOT>/cfg - for configuration

/bin - for utility scripts

/history - for historical DB

/log - for internal FBSNG logs

/slog - for section log files

/archive - for compressed log archives

/jdb - for job database

· Create initial configuration editing farm.cfg file with a text editor, or

· Rename farm.cfg.template file into farm.cfg, start BMGR (see Starting FBSNG section below), and then use FBSNG Configuration Utility or FBSNG GUI to create desired configuration.

7. Follow instructions in scripts in <FBSNG_ROOT>/bin to adjust them to your installation. In particular, comment out lines related to FUE

environment, comment in lines for non-FUE environment, and fill-in

actual <FBSNG_DIR> and <FCSLIB_DIR> locations.

fbs_env.(sh|csh) will be 'sourced' by users in order to use FBSNG. Sourcing

fbs_env script will add front-end 'fbs' command and FBSNG API to

user's PATH and PYTHONPATH respectively. Make sure that <FCSLIB_DIR>/lib

is included into PYTHONPATH.

FBSNG Configuration

FBSNG has two configuration files, farm.cfg and fbs.cfg, located in $FBSNG_ROOT/cfg directory.

File fbs.cfg contains configuration parameters for FBSNG components such as TCP/IP addresses, directory locations, time-outs, security-related parameters, etc. This file should be edited by a text editor. For any modifications of this file to take effect, corresponding FBSNG component or components must be re-started.

File farm.cfg contains description of the farm in terms of resources, process types and scheduling parameters. FBSNG comes with FBSNG Configuration Utility. FBSNG monitor includes GUI version of the configuration utility. The configuration utility allows the administrator to perform most of configuration operations without editing farm.cfg file. However, in an emergency situations, and when the configuration utility does not provide necessary functionality, manual editing of farm.cfg file may be required. In this case, BMGR daemon must be restarted after the editing in order for the modifications to take effect.

Manual editing of the farm.cfg file is required when the administrator needs to remove FBSNG objects like queues, process types, node classes, etc. except for removing a node from a node class. This later function can be performed using the FBSNG Configuration Utility or GUI. Certain rules that must be followed when manually editing farm.cfg file:

A queue should not be removed until it is empty, i.e. does not have any sections in any state in it. In order to “drain” a queue out, one can lock the queue and wait for all sections to start and complete, or kill all sections in the queue. One can use “fbs queues” or “fbs ls –q <queue>” commands to see whether the queue is empty or not.

If a non-empty queue is removed from the configuration, BMGR process will not re-start until the queue is returned back into the configuration.

A process type can be removed only if there are no sections of this type in any state. If necessary, those sections should be killed, or the administrator has to wait for all of them to exit and be removed from FBSNG memory. One can use “fbs ls” command to see what sections currently exist.

If a process type is removed from FBSNG configuration, BMGR process will not re-start until the process type is returned back into the configuration.

A node can be removed from the configuration only if no batch processes are running there. A node can be put on hold using “fbs hold node” command in order to block FBSNG from starting new processes on the node. One can use “fbs nodes –l <node>” command to see what processes are running on the node and the node status.

If a node with batch processes running is removed, the processes will continue to run, but FBSNG will consider them terminated as if the node was shut down.

A node class can be removed only if no batch processes are running on any of the nodes of the class.

A resource or resource pool can be removed from the configuration only if there are no running processes or sections in any state that allocate or require the resource or pool. Use “fbs resources” command to see what resources are currently allocated. Unfortunately, there is no easy way to find out what pending sections will require what resources. One can issue “fbs status” command for each pending job or section to find out what resources it requires.

If a resource or pool is removed, all pending sections that require the resource or pool will remain pending indefinitely.

Both configuration files are described below.

Daemons Configuration File fbs.cfg

FBSNG daemons configuration is specified in $FBSNG_ROOT/cfg/fbs.cfg file. The file has the structure described above in Format of FBSNG Configuration Files section. This file must be located in $FBSNG_ROOT/cfg directory. The following is description of each set of the fbs.cfg file and set parameters.

See Appendix B for an example of fbs.cfg file.

Set bmgr

This set defines parameters for BMGR daemon. The following table describes meaning of each parameter and default values, if any:

Parameter	Value Type	Default	Meaning of Value
api_port	Integer	(required)	TCP port number for API client.
launcher_if_port	Integer	(required)	LauncherIF port where launchers connect to. Unless BMGR is running as root, this number should 1024 or greater.
host	String	(required)	Host name where BMGR is running
allow_submit	List of Strings	(allow submit from any node)	List of patterns, determines what computers job submission is allowed from. Each pattern can be in one of 3 forms: official node name or IP address(e.g. fnpc21.fnal.gov, 123.2.3.16) Pattern with star () at the end or beginning (e.g. 123.2.3., *.fnal.gov) Regular expression with either [] or () (e.g. fnpc[a-d].fnal.gov, (fnsfo\|fnsfh).fnal.gov)
deny_submit	List of Strings	(empty list)	List of patterns, determines what computers job submission is denied from. Patterns are in the same format as for “allow_submit” parameter.
admin_list	List of Strings	(any user is an admin)	List of user names or numeric user ids. This is list of users authorized to modify FBSNG configuration.
section_log_dir	String	(required)	Directory where section logs are stored
recovery_timeout	Integer	10	BMGR recovery time-out in seconds. During initial recovery interval, BMGR waits for launchers on farm nodes to connect and ignores requests from API clients such as User Interface commands and GUI.
log_ignore	String	""	Log messages not to suppressed ‘I’,’D’,’E’,’F’ or any combination of them. Each letter suppresses log messages of types Informational, Debug, Error and Fatal respectively.
job_retention_interval	Integer	600	Job retention interval in seconds. Jobs will be kept in memory for the specified time after completion.

FBSNG Security

Parameters “allow_submit” and “deny_submit” define what computers job submission is allowed from. These computers are referred to as “trusted computers”. Submission is allowed from the node if:

Its IP address or official IP node name matches any pattern from allow_submit list, or the list is empty

and

Neither its IP address nor its official IP node name matches any pattern on the deny_submit list, or the list is empty.

Together with allow_submit and deny_submit lists, the admin_list parameter defines list of “trusted users” or “administrators”. The user is an administrator if:

It is on one of “trusted” computers

and

Its UID or username is on the admin_list, or the admin_list is empty.

Depending on username, UID and the node the user comes from, he/she can perform the following actions:

Trusted node ?	Trusted user ?	Can modify configuration	Can submit jobs
Yes	Yes	Yes	Yes
Yes	No	No	Yes
No	Yes	No	No
No	No	No	No

Any user on any computer is allowed to obtain run-time and configuration information.

Set jobdb

This set controls where FBSNG keeps its job data database. Currently, it has the only parameter:

Parameter	Value Type	Default	Meaning of the value
root	String	(required)	Path to the directory where job database is located. The directory should not contain any other information than the database created by BMGR

Set logd

This set defined parameters for FBSNG logd daemon and supplemental archiving, compression and purging scripts.

Parameter	Value Type	Default	Meaning of the value
host	String	(required)	Log daemon host name
server_port	Int	(required)	Log daemon UDP port number
log_dir	String	(required)	Log directory
email_wait	Int	1800	Time interval between sending identical e-mail messages in seconds (default=30 minutes)
email_command	String	"Mail"	Command to be used to send e-mail
arch_dir	String	(required)	Log archive storage directory
days_to_zip	Int	1	Age of log files to compress in days
days_to_archive	Int	5	Age of files to archive in days
ignor_codes	String	""	Message codes to ignore. This is a combination of single-character message type codes. Any message with code present in this string will be discarded without storing in the log files.
admin_address	String	None	e-mail address for error reports

Set history

This set defines parameters for FBSNG job history database:

Parameter	Value Type	Default	Meaning of Value
archive_period	Integer	1	Interval between moving the data from history file to archive file, in days.
hist_dir	String	(required)	Directory where history file(s) are stored
archive_file	String	archive.log	Archived history file name
hist_file	String	hist.log	Current history file name
compress_period	Integer		Interval in days between compressing history information into .tar.Z files
days_to_store	Integer		Time in days to keep information in history database

Set global

This set has two parameters:

Parameter	Value Type	Default	Meaning of Value
domain	String	(empty)	Common suffix for farm node names
mail_command	String	/bin/mail	External command to be used by FBSNG components to send e-mail messages
Farm_name	String	(empty)	Name of the farm

Value of “domain” parameter should be common IP domain name for all farm nodes. If specified, farm node name used by FBSNG will be determined by truncating the specified domain name from the “official” (obtained by gethostname() system function) IP node name.

Set launcher

Set of type “launcher” has only one integer parameter “stat_port”. Its value is UDP port number used by API clients to obtain run-time information about running processes.

Farm Configuration File farm.cfg

This file contains definition of farm resources, farm nodes, their classes, process types, and queues. See FBSNG Resources and FBSNG Scheduler documents for more details on these concepts and relationships between them. This file must be located in $FBSNG_ROOT/cfg directory. The file protection mask must allow BMGR process to write into this file. See Appendix C for an example of farm.cfg file.

Normally, FBSNG Configuration Utility or GUI should be used to modify farm configuration defined in farm.cfg file. However, under certain circumstances it may be necessary to edit this file using a text editor. In such cases, BMGR daemon must be re-started in order for the modifications to take effect.

Resources Configuration

Farm resources are defined in set “resources” of farm.cfg. The set has two parameters:

Parameter	Value Type	Default	Meaning of Value
local	List of Strings	(no local resources)	List of local resource names, including node attributes.
global	Dictionary String:Integer	(no global resources)	Dictionary with resource names as keys and corresponding integer resource capacities as values

Parameter

Value
Type

Default

Meaning of Value

local

List of Strings

(no local resources)

List of local resource names, including node attributes.

global

Dictionary

String:Integer

(no global resources)

Dictionary with resource names as keys and corresponding integer resource capacities as values

Resource pools are defined in set “resource_pools”. Each pool is represented with single set parameter with the pool name as parameter name and list of underlying resource names as value.

Node Classes Configuration

Each node class is represented as a separate set of type “node_class” and the class name as set id. Each set has two parameters:

Parameter	Value Type	Default	Meaning of Value
resources	Dictionary String:Integer	(no local resources defined for the class)	Dictionary with local resource names as keys and optional integer capacities as values. If a value is not specified, corresponding resource will be considered as a node attribute.
local_disks	Dictionary String:String	(no local disks defined for the class)	This dictionary describes correspondence between resource names used to represent local scratch disks and actual locations of the scratch disk storage. For each key-value pair, the key must be the name of one of the local resources defined for the node class, and the value – path to corresponding directory.

Parameter

Value
Type

Default

Meaning of Value

resources

Dictionary

String:Integer

(no local resources defined for the class)

Dictionary with local resource names as keys and optional integer capacities as values. If a value is not specified, corresponding resource will be considered as a node attribute.

local_disks

Dictionary

String:String

(no local disks defined for the class)

This dictionary describes correspondence between resource names used to represent local scratch disks and actual locations of the scratch disk storage. For each key-value pair, the key must be the name of one of the local resources defined for the node class, and the value – path to corresponding directory.

Local scratch disk areas are specified as their top directories. If the specified top directory does not exist, it will be created. FBSNG assumes that the specified directory will not contain any file or directory other than those created by FBSNG as temporary working directories for batch processes, and therefore, can delete any file or directory found there at any time.

Farm Nodes Configuration

Set “node_list” defines the node class each node belongs to. The set consists of

lines, one line per node. Node name for a farm node is defined as its official IP node name with domain name (as specified in set “global” of fbs.cfg, if any) truncated.

Process Types Configuration

Each process type is represented as a separate set of type “proc_type” and the process type name as the set id. Each set has 3 patameters:

Parameter	Value Type	Default	Meaning of Value
proc_rsrc_defaults	Dictionary String:Integer	(no default resource requirements)	This dictionary defines default resource requirements for processes of this type. For each key-value pair, the key must be the name of one of a resource, and the value – integer amount of the resource required. For node attributes, the value should be omitted.
sect_rsrc_defaults	Dictionary String:Integer	(no default resource requirements)	This dictionary defines default resource requirements for sections of this type. For each key-value pair, the key must be the name of one of global resources, and the value – integer amount of the resource required.
Resource_quota	Dictionary String:Integer	(no local disks defined for the class)	This dictionary resource allocation quotas for the process type. For each key-value pair, the key must be the name of one of local or global resources, and the value – the allocation limit. If a resource is not mentioned in the dictionary, the process type can allocate any amount of the resource.

Scheduler Configuration

FBSNG Scheduler is configured by defining set of Scheduler queues and their parameters. See FBSNG Scheduler document for more details on its algorithm and configuration parameters.

In farm.cfg file, each queue is represented by one set of type “queue” with the following parameters:

Parameter	Value Type	Default	Meaning of Value
proc_type	String		Default process type for the queue
s_prio_max	Int	100	Maximal section priority
s_prio_gap	Int	1000	Section priority gap
q_prio_min	Int	0	Minimal queue priority (= initial queue priority)
q_prio_max	Int	100	Maximal queue priority
q_prio_inc	Int	1	Queue priority increment
q_prio_dec	Int	10	Queue priority decrement
q_prio_gap	Int	1000	Queue priority gap
cputime	Int	(unlimited)	Process CPU time limit
realtime	Int	(unlimited)	Process run time limit

FBSNG Configuration Utility

The utility can work in 2 modes, interactive and single command. In interactive mode the utility prompts for next command after completion of previous. In single-command mode, it executes only one command and exits. Interactive mode is invoked by "fbs config" command. Command syntax is:

<command> <arguments> ...

Single-command mode is involed by typing

fbs config <command> <arguments> ...

Both modes accept the same set of commands and arguments:

Creating Objects

The following commands are used to create FBSNG objects:

Creating a Queue

create queue <queue name> <def. process type> [<parameters>]

See “set queue” command description for list of optional parameters. If no parameters are specified, the queue is created with default set of parameters and the specified process type associated with it. Queues are always created in locked state. In order to unlock a queue, use “unlock queue” command.

Creating a Node Class

create nclass <node class name>

The command creates an empty (having no nodes associated to it) node class with no resources or local disks defined. Use "set nclass" command to configure local resources and local scratch disks for the node class. Use "add/remove node" commands to manipulate list of nodes of the class.

Creating a Process Type

create ptype <proc type name> [<parameters>]

See "set ptype" command description for accepted list of optional parameters. If no parameters are specified, the new process type will have no quotas or default resource requirements defined.

Creating a Resource Pool

create pool <pool name> <resource name> ...

The command creates new resource pool combining listed resources.

Create a Global Resource

create gr <global resource name> <capacity>

The command creates new global resource with specified capacity.

Create a Local Resource

create lr <local resource name>

The command creates new local resource.

Modifying Object Parameters

The following commands can be used to modify parameters of existing objects

Setting Queue Parameters

set queue <queue name> <parameter>[:<value>] ...

The command accepts the following parameters: QPGap, QPInc, QPDec, MaxQPrio, MinQPrio, MaxSPrio, SPGap, Prio, DefProcType, RealTimeLimit, CPUTimeLimit.

Values for parameters RealTimeLimit and CPUTimeLimit can be omitted together with separating colons. In this case corresponding limits will be removed.

Setting Node Class Parameters

set nclass <node class> rsrcs <resource>:<capacity> ...

“rsrcs” keyword indicates that local resource capacity for the class should be set as specified as by the following dictionary.

set nclass <node class> disks

<disk resource name>:<root directory> ...

“disks” keyword indicates that the following is the mapping between resource names and associated local scratch disk locations. Local disk resources must already be defined as local resources for the node class.

Setting Process Type Parameters

set ptype <proc type name> <parameter> <value>

Parameters and their values are:

srdef <resource>:[<amount>] ... – default section resource requirements
prdef <resource>:[<amount>] ... – default process resource requirements
rquota <resource>:<amount> ... – resource allocation quota
maxpinc <value> - maximum allowed priority increment

Setting Global Resource Capacity

set gr <global rsrc name> <capacity>

Setting Composition of a Resource Pool

set pool <pool name> <resource name> ...

Node Manipulation

The following commands should be used to add or remove nodes to or from an existing node class. Only nodes without any processes running can be removed from a node class. When a node is added to a node class, its status becomes "held".

add node <node class name> <node name> ...

remove node <node class name> <node name> ...

Queue Locking and Unlocking

An administrator can lock a queue to disable submission of new sections to it in order to "drain" the queue down. When a new queue is created, it must be unlocked before it can be used. The commands are:

lock queue (all|<queue name>)

unlock queue (all|<queue name>)

If the keyword “all” is used instead of a queue name, the operation will be applied to existing queues.

Show Commands

In order to see information about one or more objects of certain type, one can use "show" command:

show (queue|nclass|pool|ptype|rsrc) [<name> ...]

If no object name is specified, all objects of the type will be displayed. "rsrc" keyword is used to show information about all types of resources.

Online help

If a user uses incomplete command, usage information is printed. Also a user can issue help command for a brief command summary.

Maintenance Tasks

Starting FBSNG

FBSNG has the following permanently running processes (daemons):

bmgr - runs on preferably most reliable node of the farm. This daemon can run under non-privileged account. It must have write access to log, slog, archive, history, cfg subdirectories under <FBSNG_ROOT> and files in these directories, and have at least read access to the rest of <FBSNG_ROOT> subtree.

launcher - an instance of this process must run on each worker node of the farm. Launcher must run as root and must have read access to files in <FBSNG_ROOT>/cfg.

logd - one instance, runs on preferably most reliable node of the farm. This daemon can run under non-privileged account. It must have write access to <FBSNG_ROOT>/log subdirectory, and have at least read access to the rest of <FBSNG_ROOT> subtree. Logd is not required for FBSNG to work properly. It is used only to store FBSNG internal logs. Logd and bmgr do not have to run on the same node.

Although not necessary, it is recommended to set up unprivileged account for bmgr and logd and create <FBSNG_ROOT> directory in NFS-exported home area of such account.

In order to start each daemon, administrator should use start_bmgr.sh, start_logd.sh and start_lch.sh scripts found in <FBSNG_ROOT>/bin directory. Before they can be used, the scripts must be adjusted to the particular installation following instructions found in the script files. Administrator must 'su' to proper account before using the scripts. The scripts are designed to be called from system boot-time start-up scripts.

FBSNG distribution includes “sendcmd” script. This script, located in $FBSNG_ROOT/bin directory, can be used to perform certain operations on a number of nodes using “rsh” command. “sendcmd” script can be used to start launcers on farm nodes in the following way:

cd $FBSNG_ROOT/bin

./sendcmd fnpc 101 120 /home/farms/fbsng_root/bin/start_lch.sh

This command will run the start_lch.sh script on nodes fnpc101 through fnpc120.

In cases when using start_xxx.sh scripts is not acceptable due to site-specific policies or procedures, FBSNG Administrator can use lower level commands to start FBSNG components. The following commands run corresponding daemons:

fbs bmgr # run BMGR

fbs launcher # run Laucnher

fbs logd # run Logd

How to Tell Whether FBSNG is Running

First quick check is to use UNIX “ps” command. Here is what ps output should look like for BMGR:

fnpcb> ps axw | grep bmgr

17177 pts/12 S 0:01 sh –f /home/farms/fbsng_v1_1/bin/fbs_run.sh bmgr

20163 pts/12 S 0:00 sh /fnal/ups/prd/fbsng/v1_1/Linux/bin/fbsng bmgr

20165 pts/12 S 0:09 python /fnal/ups/prd/fbsng/v1_1/Linux/bin/bmgr.py

for launcher:

fnpcb> ps axw | grep launcher

13660 ? S 0:00 sh -f /home/farms/fbsng_v1_1/bin/fbs_run.sh launcher

20488 ? S 0:00 sh /fnal/ups/prd/fbsng/v1_1/Linux/bin/fbsng launcher

20509 ? S 28:50 python /fnal/ups/prd/fbsng/v1_1/Linux/bin/launcher.py

and for logd:

fnpcb> ps axw | grep logd

18921 pts/12 S 0:00 sh -f /home/farms/fbsng_v1_1/bin/fbs_run.sh logd

19007 pts/12 S 0:00 sh /fnal/ups/prd/fbsng/v1_1/Linux/bin/fbsng logd

19008 pts/12 S 0:21 python /fnal/ups/prd/fbsng/v1_1/Linux/bin/logd.py

Also, in order to see if FBSNG is working properly, administrator can use “fbs nodes” command. If the command prints list of the farm nodes, it means that important FBSNG components are configured properly, and BMGR process is running. Here is what normal “fbs nodes” output might look like:

fnpcb> fbs nodes

HOST STATUS CLASS PROCESSES

---- ------ ----- --------

fnpcb up IO []

fnpc104 down Worker []

fnpc105 down Worker []

fnpc106 down Worker []

fnpc107 down Worker []

fnpc101 down Worker []

fnpc102 down Worker []

fnpc103 down Worker []

fnpc108 down Worker []

It should be noted that after starting, BMGR process is supposed to ignore all request while it performs initial recovery functions. The recovery time interval is defined in fbs.cfg file.

“fbs nodes” command will print status of all farm nodes. “down” node status may indicates that either the node is down, or launcher process on the node can not connect to BMGR process, or the launcher is not running there.

Re-starting FBSNG Components

If started using corresponding start-up scripts, FBSNG components will be re-started automatically in a minute after they exit. Therefore, to re-start an FBSNG component, it is sufficient to kill corresponding process. <FBSNG_ROOT>/bin directory contains kill_lch.sh, kill_bmgr.sh and kill_logd.sh scripts. They can be used to kill launcher, BMGR and logd processes respectively. Of course, they must be used on the node where the process is running, and require necessary privileges to kill the process.

Shutting Down FBSNG Components

In order to shut down FBSNG component so that it does not re-start afterwards, shutdown_lch.sh, shutdown_bmgr.sh and shutdown_logd.sh scripts found in <FBSNG_ROOT>/bin directory should be used.

FBSNG Recovery Limitations

FBSNG is designed so that it can recover from most of planned or unexpected component failures. However, when planning re-start of FBSNG components, for example after modifying FBSNG configuration, the administrator should follow certain rules:

Launcher on a farm node should never be re-started while any batch processes run on the farm node. Launcher termination is interpreted as shutting down the node it runs on, and any processes running there are considered to be terminated. If it is necessary to shut down or re-start a launcher, the administrator should hold the node using “fbs hold node” command, either kill all jobs or job sections running on the node, or wait until all processes running on the node exit, and only then re-start or shut down the launcher process.

In order to prevent FBSNG database corruption, BMGR process must not be killed with SIGKILL (usually signal 9). It should be killed only with SIGINT signal. It may take several seconds for BMGR to exit after receiving the signal depending on what state it was when the signal was received. It is recommended to always use kill_bmgr.sh to re-start the BMGR process or shutdown_bmgr.sh to shut it down permanently. As long as it is done properly, it is safe to shut down or re-start BMGR process at any time. When BMGR process re-starts, it will recover run-time information about pending and running batch jobs.

Logd process can be killed virtually any time. If logd re-starts shortly, no log information will be lost. However, if it does not re-start soon enough, some old log information may be discarded.

Appendix C: Sample farm.cfg file

#--------------------------------------------------------------------------

#- File: farm.cfg

#- FBSNG farm configuration file.

#- IMPORTANT NOTICE:

#- =================

#- This file should be edited directly only if absolutely necessary.

#- Use "fbs config" or "fbs monitor" to modify farm configuration.

#- If editing is necessary, preserve header lines starting with "#-"

#- and modification history lines starting with "##".

#- farm.cfg set description:

#- -------------------------

#- %set resources

#- local = <local resource name> ...

#- global = <global resource name>:<integer capacity> ...

#- %set resource_pools

#- <pool name> = <resource name> ...

#- ...

#- %set node_class <class name>

#- resources = <local resource name>:<capacity> ... <attribute> ...

#- local_disks = <resource name>:<root directory> ...

#- %set node_list

#- <node name> = <node class name>

#- ...

#- %set proc_type <proc.type name>

#- proc_rsrc_defaults = <resource name>:<amount> ... <attribute> ...

#- sect_rsrc_defaults = <global resource name>:<amount> ...

#- resource_quota = <resource name>:<max. amount> ...

#- max_prio_inc = <integer max. allowed priority increment>

#- %set queue <queue name>

#- s_prio_gap = <section priority gap>

#- proc_type = <default process type> (required field)

#- q_prio_max = <max. queue priority>

#- q_prio_gap = <queue priority gap>

#- s_prio_max = <max. section priority>

#- q_prio_min = <min. queue priority>

#- q_prio_inc = <queue priority increment>

#- q_prio_dec = <queue priority decrement>

#---------------------------------------------------------------------------

## Fri Sep 15 11:38:33 2000: ivm: removed node fnpc111

## Fri Sep 15 11:38:33 2000: ivm: removed node fnpc112

%set node_list

fnpc104 = Worker

fnpc105 = Worker

fnpc106 = Worker

fnpc107 = Worker

fnpcb = IO

fnpc102 = Worker

fnpc103 = Worker

fnpc108 = Worker

fnpc101 = Worker

%set resources

local = IO cpu disk1 disk2

global = nfs_disk:30

%set proc_type IO

resource_quota = cpu:1000

proc_rsrc_defaults = IO cpu:50

%set proc_type Light

resource_quota = cpu:150

proc_rsrc_defaults = cpu:1

%set proc_type Worker

proc_rsrc_defaults = cpu:100

%set node_class IO

resources = disk2:20 IO cpu:200

local_disks = disk2:/tmp/fbs_scratch

%set node_class Worker

resources = cpu:100 disk1:10

local_disks = disk1:/tmp/fbs_scratch

%set resource_pools

disk = disk1 disk2

%set queue FastQ

q_prio_max = 100

s_prio_max = 100

q_prio_gap = 5

s_prio_gap = 1000

proc_type = Worker

q_prio_inc = 3

q_prio_min = 2

%set queue IOQ

s_prio_gap = 1000

proc_type = IO

q_prio_max = 1000

q_prio_gap = 1000

s_prio_max = 100

%set queue LongQ

s_prio_gap = 10

proc_type = Worker

q_prio_max = 100

q_prio_gap = 10

s_prio_max = 100

Version 1.1

Table of Contents

Set logd

Set history

Set global

FBSNG Configuration Utility

Creating a Queue

Creating a Node Class

Creating a Process Type

Creating a Resource Pool

Create a Global Resource

Create a Local Resource

Setting Queue Parameters

Setting Node Class Parameters

Setting Process Type Parameters

Setting Global Resource Capacity

Setting Composition of a Resource Pool

Starting FBSNG