The job submission system functions as a large batch system with commands to submit a job, check its status, and to retrieve any output. The two main differences between a standard batch system (such as PBS, LSF etc.) and the EDG job submission service are that the job submission service adds a high-level layer permitting uniform access to the resources at different sites and matches the available resources to the requirements for the job automatically.
The summary below is a brief overview of the main features of the job submission system; Figure 4.1 gives a brief pictorial overview. For further information, see the Workload Management Documentationhttp://www.infn.it/workload-grid/documents.html. See Chapter 5 for a description of the job execution environment.
The relevant commands are
edg-job-submit --vo <VO> <job.jdl> edg-job-status <jobId> edg-job-get-output <jobId> edg-job-cancel <jobId>for submitting a job, querying its status, retrieving the output, and cancelling a job, respectively.
For input, the submit command takes a job description file job.jdl (see below). The submission returns a job identifier, jobId, of the form:
https://lxshare0403.cern.ch:9000/2ODSom6oFPlH0sLGvTQX4Qwhich can then be used to determine the status of the job and eventually retrieve its output.
The other commands take job identifiers as input (jobId) and perform the corresponding action on the listed jobs. The edg-job-get-output retrieves the output from the job from the resource broker machine where it is cached; the output can only be retrieved once a job has reached the ``Done'' status.
The key to the job submission and resource matching process is the job description file. This file describes the necessary inputs, generated outputs, and resource requirements of a job using the Job Description Language (JDL).
A typical example of a job description file:
Executable = "HelloScript.sh"; Arguments = "hello 200"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"HelloScript.sh"}; OutputSandbox = {"std.out","std.err"}; Requirements = other.GlueCEInfoTotalCPUs > 4; Rank = other.GlueCEStateFreeCPUs;shows that a script is passed as input to a job, which then produces standard output and error files which will be transported back to the user (eventually with a edg-job-get-output).
The input and output sandboxes are intended for relatively small files (on the order of a few megabytes) like scripts, standard input, and standard output streams. If you are using large input files or generating large output files, you should instead directly read from or write to a storage element. Abuse of the input and output sandboxes can fill the storage on the ResourceBroker and make the broker unusable.
The two parameters--Requirements and Rank--control the resource matching for the job. The expression given for the requirements specifies the constraints necessary for a job to run. In this case, a site with more than four CPUs is required. The job will only be submitted to resources which satisfy this condition. If more than one resource matches, then the rank is used to determine which is the most desirable resource and hence the one to which the job is submitted. (Higher values are more desirable.) In this case, the resource with the largest number of free CPUs is chosen. Both of these can be arbitrary expressions which use the fields published by the resources in the information system. The JDL uses the LDAP names of these attributes (which are slightly different than the SQL names used by R-GMA); see Appendix D for a list of useful attributes as well as the JDL Attributes documentshttp://server11.infn.it/workload-grid/documents.html.
JDL is based on the Condor ClassAds library. More information about the supported functions and the syntax of these expressions can be found in the ClassAds documentationhttp://www.cs.wisc.edu/condor/classad/.
ClassAds are extensible and place no restrictions on the parameter names. A side-effect of this is that misspelled parameter names are still syntatically valid, but will not act as expected. Be extremely careful spelling the JDL parameter names.
Using parameters in the JDL file one can steer jobs to sites which have a copy of a particular input file:
InputData = {"lfn:file1.txt","guid:135b7b23-4a6a-11d7-87e7-9d101f8c8b70"}; DataAccessProtocol = {"file", "gridftp"};Both the parameters are required to use this feature. Either logical file names (LFN) or file identifiers (GUID) can be specified; see Chapter 6 for details. The file catalog to use is determined from the information system and the VO specified on the command line. NOTE: If you specify the wrong VO, the wrong catalog will be used leading (usually) to a job matching failure. Similarly, one can also select a site with a particular storage element:
OutputSE = "gppse05.gridpp.rl.ac.uk";For more details, see the job submission examples (Section 4.5), further examples in the Data Management Section (Section 6), and the Workload Management Documentationhttp://www.infn.it/workload-grid/documents.html.
It is often useful to check the results of the resource matching without submitting a job. For this, one can use the edg-job-list-match command. Given the JDL file it will return a ranked list of matching resources. The highest-ranked resource will appear first.
Long jobs may outlive the validity of the initial proxy; if so and the proxy is not renewed, the job will die prematurely. To avoid this the workload management software allows the proxy to be renewed automatically if your credentials are managed by a proxy server.
To use the automatic proxy renewal mechanism, first register a proxy with the MyProxy server using the command
myproxy-init -s <server> -t <hours> -d -nwhere ``server'' is the host name of the MyProxy server and ``hours'' is the number of hours the proxy should be valid on the server (default is 7 days). Specifying the d and n options are vital to proper proxy renewal! MyProxy servers can be found in the service table with R-GMA (see Chapter 3).
As this proxy is only copied to the server, you will need to create a local short-lived proxy using grid-proxy-init to do the job submissions. The resource broker will retrieve renewed proxies from the MyProxy server for jobs which need them.
Information about your stored proxy can be obtained via the command
myproxy-info -s <server> -dand the proxy can be removed with
myproxy-destroy -s <server> -d.Once the proxy is removed from the server, running jobs will no longer receive renewed credentials.
To allow the broker to find the correct MyProxy server, you must specify it in a job's JDL file:
MyProxyServer = "myproxy.example.org";and the MyProxy server must be configured to allow renewals from the broker you use. If this isn't specified in the JDL file, the proxy will not be renewed. A default can be specified in the UI configuration file, if desired.
With the latest versions of the workload management software, it is now possible to submit interactive, MPI (Message Passing Interface), and checkpointed jobs. Interactive jobs open real time connections to the standard input, output, and error streams of the job and allow direct interaction with it. See Example 4.5 for a typical interactive job example. MPI and checkpointed jobs require specially-instrumented user executables linked against the MPI or checkpointing libraries, respectively.
Parallelized computations are typically instrumented with the MPI and are designed to run on many CPUs simulataneously. To submit an MPI job with the EDG software, one must specify two parameters in the job's JDL file:
JobType = "MpiCh"; NodeNumber = 4;The first parameter identifies this job as an MPI executable; the second parameter specifies the number of nodes needed for the executable to run.
The workload management system implements logical checkpointing for jobs. The checkpointing system handles the details for saving and retrieving a checkpoint state. However, the job itself is responsible for the content of the checkpoint state and it must be written to allow restarting from a specified checkpoint state.
Jobs linked against the checkpointing library periodically save the state of the computation; these jobs can be restarted from any of the intermediate saved states. To specify a job which uses an executable instrumented with the checkpointing library, simply specify the type of the job in the JDL file:
JobType = "checkpointable";The checkpointing framework allows you to retrieve the intermediate state of a job and to specify the state when submitting a job. For example, the commands
edg-job-get-chkpt --cs 1 -o <state.file> <jobid> edg-job-submit -chkpt <state.file> <jdl file>will retrieve the penultimate state of the job and then submit a new job which starts from that state.
The simplest job is one which echos ``Hello World'' to the standard output. To run this example prepare a file hello.jdl which contains the following:
Executable = "/bin/echo"; Arguments = "Hello World"; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"};Note that the complete path of the command is given and that the standard output and standard error are specified in the output sandbox.
Submitting the job returns the generated job identifier:
>> edg-job-submit -nomsg Hello.jdl https://boszwijn.nikhef.nl:9000/F0qoyQxCejiW5GLeaYc_Agwhere the nomsg option suppresses superfluous information.
Using edg-job-status with the job identifier above yields the following, edited output:
>> edg-job-status https://boszwijn.nikhef.nl:9000/F0qoyQxCejiW5GLeaYc_Ag Printing status info for the Job : https://boszwijn.nikhef.nl:9000/F0qoyQxCejiW5GLeaYc_Ag Current Status: Running Status Reason: Job successfully submitted to Globus Destination: grid-w2.ifae.es:2119/jobmanager-pbs-workq reached on: Wed Oct 1 13:00:40 2003When the status was requested, the job was running. States seen in the normal processing of jobs are: Accepted, Waiting, Running, and Done. Abnormal execution usually ends with an ``Aborted'' status.
To retrieve the output (once the job has reached the ``Done (Success)'' state), use the edg-job-get-output command with the job identifier as the argument. The following is returned:
>> edg-job-get-output https://boszwijn.nikhef.nl:9000/F0qoyQxCejiW5GLeaYc_Ag Output sandbox files for the job: - https://boszwijn.nikhef.nl:9000/F0qoyQxCejiW5GLeaYc_Ag have been successfully retrieved and stored in the directory: /tmp/jobOutput/F0qoyQxCejiW5GLeaYc_Agwhere the output has again been edited. The important information is the location of the output files. Within the given directory will be the std.out and std.err files specified in the output sandbox. The std.out file should contain the string ``Hello World''. The std.err file is empty in this case but would contain anything written to the standard error.
The contents of the JDL file are transformed by many different programs during the submission process. A side effect of this is that special characters in the ``Arguments'' attribute must be preceeded by triple ` \', e.g.:
Executable = "/usr/bin/tail"; Arguments = "-f file1\\\&file2";
and quotes must be escaped with the ` \' character, e.g.:
Executable = "/bin/grep"; Arguments = "-i \"my name\" *.txt";
or better, create a script as in the next example.
Directly handling the job identifiers quickly becomes tedious. To avoid this, the edg-job-submit will append the job identifier to a named file when using the o option. Job management commands which take job identifiers as an argument accept the i which allows the job identifier to be read from a file.
Note that the job management system does not limit the rate of jobs submitted either by a single user or in total. Extremely high rates have not been tested with this release, but users are asked to show some restraint in this regard.
The next example simply sends a small script with the job, executes it and returns the results. Create an executable file called HelloScript.sh which contains the following:
#!/bin/sh /bin/echo "Hello From Script" /bin/echo "Error From Script" 1>&2This will echo ``Hello From Script'' to the standard output and ``Error From Script'' to the standard error.
The appropriate JDL file for this job is the following.
Executable = "HelloScript.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"HelloScript.sh"}; OutputSandbox = {"std.out","std.err"};If the script is not in the current directory, then you must give the full path in the input sandbox line.
After submitting the job as in the previous example and retrieving the output, one finds that the std.out and std.err files contain the strings ``Hello From Script'' and ``Error From Script'', respectively.
The executable named in the JDL will automatically be set with executable permissions. Other executable files shipped in the input sandbox may need to have the permissions set explicitly.
Important note: The input and output sandboxes are intended for relatively small files (few megabytes) like scripts, standard input, and standard output streams. If you are using large input files or generating large output files, you should instead directly read from or write to a storage element. Abuse of the input and output sandboxes can fill the storage on the ResourceBroker and make the broker unusable.
By specifying job requirements, the user can steer the job to sites which have the resources necessary to run the job correctly. Incompletely specifying the requirements may cause the job to be scheduled on a resource which cannot handle the job. This will cause the job to fail, wasting computing resources and the user's time.
The requirements are specified with a ``Requirements'' attribute in the JDL description of the job. This value of this attribute is a boolean expression which specifies the necessary constraints. Nearly the full set of C operators and syntax are supported.
The values (or variables) which can be used in the requirements expression can be found by looking at the GlueCE attributes in the information system (see Appendix D). To see the values of attributes for all CEs, try the following command:
> edg-rgma -c "latest select * from GlueCE"NOTE: the names vary between the LDAP and SQL versions of the information schema. The ones specified in JDL expressions must be LDAP names (see Appendix D).
To express that a job requires at least 25 minutes of CPU time and 100 minutes of real time, the expression:
Requirements = other.GlueCEPolicyMaxCPUTime>=1500 && other.GlueCEPolicyMaxWallClockTime>=6000;would limit the matching to viable sites. The times are given in seconds. Note that the attribute names are prefixed with ``other.''; this is a remnant of the ClassAds syntax on which JDL is based. Note also that the values are not quoted. Using quotes around a numeric value will result in a string comparision which will produce an erroneous match (or none at all).
The ``GlueHostApplicationSoftwareRunTimeEnvironment'' is usually used to describe application software packages which are installed on a site. For example,
Requirements = Member("ALICE-3.07.01", other.GlueHostApplicationSoftwareRunTimeEnvironment);will choose a site with the ``ALICE-3.07.01'' tag defined. The run time environment is a multi-valued attribute and evaluates to a list. The ``Member'' function returns true if the given value is in the list. (NOTE: The order of these parameters have changed from earlier releases.)
Occasionally, one may wish to exclude or include a site manually. Forcing a job to a site can be accomplished with the resource option of the edg-job-submit command. However, this entirely bypasses the matchmaking process and will not produce a .BrokerInfo file (see the Workload Management Documentationhttp://www.infn.it/workload-grid/documents.html for information on the BrokerInfo file) with the matchmaking results. Instead one can use the matchmaking with a clause like4.1
Requirements = other.GlueCEUniqueID == "adc0006.cern.ch:2119/jobmanager-pbs-short";to do the same thing. More interestingly one can select or exclude a site:
Requirements = RegExp(".*cern.*",other.GlueCEUniqueID); Requirements = (!(RegExp(".*cern.*",other.GlueCEUniqueID)));which cannot be accomplished with the resource option. Note that the JDL is very picky about the logical ``not'' syntax. Many sites also define a run time environment variable which identifies the site.
In the UI configuration file (/opt/edg/etc/edg_wl_ui_cmd_var.conf) there is a requirements clause which is added to all JDL files by default. By default this is
Requirements = other.GlueCEStateStatus == "Production";which chooses sites marked as production. Users may create a UI configuration file of their own to specify habitual requirements (or to choose an alternate resource broker, etc.). To use a custom UI configuration file set the edg_wl_ui_config_path variable to the full path name, or specify the c option when submitting a job with the edg-job-submit command. This configuration also contains a default Virtual Organization to use if one is not specified on the command line.
If more than one resource matches the specified requirements, then the highest-ranked resource will be used. If the ``Rank'' attribute is not specified in the user's JDL description, then
rank = - other.GlueCEStateEstimatedResponseTime;will be used by default (also specified in the UI configuration file). The estimated response time is the expected time in seconds that a job will take to begin executing at the site.
This ranking is not always ideal, and the user may wish to choose some other criteria for the ranking. The rule to remember is that the larger the rank, the more desirable the resource is. If there are multiple resources with the same, highest rank, the broker will choose randomly between those resources. For example,
Rank = other.GlueCEStateFreeCPUs;will choose the site with the largest number of free CPUs. If there are no suitable sites with free CPUs, the broker will choose randomly. An expression like:
Rank = 1;will ensure a random selection between all suitable resources.
Save the following JDL to a file called ``Interactive.jdl'',
JobType = "Interactive"; Executable = "ScriptInt.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"ScriptInt.sh"}; OutputSandbox = {"std.out","std.err"}; ListenerPort = 50101; Requirements = other.GlueCEStateFreeCPUs > 0; Rank = other.GlueCEStateFreeCPUs;and the following to a script called ``ScriptInt.sh''
#!/bin/sh echo "Welcome!" sleep 1; echo "What is your name?" read A echo "Bye Bye $A" exit 0When the job starts, an X-window will be opened allowing you to see the standard output and error streams; moreover, you'll be able to provide the standard input.
Note that you must connect to the User Interface machine in such a way that X-Window input is possible. Usually this means using ssh with the X option.
Also the RB must be able to make an incoming connection to a TCP port on the UI. If your UI is behind a firewall, then specify the listener port in the JDL with the following:
ListenerPort = 50100;where you choose a free port which is accessible from outside the firewall. The environmental variable globus_tcp_port_range, if set, contains a range of ports accessible from the exterior.