PJM is a simple job manager for PBS (Portable Batch System) written by Python. The main function of PJM is inspirated by SJM. Similarlly, PJM can be used as a framework for bioinformatics analysis pipelines by automatically submitting jobs, monitoring job status and controlling the workflow according to dependencies of each job. However, jobs in PJM are defined like "functions", with all the settings including computing resources and job dependencies, which is distinct from SJM.
PJM has beed tested on Torque. But it remains unknown whether PJM also work well on OpenPBS or PBS Professional. PJM may be ported to or compatible with other schedulers in the future, such as SGE (Sun Grid Engine).
- Python 2.7
cd ~ # or anywhere you like
git clone https://github.com/zengxiaofei/PJM.git
export PATH=~/PJM:$PATH # you can add it to .bashrc or .bash_profile
Print help message:
$ pjm.py -h
usage: pjm.py [-h] [--sleep SLEEP] job_file
** A Simple Job Manager for PBS (PJM) Version 0.9 **
. Author: Xiaofei Zeng
. Email: xiaofei_zeng@whu.edu.cn
positional arguments:
job_file input job file
optional arguments:
-h, --help show this help message and exit
--sleep SLEEP monitor interval time (seconds) [default: 60]
There is only one required argument job_file
.
To run a pipeline, just:
$ pjm.py a_pipeline.job
A pipeline always contains a series of jobs. Each job has its own demand of resources (memory, processors), and running serially or in parallel in the workflow. It's neccessary and user-friendly to set up these when defining a job.
Here is an example of a job definition:
job_example(depend=another_job;nodes=1;ppn=8;queue=cu;stringency=high;dir=example_dir){
echo 'This is an example!'
}END
-
job_example
defines the JOBNAME, which is the unique identifier of the job. Avoid duplicated JOBNAME. -
depend=jobA
means thatjob_example
should be run afteranother_job
finished. Default is no dependency. -
nodes=1
,ppn=8
,queue=cu
are settings for the job resources and queue. -
stringency=high
will add a shell commandset -o errexit
at the head of generated.pbs
file of the job. If an error occurs, the job will be aborted and exit. PJM will wait for other jobs to finish if they are independent with the failed job. -
dir=example_dir
is the working directory of the job. PJM will creat a folder when this job is ready to run. Default is JOBNAME_dir. -
echo 'This is an example!'
is a shell command of the job. When you are building a pipeline, the commands can be running softwares, executing custom scripts, or even calling sub-processes.
The default settings of nodes
, ppn
, queue
, stringency
and avaliable queue list
can be edited in configuration file pjm.cfg
:
[Default]
nodes=1
ppn=1
queue=cu
stringency=high
[PBS]
QueueList=cu,fat,batch,assemble