Springe direkt zu Inhalt

Job Array

Introduction

For a large number of jobs which can be parameterised by a single integer, job arrays can be used. A job array is submitted in the following manner

sbatch --array=1-20 my_script.sh

For each value specified by the --array option, the variable ${SLURM_ARRAY_TASK_ID} in the script is replaced by that value. Therefore ${SLURM_ARRAY_TASK_ID} must be used in the job script to define the desired input.  The maximum array size is 5001.

Simple example

In the simplest case, the array index is passed as an arguement directly to the program:

#!/bin/bash

#SBATCH --job-name=my_job_array                 # replace name
#SBATCH --mail-user=example@zedat.fu-berlin.de  # replace email address
#SBATCH --mail-type=end
#SBATCH --nodes=1
#SBATCH --ntasks=1                              # replace with value for your job
#SBATCH --mem-per-cpu=4096                      # replace with value for your job
#SBATCH --time=08:00:00                         # replace with value for your job
#SBATCH --qos=standard                          # replace with value for your job

module add ExampleProg/1.2.3-foss-2018b         # replace with value for your job

cd /scratch/${USER}                             # replace with your directory
exampleprog ${SLURM_ARRAY_TASK_ID}              # replace with your program

Simple example with non-integer parameters

For non-integer parameters, an array is first definined and then the array index used to select an indiviual parameter:

#!/bin/bash

#SBATCH --job-name=my_job_array                 # replace name
#SBATCH --mail-user=example@zedat.fu-berlin.de  # replace email address
#SBATCH --mail-type=end
#SBATCH --nodes=1
#SBATCH --ntasks=1                              # replace with value for your job
#SBATCH --mem-per-cpu=4096                      # replace with value for your job
#SBATCH --time=08:00:00                         # replace with value for your job
#SBATCH --qos=standard                          # replace with value for your job

module add ExampleProg/1.2.3-foss-2018b         # replace with value for your job

declare -a fruit
fruit=(apple banana cherry damson)

cd /scratch/${USER}                             # replace with your directory
exampleprog ${fruit[${SLURM_ARRAY_TASK_ID}]}    # replace with your program

Multiparameter example

However, an array of parameter combinations can aslo be generated and the array index used to select a specifc combination:

#!/bin/bash

#SBATCH --job-name=my_job_array                 # replace name
#SBATCH --mail-user=example@zedat.fu-berlin.de  # replace email address
#SBATCH --mail-type=end
#SBATCH --nodes=1
#SBATCH --ntasks=1                              # replace with value for your job
#SBATCH --mem-per-cpu=4096                      # replace with value for your job
#SBATCH --time=08:00:00                         # replace with value for your job
#SBATCH --qos=standard                          # replace with value for your job

declare -a combinations
index=0
for run in `seq 1 10`
do
    for direction in 'north' 'south' 'east' 'west'
    do
        for size in 7 11 19 33 112
        do
            combinations[$index]="$run $direction $size"
            index=$((index + 1))
        done
    done
done

parameters=(${combinations[${SLURM_ARRAY_TASK_ID}]})

run=${parameters[0]}
direction=${parameters[1]}
size=${parameters[2]}

module add ExampleProg/1.2.3-foss-2018b         # replace with value for your job

cd /scratch/${USER}                             # replace with your directory
exampleprog ${run} ${direction} ${size}

Keywords