The links below provide some useful information about basic slurm commands and get started with using slurm to submit a job and how to specify the partition and qos:
Checking SLURM Command Options
All slurm command options can be checked by, e.g. man sinfo etc
sinfo --format "%20N %.6D %20P %.11T %.4c %.8z %.8m %.8d %.6w %.8f %20E"
will give each node’s number of CPUs, and memory per node in MB
also, you can play with sinfo, sinfo -s, sinfo -N -l,
Note: -N, -l, -s will all be overwritten by –format option
scontrol show job job_id gives the job information
scontrol show node compute-1-1 gives the node information of compute-1-1 node
sacct -j job_id --format="MaxRSS" will give you the memory usage of a job
for some particular reason, if in the partition you specified, you want to use an individual node to run the job, you can do it by in sbatch file by e.g. #SBATCH --nodelist compute-1-1
slurmtop gives a good overview of current status of each node and how current running jobs are allocated on each CPUs of all nodes
check the running jobs on a specific node, squeue -w compute-1-10
How do I submit jobs to a specific queue?
Slurm doesn’t really have queues; the closest analog would be the combination of partition and QOS. A partition is a group of nodes, and a QOS is a policy that says how many cores you can use for how long, whether jobs can be preempted, etc.
Submit with sbatch -p partition -q qosname
What partitions and QOSes are available?
The following partitions are limited to stakeholders who own the compute nodes in each: owners, highmem, ultrahigh, mh1, gpu_owners.
The following partitions can be used by anyone: main, long, threaded, gpu.
The long and threaded partitions include separate sets of nodes all owned by OIT, so anyone can use them without being preempted by a stakeholder who owns the nodes. The main partition includes all of the nodes from the long, owners, highmem, ultrahigh, and mh1 partitions, but jobs in the main partition may be subject to preemption if they end up on a node owned by a stakeholder who then submits a job in the owners, highmem, ultrahigh, or mh1 partition. The main partition allows jobs of any size, but the higher-priority jobs submitted in owners, highmem, or ultrahigh are limited to the amount of compute resources the stakeholder has purchased. The idea is that the people who own compute nodes are guaranteed access to the resources they own, within a reasonable timeframe, but others can make opportunistic use of the same hardware when it would be sitting idle.
Then there are the different options for qos. Each stakeholder has their own qos named after their username or their department, for example, avolkov1, math, etc., which allows them to use the resources they purchased for an unlimited time, preempting others. There are a few specialized qos policies that can only be used on a specific partition, for example on the threaded partition you must use the threaded qos, and on the gpu partition you must use the gpu qos. The main qos is the one you’d usually use in the main partition, allowing up to 24 hour runtime on any number of nodes / cores, but jobs are preemptable as described above.
The long qos allows up to 7 day runtime, and can be used on either the long partition or the main partition. Overall, long qos jobs are limited to about 1/4 of the cluster at any given time so that resources will be available for shorter jobs within a reasonable timeframe. If used on the main partition, it may mean that the job is preemptable, but it may be easier for SLURM to find somewhere to run the job since more nodes are included in the main partition. If used on the long partition it will mean that the job cannot be preempted but the types of nodes are more limited.
In summary:
-p main : This is the partition which we expect to be used most of the time. QOSes available to use with this are “main”, “long”, and “debug”.
-p long : This partition contains nodes that cannot be preempted by an owner. Use only the “long” QOS with this.
-p owners : This partition is for system stakeholders who own nodes. It gives a higher priority and may preempt jobs running in the main partition. Users are assigned an owner QOS based on their group.
-p highmem : This partition is only available to stakeholders who have purchased nodes with moderately increased amounts of memory per core. It gives a higher priority and may preempt jobs running in the main partition. Users are assigned an owner QOS for this if they are in these particular stakeholders groups.
-p ultrahigh : This partition is like highmem, but even moreso. It is typically reserved for those who have purchased nodes with 1TB of memory.
-p threaded : This partition contains a small number of OIT-owned nodes intended for software that uses lightweight processes / threads to parallelize, rather than a distributed framework like MPI. The nodes have slightly faster CPUs than the others, and much more memory. Use only the “threaded” QOS with this.
HPC Acknowledgement
If you use any HPC resource in your research efforts, we appreciate you adding the following statement in your Acknowledgement section: “We would like to thank The University of Alabama and the Office of Information Technology for providing high-performance computing resources and support that have contributed to these research results.”