Background
Currently I am working on a movie of our accelerator ring. The movie will be a camera flight along the ring and will consist of more than 5000 frames to realize a sequence of about 2 minutes length. The blender file is more than 800 MB of size and rendering one frame (30 samples per pixel) on my local quad-core i5 machine (I do have as well a CUDA graphics card) takes about 5 hours as I am using effects like depth of view, a couple of lights in the scene, full global illumination, etc.Fortunately, at work we have a high performance computing cluster (see https://www.gsi.de/en/work/research/it/hpc.htm) with 550 linux machines offering a maximum of 13800 CPU cores, using SLURM as job manager to distribute the jobs among the cores.
Blender@SLURM: not Always Easy
To use the possibilities of our HPC cluster, I started programming a batch script which submits the following blender command line command to the cluster system:
blender -b <myBlenderFile>.blend -o <output path> -F PNG -s <startFrame> -e <endFrame> -a
which needs to be called via a Linux bash shell script. This command line command calls Blender in batch mode ("-b") and renders an animation set ("-a") from frame "-s" to frame "-e". The output path is specified using the "-o" command.
I started in a naive approach and ended up in frustration. Simply by putting the command above in to a shell script (including the "#!/bin/bash" command as first line) led to errors: the cluster refused to accept the script and always bailed out with an error "slurmstepd: error: Exceeded step memory limit at some point."
With some help of our local experts, I found out that this error is due to the memory-management of Blender: Where the SLURM cluster is able of assign memory per CPU-core (in my case Blender wanted 4GB in total), where Blender is not able to handle memory per core, but requires to have exclusive access to a machine's memory.
Another point is the run time of the job(s). As I said, the run time on a 4 core machine is about 5 hours; the SLURM cluster machines have Xeon processors with up to 40 cores, reducing the run time to something like 15 minutes, depending on the actual hardware. However, this adds up to a total run time for the whole movie of 5000 frames to something like 1200 hours, which is 50 days of CPU time. This means, I would like to perform as much parallel processing as possible. On the other hand, our SLURM cluster by standard does not allow more than 6 hours of run time, which means I cannot render more than 24 frames per submitted batch.
SLURM Shell Script
After some tries I ended up with the following script ("renderAnim.sh") to render the movie:
#!/bin/bash
# Task name:
#SBATCH -J <default job name>
#SBATCH --mail-user=<email address for notification>
#SBATCH --mail-type=FAIL
#SBATCH --time=4:00:00
#SBATCH --exclusive
# Working Directory on shared storage
#SBATCH -D <path to working directory>
# Standard and error output
###SBATCH -o %j_%N.out.log
#SBATCH -e %j_%N.err.log
# Application code
if [ $1 -eq $2 ]
then
PATH=<if you need a local blender installation put PATH here>
blender -b <myBlendFile>.blend -o ./output/ -F PNG -s $1 -e $2 -a
else
if [ $2 -lt 5000 ]
then
OLD_END=$2
NEXT_START=$((OLD_END+1))
NEXT_END=$((OLD_END+10))
sbatch --job-name=${NEXT_START}_${NEXT_END} renderAnim.sh $NEXT_START $NEXT_END
PATH=<if you need a local blender installation put PATH here>
blender -b <myBlendFile>.blend -o ./output/ -F PNG -s $1 -e $2 -a
fi
fi
This script renders 10 frames per batch job and submits one new batch job if SLURM executes the script. By this mechanism, the script requests always as many jobs as possible until all the free machines are used. The option "#SBATCH --exclusive" requests complete machine for the job to account for the "egoistic" memory management of Blender.
I hope this help with you own project - keep on geekin'!
Your WolfiG
