|
|
**Table of contents**
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
# Pipeline Batch Config Options
|
|
|
|
|
|
We have spent significant time optimizing the run time and memory allocation of the pipeline processes. On Compute Canada clusters reducing the time and memory allocation of a job submission will reduce the time spent on the queue and therefore improve throughput all other things remaining equal. Over committing time and memory can be a huge waste of resources.
|
|
|
|
|
|
# job_name & mfile_name
|
|
|
These fields support string swapping with batch_hfn and batch_dfn see [Batch Context](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/adv-str-swap) for instructions on how to do string swapping.
|
|
|
|
|
|
However we have set them to be useful for development of the pipeline. Job names are used in the scheduler for tracking the jobs and mfile name is used when making the pipeline script for the output name and output log name.
|
|
|
|
|
|
# qsub_options (passthrough options)
|
|
|
|
|
|
Note: More accurately to be named passthrough_options or scheduler_options or additional_options
|
|
|
|
|
|
These options are not abstracted in anyway and are passed directly to the scheduler at the end of the list of parameters. This allows you to pick specific flags that Batch Context does not provide abstractions for such as partitions or other accounting information.
|
|
|
|
|
|
# Memory & Time
|
|
|
Memory and time(time_limit) are special fields that support expressions to scale the number of minutes/hours or megabyte/gigabytes used for a given job based on the number of samples and number of channels in a data file.
|
|
|
|
|
|
To use expressions provide an expression that contain the variable `s `and variable `c` for samples and channels respectively. The expressions must be valid Matlab expressions that result in a scalar, but also must end in scaling factor.
|
|
|
|
|
|
For memory the expressions must end in either
|
|
|
* m for megabytes
|
|
|
* g for gigabytes
|
|
|
|
|
|
For time_limit the expressions must end in either
|
|
|
* s for seconds
|
|
|
* m for minutes
|
|
|
* h for hours
|
|
|
|
|
|
For example in the memory field:
|
|
|
```matlab
|
|
|
3 + c* 0.1g
|
|
|
```
|
|
|
Will make jobs 3 gigabytes plus 0.1 gigabytes for each channel in the data. This would be a lot of memory so we advise against this particular expression and it is given for example. As you can see the expression can be fractional. The handling of the fractional values is handled by scheduler drivers. If the scheduler drivers do not accept a expression you believe should work please [file an issue](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/issues).
|
|
|
|
|
|
# mpi
|
|
|
Do not change this setting, amica jobs are the only jobs capable of using mpi in our pipeline, any other jobs using mpi will lead to undefined behaviour and may waste cpu time or increase queue time.
|
|
|
|
|
|
# num_processors (more accurately number of processes)
|
|
|
|
|
|
Number of tasks, processes, sometimes matches the number of processors.
|
|
|
|
|
|
For some jobs the number of processors can be increased. Do not do this for any of the Octave jobs as they will not benefit from more processors and will results in longer queue times and wasted cpu hours. For jobs like Amica(s02, s05, s07-12) additional processors will improve runtime performance. You however, should be aware that again more processors will result in longer queue times and therefore you should balance the number of processors based on the total job time (queue time + runtime). Parallel jobs that communicate between tasks have overhead associated them so there will be diminishing returns as you increase the number of processors on a parallel job in most cases.
|
|
|
|
|
|
# cpus_per_task (threads per process)
|
|
|
|
|
|
This field will likely be introduced soon, this will facilitate threading on a single process. Threading massively increases the queue times on busy clusters so you should avoid specifying a high value here as it will lead to extremely poor performance on busy clusters.
|
|
|
|
|
|
***
|
|
|
[ :house: **Return to Main Page**](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/home) |
|
|
\ No newline at end of file |