|
|
# Pipeline Cluster Configuration
|
|
|
|
|
|
We have spent significant time optimizing the run time and memory allocation of the pipeline processes. On Compute Canada clusters reducing the time and memory allocation of a job submission will reduce the time spent on the queue and therefore improve throughput all other things remaining equal. Over committing time and memory can be a huge waste of resources.
|
|
|
Below are various configuration options that may require customization if a study is particularly long or complex. Please note that significant empirical study was done to optimize the run time and memory allocations of all of the pipeline processes. As such, modifications to these parameters may dramatically increase queuing time.
|
|
|
|
|
|
# job_name & mfile_name
|
|
|
These fields support string swapping with batch_hfn and batch_dfn see [Batch Context](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/adv-str-swap) for instructions on how to do string swapping.
|
|
|
|
|
|
However we have set them to be useful for development of the pipeline. Job names are used in the scheduler for tracking the jobs and mfile name is used when making the pipeline script for the output name and output log name.
|
|
|
Note that some options have been left out as their modification is too complex or integral to the pipeline and as such cannot be changed. If you still wish to change an option not found on this page please either [contact](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/wikis/contributing#contacting-us) us or file an [issue](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/issues).
|
|
|
|
|
|
# qsub_options (passthrough options)
|
|
|
|
|
|
Note: More accurately to be named passthrough_options or scheduler_options or additional_options
|
|
|
# job_name & mfile_name
|
|
|
These fields support string swapping with batch_hfn and batch_dfn. See [Batch Context](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/adv-str-swap) for instructions on string swapping.
|
|
|
|
|
|
These options are not abstracted in anyway and are passed directly to the scheduler at the end of the list of parameters. This allows you to pick specific flags that Batch Context does not provide abstractions for such as partitions or other accounting information.
|
|
|
# Passthrough Options
|
|
|
Options in this field are arguments to be directly passed to either `srun` or `sbatch` without abstraction.
|
|
|
|
|
|
# Memory & Time
|
|
|
Memory and time(time_limit) are special fields that support expressions to scale the number of minutes/hours or megabyte/gigabytes used for a given job based on the number of samples and number of channels in a data file.
|
|
|
Memory and time (time_limit) are fields that support expressions to scale the number of minutes/hours or megabyte/gigabytes used for a given job based on the number of samples and number of channels in a data file.
|
|
|
|
|
|
To use expressions provide an expression that contain the variable `s `and variable `c` for samples and channels respectively. The expressions must be valid Matlab expressions that result in a scalar, but also must end in scaling factor.
|
|
|
Expressions contain variables `s `and `c` for samples and channels respectively. The expressions must be valid Matlab expressions that result in a scalar, but also must end in scaling factor.
|
|
|
|
|
|
For memory the expressions must end in either
|
|
|
* m for megabytes
|
... | ... | @@ -27,24 +24,20 @@ For time_limit the expressions must end in either |
|
|
* m for minutes
|
|
|
* h for hours
|
|
|
|
|
|
For example in the memory field:
|
|
|
Example:
|
|
|
```matlab
|
|
|
3 + c* 0.1g
|
|
|
```
|
|
|
Will make jobs 3 gigabytes plus 0.1 gigabytes for each channel in the data. This would be a lot of memory so we advise against this particular expression and it is given for example. As you can see the expression can be fractional. The handling of the fractional values is handled by scheduler drivers. If the scheduler drivers do not accept a expression you believe should work please [file an issue](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/issues).
|
|
|
The above expression will make jobs 3 gigabytes plus 0.1 gigabytes in size for each channel in the data.
|
|
|
|
|
|
# mpi
|
|
|
Do not change this setting, amica jobs are the only jobs capable of using mpi in our pipeline, any other jobs using mpi will lead to undefined behaviour and may waste cpu time or increase queue time.
|
|
|
**Please note that this is too much memory and is only written for example purposes**
|
|
|
|
|
|
# num_processors (more accurately number of processes)
|
|
|
# num_processors
|
|
|
|
|
|
Number of tasks, processes, sometimes matches the number of processors.
|
|
|
|
|
|
For some jobs the number of processors can be increased. Do not do this for any of the Octave jobs as they will not benefit from more processors and will results in longer queue times and wasted cpu hours. For jobs like Amica(s02, s05, s07-12) additional processors will improve runtime performance. You however, should be aware that again more processors will result in longer queue times and therefore you should balance the number of processors based on the total job time (queue time + runtime). Parallel jobs that communicate between tasks have overhead associated them so there will be diminishing returns as you increase the number of processors on a parallel job in most cases.
|
|
|
|
|
|
# cpus_per_task (threads per process)
|
|
|
|
|
|
This field will likely be introduced soon, this will facilitate threading on a single process. Threading massively increases the queue times on busy clusters so you should avoid specifying a high value here as it will lead to extremely poor performance on busy clusters.
|
|
|
For some jobs the number of processors can be increased. Do not do this for any of the Octave jobs as they will not benefit from more processors and will results in longer queue times and wasted cpu hours. For Amica jobs, additional processors will improve runtime performance. However, be aware that more processors will result in longer queue times and therefore balancing the number of processors based on the total job time (queue time + runtime) is critical. Parallel jobs that communicate between tasks have overhead associated them so there will be diminishing returns as you increase the number of processors on a parallel job in most cases.
|
|
|
|
|
|
***
|
|
|
|
|
|
[ :house: **Return to Main Page**](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/wikis/home) |
|
|
\ No newline at end of file |