| ... | ... | @@ -3,10 +3,10 @@ |
|
|
|
# Introduction
|
|
|
|
The batch_context plugin relies on two types of configuration files in order to correctly handle the job batch.
|
|
|
|
1. A **Context Config File**
|
|
|
|
This is a single config text file that specifies local and remote directories and system communications. It is unique to the *user* and the *project*. A user needs to create a new one of these for every project. :so: Name the file to includes the project, user and execution type.
|
|
|
|
This is a single config text file that specifies local and remote directories and system communication. It is unique to the *user* and the *project*. A user needs to create a new one of these for every project. :so: Name the file to includes the project, user and execution type.
|
|
|
|
(:b: *contextconfig_local.cfg* and *contextconfig_remote.cfg* )
|
|
|
|
2. **Batch Config Files**
|
|
|
|
These are config text files are are unique to each pipeline *script*, but do not need to be changed according to the user or project. These handle the data file identification, pipeline order, software and computing resources of each of the scripts. :so: It is recommended that these are named to mirror their corresponding scripts as they used hand in hand.
|
|
|
|
These are config text files are are unique to each pipeline *script*, but do not need to be changed according to the user or project. These handle the data file identification, pipeline order, software and computing resources of each of the scripts. :so: It is recommended that these are named to mirror their corresponding scripts.
|
|
|
|
(:b: *c1_.scalpart.cfg* is the config file for *s1_scalpart.htb*)
|
|
|
|
|
|
|
|
:so: Save you config files as text files, this will make them easy to view, edit and save outside of the batch_context user interface.
|
| ... | ... | @@ -18,7 +18,7 @@ You will need a unique context config file for each project. To get started go t |
|
|
|
|
|
|
|
|
|
|
|
## Editing
|
|
|
|
You can create a new or edit and old context config in this user interface. To load an existing config file click on the `| Load Context Config |` button. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely. All jobs must fill out the local dependency fields at the top of the block.
|
|
|
|
You can create a new or edit an old context config in this user interface. To load an existing config file click on the `| Load Context Config |` button. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely. All jobs must fill out the local dependency fields at the top of the block.
|
|
|
|
|
|
|
|
***
|
|
|
|
A detailed description of each of the fields can be found below:
|
| ... | ... | @@ -46,14 +46,14 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
**4. Username for the Remote Host**
|
|
|
|
* This will be your username for your remote host.
|
|
|
|
:b: :warning: Default as *user_name* must be changed to your sharcnet user name.
|
|
|
|
:b: :warning: Default as *user_name* must be changed to your cluster login user name.
|
|
|
|
|
|
|
|
**5. Host for Compute Execution**
|
|
|
|
* This is the computational system on which jobs will be submitted to the scheduler.
|
|
|
|
:b: Default set to *redfin.sharcnet.ca*
|
|
|
|
|
|
|
|
**6. Project Root Archive Address**
|
|
|
|
* This is the address to the archive copy of the project folder. Typically this is on a long term storage system that is not used for compute access but rather for back ups of persistent files. On SHARCNET systems the /freezer system is appropriate for this kind of storage.
|
|
|
|
* This is the address to the archive copy of the project folder. Typically this is on a long term storage system that is not used for compute access but rather for backup of persistent files. On SHARCNET systems the /freezer system is appropriate for this kind of storage.
|
|
|
|
:b: :warning: Default of *dtn.sharnet.ca:/freezer/user_name/project_name* needs to be adjusted for your project.
|
|
|
|
|
|
|
|
**7. Project Root Work Address**
|
| ... | ... | @@ -67,11 +67,11 @@ A detailed description of each of the fields can be found below: |
|
|
|
***
|
|
|
|
|
|
|
|
**9. Archive Mount Directory**
|
|
|
|
* This is the local path where the remote archive directory can be mounted (e.g. using [sshfs](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/remote-start-up)). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
|
|
|
|
* This is the local path where the remote archive directory can be mounted (e.g. using [sshfs](remote-start-up)). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
|
|
|
|
:b: Default set to *remote_archive*
|
|
|
|
|
|
|
|
**10. Work Mount Directory**
|
|
|
|
* This is the local path where the remote work directory can be mounted (e.g. using [sshfs](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/remote-start-up)). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
|
|
|
|
* This is the local path where the remote work directory can be mounted (e.g. using [sshfs](remote-start-up)). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
|
|
|
|
:b: Default set to *remote_work*
|
|
|
|
|
|
|
|
***
|
| ... | ... | @@ -84,8 +84,8 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
**12. System Variables**
|
|
|
|
* System variables are custom system commands you can quickly make using the other context configuration variables.
|
|
|
|
* These are strings that will have key strings swapped and then made available to editing before being passed to the "system" command for execution in the system terminal.
|
|
|
|
For more information visit the [Mounting the Project](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/remote-start-up#mounting-the-project) on the [Managing Remote Projects](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/remote-start-up) page.
|
|
|
|
* These are strings that will have key strings swapped and then made available for editing before being passed to the "system" command for execution in the system terminal.
|
|
|
|
For more information visit the [Mounting the Project](remote-start-up#mounting-the-project) on the [Managing Remote Projects](remote-start-up) page.
|
|
|
|
:b: Default set as:
|
|
|
|
*sshfs [remote_user_name]@[remote_project_archive] [cd]/[mount_archive]
|
|
|
|
sshfs [remote_user_name]@[remote_project_work] [cd]/[mount_work]
|
| ... | ... | @@ -100,7 +100,7 @@ Once you have filled out all of the fields click on the `| Save As |` button to |
|
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use this text file as a template to quickly edit and create new context config files in a text editor.
|
|
|
|
|
|
|
|
# Batch Configuration Files (*.cfg)
|
|
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user.
|
|
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependent on the project or the user.
|
|
|
|
:so: Only create one local and one remote file for each script. To get started go to:
|
|
|
|
* *File -> Batch -> Batch Configuration*
|
|
|
|
|
| ... | ... | @@ -115,44 +115,45 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
**1. Execute Function**
|
|
|
|
This is a drop down menu where you will select one of the following:
|
|
|
|
* *Current Base*
|
|
|
|
This designates the job to be completed locally.
|
|
|
|
* *Sq_Sub*
|
|
|
|
This designates the job to be completed on the Sharcnet using the sqsub wrapper scheduler.
|
|
|
|
* *q_Sub*
|
|
|
|
This designates the job to be completed on the Sharcnet using the qsub scheduler.
|
|
|
|
* Other schedulers are currently being developed including Slurm and Torque.
|
|
|
|
* *current_base*
|
|
|
|
This specifies the job be run locally.
|
|
|
|
* *sq_sub*
|
|
|
|
This specifies the job be run on Sharcnet using the sqsub wrapper scheduler.
|
|
|
|
* *sbatch*
|
|
|
|
This specifies the job be run on a Slurm scheduler that uses sbatch. Like the new Graham or Cedar clusters at Compute Canada
|
|
|
|
|
|
|
|
:b: The default depends on whether you loaded the local or remote file.
|
|
|
|
|
|
|
|
**2. Replace String**
|
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files. Refer to [Creating Scripts](https://git.sharcnet.ca/jdesjard/batch_context/wikis/script-files) for more information. Notice that each variable is identified in **[]** brackets, then proceded by a **,** comma, and finally each variable is on a separate line.
|
|
|
|
:b: Script one: *scalpart.htb* looks for many variables in the swap string field, one of which is *[in_path]* which can be seen is initialised in the in the swap string field by: *[in_path],analysis/data/1_init*. By default these are already complete.
|
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initializing variables to be used in each of the loaded data files. Refer to [Creating Scripts](script-files) for more information. Notice that each variable is identified in **[]** brackets, then proceded by a **,** comma, and finally each variable is on a separate line.
|
|
|
|
:b: Script one: *scalpart.htb* looks for many variables in the swap string field, one of which is *[in_path]* which can be seen is initialized in the swap string field by: *[in_path],analysis/data/1_init*. By default these are already complete.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
To find out which variables the config file must set as swap strings, look in the comments at the top of the scripts.
|
|
|
|
|
|
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code. You can also use them to make decisions on options like 'yes' or 'no' to preform different sections of code within the script.
|
|
|
|
* **[batch_dfn] and [batch_dfp] ** are special swap strings that are not collected from the config file. Instead they are gathered from the loaded EEG data files.
|
|
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code. You can also use them to make decisions on options like 'yes' or 'no' to perform different sections of code within the script.
|
|
|
|
* **[batch_dfn] and [batch_dfp] ** are special swap strings that are not collected from the config file. Instead they are gathered from the loaded EEG data files. Other builtin swap strings variables are available and are recorded in [builtin swap strings.](built-in-swap-strings)
|
|
|
|
|
|
|
|
**3. Order**
|
|
|
|
* The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order.When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
|
|
* The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order. When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
|
|
* [1] [2] [3] [4] and so on...
|
|
|
|
|
|
|
|
* When running a job remotely you have the option to submit jobs in parallel in order to increase the pipeline's efficiency. The field input will be an array, where the first value is the order number, which denotes the position of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next value or values in the array denote what other scripts must be complete before this one starts.
|
|
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
|
|
|
|
|
|
|
|
Note: certain configurations are impossible due to the way we submit jobs and record their job ids. You must ensure that your job is capable of being submitted in linear order and still being valid. This means that dependents must be after parents in the linear order. A way to ensure this is to draw the jobs as a graph and number them breadth first.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
* This order structure allows for infinite combinations and layouts of pipelines. As you may have noticed the first order number does not matter, **but** remember that files are loaded into the history script field alphabetically, so you may need to adjust the files names to ensure that the jobs that start first must be higher in the call order. In our case the file name for job [5,2] must be alphabetically smaller then the file name for job [6,4,5].
|
|
|
|
* This order structure allows for infinite combinations and layouts of pipelines. As you may have noticed the first order number does not matter, **but** remember that files are loaded into the history script field alphabetically, so you may need to adjust the files names to ensure that the jobs that start first must be higher in the call order. In our case the file name for job [5,2] must be lexicographically smaller then the file name for job [6,4,5].
|
|
|
|
|
|
|
|
**4. Job Name**
|
|
|
|
* This will indicate what the job name will be. [Batch swap string rules apply](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/adv-str-swap). This name is used as the .m file name that is runs the scripts on your files, as well as the name of the cluster job name. Make sure that this name is under 64 characters as matlab can not run scripts with names that are too long.
|
|
|
|
:b: Default as *[batch_hfn,.,1]_[batch_dfn,.,1]* indicating that the job name will be named after the current script followed by the the name of the data file.
|
|
|
|
* This will indicate what the job name will be. [Batch swap string rules apply](adv-str-swap). This name is used as the .m file name that is runs the scripts on your files, as well as the name of the cluster job name. Make sure that this name is under 64 characters as Matlab can not run scripts with names that are too long.
|
|
|
|
:b: Defaults to *[batch_hfn,.,1]_[batch_dfn,.,1]* indicating that the job name will be named after the current script followed by the the name of the data file.
|
|
|
|
|
|
|
|
**5. Session Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
|
| ... | ... | @@ -163,38 +164,23 @@ module unload intel/12.1.3; |
|
|
|
module unload octave;
|
|
|
|
module load octave/3.8.1;
|
|
|
|
```
|
|
|
|
If you are running a different program other than octave your session init scrip will be different. The tutorial also contains a session init file for AMICA.
|
|
|
|
If you are running a different program other than octave your session init script will be different. The tutorial also contains a session init file for AMICA.
|
|
|
|
|
|
|
|
**6. Job Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank. The job init field is used in the reference pipeline to add a line to the submit script before each job that adds the LD_RUN_PATH to the LD_LIBRARY_PATH before running the script.
|
|
|
|
:b: Default is left blank ' '.
|
|
|
|
|
|
|
|
**7. M File Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally or running a binary file you can leave this field blank.
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote server at the beginning of the batch of jobs. If you are running a project locally or running a binary/script you should leave this field blank.
|
|
|
|
:b: Remote Default for Matlab is *analysis/support/config/matlab.minit* containing:
|
|
|
|
|
|
|
|
```
|
|
|
|
pop_editoptions('option_savetwofiles', 0,'option_single',0);
|
|
|
|
```
|
|
|
|
In order to make sure your EEGLAB has its options set properly.
|
|
|
|
:b: The reference pipeline default for Octave is *analysis/support/config/octave.minit* containing various fixes for specific versions of Octave and code to include scripts into our path.
|
|
|
|
|
|
|
|
:b: Remote Default for Octave is *analysis/support/config/octave.minit* containing:
|
|
|
|
|
|
|
|
```
|
|
|
|
warning off;
|
|
|
|
pkg load signal;
|
|
|
|
addpath(genpath('[:,1,remote_dependency]'));
|
|
|
|
cd '[:,1,remote_project_work]';
|
|
|
|
```
|
|
|
|
It also contains a loop which will remove the *Fieldtrip* plugin as it will crash Octave.
|
|
|
|
**8. Submit Options**
|
|
|
|
|
|
|
|
**8. Scheduler Options**
|
|
|
|
* This section contains a series of characters to communicate the job specifications to the remote server, such as memory and time. Click on the field and look at the information box below to see the specific commands. If you are running a project locally you can leave this field blank.
|
|
|
|
:b: Remote default is *-r 4h --mpp 6G* for Octave Scripts and *-r 3h --mpp 1G --ppn 1 -q mpi -n 8* for AMICA files when selecting sqsub. For qsub the defaults would be *-q serial -l nodes=1:ppn=1,pvmem=6gb,walltime=4:00:00* for Octave Files, and *-q mpi -l nodes=1:ppn=8,pvmem=1gb,walltime=3:00:00* for AMICA.
|
|
|
|
These should be adjusted based on what server you are running your jobs on as well as the kind of process you are preforming. Some jobs will require more time and some will require more memory. These also effect the que time your jobs will have in the scheduler. Most of the time jobs with shorter time and lower memory will get into que faster, but you also don't want your job to crash as you did not give it enough resources.
|
|
|
|
|
|
|
|
The nature of the string also depends on what scheduler you are using.
|
|
|
|
:warning: The current systems use qsub or sqsub, but as of April 2017 the new Sharcnet servers will be using Slurm.
|
|
|
|
|
|
|
|
**9. Software**
|
|
|
|
This is a drop down menu where you can select one of the following:
|
| ... | ... | @@ -218,4 +204,6 @@ Now when running a job you can simply load this file and it will populate the re |
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
*Updated/Verified by Brad Kennedy on 2017-08-08*
|
|
|
|
|
|
|
|
[ :house: **Return to Main Page**](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/home) |
