Introduction
The batch_context plugin relies on two types of configuration files in order to correctly handle the job batch.
- A Context Config File
This is a single config text file that specifies local and remote directories and system communications. It is unique to the user and the project. A user needs to create a new one of these for every project.
(
-
Batch Config Files
These are config text files are are unique to each pipeline script, but do not need to be changed according to the user or project. These handle the data file identification, pipeline order, software and computing resources of each of the scripts.
(
Context configuration files (*.cfg)
You will need a unique context config file for each project. To get started go to:
-
File ->Batch-> Context Configuration
Editing
You can create a new or edit and old context config in this user interface. To load an existing config file click on the | Load Context Config | button. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely. All jobs must fill out the local dependency fields at the top of the block.
A detailed description of each of the fields can be found below:
Required for All Jobs
1. Log Path
- This is the directory location of your log folder. The log folder is used to collect the intermediate scripts created and used for each file, as well as collecting any error messages.
2. Project Root Directory
- This is the directory location of your project folder.
3. Dependency Root Directory
- This is the directory location of your project dependencies. These included EEGLAB and any plugins.
Only Required for Remote Jobs
4. Username for the Remote Host
- This will be your username for your remote host.
5. Host for Compute Execution
- This is the computational system on which jobs will be submitted to the scheduler.
6. Project Root Archive Address
- This is the address to the archive copy of the project folder. Typically this is on a long term storage system that is not used for compute access but rather for back ups of persistent files. On SHARCNET systems the /freezer system is appropriate for this kind of storage.
7. Project Root Work Address
- This is the address to the working copy of the project folder. Typically this is on a short term storage system that has good performance with compute nodes. On SHARCNET systems this is typically the global access /work file system or the cluster specific /scratch systems.
8. Dependency Address
- This is the address to the path where necessary files (e.g. the EEGLAB file distribution) are located for access by the compute nodes. Typically this folder is included within each project's file distribution but it can also be a central location shared among several projects.
9. Archive Mount Directory
- This is the local path where the remote archive directory can be mounted (e.g. using sshfs). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
10. Work Mount Directory
- This is the local path where the remote work directory can be mounted (e.g. using sshfs). Relative paths are prefixed with [local_project] or the current working directory (pwd) if [Local_project] is empty.
11. Miscellaneous Locations
- This field allows allows the user to add new location names and addresses. The inputs are expected to be key strings and address/paths of other places (e.g. '[external_backup],/media/user/external/backup/project_name]').
12. System Variables
- System variables are custom system commands you can quickly make using the other context configuration variables.
- These are strings that will have key strings swapped and then made available to editing before being passed to the "system" command for execution in the system terminal.
For more information visit the Mounting the Project on the Managing Remote Projects page.
sshfs [remote_user_name]@[remote_project_archive] [cd]/[mount_archive]
sshfs [remote_user_name]@[remote_project_work] [cd]/[mount_work]
meld [local_project]/analysis [cd]/[mount_work]/analysis &
meld [local_project]/analysis [cd]/[mount_archive]/analysis [cd]/[mount_work]/analysis &
fusermount -u [cd]/[mount_archive]
fusermount -u [cd]/[mount_work]
Saving
Once you have filled out all of the fields click on the | Save As | button to save the file.
(
Batch Configuration Files (*.cfg)
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user.
-
File -> Batch -> Batch Configuration
Editing
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the | Get Batch Config File Names | button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a designated configuration file for it. If you are loading you can adjust the order of the files by changing the names of them in the the load window. If you do not change anything they will be loaded alphabetically. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
A detailed description of each of the fields can be found below:
1. Execute Function
This is a drop down menu where you will select one of the following:
-
Current Base
This designates the job to be completed locally. -
Sq_Sub
This designates the job to be completed on the Sharcnet using the sqsub wrapper scheduler. -
q_Sub
This designates the job to be completed on the Sharcnet using the qsub scheduler.- Other schedulers are currently being developed including Slurm and Torque.
2. Replace String
- Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files. Refer to Creating Scripts for more information. Notice that each variable is identified in [] brackets, then proceded by a , comma, and finally each variable is on a separate line.
To find out which variables the config file must set as swap strings, look in the comments at the top of the scripts.
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code. You can also use them to make decisions on options like 'yes' or 'no' to preform different sections of code within the script.
- **[batch_dfn] and [batch_dfp] ** are special swap strings that are not collected from the config file. Instead they are gathered from the loaded EEG data files.
3. Order
-
The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order.When running a job locally your computer will focus on one task at a time so your script order will be:
-
[1] [2] [3] [4] and so on...
-
When running a job remotely you have the option to submit jobs in parallel in order to increase the pipeline's efficiency. The field input will be an array, where the first value is the order number, which denotes the position of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next value or values in the array denote what other scripts must be complete before this one starts.
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
- This order structure allows for infinite combinations and layouts of pipelines. As you may have noticed the first order number does not matter, but remember that files are loaded into the history script field alphabetically, so you may need to adjust the files names to ensure that the jobs that start first must be higher in the call order. In our case the file name for job [5,2] must be alphabetically smaller then the file name for job [6,4,5].
4. Job Name
- This will indicate what the job name will be. Batch swap string rules apply.
5. Session Init
- This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
module unload intel/12.1.3;
module unload octave;
module load octave/3.8.1;If you are running a different program other than octave your session init scrip will be different. The tutorial also contains a session init file for AMICA.
6. Job Init
- This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
7. M File Init
- This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally or running a binary file you can leave this field blank.
warning off;
pkg load signal;
addpath(genpath('[:,1,remote_dependency]'));
cd '[:,1,remote_project_work]';It also contains a loop which will remove the Fieldtrip plugin as it will crash Octave.
8. Scheduler Options
- This section contains a series of characters to communicate the job specifications to the remote server, such as memory and time. Click on the field and look at the information box below to see the specific commands. If you are running a project locally you can leave this field blank.
The nature of the string also depends on what scheduler you are using.
9. Software
This is a drop down menu where you can select one of the following:
- Matlab : select matlab for every script when running a job locally
- Octave : select octave when you are running matlab scripts remotely, Octave is the free alternative found on many servers.
- None : select none when you are running a binary process remotely such as AMICA.
10. Program Options
- Use the text --traditional when running Octave on a remote cluster. This will help octave run the same as matlab. Otherwise this field can be left blank.
These fields once filled out play a part in creating a submit string that is used by the remote servers to correctly process the scripts. These strings are created for each data file for each scrip, and stored in the log folder. Below is an example of a sqsub submit string.
Saving
Once you have filled out all of the fields click on the | Save As | button to save. If you have multiple files open Ctrl - Click to select as many as you would like to save. These will not be combined into one file. It is recommended that the files are named to mirror their corresponding scripts as they used hand in hand. (ie. S1_Script1.htb and C1_Script1.cfg )
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use the text file created as a template to quickly edit and create new batch config files in a text editor.
