| ... | ... | @@ -82,10 +82,13 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
**12. System Variables**
|
|
|
|
* System variables are custom system commands you can quickly make using the other context configuration variables.
|
|
|
|
* These are strings that will have key strings swapped and then made available for editing before being passed to the "system" command for execution in the system terminal.
|
|
|
|
12. System Variables
|
|
|
|
System variables are custom system commands you can quickly make using the other context configuration variables.
|
|
|
|
|
|
|
|
These are strings that will have key strings swapped and then made available for editing before being passed to the "system" command for execution in the system terminal.
|
|
|
|
|
|
|
|
For more information visit the [Mounting the Project](remote-start-up#mounting-the-project) on the [Managing Remote Projects](remote-start-up) page.
|
|
|
|
|
|
|
|
:b: Default set as:
|
|
|
|
*sshfs [remote_user_name]@[remote_project_archive] [cd]/[mount_archive]
|
|
|
|
sshfs [remote_user_name]@[remote_project_work] [cd]/[mount_work]
|
| ... | ... | @@ -106,97 +109,46 @@ There needs to be a unique batch config file created for each script in the pipe |
|
|
|
|
|
|
|
|
|
|
|
## Editing
|
|
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the `| Get Batch Config File Names |` button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a designated configuration file for it. If you are loading you can adjust the order of the files by changing the names of them in the the load window. If you do not change anything they will be loaded alphabetically. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the `| Get Batch Config File Names |` button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualize the pipeline order, and ensure that every script you are using has a designated configuration file for it. If you are loading you can adjust the order of the files by changing the names of them in the the load window. If you do not change anything they will be loaded alphabetically. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
A detailed description of each of the fields can be found below:
|
|
|
|
***
|
|
|
|
|
|
|
|
**1. Execute Function**
|
|
|
|
This is a drop down menu where you will select one of the following:
|
|
|
|
* *current_base*
|
|
|
|
This specifies the job be run locally.
|
|
|
|
* *sq_sub*
|
|
|
|
This specifies the job be run on Sharcnet using the sqsub wrapper scheduler.
|
|
|
|
* *sbatch*
|
|
|
|
This specifies the job be run on a Slurm scheduler that uses sbatch. Like the new Graham or Cedar clusters at Compute Canada
|
|
|
|
|
|
|
|
:b: The default depends on whether you loaded the local or remote file.
|
|
|
|
|
|
|
|
**2. Replace String**
|
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initializing variables to be used in each of the loaded data files. Refer to [Creating Scripts](script-files) for more information. Notice that each variable is identified in **[]** brackets, then proceded by a **,** comma, and finally each variable is on a separate line.
|
|
|
|
:b: Script one: *scalpart.htb* looks for many variables in the swap string field, one of which is *[in_path]* which can be seen is initialized in the swap string field by: *[in_path],analysis/data/1_init*. By default these are already complete.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
To find out which variables the config file must set as swap strings, look in the comments at the top of the scripts.
|
|
|
|
|
|
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code. You can also use them to make decisions on options like 'yes' or 'no' to perform different sections of code within the script.
|
|
|
|
* **[batch_dfn] and [batch_dfp] ** are special swap strings that are not collected from the config file. Instead they are gathered from the loaded EEG data files. Other builtin swap strings variables are available and are recorded in [builtin swap strings.](built-in-swap-strings)
|
|
|
|
|
|
|
|
**3. Order**
|
|
|
|
* The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order. When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
|
|
* [1] [2] [3] [4] and so on...
|
|
|
|
|
|
|
|
* When running a job remotely you have the option to submit jobs in parallel in order to increase the pipeline's efficiency. The field input will be an array, where the first value is the order number, which denotes the position of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next value or values in the array denote what other scripts must be complete before this one starts.
|
|
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
|
|
|
|
|
|
|
|
Note: certain configurations are impossible due to the way we submit jobs and record their job ids. You must ensure that your job is capable of being submitted in linear order and still being valid. This means that dependents must be after parents in the linear order. A way to ensure this is to draw the jobs as a graph and number them breadth first.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
* This order structure allows for infinite combinations and layouts of pipelines. As you may have noticed the first order number does not matter, **but** remember that files are loaded into the history script field alphabetically, so you may need to adjust the files names to ensure that the jobs that start first must be higher in the call order. In our case the file name for job [5,2] must be lexicographically smaller then the file name for job [6,4,5].
|
|
|
|
|
|
|
|
**4. Job Name**
|
|
|
|
* This will indicate what the job name will be. [Batch swap string rules apply](adv-str-swap). This name is used as the .m file name that is runs the scripts on your files, as well as the name of the cluster job name. Make sure that this name is under 64 characters as Matlab can not run scripts with names that are too long.
|
|
|
|
:b: Defaults to *[batch_hfn,.,1]_[batch_dfn,.,1]* indicating that the job name will be named after the current script followed by the the name of the data file.
|
|
|
|
|
|
|
|
**5. Session Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
|
|
|
|
:b: Remote Default for Octave is *analysis/support/config/octave.sesinit* containing a few lines to unload and load the correct version of Octave.
|
|
|
|
|
|
|
|
```
|
|
|
|
module unload intel/12.1.3;
|
|
|
|
module unload octave;
|
|
|
|
module load octave/3.8.1;
|
|
|
|
```
|
|
|
|
If you are running a different program other than octave your session init script will be different. The tutorial also contains a session init file for AMICA.
|
|
|
|
|
|
|
|
**6. Job Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank. The job init field is used in the reference pipeline to add a line to the submit script before each job that adds the LD_RUN_PATH to the LD_LIBRARY_PATH before running the script.
|
|
|
|
:b: Default is left blank ' '.
|
|
|
|
|
|
|
|
**7. M File Init**
|
|
|
|
* This field will contain the location of any scripts you need to run on the remote server at the beginning of the batch of jobs. If you are running a project locally or running a binary/script you should leave this field blank.
|
|
|
|
:b: Remote Default for Matlab is *analysis/support/config/matlab.minit* containing:
|
|
|
|
|
|
|
|
:b: The reference pipeline default for Octave is *analysis/support/config/octave.minit* containing various fixes for specific versions of Octave and code to include scripts into our path.
|
|
|
|
|
|
|
|
|
|
|
|
**8. Submit Options**
|
|
|
|
|
|
|
|
|
|
|
|
The nature of the string also depends on what scheduler you are using.
|
|
|
|
|
|
|
|
**9. Software**
|
|
|
|
This is a drop down menu where you can select one of the following:
|
|
|
|
* *Matlab* : select matlab for every script when running a job locally
|
|
|
|
* *Octave* : select octave when you are running matlab scripts remotely, Octave is the free alternative found on many servers.
|
|
|
|
* *None* : select none when you are running a binary process remotely such as AMICA.
|
|
|
|
|
|
|
|
**10. Program Options**
|
|
|
|
* Use the text *--traditional* when running Octave on a remote cluster. This will help octave run the same as matlab. Otherwise this field can be left blank.
|
|
|
|
:b: Default set as *--traditional*.
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
These fields once filled out play a part in creating a *submit string* that is used by the remote servers to correctly process the scripts. These strings are created for each data file for each scrip, and stored in the log folder. Below is an example of a sqsub submit string.
|
|
|
|
|
|
|
|
***
|
|
|
|
## Batch Config Fields
|
|
|
|
|
|
|
|
* Execution Function (exec_func)
|
|
|
|
* Specifies the execution function this is generated from a list of installed execution function/scheduler plugins. See [scheduler plugins](scheduler-plugins) for more details.
|
|
|
|
* Replace String (replace_string)
|
|
|
|
* User supplied string-swap variables. Uses the format of [key],value with each set separated by new lines. Some defaults are supplied and can be seen in [builtin string-swaps](builtin-string-swaps)
|
|
|
|
* Order (order)
|
|
|
|
* Specifies the running order of the scripts for each data file. Takes the form of '['a [b ...]']' where a is the current order number, b and any following are the jobs that a is dependent on.
|
|
|
|
* Session Initialization (session_init)
|
|
|
|
* Session Initialization is run once per history file in the submit script. This contains shell code the setups up the environment on the remote for the current job. We use this on the cluster to load modules (programs) and any other environment business.
|
|
|
|
* Job Name (job_name)
|
|
|
|
* Sets the name that is submitted to the scheduler. This can be used for reporting and monitoring
|
|
|
|
* Job Initialization (job_init)
|
|
|
|
* Is inserted before each datafile in a specific submit.sh script. We use this to add LD_RUN_PATH to LD_LIBRARY_PATH before running Amica in this reference pipeline.
|
|
|
|
* M file name (mfile_name)
|
|
|
|
* Used for the Octave/Matlab script file name, also used as the log name. This allows us to make log and mfile names that are a specific as we need.
|
|
|
|
* Script Prefix / M File Init (m_init)
|
|
|
|
* Is run in each m file that batch_context generates, used to setup environment specific settings and imports in Octave/Matlab
|
|
|
|
* Submit Options / Custom scheduler options (submit_options)
|
|
|
|
* Used for generic options to the scheduler, use this if you need to input scheduler or environment specific flags to the scheduler.
|
|
|
|
* Memory (memory)
|
|
|
|
* Computed value of memory allowance for the job. Uses variables `c` and `s`for calculation. The variables `c` and `s` and the whole expression are calculated and assumed to be in 'g' or 'm' based on the last character in the field.
|
|
|
|
* i.e. `100 + 30*c + s*0.01m`
|
|
|
|
Note that `s*0.01` is used in typically reversed order to make it more clear that m is unrelated to the expression.
|
|
|
|
* Time Limit (time_limit)
|
|
|
|
* Similar to memory it is a computed value of time allowance. Uses variables `c` and `s` for calculation. The variable `c` and `s` and the whole expression are calculated and assumed to be in 's', 'm' or 'h' based on the last character in the field.
|
|
|
|
i.e. `100 + 30*c + s*0.1m`
|
|
|
|
* MPI (mpi)
|
|
|
|
* Specifies if we need to insert the special flags or strings to use mpi
|
|
|
|
* Number of tasks (num_tasks)
|
|
|
|
* Specifies the number of processes to use, this field does not specify that the processes will be packed onto on machine.
|
|
|
|
* Threads Per Task (threads_per_task)
|
|
|
|
* Specifies the number of threads per task. This will request additional cpus for each task/process.
|
|
|
|
* Software (software)
|
|
|
|
* Specifies if we are using Octave/Matlab or a script. Use Octave to set flags for Octave, leave as none if you want to use another executable file.
|
|
|
|
* Program Options (program_options)
|
|
|
|
* Arbitrary options to pass after the software field, often used for --traditional in Octave, we moved away from --traditional with out reference pipeline and instead added the specific options to the octave.minit due to bugs that exist in the current stable version of octave.
|
|
|
|
|
|
|
|
## Saving
|
|
|
|
Once you have filled out all of the fields click on the `| Save As |` button to save. If you have multiple files open *Ctrl - Click* to select as many as you would like to save. These will **not** be combined into one file. It is recommended that the files are named to mirror their corresponding scripts as they used hand in hand. (ie. *S1_Script1.htb* and *C1_Script1.cfg* )
|
| ... | ... | |
