... | ... | @@ -4,14 +4,15 @@ |
|
|
The batch_context plugin relies on two types of configuration files in order to correctly handle the job batch.
|
|
|
1. A **Context Config File**
|
|
|
This is a single config text file that specifies local and remote directories and system communications. It is unique to the *user* and the *project*. A user needs to create a new one of these for every project. :so: Name the file to includes the project, user and execution type.
|
|
|
(:b: *contextconfig_local.txt* and *contextconfig_remote* )
|
|
|
(:b: *contextconfig_local.txt* and *contextconfig_remote.txt* )
|
|
|
2. **Batch Config Files**
|
|
|
These are config text files are are unique to each pipeline *script*, but do not need to be changed according to the user or project. These handle the data file identification, pipeline order, software and computing resources of each of the scripts. :so: It is recommended that these are named to mirror their corresponding scripts as they used hand in hand.
|
|
|
(ie. *S1_Script1.htb* and *C1_Config_for_Script1.cfg* )
|
|
|
(:b: *c1_.scalpart.txt* is the config file for *s1_scalpart.htb*)
|
|
|
|
|
|
:so: It is highly suggested that you always save you config files in the support/config directory as text files. This will make them easy to edit and save outside of the batch_context user interface, as well as having an easy to load file when [Running Scripts](https://git.sharcnet.ca/jdesjard/batch_context/wikis/running-scripts).
|
|
|
:so: Save you config files as text files, this will make them easy to view, edit and save outside of the batch_context user interface.
|
|
|
:b: All the configuration files are saved as .txt and located in the *analysis/support/config* directory.
|
|
|
# Context configuration files (*.cfg)
|
|
|
A user will need a unique context config file for each project. To get started go to:
|
|
|
You will need a unique context config file for each project. To get started go to:
|
|
|
* *File -> Context Configuration*
|
|
|
![smallestcreatecontextconfig](/uploads/d1632e0d4616b94e5235e3cca0821112/smallestcreatecontextconfig.png)
|
|
|
|
... | ... | @@ -19,20 +20,23 @@ A user will need a unique context config file for each project. To get started g |
|
|
You can create a new or edit and old context config in this user interface. To load an existing config file click on the |`Load Context Config`| button. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely. All jobs must fill out the local dependency fields at the top of the block.
|
|
|
![contextedit](/uploads/936947f6c07f2c7ba518432524a44d4d/contextedit.png)
|
|
|
***
|
|
|
A detailed description of each of the fields can be found below:
|
|
|
A detailed description of each of the fields can be found below:
|
|
|
:so: Set the matlab workspace as your project_name directory that way these fields can be filled out neater.
|
|
|
***
|
|
|
*Required for All Jobs*
|
|
|
***
|
|
|
**1. Log Path**
|
|
|
* text
|
|
|
|
|
|
***
|
|
|
* This is the directory location of your log folder. The log folder is used to collect the intermediate scripts created and used for each file, as well as collecting any error messages.
|
|
|
:b: Default set to the *analysis/log* directory.
|
|
|
|
|
|
***
|
|
|
**2. Project Root Directory**
|
|
|
* text
|
|
|
* This is the directory location of your project folder.
|
|
|
:b: Default left as ' ' as you will have set the matlab workspace to your project_name directory.
|
|
|
|
|
|
**3. Dependency Root Directory**
|
|
|
* text
|
|
|
* This is the directory location of your project dependencies. These included EEGLAB and any plugins.
|
|
|
:b: Default set to the *analysis/support/dependencies* directory.
|
|
|
|
|
|
***
|
|
|
*Only Required for Remote Jobs*
|
... | ... | @@ -40,49 +44,63 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
|
**4. Username for the Remote Host**
|
|
|
* text
|
|
|
* This will be your username for your remote host.
|
|
|
:b: :warning: Default as *user_name* must be changed to your sharcnet user name.
|
|
|
|
|
|
**5. Host for Compute Execution**
|
|
|
* text
|
|
|
:b: Default set to *redfin.sharcnet.ca*
|
|
|
|
|
|
**6. Project Root Archive Address**
|
|
|
* text
|
|
|
* text --------------------------------------------
|
|
|
:b: :warning: Default of *dtn.sharnet.ca:/work/user_name/project_name* needs to be adjusted for your job.
|
|
|
|
|
|
**7. Project Root Work Address**
|
|
|
* text
|
|
|
* text ----------------------------------------------------------------
|
|
|
:b: :warning: Default of *dtn.sharnet.ca:/work/user_name/project_name* needs to be adjusted for your job.
|
|
|
|
|
|
**8. Dependency Address**
|
|
|
* text
|
|
|
* text ----------------------------------------------------------
|
|
|
:b: :warning: Default of *dtn.sharnet.ca:/work/user_name/project_name/analysis/support/dependencies* needs to be adjusted for your job.
|
|
|
|
|
|
***
|
|
|
|
|
|
**9. Archive Mount Directory**
|
|
|
* text
|
|
|
* text -------------------------------------------------------------------
|
|
|
:b: Default set to *remote_archive*
|
|
|
|
|
|
**10. Work Mount Directory**
|
|
|
* text
|
|
|
* text ----------------------------------------
|
|
|
:b: Default set to *remote_work*
|
|
|
|
|
|
***
|
|
|
|
|
|
**11. Miscellaneous Locations**
|
|
|
* text
|
|
|
|
|
|
***
|
|
|
* text ------------------------------------------
|
|
|
:b: Default left as ' '.
|
|
|
***
|
|
|
|
|
|
**12. System Variables**
|
|
|
* text
|
|
|
:b: Default set as:
|
|
|
*sshfs [remote_user_name]@[remote_project_archive] [cd]/[mount_archive]
|
|
|
sshfs [remote_user_name]@[remote_project_work] [cd]/[mount_work]
|
|
|
meld [local_project]/analysis [cd]/[mount_work]/analysis &
|
|
|
meld [local_project]/analysis [cd]/[mount_archive]/analysis [cd]/[mount_work]/analysis &
|
|
|
fusermount -u [cd]/[mount_archive]
|
|
|
fusermount -u [cd]/[mount_work]*
|
|
|
|
|
|
## Saving
|
|
|
Once you have filled out all of the fields click on the |`Save As`| button to save the file. :so: It is recommended that this file's name includes the project and the user. (ie. *adlofts_eyespilot2016_contextconfig.cfg* ) Make sure to place it in support/config directory.
|
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use the text file created as a template to quickly edit and create new context config files in a text editor.
|
|
|
Once you have filled out all of the fields click on the |`Save As`| button to save the file. :so: Name this file to includes the project and the user (:b: *contextconfig_local.txt* and *contextconfig_remote.txt* can be found in the *analysis/support/config* directory).
|
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use this text file as a template to quickly edit and create new context config files in a text editor.
|
|
|
|
|
|
# Job configuration files (*.cfg)
|
|
|
There needs to be a unique file created for each script in the pipeline. As the batch config files are not dependant on the project or the user, :so: it is recommended that you only create one local and one remote file for each script. To get started go to:
|
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user. :so: Only create one local and one remote file for each script. To get started go to:
|
|
|
* *File -> Batch -> Batch Configuration*
|
|
|
![smallestcreatebatchconfig](/uploads/48f5df1610f2c60bbad7158a49b01dcd/smallestcreatebatchconfig.png)
|
|
|
|
|
|
## Editing
|
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the |`Get Batch Config File Names`| button. This interface allows you to look at and edit multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a configuration file for it. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the |`Get Batch Config File Names`| button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a designated configuration file for it. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
|
![editconfigbuttons](/uploads/15cfea01e62161d13011548c19254ffe/editconfigbuttons.png)
|
|
|
***
|
|
|
|
... | ... | @@ -98,7 +116,7 @@ This designates the job to be completed on the ____________server. |
|
|
* Other options will not work at the moment :cry:
|
|
|
|
|
|
**2. Replace String**
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config file, the script and the data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files.
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files. Refer to [Creating Scripts] for more information.
|
|
|
For Example, the BUCANL pipeline *scalpart.htb* script looks for the variable *[in_path]* which you will have created in the swap string field like so:
|
|
|
![swapstringpopupgoodquality](/uploads/004e1ce8d684ed5c9315bc2ca34b3d41/swapstringpopupgoodquality.png)
|
|
|
|
... | ... | |