... | ... | @@ -26,7 +26,7 @@ A detailed description of each of the fields can be found below: |
|
|
*Required for All Jobs*
|
|
|
***
|
|
|
**1. Log Path**
|
|
|
* This is the directory location of your log folder. The log folder is used to collect the intermediate scripts created and used for each file, as well as collecting any error messages.
|
|
|
* This is the directory location of your log folder. The log folder is used to collect the intermediate scripts created and used for each file, as well as collecting any error messages.
|
|
|
:b: Default set to the *analysis/log* directory.
|
|
|
|
|
|
***
|
... | ... | @@ -44,8 +44,8 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
|
|
|
**4. Username for the Remote Host**
|
|
|
* This will be your username for your remote host.
|
|
|
:b: :warning: Default as *user_name* must be changed to your sharcnet user name.
|
|
|
* This will be your username for your remote host.
|
|
|
:b: :warning: Default as *user_name* must be changed to your sharcnet user name.
|
|
|
|
|
|
**5. Host for Compute Execution**
|
|
|
* text
|
... | ... | @@ -77,11 +77,12 @@ A detailed description of each of the fields can be found below: |
|
|
|
|
|
**11. Miscellaneous Locations**
|
|
|
* text ------------------------------------------
|
|
|
:b: Default left as ' '.
|
|
|
:b: Default left as ' '.
|
|
|
|
|
|
***
|
|
|
|
|
|
**12. System Variables**
|
|
|
* text
|
|
|
* text______________________________________________
|
|
|
:b: Default set as:
|
|
|
*sshfs [remote_user_name]@[remote_project_archive] [cd]/[mount_archive]
|
|
|
sshfs [remote_user_name]@[remote_project_work] [cd]/[mount_work]
|
... | ... | @@ -91,11 +92,13 @@ A detailed description of each of the fields can be found below: |
|
|
fusermount -u [cd]/[mount_work]*
|
|
|
|
|
|
## Saving
|
|
|
Once you have filled out all of the fields click on the |`Save As`| button to save the file. :so: Name this file to includes the project and the user (:b: *contextconfig_local.txt* and *contextconfig_remote.txt* can be found in the *analysis/support/config* directory).
|
|
|
Once you have filled out all of the fields click on the |`Save As`| button to save the file. :so: Name this file to includes the project and the user
|
|
|
(:b: *contextconfig_local.txt* and *contextconfig_remote.txt* can be found in the *analysis/support/config* directory).
|
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use this text file as a template to quickly edit and create new context config files in a text editor.
|
|
|
|
|
|
# Job configuration files (*.cfg)
|
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user. :so: Only create one local and one remote file for each script. To get started go to:
|
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user.
|
|
|
:so: Only create one local and one remote file for each script. To get started go to:
|
|
|
* *File -> Batch -> Batch Configuration*
|
|
|
![smallestcreatebatchconfig](/uploads/48f5df1610f2c60bbad7158a49b01dcd/smallestcreatebatchconfig.png)
|
|
|
|
... | ... | @@ -113,18 +116,19 @@ This is a drop down menu where you will select one of the following: |
|
|
This designates the job to be completed locally.
|
|
|
* *Sq_Sub*
|
|
|
This designates the job to be completed on the ____________server.
|
|
|
* Other options will not work at the moment :cry:
|
|
|
* Other options will not work at the moment :cry:
|
|
|
:b: The default depends on whether you loaded the local or remote file.
|
|
|
|
|
|
**2. Replace String**
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files. Refer to [Creating Scripts] for more information.
|
|
|
For Example, the BUCANL pipeline *scalpart.htb* script looks for the variable *[in_path]* which you will have created in the swap string field like so:
|
|
|
* Replace strings is one of the most important aspects of batch_context. Replace string is the communication between the config, script and data files. You will need to create a string similar to the one below, initialising variables to be used in each of the loaded data files. Refer to [Creating Scripts](https://git.sharcnet.ca/jdesjard/batch_context/wikis/script-files) for more information. Notice that each varible is identified in **[]** brackets, then proceded by a **,** comma, and finally each variable is on a separate line.
|
|
|
:b: Script one: *scalpart.htb* looks for many variables in the swap string field, one of which is *[in_path]* which can be seen is initialised in the in the swap string field by: *[in_path],analysis/data/1_init*. By default these are already complete.
|
|
|
![swapstringpopupgoodquality](/uploads/004e1ce8d684ed5c9315bc2ca34b3d41/swapstringpopupgoodquality.png)
|
|
|
|
|
|
***
|
|
|
|
|
|
The script can now load the correct files as it has the in path directory. To find out which variables the config file must set, look in the comments of the scripts.
|
|
|
To find out which variables the config file must set, look in the comments of the scripts.
|
|
|
![swpstrpart2](/uploads/5d92049f629fcc35bdcbbdf2a4193fa1/swpstrpart2.png)
|
|
|
Swap string is also helpful as you can change variables like filter strengths by simply editing the text config files, rather then changing the script code.
|
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code.
|
|
|
* **[batch_dfn]** is a special swap string as it is not collected from the config file. Instead it is the name of the current data file the script is analysing.
|
|
|
|
|
|
**3. Order**
|
... | ... | @@ -132,26 +136,34 @@ Swap string is also helpful as you can change variables like filter strengths by |
|
|
When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
|
* [1] [2] [3] [4] and so on...
|
|
|
|
|
|
* When running a job remotely you have the option to sub jobs in parallel in order to increase the jobs efficiency. The field input will be an array, where the first value is an arbitrary number, that helps denotes the location of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier. The next values in the array denote what other scripts must be complete before this one starts.
|
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parrallel [4,3]'s have been completed and [5,1] has been completed.
|
|
|
* When running a job remotely you have the option to sub jobs in parallel in order to increase the jobs efficiency. The field input will be an array, where the first value is the order number, that helps denotes the location of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next values in the array denote what other scripts must be complete before this one starts.
|
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
|
|
|
|
|
|
![smallerorderchart](/uploads/9d8f415d657d92c96647a31efe17245c/smallerorderchart.png)
|
|
|
|
|
|
***
|
|
|
|
|
|
* This order structure allows for infinite combinations and layouts of pipelines.
|
|
|
* This order structure allows for infinite combinations and layouts of pipelines. As you may have noticed the first order number does not matter, **but** remember that files are loaded into the history script field alphabetically, so you may need to adjust the files names to ensure that the jobs that start first must be higher in the call order. In our case the file name for job [5,2] must be alphabetically smaller then the file name for job [6,4,5].
|
|
|
|
|
|
**4. Job Name**
|
|
|
* text
|
|
|
* This will indicate what the job name will be. Batch swap string rules apply.
|
|
|
:b: Default as *[batch_hfn]* indicating that the job name will be the current script name.
|
|
|
|
|
|
**5. Session Init**
|
|
|
* text
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
|
|
|
:b: Remote Default is *analysis/support/comfig/octave.sesinit* a script file that initialises the octave software on the remote server.
|
|
|
|
|
|
**6. M File Init**
|
|
|
* text
|
|
|
**6. Job Init**
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
|
:b: Default is left blank ' '.
|
|
|
|
|
|
**7. Scheduler Options**
|
|
|
* text
|
|
|
**7. M File Init**
|
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
|
:b: Default is left blank ' '.
|
|
|
|
|
|
**8. Scheduler Options**
|
|
|
* This section contains a series of characters to communicate the job specifications to the remote server, such as memory and time. Click on the field and look at the information box below to see the sepecific commands. If you are running a project locally you can leave this field blank.
|
|
|
:b: Remote default is *-r 4h -o [batch_hfn]*
|
|
|
|
|
|
**8. Software**
|
|
|
This is a drop down menu where you can select one of the following:
|
... | ... | @@ -161,6 +173,7 @@ This is a drop down menu where you can select one of the following: |
|
|
|
|
|
**8. Program Options**
|
|
|
* Use the text *--traditional* when running Octave on a remote cluster. This will help octave run the same as matlab. Otherwise this field can be left blank.
|
|
|
:b: Default set as *--traditional*.
|
|
|
|
|
|
***
|
|
|
|
... | ... | |