... | @@ -98,14 +98,14 @@ Once you have filled out all of the fields click on the `| Save As |` button to |
... | @@ -98,14 +98,14 @@ Once you have filled out all of the fields click on the `| Save As |` button to |
|
(:b: *contextconfig_local.txt* and *contextconfig_remote.txt* can be found in the *analysis/support/config* directory).
|
|
(:b: *contextconfig_local.txt* and *contextconfig_remote.txt* can be found in the *analysis/support/config* directory).
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use this text file as a template to quickly edit and create new context config files in a text editor.
|
|
Now when running a job you can simply load this file and it will populate the required fields for you. You can also use this text file as a template to quickly edit and create new context config files in a text editor.
|
|
|
|
|
|
# Job configuration files (*.cfg)
|
|
# Batch Configuration Files (*.cfg)
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user.
|
|
There needs to be a unique batch config file created for each script in the pipeline. Batch config files are not dependant on the project or the user.
|
|
:so: Only create one local and one remote file for each script. To get started go to:
|
|
:so: Only create one local and one remote file for each script. To get started go to:
|
|
* *File -> Batch -> Batch Configuration*
|
|
* *File -> Batch -> Batch Configuration*
|
|
![smallestcreatebatchconfig](/uploads/48f5df1610f2c60bbad7158a49b01dcd/smallestcreatebatchconfig.png)
|
|
![smallestcreatebatchconfig](/uploads/48f5df1610f2c60bbad7158a49b01dcd/smallestcreatebatchconfig.png)
|
|
|
|
|
|
## Editing
|
|
## Editing
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the `| Get Batch Config File Names |` button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a designated configuration file for it. If you are loading you can adjust the order of the files by changing the names of them in the the load window. If you dont change anything they will be loaded alphabetically. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
You can create a new or edit and old batch config in this user interface. To load existing batch files click on the `| Get Batch Config File Names |` button. This interface allows you to look at, and edit, multiple config files at once. This helps you visualise the pipeline order, and ensure that every script you are using has a designated configuration file for it. If you are loading you can adjust the order of the files by changing the names of them in the the load window. If you do not change anything they will be loaded alphabetically. You will need to fill out a number of fields depending on whether you will be running the job locally or remotely.
|
|
![editconfigbuttons](/uploads/15cfea01e62161d13011548c19254ffe/editconfigbuttons.png)
|
|
![editconfigbuttons](/uploads/15cfea01e62161d13011548c19254ffe/editconfigbuttons.png)
|
|
***
|
|
***
|
|
|
|
|
... | @@ -117,7 +117,7 @@ This is a drop down menu where you will select one of the following: |
... | @@ -117,7 +117,7 @@ This is a drop down menu where you will select one of the following: |
|
* *Current Base*
|
|
* *Current Base*
|
|
This designates the job to be completed locally.
|
|
This designates the job to be completed locally.
|
|
* *Sq_Sub*
|
|
* *Sq_Sub*
|
|
This designates the job to be completed on the sharcnet.
|
|
This designates the job to be completed on the Sharcnet.
|
|
* Other schedulers are currently being developed including qSub.
|
|
* Other schedulers are currently being developed including qSub.
|
|
:b: The default depends on whether you loaded the local or remote file.
|
|
:b: The default depends on whether you loaded the local or remote file.
|
|
|
|
|
... | @@ -128,16 +128,16 @@ This designates the job to be completed on the sharcnet. |
... | @@ -128,16 +128,16 @@ This designates the job to be completed on the sharcnet. |
|
|
|
|
|
***
|
|
***
|
|
|
|
|
|
To find out which variables the config file must set, look in the comments of the scripts.
|
|
To find out which variables the config file must setas swap strings, look in the comments at the top of the scripts.
|
|
![swpstrpart2](/uploads/5d92049f629fcc35bdcbbdf2a4193fa1/swpstrpart2.png)
|
|
![swpstrpart2](/uploads/5d92049f629fcc35bdcbbdf2a4193fa1/swpstrpart2.png)
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code.
|
|
Swap string is also helpful as you can change variables like filtered frequencies by simply editing the text config files, rather then changing the script code.
|
|
* **[batch_dfn]** is a special swap string as it is not collected from the config file. Instead it is the name of the current data file the script is analysing.
|
|
* **[batch_dfn] and [batch_dfp] []** are special swap strings that are not collected from the config file. Instead it is the name of the current data file and the current script.
|
|
|
|
|
|
**3. Order**
|
|
**3. Order**
|
|
* The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order.When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
* The order of the script is an important part of creating a script pipeline. If you leave this section empty your scripts will be run in a default linear order same as the local job order.When running a job locally your computer will focus on one task at a time so your script order will be:
|
|
* [1] [2] [3] [4] and so on...
|
|
* [1] [2] [3] [4] and so on...
|
|
|
|
|
|
* When running a job remotely you have the option to sub jobs in parallel in order to increase the jobs efficiency. The field input will be an array, where the first value is the order number, that helps denotes the location of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next values in the array denote what other scripts must be complete before this one starts.
|
|
* When running a job remotely you have the option to submit jobs in parallel in order to increase the pipeline's efficiency. The field input will be an array, where the first value is the order number, which denotes the position of the script in the pipeline. Where number 1 is first and the largest number is last. Two scripts can have the same identifier if they are completed in parallel. The next value or values in the array denote what other scripts must be complete before this one starts.
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
|
|
In the example below the first script is number [1]. After it is complete both script [2,1] and [5,1] start. Once [2,1] is finished [3,2] starts. Once [3,2] finishes both parallel processes [4,3] start. By this time our longer script [5,1] will finish. Script [6,4,5] will wait until both parallel [4,3]'s have been completed and [5,1] has been completed.
|
|
|
|
|
|
![smallerorderchart](/uploads/9d8f415d657d92c96647a31efe17245c/smallerorderchart.png)
|
|
![smallerorderchart](/uploads/9d8f415d657d92c96647a31efe17245c/smallerorderchart.png)
|
... | @@ -148,31 +148,44 @@ In the example below the first script is number [1]. After it is complete both s |
... | @@ -148,31 +148,44 @@ In the example below the first script is number [1]. After it is complete both s |
|
|
|
|
|
**4. Job Name**
|
|
**4. Job Name**
|
|
* This will indicate what the job name will be. Batch swap string rules apply.
|
|
* This will indicate what the job name will be. Batch swap string rules apply.
|
|
:b: Default as *[batch_hfn]* indicating that the job name will be the current script name.
|
|
:b: Default as *[batch_hfn,.,1]_[batch_dfn,.,1]* indicating that the job name will be named after the current script followed by the subject file number.
|
|
|
|
|
|
**5. Session Init**
|
|
**5. Session Init**
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the session. If you are running a project locally you can leave this field blank.
|
|
:b: Remote Default is *analysis/support/config/octave.sesinit* a script file that initialises the octave software on the remote server.
|
|
:b: Remote Default for Octave is *analysis/support/config/octave.sesinit* containing:
|
|
|
|
|
|
|
|
```
|
|
|
|
module unload intel/12.1.3;
|
|
|
|
module unload octave;
|
|
|
|
module load octave/3.8.1;
|
|
|
|
```
|
|
|
|
|
|
**6. Job Init**
|
|
**6. Job Init**
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
:b: Default is left blank ' '.
|
|
:b: Default is left blank ' '.
|
|
|
|
|
|
**7. M File Init**
|
|
**7. M File Init**
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally you can leave this field blank.
|
|
* This field will contain the location of any scripts you need to run on the remote sever at the beginning of the job. If you are running a project locally or running a binary file you can leave this field blank.
|
|
:b: Default is left blank ' '.
|
|
:b: Remote Default for Octave is *analysis/support/config/octave.minit* containing:
|
|
|
|
|
|
|
|
```
|
|
|
|
warning off;
|
|
|
|
pkg load signal;
|
|
|
|
addpath(genpath('[:,1,remote_dependency]'));
|
|
|
|
cd '[:,1,remote_project_work]';
|
|
|
|
```
|
|
|
|
|
|
**8. Scheduler Options**
|
|
**8. Scheduler Options**
|
|
* This section contains a series of characters to communicate the job specifications to the remote server, such as memory and time. Click on the field and look at the information box below to see the specific commands. If you are running a project locally you can leave this field blank.
|
|
* This section contains a series of characters to communicate the job specifications to the remote server, such as memory and time. Click on the field and look at the information box below to see the specific commands. If you are running a project locally you can leave this field blank.
|
|
:b: Remote default is *-r 4h -o [batch_hfn]*
|
|
:b: Remote default is *-r 4h --mpp 6G* for Octave Scripts and *-r 4h -q -mpi 1G* for amica files.
|
|
|
|
|
|
**8. Software**
|
|
**9. Software**
|
|
This is a drop down menu where you can select one of the following:
|
|
This is a drop down menu where you can select one of the following:
|
|
* *Matlab* : select matlab for every script when running a job locally
|
|
* *Matlab* : select matlab for every script when running a job locally
|
|
* *Octave* : select octave when you are running matlab scripts remotely, octave is a free software than can be downloaded onto the remote system.
|
|
* *Octave* : select octave when you are running matlab scripts remotely, Octave is the free alternative found on many servers.
|
|
* *None* : select none when you are running a binary process remotely such as AMICA.
|
|
* *None* : select none when you are running a binary process remotely such as AMICA.
|
|
|
|
|
|
**8. Program Options**
|
|
**10. Program Options**
|
|
* Use the text *--traditional* when running Octave on a remote cluster. This will help octave run the same as matlab. Otherwise this field can be left blank.
|
|
* Use the text *--traditional* when running Octave on a remote cluster. This will help octave run the same as matlab. Otherwise this field can be left blank.
|
|
:b: Default set as *--traditional*.
|
|
:b: Default set as *--traditional*.
|
|
|
|
|
... | | ... | |