... | @@ -9,11 +9,12 @@ After being presented with the first GUI, navigate to File > Batch > Run history |
... | @@ -9,11 +9,12 @@ After being presented with the first GUI, navigate to File > Batch > Run history |
|
|
|
|
|
This new GUI combines both context config and batch config into one interface.
|
|
This new GUI combines both context config and batch config into one interface.
|
|
|
|
|
|
For a job we must specify the following four things:
|
|
For a job we must specify the following five things:
|
|
1. Context configuration
|
|
1. Context configuration
|
|
2. Batch configuration
|
|
2. Batch configuration
|
|
3. Script files
|
|
3. History and script files
|
|
4. Data files
|
|
4. Data files
|
|
|
|
5. Submission
|
|
|
|
|
|
Instructions on how to complete each step in order and brief explanations are found below.
|
|
Instructions on how to complete each step in order and brief explanations are found below.
|
|
|
|
|
... | @@ -25,7 +26,7 @@ As specified in the context [tutorial](configuring-context-config) page, this co |
... | @@ -25,7 +26,7 @@ As specified in the context [tutorial](configuring-context-config) page, this co |
|
Begin by clicking the **Load context configuration** button on the GUI. You should be presented with a file chooser. Either navigate to your personally saved configuration or navigate to `/bids/derivatives/lossless/code/config` and select `contextconfig.cfg`. This will load some default options assuming the graham cluster's specifications. From this point you will need to modify the options as they only contain sample paths. For an in depth explanation of what needs to be modified, see the [tutorial](configuring-context-config).
|
|
Begin by clicking the **Load context configuration** button on the GUI. You should be presented with a file chooser. Either navigate to your personally saved configuration or navigate to `/bids/derivatives/lossless/code/config` and select `contextconfig.cfg`. This will load some default options assuming the graham cluster's specifications. From this point you will need to modify the options as they only contain sample paths. For an in depth explanation of what needs to be modified, see the [tutorial](configuring-context-config).
|
|
|
|
|
|
# 2. Batch Configuration
|
|
# 2. Batch Configuration
|
|
The next step is to load the configuration files that specify how the scripts will interact with the cluster and its scheduler. First, click **Load batch config**. A small GUI should be presented. Click **Batch configuration file** to open a file chooser. Either navigate to your own personalized scripts, or navigate to `/bids/derivatives/lossless/code/config/remote-sbatch`. Select all files beginning with the **c** prefix and press **open**. The lower text area of the small GUI should now have entries in it corresponding to the selected filed. Press **ok** to load these configurations into Matlab.
|
|
The next step is to load the configuration files that specify how the scripts will interact with the cluster and its scheduler. First, click **Load batch config**. A small GUI should be presented. Click **Batch configuration file** to open a file chooser. Either navigate to your own personalized scripts, or navigate to `/bids/derivatives/lossless/code/config/remote-sbatch`. Select all files beginning with the **c** prefix and press **open**. Selection of multiple files can typically be done by holding down the ctrl key. The lower text area of the small GUI should now have entries in it corresponding to the selected filed. Press **ok** to load these configurations into Matlab.
|
|
|
|
|
|
The batch configuration area should now be filled with texts.
|
|
The batch configuration area should now be filled with texts.
|
|
|
|
|
... | @@ -33,47 +34,34 @@ The batch configuration area should now be filled with texts. |
... | @@ -33,47 +34,34 @@ The batch configuration area should now be filled with texts. |
|
|
|
|
|
**NOTE:** These configurations assume your testing session is of average length. Increasing various parameters can cause a **major** increase in queue time. See the batch configuration [page](tuning-pipeline-configs) on how to customize memory and time parameters safely.
|
|
**NOTE:** These configurations assume your testing session is of average length. Increasing various parameters can cause a **major** increase in queue time. See the batch configuration [page](tuning-pipeline-configs) on how to customize memory and time parameters safely.
|
|
|
|
|
|
# Selecting the correct history files
|
|
# 3. History and script files
|
|
Once you have your context configuration loaded clicking "History file" will open you directly to your scripts directory.
|
|
These files specify what the pipeline actually does and how to accomplish it. For an in depth breakdown of what each section of the pipeline does, see [here](pipeline-scripts).
|
|
|
|
|
|
Here is an example of what you may see:
|
|
To select the pipeline's history files, click **History file** and navigate to `/bids/derivatives/lossless/code/scripts/` and select all files beginning with **s** and press **open**. Selection of multiple files can be done by holding down the ctrl key.
|
|
|
|
|
|
![htb_example](/uploads/82857a1eebce1d279d1daa9510079276/htb_example.png)
|
|
# 4. Data files
|
|
|
|
|
|
The scripts have prefixes of s followed by the number of the pipeline step. We can select multiple steps of the pipeline by ctrl clicking.
|
|
Select your data files by clicking "Data files or ESS Capsule" Navigate into `1_init`.
|
|
|
|
|
|
Normally we don't need to run the whole pipeline we can just run half the pipeline so long as the data is reasonably clean.
|
|
|
|
|
|
|
|
**Full pipeline run**
|
|
|
|
Select 1-5, 7-11, 6
|
|
|
|
|
|
|
|
**Data side pipeline run**
|
|
|
|
Select 1-6
|
|
|
|
|
|
|
|
# Selecting the correct batch configuration files
|
|
|
|
The configuration files specify the execution function or scheduler used by the remote server. You must select the correct type of configuration files given your chosen remote host.
|
|
|
|
|
|
|
|
Click the "Load batch config" button and in the next interface select "Batch configuration file" select remote-sbatch if you are using an sbatch scheduler or remote-sqsub if you are using a sqsub based scheduler.
|
|
|
|
|
|
|
|
Note: For Sharcnet/Compute Canada users `sqsub` is used in the legacy systems like kraken, orca, and saw. `sbatch` is used in Graham and Cedar.
|
|
Tip: If you are using eeglab files (.set) you can change File Name: at the bottom from `*.*` to `*.set`in order to reduce the clutter in the folder.
|
|
|
|
|
|
Now in these folders you need to select the corresponding config files that match the files in the [previous step](#selecting-the-correct-history-files)
|
|
If you are running only a later section of the pipeline you should still only select .set files as the rest of our scripts are designed to find the correct process file from the preproc directory.
|
|
|
|
|
|
We will talk about [tuning these config files in another page.](tuning-pipeline-configs)
|
|
# 5. Submission
|
|
|
|
|
|
# Selecting your data files
|
|
At the bottom of GUI is a drop down menu allowing for the choice of remote connection type.
|
|
|
|
|
|
Select your data files by clicking "Data files or ESS Capsule" Navigate into `1_init`.
|
|
If system is chosen, you will need to enter your password into the Matlab command window. Note that the password will be hidden from view and is secure. This option will also take advantage of RSA keys.
|
|
|
|
|
|
Tip: If you are using eeglab files (.set) you can change File Name: at the bottom from `*.*` to `*.set`in order to reduce the clutter in the folder.
|
|
If ssh2 is picked, a small GUI allowing password entry will appear. Enter your password and **click** ok.
|
|
|
|
|
|
If you are running only a later section of the pipeline you should still only select .set files as the rest of our scripts are designed to find the correct process file from the preproc directory.
|
|
Do not select none.
|
|
|
|
|
|
# Submission
|
|
Submission is then finally done by pressing **Ok**.
|
|
|
|
|
|
Submission is done by pressing "Ok"
|
|
The Matlab Command Window will detail the process of submitting the job and display any errors the occur.
|
|
|
|
|
|
The Matlab Command Window will detail the process of submitting the job and display any errors the occur. We have spent a significant amount of time improving the logging of the pipeline so if you choose to submit a Issue about your problem please include a sizable chunk of logs as an attachment.
|
|
Below is an example of a completely filled in GUI with two participants.
|
|
|
|
|
|
# Pipeline Diagram
|
|
# Pipeline Diagram
|
|
![pipe_w_init](/uploads/9ae98545cbe1afbf43cedde93e913aaf/BUCANL_pipeline.png)
|
|
![pipe_w_init](/uploads/9ae98545cbe1afbf43cedde93e913aaf/BUCANL_pipeline.png)
|
... | | ... | |