History Batch File (.htb) is a matlab code framework that contains all of the function calls but where all of the specific function inputs are [swap_strings]. These frameworks act as templates for your batches and they have their unique specifics modified by adjusting a batch configuration file, so they can rapidly be adjusted depending on the batch you are running.
Script (.m) is also matlab code but has a .m file type. It is the History Batch File framework but has been successfully string swapped according to the configuration file so that its specifics are unique for each data file. These contain the actual code that matlab will be preforming.
Pipeline refered here to a series of History Batch File scripts that are loaded in order such that the output of one is the input for the next. They can streamline what would be multiple processes into into one single batch execution.
A simple pipeline is depicted below.
This pipeline has 7 .htb scripts and thus makes a .m file for each script containing the unique information from each data file.
If we were to run 1 data files through this pipeline we would automatically generate 1 x 7 = 7 unique .m scripts and run them each in succession. Locally these would be executed in matlab one at a time.
String swapping is an essential aspect of creating an efficient pipeline. It works by writing the .htb scripts filled with strings in [ ] brackets rather than values. The script will not be able to run on its own, but when used in conjunction with a configuration file in the batch context plugin you can rapidly create unique .m scripts for each of your data .set files. You can also adjust the the script function parameters without permanently changing the code.
Below we have a .htb pipeline script that is looking for 5 swap string inputs 4 from the custom configuration file, and 1 from the dataset file name. The configuration swap string should look like this:
The swap string [batch_dfn] is one of a few special strings as it automatically takes its information from the loaded data set files.
When these .m files are executed in matlab they will perform the code that is now complete.
You can have any number of swap strings and for any type of variable, as the strings are swapped out for their values before the code is run.
Please refer to Configuration for more information on how to give your swap strings values from inside the configuration file. Additional swapping information can be found here.
Looking into the Example Further
Below is the code representing the graphic above. As you can see the [strings] have been replace with the correct 'strings' or values. The first example is switching out for a string and the second is when switching out for a numeric. Loading Data Files:
Before string swap: The strings [batch_dfn] and [in_path] are waiting to be replaced.
%%LOAD DATASET... disp('Loading set file: [batch_dfn]...');tic;EEG=pop_loadset('filename','[batch_dfn]','filepath','[in_path]');EEG=eeg_checkset(EEG);disp('TIME TO: LOAD DATASET...');toc
After string swap: You can see that the previous [batch_dfn] and [in_path] have been exchanged out for the correct unique strings.
%%LOAD DATASET... disp('Loading set file: datafile1.set...');tic;EEG=pop_loadset('filename','datafile1_mrg.set','filepath','analysis/data/1_init');EEG=eeg_checkset(EEG);disp('TIME TO: LOAD DATASET...');toc
Filling in Variables:
Before string swap: The string [low_bound] is waiting to be replaced with a number.
%%FILTER HIGH PASS...disp('High pass filtering the data...');tic;EEG=pop_eegfiltnew(EEG,,[low_bound_hz],,true,,0);EEG.setname='filtHP';EEG=eeg_checkset(EEG);disp('TIME TO: FILTER HIGH PASS...');toc
After string swap: You can see that the previous [low_bound] has been replaced with a number .
%%FILTER HIGH PASS...disp('High pass filtering the data...');tic;EEG=pop_eegfiltnew(EEG,,0.1,,true,,0);EEG.setname='filtHP';EEG=eeg_checkset(EEG);disp('TIME TO: FILTER HIGH PASS...');toc
Building Manual Steps into your Pipeline
Creating a pipeline with manual steps is also possible when running the batch locally. In the example below the batch will automatically open the file and do a plot. It will then wait for the user to type in the ms milliseconds time they would like to investigate. It then proceeds to open that time frame for inspection. Once the first plot is closed, the next .m file will run and load the next dataset. This aspect speeds up any manual steps that you need to do, such as investigating time frame, component, channel or location differences between subjects.
Please note that if you use manual steps / user input during a pipeline script you cannot run this remotely on a parallel cluster without additional modification/tooling.
% LOAD DATASET FILE...EEG=pop_loadset('filename','[batch_dfn]','filepath','[batch_dfp]');EEG=eeg_checkset(EEG);h1=figure;pop_timtopo(EEG,[-200600],,'ERP data and scalp maps of En');ms=input('TYPE IN MS TO PLOT TOPO\n');% Stops and waits for inputh2=figure;pop_topoplot(EEG,1,ms,'[batch_dfn],[1 1]',0,'electrodes','labels');% Stops and waits until window is closeduiwait(gcf);% Proceeds to next data file