Table of Contents
- Directories and Structure
Directories and Structure
The BIDS Lossless EEG pipeline is unsurprisingly, BIDS-compliant. As such, data is to be stored in a subject-wise manner and should follow the standards introduced by BIDS. See below for examples, and here for full BIDS documentation.
The goal of this page is to provide a working understanding of how the pipeline is configured. As such, when folders contain external dependencies please refer to their respective documentation.
Main Pipeline Directory
The main pipeline directory initially contains the
derivatives folders. Initialization scripts and remote configuration setup scripts can be found in the
code folder. The files that have been processed by the pipeline should end up in the
derivatives/lossless/ folder in a BIDS-compliant subject-wise folder structure (e.g.
sub-01/ses-01/eeg/sub-01_ses-01_eeg_ll.set). The initialized BIDS-compliant files should be in an identical subject-wise folder structure in the top-level directory along with the
|code||This folder should contain the source code of scripts that prepared the dataset for running on the pipeline.|
|derivatives||Derived data storage location. Can contain multiple derivatives from one set of subjects. The pipeline is one such derivative (lossless), and is contained within. See below for pipeline information.|
|sourcedata||This folder can be created by the user and should contain all the raw data files and accompanying articles, scripts, and other documentation related to acquisition. Currently, there is no standard for the folder structure within this directory.|
|sub-*||Subject folders containing the initialized BIDS-compliant data files.|
|code||Contains all necessary scripts and configuration files required to run the lossless pipeline. See section following the log section.|
|log||Contains useful debugging information and necessary intermediate pipeline files. See below section for more details.|
|sub-*||Subject folders containing the final output of the pipeline. Refer to bash config scripts to determine which .set file is which.|
Log is automatically populated with scripts when you execute the pipeline. Folders are generated for every script identified by the script that was run followed by the date and time of the scripts execution. These folders contain:
- A specific .m file for each subject data file.
- A specific .err file for each subject if an error occurred.
- A specific .log file for each subject that records the remote system command window.
- A single .sh submit script that contains the job commands that were sent to the remote servers. (Remote Jobs Only)
Note: When debugging errors, this is the most important folder as you can see what is happening by using these four files.
This folder contains the body of the pipeline and houses all scripts and configurations.
|config||This folder contains all of the configuration files and remote initialization files that are needed for the EEGLAB plugins. Configuration files will have a .cfg or .txt extension. The naming of each configuration file should mirror the script that it goes with.|
|dependencies||Contains packages that the pipeline is dependent on. All folders within should be their own git repository, with their own documentation.|
|misc||Miscellaneous support pieces such amica parameter files, EEG montage files, and event renaming files are stored here.|
|scripts||Folder containing all of the script files that make up processes behind the pipeline. These have the extension .htb and are called History Template Batch as they are executed in the batch_context extensions.|
|tools||Various script files to be used as in-house optimizers, diagnostics, or cleanup scripts for remotes.|
Note: Excluded files at this level are git repository specific and have no actual effect on the pipeline.