|
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
# Overview
|
|
|
|
This script takes in raw data files and preforms all of the necessary steps to prepare the raw files for the [eeg_pipe_asr_amica pipeline](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/home) (this pipeline). This script was designed to be very versatile for any level of automation and knowledge base of configuration scripts.
|
|
|
|
This script is essentially an attemp to create a standardized import function for the BUCANL lossless pipeline. It takes in raw data files and preforms all of the necessary steps to prepare the raw files for the [eeg_pipe_asr_amica pipeline](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/home) (this pipeline). This script was designed to be very versatile for any level of automation and knowledge base of configuration scripts.
|
|
|
|
|
|
|
|
Raw files can vary greatly depending on the modality used to collect the data, while init files going into the pipeline need to to prepared just right for smooth processing. This script current can prepare data from two different styles:
|
|
|
|
Raw files can vary greatly depending on the modality used to collect the data, while init files going into the pipeline need to to prepared just right for smooth processing during the initial stages and for ICA. This script currently can prepare data from two different styles:
|
|
|
|
|
|
|
|
**1. ESS Capsule**
|
|
|
|
To load a capsule make sure it it is at the level 1 stage, and load the study_description.xml file into the [batch config](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/home) file upload. The metadata stored in the capsule will be automatically accessed in the script. Using the [file structure](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/directories) for this pipeline is best as the saved files will be directed into the *analysis/data/1_init* folder when they are complete.
|
|
|
|
To load a capsule make sure it it is at the level 1 stage, and load the study_description.xml file into the [batch config](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/home) under the file upload section. The metadata stored in the capsule will be automatically accessed in the script and be used to find all the files the study contains. Using the [file structure](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/directories) designed for this pipeline is best as the saved files will be directed into the *analysis/data/1_init* folder when they are complete.
|
|
|
|
|
|
|
|
:so: We recommend storing the *Main_Study_Folder* inside of *analysis/data/0_raw* making the study_description.xml available in *analysis/data/0_raw/Main_Study_Folder/level_1/study_description.xml*
|
|
|
|
:so: We recommend storing the *Main_Study_Folder* inside of *analysis/data/0_raw* making the study_description.xml available in *analysis/data/0_raw/Main_Study_Folder/level_1/study_description.xml*
|
|
|
|
|
|
|
|
**2. Raw Files**
|
|
|
|
You can also load raw files into the [batch config](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/home) file upload. Using the [file structure](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/directories) for this pipeline is best as the saved files will be directed into the *analysis/data/1_init* folder when they are complete. This means uploading your raw files from the *analysis/data/0_raw* folder.
|
|
|
|
You can also load raw files into the [batch config](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/home) file upload as normal. Using the [file structure](https://git.sharcnet.ca/bucanl_pipelines/eeg_pipe_asr_amica/wikis/directories) for this pipeline is best as the saved files will be directed into the *analysis/data/1_init* folder when they are complete. This means uploading your raw files from the *analysis/data/0_raw* folder.
|
|
|
|
|
|
|
|
:so: On a side note make sure your EEGLAB work space is set to the *eeglab_asr_amica* folder so that *analysis* is the first folder on your path.
|
|
|
|
|
|
|
|
This script takes the raw data file through 3 process:
|
|
|
|
1. Loading the File
|
|
|
|
2. Warping Channel locations to MNI head
|
|
|
|
3. Creating events and marks to indicate In/Out Task Time
|
|
|
|
1. Loading the File (LOAD)
|
|
|
|
2. Warping Channel locations to MNI head (WARP)
|
|
|
|
3. Creating events and marks to indicate In/Out Task Time (EVENTS)
|
|
|
|
|
|
|
|
The picture below shows how each section develops the file.
|
|
|
|
|
| ... | ... | @@ -25,9 +26,15 @@ The picture below shows how each section develops the file. |
|
|
|
|
|
|
|
|
|
|
|
# Configuration and GUI
|
|
|
|
This script will only produce GUI's if it is missing a necessary piece of information. This means that if you carefully fill out the configuration file then you can run the complete pipeline autonomously. On the other hand if you are not comfortable with the configuration files this file can be run with no configuration and will use GUI's to prompt you for the information as needed. This process is more time consuming but will help you understand what is needed in the configuration, and you can always run a smaller batch first to sort out these details. Refer to the table below to plan your script usage.
|
|
|
|
This script will only produce GUI's if it is missing a necessary piece of information. This means that if you carefully fill out the configuration file then you can run the complete pipeline automatically! On the other hand if you are:
|
|
|
|
* Not comfortable with the configuration files
|
|
|
|
* Unsure of some of the this file information
|
|
|
|
* Missing clear events to mark the start and stop of recording
|
|
|
|
Then the script can be run with less or no configuration setup and will use GUI's to prompt you for the information as needed. This process is more time consuming but will help you understand what is needed in the configuration.
|
|
|
|
|
|
|
|
| Method | Varibles | Run Style |
|
|
|
|
:so: You can always run a smaller batch first to sort out these details then run the whole set once the config is complete. Refer to the table below to help plan your script use.
|
|
|
|
|
|
|
|
| Method | Variables | Run Style |
|
|
|
|
|:------:|:-------------------------:|:-----------:|
|
|
|
|
| ESS | | |
|
|
|
|
| 1 | ESS and Config | Auto |
|
| ... | ... | @@ -41,10 +48,10 @@ This script will only produce GUI's if it is missing a necessary piece of inform |
|
|
|
# Mandatory Variables
|
|
|
|
If you want to run this script fully automatic or on a remote cluster you need to ensure that each of the mandatory variables described below are denoted in either the batch config or the ESS capsule metadata.
|
|
|
|
|
|
|
|
**ESS LoadType**
|
|
|
|
**ESS LoadType - Automatic**
|
|
|
|
* Mandatory variables are split between the ESS capsule metadata and the config file. Do not worry about the XML variables in this case as they are automatically found by reading the ESS study_description.xml file that you loaded when batching.
|
|
|
|
|
|
|
|
**OTHER LoadType**
|
|
|
|
**OTHER LoadType - Automatic**
|
|
|
|
* All of the mandatory variables need to be in the config file.
|
|
|
|
|
|
|
|
The table below indicates with an **x** the methods in which you can describe each variables depending on your load type. You will see that all variables can be described in the Config or the ESS metadata. If those are not filled out, then some variable allow you to enter the values later, while others will revert to defaults.
|
| ... | ... | @@ -66,10 +73,11 @@ The table below indicates with an **x** the methods in which you can describe ea |
|
|
|
| Inevents | | x | x | x | x | Events that represent HED tag 'Custom/Marks/InEvent' |
|
|
|
|
|
|
|
|
# Loading Variables
|
|
|
|
This next section contains a short description on each of the variables and examples of what you might want to put in the config. If you are using the configs that come with this git project, you will have 2 templates one for ESS and one for OTHER.
|
|
|
|
|
|
|
|
**loadType**
|
|
|
|
The load type refers to whether you are starting from an ESS capsule or from a list of raw files. Use
|
|
|
|
```[loadType],ESS``` or ``` [loadType],OTHER ``` in the config replace_string. If you leave config empty the following GUI will be displayed. Simply checkbox your answer.
|
|
|
|
```[loadType],ESS``` or ``` [loadType],OTHER ``` in the config replace_string. If you leave config empty the following GUI will be displayed. Simply checkbox your answer.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
| ... | ... | @@ -91,24 +99,24 @@ Use ```[oneloc],0``` or ``` [oneloc],1 ``` in the config replace_string. If you |
|
|
|
***
|
|
|
|
|
|
|
|
**locs_file_name**
|
|
|
|
This variable is a continuation of your **oneloc** answer above. ESS capsules use the built in metadata and the location file names will be found automatically.
|
|
|
|
This variable is a continuation of your **oneloc** answer above. ESS capsules use the built in metadata and the location file names will be found automatically. You will have to change this if you are using OTHER.
|
|
|
|
* If you chose **oneloc 0** , use a naming convention in the config such as ```[batch_dfn,.,-1]_locs.elc``` see [swap strings](https://git.sharcnet.ca/bucanl_eeglab_extensions/batch_context/wikis/script-files#swap-string) for more information.
|
|
|
|
|
|
|
|
* If you chose **oneloc 1** , write the path and file name of the montage file such as ```analysis/support/misc/BioSemi_BUCANL_7Eyes.sfp```.
|
|
|
|
|
|
|
|
If you leave config empty the following GUI will be displayed. Simply write the same thing as above.
|
|
|
|
If you leave the config empty the following GUI will be displayed. Simply type the same thing as above.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
**srate**
|
|
|
|
State the sampling rate of the system that you used. ESS capsules will find this data automatically. Common rates are 280, 500, and 512 point per second. An example would be ```[srate],512``` in the config replace_string. If you leave config empty the following GUI will be displayed. Simply type in a number.
|
|
|
|
State the sampling rate of the system that you used. ESS capsules will find this data automatically and nothing is needed in the config. Common rates are 280, 500, and 512 point per second. An example would be ```[srate],512``` in the config replace_string. If you leave config empty the following GUI will be displayed. Simply type in a number.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
**NotEEG_chans**
|
|
|
|
These are the channel indices that do not contain EEG information this includes unplugged channels **not including fiducials**. ESS will by default not load any channels that do not contain EEG, but if you are using config you will have to specify these. For example our lab never uses the EXG8 channel of our BioSemi, this corresponds to using ```[NotEEG_chans],[140]``` to remove the very last channel. This item is one of the few that does not contain a backup GUI as an empty config is a acceptable answer. The default will remove no channels.
|
|
|
|
These are the channel indices that do not contain EEG information this includes unplugged channels but will **not include fiducials**. ESS will by default not load any channels that do not contain EEG. For example our lab never uses the EXG8 channel of our BioSemi, this corresponds to using ```[NotEEG_chans],[140]``` to remove the very last channel. This item is one of the few that does not contain a backup GUI as an empty config is a acceptable answer and this default will remove no channels.
|
|
|
|
|
|
|
|
Note that the channels that are purged are: **NotEEG_chans + Fid_chans(see below)**, so channels should only be included in one or the other.
|
|
|
|
|
| ... | ... | @@ -117,28 +125,28 @@ Note that the channels that are purged are: **NotEEG_chans + Fid_chans(see below |
|
|
|
# Warping Variables
|
|
|
|
|
|
|
|
**Fid_chans**
|
|
|
|
This item is very similar to **NotEEG_chans** but the indices you are listing in this case are the fiducials. ESS capsules denote this information by specifying the channel *Type* as *FID*. ESS will automatically use these when warping , but if you are using config you will have to specify these. For example our lab never uses the LPA,NZ and RPA fiducials recorded as the first three, using```[Fid_chans],[1 2 3]``` to set these as the landmarks for warping and then removing them from EEG.data afterwards. If you leave config empty the following GUI will be displayed. Simply type in a indices of the channels, you do not have to do [ ] brackets.
|
|
|
|
This item is very similar to **NotEEG_chans** but is a list of the indices of the fiducials. ESS capsules denote this information by specifying the channel *Type* as *FID* and will automatically use these when warping , but if you are using config you will have to specify these. For example our lab uses the first three channel locations as the LPA, NZ and RPA fiducials, using```[Fid_chans],[1 2 3]``` will set these as the landmarks for warping and then remove them from EEG.data afterwards. If you the leave config empty the following GUI will be displayed. Simply type in a indices of the channels, you do not have to do [ ] brackets in the GUI as it is already expecting a matrix. For example type```1 2 3```.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
**ref_loc_file**
|
|
|
|
The reference Location file is the montage that you want to warp your channel locations to. For this pipeline you will be warping to the MNI head. The config should point to the location file. The config should be```[ref_loc_file],analysis/support/misc/standard_1020.elc```. If the config is empty it will default back to this file as well. In the future this may want to be changed and thus was made a string_swap variable.
|
|
|
|
The reference location file is the montage that you want to warp your channel locations to. For this pipeline you will be warping to the MNI head. The config should point to the location file. The config should be```[ref_loc_file],analysis/support/misc/standard_1020.elc```. If the config is empty it will default back to this file as well. In the future this may want to be changed and thus was made a string_swap variable.
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
**tMatrix**
|
|
|
|
Warping or transforming, ensures that the location file that you loaded aligns with the standard MNI head model that is used later on in the pipeline for referencing. Both ESS capsules and Other sets need to provide this information through the config or through the interactive GUI. Essentially the pipeline needs a transformation matrix of 9 values to expand,translate and rotate in each dimension to get your locations to our positions. If you are using a standard transformation matrix for all of your subjects you can place the matrix in the config, for example ```[tMatrix],[ 0 , -23 , -50 , 0 , 0 , -240.3 , 1100 , 1100 , 1100 ] ```. If you you would like to do exact transformations for each file you will have to go through the GUIs. The first GUI allows you to input a file specific transformation matrix for that subject if you know it. If you do not know what it is yet, then click the manual checkbox to open the EEGLAB co-register GUI.
|
|
|
|
Warping and transforming, ensures that the location file that you loaded aligns with the standard MNI head model that is used later on in the pipeline for referencing. Both ESS capsules and OTHER sets need to provide this information through the config or through the interactive GUI. Essentially the pipeline needs a transformation matrix of 9 values to expand,translate and rotate in each dimension to get your locations to our positions. If you are using a standard transformation matrix for all of your subjects you can place the matrix in the config, for example ```[tMatrix],[ 0 , -23 , -50 , 0 , 0 , -240.3 , 1100 , 1100 , 1100 ] ```. If you you would like to do exact transformations for each file you will have to go through the GUIs. The first GUI allows you to input a file specific transformation matrix for that subject if you know it. If you do not know what it is yet, then click the manual checkbox to open the EEGLAB co-register GUI.
|
|
|
|
|
|
|
|
Adjust the matrix that is displayed at the bottom until you find the right fit and press ```[ OK ]```.
|
|
|
|
|
|
|
|
:so: After doing you first file by manually warping, you will be able to see the transformation matrix that will be used. You can use this matrix to help you get a *head start* ( :smile: ) on the next file, or use this in the config to morph all of the files the same way.
|
|
|
|
:so: After doing you first file by manually warping, you will be able to see the transformation matrix that will be used. You can use this matrix to help you get a *head start* ( :smile: ) on the next file, or use this in the config to morph all of the files the same way for a faster approximation.
|
|
|
|
|
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
# Event and Marks Variables
|
|
|
|
The pipeline proceeding this script needs to be contain specific events that indicate in-task time and out of task time. This is because ICA analysis is sensitive to messy task time that could contain movement artifacts or non-stationary so we only want to submit the essential data to ICA. To do this the out of task time is flagged using our Out/On/In events to [create removal marks](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/marks). This system uses the [Vised Marks plugin](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/home) to do the data flagging.
|
|
|
|
The pipeline proceeding this script needs to be contain specific events that indicate in-task time and out of task time. This is because ICA analysis is sensitive to messy task time that could contain movement or non-stationary so we only want to submit the essential data to ICA. To do this the out of task time is flagged using our Out/On/In events to [create removal marks](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/marks). This system uses the [Vised Marks plugin](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/home) to do the data flagging.
|
|
|
|
|
|
|
|
:so: If you are building your own ESS capsule or you are lucky these events can be built into the capsule already. Use the HED tags:
|
|
|
|
* Custom/Marks/RecStart
|
| ... | ... | @@ -150,17 +158,16 @@ These tags will be identified and used as your On/Off/In events respectively. |
|
|
|
**Onevents/Offevents/Inevents**
|
|
|
|
To tell the script where to flag, you need to give it Onevents and Offevents **OR** Inevents.
|
|
|
|
|
|
|
|
* **In events** are events that need to be surrounded in In task time. Any time that is not within 3 sec of an Inevent is considered out of task. This is ideal for studies with many repeated tasks that are within a few seconds of each other.
|
|
|
|
* **In events** are events that need to be surrounded within In task time. Any time that is not within 3 sec of an Inevent is considered out of task. This is ideal for studies with many repeated tasks that are within a few seconds of each other.
|
|
|
|
* Config Example: A Go/NoGo task with 1 as Go and 2 as NoGo with one or the other every 2 seconds. This task has many short breaks between the sessions. Inevents would be ideal as you could specify ```[Inevent], 1 2```and that would ensure that all of the Go and NoGo tasks are included for ICA, while the breaks and lead up times would be removed.
|
|
|
|
|
|
|
|
|
|
|
|
* **On/Off events** can also be thought of as *Record On* and *Record Off*. They are specified in pairs to act as boundaries for in task time. Any time on the outside of these events will be marked black and out of task.
|
|
|
|
|
|
|
|
* Config Example: A study where participants are asked to clear their mind for 5 min. The start of this task is event 11 and the end is 22. In this case Inevents would not work, as they would only gather 6sec of the task. Instead if you specify ```[Onevent], 11```and ```[Offevent], 22``` that would ensure that the whole 5min is set as in task recording. Notice how 11 and 22 are a paired set. If 11 was always on but off alternated between 22 and 33, then you would specify the following in the config:```[Onevent], 11 11```and ```[Offevent], 22 33```. 11 is recorded twice but each time it has a different stop indicator. You also have to be weary about the number of events of each type you have in the file. **For every On there must be an Off**, since the marking function uses intervals.
|
|
|
|
|
|
|
|
If you do not have the prebuilt ESS and you don not fill out the config file you will prompted with the following GUI that asks for the same information.
|
|
|
|
Depending on which method you are using simply type in the event numbers separated by spaces. If your study does not contain events that suit either of these methods you can manually add *Recording Start* and *Recording Stop* marks. To do this select the *Manual Mode* checkbox.
|
|
|
|
* Config Example: A study where participants are asked to clear their mind for 5 min. The start of this task is event 11 and the end is 22. In this case Inevents would not work, as they would only gather 12 sec of the task,6 at the start and 6 at the end on either side of the start and stop events. Instead if you specify the pair ```[Onevent], 11```and ```[Offevent], 22``` that would ensure that the whole 5min is set as in task recording. Notice how 11 and 22 are a paired set. If 11 was always on but off alternated between 22 and 33, then you would specify the following in the config:```[Onevent], 11 11```and ```[Offevent], 22 33```. 11 is recorded twice but each time it has a different stop indicator. You also have to be weary about the number of events of each type you have in the file. **For every On there must be an Off**, since the marking function uses intervals.
|
|
|
|
|
|
|
|
If you do not have a prebuilt ESS and you don not fill out the config file you will prompted with the following GUI that asks for the same information.
|
|
|
|
Depending on which method you are using simply type in the event numbers separated by spaces. For example ```22 33```. If your study does not contain events that suit either of these methods you can manually add *Recording Start* and *Recording Stop* marks. To do this select the *Manual Mode* checkbox.
|
|
|
|
|
|
|
|
|
|
|
|
If you selected *Manual Mode* then this little GUI will pop up. This GUI simply warns you that you are entering the [Vised-Marks editor](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/manual-vised-edit) and that any annotations you add here will be added to the [marks structure](https://git.sharcnet.ca/bucanl_eeglab_extensions/vised_marks/wikis/marks) permanently. The two commands you will be using are:
|
| ... | ... | @@ -176,7 +183,7 @@ See the example below where we placed events before and after each block of task |
|
|
|
|
|
|
|
***
|
|
|
|
|
|
|
|
After you click *Update Marks Structure* the events that you dropped into the data will be saved into the events information structure. Vised Marks will then proceed to place *Out of task Marks* everywhere outside your boundaries. Below is a example of what the final result will look like if you were to plot the data afterwards, note that below we plotted the whole file at once so you could get a birds eye view.
|
|
|
|
After you click *Update Marks Structure* the events that you dropped into the data will be saved into the events information structure. Vised Marks will then proceed to place *Out of task marks* everywhere outside your boundaries. Below is a example of what the final result will look like if you were to plot the data afterwards, note that below we plotted the whole file at once so you could get a birds eye view.
|
|
|
|
|
|
|
|
|
|
|
|
These methods may seem like a lot of work but they can easily be simplified by using the events that already exist in the experiment. For the example above if we used Inevents we could have listed all of the event types in the experiment in the config file. We would get the same result as there were no events during the break times. That method would have run automatically and would have saved bunch of time. Consider your experiment when choosing a marking method.
|
| ... | ... | |
