Changes

Page history
mikec1 modified page: Running your first job authored by Mike Cichonski's avatar Mike Cichonski
Show whitespace changes
Inline Side-by-side
pipeline-scripts.md
View page @ 4a628309
...@@ -2,8 +2,8 @@ ...@@ -2,8 +2,8 @@
# Pipeline Scripts # Pipeline Scripts
This section is designed to outline the key procedures of each of the pipeline scripts. This section is designed to outline the key procedures of each of the pipeline scripts.
# Scalpart # Scalpart (s01)
1. Load an _init.set file 1. Load an initialized *_eeg.set file
2. Standard Deviation or quantiles used to reject comically bad epochs (% goes to bad channel) 2. Standard Deviation or quantiles used to reject comically bad epochs (% goes to bad channel)
3. Re-reference to interpolated average site of MNI surface: without flagged channels 3. Re-reference to interpolated average site of MNI surface: without flagged channels
4. High pass filter 4. High pass filter
...@@ -12,36 +12,55 @@ This section is designed to outline the key procedures of each of the pipeline s ...@@ -12,36 +12,55 @@ This section is designed to outline the key procedures of each of the pipeline s
* low correlation * low correlation
* too high * too high
7. Rank channel - highest correlation removed 7. Rank channel - highest correlation removed
8. ALL BAD CHANNELS GONE 8. ALL BAD CHANNELS FLAGGED
9. Re-reference to interpolated average site of MNI surface: without all flagged channels 9. Re-reference to interpolated average site of MNI surface: without all flagged channels
10. Nearest Neighbour R on epochs for time removal 10. Nearest Neighbour R on epochs for time removal
* low correlation only by time segment * low correlation only by time segment
11. Save an _sa.set file 11. Save an *_sa.set file
| Variable | Default in Config File | Description | | Variable | Default in Config File | Description |
|--------------------|--------------------------------------------------|-------------| |--------------------|--------------------------------------------------|-------------|
| [in_path] | analysis/data/1_init | Relative path to input data files assuming cd = work_path | | [in_path] | [] | Relative path to input data files assuming cd = work_path |
| [montage_info] | derivatives/lossless/code/scripts/staging_script.m | Montage information to fit channel locations on a standard surface. This can be a transformations matrix (e.g. [-.14,-22,-48,-.07,0,-1.57,1080,1260,1240]), or a a file containing a transformation matrix (e.g. [batch_dfn]_transmat.mat), or a BIDS channel location tsv file (assuming that during BIDS init the channels were coregistered to the standard surface). |
| [recur_sec] | 1 | Recurrence (sec) for artifact detection epoching | | [recur_sec] | 1 | Recurrence (sec) for artifact detection epoching |
| [limit_sec] | [0 1] | Limits (sec) for artifact detection epoching | | [limit_sec] | [0 1] | Limits (sec) for artifact detection epoching |
| [sd_chan_z] | 2.326 | z threshold for flagging channel during standard deviation criteria | | [sd_t_meth] | q | Method used for flagging epochs (e.g. 'q' (quantiles), or 'na') |
| [sd_chan_p] | 0.1 | Percentage of flagged epochs required to flag a channel during standard | | [sd_t_vals] | [.3 .7] | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7]) |
| [sd_chan_trim] | 10 | Percentage trim for confidence intervals during channel neighbour r criteria (10 = 5% top and bottom) | | [sd_t_o] | 6 | z threshold for flagging epochs during standard deviation criteria |
| [ref_loc_file] | analysis/support/misc/standard_1020_bucanl19.elc | Name of file containing the reference locations (including the relative path) | | [sd_t_f_meth] | fixed | Fixed method used for flagging epochs (should be 'fixed') |
| [low_bound_hz] | 2 | Lower bound of the filter bass-band | | [sd_t_f_vals] | | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
| [high_bound_hz] | 30 | Upper bound of the filter bass-band | | [sd_t_f_o] | 0.2 | z threshold for flagging epochs during fixed standard deviation criteria |
| [n_neighb_chan] | 3 | Number of channels to use in nearest neighbour r calculation (for channel criteria) | | [sd_t_pad] | 1 | Number of windows to pad onto each side of the ch_sd time flag |
| [r_chan_z] | 2.326 | z threshold for flagging channel during neighbour r criteria | | [sd_ch_meth] | q | Method used for flagging channels (e.g. 'q' (quantiles), or 'na') |
| [r_chan_p] | 0.1 | Percentage of flagged epochs required to flag a channel during neighbour r criteria | | [sd_ch_vals] | [.3 .7] | Percentage trim for confidence intervals during channel standard deviation criteria (e.g. [.3 .7]) |
| [r_chan_trim] | 10 | Percentage trim for confidence intervals during channel neighbour r criteria (10 = 5% top and bottom) | | [sd_ch_o] | 6 | z threshold for flagging channels during standard deviation criteria |
| [sd_ch_f_meth] | fixed | Fixed method used for flagging channels (should be 'fixed') |
| [sd_ch_f_vals] | | Percentage trim for confidence intervals during channel standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
| [sd_ch_f_o] | 0.2 | z threshold for flagging channel during fixed standard deviation criteria
| [ref_loc_file] | derivatives/lossless/code/misc/standard_1020_bucanl19.elc | Name of file containing the reference locations (including the relative path) |
| [low_bound_hz] | 1 | Lower bound of the filter bass-band |
| [high_bound_hz] | | Upper bound of the filter bass-band |
| [save_f_res] | 1 | 1 if you want to save the filter residuals |
| [n_nbr_ch] | 3 | Number of channels to use in nearest neighbour r calculation (for channel criteria) |
| [r_ch_meth] | q | Method used for flagging low r channels (e.g. 'q' (quantiles), or 'na') |
| [r_ch_vals] | [.3 .7] | Percentage trim for confidence intervals during low r channel standard deviation criteria (e.g. [.3 .7]) |
| [r_ch_o] | 6 | z threshold for flagging channel during neighbour r criteria (e.g. 3) |
| [r_ch_f_meth] | fixed | Fixed method used for flagging low r channels (should be 'fixed') |
| [r_ch_f_vals] | | Percentage trim for confidence intervals during low r channel standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
| [r_ch_f_o] | 0.2 | z threshold for flagging channel during fixed neighbour r criteria (e.g. 3) |
| [bridge_trim] | 40 | Percentage trim for z calculation of bridged channels (e.g. 40 = 20% top and bottom) | | [bridge_trim] | 40 | Percentage trim for z calculation of bridged channels (e.g. 40 = 20% top and bottom) |
| [bridge_z] | 3 | z threshold for flagging channel during bridging criteria | | [bridge_z] | 6 | z threshold for flagging channel during bridging criteria |
| [n_neighb_epoch] | 3 | Number of channels to use in nearest neighbour r calculation (for epoch criteria) | | [n_nbr_t] | 3 | Number of channels to use in nearest neighbour r calculation (for epoch criteria) |
| [epoch_z] | 2.32 | z threshold for flagging epochs during neighbour r criteria | | [r_t_meth] | q | Method used for flagging low r epochs (e.g. 'q' (quantiles), or 'na') |
| [epoch_p] |0. 2 | Percentage of flagged channels required to flag an epoch during neighbour r criteria | | [r_t_vals] | [.3 .7] | Percentage trim for confidence intervals during low r epoch standard deviation criteria (e.g. [.3 .7]) |
| [epoch_trim] | 20 | Percentage trim for confidence intervals during epoch neighbour r criteria (20 = 10% top and bottom) | | [r_t_o] | 6 | z threshold for flagging epochs during neighbour r criteria (e.g. 3) |
| [out_path] | analysis/data/2_preproc | Relative path to output data files assuming cd = work_path | | [r_t_f_meth] | fixed | Fixed method used for flagging low r epochs (should be 'fixed') |
| [amica_param_file] | analysis/support/misc/amica_default.param | template amicadefs.param file to modify | | [r_t_f_vals] | | Percentage trim for confidence intervals during low r epoch standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
| [savediff] | NO | YES if you want to save the filter residuals | | [r_t_f_o] | 0.2 | z threshold for flagging epochs during fixed neighbour r criteria |
| [min_gap_ms] | 2000 | Minimum time (ms) to allow between periods marked for rejection |
| [out_path] | [] | Relative path to output data files assuming cd = work_path |
| [amica_param_file] | derivatives/lossless/code/misc/amica15_default.param | template amicadefs.param file to modify |
| [amica_threads_s02] | 8 | number of threads to use for running s02 amica script |
![diag111](/uploads/a9c836eef4244f3c251a3f36ce19cc4f/diag111.png) ![diag111](/uploads/a9c836eef4244f3c251a3f36ce19cc4f/diag111.png)
...@@ -49,91 +68,78 @@ This section is designed to outline the key procedures of each of the pipeline s ...@@ -49,91 +68,78 @@ This section is designed to outline the key procedures of each of the pipeline s
![diag333](/uploads/54b21404919e9a41a4663b36a4376179/diag333.png) ![diag333](/uploads/54b21404919e9a41a4663b36a4376179/diag333.png)
# Amica # Amica (s02, s04a, s04b, s04c)
The Amica script is run once initially, and then three more times in parallel after the component artefact rejection to ensure that the Amica procedure is replicable if run multiple times on the same dataset.
Versions Versions
* amica_init * amica_init (s02)
* amica_asrinit * amica_A,B,C (s04a, s04b, s04c)
* amica_A,B,C
* amica_D,E,F
| Variable | Default in Config File | Description | | Variable | Default in Config File | Description |
|-------------|--------------------------------------------------------------|-------------| |-------------|--------------------------------------------------------------|-------------|
| [dep_path] | analysis/support/dependencies/eeglab_asr_amica/plugins/amica | Relative path to amica executable program assuming cd = work_path | | [dep_path] | derivatives/lossless/code/dependencies/eeglab_asr_amica/plugins/amica | Relative path to amica executable program assuming cd = work_path |
| [in_path] | analysis/data/2_preproc | Relative path to input data files assuming cd = work_path | | [in_path] | [] | Relative path to input data files assuming cd = work_path |
| [amica_ver] | amica15ub | Name of the amica executable program version | | [amica_ver] | amica15 | Name of the amica executable program version |
# Compart # Compart (s03)
Versions Versions
* compart_data * compart_data (s03)
* compart_asrdata
| Variable | Default in Config | Description | | Variable | Default in Config | Description |
|--------------------|-------------------------------------------|-------------| |--------------------|-------------------------------------------|-------------|
| [in_path] | analysis/data/2_preproc | Path to input and output data files assuming cd = work_path | | [in_path] | derivatives/lossless | Path to input and output data files assuming cd = work_path |
| [recur_sec] | 1 | Recurrence (sec) for artifact detection epoching | | [recur_sec] | 1 | Recurrence (sec) for artifact detection epoching |
| [limit_sec] | [0 1] | Limits (sec) for artifact detection epoching | | [limit_sec] | [0 1] | Limits (sec) for artifact detection epoching |
| [epoch_z] | 2.326 | z threshold for flagging channel during ic_sd criteria | | [sd_t_meth] | q | Method used for flagging epochs (e.g. 'q' (quantiles), or 'na') |
| [epoch_p] | 0.1 | Percentage of flagged components required to flag an epoch during ic_sd criteria | | [sd_t_vals] | [.3 .7] | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7]) |
| [epoch_trim] | 20 | Percentage trim for confidence intervals during channel neighbour r criteria (20 = 10% top and bottom) | | [sd_t_o] | 6 | z threshold for flagging epochs during standard deviation criteria (e.g. 6) |
| [out_path] | analysis/data/2_preproc | Relative path to output data files assuming cd = work_path | | [sd_t_f_meth] | fixed | Fixed method used for flagging epochs (should be 'fixed') |
| [amica_param_file] | analysis/support/misc/amica_default.param | Template amicadefs.param file to modify. | | [sd_t_f_vals] | | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
| [min_gap_ms] | 4000 | Minimum time (ms) to allow between periods marked for rejection | | [sd_t_f_o] | 0.2 | z threshold for flagging epochs during fixed standard deviation criteria (e.g. .2) |
| [out_path] | derivatives/lossless | Relative path to output data files assuming cd = work_path |
![diag444](/uploads/bc498981a3b21de4b3a2252d4cd1031c/diag444.png) | [min_gap_ms] | 2000 | Minimum time (ms) to allow between periods marked for rejection |
| [amica_param_file] | derivatives/lossless/code/misc/amica_default.param | Template amicadefs.param file to modify |
| [amica_threads_s04] | 8 | number of threads to use for running s04a, s04b, s04c amica scripts |
# ASR
ASR learning time and model selection ![diag444](/uploads/bc498981a3b21de4b3a2252d4cd1031c/diag444.png)
The selection of the learning model is based of segments of time that have a high AMICA log likelihood values.
ASR learn time ticks through each of the data points of the log likelihood and looks at the Lt values. By default for the time to be considered the one second block must have 80% of its points above them mean. This percentage can be adjusted, but its intent is prevent time segments where the model may be bad for part but the other half is very strong. If the following second has enough values above your critical mean then it is considered viable for the learning time. By progressing through each data point, sections of time will only be flagged for time sections for 1 sec or longer, but is not limited to 1 second intervals.
Viable time is then sorted so that the highest blocks log likelihood time are added first to the total time.Segments are continued to be added until the total time value defaulted at 1 minute is filled. This EEG data is added up and then sent to the *clear_asr* function. # Concat and Dipfit (s05)
| Variable | Default in Config File | Description |
|--------------------|-------------------------------------------|-------------|
| [in_path] | analysis/data/2_preproc | Relative path to input data files assuming cd = work_path |
| [out_path] | analysis/data/2_preproc | Relative path to output data files assuming cd = work_path |
| [amica_param_file] | analysis/support/misc/amica_default.param | template amicadefs.param file to modify. |
| [asr_sd_cutoff] | 20 | Any data over this standard deviation will be excluded |
| [asr_winlen] | [] | Window length, does not normally need adjusting |
| [asr_stepsize] | [] | Processing step size, does not normally need adjusting |
| [asr_maxdims] | [] | Max matrix dimension reconstruction size faction or value (eg. 2/3), does not normally need adjusting |
| [asr_learningtime] | 60 | How long of data ASR trains on in sec |
| [asr_reftol] | [] | ASR reference tolerence by standard deviation (eg. [-3.5 5.5]) not needed as learning data is supplied |
| [asr_refwinlen] | [] | Window length of references. (eg. 1), does not normally need adjusting |
| [asr_useGPU] | false | Run on the GPU or not (true or false). |
| [min_gap_ms] | 4000 | Minimum time (ms) to allow between periods marked for rejection |
| [savediff] | NO | YES if you want to save the filter residuals |
| [critmean] | 0.8 | Fraction of training points that need to be above the mean to be included in ASR training |
| [asr_mem] | 5000 | Free memory allotted to asr in MB must be >256 (Default: 5000), for remote should be less then mpp allotted. |
![diag555](/uploads/3a7bb2cec77064cbada8e45194c93ca1/diag555.png)
# Concat
Versions Versions
* concat_data * concat_data (s05)
* concat_asrdata
| Variables | Default in Config | Description | | Variables | Default in Config | Description |
|--------------|-------------------------|-------------| |--------------|-------------------------|-------------|
| [in_path] | analysis/data/2_preproc | Relative path to input data files assuming cd = work_path | | [in_path] | derivatives/lossless | Relative path to input data files assuming cd = work_path |
| [out_path] | analysis/data/2_preproc | Relative path to output data files assuming cd = work_path |
| [recur_sec] | 1 | Recurrence (sec) for the current artifact detection epoching | | [recur_sec] | 1 | Recurrence (sec) for the current artifact detection epoching |
| [limit_sec] | [0 1] | Limits (sec) for the current artifact detection epoching | | [limit_sec] | [0 1] | Limits (sec) for the current artifact detection epoching |
| [epoch_z] | 2.326 | z threshold for flagging channel during icsd criteria | | [sd_t_meth] | q | Method used for flagging epochs (e.g. 'q' (quantiles), or 'na') |
| [epoch_p] | 0.1 | Percentage of flagged components required to flag an epoch during icsd criteria | | [sd_t_vals] | [.3 .7] | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7]) |
| [epoch_trim] | 20 | Percentage trim for confidence intervals during channel neighbour r criteria (20 = 10% top and bottom) | | [sd_t_o] | 6 | z threshold for flagging epochs during standard deviation criteria (e.g. 6) |
| [min_gap_ms] | 4000 | Minimum time (ms) to allow between periods marked for rejection | | [sd_t_f_meth] | fixed | Fixed method used for flagging epochs (should be 'fixed') |
| [sd_t_f_vals] | | Percentage trim for confidence intervals during epoch standard deviation criteria (e.g. [.3 .7], leave empty for 'fixed') |
# Dipfit | [sd_t_f_o] | 0.2 | z threshold for flagging epochs during fixed standard deviation criteria (e.g. .2) |
| [min_gap_ms] | 2000 | Minimum time (ms) to allow between periods marked for rejection |
| [icr_crit] | 0.05 | False discovery rate of component replicatablilty, measure of sensitivity % |
| [dip_vol] | derivatives/lossless/code/misc/standard_vol.mat | file path/name of the standard volume file used in dipole fitting |
| [dip_mri] | derivatives/lossless/code/misc/standard_mri.mat | file path/name of the standard mri file used in dipole fitting |
| [dip_elc] | derivatives/lossless/code/misc/standard_1020.elc | file path/name of the standard electrode file used in dipole fitting |
| [dip_transmat] | [0 0 0 0 0 0 1 1 1] | Transformation matrix to apply during dipole fit |
| [cor_transmat] | [0,0,0,0,0,1.57,1,1,1] | The transformation matrix to apply between dipole fitting to the standard head and performing ICLabel to the EEGLAB default orientation |
| [cor_nosedir] | +X | Nose direction of montage to apply between dipole fitting to the standard head and performing ICLabel to the EEGLAB default orientation (e.g. +Y) |
| [out_path] | derivatives/lossless | Relative path to output data files assuming cd = work_path |
*** ***
*Updated/Verified by Mike Cichonski on 2019-03-13*
[ :house: **Return to Main Page**](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/wikis/home) [ :house: **Return to Main Page**](https://git.sharcnet.ca/bucanl_pipelines/bids_lossless_eeg/wikis/home)
\ No newline at end of file