ndi.fun.probe.import.kilosort.probe
NDI.FUN.PROBE.IMPORT.KILOSORT.PROBE - import curated Kilosort spike sorting results into NDI
NDI.FUN.PROBE.IMPORT.KILOSORT.PROBE(S, PROBE, ...)
Imports curated Kilosort/Phy output for an ndi.probe (or ndi.element) PROBE that
is part of the ndi.session S. For each curated cluster that passes the quality
filter, this function creates:
1) an ndi.neuron element named [PROBE.name '_' N], where N is the cluster id,
with spike times added as epochs (mapped back from the concatenated Kilosort
sample stream into each NDI epoch's local time), and
2) an 'neuron_extracellular' ndi.document holding the mean waveform, sample
counts, cluster index, and quality (label/number) for that neuron.
This function is the import-side analog of NDI.FUN.PROBE.EXPORT.ALL_BINARY /
NDI.FUN.PROBE.EXPORT.BINARY: it expects the Kilosort output to live in the same
directory the binary was exported to, namely
[S.path]/[kilosort_dir]/[probe_directory]/[subdir]/
The [probe_directory] name comes from ndi.fun.file.elementDirectoryName;
for a probe named 'ctx' with reference 1 it is 'ctx_-_1'. Folders written
by older versions of NDI, which used a '|' separator ('ctx_|_1'), are still
found and used if they are present.
(spaces in the element string are replaced by underscores, matching the export).
By default 'subdir' is 'kilosort_output', so the curated files are expected in
a 'kilosort_output' subfolder of the probe's directory. Set 'subdir' to a
different name to use a different subfolder, or pass 'noSubFolder',true to look
for the files directly in the probe's directory (no subfolder).
The chosen directory is expected to contain the curated Kilosort/Phy files:
spike_times.npy - global (concatenated) sample index of each spike
spike_clusters.npy - curated cluster id of each spike
spike_templates.npy - template id of each spike (for waveforms)
templates.npy - nTemplates x nSamples x nChannels template shapes
amplitudes.npy - per-spike template scaling amplitude
cluster_group.tsv - (or cluster_KSLabel.tsv / cluster_info.tsv) curation labels
whitening_mat_inv.npy - (optional) used to un-whiten template waveforms
params.py - (optional) Phy parameters; its 'n_channels_dat',
'dtype' and 'offset' describe the raw recording's
layout (used when recalculating wide mean waveforms,
see below). Its 'dat_path' is NOT used to open a
binary (see the note below).
Note: the raw recording's file name is NOT stored in any of the .npy files
(they hold only numeric arrays). Phy records a 'dat_path' in params.py, but for
externally sorted data that path frequently points at Kilosort's whitened,
filtered temporary file (e.g. 'temp_wh.dat') rather than the true raw recording,
and cannot be relied on to exist - so this importer never opens it.
The spike sample indices in spike_times.npy are treated as positions in the
concatenated stream of the probe's epochs (in probe.epochtable() order), the same
ordering used by ndi.fun.probe.export.binary. The function checks that all spike
indices fall within the total sample count of the probe's epochs and errors
(ndi:fun:probe:import:kilosort:probe:sampleOutOfRange) if any fall outside, which
indicates the sort does not correspond to this probe's epochs.
A 'kilosort_clusters' ndi.document is created that depends on PROBE and stores the
MD5 checksum of spike_clusters.npy. This is used to detect whether the curation has
changed since a previous import: if the checksum is unchanged the function does
nothing (unless 'force' is 1); if it has changed, the previously imported neurons
and documents are removed and the import is repeated.
MEAN WAVEFORMS AND THE RAW BINARY
By default ('RecalculateMeanWaveforms' true) each neuron's mean waveform is
recomputed over a wide window ('RecalculateMeanWaveformT0' to
'RecalculateMeanWaveformT1', default -5 ms to +5 ms) by reading the raw binary
recording directly, because the Kilosort templates are only ~2 ms wide. All
clusters are recomputed together in a single streaming pass over the binary (see
ndi.fun.probe.import.kilosort.recalculatemeanwaveforms), so the file is read - and,
when high-pass filtering, filtered - once rather than once per cluster; pass
'progressbar',true to watch its progress. The raw binary is located automatically,
in this order:
1) an explicit 'binary_file' option, if given;
2) the '.metadata' sidecar written next to the binary by
ndi.fun.probe.export.binary (present when the data were exported from NDI).
If neither is available (e.g. data sorted OUTSIDE NDI, so there is no '.metadata'
sidecar), the importer PROMPTS you to select the raw recording and its Neuropixels
generation. It deliberately does NOT read Kilosort's params.py 'dat_path', because
for external sorts that often points at Kilosort's whitened, filtered temporary
file (e.g. 'temp_wh.dat') - which is not the raw data and may not exist. On a
headless/automated run pass 'RawFile' and 'ProbeType' to supply these without a
dialog (and 'PromptForRawFile',false to fail rather than block on a dialog).
Because a hand-selected raw recording is UNFILTERED (it carries the DC offset and
LFP band that Kilosort's temp file had already removed), the raw data are high-pass
filtered before the spike shapes are extracted ('HighPassFilter' true by default:
a zero-phase Chebyshev type I filter, 'HighPassCutoff' 300 Hz, 'HighPassOrder' 4,
'HighPassRipple' 0.8 dB). The same filtering is applied when reading the NDI
'.metadata' binary (also raw), so both routes yield comparably conditioned shapes.
NEUROPIXELS UNITS: a raw SpikeGLX recording stores int16 ADC counts; the decode to
volts is fixed by the probe generation ('ProbeType'): 'NP1' (Neuropixels 1.0) uses
volts = int16 * 0.6/(512*500); 'NP2' (Neuropixels 2.0) uses volts = int16 *
0.5/(8192*80). See ndi.fun.probe.import.kilosort.neuropixelsmultiplier. The read
stride (channel count) of a user-selected raw file is taken from its SpikeGLX
'.meta' sidecar (nSavedChans), else 'RawNumChannels', else a prompt - NOT from
n_channels_dat, which is the count of the (missing) sorted file and usually differs
(raw AP band 385 = 384 electrodes + 1 sync, vs an NDI export's 384 electrodes).
If the binary cannot be found and no raw file is selected, the mean waveforms fall
back to the narrow Kilosort templates and a warning is issued.
This function takes name/value pairs that modify its operation:
---------------------------------------------------------------------------------
| Parameter (default) | Description |
|--------------------------|-----------------------------------------------------|
| kilosort_dir ('kilosort')| Name of the directory holding the kilosort output |
| subdir | Subfolder within the probe's directory that holds |
| ('kilosort_output') | the curated files. Set to '' or use noSubFolder |
| | to read directly from the probe's directory. |
| noSubFolder (false) | If true, ignore 'subdir' and read the curated files |
| | directly from the probe's directory. |
| quality_labels | String array of curation labels to import. Labels |
| (["good" "mua"]) | are matched case-insensitively. You may pass your |
| | own custom tags here. |
| quality_values ([1 4]) | Numeric quality_number assigned to each label in |
| | quality_labels (parallel array). Defaults follow |
| | the convention single=1, multi=4. |
| kilosort_version ('2.5') | Version of Kilosort that produced the sort. Recorded |
| | in the 'app' provenance of the created documents |
| | (app.name = 'Kilosort<ver> to phy to |
| | ndi.fun.probe.import.kilosort', app.version=<ver>). |
| | Defaults to '2.5' (the assumption for MATLAB data).|
| waveform_source | 'templates' (amplitude-weighted average of the |
| ('templates') | contributing Kilosort templates) or 'none'. |
| RecalculateMeanWaveforms | If true (and waveform_source is 'templates'), |
| (true) | recompute each mean waveform by reading a wide |
| | window directly from the raw binary recording |
| | instead of using the (~2 ms) Kilosort templates. |
| | Falls back to templates (with a warning) if the |
| | binary cannot be located. |
| RecalculateMeanWaveformT0| Window start relative to each spike, in seconds, |
| (-0.005) | used when recalculating from the binary. |
| RecalculateMeanWaveformT1| Window end relative to each spike, in seconds, |
| (+0.005) | used when recalculating from the binary. |
| RecalculateMeanWaveform- | Maximum number of spikes averaged per cluster when |
| MaxSpikes (1000) | recalculating (an evenly spaced subset is used |
| | beyond this; Inf uses every spike). |
| RecalculateChunkMemory- | Peak-memory ceiling (bytes) for the single-pass |
| Bytes (2e9) | recalculation. The streaming chunk size is derived |
| | from it (accounting for the filtfilt overhead), so |
| | the whole pass stays under it. Raise it if the |
| | machine has spare RAM. |
| binary_file ('') | Explicit path to the raw binary recording. When |
| | empty, it is located from the export '.metadata' |
| | sidecar, else the user is prompted (see below). |
| HighPassFilter (true) | High-pass filter the raw data before extracting |
| | spike shapes (raw recordings are unfiltered). Uses |
| | a zero-phase Chebyshev type I filter. |
| HighPassCutoff (300) | High-pass cutoff frequency, Hz. |
| HighPassOrder (4) | Chebyshev type I filter order. |
| HighPassRipple (0.8) | Passband ripple Rp, dB. |
| PromptForRawFile (true) | If the binary is not found automatically, prompt |
| | (uigetfile + probe generation) for the raw file. |
| | Set false for headless runs (then RawFile and |
| | ProbeType must be supplied). |
| RawFile ('') | Path to the raw recording to use when the binary is |
| | not found automatically (skips the file dialog). |
| ProbeType ('') | 'NP1' or 'NP2': the Neuropixels generation of a |
| | selected raw file, fixing the int16->volts units |
| | (skips the probe-generation dialog). |
| RawNumChannels (NaN) | Override the read stride (channel count) of a |
| | selected raw file. When NaN it is read from the |
| | SpikeGLX .meta sidecar (nSavedChans), else the |
| | user is prompted. NOT n_channels_dat (the count |
| | of the sorted file, usually 384 vs raw 385). |
| force (0) | Re-import even if the checksum is unchanged. |
| dryRun (false) | If true, report what would be imported (neurons, |
| | spike counts, documents that would be removed) |
| | without making any changes to the database. |
| progressbar (false) | If true, show an ndi.gui.component.ProgressBarWindow |
| | tracking the single-pass mean-waveform recalcula- |
| | tion and the cluster import loop. Degrades quietly |
| | if no display is available. |
| verbose (1) | 0/1 Should we be verbose? |
---------------------------------------------------------------------------------
See also: NDI.FUN.PROBE.IMPORT.KILOSORT.SESSION, NDI.FUN.PROBE.EXPORT.ALL_BINARY
Example:
S = ndi.session.dir('/path/to/session');
p = S.getprobes('type','n-trode');
ndi.fun.probe.import.kilosort.probe(S, p{1});