Utilities on SQLite Browser

Creating an SQLite file directly from a BIDS dataset

Mon, 01 Jan 0001 00:00:00 +0000

utilities/bids2db.py indexes a BIDS dataset with bids2table (the same as b2t2 index), adds each file’s JSON sidecar metadata, and writes the result to a SQLite file that SQLite Browser can open.

Use this instead of parquets2db.py when you want sidecar metadata, because bids2table v2 Parquet files don’t include it.

Prerequisites

SQLite Browser dependencies installed (uv sync in the repo root), which includes bids2table
A BIDS dataset folder containing dataset_description.json and sub-* folders

Usage

uv run python utilities/bids2db.py PATH -o SQLITE_FILE [options]

Argument	Required	Description
`PATH`	Yes	Root folder of the BIDS dataset
`-o` / `--output`	Yes	Path for the output `.sqlite` file (must end in `.sqlite`)
`--subjects`	No	Subject folder names or glob patterns to include, with the `sub-` prefix (e.g. `sub-01 'sub-1*'`)
`-j` / `--workers`	No	Worker processes for loading metadata (`0` = main process (the default), `-1` = all cores)
`--use-threads`	No	Use threads instead of processes when `--workers` > 0
`-q` / `--no-progress`	No	Disable the progress bar
`-v` / `--verbose`	No	`-v` for warnings, `-vv` for more logging

Example:

Concatenating SQLite files

Mon, 01 Jan 0001 00:00:00 +0000

utilities/db_concatenate.py appends the rows of one table (default data) from several .sqlite files into a single output .sqlite file. This is useful when a dataset was indexed in pieces, for example one file per site produced by bids2db.py or parquets2db.py.

Usage

uv run python utilities/db_concatenate.py SQLITE_FILE [SQLITE_FILE ...] -o OUTPUT_FILE [options]

Argument	Default	Description
`SQLITE_FILE ...`	required	Space-separated input files; each must end in `.sqlite`
`-o` / `--output`	required	Output file; must end in `.sqlite` and must not be one of the inputs
`-t` / `--table`	`data`	Table to concatenate from every input

Example:

Enriching an SQLite database with a TSV join

Mon, 01 Jan 0001 00:00:00 +0000

After creating a SQLite file (see Indexing a BIDS dataset or Creating a SQLite file from Parquet files), you may want to enrich the data table with additional per-subject or per-session metadata stored in a separate TSV or CSV file — for example, a phenotype file or an ID-mapping file. utilities/db_join.py performs that join and writes the result back into the database.

Prerequisites

An existing .sqlite database (e.g., produced by bids2db.py or parquets2db.py)
A TSV or CSV file with a column that shares values with a column in the database table

Usage

uv run python utilities/db_join.py DATABASE CSV_FILE -k DB_KEY -c CSV_KEY [options]

Required arguments

Argument	Description
`DATABASE`	Path to the existing `.sqlite` file
`CSV_FILE`	Path to the TSV or CSV file to join in
`-k` / `--db-key`	Column name in the database table to join on
`-c` / `--csv-key`	Column name in the TSV/CSV file to join on

Optional arguments

Argument	Default	Description
`-t` / `--table`	`data`	Name of the database table to read from
`-j` / `--join-type`	`outer`	Join type: `left`, `right`, `inner`, or `outer`
`-o` / `--output-table`	`data`	Name of the table to write the joined result to
`--replace`	off	Replace the output table if it already exists
`-s` / `--separator`	auto	Column delimiter; auto-detected from file extension (`.tsv`/`.tab` → tab, everything else → comma)

Step-by-step

1. Identify your join columns

Open db.sqlite in SQLite Browser and note the column name you want to join on. Then check your TSV or CSV file for the matching column name — the two do not need to have the same name.

Creating an SQLite file from Parquet files

Mon, 01 Jan 0001 00:00:00 +0000

utilities/parquets2db.py collects one or more Parquet files, concatenates them into a single table, and writes the result to a SQLite file that can be opened directly in SQLite Browser. While it was originally built around bids2table outputs, it works with any Parquet files.

Sidecar metadata: bids2table v2 Parquet files don’t include JSON sidecar metadata. To include it, build the SQLite file directly from the BIDS dataset with bids2db.py.

Prerequisites

SQLite Browser dependencies installed (uv sync in the repo root)
One or more .parquet files to convert

Usage

uv run python utilities/parquets2db.py PARQUET_GLOB -o SQLITE_FILE

Argument	Required	Description
`PARQUET_GLOB`	Yes	Python glob pattern matching all Parquet files to include
`-o` / `--output`	Yes	Path for the output `.sqlite` file (must end in `.sqlite`)

Step-by-step

1. Locate your Parquet files

Here is an example of how Parquet files might look on the filesystem: