Merge pull request #158 from fedorov/update-idc-skill-v1.6.2
Skill Spec Validation / Validate skills against the Agent Skills spec (push) Has been cancelled
Skill Tests / Repo-wide contract and coverage guard (push) Has been cancelled
Skill Tests / Standard-library-only skill suites (push) Has been cancelled
Skill Spec Validation / Validate skills against the Agent Skills spec (push) Has been cancelled
Skill Tests / Repo-wide contract and coverage guard (push) Has been cancelled
Skill Tests / Standard-library-only skill suites (push) Has been cancelled
Update imaging-data-commons skill from v1.4.0 to v1.8.1 upstream
This commit is contained in:
Timothy Kassis
@@ -1,82 +1,93 @@
|
||||
---
|
||||
name: imaging-data-commons
|
||||
description: Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses.
|
||||
description: Query and download public cancer imaging data from NCI Imaging Data Commons. Invoke for any question about IDC collections, cancer imaging datasets, DICOM data access, radiology (CT, MR, PET) or pathology AI training sets, metadata queries, visualization, or license checks — even when the user doesn't explicitly mention "IDC". No authentication required.
|
||||
license: This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data.
|
||||
metadata:
|
||||
version: "1.4"
|
||||
source-skill-version: 1.4.0
|
||||
version: "1.5"
|
||||
source-skill-version: 1.8.1
|
||||
skill-author: Andrey Fedorov, @fedorov
|
||||
idc-index: 0.11.14
|
||||
idc-data-version: v23
|
||||
repository: https://github.com/ImagingDataCommons/idc-claude-skill
|
||||
idc-index: "0.12.5"
|
||||
idc-data-version: "v24"
|
||||
repository: https://github.com/ImagingDataCommons/imaging-data-commons-skill
|
||||
---
|
||||
|
||||
# Imaging Data Commons
|
||||
|
||||
## Overview
|
||||
|
||||
Use the `idc-index` Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
|
||||
Query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
|
||||
|
||||
**Current IDC Data Version: v23** (always verify with `IDCClient().get_idc_version()`)
|
||||
**Expected network access:** IDC metadata is reachable three ways — a local DuckDB index shipped with the `idc-index` Python package (no network), or the hosted IDC service over MCP or REST (`api.imaging.datacommons.cancer.gov`, no authentication). File downloads use public GCS (`storage.googleapis.com`) and AWS S3 (`s3.amazonaws.com`) — no authentication required. DICOMweb access uses either the public IDC proxy (`proxy.imaging.datacommons.cancer.gov`, no auth) or the Google Cloud Healthcare API (`healthcare.googleapis.com`, requires GCP authentication). Optional BigQuery queries (`bigquery.googleapis.com`) also require GCP authentication. No credentials or environment variables are accessed by this skill.
|
||||
|
||||
**Primary tool:** `idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index))
|
||||
**Current IDC Data Version: v24** (always verify — see *Best Practices*)
|
||||
|
||||
**CRITICAL - Check package version before anything else (run this FIRST):**
|
||||
**Choose the access path first.** There is no single default: the cheapest correct path depends
|
||||
on the session and the task.
|
||||
|
||||
This block only *reports*. It never installs. If the version is too old, show the user the
|
||||
suggested command and wait for them to approve it — do not run an install on their behalf.
|
||||
1. **Session already has the IDC MCP server?** Route discovery and metadata there — see *IDC
|
||||
MCP Server*.
|
||||
2. **Otherwise, is `idc-index` installed?** Run `python scripts/check_version.py`. If it passes,
|
||||
use `idc-index` for everything.
|
||||
3. **Not installed, and the task is read-only metadata** — counts, attribute values, collection
|
||||
lookups, SQL under 10 000 rows, licenses, citations, viewer URLs? **Use the REST API over
|
||||
`curl`; do not install anything.** Installing costs ~77 MB of packaged index data plus
|
||||
pandas, pyarrow, and duckdb, which a metadata question does not need. See *Data Access
|
||||
Options*.
|
||||
4. **Not installed, and the task needs more than metadata** — downloading files, pandas or
|
||||
plotting, pydicom/SimpleITK, pathology tiling, results past 10 000 rows, or a version-pinned
|
||||
script the user re-runs? Install `idc-index`: `check_version.py` exits non-zero and prints
|
||||
the exact install command for the running interpreter. Prefer a virtual environment, then
|
||||
restart Python.
|
||||
|
||||
```python
|
||||
import idc_index
|
||||
`idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index)) is still the most
|
||||
capable path and the only one that moves image bytes; the rule is just not to pay for it before
|
||||
the task calls for it. `check_version.py` never installs anything itself — it also flags a newer
|
||||
`idc-index` or skill release when one exists.
|
||||
|
||||
REQUIRED_VERSION = "0.11.14" # Must match metadata.idc-index in this file
|
||||
installed = idc_index.__version__
|
||||
|
||||
def _parts(version):
|
||||
# Compare numerically: "0.9.0" < "0.11.14" is False as a string comparison.
|
||||
return tuple(int(p) if p.isdigit() else 0 for p in version.split(".")[:3])
|
||||
|
||||
if _parts(installed) < _parts(REQUIRED_VERSION):
|
||||
print(f"idc-index {installed} is older than the tested {REQUIRED_VERSION}.")
|
||||
print("Ask the user before installing. Suggested command, in a virtual environment:")
|
||||
print(f" uv pip install 'idc-index=={REQUIRED_VERSION}'")
|
||||
else:
|
||||
print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")
|
||||
```
|
||||
|
||||
**Never** install into a system-managed Python with `--break-system-packages`. That flag exists to
|
||||
override a protection the distribution put there deliberately, and a skill has no business
|
||||
switching it off unattended. Install into a virtual environment, and pin the version you tested
|
||||
against so a later IDC release cannot silently change query results underneath a saved analysis.
|
||||
|
||||
**Verify IDC data version and check current data scale:**
|
||||
**Setup for the `idc-index` path:**
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
|
||||
# Verify IDC data version (should be "v23")
|
||||
# Verify IDC data version (should be "v24")
|
||||
print(f"IDC data version: {client.get_idc_version()}")
|
||||
|
||||
# Get collection count and total series
|
||||
stats = client.sql_query("""
|
||||
SELECT
|
||||
COUNT(DISTINCT collection_id) as collections,
|
||||
COUNT(DISTINCT analysis_result_id) as analysis_results,
|
||||
COUNT(DISTINCT PatientID) as patients,
|
||||
COUNT(DISTINCT StudyInstanceUID) as studies,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series,
|
||||
SUM(instanceCount) as instances,
|
||||
SUM(series_size_MB)/1000000 as size_TB
|
||||
FROM index
|
||||
""")
|
||||
print(stats)
|
||||
```
|
||||
|
||||
**Core workflow:**
|
||||
1. Query metadata → `client.sql_query()`
|
||||
2. Download DICOM files → `client.download_from_selection()`
|
||||
3. Visualize in browser → `client.get_viewer_URL(seriesInstanceUID=...)`
|
||||
**Core workflow:** query metadata with `client.sql_query()` → download with
|
||||
`client.download_from_selection()` → visualize with `client.get_viewer_URL()`. Python examples
|
||||
below assume this `client`; *Data Access Options* has the REST equivalents. For current data
|
||||
scale, run the summary query in `references/sql_patterns.md` or `GET /v3/stats`.
|
||||
|
||||
## IDC MCP Server
|
||||
|
||||
IDC operates a hosted MCP server at `https://api.imaging.datacommons.cancer.gov/mcp`
|
||||
(streamable HTTP, no authentication). Where it is available it complements — it does not
|
||||
replace — the `idc-index` workflow below.
|
||||
|
||||
**Identify it** by the MCP resource `idc://guide`, or by three or more of the tool names
|
||||
`build_cohort`, `get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Generic
|
||||
names such as `run_sql` are not evidence on their own. If identification is ambiguous, use
|
||||
`idc-index`.
|
||||
|
||||
**If this session has the server**, treat it as authoritative for discovery and metadata —
|
||||
IDC version, counts, attribute values, cohort building, metadata SQL — and follow the
|
||||
server's own instructions rather than re-deriving them from this file. Its data version is
|
||||
whatever the server reports: call `get_idc_version` instead of relying on the version pinned
|
||||
in this file.
|
||||
|
||||
Return here for what the server does not do: downloading files, local pandas/notebook
|
||||
analysis, DICOMweb, BigQuery, digital pathology tiling, and reproducible scripts. Hand off by
|
||||
passing SeriesInstanceUIDs from the server to `client.download_from_selection(...)`, and run
|
||||
`scripts/check_version.py` at that point.
|
||||
|
||||
**If it is not available**, the identical service is reachable with no configuration as a REST
|
||||
API at `https://api.imaging.datacommons.cancer.gov/v3` — use it for read-only metadata rather
|
||||
than installing `idc-index`, per the routing gate in *Overview*. Suggest connecting the MCP
|
||||
server at most once, only for repeated interactive discovery, and never change the user's
|
||||
configuration yourself.
|
||||
|
||||
See `references/mcp_guide.md` for the tool inventory, handoff patterns, and per-host notes.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
@@ -88,37 +99,34 @@ print(stats)
|
||||
|
||||
## Quick Navigation
|
||||
|
||||
**Core Sections (inline):**
|
||||
- IDC Data Model - Collection and analysis result hierarchy
|
||||
- Index Tables - Available tables and joining patterns
|
||||
- Installation - Package setup and version verification
|
||||
- Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)
|
||||
- Best Practices - Usage guidelines
|
||||
- Troubleshooting - Common issues and solutions
|
||||
Inline below: the MCP/REST routing rules, the IDC data model, the index tables and how they
|
||||
join, the core API patterns (query, download, visualize, license, cite), best practices, and
|
||||
troubleshooting.
|
||||
|
||||
**Reference Guides (load on demand):**
|
||||
|
||||
| Guide | When to Load |
|
||||
|-------|--------------|
|
||||
| `index_tables_guide.md` | Complex JOINs, schema discovery, DataFrame access |
|
||||
| `use_cases.md` | End-to-end workflow examples (training datasets, batch downloads) |
|
||||
| `use_cases.md` | End-to-end workflows: training datasets, batch downloads, DICOM reading with pydicom/SimpleITK, pipeline integration |
|
||||
| `sql_patterns.md` | Quick SQL patterns for filter discovery, annotations, size estimation |
|
||||
| `clinical_data_guide.md` | Clinical/tabular data, imaging+clinical joins, value mapping |
|
||||
| `licensing_and_citation.md` | Commercial-use questions, mixed-license cohorts, citation formats |
|
||||
| `cloud_storage_guide.md` | Direct S3/GCS access, versioning, UUID mapping |
|
||||
| `dicomweb_guide.md` | DICOMweb endpoints, PACS integration |
|
||||
| `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows |
|
||||
| `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) |
|
||||
| `cli_guide.md` | Command-line tools (`idc download`, manifest files) |
|
||||
| `parquet_access_guide.md` | Direct Parquet queries via GCS (no idc-index install needed) |
|
||||
| `mcp_guide.md` | Hosted IDC MCP server: tool inventory, identification, handoff to `idc-index` |
|
||||
| `rest_api_guide.md` | Hosted IDC REST API: endpoints, filter syntax, SQL over HTTP, manifests |
|
||||
|
||||
## IDC Data Model
|
||||
|
||||
IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):
|
||||
|
||||
- **collection_id**: Groups patients by disease, modality, or research focus (e.g., `tcga_luad`, `nlst`). A patient belongs to exactly one collection.
|
||||
- **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.
|
||||
|
||||
Use `collection_id` to find original imaging data, may include annotations deposited along with the images; use `analysis_result_id` to find AI-generated or expert annotations.
|
||||
- **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections. Use it to find AI-generated or expert annotations, while `collection_id` finds original imaging data (which may itself include deposited annotations).
|
||||
|
||||
**Key identifiers for queries:**
|
||||
| Identifier | Scope | Use for |
|
||||
@@ -130,278 +138,359 @@ Use `collection_id` to find original imaging data, may include annotations depos
|
||||
|
||||
## Index Tables
|
||||
|
||||
The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.
|
||||
The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames. The REST API exposes the same tables through `GET /tables` and `POST /sql`.
|
||||
|
||||
**Complete index table documentation:** Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.
|
||||
|
||||
**Important:** Use `client.indices_overview` to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.
|
||||
**Important:** `client.indices_overview` is the authoritative source for current table descriptions, available columns, and their types — query it when writing SQL or exploring data structure. It also answers "which table contains column X"; see `references/index_tables_guide.md` for that search pattern and full schema discovery.
|
||||
|
||||
### Available Tables
|
||||
|
||||
| Table | Row Granularity | Loaded | Description |
|
||||
|-------|-----------------|--------|-------------|
|
||||
| `index` | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data |
|
||||
| `prior_versions_index` | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data |
|
||||
| `collections_index` | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions |
|
||||
| `analysis_results_index` | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) |
|
||||
| `clinical_index` | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections |
|
||||
| `sm_index` | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata |
|
||||
| `sm_instance_index` | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy |
|
||||
| `seg_index` | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series |
|
||||
| `ann_index` | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
|
||||
| `ann_group_index` | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
|
||||
| `contrast_index` | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
|
||||
| `volume_geometry_index` | 1 row = 1 CT/MR/PT series | fetch_index() | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag |
|
||||
| `rtstruct_index` | 1 row = 1 RTSTRUCT series | fetch_index() | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
|
||||
Always call `client.fetch_index("table_name")` before querying any index table — it is safe and idempotent for all tables, including those loaded automatically at startup.
|
||||
|
||||
**Auto** = loaded automatically when `IDCClient()` is instantiated
|
||||
**fetch_index()** = requires `client.fetch_index("table_name")` to load
|
||||
| Family | Tables | Granularity |
|
||||
|--------|--------|-------------|
|
||||
| Core | `index` (primary metadata for all current data), `collections_index`, `analysis_results_index` | series / collection / analysis result |
|
||||
| Modality acquisition parameters | `ct_index`, `mr_index`, `pt_index`, `contrast_index` | 1 row = 1 series of that modality |
|
||||
| Derived objects | `seg_index`, `rtstruct_index`, `ann_index`, `ann_group_index` | 1 row = 1 series (or annotation group) |
|
||||
| Microscopy | `sm_index`, `sm_instance_index` | 1 row = 1 SM series / instance |
|
||||
| Geometry, clinical, history | `volume_geometry_index`, `clinical_index`, `version_metadata_index`, `prior_versions_index` | see guide |
|
||||
|
||||
`references/index_tables_guide.md` has the full inventory with each table's columns and
|
||||
contents — load it when you need to know what a specialized table actually holds.
|
||||
|
||||
**`prior_versions_index` is for reproducibility only.** It contains series permanently *removed*
|
||||
from IDC, with zero overlap with `index`. Use it only to reproduce work against a prior IDC
|
||||
version. Do NOT use it for version history or "what's new" questions — those use
|
||||
`series_init_idc_version` / `series_revised_idc_version` in the main `index` table, which are
|
||||
not equivalent to this table's `min_idc_version` / `max_idc_version`.
|
||||
|
||||
### Joining Tables
|
||||
|
||||
**Key columns are not explicitly labeled, the following is a subset that can be used in joins.**
|
||||
**`SeriesInstanceUID` is the universal join key** for all series-level specialized tables: `sm_index`, `sm_instance_index`, `seg_index`, `ann_index`, `ann_group_index`, `contrast_index`, `volume_geometry_index`, `rtstruct_index`, `ct_index`, `mr_index`, `pt_index`. Always join these to `index` on `SeriesInstanceUID`. The exceptions below use different column names.
|
||||
|
||||
| Join Column | Tables | Use Case |
|
||||
|-------------|--------|----------|
|
||||
| `collection_id` | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data |
|
||||
| `SeriesInstanceUID` | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details |
|
||||
| `StudyInstanceUID` | index, prior_versions_index | Link studies across current and historical data |
|
||||
| `PatientID` | index, prior_versions_index | Link patients across current and historical data |
|
||||
| `analysis_result_id` | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) |
|
||||
| `source_DOI` | index, analysis_results_index | Link by publication DOI |
|
||||
| `crdc_series_uuid` | index, prior_versions_index | Link by CRDC unique identifier |
|
||||
| `Modality` | index, prior_versions_index | Filter by imaging modality |
|
||||
| `SeriesInstanceUID` | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation/annotation/contrast series to its index metadata |
|
||||
| `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |
|
||||
| `referenced_SeriesInstanceUID` | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |
|
||||
| `SeriesInstanceUID` | index, volume_geometry_index | Link series to its 3D geometry validation result (join index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID) |
|
||||
| `SeriesInstanceUID` / `referenced_SeriesInstanceUID` | index, rtstruct_index | Join RTSTRUCT series to its metadata (index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID); use rtstruct_index.referenced_SeriesInstanceUID to find the source image series |
|
||||
| `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (`seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID`) |
|
||||
| `referenced_SeriesInstanceUID` | ann_index → index, rtstruct_index → index | Link annotation or RTSTRUCT to its source image series |
|
||||
|
||||
**Note:** `Subjects`, `Updated`, and `Description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).
|
||||
**Note:** `subjects`, `updated`, and `description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts). Joining `prior_versions_index` to `index` on `SeriesInstanceUID` always returns zero rows — see the warning above.
|
||||
|
||||
For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see `references/index_tables_guide.md`.
|
||||
|
||||
### Clinical Data Access
|
||||
|
||||
```python
|
||||
# Fetch clinical index (also downloads clinical data tables)
|
||||
client.fetch_index("clinical_index")
|
||||
Clinical (non-imaging) attributes — staging, demographics, therapy — live in per-collection
|
||||
tables. `client.fetch_index("clinical_index")` loads the dictionary mapping columns to
|
||||
collections; `client.get_clinical_table(name)` returns one table as a DataFrame.
|
||||
|
||||
# Query clinical index to find available tables and their columns
|
||||
tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")
|
||||
|
||||
# Load a specific clinical table as DataFrame
|
||||
clinical_df = client.get_clinical_table("table_name")
|
||||
```
|
||||
|
||||
See `references/clinical_data_guide.md` for detailed workflows including value mapping patterns and joining clinical data with imaging.
|
||||
See `references/clinical_data_guide.md` for the discovery workflow, coded-value mapping, and
|
||||
joining clinical data with imaging.
|
||||
|
||||
## Data Access Options
|
||||
|
||||
| Method | Auth Required | Best For |
|
||||
|--------|---------------|----------|
|
||||
| `idc-index` | No | Key queries and downloads (recommended) |
|
||||
| Direct Parquet (GCS) | No | Quick queries without installing idc-index; always uses latest data |
|
||||
| IDC Portal | No | Interactive exploration, manual selection, browser-based download |
|
||||
| BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |
|
||||
| DICOMweb proxy | No | Tool integration via DICOMweb API |
|
||||
| Cloud storage (S3/GCS) | No | Direct file access, bulk downloads, custom pipelines |
|
||||
| Method | Auth | Best For | Reference |
|
||||
|--------|------|----------|-----------|
|
||||
| `idc-index` | No | Downloads, pandas analysis, unbounded queries — the most capable path | This document |
|
||||
| IDC MCP server | No | Discovery, cohort building, metadata when the session already has it | `mcp_guide.md` |
|
||||
| IDC REST API | No | Metadata with no install, from any language or shell — the default when `idc-index` is absent | `rest_api_guide.md` |
|
||||
| Direct Parquet (GCS) | No | Version-pinned queries, or results past the REST row cap | `parquet_access_guide.md` |
|
||||
| Cloud storage (S3/GCS) | No | Direct file access, bulk transfer, custom pipelines | `cloud_storage_guide.md` |
|
||||
| DICOMweb via IDC proxy | No | Tool and PACS integration; daily quota, so testing and moderate use | `dicomweb_guide.md` |
|
||||
| DICOMweb via Google Healthcare | Yes (GCP) | The same DICOMweb API at production volume, without the proxy quota | `dicomweb_guide.md` |
|
||||
| SlicerIDCBrowser | No | 3D visualization and analysis in 3D Slicer | https://github.com/ImagingDataCommons/SlicerIDCBrowser |
|
||||
| BigQuery | Yes (GCP) | Full DICOM metadata, private elements, SR measurements — last resort | `bigquery_guide.md` |
|
||||
|
||||
**The IDC Portal (https://portal.imaging.datacommons.cancer.gov/) is interactive only** —
|
||||
browser-based exploration, manual cohort selection, and download. Unlike every option above it
|
||||
has no programmatic interface, so point a user there to browse or click through data
|
||||
themselves; never use it as a step in a script or workflow.
|
||||
|
||||
**REST API — the no-install metadata path**
|
||||
|
||||
`https://api.imaging.datacommons.cancer.gov/v3`, no authentication: discovery, cohort counts and
|
||||
manifests, read-only SQL, clinical tables, viewer URLs, licenses, citations. It is the same
|
||||
service as the MCP server over plain HTTP, so it needs no configuration. It never moves image
|
||||
bytes — switch to `idc-index` to download, to get a DataFrame, or for results past 10 000 rows.
|
||||
|
||||
```bash
|
||||
B=https://api.imaging.datacommons.cancer.gov/v3
|
||||
curl -s $B/version # idc_version, idc_index_data_version, api_version
|
||||
curl -s $B/stats # collections, patients, studies, series, instances, size_TB
|
||||
curl -s "$B/attributes/Modality/values?limit=5" # real filter values, with counts
|
||||
curl -s $B/sql -H 'content-type: application/json' \
|
||||
-d '{"sql":"SELECT collection_id, COUNT(*) n FROM index GROUP BY 1 ORDER BY n DESC LIMIT 3"}'
|
||||
curl -s $B/cohort/counts -H 'content-type: application/json' \
|
||||
-d '{"filters":{"terms":{"collection_id":["rider_pilot"]}}}'
|
||||
```
|
||||
|
||||
**The filter object always goes under `filters`** — on `cohort/counts`, `cohort/manifest`,
|
||||
`cohort/manifest.txt`, `licenses`, and `citations` alike. A bare filter or an unrecognized key is
|
||||
a 422 naming the fix; an unfiltered series-enumerating request is a 400, not the whole archive.
|
||||
Every filtered response echoes `filters_applied` and `warnings` — read them, because they name
|
||||
any predicate the server dropped. A zero count with empty `warnings` therefore means the filter
|
||||
matched nothing, not that a value was miscased; miscasing produces a warning that says so.
|
||||
|
||||
`POST /sql` takes one read-only `SELECT`/`WITH` over the tables `idc-index` exposes plus
|
||||
`clinical.<table>`; `max_rows` defaults to 5 000, caps at 10 000, and `truncated` flags clipping.
|
||||
`GET /attributes` lists the 19 filterable attributes — clinical values, segmented anatomy, and
|
||||
acquisition parameters are not among them and need SQL. There is no rate limit or quota. **Use
|
||||
v3 only:** V1 and V2 are superseded and scheduled for shutdown, so port any `/v1/`- or
|
||||
`Modality_btw`-style example a user brings rather than extending it.
|
||||
|
||||
Both sides build on `idc-index-data`, so compare the API's `idc_index_data_version` against local
|
||||
`idc_index_data.__version__` before mixing them: the **major is the IDC data release** (`24.x.y`
|
||||
serves `v24`), so differing minor/patch means the series are identical. If the API is a whole
|
||||
release ahead, `idc-index` **cannot download the extra series** — it silently skips what its own
|
||||
index does not list — so either upgrade it (run `scripts/check_version.py` for the right command)
|
||||
or transfer directly from the bucket with `s5cmd --no-sign-request`.
|
||||
|
||||
See `references/rest_api_guide.md` for the endpoint reference, filter grounding, limits, and the
|
||||
manifest-based download flow.
|
||||
|
||||
**Cloud storage organization**
|
||||
|
||||
IDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.
|
||||
|
||||
| Bucket (AWS / GCS) | License | Content |
|
||||
|--------------------|---------|---------|
|
||||
| `idc-open-data` / `idc-open-data` | No commercial restriction | >90% of IDC data |
|
||||
| `idc-open-data-two` / `idc-open-idc1` | No commercial restriction | Collections with potential head scans |
|
||||
| `idc-open-data-cr` / `idc-open-cr` | Commercial use restricted (CC BY-NC) | ~4% of data |
|
||||
|
||||
Files are stored as `<crdc_series_uuid>/<crdc_instance_uuid>.dcm`. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use `series_aws_url` column from the index for S3 URLs; GCS uses the same path structure.
|
||||
|
||||
See `references/cloud_storage_guide.md` for bucket details, access commands, UUID mapping, and versioning.
|
||||
All DICOM files live in public buckets mirrored between AWS S3 and GCS, organized by CRDC UUIDs
|
||||
(not DICOM UIDs) to support versioning, as `<crdc_series_uuid>/<crdc_instance_uuid>.dcm`. Access
|
||||
is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access; use the
|
||||
`series_aws_url` column for S3 URLs. Note that `idc-open-data-cr` / `idc-open-cr` (~4% of data)
|
||||
is commercial-use restricted (CC BY-NC). See `references/cloud_storage_guide.md` for the full
|
||||
bucket list and UUID mapping.
|
||||
|
||||
**DICOMweb access**
|
||||
|
||||
IDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.
|
||||
|
||||
| Endpoint | Auth | Use Case |
|
||||
|----------|------|----------|
|
||||
| Public proxy | No | Testing, moderate queries, daily quota |
|
||||
| Google Healthcare | Yes (GCP) | Production use, higher quotas |
|
||||
|
||||
See `references/dicomweb_guide.md` for endpoint URLs, code examples, supported operations, and implementation details.
|
||||
IDC data is available via DICOMweb (Google Cloud Healthcare API) for PACS integration and
|
||||
DICOMweb-compatible tools: a public proxy (no auth, daily quota) for testing and moderate
|
||||
queries, or Google Healthcare (GCP auth) for production volumes. See
|
||||
`references/dicomweb_guide.md`.
|
||||
|
||||
**Direct Parquet access**
|
||||
|
||||
All idc-index metadata tables are published as Parquet files to a public GCS bucket (`idc-index-data-artifacts`) with unrestricted CORS. This enables DuckDB or pandas queries without installing idc-index, including cross-table joins and queries against `volume_geometry_index` and `rtstruct_index`.
|
||||
|
||||
See `references/parquet_access_guide.md` for URL patterns, available files, and DuckDB query examples.
|
||||
|
||||
## Installation and Setup
|
||||
|
||||
**Required (for basic access):** install into a virtual environment, pinned to the tested release:
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
|
||||
**Important:** every new IDC data release ships a new `idc-index`. Moving to a newer version
|
||||
changes which data your queries see, so treat it as a deliberate step: check the release notes,
|
||||
then pin the new version here. An unpinned `--upgrade` makes the data version a moving target and
|
||||
silently breaks reproducibility of an analysis you ran last month.
|
||||
|
||||
**IMPORTANT:** IDC data version v23 is current. Always verify your version:
|
||||
```python
|
||||
print(client.get_idc_version()) # Should return "v23"
|
||||
```
|
||||
If it returns an older version, tell the user which version they have and which one this skill was
|
||||
tested against, and let them decide whether to upgrade.
|
||||
|
||||
**Tested with:** idc-index 0.11.14 (IDC data version v23)
|
||||
|
||||
**Optional (for data analysis):**
|
||||
```bash
|
||||
uv pip install pandas numpy pydicom
|
||||
```
|
||||
The idc-index metadata tables are also published as Parquet on a public GCS bucket
|
||||
(`idc-index-data-artifacts`), queryable with DuckDB or pandas. This needs DuckDB installed
|
||||
and cannot reach the per-collection clinical tables, so prefer REST `/sql` for ad-hoc metadata;
|
||||
choose Parquet to pin a data version or for results past the REST row cap. See
|
||||
`references/parquet_access_guide.md`.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
Nine capability areas, each with worked code, are documented in
|
||||
[references/core_capabilities.md](references/core_capabilities.md):
|
||||
The patterns below are the ones that go wrong when recalled from memory rather than checked.
|
||||
Worked examples for each area live in the reference guides named inline.
|
||||
|
||||
1. **Data discovery and exploration** — summary statistics, and enumerating the actual
|
||||
`Modality` and `BodyPartExamined` values before filtering on them.
|
||||
2. **Querying metadata with SQL** — against `index`, `collections_index`, and
|
||||
`analysis_results_index`, returning pandas DataFrames.
|
||||
3. **Downloading DICOM files** — Python and CLI, by collection, series UID, or manifest,
|
||||
with control over directory hierarchy (full, simplified, or flat).
|
||||
4. **Visualizing IDC images** — single series or a whole study in the OHIF viewer.
|
||||
5. **Licenses and citations** — per-collection license checks, and citations in APA or
|
||||
BibTeX.
|
||||
6. **Batch processing and filtering** — scanner- and protocol-level filters, manifests,
|
||||
and batched downloads that avoid timeouts.
|
||||
7. **Advanced BigQuery queries** — for joins and aggregations beyond the index API.
|
||||
8. **Tool selection guide** — which access path fits which task.
|
||||
9. **Integration with analysis pipelines** — reading series with pydicom, processing with
|
||||
SimpleITK, and converting to NIfTI.
|
||||
### 1. Discovery — enumerate values before filtering on them
|
||||
|
||||
Always explore the real column values first: filtering on a guessed `Modality` or
|
||||
`BodyPartExamined` string is the most common cause of an empty result set.
|
||||
Filtering on a guessed `Modality` or `BodyPartExamined` string is the most common cause of an
|
||||
empty result set. Enumerate first:
|
||||
|
||||
## Common Use Cases
|
||||
```python
|
||||
modalities = client.sql_query("""
|
||||
SELECT DISTINCT Modality, COUNT(*) as series_count
|
||||
FROM index
|
||||
GROUP BY Modality
|
||||
ORDER BY series_count DESC
|
||||
""")
|
||||
print(modalities)
|
||||
```
|
||||
|
||||
See `references/use_cases.md` for complete end-to-end workflow examples including:
|
||||
- Building deep learning training datasets from lung CT scans
|
||||
- Comparing image quality across scanner manufacturers
|
||||
- Previewing data in browser before downloading
|
||||
- License-aware batch downloads for commercial use
|
||||
The same pattern works for any filter column, optionally narrowed by another —
|
||||
`BodyPartExamined` within a `Modality`, `Manufacturer`, `collection_id`. On the REST path this
|
||||
grounding is a single call — `GET /attributes/{attr}/values` returns values with counts — and the
|
||||
cohort endpoints report a miscased value in `warnings` rather than as an empty result.
|
||||
|
||||
Two indices carry curated collection-level metadata the primary `index` does not, both
|
||||
requiring `client.fetch_index(...)` first: `collections_index` (cancer types, tumor locations,
|
||||
species, subject counts) and `analysis_results_index` (derived datasets — AI segmentations,
|
||||
expert annotations, radiomics — with their source collections and modalities).
|
||||
|
||||
**Cancer type lives in `collections_index.cancer_types`, not in `index`** — filtering by
|
||||
cancer type requires a join:
|
||||
|
||||
```python
|
||||
client.fetch_index("collections_index")
|
||||
results = client.sql_query("""
|
||||
SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
|
||||
FROM index i
|
||||
JOIN collections_index c ON i.collection_id = c.collection_id
|
||||
WHERE c.cancer_types LIKE '%Breast%'
|
||||
AND i.Modality = 'MR'
|
||||
LIMIT 20
|
||||
""")
|
||||
```
|
||||
|
||||
`client.sql_query()` returns a pandas DataFrame. Confirm column names with
|
||||
`client.get_index_schema('index')` or `client.indices_overview` before writing a query rather
|
||||
than assuming them.
|
||||
|
||||
See `references/sql_patterns.md` for filter-value discovery, annotation and segmentation
|
||||
queries, size estimation, clinical linking, and version tracking ("what's new in vX" — use
|
||||
`series_init_idc_version` / `series_revised_idc_version` in `index`, never
|
||||
`prior_versions_index`).
|
||||
|
||||
### 2. Downloading DICOM files
|
||||
|
||||
**The two download methods take their first two arguments in opposite order.** This is the
|
||||
most common source of broken IDC code — check it rather than recalling it:
|
||||
|
||||
| Method | First arg | Second arg | Use when |
|
||||
|--------|-----------|------------|----------|
|
||||
| `download_from_selection` | `downloadDir` (required) | filter kwargs (optional) | Filtering by collection, patient, study, or series |
|
||||
| `download_dicom_series` | `seriesInstanceUID` (required) | `downloadDir` (required) | Downloading specific series by UID only |
|
||||
|
||||
**`download_from_selection` takes filter keyword arguments, NOT a DataFrame.** The name
|
||||
"from_selection" refers to filtering the IDC index by criteria — not to accepting a pandas
|
||||
DataFrame. To download query results, extract the UIDs into a list first:
|
||||
|
||||
```python
|
||||
# Step 1: Query for series UIDs
|
||||
series_df = client.sql_query("""
|
||||
SELECT SeriesInstanceUID
|
||||
FROM index
|
||||
WHERE Modality = 'CT'
|
||||
AND BodyPartExamined = 'CHEST'
|
||||
AND collection_id = 'nlst'
|
||||
LIMIT 5
|
||||
""")
|
||||
|
||||
# Step 2: Extract UIDs as a list from the DataFrame
|
||||
uids = list(series_df['SeriesInstanceUID'].values)
|
||||
|
||||
# Step 3: Pass the list to download_from_selection (NOT the DataFrame itself)
|
||||
client.download_from_selection(
|
||||
downloadDir="./data/lung_ct",
|
||||
seriesInstanceUID=uids # list of strings, not a DataFrame
|
||||
)
|
||||
|
||||
# Alternative: download_dicom_series has seriesInstanceUID as FIRST arg (different order!)
|
||||
client.download_dicom_series(
|
||||
seriesInstanceUID=uids, # FIRST arg here
|
||||
downloadDir="./data/lung_ct"
|
||||
)
|
||||
|
||||
# Whole collection: downloadDir is still the FIRST positional argument
|
||||
client.download_from_selection(downloadDir="./data/rider", collection_id="rider_pilot")
|
||||
```
|
||||
|
||||
Both methods default to AWS; pass `source_bucket_location="gcs"` to pull from Google Storage.
|
||||
|
||||
**Downloaded files are named `<crdc_instance_uuid>.dcm`, not by SOPInstanceUID.** The DICOM
|
||||
UIDs are preserved inside the file metadata, not in the filename. Use the `crdc_instance_uuid`
|
||||
column to map files back to the series they came from.
|
||||
|
||||
`idc download <collection|series-uid|manifest> --download-dir ./data` does the same from a
|
||||
shell. See `references/cli_guide.md` for the `dirTemplate` hierarchy options (Python default:
|
||||
`%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`; `dirTemplate=""`
|
||||
flattens), manifest downloads with resume, and dry-run size estimation.
|
||||
|
||||
### 3. Visualizing IDC images
|
||||
|
||||
```python
|
||||
viewer_url = client.get_viewer_URL(seriesInstanceUID=uid) # one series
|
||||
viewer_url = client.get_viewer_URL(studyInstanceUID=study_uid) # all series in a study
|
||||
```
|
||||
|
||||
Returns a browser URL — nothing is downloaded. The method selects OHIF v3 for radiology or
|
||||
SLIM for slide microscopy automatically. Viewing by study is useful when a single DICOM Study
|
||||
holds several Series (T1, T2, and DWI from one MRI session).
|
||||
|
||||
### 4. Licenses and citations — obligations, not optional steps
|
||||
|
||||
IDC data carries license terms and attribution requirements that follow it into any downstream
|
||||
publication or product, and neither is inferable from the pixel data. **Check the license
|
||||
before use, and generate citations for whatever you download.**
|
||||
|
||||
```python
|
||||
# License breakdown for a selection
|
||||
licenses = client.sql_query("""
|
||||
SELECT DISTINCT collection_id, license_short_name,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series_count
|
||||
FROM index GROUP BY collection_id, license_short_name
|
||||
""")
|
||||
|
||||
# Citations for the same selection you downloaded (APA by default)
|
||||
for citation in client.citations_from_selection(collection_id="rider_pilot"):
|
||||
print(citation)
|
||||
```
|
||||
|
||||
About 97% of IDC data is CC BY (commercial use allowed with attribution) and about 3% is
|
||||
CC BY-NC (non-commercial only). **Licenses attach to series, not collections** — 39 of 176
|
||||
collections carry more than one — so check the selection you actually intend to use, and note
|
||||
that the most restrictive term governs a mixed cohort.
|
||||
|
||||
Both tasks are available from all three access paths, so stay on whichever one the session is
|
||||
already using: `idc-index` as above, `POST /v3/licenses` and `POST /v3/citations` over REST,
|
||||
or the `get_licenses` and `get_citations` MCP tools. See
|
||||
`references/licensing_and_citation.md` for the full license inventory, all three routes, the
|
||||
citation formats (APA, BibTeX, CSL JSON, RDF Turtle), and what to include when publishing.
|
||||
|
||||
### 5. Reaching past the index
|
||||
|
||||
Pick the access path with the routing gate in *Overview*; *Data Access Options* above is the
|
||||
full routing table.
|
||||
|
||||
Before reaching for BigQuery (which needs a billing-enabled GCP account), check whether a
|
||||
specialized index table already has the column you want: search `client.indices_overview`,
|
||||
then `client.fetch_index(...)` and query locally for free. BigQuery is required only for
|
||||
private DICOM elements, per-segment anatomy (`segmentations`), and pre-extracted SR
|
||||
measurements (`quantitative_measurements`, `qualitative_measurements`) — these have no
|
||||
idc-index equivalent.
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Verify IDC version before generating responses** - Always call `client.get_idc_version()` at the start of a session to confirm you're using the expected data version (currently v23). If using an older version, report it and let the user decide whether to install a newer pinned release; never install on their behalf
|
||||
- **Check licenses before use** - Always query the `license_short_name` field and respect licensing terms (CC BY vs CC BY-NC)
|
||||
- **Generate citations for attribution** - Use `citations_from_selection()` to get properly formatted citations from `source_DOI` values; include these in publications
|
||||
- **Start with small queries** - Use `LIMIT` clause when exploring to avoid long downloads and understand data structure
|
||||
- **Use mini-index for simple queries** - Only use BigQuery when you need comprehensive metadata or complex JOINs
|
||||
- **Organize downloads with dirTemplate** - Use meaningful directory structures like `%collection_id/%PatientID/%Modality`
|
||||
- **Cache query results** - Save DataFrames to CSV files to avoid re-querying and ensure reproducibility
|
||||
- **Estimate size first** - Check collection size before downloading - some collection sizes are in terabytes!
|
||||
- **Save manifests** - Always save query results with Series UIDs for reproducibility and data provenance
|
||||
- **Read documentation** - IDC data structure and metadata fields are documented at https://learn.canceridc.dev/
|
||||
- **Use IDC forum** - Search for questions/answers and ask your questions to the IDC maintainers and users at https://discourse.canceridc.dev/
|
||||
- **Check schema before writing queries** — Use `client.get_index_schema('index')` (reads cached metadata, no SQL executed) or `client.indices_overview` to see all available columns and their descriptions. The version-tracking columns `series_init_idc_version` and `series_revised_idc_version` in the main `index` table directly answer "what's new / when was this added" questions without touching `prior_versions_index`.
|
||||
- **Never use web search for IDC data content questions** - Always query the IDC index directly, via `client.sql_query()` locally or `POST /v3/sql` over HTTP. Web sources (release notes, blog posts, documentation pages) are frequently out of date and will produce incorrect answers. The index is the authoritative source; use it even when web search is available.
|
||||
- **Verify the IDC data version at the start of a session** - `client.get_idc_version()`, `GET /v3/version`, or the MCP `get_idc_version` tool, depending on the path in use (currently v24). For a stale local index, run `scripts/check_version.py` and use the upgrade command it prints
|
||||
- **Check licenses and generate citations** - Query `license_short_name` and respect CC BY vs CC BY-NC terms; use `citations_from_selection()` to produce citations from `source_DOI` for publications
|
||||
- **Explore small, then commit** - Use `LIMIT` (or a low `max_rows`) while exploring, and check collection size before downloading — some collections are terabytes. See `references/cli_guide.md`
|
||||
- **Keep downloads reproducible** - Organize with `dirTemplate` (e.g. `%collection_id/%PatientID/%Modality`) and save the Series UIDs or manifest behind any dataset you build
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue: `ModuleNotFoundError: No module named 'idc_index'`**
|
||||
- **Cause:** idc-index package not installed
|
||||
- **Solution:** with the user's agreement, `uv pip install 'idc-index==0.11.14'` in a virtual environment
|
||||
- **Solution:** If the task is read-only metadata, do not install it — use the REST API instead (*Data Access Options*). Otherwise run `scripts/check_version.py` and use the install command it prints, which targets the running interpreter and pins the vetted version. For data analysis also add pandas, numpy, and pydicom (tested with pandas>=1.5, numpy>=1.23, pydicom>=2.3)
|
||||
|
||||
**Issue: Download fails with connection timeout**
|
||||
- **Cause:** Network instability or large download size
|
||||
- **Solution:**
|
||||
- Download smaller batches (e.g., 10-20 series at a time)
|
||||
- Check network connection
|
||||
- Use `dirTemplate` to organize downloads by batch
|
||||
- Implement retry logic with delays
|
||||
- **Solution:** Download in smaller batches (10-20 series); see `references/cli_guide.md` for
|
||||
`--use-s5cmd-sync` resume and retry guidance
|
||||
|
||||
**Issue: `BigQuery quota exceeded` or billing errors**
|
||||
- **Cause:** BigQuery requires billing-enabled GCP project
|
||||
- **Solution:** Use idc-index mini-index for simple queries (no billing required), or see `references/bigquery_guide.md` for cost optimization tips
|
||||
|
||||
**Issue: Series UID not found or no data returned**
|
||||
- **Cause:** Typo in UID, data not in current IDC version, or wrong field name
|
||||
- **Solution:**
|
||||
- Check if data is in current IDC version (some old data may be deprecated)
|
||||
- Use `LIMIT 5` to test query first
|
||||
- Check field names against metadata schema documentation
|
||||
- **Cause:** Typo in UID, data not in the current IDC version, or wrong field name
|
||||
- **Solution:** Test with `LIMIT 5` first, check field names against `client.indices_overview`,
|
||||
and confirm the series is in the current version (some old data is deprecated)
|
||||
|
||||
**Issue: Column not found in `index` table (e.g., `SliceThickness`, `PixelSpacing`, `KVP`, `EchoTime`, `InjectedDose`)**
|
||||
- **Cause:** The `index` table contains series-level metadata only; modality-specific acquisition and reconstruction parameters live in dedicated tables (`ct_index`, `mr_index`, `pt_index`)
|
||||
- **Solution:** Search `client.indices_overview` for the column to find its table — the loop is under *Finding which table contains a column* in `references/index_tables_guide.md` — then fetch and join on `SeriesInstanceUID`:
|
||||
```python
|
||||
client.fetch_index("ct_index")
|
||||
result = client.sql_query("""
|
||||
SELECT i.SeriesInstanceUID, i.Modality, c.SliceThickness, c.KVP, c.PixelSpacing_row_mm
|
||||
FROM index i
|
||||
JOIN ct_index c USING (SeriesInstanceUID)
|
||||
WHERE i.collection_id = 'your_collection'
|
||||
""")
|
||||
```
|
||||
|
||||
**Issue: Downloaded DICOM files won't open**
|
||||
- **Cause:** Corrupted download or incompatible viewer
|
||||
- **Solution:**
|
||||
- Check DICOM object type (Modality and SOPClassUID attributes) - some object types require specialized tools
|
||||
- Verify file integrity (check file sizes)
|
||||
- Use pydicom to validate: `pydicom.dcmread(file, force=True)`
|
||||
- Try different DICOM viewer (3D Slicer, Horos, RadiAnt, QuPath)
|
||||
- Re-download the series
|
||||
|
||||
## Common SQL Query Patterns
|
||||
|
||||
See `references/sql_patterns.md` for quick-reference SQL patterns including:
|
||||
- Filter value discovery (modalities, body parts, manufacturers)
|
||||
- Annotation and segmentation queries (including seg_index, ann_index joins)
|
||||
- Slide microscopy queries (sm_index patterns)
|
||||
- Download size estimation
|
||||
- Clinical data linking
|
||||
|
||||
For segmentation and annotation details, also see `references/digital_pathology_guide.md`.
|
||||
|
||||
## Related Skills
|
||||
|
||||
The following skills complement IDC workflows for downstream analysis and visualization:
|
||||
|
||||
### DICOM Processing
|
||||
- **pydicom** - Read, write, and manipulate downloaded DICOM files. Use for extracting pixel data, reading metadata, anonymization, and format conversion. Essential for working with IDC radiology data (CT, MR, PET).
|
||||
|
||||
### Pathology and Slide Microscopy
|
||||
See `references/digital_pathology_guide.md` for DICOM-compatible tools (highdicom, wsidicom, TIA-Toolbox, Slim viewer).
|
||||
|
||||
### Metadata Visualization
|
||||
- **matplotlib** - Low-level plotting for full customization. Use for creating static figures summarizing IDC query results (bar charts of modalities, histograms of series counts, etc.).
|
||||
- **seaborn** - Statistical visualization with pandas integration. Use for quick exploration of IDC metadata distributions, relationships between variables, and categorical comparisons with attractive defaults.
|
||||
- **plotly** - Interactive visualization. Use when you need hover info, zoom, and pan for exploring IDC metadata, or for creating web-embeddable dashboards of collection statistics.
|
||||
|
||||
### Data Exploration
|
||||
- **exploratory-data-analysis** - Comprehensive EDA on scientific data files. Use after downloading IDC data to understand file structure, quality, and characteristics before analysis.
|
||||
- **Cause:** Corrupted download, or an object type the viewer does not handle — SEG, RTSTRUCT,
|
||||
SR, and slide microscopy all need specialized tools
|
||||
- **Solution:** Check `Modality` and `SOPClassUID` first, validate with
|
||||
`pydicom.dcmread(file, force=True)`, try another viewer (3D Slicer, QuPath for pathology),
|
||||
then re-download
|
||||
|
||||
## Resources
|
||||
|
||||
### Schema Reference (Primary Source)
|
||||
|
||||
**Always use `client.indices_overview` for current column schemas.** This ensures accuracy with the installed idc-index version:
|
||||
|
||||
```python
|
||||
# Get all column names and types for any table
|
||||
schema = client.indices_overview["index"]["schema"]
|
||||
columns = [(c['name'], c['type'], c.get('description', '')) for c in schema['columns']]
|
||||
```
|
||||
|
||||
### Reference Documentation
|
||||
|
||||
See the Quick Navigation section at the top for the full list of reference guides with decision triggers.
|
||||
|
||||
- **[indices_reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html)** - External documentation for index tables (may be ahead of the installed version)
|
||||
|
||||
### External Links
|
||||
Reference guides and their decision triggers are listed in *Quick Navigation* above.
|
||||
|
||||
- **IDC Portal**: https://portal.imaging.datacommons.cancer.gov/explore/
|
||||
- **Documentation**: https://learn.canceridc.dev/
|
||||
- **Tutorials**: https://github.com/ImagingDataCommons/IDC-Tutorials
|
||||
- **User Forum**: https://discourse.canceridc.dev/
|
||||
- **idc-index GitHub**: https://github.com/ImagingDataCommons/idc-index
|
||||
- **Documentation**: https://learn.canceridc.dev/ — **Tutorials**: https://github.com/ImagingDataCommons/IDC-Tutorials
|
||||
- **User Forum**: https://discourse.canceridc.dev/ — **idc-index**: https://github.com/ImagingDataCommons/idc-index
|
||||
- **[indices_reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html)** — external index-table docs (may be ahead of the installed version)
|
||||
- **Citation**: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180
|
||||
|
||||
### Skill Updates
|
||||
|
||||
This skill version is available in skill metadata. To check for updates:
|
||||
- Visit the [releases page](https://github.com/ImagingDataCommons/idc-claude-skill/releases)
|
||||
- Watch the repository on GitHub (Watch → Custom → Releases)
|
||||
- **Skill updates**: [releases page](https://github.com/ImagingDataCommons/imaging-data-commons-skill/releases); watch the repository (Watch → Custom → Releases)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# BigQuery Guide for IDC
|
||||
|
||||
**Tested with:** IDC data version v23
|
||||
**Tested with:** `bigquery-public-data.idc_current` and idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
For most queries and downloads, use `idc-index` (see main SKILL.md). This guide covers BigQuery for advanced use cases requiring full DICOM metadata or complex joins.
|
||||
|
||||
@@ -648,7 +648,7 @@ ORDER BY collection_id
|
||||
- [Part 15 Appendix E - De-identification Profiles](https://dicom.nema.org/medical/dicom/current/output/chtml/part15/chapter_e.html)
|
||||
|
||||
**Community Resources:**
|
||||
- [NAMIC Wiki: DWI/DTI DICOM](https://www.na-mic.org/wiki/NAMIC_Wiki:DTI:DICOM_for_DWI_and_DTI) - comprehensive vendor comparison for diffusion imaging
|
||||
- [NAMIC Wiki: DWI/DTI DICOM](https://web.archive.org/web/20260520044207/https://www.na-mic.org/wiki/NAMIC_Wiki:DTI:DICOM_for_DWI_and_DTI) - comprehensive vendor comparison for diffusion imaging (archived; original NA-MIC wiki retired)
|
||||
- [StandardizeBValue](https://github.com/nslay/StandardizeBValue) - tool to extract vendor b-values to standard tags
|
||||
|
||||
## Using Query Results with idc-index
|
||||
@@ -660,7 +660,7 @@ from google.cloud import bigquery
|
||||
from idc_index import IDCClient
|
||||
|
||||
# Initialize BigQuery client
|
||||
# Requires: uv pip install google-cloud-bigquery
|
||||
# Requires: the Python google-cloud-bigquery package
|
||||
# Auth: gcloud auth application-default login
|
||||
# Project: needed for billing even on public datasets (free tier applies)
|
||||
bq_client = bigquery.Client(project="your-gcp-project-id")
|
||||
|
||||
@@ -4,9 +4,8 @@ The `idc-index` package provides command-line tools for downloading DICOM data f
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
|
||||
version and prints the install command for the interpreter you are running.
|
||||
|
||||
After installation, the `idc` command is available in your terminal.
|
||||
|
||||
@@ -50,6 +49,22 @@ idc download manifest.txt --download-dir ./data
|
||||
|
||||
### Directory Template Variables
|
||||
|
||||
The same templates apply in Python, where the argument is `dirTemplate=` rather than the
|
||||
`--dir-template` flag. The default is
|
||||
`%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`:
|
||||
|
||||
```python
|
||||
# Simplified hierarchy (omit StudyInstanceUID level)
|
||||
client.download_from_selection(
|
||||
downloadDir="./data",
|
||||
collection_id="tcga_luad",
|
||||
dirTemplate="%collection_id/%PatientID/%Modality"
|
||||
)
|
||||
# Results in: ./data/tcga_luad/TCGA-05-4244/CT/
|
||||
|
||||
# dirTemplate="" disables the hierarchy, writing every file straight into downloadDir
|
||||
```
|
||||
|
||||
Use these variables in `--dir-template` to organize downloads:
|
||||
|
||||
- `%collection_id` - Collection identifier
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Clinical Data Guide for IDC
|
||||
|
||||
**Tested with:** idc-index 0.11.7 (IDC data version v23)
|
||||
**Tested with:** idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
Clinical data (demographics, diagnoses, therapies, lab tests, staging) accompanies many IDC imaging collections. This guide covers how to discover, access, and integrate clinical data with imaging data using `idc-index`.
|
||||
|
||||
@@ -16,9 +16,8 @@ For basic clinical data access, see the "Clinical Data Access" section in the ma
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
|
||||
version and prints the install command for the interpreter you are running.
|
||||
|
||||
No BigQuery credentials required - clinical data is packaged with `idc-index`.
|
||||
|
||||
|
||||
@@ -205,7 +205,7 @@ IDC releases new data versions every 2-4 months. The versioning system ensures r
|
||||
|
||||
### How Versioning Works
|
||||
|
||||
1. **Snapshots**: Each IDC version (v1, v2, ..., v23, etc.) represents a complete snapshot of all data at release time
|
||||
1. **Snapshots**: Each IDC version (v1, v2, ..., v24, etc.) represents a complete snapshot of all data at release time
|
||||
2. **UUID-based**: When data changes, new CRDC UUIDs are assigned; old UUIDs remain accessible
|
||||
3. **Cumulative buckets**: All versions coexist in the same buckets—old series folders
|
||||
|
||||
@@ -223,7 +223,7 @@ IDC releases new data versions every 2-4 months. The versioning system ensures r
|
||||
|
||||
For querying version-specific metadata, BigQuery provides versioned tables. See `bigquery_guide.md` for details.
|
||||
- `bigquery-public-data.idc_current` — alias to latest version
|
||||
- `bigquery-public-data.idc_v23` — specific version (replace 23 with desired version)
|
||||
- `bigquery-public-data.idc_v24` — specific version (replace 24 with desired version)
|
||||
|
||||
### Reproducing a Previous Analysis
|
||||
|
||||
|
||||
@@ -1,502 +0,0 @@
|
||||
# IDC Core Capabilities
|
||||
|
||||
The nine capability areas in full, with worked code: data discovery and exploration,
|
||||
querying metadata with SQL, downloading DICOM files (Python and command line),
|
||||
visualizing images, checking licenses and generating citations, batch processing and
|
||||
filtering, advanced BigQuery use, the tool selection guide, and integration with
|
||||
analysis pipelines (pydicom, SimpleITK, NIfTI conversion).
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### 1. Data Discovery and Exploration
|
||||
|
||||
Discover what imaging collections and data are available in IDC:
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Get summary statistics from primary index
|
||||
query = """
|
||||
SELECT
|
||||
collection_id,
|
||||
COUNT(DISTINCT PatientID) as patients,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series,
|
||||
SUM(series_size_MB) as size_mb
|
||||
FROM index
|
||||
GROUP BY collection_id
|
||||
ORDER BY patients DESC
|
||||
"""
|
||||
collections_summary = client.sql_query(query)
|
||||
|
||||
# For richer collection metadata, use collections_index
|
||||
client.fetch_index("collections_index")
|
||||
collections_info = client.sql_query("""
|
||||
SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData
|
||||
FROM collections_index
|
||||
""")
|
||||
|
||||
# For analysis results (annotations, segmentations), use analysis_results_index
|
||||
client.fetch_index("analysis_results_index")
|
||||
analysis_info = client.sql_query("""
|
||||
SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities
|
||||
FROM analysis_results_index
|
||||
""")
|
||||
```
|
||||
|
||||
**`collections_index`** provides curated metadata per collection: cancer types, tumor locations, species, subject counts, and supporting data types — without needing to aggregate from the primary index.
|
||||
|
||||
**`analysis_results_index`** lists derived datasets (AI segmentations, expert annotations, radiomics features) with their source collections and modalities.
|
||||
|
||||
### 2. Querying Metadata with SQL
|
||||
|
||||
Query the IDC mini-index using SQL to find specific datasets.
|
||||
|
||||
**First, explore available values for filter columns:**
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Check what Modality values exist
|
||||
modalities = client.sql_query("""
|
||||
SELECT DISTINCT Modality, COUNT(*) as series_count
|
||||
FROM index
|
||||
GROUP BY Modality
|
||||
ORDER BY series_count DESC
|
||||
""")
|
||||
print(modalities)
|
||||
|
||||
# Check what BodyPartExamined values exist for MR modality
|
||||
body_parts = client.sql_query("""
|
||||
SELECT DISTINCT BodyPartExamined, COUNT(*) as series_count
|
||||
FROM index
|
||||
WHERE Modality = 'MR' AND BodyPartExamined IS NOT NULL
|
||||
GROUP BY BodyPartExamined
|
||||
ORDER BY series_count DESC
|
||||
LIMIT 20
|
||||
""")
|
||||
print(body_parts)
|
||||
```
|
||||
|
||||
**Then query with validated filter values:**
|
||||
```python
|
||||
# Find breast MRI scans (use actual values from exploration above)
|
||||
results = client.sql_query("""
|
||||
SELECT
|
||||
collection_id,
|
||||
PatientID,
|
||||
SeriesInstanceUID,
|
||||
Modality,
|
||||
SeriesDescription,
|
||||
license_short_name
|
||||
FROM index
|
||||
WHERE Modality = 'MR'
|
||||
AND BodyPartExamined = 'BREAST'
|
||||
LIMIT 20
|
||||
""")
|
||||
|
||||
# Access results as pandas DataFrame
|
||||
for idx, row in results.iterrows():
|
||||
print(f"Patient: {row['PatientID']}, Series: {row['SeriesInstanceUID']}")
|
||||
```
|
||||
|
||||
**To filter by cancer type, join with `collections_index`:**
|
||||
```python
|
||||
client.fetch_index("collections_index")
|
||||
results = client.sql_query("""
|
||||
SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
|
||||
FROM index i
|
||||
JOIN collections_index c ON i.collection_id = c.collection_id
|
||||
WHERE c.CancerTypes LIKE '%Breast%'
|
||||
AND i.Modality = 'MR'
|
||||
LIMIT 20
|
||||
""")
|
||||
```
|
||||
|
||||
**Available metadata fields** (use `client.indices_overview` for complete list):
|
||||
- Identifiers: collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID
|
||||
- Imaging: Modality, BodyPartExamined, Manufacturer, ManufacturerModelName
|
||||
- Clinical: PatientAge, PatientSex, StudyDate
|
||||
- Descriptions: StudyDescription, SeriesDescription
|
||||
- Licensing: license_short_name
|
||||
|
||||
**Note:** Cancer type is in `collections_index.CancerTypes`, not in the primary `index` table.
|
||||
|
||||
### 3. Downloading DICOM Files
|
||||
|
||||
Download imaging data efficiently from IDC's cloud storage:
|
||||
|
||||
**Download entire collection:**
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Download small collection (RIDER Pilot ~1GB)
|
||||
client.download_from_selection(
|
||||
collection_id="rider_pilot",
|
||||
downloadDir="./data/rider"
|
||||
)
|
||||
```
|
||||
|
||||
**Download specific series:**
|
||||
```python
|
||||
# First, query for series UIDs
|
||||
series_df = client.sql_query("""
|
||||
SELECT SeriesInstanceUID
|
||||
FROM index
|
||||
WHERE Modality = 'CT'
|
||||
AND BodyPartExamined = 'CHEST'
|
||||
AND collection_id = 'nlst'
|
||||
LIMIT 5
|
||||
""")
|
||||
|
||||
# Download only those series
|
||||
client.download_from_selection(
|
||||
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
|
||||
downloadDir="./data/lung_ct"
|
||||
)
|
||||
```
|
||||
|
||||
**Custom directory structure:**
|
||||
|
||||
Default `dirTemplate`: `%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`
|
||||
|
||||
```python
|
||||
# Simplified hierarchy (omit StudyInstanceUID level)
|
||||
client.download_from_selection(
|
||||
collection_id="tcga_luad",
|
||||
downloadDir="./data",
|
||||
dirTemplate="%collection_id/%PatientID/%Modality"
|
||||
)
|
||||
# Results in: ./data/tcga_luad/TCGA-05-4244/CT/
|
||||
|
||||
# Flat structure (all files in one directory)
|
||||
client.download_from_selection(
|
||||
seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
|
||||
downloadDir="./data/flat",
|
||||
dirTemplate=""
|
||||
)
|
||||
# Results in: ./data/flat/*.dcm
|
||||
```
|
||||
|
||||
**Downloaded file names:**
|
||||
|
||||
Individual DICOM files are named using their CRDC instance UUID: `<crdc_instance_uuid>.dcm` (e.g., `0d73f84e-70ae-4eeb-96a0-1c613b5d9229.dcm`). This UUID-based naming:
|
||||
- Enables version tracking (UUIDs change when file content changes)
|
||||
- Matches cloud storage organization (`s3://idc-open-data/<crdc_series_uuid>/<crdc_instance_uuid>.dcm`)
|
||||
- Differs from DICOM UIDs (SOPInstanceUID) which are preserved inside the file metadata
|
||||
|
||||
To identify files, use the `crdc_instance_uuid` column in queries or read DICOM metadata (SOPInstanceUID) from the files.
|
||||
|
||||
### Command-Line Download
|
||||
|
||||
The `idc download` command provides command-line access to download functionality without writing Python code. Available after installing `idc-index`.
|
||||
|
||||
**Auto-detects input type:** manifest file path, or identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).
|
||||
|
||||
```bash
|
||||
# Download entire collection
|
||||
idc download rider_pilot --download-dir ./data
|
||||
|
||||
# Download specific series by UID
|
||||
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data
|
||||
|
||||
# Download multiple items (comma-separated)
|
||||
idc download "tcga_luad,tcga_lusc" --download-dir ./data
|
||||
|
||||
# Download from manifest file (auto-detected)
|
||||
idc download manifest.txt --download-dir ./data
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--download-dir` | Output directory (default: current directory) |
|
||||
| `--dir-template` | Directory hierarchy template (default: `%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`) |
|
||||
| `--log-level` | Verbosity: debug, info, warning, error, critical |
|
||||
|
||||
**Manifest files:**
|
||||
|
||||
Manifest files contain S3 URLs (one per line) and can be:
|
||||
- Exported from the IDC Portal after cohort selection
|
||||
- Shared by collaborators for reproducible data access
|
||||
- Generated programmatically from query results
|
||||
|
||||
Format (one S3 URL per line):
|
||||
```
|
||||
s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
|
||||
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*
|
||||
```
|
||||
|
||||
**Example: Generate manifest from Python query:**
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Query for series URLs
|
||||
results = client.sql_query("""
|
||||
SELECT series_aws_url
|
||||
FROM index
|
||||
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
|
||||
""")
|
||||
|
||||
# Save as manifest file
|
||||
with open('ct_manifest.txt', 'w') as f:
|
||||
for url in results['series_aws_url']:
|
||||
f.write(url + '\n')
|
||||
```
|
||||
|
||||
Then download:
|
||||
```bash
|
||||
idc download ct_manifest.txt --download-dir ./ct_data
|
||||
```
|
||||
|
||||
### 4. Visualizing IDC Images
|
||||
|
||||
View DICOM data in browser without downloading:
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
import webbrowser
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# First query to get valid UIDs
|
||||
results = client.sql_query("""
|
||||
SELECT SeriesInstanceUID, StudyInstanceUID
|
||||
FROM index
|
||||
WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
|
||||
LIMIT 1
|
||||
""")
|
||||
|
||||
# View single series
|
||||
viewer_url = client.get_viewer_URL(seriesInstanceUID=results.iloc[0]['SeriesInstanceUID'])
|
||||
webbrowser.open(viewer_url)
|
||||
|
||||
# View all series in a study (useful for multi-series exams like MRI protocols)
|
||||
viewer_url = client.get_viewer_URL(studyInstanceUID=results.iloc[0]['StudyInstanceUID'])
|
||||
webbrowser.open(viewer_url)
|
||||
```
|
||||
|
||||
The method automatically selects OHIF v3 for radiology or SLIM for slide microscopy. Viewing by study is useful when a DICOM Study contains multiple Series (e.g., T1, T2, DWI sequences from a single MRI session).
|
||||
|
||||
### 5. Understanding and Checking Licenses
|
||||
|
||||
Check data licensing before use (critical for commercial applications):
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Check licenses for all collections
|
||||
query = """
|
||||
SELECT DISTINCT
|
||||
collection_id,
|
||||
license_short_name,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series_count
|
||||
FROM index
|
||||
GROUP BY collection_id, license_short_name
|
||||
ORDER BY collection_id
|
||||
"""
|
||||
|
||||
licenses = client.sql_query(query)
|
||||
print(licenses)
|
||||
```
|
||||
|
||||
**License types in IDC:**
|
||||
- **CC BY 4.0** / **CC BY 3.0** (~97% of data) - Allows commercial use with attribution
|
||||
- **CC BY-NC 4.0** / **CC BY-NC 3.0** (~3% of data) - Non-commercial use only
|
||||
- **Custom licenses** (rare) - Some collections have specific terms (e.g., NLM Terms and Conditions)
|
||||
|
||||
**Important:** Always check the license before using IDC data in publications or commercial applications. Each DICOM file is tagged with its specific license in metadata.
|
||||
|
||||
### Generating Citations for Attribution
|
||||
|
||||
The `source_DOI` column contains DOIs linking to publications describing how the data was generated. To satisfy attribution requirements, use `citations_from_selection()` to generate properly formatted citations:
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Get citations for a collection (APA format by default)
|
||||
citations = client.citations_from_selection(collection_id="rider_pilot")
|
||||
for citation in citations:
|
||||
print(citation)
|
||||
|
||||
# Get citations for specific series
|
||||
results = client.sql_query("""
|
||||
SELECT SeriesInstanceUID FROM index
|
||||
WHERE collection_id = 'tcga_luad' LIMIT 5
|
||||
""")
|
||||
citations = client.citations_from_selection(
|
||||
seriesInstanceUID=list(results['SeriesInstanceUID'].values)
|
||||
)
|
||||
|
||||
# Alternative format: BibTeX (for LaTeX documents)
|
||||
bibtex_citations = client.citations_from_selection(
|
||||
collection_id="tcga_luad",
|
||||
citation_format=IDCClient.CITATION_FORMAT_BIBTEX
|
||||
)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `collection_id`: Filter by collection(s)
|
||||
- `patientId`: Filter by patient ID(s)
|
||||
- `studyInstanceUID`: Filter by study UID(s)
|
||||
- `seriesInstanceUID`: Filter by series UID(s)
|
||||
- `citation_format`: Use `IDCClient.CITATION_FORMAT_*` constants:
|
||||
- `CITATION_FORMAT_APA` (default) - APA style
|
||||
- `CITATION_FORMAT_BIBTEX` - BibTeX for LaTeX
|
||||
- `CITATION_FORMAT_JSON` - CSL JSON
|
||||
- `CITATION_FORMAT_TURTLE` - RDF Turtle
|
||||
|
||||
**Best practice:** When publishing results using IDC data, include the generated citations to properly attribute the data sources and satisfy license requirements.
|
||||
|
||||
### 6. Batch Processing and Filtering
|
||||
|
||||
Process large datasets efficiently with filtering:
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
import pandas as pd
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Find chest CT scans from GE scanners
|
||||
query = """
|
||||
SELECT
|
||||
SeriesInstanceUID,
|
||||
PatientID,
|
||||
collection_id,
|
||||
ManufacturerModelName
|
||||
FROM index
|
||||
WHERE Modality = 'CT'
|
||||
AND BodyPartExamined = 'CHEST'
|
||||
AND Manufacturer = 'GE MEDICAL SYSTEMS'
|
||||
AND license_short_name = 'CC BY 4.0'
|
||||
LIMIT 100
|
||||
"""
|
||||
|
||||
results = client.sql_query(query)
|
||||
|
||||
# Save manifest for later
|
||||
results.to_csv('lung_ct_manifest.csv', index=False)
|
||||
|
||||
# Download in batches to avoid timeout
|
||||
batch_size = 10
|
||||
for i in range(0, len(results), batch_size):
|
||||
batch = results.iloc[i:i+batch_size]
|
||||
client.download_from_selection(
|
||||
seriesInstanceUID=list(batch['SeriesInstanceUID'].values),
|
||||
downloadDir=f"./data/batch_{i//batch_size}"
|
||||
)
|
||||
```
|
||||
|
||||
### 7. Advanced Queries with BigQuery
|
||||
|
||||
For queries requiring full DICOM metadata, complex JOINs, clinical data tables, or private DICOM elements, use Google BigQuery. Requires GCP account with billing enabled.
|
||||
|
||||
**Quick reference:**
|
||||
- Dataset: `bigquery-public-data.idc_current.*`
|
||||
- Main table: `dicom_all` (combined metadata)
|
||||
- Full metadata: `dicom_metadata` (all DICOM tags)
|
||||
- Private elements: `OtherElements` column (vendor-specific tags like diffusion b-values)
|
||||
|
||||
See `references/bigquery_guide.md` for setup, table schemas, query patterns, private element access, and cost optimization.
|
||||
|
||||
**Before using BigQuery**, always check if a specialized index table already has the metadata you need:
|
||||
1. Use `client.indices_overview` or the [idc-index indices reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html) to discover all available tables and their columns
|
||||
2. Fetch the relevant index: `client.fetch_index("table_name")`
|
||||
3. Query locally with `client.sql_query()` (free, no GCP account needed)
|
||||
|
||||
Common specialized indices: `seg_index` (segmentations), `ann_index` / `ann_group_index` (microscopy annotations), `sm_index` (slide microscopy), `collections_index` (collection metadata). Only use BigQuery if you need private DICOM elements or attributes not in any index.
|
||||
|
||||
**Use cases that require BigQuery (no idc-index equivalent):**
|
||||
- **Per-segment anatomy search** — `seg_index` gives series-level SEG metadata, but the BigQuery `segmentations` table exposes each segment individually with its DICOM coded structure name (e.g., find all SEG series containing a "Liver" or "Neoplasm" segment)
|
||||
- **Quantitative measurements from SR** — the `quantitative_measurements` BigQuery table contains pre-extracted radiomics features (volume, diameter, shape descriptors, texture, intensity statistics) from DICOM SR TID1500 objects; no idc-index equivalent
|
||||
- **Qualitative measurements from SR** — the `qualitative_measurements` BigQuery table contains coded assessments (malignancy rating, calcification, texture, margin) from DICOM SR TID1500; no idc-index equivalent
|
||||
|
||||
See `references/bigquery_guide.md` for schemas, column descriptions, and query examples for these tables.
|
||||
|
||||
### 8. Tool Selection Guide
|
||||
|
||||
| Task | Tool | Reference |
|
||||
|------|------|-----------|
|
||||
| Programmatic queries & downloads | `idc-index` | This document |
|
||||
| Interactive exploration | IDC Portal | https://portal.imaging.datacommons.cancer.gov/ |
|
||||
| Complex metadata queries | BigQuery | `references/bigquery_guide.md` |
|
||||
| 3D visualization & analysis | SlicerIDCBrowser | https://github.com/ImagingDataCommons/SlicerIDCBrowser |
|
||||
|
||||
**Default choice:** Use `idc-index` for most tasks (no auth, easy API, batch downloads).
|
||||
|
||||
### 9. Integration with Analysis Pipelines
|
||||
|
||||
Integrate IDC data into imaging analysis workflows:
|
||||
|
||||
**Read downloaded DICOM files:**
|
||||
```python
|
||||
import pydicom
|
||||
import os
|
||||
|
||||
# Read DICOM files from downloaded series
|
||||
series_dir = "./data/rider/rider_pilot/RIDER-1007893286/CT_1.3.6.1..."
|
||||
|
||||
dicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir)
|
||||
if f.endswith('.dcm')]
|
||||
|
||||
# Load first image
|
||||
ds = pydicom.dcmread(dicom_files[0])
|
||||
print(f"Patient ID: {ds.PatientID}")
|
||||
print(f"Modality: {ds.Modality}")
|
||||
print(f"Image shape: {ds.pixel_array.shape}")
|
||||
```
|
||||
|
||||
**Build 3D volume from CT series:**
|
||||
```python
|
||||
import pydicom
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
def load_ct_series(series_path):
|
||||
"""Load CT series as 3D numpy array"""
|
||||
files = sorted(Path(series_path).glob('*.dcm'))
|
||||
slices = [pydicom.dcmread(str(f)) for f in files]
|
||||
|
||||
# Sort by slice location
|
||||
slices.sort(key=lambda x: float(x.ImagePositionPatient[2]))
|
||||
|
||||
# Stack into 3D array
|
||||
volume = np.stack([s.pixel_array for s in slices])
|
||||
|
||||
return volume, slices[0] # Return volume and first slice for metadata
|
||||
|
||||
volume, metadata = load_ct_series("./data/lung_ct/series_dir")
|
||||
print(f"Volume shape: {volume.shape}") # (z, y, x)
|
||||
```
|
||||
|
||||
**Integrate with SimpleITK:**
|
||||
```python
|
||||
import SimpleITK as sitk
|
||||
from pathlib import Path
|
||||
|
||||
# Read DICOM series
|
||||
series_path = "./data/ct_series"
|
||||
reader = sitk.ImageSeriesReader()
|
||||
dicom_names = reader.GetGDCMSeriesFileNames(series_path)
|
||||
reader.SetFileNames(dicom_names)
|
||||
image = reader.Execute()
|
||||
|
||||
# Apply processing
|
||||
smoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5)
|
||||
|
||||
# Save as NIfTI
|
||||
sitk.WriteImage(smoothed, "processed_volume.nii.gz")
|
||||
```
|
||||
@@ -39,7 +39,7 @@ Replace `{VERSION}` with the IDC release number. To find the current version:
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
print(client.get_idc_version()) # e.g., "23" for v23
|
||||
print(client.get_idc_version()) # e.g., "v24" for current version
|
||||
```
|
||||
|
||||
- **~96% data coverage** - Only replicates data from `idc-open-data` bucket (missing ~4% from other buckets)
|
||||
@@ -334,7 +334,7 @@ credentials, project = default()
|
||||
credentials.refresh(Request())
|
||||
|
||||
# Build authenticated request
|
||||
base_url = "https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v23/dicomWeb"
|
||||
base_url = "https://healthcare.googleapis.com/v1/projects/nci-idc-data/locations/us-central1/datasets/idc/dicomStores/idc-store-v24/dicomWeb"
|
||||
|
||||
response = requests.get(
|
||||
f"{base_url}/studies",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Digital Pathology Guide for IDC
|
||||
|
||||
**Tested with:** IDC data version v23, idc-index 0.11.10
|
||||
**Tested with:** idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
For general IDC queries and downloads, use `idc-index` (see main SKILL.md). This guide covers slide microscopy (SM) imaging, microscopy bulk simple annotations (ANN), and segmentations (SEG) in the context of digital pathology in IDC.
|
||||
|
||||
@@ -251,12 +251,12 @@ client.sql_query("""
|
||||
SELECT
|
||||
ar.analysis_result_id,
|
||||
ar.analysis_result_title,
|
||||
ar.Modalities,
|
||||
ar.Subjects,
|
||||
ar.Collections
|
||||
ar.modalities,
|
||||
ar.subjects,
|
||||
ar.collections
|
||||
FROM analysis_results_index ar
|
||||
WHERE ar.Modalities LIKE '%ANN%' OR ar.Modalities LIKE '%SEG%'
|
||||
ORDER BY ar.Subjects DESC
|
||||
WHERE ar.modalities LIKE '%ANN%' OR ar.modalities LIKE '%SEG%'
|
||||
ORDER BY ar.subjects DESC
|
||||
""")
|
||||
```
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Index Tables Guide for IDC
|
||||
|
||||
**Tested with:** idc-index 0.11.14 (IDC data version v23)
|
||||
**Tested with:** idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
This guide covers the structure and access patterns for IDC index tables: programmatic schema discovery, DataFrame access, and join column references. For the overview of available tables and their purposes, see the "Index Tables" section in the main SKILL.md.
|
||||
|
||||
@@ -17,9 +17,35 @@ For SQL query examples (filter discovery, finding annotations, size estimation),
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
|
||||
version and prints the install command for the interpreter you are running.
|
||||
|
||||
## Available Tables
|
||||
|
||||
`SKILL.md` carries a compact map of the table families. This is the full inventory with row
|
||||
granularity and contents. Always call `client.fetch_index("table_name")` before querying any
|
||||
of them — it is safe and idempotent for all tables, including those loaded automatically at
|
||||
startup.
|
||||
|
||||
| Table | Row Granularity | Description |
|
||||
|-------|-----------------|-------------|
|
||||
| `index` | 1 row = 1 DICOM series | Primary metadata for all current IDC data |
|
||||
| `version_metadata_index` | 1 row = 1 IDC release version | IDC version release timestamps; join on `idc_version` to correlate series with their release date |
|
||||
| `collections_index` | 1 row = 1 collection | Collection-level metadata and descriptions |
|
||||
| `analysis_results_index` | 1 row = 1 analysis result collection | Metadata about derived datasets (annotations, segmentations) |
|
||||
| `clinical_index` | 1 row = 1 (collection, table, column) triple | Dictionary mapping clinical data table columns to collections |
|
||||
| `sm_index` | 1 row = 1 slide microscopy series | Slide Microscopy (pathology) series metadata |
|
||||
| `sm_instance_index` | 1 row = 1 slide microscopy instance | Instance-level (SOPInstanceUID) metadata for slide microscopy |
|
||||
| `seg_index` | 1 row = 1 DICOM Segmentation series | Segmentation metadata: algorithm, segment count, reference to source image series |
|
||||
| `ann_index` | 1 row = 1 DICOM ANN series | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
|
||||
| `ann_group_index` | 1 row = 1 annotation group | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
|
||||
| `contrast_index` | 1 row = 1 series with contrast info | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
|
||||
| `volume_geometry_index` | 1 row = 1 CT/MR/PT series | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag |
|
||||
| `rtstruct_index` | 1 row = 1 RTSTRUCT series | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
|
||||
| `ct_index` | 1 row = 1 CT series | CT acquisition/reconstruction parameters: pixel spacing, slice thickness, kVp, convolution kernel, tube current (min/max for dose-modulated), exposure, spiral pitch, scan options |
|
||||
| `mr_index` | 1 row = 1 MR series | MR acquisition/sequence parameters: field strength, scanning sequence, TE (array for multi-echo), TR, flip angle, DiffusionBValue (array for DWI), pixel bandwidth, receive coil, number of temporal positions |
|
||||
| `pt_index` | 1 row = 1 PET series | PET acquisition/reconstruction/radiopharmaceutical parameters: series type, units, decay/scatter/attenuation correction, reconstruction method, radionuclide, injected dose, frame duration (array for dynamic PET) |
|
||||
| `prior_versions_index` | 1 row = 1 DICOM series | **Reproducibility only.** Contains series permanently removed from IDC (all `max_idc_version` < current version; zero overlap with `index`). Use ONLY when a user explicitly needs to reproduce work from a prior IDC version using data no longer in the current release. Do NOT use for version history or "what's new" questions — those use `series_init_idc_version`/`series_revised_idc_version` in the main `index` table. Column names `min_idc_version`/`max_idc_version` here are NOT equivalent to `series_init_idc_version`/`series_revised_idc_version` in `index`. |
|
||||
|
||||
## Accessing Index Tables
|
||||
|
||||
@@ -34,7 +60,7 @@ results = client.sql_query("SELECT * FROM index WHERE Modality = 'CT' LIMIT 10")
|
||||
|
||||
# Fetch and query additional indices
|
||||
client.fetch_index("collections_index")
|
||||
collections = client.sql_query("SELECT collection_id, CancerTypes, TumorLocations FROM collections_index")
|
||||
collections = client.sql_query("SELECT collection_id, cancer_types, tumor_locations FROM collections_index")
|
||||
|
||||
client.fetch_index("analysis_results_index")
|
||||
analysis = client.sql_query("SELECT * FROM analysis_results_index LIMIT 5")
|
||||
@@ -87,6 +113,29 @@ schema = client.get_index_schema("index")
|
||||
# Returns same schema dict: {'table_description': ..., 'columns': [...]}
|
||||
```
|
||||
|
||||
### Finding which table contains a column
|
||||
|
||||
The most common schema question is "where does `SliceThickness` live?" — the primary `index`
|
||||
holds series-level metadata only, so modality-specific acquisition parameters are in dedicated
|
||||
tables. Search the overview rather than guessing; neither call fetches anything:
|
||||
|
||||
```python
|
||||
# Find which table(s) contain a specific column (no fetch required)
|
||||
target = "SliceThickness"
|
||||
for table_name, info in client.indices_overview.items():
|
||||
if any(c["name"] == target for c in info["schema"]["columns"]):
|
||||
print(f"'{target}' is in: {table_name}")
|
||||
# → 'SliceThickness' is in: ct_index
|
||||
|
||||
# List all columns in a table from the schema (no fetch required)
|
||||
ct_cols = [c["name"] for c in client.indices_overview["ct_index"]["schema"]["columns"]]
|
||||
print("ct_index columns:", ct_cols)
|
||||
# → ['SeriesInstanceUID', 'PixelSpacing_row_mm', 'PixelSpacing_col_mm', 'Rows',
|
||||
# 'Columns', 'SliceThickness', 'KVP', 'ConvolutionKernel', ...]
|
||||
```
|
||||
|
||||
Then `client.fetch_index("ct_index")` and join to `index` on `SeriesInstanceUID`.
|
||||
|
||||
## Key Columns Reference
|
||||
|
||||
Most common columns in the primary `index` table (use `indices_overview` for complete list and descriptions):
|
||||
@@ -130,6 +179,9 @@ Use this table to identify join columns between index tables. Always call `clien
|
||||
| `index` | `volume_geometry_index` | `index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID` |
|
||||
| `index` | `rtstruct_index` | `index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID` |
|
||||
| `rtstruct_index` | `index` (source images) | `rtstruct_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID` |
|
||||
| `index` | `ct_index` | `index.SeriesInstanceUID = ct_index.SeriesInstanceUID` |
|
||||
| `index` | `mr_index` | `index.SeriesInstanceUID = mr_index.SeriesInstanceUID` |
|
||||
| `index` | `pt_index` | `index.SeriesInstanceUID = pt_index.SeriesInstanceUID` |
|
||||
|
||||
For complete query examples using these joins, see `references/sql_patterns.md`.
|
||||
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
# Licensing and Citation Guide for IDC
|
||||
|
||||
## When to Use This Guide
|
||||
|
||||
Load this guide when:
|
||||
- A user asks whether IDC data can be used commercially, redistributed, or included in a product
|
||||
- You are assembling a cohort that mixes collections and need to know which terms govern it
|
||||
- A user is publishing results and needs formatted citations (APA, BibTeX, CSL JSON, RDF Turtle)
|
||||
- You need the parameters or output formats for citation generation
|
||||
|
||||
The obligation summary — check the license, generate citations — lives in `SKILL.md`. This
|
||||
guide holds the detail behind it.
|
||||
|
||||
**These are the two IDC tasks least tied to any one access path.** Licenses and citations are
|
||||
available identically from `idc-index`, the REST API, and the hosted MCP server. Use whichever
|
||||
route the session is already on rather than installing Python to answer a licensing question,
|
||||
or dropping out of an MCP session to run a script.
|
||||
|
||||
| Task | `idc-index` (Python) | REST API | MCP server |
|
||||
|------|----------------------|----------|------------|
|
||||
| License breakdown for a selection | `sql_query` on `license_short_name` | `POST /v3/licenses` | `get_licenses` |
|
||||
| Citations for a selection | `citations_from_selection()` | `POST /v3/citations` | `get_citations` |
|
||||
|
||||
Route-specific detail lives in `references/rest_api_guide.md` (endpoint reference, filter
|
||||
syntax, and the body-shape pitfall that makes a mis-shaped filter return all of IDC) and
|
||||
`references/mcp_guide.md` (tool inventory). The license semantics below apply to all three.
|
||||
|
||||
## Licenses in IDC
|
||||
|
||||
Every DICOM file in IDC is tagged with its license in the file metadata, and every row in the
|
||||
`index` table carries a `license_short_name` column. There is no single IDC-wide license.
|
||||
|
||||
| License | Share of data | Commercial use | Attribution required |
|
||||
|---------|---------------|----------------|----------------------|
|
||||
| CC BY 4.0 | 74.7% | Yes | Yes |
|
||||
| CC BY 3.0 | 22.1% | Yes | Yes |
|
||||
| CC BY-NC 4.0 | 2.1% | **No** | Yes |
|
||||
| CC BY-NC 3.0 | 0.8% | **No** | Yes |
|
||||
| NLM Terms and Conditions | 0.3% | Read the terms | Yes |
|
||||
|
||||
About 97% of IDC data by size permits commercial reuse; just under 3% is non-commercial. Treat
|
||||
any `license_short_name` that is not a recognizable Creative Commons string as custom, and
|
||||
report the exact value to the user rather than assuming it permits reuse.
|
||||
|
||||
**Licenses attach to individual series, not to whole collections.** 39 of IDC's 176 collections
|
||||
carry more than one license — analysis results and original images within one collection can
|
||||
differ, as can series from different sources. Never conclude that a collection is
|
||||
commercially usable from one series, or from the collection's headline license: group by
|
||||
`license_short_name` over the exact selection you intend to use.
|
||||
|
||||
**When a cohort mixes licenses, the most restrictive term governs the combined dataset.** If a
|
||||
selection contains any CC BY-NC series, either drop those series or tell the user the whole
|
||||
derived dataset is non-commercial.
|
||||
|
||||
Commercially restricted data is also physically separated in cloud storage: the
|
||||
`idc-open-data-cr` (AWS) / `idc-open-cr` (GCS) buckets hold the CC BY-NC collections. See
|
||||
`references/cloud_storage_guide.md` for bucket details.
|
||||
|
||||
## Checking licenses
|
||||
|
||||
### Via `idc-index`
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
|
||||
# Licenses across all collections
|
||||
licenses = client.sql_query("""
|
||||
SELECT DISTINCT
|
||||
collection_id,
|
||||
license_short_name,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series_count
|
||||
FROM index
|
||||
GROUP BY collection_id, license_short_name
|
||||
ORDER BY collection_id
|
||||
""")
|
||||
print(licenses)
|
||||
```
|
||||
|
||||
```python
|
||||
# Licenses present in one specific cohort — run this before handing a dataset to a user
|
||||
cohort_licenses = client.sql_query("""
|
||||
SELECT license_short_name, COUNT(DISTINCT SeriesInstanceUID) as series_count
|
||||
FROM index
|
||||
WHERE Modality = 'MR' AND BodyPartExamined = 'BREAST'
|
||||
GROUP BY license_short_name
|
||||
""")
|
||||
print(cohort_licenses)
|
||||
```
|
||||
|
||||
```python
|
||||
# Commercial-safe subset: exclude non-commercial collections outright
|
||||
commercial_ok = client.sql_query("""
|
||||
SELECT collection_id, SeriesInstanceUID
|
||||
FROM index
|
||||
WHERE Modality = 'CT'
|
||||
AND license_short_name NOT LIKE '%NC%'
|
||||
LIMIT 20
|
||||
""")
|
||||
```
|
||||
|
||||
### Via the REST API
|
||||
|
||||
`POST /v3/licenses` takes the filter object **directly** (not wrapped in a `filters` key) and
|
||||
returns the per-license breakdown with series counts and sizes:
|
||||
|
||||
```bash
|
||||
B=https://api.imaging.datacommons.cancer.gov/v3
|
||||
curl -s $B/licenses \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}'
|
||||
```
|
||||
|
||||
Response shape: `licenses[{license_short_name, series, size_TB}]`. A collection's licenses are
|
||||
also included in `GET /v3/collections/{id}`.
|
||||
|
||||
### Via the MCP server
|
||||
|
||||
Call `get_licenses` with the same selection you built with `build_cohort`. The result carries
|
||||
the same per-license breakdown; the CC BY vs CC BY-NC distinction above applies unchanged.
|
||||
|
||||
## Citations and attribution
|
||||
|
||||
The `source_DOI` column links to the publications describing how each dataset was generated.
|
||||
All three routes turn a selection into formatted citations that satisfy the attribution
|
||||
requirement common to every IDC license.
|
||||
|
||||
Generate citations from the *same* selection you downloaded, not from the collection as a
|
||||
whole — a five-series subset of a collection that spans several source publications should
|
||||
cite only the publications it actually draws on.
|
||||
|
||||
### Via `idc-index`
|
||||
|
||||
```python
|
||||
# Citations for a collection (APA is the default format)
|
||||
citations = client.citations_from_selection(collection_id="rider_pilot")
|
||||
for citation in citations:
|
||||
print(citation)
|
||||
```
|
||||
|
||||
```python
|
||||
# Citations for a specific set of series — matches what you actually downloaded
|
||||
results = client.sql_query("""
|
||||
SELECT SeriesInstanceUID FROM index
|
||||
WHERE collection_id = 'tcga_luad' LIMIT 5
|
||||
""")
|
||||
citations = client.citations_from_selection(
|
||||
seriesInstanceUID=list(results['SeriesInstanceUID'].values)
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# BibTeX, for LaTeX manuscripts
|
||||
bibtex_citations = client.citations_from_selection(
|
||||
collection_id="tcga_luad",
|
||||
citation_format=IDCClient.CITATION_FORMAT_BIBTEX
|
||||
)
|
||||
```
|
||||
|
||||
`citations_from_selection()` takes the same selection filters as the download methods —
|
||||
`collection_id`, `patientId`, `studyInstanceUID`, `seriesInstanceUID` — plus `citation_format`.
|
||||
|
||||
### Via the REST API
|
||||
|
||||
`POST /v3/citations` **wraps** the filter in a `filters` key (unlike `/v3/licenses` — this
|
||||
asymmetry is the single most common REST mistake; see `references/rest_api_guide.md`):
|
||||
|
||||
```bash
|
||||
curl -s $B/citations \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}, "citation_format": "bibtex"}'
|
||||
```
|
||||
|
||||
The response separates the per-dataset `citations[]` from `idc_acknowledgment` (the IDC paper)
|
||||
and `recommendation`. Include both parts — see *What to include when publishing* below.
|
||||
|
||||
### Via the MCP server
|
||||
|
||||
Call `get_citations` for a selection. It returns the per-dataset citations plus the IDC paper
|
||||
to acknowledge IDC itself, matching the REST response.
|
||||
|
||||
### Citation formats
|
||||
|
||||
| `idc-index` constant | REST / MCP `citation_format` | Output |
|
||||
|----------------------|------------------------------|--------|
|
||||
| `IDCClient.CITATION_FORMAT_APA` | `apa` (default) | APA string |
|
||||
| `IDCClient.CITATION_FORMAT_BIBTEX` | `bibtex` | BibTeX entry, for LaTeX |
|
||||
| `IDCClient.CITATION_FORMAT_JSON` | `csl-json` | CSL JSON |
|
||||
| `IDCClient.CITATION_FORMAT_TURTLE` | `turtle` | RDF Turtle |
|
||||
|
||||
## What to include when publishing
|
||||
|
||||
1. **The dataset citations** for every collection or series set used.
|
||||
2. **The IDC data version** — `client.get_idc_version()`, `GET /v3/version`, or the MCP
|
||||
`get_idc_version` tool. IDC releases are versioned and series are added and revised between
|
||||
them, so the version is what makes the selection reproducible.
|
||||
3. **The IDC platform citation**, to acknowledge IDC itself. The REST and MCP routes return
|
||||
this as `idc_acknowledgment`; when using `idc-index`, add it yourself:
|
||||
|
||||
> Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency,
|
||||
> Reproducibility, and Scalability in Imaging Artificial Intelligence." *RadioGraphics* 43.12
|
||||
> (2023). https://doi.org/10.1148/rg.230180
|
||||
|
||||
4. **The series manifest** — save the `SeriesInstanceUID` list alongside the analysis so the
|
||||
exact cohort can be rebuilt.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Fewer citations returned than collections selected
|
||||
|
||||
- **Cause:** Citations are derived from `source_DOI`, and several collections can share one
|
||||
DOI, so a multi-collection selection may legitimately produce a shorter list.
|
||||
- **Solution:** Query `SELECT DISTINCT collection_id, source_DOI FROM index WHERE ...` to see
|
||||
the mapping directly.
|
||||
|
||||
### Issue: `POST /v3/citations` returns citations for all of IDC
|
||||
|
||||
- **Cause:** The filter was passed directly instead of wrapped in `filters`. `/v3/licenses`
|
||||
takes the filter directly; `/v3/citations` wraps it. A mis-shaped body is not an error — it
|
||||
is treated as an empty filter.
|
||||
- **Solution:** Check the response counts against a `POST /v3/cohort/counts` for the same
|
||||
selection. See `references/rest_api_guide.md`.
|
||||
|
||||
## Resources
|
||||
|
||||
- **IDC Portal** — https://portal.imaging.datacommons.cancer.gov/
|
||||
- **IDC data licensing documentation** — https://learn.canceridc.dev/data/licensing
|
||||
- **`references/rest_api_guide.md`** — `/v3/licenses` and `/v3/citations` endpoint reference
|
||||
- **`references/mcp_guide.md`** — `get_licenses` and `get_citations` tool inventory
|
||||
- **`references/cloud_storage_guide.md`** — bucket separation for commercially restricted data
|
||||
@@ -0,0 +1,181 @@
|
||||
# IDC MCP Server Guide
|
||||
|
||||
IDC operates a hosted [Model Context Protocol](https://modelcontextprotocol.io/) server that
|
||||
exposes IDC discovery and metadata as agent tools. This guide covers how to recognize it, how
|
||||
to divide work between it and `idc-index`, and what to hand off across that boundary.
|
||||
|
||||
The server is optional. Everything in `SKILL.md` works without it.
|
||||
|
||||
## Endpoint
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| URL | `https://api.imaging.datacommons.cancer.gov/mcp` |
|
||||
| Transport | Streamable HTTP (`streamable-http`, sometimes spelled `http`) |
|
||||
| Authentication | None |
|
||||
| Server identity | `IDC (Imaging Data Commons)` |
|
||||
|
||||
## Identifying the server
|
||||
|
||||
Tool names and resource URIs are defined by the server, so they are the same on every host.
|
||||
Use them, not host-specific naming conventions, to decide whether the server is present.
|
||||
|
||||
**Strongest signal — resource URIs.** The server publishes two resources under an `idc://`
|
||||
scheme:
|
||||
|
||||
| URI | Content |
|
||||
|-----|---------|
|
||||
| `idc://guide` | Data model and recommended workflow (Markdown) |
|
||||
| `idc://tables` | Tables available to `run_sql`, with descriptions and column counts (JSON) |
|
||||
|
||||
If the host can enumerate MCP resources, a resource with URI `idc://guide` identifies the
|
||||
server unambiguously.
|
||||
|
||||
**Fallback — tool-name fingerprint.** Require three or more of `build_cohort`,
|
||||
`get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Do not treat `run_sql`,
|
||||
`get_stats`, `list_tables`, or `get_citations` as evidence on their own; those names are
|
||||
generic enough that another server could expose them.
|
||||
|
||||
**This is disambiguation, not authentication.** No runtime check can prove the server on the
|
||||
other end is operated by NCI — a hostile server could serve `idc://guide` and name its tools
|
||||
anything. The trust anchor is the URL the user configured plus TLS, which is established when
|
||||
the server is added, not when the skill runs. That is sufficient for routing: the check only
|
||||
has to distinguish IDC from the user's other installed servers.
|
||||
|
||||
**Fail soft.** If identification is ambiguous, or a tool call fails, fall back rather than
|
||||
reporting an error — to the REST API (`rest_api_guide.md`) for read-only metadata, which is the
|
||||
same service with no configuration, or to `idc-index` when it is already installed or the task
|
||||
needs downloads or local analysis. Tool names may change as the server matures.
|
||||
|
||||
## Tool inventory
|
||||
|
||||
Verified against server version `3.0.0b3`. Treat this as a snapshot, not a contract — call
|
||||
the server's own listing rather than assuming this list is current.
|
||||
|
||||
| Group | Tools |
|
||||
|-------|-------|
|
||||
| Version and scale | `get_idc_version`, `get_stats` |
|
||||
| Collections | `list_collections`, `get_collection`, `list_analysis_results` |
|
||||
| Attribute grounding | `list_attributes`, `get_attribute_values` |
|
||||
| Cohorts | `build_cohort`, `get_cohort_urls` |
|
||||
| SQL | `list_tables`, `get_table_schema`, `run_sql` |
|
||||
| Clinical data | `list_clinical_tables`, `get_clinical_table_schema`, `get_clinical_table` |
|
||||
| Attribution | `get_citations`, `get_licenses` |
|
||||
| Visualization | `get_viewer_url` |
|
||||
|
||||
The server ships its own usage instructions, which most hosts inject automatically. Follow
|
||||
those instructions for tool sequencing (ground with `list_attributes` /
|
||||
`get_attribute_values` before filtering; check `list_tables` before writing SQL). Do not
|
||||
re-derive that workflow from `SKILL.md` — the two would drift apart on the server's next
|
||||
release.
|
||||
|
||||
**Cohort results report their own filters.** `build_cohort` and `get_cohort_urls` require at
|
||||
least one filter predicate and fail cleanly without one, rather than returning the whole archive;
|
||||
results echo the filters actually applied along with warnings for any predicate that was dropped
|
||||
or any value whose casing did not match. Read those warnings before reporting a count — a zero
|
||||
with no warning means the filter matched nothing, which is a real answer. Same contract as the
|
||||
REST endpoints they wrap; see `rest_api_guide.md`.
|
||||
|
||||
## Division of labor
|
||||
|
||||
The server and `idc-index` overlap on metadata queries and diverge everywhere else.
|
||||
|
||||
| Task | Use |
|
||||
|------|-----|
|
||||
| IDC data version, collection and series counts | Server (`get_idc_version`, `get_stats`) |
|
||||
| Valid filter values before building a query | Server (`get_attribute_values`) |
|
||||
| Cohort selection by attribute filters | Server (`build_cohort`) |
|
||||
| One-off metadata SQL, answer consumed as prose | Server (`run_sql`) |
|
||||
| Metadata SQL whose result feeds local Python | `idc-index` (`client.sql_query`) |
|
||||
| Downloading DICOM files | `idc-index` (`client.download_from_selection`) |
|
||||
| pandas / notebook analysis, plotting | `idc-index` |
|
||||
| Reading pixel data (pydicom, SimpleITK) | `idc-index` + local files |
|
||||
| DICOMweb, BigQuery, direct S3/GCS, Parquet | `idc-index` and the relevant reference guide |
|
||||
| Digital pathology tiling and annotation workflows | `idc-index` + `digital_pathology_guide.md` |
|
||||
| Reproducible scripts a user will re-run | `idc-index` (a script outlives the session) |
|
||||
|
||||
Two rules resolve the overlap:
|
||||
|
||||
- **Prefer the server for discovery.** It is hosted against a current IDC release, so it does
|
||||
not depend on the `idc-index` version pinned in `SKILL.md`.
|
||||
- **Prefer `idc-index` when the result must become a Python object.** Round-tripping a
|
||||
DataFrame through tool output wastes context and loses types.
|
||||
|
||||
## Handing off from the server to `idc-index`
|
||||
|
||||
The boundary artifact is a list of `SeriesInstanceUID` values.
|
||||
|
||||
```python
|
||||
# UIDs obtained from the MCP server's build_cohort / run_sql output
|
||||
series_uids = [
|
||||
"1.3.6.1.4.1.14519.5.2.1.7009.2403.334240657131972136850343327463",
|
||||
# ...
|
||||
]
|
||||
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
|
||||
# Confirm size before downloading — the server reports size_TB, but re-check locally
|
||||
sizes = client.sql_query(f"""
|
||||
SELECT COUNT(*) AS series, SUM(series_size_MB)/1000 AS size_GB
|
||||
FROM index
|
||||
WHERE SeriesInstanceUID IN ({','.join(f"'{u}'" for u in series_uids)})
|
||||
""")
|
||||
print(sizes)
|
||||
|
||||
client.download_from_selection(
|
||||
downloadDir="./data",
|
||||
seriesInstanceUID=series_uids, # a list, not a DataFrame
|
||||
dirTemplate="%collection_id/%PatientID/%Modality",
|
||||
)
|
||||
```
|
||||
|
||||
Run `python scripts/check_version.py` before the first `idc-index` call in a session, even if
|
||||
discovery happened server-side — the two components version independently.
|
||||
|
||||
`get_cohort_urls` also returns ready-made `idc` CLI commands. Those are the better handoff
|
||||
when the user wants a shell command they can re-run outside the session; see
|
||||
`references/cli_guide.md`.
|
||||
|
||||
Going the other direction, `idc-index` results are already local, so there is rarely a reason
|
||||
to send them back to the server.
|
||||
|
||||
## Version authority
|
||||
|
||||
When the server is present, it is the authority on the IDC data version: call
|
||||
`get_idc_version` rather than quoting the `idc-data-version` value in the `SKILL.md`
|
||||
frontmatter, which records the release the skill was last verified against.
|
||||
|
||||
If the server and a locally installed `idc-index` report different versions, say so and name
|
||||
both. The mismatch is real — the hosted server tracks IDC releases independently of the user's
|
||||
installed package — and it changes which answers about "what's new" are correct.
|
||||
|
||||
## Host-specific notes
|
||||
|
||||
Everything above is portable. The items below are not, and apply only to specific agent
|
||||
environments.
|
||||
|
||||
### Claude Code
|
||||
|
||||
- **Tool naming.** MCP tools are exposed as `mcp__<server>__<tool>`, where `<server>` is the
|
||||
configured server name with every character outside `A-Za-z0-9_-` replaced by `_`. A CLI
|
||||
install named `idc` yields `mcp__idc__build_cohort`; a claude.ai connector named
|
||||
`IDC MCP prod` yields `mcp__claude_ai_IDC_MCP_prod__build_cohort`.
|
||||
- **Enumerating resources.** `ListMcpResourcesTool` returns each resource with a `server`
|
||||
field, which is how to find the `idc://guide` resource and the owning server name in one
|
||||
call.
|
||||
- **Adding the server.**
|
||||
`claude mcp add --transport http idc https://api.imaging.datacommons.cancer.gov/mcp`
|
||||
- **Permission rules.** Allow rules need a literal, glob-free server segment: `mcp__idc__*`
|
||||
works, `mcp__*` does not. Connector installs need their own
|
||||
`mcp__claude_ai_<name>__*` rule, so the rule differs by install path.
|
||||
- **Detecting via the CLI does not work.** `claude mcp list` reads only file-based
|
||||
configuration (`~/.claude.json`, `.mcp.json`). It reports "No MCP servers configured" for a
|
||||
claude.ai connector that is connected and working in the same session, so it cannot be used
|
||||
as a presence check.
|
||||
|
||||
### Other hosts
|
||||
|
||||
Any agent that supports MCP over streamable HTTP can use the server. Consult that agent's
|
||||
documentation for how servers are registered and how tool names are namespaced; the endpoint
|
||||
URL and the absence of authentication are all the configuration it needs.
|
||||
@@ -1,19 +1,22 @@
|
||||
# Direct Parquet Access Guide for IDC
|
||||
|
||||
**Tested with:** idc-index-data 23.10.1, DuckDB 1.x
|
||||
**Tested with:** idc-index-data 24.2.2 (IDC data version v24), DuckDB 1.5
|
||||
|
||||
All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index — useful for quick exploration or environments where package installation is unavailable.
|
||||
All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index.
|
||||
|
||||
**Limitation:** download helpers (`download_from_selection()`), viewer URLs (`get_viewer_URL()`), and citation generation require the idc-index client and are not available from raw Parquet files.
|
||||
|
||||
**This is not the first no-install option to reach for.** It still needs DuckDB installed, and the per-collection clinical tables are not published here — only the `clinical_index` dictionary. For ad-hoc metadata with nothing installed, the REST API (`rest_api_guide.md`) needs no install at all and reaches `clinical.<table>` through `POST /sql`.
|
||||
|
||||
## When to Use This Guide
|
||||
|
||||
Load this guide when you need to:
|
||||
- Query IDC metadata without installing idc-index
|
||||
- Run ad-hoc DuckDB queries against the latest index files
|
||||
- Access `volume_geometry_index` or `rtstruct_index` for geometry validation or RT structure queries
|
||||
- Pin queries to a specific IDC data version (see *Pinning to a Specific Version* below) rather than whatever the hosted API currently serves
|
||||
- Return more rows than the REST `/sql` ceiling of 10 000
|
||||
- Run heavy or repeated local DuckDB analysis without driving the hosted API
|
||||
- Query IDC metadata where DuckDB is available but idc-index is not
|
||||
|
||||
For full API access (downloads, viewer, citations), use idc-index as documented in the main SKILL.md.
|
||||
For downloads, viewer URLs, and citations, use idc-index as documented in the main SKILL.md.
|
||||
|
||||
## URL Pattern
|
||||
|
||||
@@ -38,16 +41,17 @@ https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifact
|
||||
| `collections_index.parquet` | — | Collection-level metadata |
|
||||
| `analysis_results_index.parquet` | — | Derived dataset metadata |
|
||||
| `clinical_index.parquet` | ~0.2 MB | Clinical data column dictionary |
|
||||
| `ct_index.parquet` | — | CT acquisition/reconstruction parameters |
|
||||
| `mr_index.parquet` | — | MR sequence/acquisition parameters |
|
||||
| `pt_index.parquet` | — | PET acquisition/radiopharmaceutical parameters |
|
||||
| `prior_versions_index.parquet` | — | Series from previous IDC releases |
|
||||
|
||||
**Note:** the main index file is named `idc_index.parquet`, not `index.parquet`. Reference it with an alias in SQL queries (e.g., `FROM read_parquet(...) AS index`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install duckdb
|
||||
# or: uv add duckdb
|
||||
```
|
||||
Install the Python `duckdb` package, using whatever installer manages the environment you are
|
||||
running in.
|
||||
|
||||
DuckDB reads Parquet directly from HTTPS URLs using HTTP range requests — no GCS client library or authentication required.
|
||||
|
||||
|
||||
@@ -0,0 +1,612 @@
|
||||
# IDC REST API Guide
|
||||
|
||||
**Tested with:** API `3.0.0b3` (build `0640860`), IDC data version v24, `idc_index_data_version` 24.2.2
|
||||
|
||||
IDC operates a hosted REST API that exposes discovery, cohort building, metadata SQL, and
|
||||
download manifests over plain HTTP. No authentication, account, or credentials are required —
|
||||
every example on this page can be run from any terminal with `curl`.
|
||||
|
||||
The API and the [MCP server](mcp_guide.md) are the same service behind two transports: the MCP
|
||||
tools wrap these endpoints. Both are hosted against a current IDC release, independently of the
|
||||
`idc-index` version installed locally.
|
||||
|
||||
## When to Use This Guide
|
||||
|
||||
Load this guide when you need to:
|
||||
- Query IDC from a language or environment with no `idc-index` install (shell, R, Java, JS, a
|
||||
notebook without pip access)
|
||||
- Build cohorts and manifests over HTTP for a pipeline or web application
|
||||
- Run metadata SQL against a current IDC release without downloading local index files
|
||||
- Hand a user copy-pasteable `curl` commands they can run anywhere
|
||||
|
||||
For downloads, pandas/notebook analysis, reading pixel data, DICOMweb, BigQuery, or digital
|
||||
pathology tiling, use `idc-index` as documented in `SKILL.md`. The API never moves image bytes:
|
||||
it returns public `s3://` URLs and manifests, and the transfer happens directly from cloud
|
||||
storage to the client.
|
||||
|
||||
**Choosing among the three interfaces:**
|
||||
|
||||
| Situation | Use |
|
||||
|-----------|-----|
|
||||
| Session already has the hosted MCP server | MCP tools (`mcp_guide.md`) — same data, no HTTP plumbing |
|
||||
| `idc-index` already installed, results feed pandas / downloads | `idc-index` (`SKILL.md`) |
|
||||
| `idc-index` **not** installed and the task is read-only metadata | This REST API — do not install to answer a metadata question |
|
||||
| No Python, or a non-Python client, or shell commands the user re-runs | This REST API |
|
||||
| Installed `idc-index` is a whole IDC data release behind and cannot be upgraded here | REST API for the query, **direct bucket transfer** may be needed for the download — `idc-index` cannot fetch what its index does not list |
|
||||
|
||||
Being hosted, the API always serves a current IDC release — but so does an up-to-date
|
||||
`idc-index`. "I want the newest data" is not on its own a reason to prefer the API: the normal
|
||||
fix for a stale local index is to upgrade it. Reach for the API on version grounds only when
|
||||
upgrading is not an option in that environment. Avoiding the install *is* a reason, though: the
|
||||
packaged index data is ~77 MB before pandas, pyarrow, and duckdb, which a metadata question does
|
||||
not need.
|
||||
|
||||
## Endpoint and Versioning
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Base URL | `https://api.imaging.datacommons.cancer.gov/v3` |
|
||||
| Authentication | None |
|
||||
| Content type | `application/json` (request and response), except `cohort/manifest.txt` → `text/plain` |
|
||||
| Interactive docs | https://api.imaging.datacommons.cancer.gov/v3/docs (Swagger UI) |
|
||||
| OpenAPI spec | https://api.imaging.datacommons.cancer.gov/v3/openapi.json |
|
||||
| Source | https://github.com/ImagingDataCommons/IDC-REST-MCP |
|
||||
|
||||
**v3 is in beta.** The contract may still change before the final `3.0.0` release, so verify the
|
||||
running build rather than assuming the values in this guide:
|
||||
|
||||
```bash
|
||||
curl -s https://api.imaging.datacommons.cancer.gov/v3/version
|
||||
# {"idc_version":"v24","idc_index_data_version":"24.2.2","api_version":"3.0.0b3","build":"0640860"}
|
||||
```
|
||||
|
||||
`idc_version` is the IDC data release the API serves and is the authority when the API is in
|
||||
use — prefer it over the `idc-data-version` pinned in the `SKILL.md` frontmatter, which records
|
||||
the release the skill was last verified against.
|
||||
|
||||
### Checking the API against a local idc-index
|
||||
|
||||
The API and `idc-index` are built on the same `idc-index-data` package, and both report its
|
||||
version, so consistency is an exact check rather than a guess:
|
||||
|
||||
| Side | Data release (coarse) | `idc-index-data` version (exact) |
|
||||
|------|-----------------------|----------------------------------|
|
||||
| API | `GET /v3/version` → `idc_version` | `GET /v3/version` → `idc_index_data_version` |
|
||||
| Local | `IDCClient().get_idc_version()` | `idc_index_data.__version__` |
|
||||
|
||||
```python
|
||||
import idc_index_data
|
||||
import requests
|
||||
|
||||
api = requests.get("https://api.imaging.datacommons.cancer.gov/v3/version", timeout=30).json()
|
||||
api_version, local_version = api["idc_index_data_version"], idc_index_data.__version__
|
||||
|
||||
if api_version.split(".")[0] != local_version.split(".")[0]:
|
||||
print(f"Different IDC data release: API {api_version}, local {local_version} — upgrade")
|
||||
elif api_version != local_version:
|
||||
print(f"Same data release, different index build: API {api_version}, local {local_version}")
|
||||
# → Same data release, different index build: API 24.2.2, local 24.2.0
|
||||
```
|
||||
|
||||
**The major is the IDC data release; the rest is the index build.** `idc-index-data` `24.x.y`
|
||||
serves IDC `v24` — 24.0.0 shipped with the v24 release, and 24.1.0 / 24.2.x are later builds of
|
||||
the *same* release. What differs between them is the index itself (added tables and columns,
|
||||
corrected metadata), never which series IDC contains.
|
||||
|
||||
So read a mismatch by its position:
|
||||
|
||||
| Difference | Means | Consequence |
|
||||
|------------|-------|-------------|
|
||||
| Major (24.x.y vs 25.x.y) | Different IDC data release | Series added, revised, or removed. Counts legitimately differ, and `idc-index` **cannot download** what its index does not list |
|
||||
| Minor or patch (24.2.0 vs 24.2.2) | Same data release, different index build | Same series everywhere; downloads are unaffected. A metadata query can still differ if it touches a column that was added or corrected |
|
||||
|
||||
Comparing `idc_version` alone cannot make this distinction in the other direction either — the
|
||||
`vNN` label is exactly the major, so matching `v24` on both sides tells you the release agrees
|
||||
but says nothing about the index build.
|
||||
|
||||
When the two disagree, say so and name both versions, then reconcile rather than mixing
|
||||
results: upgrading `idc-index` brings the local side to the newer `idc-index-data`, and
|
||||
`python scripts/check_version.py` reports whether an upgrade is available and prints the
|
||||
command for the interpreter you are running. Do not present API-derived and locally-derived counts side by side as if they came
|
||||
from one index.
|
||||
|
||||
**A major behind also breaks downloads.** `idc-index` resolves every `s3://` URL it is given
|
||||
against its *own* index, so a manifest built from a newer IDC data release can name series it
|
||||
has never heard of — see *When the local index is a data release behind the API* under
|
||||
**Getting the Data**.
|
||||
|
||||
### Use v3 only — V1 and V2 are being retired
|
||||
|
||||
**Do not write new code against the V1 or V2 IDC APIs, and do not follow V1/V2 examples found
|
||||
in older tutorials, notebooks, blog posts, or forum answers.** Both are superseded by v3 and
|
||||
are scheduled to be deprecated and shut down; code written against them will stop working. If
|
||||
a user brings V1 or V2 code, say so and port it to v3 rather than extending it.
|
||||
|
||||
Two signals that a snippet is V1/V2 rather than v3, both of which will fail against `/v3`:
|
||||
|
||||
- a base URL other than `https://api.imaging.datacommons.cancer.gov/v3` (for example a
|
||||
`/v1/` or `/v2/` path segment)
|
||||
- per-attribute filter suffixes such as `Modality_btw` or `_lt` / `_gt`, instead of v3's
|
||||
separate `terms` and `ranges` objects
|
||||
|
||||
V1/V2 documentation survives only in the [IDC docs archive](https://learn.canceridc.dev/archive/archive)
|
||||
for historical reference. Treat it as read-only history, not as a source of working examples.
|
||||
|
||||
## The Query Surfaces
|
||||
|
||||
The endpoints group into five surfaces that build on each other:
|
||||
|
||||
| Surface | Answers | Endpoints |
|
||||
|---------|---------|-----------|
|
||||
| Discovery | What exists? What can I filter on? | `GET /version`, `/stats`, `/collections`, `/collections/{id}`, `/analysis_results`, `/attributes`, `/attributes/{attr}/values` |
|
||||
| Cohort | How big is my selection, and what's in it? | `POST /cohort/counts`, `POST /cohort/manifest` |
|
||||
| Retrieval | Give me the download links | `POST /cohort/manifest.txt` |
|
||||
| SQL | Anything relational or aggregate | `GET /tables`, `GET /tables/{table}`, `POST /sql` |
|
||||
| Side tools | View / cite / license-check a cohort | `GET /viewer-url`, `POST /citations`, `POST /licenses` |
|
||||
|
||||
Discovery supplies the vocabulary (attribute names and their valid values) that the cohort
|
||||
filters consume. SQL is the escape hatch for questions structured filters cannot express.
|
||||
|
||||
Clinical data has its own discovery endpoints (`GET /clinical/tables`,
|
||||
`/clinical/tables/{table}`, `/clinical/tables/{table}/rows`) and is filtered or joined through
|
||||
SQL against the `clinical` schema.
|
||||
|
||||
## Endpoint Reference
|
||||
|
||||
All paths are relative to `https://api.imaging.datacommons.cancer.gov/v3`.
|
||||
|
||||
| Method & path | Purpose | Key response fields |
|
||||
|---------------|---------|---------------------|
|
||||
| `GET /version` | IDC data release + API build | `idc_version`, `idc_index_data_version`, `api_version`, `build` |
|
||||
| `GET /stats` | Headline totals | `collections`, `patients`, `studies`, `series`, `instances`, `size_TB` |
|
||||
| `GET /collections` | List collections (JSON array) | `collection_id`, `collection_name`, `cancer_types`, `tumor_locations`, `species`, `subjects`, `description` |
|
||||
| `GET /collections/{id}` | Collection detail | the above plus `patients`, `studies`, `series`, `instances`, `size_TB`, `modalities[]`, `licenses[]` |
|
||||
| `GET /analysis_results` | Derived datasets (JSON array) | `analysis_result_id`, `analysis_result_title`, `source_DOI`, `subjects`, `collections`, `modalities`, `license_short_name` |
|
||||
| `GET /attributes` | Filterable attributes | `name`, `table`, `data_type`, `kind` (`term` \| `range`), `categorical`, `description` |
|
||||
| `GET /attributes/{attr}/values?limit=` | Distinct values with counts | `attribute`, `values[{value,count}]`, `truncated`, `note` |
|
||||
| `GET /tables` | Tables available to SQL | `tables[{name,description,column_count}]` |
|
||||
| `GET /tables/{table}` | Column schema | `name`, `description`, `columns[{name,type,description}]` |
|
||||
| `GET /clinical/tables?collection_id=` | Clinical tables, optionally one collection | `tables[{table_name,sql_path,collection_id,column_count}]` |
|
||||
| `GET /clinical/tables/{table}` | Clinical columns + labels | `name`, `columns[{name,type,description}]` |
|
||||
| `GET /clinical/tables/{table}/rows?max_rows=` | Clinical rows (capped) | `columns`, `rows`, `row_count`, `truncated`, `max_rows` |
|
||||
| `POST /cohort/counts` | Distinct counts for a filter (cheap) | `patients`, `studies`, `series`, `instances`, `size_TB` |
|
||||
| `POST /cohort/manifest` | Counts + page of series + download payload | `counts`, `page`, `page_size`, `returned`, `total_series`, `series[]`, `download` |
|
||||
| `POST /cohort/manifest.txt` | Full manifest as `text/plain` | one `s3://…/*` URL per line |
|
||||
| `POST /sql` | Guarded read-only SQL (DuckDB) | `columns`, `rows`, `row_count`, `truncated`, `max_rows` |
|
||||
| `GET /viewer-url` | OHIF / Slim viewer link | `viewer_url`, `viewer`, `study_instance_uid`, `series_instance_uid` |
|
||||
| `POST /citations` | Citations for a cohort | `format`, `citations[]`, `idc_acknowledgment`, `recommendation` |
|
||||
| `POST /licenses` | License breakdown for a cohort | `licenses[{license_short_name,series,size_TB}]` |
|
||||
|
||||
`GET /health` and `GET /v3` (API root) also exist for liveness checks.
|
||||
|
||||
## Filter Syntax
|
||||
|
||||
Cohort filters are shared by `cohort/counts`, `cohort/manifest`, `cohort/manifest.txt`,
|
||||
`licenses`, and `citations`. **The filter object always goes under a `filters` key**, on every one
|
||||
of them. It has two parts:
|
||||
|
||||
- **`terms`** — `{attribute: [values]}` for equality/membership. Values are **OR**'d within an
|
||||
attribute and **AND**'d across attributes.
|
||||
- **`ranges`** — `{attribute: {"gte": x, "lte": y}}` for numeric and date attributes. Either
|
||||
bound may be omitted for an open-ended range.
|
||||
|
||||
```json
|
||||
{
|
||||
"filters": {
|
||||
"terms": {"Modality": ["CT"], "collection_id": ["nlst"]},
|
||||
"ranges": {"instanceCount": {"gte": 100, "lte": 200}}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Filters operate only on the `index` table's filterable attributes — 19 of them as of `3.0.0b3`:
|
||||
|
||||
| Kind | Attributes |
|
||||
|------|------------|
|
||||
| `term` | `collection_id`, `analysis_result_id`, `PatientID`, `StudyInstanceUID`, `SeriesInstanceUID`, `Modality`, `BodyPartExamined`, `Manufacturer`, `ManufacturerModelName`, `PatientSex`, `sop_class_name`, `license_short_name`, `source_DOI` |
|
||||
| `range` | `instanceCount`, `series_size_MB`, `series_init_idc_version`, `series_revised_idc_version`, `StudyDate`, `SeriesDate` |
|
||||
|
||||
`SeriesInstanceUID`, `StudyInstanceUID`, and `PatientID` being filterable is what makes the
|
||||
side tools work at any granularity — the licenses or citations for a single series are just a
|
||||
one-value filter.
|
||||
|
||||
Anything outside this list — clinical values, segmented anatomy, per-modality acquisition
|
||||
parameters — is not filterable here; use the SQL surface instead.
|
||||
|
||||
**Ground values before filtering.** Call `GET /attributes` for what is filterable and whether
|
||||
it is a term or a range, then `GET /attributes/{attr}/values` for real values and their casing —
|
||||
values are matched case-sensitively.
|
||||
|
||||
### The server reports what it filtered on
|
||||
|
||||
Every filtered response echoes `filters_applied` and `warnings`, and misuse is refused rather
|
||||
than ignored. Together these make a result self-describing, so you do not have to sanity-check a
|
||||
count against a number you happen to remember.
|
||||
|
||||
| Mistake | Response |
|
||||
|---------|----------|
|
||||
| Bare filter object, no `filters` key | `422` naming the correct shape |
|
||||
| Unrecognized key at any depth (`term` for `terms`, a range bound spelled `min`) | `422` pointing at the key |
|
||||
| Unknown filter attribute, or a range attribute used as a term | `400` naming the discovery call |
|
||||
| Unfiltered `cohort/manifest` or `manifest.txt` | `400` — it will not enumerate the whole archive |
|
||||
| Unfiltered `cohort/counts` or `licenses` | `200` plus an explicit "ENTIRE IDC archive" warning |
|
||||
| A predicate that constrains nothing (`{"collection_id": []}`, `{"instanceCount": {}}`) | `200`, and `warnings` names the ignored predicate |
|
||||
| Miscased value (`mr` for `MR`) | `200`, zero counts, and a warning naming the casing that exists |
|
||||
|
||||
```bash
|
||||
curl -s $B/cohort/counts -H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}'
|
||||
# {"patients":8,"studies":154,"series":774,"instances":21111,"size_TB":0.011,
|
||||
# "filters_applied":{"terms":{"collection_id":["rider_pilot"]},"ranges":{}},"warnings":[]}
|
||||
```
|
||||
|
||||
**Read `warnings` before reporting any count.** A zero count with `warnings: []` means the filter
|
||||
applied and matched nothing — that is a real answer. A zero count *with* a casing warning means
|
||||
the filter was wrong. And `filters_applied` covers the case no shape check can: a request
|
||||
carrying one good predicate plus one that constrains nothing returns a perfectly plausible
|
||||
number, and only `warnings` reveals the dropped half.
|
||||
|
||||
`cohort/manifest.txt` returns `text/plain` and so cannot carry these fields; there the
|
||||
required-predicate `400` does the same job, appending the ignored-predicate reason to its message.
|
||||
|
||||
## Worked Examples
|
||||
|
||||
### Discovery
|
||||
|
||||
```bash
|
||||
B=https://api.imaging.datacommons.cancer.gov/v3
|
||||
|
||||
curl -s $B/version # data release + API build
|
||||
curl -s $B/stats # headline totals
|
||||
curl -s $B/collections # all 176 collections
|
||||
curl -s $B/collections/rider_pilot # one collection: counts, modalities, licenses
|
||||
curl -s $B/analysis_results # derived datasets (segmentations, annotations)
|
||||
curl -s $B/attributes # what can be filtered, and how
|
||||
curl -s "$B/attributes/Modality/values?limit=10"
|
||||
```
|
||||
|
||||
### Cohort building
|
||||
|
||||
Check size first — `counts` is cheap and answers "is this download sane?":
|
||||
|
||||
```bash
|
||||
curl -s $B/cohort/counts \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}}}'
|
||||
# {"patients":3718,"studies":5689,"series":47986,"instances":6493262,"size_TB":2.421,
|
||||
# "filters_applied":{...},"warnings":[]}
|
||||
```
|
||||
|
||||
Then request a page of series plus the download payload — same filter, same `filters` key.
|
||||
|
||||
```bash
|
||||
curl -s $B/cohort/manifest \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"Modality": ["MR"], "BodyPartExamined": ["BREAST"]}},
|
||||
"page": 0, "page_size": 3}'
|
||||
```
|
||||
|
||||
Each `series[]` row carries `collection_id`, `PatientID`, `StudyInstanceUID`,
|
||||
`SeriesInstanceUID`, `Modality`, `SeriesDescription`, `instanceCount`, `series_size_MB`,
|
||||
`aws_bucket`, `crdc_series_uuid`, and `series_aws_url`. Set `"include_rows": false` to get
|
||||
counts and the download payload without the rows.
|
||||
|
||||
### SQL
|
||||
|
||||
```bash
|
||||
curl -s $B/sql \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"sql": "SELECT Modality, count(*) n FROM index GROUP BY 1 ORDER BY n DESC", "max_rows": 20}'
|
||||
```
|
||||
|
||||
### Viewer, licenses, citations
|
||||
|
||||
```bash
|
||||
curl -s "$B/viewer-url?study_instance_uid=1.3.6.1.4.1.14519.5.2.1.7695.4164.129908397467389975396031099306"
|
||||
|
||||
curl -s $B/licenses \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}'
|
||||
|
||||
curl -s $B/citations \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}, "citation_format": "bibtex"}'
|
||||
```
|
||||
|
||||
`viewer-url` takes `series_instance_uid` or `study_instance_uid` (and an optional `viewer`
|
||||
override); it picks OHIF v3 for radiology and Slim for slide microscopy automatically.
|
||||
|
||||
`citation_format` is one of `apa` (default), `bibtex`, `csl-json`, `turtle`. The response
|
||||
separates the per-dataset `citations[]` from `idc_acknowledgment`, the IDC paper — include
|
||||
both when publishing.
|
||||
|
||||
### Python client
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
BASE = "https://api.imaging.datacommons.cancer.gov/v3"
|
||||
session = requests.Session()
|
||||
|
||||
def get(path, **params):
|
||||
r = session.get(f"{BASE}{path}", params=params, timeout=60)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
def post(path, payload):
|
||||
r = session.post(f"{BASE}{path}", json=payload, timeout=120)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
# 1. Confirm which IDC release the API is serving
|
||||
print(get("/version")["idc_version"])
|
||||
|
||||
# 2. Ground the filter values before using them
|
||||
modalities = {v["value"] for v in get("/attributes/Modality/values", limit=1000)["values"]}
|
||||
assert "MR" in modalities
|
||||
|
||||
# 3. Size the cohort, then page through it — the filter always goes under `filters`
|
||||
filters = {"terms": {"collection_id": ["rider_pilot"], "Modality": ["CT"]}}
|
||||
counts = post("/cohort/counts", {"filters": filters})
|
||||
assert not counts["warnings"], counts["warnings"] # nothing was silently dropped
|
||||
print(counts)
|
||||
|
||||
manifest = post("/cohort/manifest", {"filters": filters, "page": 0, "page_size": 100})
|
||||
uids = [row["SeriesInstanceUID"] for row in manifest["series"]]
|
||||
|
||||
# 4. Hand off to idc-index for the download (see "Handing Off to idc-index" below)
|
||||
```
|
||||
|
||||
Error responses raise through `raise_for_status()`; read `r.json()["error"]["message"]` for the
|
||||
reason before retrying.
|
||||
|
||||
## The SQL Surface
|
||||
|
||||
`POST /sql` runs read-only DuckDB SQL against the same tables `idc-index` exposes locally.
|
||||
Ground the schema with `GET /tables` and `GET /tables/{table}` — do not guess table or column
|
||||
names.
|
||||
|
||||
**Guardrails** (verified against `3.0.0b3`):
|
||||
|
||||
- Only single read-only `SELECT` / `WITH … SELECT` statements are accepted. Anything else is
|
||||
rejected with `{"error": {"code": "invalid_query", "message": "Only read-only SELECT (or WITH ... SELECT) statements are allowed."}}` and HTTP 400.
|
||||
- A server row cap and per-query timeout apply. `max_rows` defaults to **5000** and is clamped
|
||||
to **10000**; results carry `truncated: true` when the cap was hit, and echo the `max_rows`
|
||||
actually applied.
|
||||
- Invalid SQL returns DuckDB's own error text, including its "Candidate bindings" suggestions —
|
||||
useful for fixing a misspelled column without another schema round-trip.
|
||||
|
||||
**Tables reachable from SQL** are the `index` table plus the specialized indices documented in
|
||||
`SKILL.md` (`collections_index`, `analysis_results_index`, `version_metadata_index`,
|
||||
`prior_versions_index`, `seg_index`, `ann_index`, `ann_group_index`, `rtstruct_index`,
|
||||
`ct_index`, `mr_index`, `pt_index`, `sm_index`, `sm_instance_index`, `contrast_index`,
|
||||
`volume_geometry_index`, `clinical_index`). Join them to `index` on `SeriesInstanceUID`, except
|
||||
where `SKILL.md` documents a different key (`segmented_SeriesInstanceUID`,
|
||||
`referenced_SeriesInstanceUID`, `collection_id`, `analysis_result_id`).
|
||||
|
||||
The SQL patterns in `references/sql_patterns.md` are written for `client.sql_query()` but the
|
||||
SQL itself transfers unchanged — send it as the `sql` field.
|
||||
|
||||
**Array columns:** columns typed `STRING[]` (e.g. `SegmentedPropertyType_CodeMeanings`) hold a
|
||||
list per row. Match with `list_contains(col, 'value')`, not `=` or `LIKE`:
|
||||
|
||||
```bash
|
||||
curl -s $B/sql -H 'content-type: application/json' -d '{
|
||||
"sql": "SELECT i.collection_id, count(DISTINCT i.SeriesInstanceUID) AS slides FROM index i JOIN seg_index seg ON seg.segmented_SeriesInstanceUID = i.SeriesInstanceUID WHERE i.Modality = '\''SM'\'' AND list_contains(seg.SegmentedPropertyType_CodeMeanings, '\''Nucleus'\'') GROUP BY 1 ORDER BY slides DESC",
|
||||
"max_rows": 20}'
|
||||
```
|
||||
|
||||
**A SQL result can be a manifest.** Selecting `series_aws_url` gives download URLs directly,
|
||||
and unlike `cohort/manifest.txt` you can carry extra columns alongside them — a per-row
|
||||
`license_short_name`, for instance, which the plain manifest does not include:
|
||||
|
||||
```sql
|
||||
SELECT SeriesInstanceUID, license_short_name, series_aws_url
|
||||
FROM index WHERE collection_id = 'rider_pilot'
|
||||
```
|
||||
|
||||
For bulk series, still prefer `cohort/manifest.txt` — it is not subject to the SQL row cap.
|
||||
|
||||
### Clinical data
|
||||
|
||||
Clinical data comes in two layers:
|
||||
|
||||
- **`clinical_index`** — a data dictionary, one row per (collection, table, column) with a
|
||||
human-readable `column_label` and coded `values`. It is an ordinary SQL table and joins to
|
||||
`index` on `collection_id`.
|
||||
- **Per-collection clinical tables** (e.g. `nlst_canc`) — the actual rows. There are ~150, so
|
||||
they are kept out of `GET /tables` and discovered through the `/clinical/tables` endpoints
|
||||
instead. In SQL they live under a separate schema and are addressed as `clinical.<table>`.
|
||||
|
||||
Clinical tables join to imaging on **`dicom_patient_id = index.PatientID`**, not on a series
|
||||
UID. Clinical data is not harmonized across collections — table and column names vary, so
|
||||
always discover before querying.
|
||||
|
||||
```bash
|
||||
curl -s "$B/clinical/tables?collection_id=nlst" # which tables this collection has
|
||||
curl -s $B/clinical/tables/nlst_canc # columns + labels
|
||||
curl -s "$B/clinical/tables/nlst_canc/rows?max_rows=100" # rows, capped
|
||||
|
||||
curl -s $B/sql -H 'content-type: application/json' -d '{
|
||||
"sql": "SELECT count(DISTINCT i.PatientID) AS patients FROM index i JOIN clinical.nlst_canc c ON c.dicom_patient_id = i.PatientID WHERE i.collection_id = '\''nlst'\'' AND i.Modality = '\''CT'\'' AND c.clinical_stag = '\''400'\''"}'
|
||||
```
|
||||
|
||||
See `references/clinical_data_guide.md` for value mapping and the wider clinical data model.
|
||||
|
||||
## Getting the Data
|
||||
|
||||
The API returns manifests; a client moves the bytes. Every URL points at public AWS S3 or GCS
|
||||
buckets and needs no credentials.
|
||||
|
||||
```bash
|
||||
# save the full manifest
|
||||
curl -s $B/cohort/manifest.txt \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"filters": {"terms": {"collection_id": ["rider_pilot"]}}}' > idc_manifest.txt
|
||||
|
||||
# download it (needs idc-index installed)
|
||||
idc download-from-manifest idc_manifest.txt --download-dir ./idc-data
|
||||
```
|
||||
|
||||
For a filter that is a single `collection_id`, the `download` payload of `cohort/manifest`
|
||||
emits the simpler `idc download <collection_id> --download-dir ./idc-data` form.
|
||||
|
||||
### When the local index is a data release behind the API
|
||||
|
||||
This applies when the two `idc-index-data` **majors** differ — the API serving `25.x.y` against
|
||||
a local `24.x.y`, say. A newer build of the same release (24.2.2 vs 24.2.0) covers the same
|
||||
series, so manifests from it resolve locally and downloads are unaffected.
|
||||
|
||||
`idc download-from-manifest` does not simply hand the URLs to a transfer client: it extracts
|
||||
each `crdc_series_uuid` from the manifest and joins it against the **local** index (then
|
||||
against `prior_versions_index`) to compute sizes and build the output hierarchy. A manifest
|
||||
produced by an API serving a newer IDC data release can therefore contain series the local
|
||||
index has never heard of.
|
||||
|
||||
Those rows are **not** downloaded, and the command does not fail — it logs, then continues
|
||||
with the rest:
|
||||
|
||||
```
|
||||
The total of N copy commands are not recognized as referencing any associated series in the
|
||||
main index. ... they may correspond to files available in a release of IDC different from v24
|
||||
used in this version of idc-index.
|
||||
...
|
||||
The corresponding files could not be downloaded.
|
||||
```
|
||||
|
||||
The result is a partial download that otherwise looks successful. `download_from_selection(seriesInstanceUID=…)`
|
||||
has the same blind spot from the other direction: it filters the local index, so UIDs it does
|
||||
not contain are silently dropped from the selection.
|
||||
|
||||
**Fix it one of two ways:**
|
||||
|
||||
1. **Upgrade** — upgrade `idc-index` (`python scripts/check_version.py` prints the command),
|
||||
then re-run. This is the right answer
|
||||
whenever it is possible; it restores the hierarchy, size checks, and progress reporting.
|
||||
2. **Bypass the index** — transfer directly from the bucket. The manifest URLs are
|
||||
self-contained (`s3://<bucket>/<crdc_series_uuid>/*`), so no index is needed at all:
|
||||
|
||||
```bash
|
||||
# one directory per series, named by crdc_series_uuid
|
||||
awk -F/ '{print "cp " $0 " ./idc-data/" $4 "/"}' idc_manifest.txt > s5cmd_commands.txt
|
||||
s5cmd --no-sign-request run s5cmd_commands.txt
|
||||
|
||||
# for source=gcs manifests, add the GCS endpoint
|
||||
s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run s5cmd_commands.txt
|
||||
```
|
||||
|
||||
`s5cmd run` expects one *command* per line, which is why the bare URLs are rewritten as `cp`
|
||||
commands; drop the `$4` segment to land every file flat in `./idc-data/`. `aws s3 cp
|
||||
--no-sign-request --recursive` works the same way per URL.
|
||||
|
||||
What you give up by bypassing `idc-index` is convenience, not data: files are laid out under
|
||||
CRDC UUIDs instead of `%collection_id/%PatientID/%Modality`, and there is no local size or
|
||||
disk-space check — so read `size_TB` from `cohort/counts` first. See
|
||||
`references/cloud_storage_guide.md` for bucket layout, `aws`/`gsutil` equivalents, and
|
||||
UUID-to-DICOM-UID mapping.
|
||||
|
||||
**`source`: `aws` (default) vs `gcs`.** Both return `s3://` URLs — GCS is reached through its
|
||||
S3-compatible endpoint, never a `gs://` URL. That is why `idc download-from-manifest` only
|
||||
recognizes `s3://` lines. Driving `s5cmd` yourself, use `--no-sign-request`, and for
|
||||
`source=gcs` add `--endpoint-url https://storage.googleapis.com`.
|
||||
|
||||
IDC is ~99 TB across 176 collections. Always report `series` and `size_TB` from
|
||||
`cohort/counts` and confirm with the user before starting a broad download.
|
||||
|
||||
## Limits, Defaults, and Errors
|
||||
|
||||
Measured against `3.0.0b3`. Values above a cap are silently clamped — the response echoes the
|
||||
value actually used (`max_rows`, `page_size`), so read it back rather than assuming the request
|
||||
was honored.
|
||||
|
||||
| Endpoint | Parameter | Default | Cap |
|
||||
|----------|-----------|---------|-----|
|
||||
| `GET /attributes/{attr}/values` | `limit` | 100 | 10000 |
|
||||
| `POST /sql` | `max_rows` | 5000 | 10000 |
|
||||
| `GET /clinical/tables/{table}/rows` | `max_rows` | 5000 | 100000 |
|
||||
| `POST /cohort/manifest` | `page_size` | 100 | 5000 |
|
||||
| `POST /cohort/manifest` | `page` | 0 | — |
|
||||
| `POST /cohort/manifest` | `include_rows` | `true` | — |
|
||||
| `POST /cohort/manifest.txt` | `limit` | 100000 | — |
|
||||
| `POST /cohort/manifest.txt` | `source` | `aws` | — |
|
||||
| `POST /citations` | `citation_format` | `apa` | — |
|
||||
|
||||
`cohort/manifest.txt` is the surface that is not *row*-capped the way `/sql` is — it returned all
|
||||
774 lines for `rider_pilot` with no `limit` set, and enumerates up to 100 000 series. That is why
|
||||
bulk series belong there rather than in a `/sql` dump.
|
||||
|
||||
There is **no per-caller rate limit or quota** and no `429`. What is bounded is the individual
|
||||
request: a 30 s SQL statement timeout, 4 GB query memory, and the caps above. A burst is absorbed
|
||||
by autoscaling and surfaces as slower responses or a `503` — back off and retry rather than
|
||||
treating it as permanent. For sustained heavy metadata access, query the `idc-index` Parquet files
|
||||
(`references/parquet_access_guide.md`) or BigQuery instead of driving this API hard.
|
||||
|
||||
Size-capped responses carry a `truncated` boolean: `false` means the result is complete, `true`
|
||||
means raise the limit or aggregate/narrow instead. Explore narrow, then widen.
|
||||
|
||||
Errors come in two shapes. Semantic problems are HTTP 400 with a uniform body:
|
||||
|
||||
```json
|
||||
{"error": {"code": "invalid_query", "message": "Unknown or non-term filter attribute: 'NotAnAttribute'. Use list_attributes to see valid attributes."}}
|
||||
```
|
||||
|
||||
Request-shape problems are HTTP 422 with FastAPI's `detail[]` array, which names the offending
|
||||
key — a bare filter object, an unrecognized key, or a misspelled range bound all land here. Both
|
||||
kinds are actionable: an unknown attribute names the discovery call to make, and a bad column name
|
||||
carries DuckDB's candidate bindings. Read the message and fix the request rather than retrying it
|
||||
unchanged.
|
||||
|
||||
What does **not** produce an error is a filter that is valid but empty or over-broad. Those return
|
||||
HTTP 200 with a `warnings` entry saying so — see *The server reports what it filtered on*. Read
|
||||
`warnings`; do not infer from the count alone.
|
||||
|
||||
## Handing Off to idc-index
|
||||
|
||||
The boundary artifact is a list of `SeriesInstanceUID` values (or a saved `manifest.txt`).
|
||||
|
||||
```python
|
||||
# UIDs from a cohort/manifest or /sql response
|
||||
series_uids = [row["SeriesInstanceUID"] for row in manifest["series"]]
|
||||
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
|
||||
client.download_from_selection(
|
||||
downloadDir="./data",
|
||||
seriesInstanceUID=series_uids, # a list, not a DataFrame
|
||||
dirTemplate="%collection_id/%PatientID/%Modality",
|
||||
)
|
||||
```
|
||||
|
||||
Run `python scripts/check_version.py` before the first `idc-index` call in a session, even when
|
||||
discovery happened over the API — the two version independently. Compare
|
||||
`idc_index_data_version` on both sides first (see *Checking the API against a local
|
||||
idc-index*).
|
||||
|
||||
**This handoff is only valid while the two are on the same IDC data release** — the same
|
||||
`idc-index-data` major. `idc-index` can only download series its own index lists, so when the
|
||||
API is a release ahead, UIDs and manifest URLs it returned may resolve to nothing locally:
|
||||
`download_from_selection` silently drops them and `download-from-manifest` logs them as
|
||||
unrecognized and skips them. Do not report that as "no data": name both versions, then either
|
||||
upgrade `idc-index` or download straight from the bucket, as described in *When the local index
|
||||
is a data release behind the API*.
|
||||
|
||||
## What the API Does Not Cover
|
||||
|
||||
- **Image bytes.** The API returns URLs and manifests only; files transfer from S3/GCS.
|
||||
- **Pixel data access and DICOMweb.** Use `references/dicomweb_guide.md`.
|
||||
- **Full DICOM metadata, per-segment detail, SR quantitative/qualitative measurements, private
|
||||
DICOM elements.** Still BigQuery-only — see `references/bigquery_guide.md`.
|
||||
- **Writes.** The service is read-only by construction; there are no POST endpoints that mutate
|
||||
state, and the SQL connection rejects anything but `SELECT`.
|
||||
- **Local analysis.** DataFrames, plotting, pydicom/SimpleITK, pathology tiling all stay with
|
||||
`idc-index`.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- IDC REST API docs: https://learn.canceridc.dev/rest-api/api
|
||||
- Swagger UI: https://api.imaging.datacommons.cancer.gov/v3/docs
|
||||
- API and MCP server source: https://github.com/ImagingDataCommons/IDC-REST-MCP
|
||||
- `references/mcp_guide.md` — the same capabilities as agent tools
|
||||
- `references/sql_patterns.md` — SQL that transfers unchanged to `POST /sql`
|
||||
- `references/cli_guide.md` — `idc download-from-manifest` and the rest of the CLI
|
||||
@@ -1,6 +1,6 @@
|
||||
# SQL Query Patterns for IDC
|
||||
|
||||
**Tested with:** idc-index 0.11.14 (IDC data version v23)
|
||||
**Tested with:** idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
Quick reference for common SQL query patterns when working with IDC data. For detailed examples with context, see the "Core Capabilities" section in the main SKILL.md.
|
||||
|
||||
@@ -14,20 +14,73 @@ Load this guide when you need quick-reference SQL patterns for:
|
||||
- Linking imaging data to clinical data
|
||||
- Filtering by 3D volume geometry validity (volume_geometry_index)
|
||||
- Finding RT Structure Set series and ROI metadata (rtstruct_index)
|
||||
- Filtering by CT/MR/PET acquisition parameters (ct_index, mr_index, pt_index)
|
||||
|
||||
For table schemas, DataFrame access, and join column references, see `references/index_tables_guide.md`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
|
||||
version and prints the install command for the interpreter you are running.
|
||||
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
client = IDCClient()
|
||||
```
|
||||
|
||||
## Overall Data Scale
|
||||
|
||||
Counts and total size across all of IDC — useful for orienting a user, and for sanity-checking
|
||||
that the index loaded the release you expect:
|
||||
|
||||
```python
|
||||
stats = client.sql_query("""
|
||||
SELECT
|
||||
COUNT(DISTINCT collection_id) as collections,
|
||||
COUNT(DISTINCT analysis_result_id) as analysis_results,
|
||||
COUNT(DISTINCT PatientID) as patients,
|
||||
COUNT(DISTINCT StudyInstanceUID) as studies,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series,
|
||||
SUM(instanceCount) as instances,
|
||||
SUM(series_size_MB)/1000000 as size_TB
|
||||
FROM index
|
||||
""")
|
||||
print(stats)
|
||||
```
|
||||
|
||||
### Per-collection breakdown
|
||||
|
||||
```python
|
||||
# Get summary statistics from primary index
|
||||
collections_summary = client.sql_query("""
|
||||
SELECT collection_id,
|
||||
COUNT(DISTINCT PatientID) as patients,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as series,
|
||||
SUM(series_size_MB) as size_mb
|
||||
FROM index
|
||||
GROUP BY collection_id
|
||||
ORDER BY patients DESC
|
||||
""")
|
||||
```
|
||||
|
||||
For richer per-collection metadata — cancer types, tumor locations, species, supporting data —
|
||||
query `collections_index` instead; for derived datasets, `analysis_results_index`. Both need
|
||||
`client.fetch_index(...)` first:
|
||||
|
||||
```python
|
||||
client.fetch_index("collections_index")
|
||||
collections_info = client.sql_query("""
|
||||
SELECT collection_id, cancer_types, tumor_locations, species, subjects, supporting_data
|
||||
FROM collections_index
|
||||
""")
|
||||
|
||||
client.fetch_index("analysis_results_index")
|
||||
analysis_info = client.sql_query("""
|
||||
SELECT analysis_result_id, analysis_result_title, subjects, collections, modalities
|
||||
FROM analysis_results_index
|
||||
""")
|
||||
```
|
||||
|
||||
## Discover Available Filter Values
|
||||
|
||||
```python
|
||||
@@ -74,7 +127,7 @@ client.sql_query("""
|
||||
# List analysis result collections (curated derived datasets)
|
||||
client.fetch_index("analysis_results_index")
|
||||
client.sql_query("""
|
||||
SELECT analysis_result_id, analysis_result_title, Collections, Modalities
|
||||
SELECT analysis_result_id, analysis_result_title, collections, modalities
|
||||
FROM analysis_results_index
|
||||
""")
|
||||
|
||||
@@ -187,6 +240,51 @@ client.sql_query("""
|
||||
|
||||
See `references/clinical_data_guide.md` for complete patterns including value mapping and patient cohort selection.
|
||||
|
||||
## Version Tracking — "What's New in IDC vX?"
|
||||
|
||||
Use `series_init_idc_version` and `series_revised_idc_version` in the main `index` table. Do NOT
|
||||
use `prior_versions_index` for this — it contains only removed series.
|
||||
|
||||
```python
|
||||
VERSION = 24 # Replace with target version
|
||||
|
||||
# Series added for the first time in vVERSION
|
||||
client.sql_query(f"""
|
||||
SELECT collection_id,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as new_series,
|
||||
ROUND(SUM(series_size_MB)/1000, 2) as size_GB
|
||||
FROM index
|
||||
WHERE series_init_idc_version = {VERSION}
|
||||
GROUP BY collection_id
|
||||
ORDER BY new_series DESC
|
||||
""")
|
||||
|
||||
# Series revised (updated content) in vVERSION but originally added earlier
|
||||
client.sql_query(f"""
|
||||
SELECT collection_id,
|
||||
COUNT(DISTINCT SeriesInstanceUID) as revised_series
|
||||
FROM index
|
||||
WHERE series_revised_idc_version = {VERSION}
|
||||
AND series_init_idc_version < {VERSION}
|
||||
GROUP BY collection_id
|
||||
ORDER BY revised_series DESC
|
||||
""")
|
||||
|
||||
# When was each collection first added to IDC?
|
||||
client.fetch_index("version_metadata_index")
|
||||
client.sql_query("""
|
||||
WITH first_versions AS (
|
||||
SELECT collection_id, MIN(series_init_idc_version) as first_version
|
||||
FROM index
|
||||
GROUP BY collection_id
|
||||
)
|
||||
SELECT f.collection_id, f.first_version, v.version_timestamp as first_release_date
|
||||
FROM first_versions f
|
||||
JOIN version_metadata_index v ON f.first_version = v.idc_version
|
||||
ORDER BY f.first_version DESC
|
||||
""")
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Query returns error "table not found"
|
||||
@@ -277,6 +375,84 @@ client.sql_query("""
|
||||
""")
|
||||
```
|
||||
|
||||
## Modality Acquisition Parameters
|
||||
|
||||
`ct_index`, `mr_index`, and `pt_index` (added in idc-index 0.12.3) expose acquisition and reconstruction parameters for CT, MR, and PET series. All join on `SeriesInstanceUID`. Dose-modulated CT acquisitions have `_min`/`_max` columns for tube current, exposure, and exposure time.
|
||||
|
||||
```python
|
||||
client.fetch_index("ct_index")
|
||||
client.fetch_index("mr_index")
|
||||
client.fetch_index("pt_index")
|
||||
|
||||
# CT: thin-slice series (≤2mm) with standard reconstruction
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, i.SeriesInstanceUID, i.BodyPartExamined,
|
||||
c.SliceThickness, c.ConvolutionKernel, c.KVP
|
||||
FROM index i
|
||||
JOIN ct_index c ON i.SeriesInstanceUID = c.SeriesInstanceUID
|
||||
WHERE c.SliceThickness <= 2.0
|
||||
AND c.ConvolutionKernel IS NOT NULL
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# CT: dose-modulated acquisitions (tube current varies across slices)
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, c.SeriesInstanceUID,
|
||||
c.XRayTubeCurrent_min, c.XRayTubeCurrent_max, c.SliceThickness
|
||||
FROM ct_index c
|
||||
JOIN index i ON c.SeriesInstanceUID = i.SeriesInstanceUID
|
||||
WHERE c.XRayTubeCurrent_min != c.XRayTubeCurrent_max
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# MR: DWI series (have non-null DiffusionBValue) at 3T
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, i.SeriesInstanceUID, i.SeriesDescription,
|
||||
m.MagneticFieldStrength, m.DiffusionBValue
|
||||
FROM index i
|
||||
JOIN mr_index m ON i.SeriesInstanceUID = m.SeriesInstanceUID
|
||||
WHERE m.DiffusionBValue IS NOT NULL
|
||||
AND m.MagneticFieldStrength >= 2.9
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# MR: multi-echo series (EchoTime stored as array with multiple values)
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, i.SeriesInstanceUID,
|
||||
m.EchoTime, m.EchoTrainLength, m.ScanningSequence
|
||||
FROM index i
|
||||
JOIN mr_index m ON i.SeriesInstanceUID = m.SeriesInstanceUID
|
||||
WHERE m.EchoTrainLength > 1
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# PET: FDG studies with specific reconstruction method
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, i.SeriesInstanceUID,
|
||||
p.RadionuclideCodeMeaning, p.ReconstructionMethod,
|
||||
p.Units, p.DecayCorrection
|
||||
FROM index i
|
||||
JOIN pt_index p ON i.SeriesInstanceUID = p.SeriesInstanceUID
|
||||
WHERE p.RadionuclideCodeMeaning LIKE '%fluorodeoxyglucose%'
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# PET: dynamic acquisitions (ActualFrameDuration is array with multiple values)
|
||||
client.sql_query("""
|
||||
SELECT i.collection_id, i.SeriesInstanceUID,
|
||||
p.NumberOfTimeSlices, p.ActualFrameDuration
|
||||
FROM index i
|
||||
JOIN pt_index p ON i.SeriesInstanceUID = p.SeriesInstanceUID
|
||||
WHERE p.NumberOfTimeSlices > 1
|
||||
LIMIT 10
|
||||
""")
|
||||
```
|
||||
|
||||
Key columns by table (use `client.indices_overview["ct_index"]["schema"]` for the full list):
|
||||
- **ct_index**: `SliceThickness`, `KVP`, `ConvolutionKernel`, `SpiralPitchFactor`, `XRayTubeCurrent_min/max`, `Exposure_min/max`, `PixelSpacing_row_mm/col_mm`, `Rows`, `Columns`
|
||||
- **mr_index**: `MagneticFieldStrength`, `ScanningSequence`, `SequenceVariant`, `MRAcquisitionType`, `EchoTime` (array), `RepetitionTime`, `FlipAngle`, `DiffusionBValue` (array), `NumberOfTemporalPositions`, `ReceiveCoilName`
|
||||
- **pt_index**: `RadionuclideCodeMeaning`, `Radiopharmaceutical`, `RadionuclideTotalDose`, `ReconstructionMethod`, `DecayCorrection`, `AttenuationCorrectionMethod`, `ActualFrameDuration` (array), `NumberOfTimeSlices`
|
||||
|
||||
## Resources
|
||||
|
||||
- `references/index_tables_guide.md` for table schemas, DataFrame access, and join column references
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Common Use Cases for IDC
|
||||
|
||||
**Tested with:** idc-index 0.11.9 (IDC data version v23)
|
||||
**Tested with:** idc-index 0.12.5 (IDC data version v24)
|
||||
|
||||
This guide provides complete end-to-end workflow examples for common IDC use cases. Each use case demonstrates the full workflow from query to download with best practices.
|
||||
|
||||
@@ -16,9 +16,8 @@ For core API patterns (query, download, visualize, citations), see the "Core Cap
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install 'idc-index==0.11.14'
|
||||
```
|
||||
Needs `idc-index` installed — run `python scripts/check_version.py`, which reports the installed
|
||||
version and prints the install command for the interpreter you are running.
|
||||
|
||||
## Use Case 1: Find and Download Lung CT Scans for Deep Learning
|
||||
|
||||
@@ -178,6 +177,98 @@ client.download_from_selection(
|
||||
cc_by_data.to_csv('commercial_dataset_manifest_CC-BY_ONLY.csv', index=False)
|
||||
```
|
||||
|
||||
## Use Case 5: Batch Download with Filtering
|
||||
|
||||
**Objective:** Download a large filtered dataset in batches to avoid timeouts
|
||||
|
||||
**Steps:**
|
||||
```python
|
||||
from idc_index import IDCClient
|
||||
import pandas as pd
|
||||
|
||||
client = IDCClient()
|
||||
|
||||
# Find chest CT scans from GE scanners with a permissive license
|
||||
query = """
|
||||
SELECT
|
||||
SeriesInstanceUID,
|
||||
PatientID,
|
||||
collection_id,
|
||||
ManufacturerModelName
|
||||
FROM index
|
||||
WHERE Modality = 'CT'
|
||||
AND BodyPartExamined = 'CHEST'
|
||||
AND Manufacturer = 'GE MEDICAL SYSTEMS'
|
||||
AND license_short_name = 'CC BY 4.0'
|
||||
LIMIT 100
|
||||
"""
|
||||
|
||||
results = client.sql_query(query)
|
||||
|
||||
# Save manifest for reproducibility
|
||||
results.to_csv('lung_ct_manifest.csv', index=False)
|
||||
|
||||
# Download in batches to avoid timeout
|
||||
batch_size = 10
|
||||
for i in range(0, len(results), batch_size):
|
||||
batch = results.iloc[i:i+batch_size]
|
||||
client.download_from_selection(
|
||||
seriesInstanceUID=list(batch['SeriesInstanceUID'].values),
|
||||
downloadDir=f"./data/batch_{i//batch_size}"
|
||||
)
|
||||
```
|
||||
|
||||
## Use Case 6: Integration with Analysis Pipelines
|
||||
|
||||
**Objective:** Load downloaded DICOM files into Python for processing
|
||||
|
||||
**Read individual DICOM files with pydicom:**
|
||||
```python
|
||||
import pydicom
|
||||
import os
|
||||
|
||||
series_dir = "./data/rider/rider_pilot/RIDER-1007893286/CT_1.3.6.1..."
|
||||
|
||||
dicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir)
|
||||
if f.endswith('.dcm')]
|
||||
|
||||
ds = pydicom.dcmread(dicom_files[0])
|
||||
print(f"Patient ID: {ds.PatientID}")
|
||||
print(f"Modality: {ds.Modality}")
|
||||
print(f"Image shape: {ds.pixel_array.shape}")
|
||||
```
|
||||
|
||||
**Build 3D volume from CT series:**
|
||||
```python
|
||||
import pydicom
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
def load_ct_series(series_path):
|
||||
files = sorted(Path(series_path).glob('*.dcm'))
|
||||
slices = [pydicom.dcmread(str(f)) for f in files]
|
||||
slices.sort(key=lambda x: float(x.ImagePositionPatient[2]))
|
||||
volume = np.stack([s.pixel_array for s in slices])
|
||||
return volume, slices[0]
|
||||
|
||||
volume, metadata = load_ct_series("./data/lung_ct/series_dir")
|
||||
print(f"Volume shape: {volume.shape}") # (z, y, x)
|
||||
```
|
||||
|
||||
**Load DICOM series with SimpleITK (recommended for correct geometry):**
|
||||
```python
|
||||
import SimpleITK as sitk
|
||||
|
||||
series_path = "./data/ct_series"
|
||||
reader = sitk.ImageSeriesReader()
|
||||
dicom_names = reader.GetGDCMSeriesFileNames(series_path)
|
||||
reader.SetFileNames(dicom_names)
|
||||
image = reader.Execute()
|
||||
|
||||
smoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5)
|
||||
sitk.WriteImage(smoothed, "processed_volume.nii.gz")
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
- Main SKILL.md for core API patterns (query, download, visualize)
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check the idc-index package and this skill for required/available updates.
|
||||
|
||||
Run FIRST at the start of an IDC session: python scripts/check_version.py
|
||||
|
||||
- Verifies that idc-index is installed and at least MIN_VERSION. It never
|
||||
installs or upgrades anything itself: if the requirement is not met it prints
|
||||
the command to run — targeting the interpreter that ran this script, via uv
|
||||
when uv is available — and exits non-zero, leaving the choice of Python
|
||||
environment to the caller.
|
||||
- Notifies (only) when a newer idc-index (PyPI) or skill release (GitHub) is
|
||||
available. Network checks are best-effort and silently skipped offline.
|
||||
|
||||
Keep MIN_VERSION and SKILL_VERSION in sync with the SKILL.md frontmatter.
|
||||
"""
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
MIN_VERSION = "0.12.5" # keep in sync with metadata.idc-index in SKILL.md
|
||||
SKILL_VERSION = "1.8.1" # keep in sync with metadata.version in SKILL.md
|
||||
REPO = "ImagingDataCommons/imaging-data-commons-skill"
|
||||
|
||||
_LEADING_DIGITS = re.compile(r"\d+")
|
||||
|
||||
|
||||
def parse_version(v):
|
||||
"""Numeric 3-tuple for comparison (string comparison misorders multi-digit parts).
|
||||
|
||||
Tolerates pre-release and suffixed tags by taking the leading digits of each
|
||||
component: "0.13.0rc1" and "v1.7.0-beta" parse as (0, 13, 0) and (1, 7, 0)
|
||||
rather than raising. A pre-release therefore compares equal to its base
|
||||
release, which keeps the update notices conservative instead of advertising
|
||||
unreleased versions.
|
||||
"""
|
||||
parts = []
|
||||
for part in v.lstrip("v").split(".")[:3]:
|
||||
match = _LEADING_DIGITS.match(part)
|
||||
parts.append(int(match.group()) if match else 0)
|
||||
return tuple(parts + [0] * (3 - len(parts)))
|
||||
|
||||
|
||||
def fetch_json(url, *keys):
|
||||
"""Best-effort JSON fetch, drilling into nested keys; None if unreachable."""
|
||||
import json
|
||||
import urllib.request
|
||||
try:
|
||||
data = json.load(urllib.request.urlopen(url, timeout=5))
|
||||
for key in keys:
|
||||
data = data[key]
|
||||
return data
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def install_commands(spec, upgrade=False):
|
||||
"""Install commands for the interpreter running this script, preferred first.
|
||||
|
||||
`-m pip` is always offered: every standard interpreter ships it, and naming the
|
||||
interpreter explicitly keeps the install out of whatever other environment a bare
|
||||
`pip` on PATH would resolve to. `uv` is offered ahead of it when it is on PATH, with
|
||||
`--python` for the same reason — otherwise `uv pip install` targets the active
|
||||
virtual environment, which is not necessarily this one.
|
||||
|
||||
Neither form overrides the PEP 668 guard on an externally managed interpreter. Both
|
||||
refuse there, which is the intended outcome, not a gap to work around.
|
||||
"""
|
||||
flag = "--upgrade " if upgrade else ""
|
||||
commands = [f"{sys.executable} -m pip install {flag}'{spec}'"]
|
||||
if shutil.which("uv"):
|
||||
commands.insert(0, f"uv pip install --python {sys.executable} {flag}'{spec}'")
|
||||
return commands
|
||||
|
||||
|
||||
def print_install_instructions(spec):
|
||||
"""Print how to install `spec` — this script never modifies the environment."""
|
||||
print("\nInstall the vetted version with:\n")
|
||||
for command in install_commands(spec):
|
||||
print(f" {command}")
|
||||
print()
|
||||
print("Use a virtual environment where you can; on an externally managed system Python")
|
||||
print("(PEP 668) the install is refused until you use a virtual environment or `--user`.")
|
||||
print("Re-run this script once the install finishes.")
|
||||
|
||||
|
||||
def check_minimum():
|
||||
"""Check the installed idc-index against MIN_VERSION.
|
||||
|
||||
Returns the installed version string, or None if idc-index is missing or
|
||||
older than MIN_VERSION — in which case install instructions are printed and
|
||||
the caller should not proceed until they have been followed.
|
||||
"""
|
||||
try:
|
||||
import idc_index
|
||||
except ImportError:
|
||||
print(f"idc-index is not installed; this skill requires {MIN_VERSION} or newer.")
|
||||
print_install_instructions(f"idc-index=={MIN_VERSION}")
|
||||
return None
|
||||
|
||||
installed = idc_index.__version__
|
||||
if parse_version(installed) < parse_version(MIN_VERSION):
|
||||
print(f"idc-index {installed} is below the pinned minimum {MIN_VERSION}.")
|
||||
print_install_instructions(f"idc-index=={MIN_VERSION}")
|
||||
return None
|
||||
|
||||
print(f"idc-index {installed} meets pinned minimum ({MIN_VERSION})")
|
||||
return installed
|
||||
|
||||
|
||||
def notify_updates(installed):
|
||||
"""Print notices when newer idc-index or skill versions are available."""
|
||||
if installed:
|
||||
pkg = fetch_json("https://pypi.org/pypi/idc-index/json", "info", "version")
|
||||
if pkg and parse_version(pkg) > parse_version(installed):
|
||||
print(f"ℹ️ idc-index {pkg} available — to update: "
|
||||
f"{install_commands('idc-index', upgrade=True)[0]}")
|
||||
|
||||
tag = fetch_json(f"https://api.github.com/repos/{REPO}/releases/latest", "tag_name")
|
||||
if tag and parse_version(tag) > parse_version(SKILL_VERSION):
|
||||
print(f"ℹ️ Skill {tag.lstrip('v')} available (you have {SKILL_VERSION}): "
|
||||
f"https://github.com/{REPO}/releases/latest")
|
||||
|
||||
|
||||
def main():
|
||||
"""Exit 0 when the pinned minimum is installed, 1 when the caller must install it."""
|
||||
installed = check_minimum()
|
||||
notify_updates(installed)
|
||||
return 0 if installed else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Contract for the one script imaging-data-commons bundles, `scripts/check_version.py`.
|
||||
|
||||
The skill's startup step. It reports whether `idc-index` is installed and at least the
|
||||
version the skill was tested against, prints the install command for the interpreter
|
||||
that ran it, and exits non-zero so the agent stops rather than querying a stale index.
|
||||
It never installs anything — see `TestNeverInstalls` for why that is load-bearing.
|
||||
|
||||
Offline and dependency-free: no network, no `idc-index`, standard library plus pytest.
|
||||
Importing the module is side-effect free — `idc_index`, `json`, and `urllib` are all
|
||||
imported inside the functions that need them — so this suite runs in the bare project
|
||||
environment, not only under `tests/run_all.py --isolated`.
|
||||
|
||||
Kept in sync with the upstream copy at
|
||||
https://github.com/ImagingDataCommons/imaging-data-commons-skill/blob/main/tests/test_check_version.py
|
||||
which is written to be vendored: only the two paths below differ. Re-copy it when the
|
||||
skill is synced.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
SKILL_ROOT = os.path.join(
|
||||
os.path.dirname(__file__), "..", "..", "skills", "imaging-data-commons"
|
||||
)
|
||||
sys.path.insert(0, os.path.join(SKILL_ROOT, "scripts"))
|
||||
import check_version # noqa: E402
|
||||
|
||||
_SKILL_MD = os.path.join(SKILL_ROOT, "SKILL.md")
|
||||
|
||||
|
||||
class TestParseVersion:
|
||||
def test_orders_numerically(self):
|
||||
# String comparison would (wrongly) order "0.12.0" < "0.9.0".
|
||||
assert check_version.parse_version("0.12.0") > check_version.parse_version("0.9.0")
|
||||
|
||||
def test_strips_v_prefix(self):
|
||||
assert check_version.parse_version("v1.6.5") == (1, 6, 5)
|
||||
|
||||
def test_tolerates_prereleases(self):
|
||||
# A pre-release tag upstream must not crash the startup check. It compares equal
|
||||
# to its base release, so the update notices stay conservative.
|
||||
assert check_version.parse_version("0.13.0rc1") == (0, 13, 0)
|
||||
assert check_version.parse_version("v1.7.0-beta") == (1, 7, 0)
|
||||
assert check_version.parse_version("0.13.0rc1") == check_version.parse_version("0.13.0")
|
||||
|
||||
def test_pads_short_versions(self):
|
||||
assert check_version.parse_version("1.7") == (1, 7, 0)
|
||||
assert check_version.parse_version("2") == (2, 0, 0)
|
||||
|
||||
def test_ignores_components_past_the_third(self):
|
||||
assert check_version.parse_version("1.2.3.4") == (1, 2, 3)
|
||||
|
||||
|
||||
class TestInstallCommands:
|
||||
"""The printed command must name this interpreter, and must not override PEP 668."""
|
||||
|
||||
def test_pip_form_targets_the_running_interpreter(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version.shutil, "which", lambda _: None)
|
||||
commands = check_version.install_commands("idc-index==0.12.5")
|
||||
assert commands == [f"{sys.executable} -m pip install 'idc-index==0.12.5'"]
|
||||
|
||||
def test_uv_form_is_preferred_when_uv_is_available(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version.shutil, "which", lambda name: f"/usr/bin/{name}")
|
||||
commands = check_version.install_commands("idc-index==0.12.5")
|
||||
assert len(commands) == 2
|
||||
# uv pip install without --python targets the *active* environment, which is not
|
||||
# necessarily the one that failed to import idc_index.
|
||||
assert commands[0] == (
|
||||
f"uv pip install --python {sys.executable} 'idc-index==0.12.5'"
|
||||
)
|
||||
assert commands[1].startswith(f"{sys.executable} -m pip install")
|
||||
|
||||
def test_upgrade_flag_applies_to_every_form(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version.shutil, "which", lambda name: f"/usr/bin/{name}")
|
||||
commands = check_version.install_commands("idc-index", upgrade=True)
|
||||
assert all("--upgrade" in command for command in commands)
|
||||
|
||||
def test_no_command_bypasses_an_externally_managed_interpreter(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version.shutil, "which", lambda name: f"/usr/bin/{name}")
|
||||
for spec in ("idc-index", "idc-index==0.12.5"):
|
||||
for command in check_version.install_commands(spec, upgrade=True):
|
||||
assert "--break-system-packages" not in command
|
||||
assert "--system" not in command
|
||||
|
||||
def test_instructions_name_the_version_and_the_interpreter(self, capsys):
|
||||
check_version.print_install_instructions(f"idc-index=={check_version.MIN_VERSION}")
|
||||
out = capsys.readouterr().out
|
||||
assert check_version.MIN_VERSION in out
|
||||
assert sys.executable in out
|
||||
assert "virtual environment" in out
|
||||
|
||||
|
||||
class TestNeverInstalls:
|
||||
"""The script reports and instructs; it must not mutate the environment.
|
||||
|
||||
Auto-installing into whatever interpreter `pip` happens to resolve to can silently
|
||||
rewrite a user's global site-packages — and, with a capped dependency, downgrade a
|
||||
package the environment needs. Installation is the caller's decision.
|
||||
"""
|
||||
|
||||
def _source(self):
|
||||
with open(check_version.__file__, encoding="utf-8") as handle:
|
||||
return handle.read()
|
||||
|
||||
def test_source_has_no_installer_call(self):
|
||||
for forbidden in ("subprocess", "--break-system-packages", "pip3", "os.system"):
|
||||
assert forbidden not in self._source(), (
|
||||
f"check_version.py must not use {forbidden}"
|
||||
)
|
||||
|
||||
def test_source_has_no_dynamic_execution(self):
|
||||
# Downstream registries reject bundled scripts that call eval/exec.
|
||||
for forbidden in ("eval(", "exec("):
|
||||
assert forbidden not in self._source()
|
||||
|
||||
|
||||
class TestMinimumCheck:
|
||||
"""check_minimum() reports the installed version, or None with instructions."""
|
||||
|
||||
def test_missing_package_reports_none_and_instructs(self, monkeypatch, capsys):
|
||||
# `None` in sys.modules makes `import idc_index` raise ImportError, so this test
|
||||
# exercises the not-installed path whether or not the package is present.
|
||||
monkeypatch.setitem(sys.modules, "idc_index", None)
|
||||
assert check_version.check_minimum() is None
|
||||
out = capsys.readouterr().out
|
||||
assert "not installed" in out
|
||||
assert check_version.MIN_VERSION in out
|
||||
|
||||
def test_older_version_reports_none(self, monkeypatch, capsys):
|
||||
monkeypatch.setitem(sys.modules, "idc_index", _FakeIdcIndex("0.9.0"))
|
||||
assert check_version.check_minimum() is None
|
||||
assert "below the pinned minimum" in capsys.readouterr().out
|
||||
|
||||
def test_current_version_is_returned(self, monkeypatch, capsys):
|
||||
monkeypatch.setitem(sys.modules, "idc_index", _FakeIdcIndex("99.0.0"))
|
||||
assert check_version.check_minimum() == "99.0.0"
|
||||
assert "meets pinned minimum" in capsys.readouterr().out
|
||||
|
||||
def test_main_exit_code_follows_the_check(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version, "notify_updates", lambda _: None)
|
||||
monkeypatch.setattr(check_version, "check_minimum", lambda: None)
|
||||
assert check_version.main() == 1
|
||||
monkeypatch.setattr(check_version, "check_minimum", lambda: "0.12.5")
|
||||
assert check_version.main() == 0
|
||||
|
||||
|
||||
class TestNetworkChecksAreBestEffort:
|
||||
def test_fetch_json_returns_none_when_unreachable(self):
|
||||
# .invalid never resolves (RFC 2606), so this stays offline.
|
||||
assert check_version.fetch_json("https://pypi.invalid/pypi/idc-index/json", "info") is None
|
||||
|
||||
def test_notify_updates_survives_an_unreachable_network(self, monkeypatch):
|
||||
monkeypatch.setattr(check_version, "fetch_json", lambda *args, **kwargs: None)
|
||||
check_version.notify_updates("0.12.5") # must not raise
|
||||
|
||||
def test_skill_update_notice_is_conservative(self, monkeypatch, capsys):
|
||||
# A GitHub tag equal to the shipped version is not an update.
|
||||
monkeypatch.setattr(
|
||||
check_version, "fetch_json", lambda *args, **kwargs: f"v{check_version.SKILL_VERSION}"
|
||||
)
|
||||
check_version.notify_updates(None)
|
||||
assert "available" not in capsys.readouterr().out
|
||||
|
||||
|
||||
class TestVersionsMatchFrontmatter:
|
||||
"""The pins in the script and in SKILL.md are read by different consumers."""
|
||||
|
||||
def _frontmatter(self):
|
||||
with open(_SKILL_MD, encoding="utf-8") as handle:
|
||||
return handle.read().split("---", 2)[1]
|
||||
|
||||
def test_min_version_matches_metadata(self):
|
||||
meta = re.search(r'idc-index:\s*"?([\d.]+)"?', self._frontmatter()).group(1)
|
||||
assert check_version.MIN_VERSION == meta
|
||||
|
||||
def test_skill_version_matches_metadata(self):
|
||||
# `metadata.version` is this repository's skill version; the script pins the
|
||||
# upstream release it came from, recorded as `source-skill-version`. The script
|
||||
# compares it against upstream's GitHub releases to notify about skill updates,
|
||||
# so a stale value points users at a release they already have.
|
||||
meta = re.search(
|
||||
r"source-skill-version:\s*\"?([\d.]+)\"?", self._frontmatter()
|
||||
).group(1)
|
||||
assert check_version.SKILL_VERSION == meta
|
||||
|
||||
|
||||
class _FakeIdcIndex:
|
||||
"""Stand-in for the real package, so the check runs with nothing installed."""
|
||||
|
||||
def __init__(self, version):
|
||||
self.__version__ = version
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(pytest.main([__file__, "-v"]))
|
||||
Reference in New Issue
Block a user