Skip to content

Commit c6c45de

Browse files
fedorovfedorov
committed
enh: fix removal of seg directory
1 parent b728c2d commit c6c45de

1 file changed

Lines changed: 203 additions & 8 deletions

File tree

‎modules/idc_dataset.ipynb‎

Lines changed: 203 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,63 @@
55
"metadata": {
66
"id": "eFLP44iEFCpB"
77
},
8-
"source": "Copyright (c) MONAI Consortium \nLicensed under the Apache License, Version 2.0 (the \"License\"); \nyou may not use this file except in compliance with the License. \nYou may obtain a copy of the License at \n    http://www.apache.org/licenses/LICENSE-2.0 \nUnless required by applicable law or agreed to in writing, software \ndistributed under the License is distributed on an \"AS IS\" BASIS, \nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \nSee the License for the specific language governing permissions and \nlimitations under the License.\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ImagingDataCommons/idc-monai/blob/main/monai_contribution/idc_dataset.ipynb)\n\n# Using NCI Imaging Data Commons with MONAI\n\n## What is IDC?\n\n[NCI Imaging Data Commons (IDC)](https://portal.imaging.datacommons.cancer.gov/) is a free, cloud-hosted repository of publicly available cancer imaging data maintained by the National Cancer Institute (NCI). It provides:\n\n- **~100 TB** of radiology (CT, MR, PET) and pathology images across 160+ cancer collections\n- **No sign-up or authentication required** — data is openly accessible\n- **Expert and AI-generated annotations** (e.g., organ segmentations) paired with images\n- **Standardized format** — all data uses DICOM, the medical imaging industry standard\n- **Cloud-native storage** — data lives in Google Cloud Storage (GCS) buckets, so downloads are fast\n- **Accompanying tools** - you can search, visualize, and subset the data\n\n## What is `idc-index`?\n\n[`idc-index`](https://github.com/ImagingDataCommons/idc-index) is a lightweight Python package that lets you search and download IDC data without any cloud account or special credentials. It ships with a local metadata index — a set of DuckDB tables describing every image series in IDC — so you can run SQL queries locally to find exactly the data you need before downloading anything.\n\n## What this tutorial covers\n\nThis tutorial shows how to:\n1. Query IDC metadata with SQL to find cancer imaging data\n2. Download DICOM images and segmentations with one function call\n3. Load the data into MONAI for AI/ML preprocessing\n4. Work with DICOM Segmentation (DICOM-SEG) objects and their rich metadata\n\n> **Tip**: This tutorial was created using [idc-claude-skill](https://github.com/ImagingDataCommons/idc-claude-skill) — an AI assistant skill for navigating IDC data and the `idc-index` API."
8+
"source": [
9+
"Copyright (c) MONAI Consortium \n",
10+
"Licensed under the Apache License, Version 2.0 (the \"License\"); \n",
11+
"you may not use this file except in compliance with the License. \n",
12+
"You may obtain a copy of the License at \n",
13+
"    http://www.apache.org/licenses/LICENSE-2.0 \n",
14+
"Unless required by applicable law or agreed to in writing, software \n",
15+
"distributed under the License is distributed on an \"AS IS\" BASIS, \n",
16+
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n",
17+
"See the License for the specific language governing permissions and \n",
18+
"limitations under the License.\n",
19+
"\n",
20+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ImagingDataCommons/idc-monai/blob/main/monai_contribution/idc_dataset.ipynb)\n",
21+
"\n",
22+
"# Using NCI Imaging Data Commons with MONAI\n",
23+
"\n",
24+
"## What is IDC?\n",
25+
"\n",
26+
"[NCI Imaging Data Commons (IDC)](https://portal.imaging.datacommons.cancer.gov/) is a free, cloud-hosted repository of publicly available cancer imaging data maintained by the National Cancer Institute (NCI). It provides:\n",
27+
"\n",
28+
"- **~100 TB** of radiology (CT, MR, PET) and pathology images across 160+ cancer collections\n",
29+
"- **No sign-up or authentication required** — data is openly accessible\n",
30+
"- **Expert and AI-generated annotations** (e.g., organ segmentations) paired with images\n",
31+
"- **Standardized format** — all data uses DICOM, the medical imaging industry standard\n",
32+
"- **Cloud-native storage** — data lives in Google Cloud Storage (GCS) buckets, so downloads are fast\n",
33+
"- **Accompanying tools** - you can search, visualize, and subset the data\n",
34+
"\n",
35+
"## What is `idc-index`?\n",
36+
"\n",
37+
"[`idc-index`](https://github.com/ImagingDataCommons/idc-index) is a lightweight Python package that lets you search and download IDC data without any cloud account or special credentials. It ships with a local metadata index — a set of DuckDB tables describing every image series in IDC — so you can run SQL queries locally to find exactly the data you need before downloading anything.\n",
38+
"\n",
39+
"## What this tutorial covers\n",
40+
"\n",
41+
"This tutorial shows how to:\n",
42+
"1. Query IDC metadata with SQL to find cancer imaging data\n",
43+
"2. Download DICOM images and segmentations with one function call\n",
44+
"3. Load the data into MONAI for AI/ML preprocessing\n",
45+
"4. Work with DICOM Segmentation (DICOM-SEG) objects and their rich metadata\n",
46+
"\n",
47+
"> **Tip**: This tutorial was created using [idc-claude-skill](https://github.com/ImagingDataCommons/idc-claude-skill) — an AI assistant skill for navigating IDC data and the `idc-index` API."
48+
]
949
},
1050
{
1151
"cell_type": "markdown",
1252
"metadata": {
1353
"id": "iudwt5GoFCpC"
1454
},
15-
"source": "## Setup environment\n\nInstall required packages:\n- `monai` — Medical Open Network for AI, the ML framework used here\n- `idc-index` — Query and download IDC data (includes local metadata index)\n- `itk` / `itkwasm-dicom` — ITK-based DICOM readers used by MONAI's `ITKReader` and our custom DICOM-SEG loader\n\n> **Colab users**: After running the cell below, restart the runtime before continuing (Runtime → Restart runtime). ITK requires a fresh runtime to load correctly after installation."
55+
"source": [
56+
"## Setup environment\n",
57+
"\n",
58+
"Install required packages:\n",
59+
"- `monai` — Medical Open Network for AI, the ML framework used here\n",
60+
"- `idc-index` — Query and download IDC data (includes local metadata index)\n",
61+
"- `itk` / `itkwasm-dicom` — ITK-based DICOM readers used by MONAI's `ITKReader` and our custom DICOM-SEG loader\n",
62+
"\n",
63+
"> **Colab users**: After running the cell below, restart the runtime before continuing (Runtime → Restart runtime). ITK requires a fresh runtime to load correctly after installation."
64+
]
1665
},
1766
{
1867
"cell_type": "code",
@@ -21,7 +70,9 @@
2170
"id": "T4ujv4U7FCpC"
2271
},
2372
"outputs": [],
24-
"source": "!pip install -q monai idc-index itk itkwasm-dicom"
73+
"source": [
74+
"!pip install -q monai idc-index itk itkwasm-dicom"
75+
]
2576
},
2677
{
2778
"cell_type": "markdown",
@@ -43,7 +94,35 @@
4394
"outputId": "f63082ca-81d8-4b02-8de0-662710f6e508"
4495
},
4596
"outputs": [],
46-
"source": "import os\nimport tempfile\nfrom pathlib import Path\nfrom typing import Hashable, Mapping\n\nimport itkwasm_dicom\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport torch\nfrom idc_index import IDCClient\nfrom matplotlib.colors import ListedColormap\n\nimport monai\nfrom monai.config import KeysCollection\nfrom monai.data import Dataset, MetaTensor\nfrom monai.data.image_reader import ITKReader\nfrom monai.transforms import (\n Compose,\n EnsureChannelFirstd,\n LoadImaged,\n MapTransform,\n Orientationd,\n ScaleIntensityRanged,\n Spacingd,\n)\n\nmonai.config.print_config()"
97+
"source": [
98+
"import os\n",
99+
"import tempfile\n",
100+
"from pathlib import Path\n",
101+
"from typing import Hashable, Mapping\n",
102+
"\n",
103+
"import itkwasm_dicom\n",
104+
"import matplotlib.pyplot as plt\n",
105+
"import numpy as np\n",
106+
"import torch\n",
107+
"from idc_index import IDCClient\n",
108+
"from matplotlib.colors import ListedColormap\n",
109+
"\n",
110+
"import monai\n",
111+
"from monai.config import KeysCollection\n",
112+
"from monai.data import Dataset, MetaTensor\n",
113+
"from monai.data.image_reader import ITKReader\n",
114+
"from monai.transforms import (\n",
115+
" Compose,\n",
116+
" EnsureChannelFirstd,\n",
117+
" LoadImaged,\n",
118+
" MapTransform,\n",
119+
" Orientationd,\n",
120+
" ScaleIntensityRanged,\n",
121+
" Spacingd,\n",
122+
")\n",
123+
"\n",
124+
"monai.config.print_config()"
125+
]
47126
},
48127
{
49128
"cell_type": "markdown",
@@ -477,7 +556,121 @@
477556
"outputId": "386c615d-5349-49fb-f412-7a92cfc421d9"
478557
},
479558
"outputs": [],
480-
"source": "class LoadDicomSegd(MapTransform):\n \"\"\"Load DICOM Segmentation (DICOM-SEG) files using ITKWasm.\n\n DICOM-SEG is an enhanced multiframe DICOM format that stores segmentation\n masks with segment metadata including recommended display colors.\n\n The affine matrix is derived directly from DICOM metadata (direction cosines,\n spacing, origin) with LPS→RAS conversion applied to match MONAI's ITKReader\n convention. No axis flipping is performed — orientation is fully encoded in\n the affine via the direction cosine matrix.\n \"\"\"\n\n def __init__(self, keys: KeysCollection, allow_missing_keys: bool = False):\n super().__init__(keys, allow_missing_keys)\n\n def _find_dcm_file(self, path: Path) -> Path:\n \"\"\"Find .dcm file in directory or return path if already a file.\"\"\"\n if path.is_file():\n return path\n dcm_files = list(path.glob(\"*.dcm\"))\n if not dcm_files:\n raise FileNotFoundError(f\"No .dcm files found in {path}\")\n return dcm_files[0]\n\n def _build_affine(self, spacing, origin, direction) -> np.ndarray:\n \"\"\"Build 4x4 affine matrix from DICOM spatial metadata.\n\n Converts from ITK/DICOM LPS convention to MONAI's RAS-like convention\n by negating X and Y world coordinates (LPS→RAS). No axis flips are\n applied — orientation is fully encoded in the affine via the direction\n cosine matrix.\n\n Args:\n spacing: Voxel spacing (X, Y, Z) as returned by itkwasm\n origin: Physical coordinates of voxel [0,0,0] in LPS\n direction: 3x3 direction cosine matrix D where D[i,j] is the\n component of voxel-axis-j's unit vector along LPS\n physical axis i. ITK affine formula:\n world_lps = D @ diag(spacing) @ voxel + origin\n \"\"\"\n lps_to_ras = np.diag([-1.0, -1.0, 1.0])\n affine = np.eye(4)\n affine[:3, :3] = lps_to_ras @ direction @ np.diag(spacing)\n affine[:3, 3] = lps_to_ras @ origin\n return affine\n\n def __call__(self, data: Mapping[Hashable, any]) -> dict[Hashable, any]:\n d = dict(data)\n for key in self.key_iterator(d):\n path = Path(d[key])\n dcm_file = self._find_dcm_file(path)\n\n # Read using ITKWasm\n seg_image, overlay_info = itkwasm_dicom.read_segmentation(dcm_file)\n\n # ITKWasm returns array in (Z, Y, X) order but metadata in (X, Y, Z) order.\n # Transpose to (X, Y, Z) to match metadata — this is a layout convention,\n # not an orientation flip.\n seg_array = np.asarray(seg_image.data).copy()\n seg_array = np.transpose(seg_array, (2, 1, 0))\n\n # Build affine from spatial metadata\n spacing = np.array(seg_image.spacing)\n origin = np.array(seg_image.origin)\n direction = np.array(seg_image.direction).reshape(3, 3)\n\n affine = self._build_affine(spacing, origin, direction)\n\n # Make contiguous (array may be non-contiguous after transpose)\n seg_array = np.ascontiguousarray(seg_array)\n\n # Create MONAI MetaTensor with metadata\n meta_tensor = MetaTensor(seg_array)\n meta_tensor.affine = affine\n meta_tensor.meta[\"filename_or_obj\"] = str(dcm_file)\n meta_tensor.meta[\"overlay_info\"] = overlay_info\n meta_tensor.meta[\"original_channel_dim\"] = \"no_channel\"\n\n d[key] = meta_tensor\n d[f\"{key}_meta_dict\"] = dict(meta_tensor.meta)\n\n return d\n\n\n# Load CT with MONAI's ITKReader\nct_transforms = Compose(\n [\n LoadImaged(keys=[\"image\"], reader=ITKReader()),\n EnsureChannelFirstd(keys=[\"image\"]),\n ]\n)\n\n# Load SEG with our custom LoadDicomSegd\nseg_transforms = Compose(\n [\n LoadDicomSegd(keys=[\"label\"]),\n EnsureChannelFirstd(keys=[\"label\"]),\n ]\n)\n\n# Load both\nimage_path = os.path.join(seg_dir, demo_pair[\"image_uid\"])\nseg_path = os.path.join(seg_dir, demo_pair[\"seg_uid\"])\n\nct_data = ct_transforms({\"image\": image_path})\nseg_data = seg_transforms({\"label\": seg_path})\n\nct_image = ct_data[\"image\"]\nseg_label = seg_data[\"label\"]\n\nprint(f\"CT image shape: {ct_image.shape}\")\nprint(f\"Segmentation shape: {seg_label.shape}\")\nprint(f\"Unique labels: {torch.unique(seg_label)[:10].tolist()}...\") # First 10 labels"
559+
"source": [
560+
"class LoadDicomSegd(MapTransform):\n",
561+
" \"\"\"Load DICOM Segmentation (DICOM-SEG) files using ITKWasm.\n",
562+
"\n",
563+
" DICOM-SEG is an enhanced multiframe DICOM format that stores segmentation\n",
564+
" masks with segment metadata including recommended display colors.\n",
565+
"\n",
566+
" The affine matrix is derived directly from DICOM metadata (direction cosines,\n",
567+
" spacing, origin) with LPS→RAS conversion applied to match MONAI's ITKReader\n",
568+
" convention. No axis flipping is performed — orientation is fully encoded in\n",
569+
" the affine via the direction cosine matrix.\n",
570+
" \"\"\"\n",
571+
"\n",
572+
" def __init__(self, keys: KeysCollection, allow_missing_keys: bool = False):\n",
573+
" super().__init__(keys, allow_missing_keys)\n",
574+
"\n",
575+
" def _find_dcm_file(self, path: Path) -> Path:\n",
576+
" \"\"\"Find .dcm file in directory or return path if already a file.\"\"\"\n",
577+
" if path.is_file():\n",
578+
" return path\n",
579+
" dcm_files = list(path.glob(\"*.dcm\"))\n",
580+
" if not dcm_files:\n",
581+
" raise FileNotFoundError(f\"No .dcm files found in {path}\")\n",
582+
" return dcm_files[0]\n",
583+
"\n",
584+
" def _build_affine(self, spacing, origin, direction) -> np.ndarray:\n",
585+
" \"\"\"Build 4x4 affine matrix from DICOM spatial metadata.\n",
586+
"\n",
587+
" Converts from ITK/DICOM LPS convention to MONAI's RAS-like convention\n",
588+
" by negating X and Y world coordinates (LPS→RAS). No axis flips are\n",
589+
" applied — orientation is fully encoded in the affine via the direction\n",
590+
" cosine matrix.\n",
591+
"\n",
592+
" Args:\n",
593+
" spacing: Voxel spacing (X, Y, Z) as returned by itkwasm\n",
594+
" origin: Physical coordinates of voxel [0,0,0] in LPS\n",
595+
" direction: 3x3 direction cosine matrix D where D[i,j] is the\n",
596+
" component of voxel-axis-j's unit vector along LPS\n",
597+
" physical axis i. ITK affine formula:\n",
598+
" world_lps = D @ diag(spacing) @ voxel + origin\n",
599+
" \"\"\"\n",
600+
" lps_to_ras = np.diag([-1.0, -1.0, 1.0])\n",
601+
" affine = np.eye(4)\n",
602+
" affine[:3, :3] = lps_to_ras @ direction @ np.diag(spacing)\n",
603+
" affine[:3, 3] = lps_to_ras @ origin\n",
604+
" return affine\n",
605+
"\n",
606+
" def __call__(self, data: Mapping[Hashable, any]) -> dict[Hashable, any]:\n",
607+
" d = dict(data)\n",
608+
" for key in self.key_iterator(d):\n",
609+
" path = Path(d[key])\n",
610+
" dcm_file = self._find_dcm_file(path)\n",
611+
"\n",
612+
" # Read using ITKWasm\n",
613+
" seg_image, overlay_info = itkwasm_dicom.read_segmentation(dcm_file)\n",
614+
"\n",
615+
" # ITKWasm returns array in (Z, Y, X) order but metadata in (X, Y, Z) order.\n",
616+
" # Transpose to (X, Y, Z) to match metadata — this is a layout convention,\n",
617+
" # not an orientation flip.\n",
618+
" seg_array = np.asarray(seg_image.data).copy()\n",
619+
" seg_array = np.transpose(seg_array, (2, 1, 0))\n",
620+
"\n",
621+
" # Build affine from spatial metadata\n",
622+
" spacing = np.array(seg_image.spacing)\n",
623+
" origin = np.array(seg_image.origin)\n",
624+
" direction = np.array(seg_image.direction).reshape(3, 3)\n",
625+
"\n",
626+
" affine = self._build_affine(spacing, origin, direction)\n",
627+
"\n",
628+
" # Make contiguous (array may be non-contiguous after transpose)\n",
629+
" seg_array = np.ascontiguousarray(seg_array)\n",
630+
"\n",
631+
" # Create MONAI MetaTensor with metadata\n",
632+
" meta_tensor = MetaTensor(seg_array)\n",
633+
" meta_tensor.affine = affine\n",
634+
" meta_tensor.meta[\"filename_or_obj\"] = str(dcm_file)\n",
635+
" meta_tensor.meta[\"overlay_info\"] = overlay_info\n",
636+
" meta_tensor.meta[\"original_channel_dim\"] = \"no_channel\"\n",
637+
"\n",
638+
" d[key] = meta_tensor\n",
639+
" d[f\"{key}_meta_dict\"] = dict(meta_tensor.meta)\n",
640+
"\n",
641+
" return d\n",
642+
"\n",
643+
"\n",
644+
"# Load CT with MONAI's ITKReader\n",
645+
"ct_transforms = Compose(\n",
646+
" [\n",
647+
" LoadImaged(keys=[\"image\"], reader=ITKReader()),\n",
648+
" EnsureChannelFirstd(keys=[\"image\"]),\n",
649+
" ]\n",
650+
")\n",
651+
"\n",
652+
"# Load SEG with our custom LoadDicomSegd\n",
653+
"seg_transforms = Compose(\n",
654+
" [\n",
655+
" LoadDicomSegd(keys=[\"label\"]),\n",
656+
" EnsureChannelFirstd(keys=[\"label\"]),\n",
657+
" ]\n",
658+
")\n",
659+
"\n",
660+
"# Load both\n",
661+
"image_path = os.path.join(seg_dir, demo_pair[\"image_uid\"])\n",
662+
"seg_path = os.path.join(seg_dir, demo_pair[\"seg_uid\"])\n",
663+
"\n",
664+
"ct_data = ct_transforms({\"image\": image_path})\n",
665+
"seg_data = seg_transforms({\"label\": seg_path})\n",
666+
"\n",
667+
"ct_image = ct_data[\"image\"]\n",
668+
"seg_label = seg_data[\"label\"]\n",
669+
"\n",
670+
"print(f\"CT image shape: {ct_image.shape}\")\n",
671+
"print(f\"Segmentation shape: {seg_label.shape}\")\n",
672+
"print(f\"Unique labels: {torch.unique(seg_label)[:10].tolist()}...\") # First 10 labels"
673+
]
481674
},
482675
{
483676
"cell_type": "code",
@@ -831,7 +1024,7 @@
8311024
},
8321025
{
8331026
"cell_type": "code",
834-
"execution_count": 57,
1027+
"execution_count": null,
8351028
"metadata": {
8361029
"colab": {
8371030
"base_uri": "https://localhost:8080/"
@@ -849,8 +1042,10 @@
8491042
}
8501043
],
8511044
"source": [
852-
"# import shutil; shutil.rmtree(data_dir) # Uncomment to delete\n",
853-
"print(f\"Data at: {data_dir}\")"
1045+
"# Uncomment to delete\n",
1046+
"# import shutil; shutil.rmtree(data_dir); shutil.rmtree(seg_dir)\n",
1047+
"print(f\"Data at: {data_dir}\")\n",
1048+
"print(f\"Segmentations at: {seg_dir}\")"
8541049
]
8551050
},
8561051
{

0 commit comments

Comments
 (0)