add data resource to readme

README.md

@@ -5,19 +5,16 @@ tags:
 library_name: monai
 license: apache-2.0
 ---
-# Description
-A pre-trained model for volumetric (3D) detection of the lung lesion from CT image.
-
 # Model Overview
-This model is trained on LUNA16 dataset (https://luna16.grand-challenge.org/Home/), using the RetinaNet (Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002).
+A pre-trained model for volumetric (3D) detection of the lung nodule from CT image.
 
-LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules, and to assign a probability for being a nodule to each location.
+This model is trained on LUNA16 dataset (https://luna16.grand-challenge.org/Home/), using the RetinaNet (Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002).
 
-Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset!
+![model workflow](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_workflow.png)
 
 ## 1. Data
 ### 1.1 Data description
-The dataset we are experimenting in this example is LUNA16 (https://luna16.grand-challenge.org/Home/), which is based on [LIDC/IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5].
+The dataset we are experimenting in this example is LUNA16 (https://luna16.grand-challenge.org/Home/), which is based on [LIDC-IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5].
 
 LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules, and to assign a probability for being a nodule to each location.
 
@@ -36,31 +33,44 @@ In this model, we resampled them into 0.703125 x 0.703125 x 1.25 mm.
 
 Please following the instruction in Section 3.1 of https://github.com/Project-MONAI/tutorials/tree/main/detection to do the resampling.
 
+### 1.4 Data download
+The mhd/raw original data can be downloaded from [LUNA16](https://luna16.grand-challenge.org/Home/). The DICOM original data can be downloaded from [LIDC-IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5]. You will need to resample the original data to start training.
+
+Alternatively, we provide [resampled nifti images](https://drive.google.com/drive/folders/1JozrufA1VIZWJIc5A1EMV3J4CNCYovKK?usp=share_link) and a copy of [original mhd/raw images](https://drive.google.com/drive/folders/1-enN4eNEnKmjltevKg3W2V-Aj0nriQWE?usp=share_link) from [LUNA16](https://luna16.grand-challenge.org/Home/) for users to download.
+
 ## 2. Training configuration
-The training was performed with at least 12GB-memory GPUs.
+The training was the following:
+
+GPU: at least 16GB GPU memory
 
 Actual Model Input: 192 x 192 x 80
 
-## 3. Input and output formats
-Input: list of 1 channel 3D CT patches
+AMP: True
+
+Optimizer: Adam
+
+Learning Rate: 1e-2
+
+Loss: BCE loss and L1 loss
+
+### Input
+list of 1 channel 3D CT patches
+
+### Output
+In training mode: dictionary of classification and box regression loss in training mode;
 
-Output: dictionary of classification and box regression loss in training mode;
-list of dictionary of predicted box, classification label, and classification score in evaluation mode.
+In evaluation mode: list of dictionary of predicted box, classification label, and classification score in evaluation mode.
 
-## 4. Results and Scores
-The script to compute FROC sensitivity value on inference results can be found in https://github.com/Project-MONAI/tutorials/tree/main/detection
+## 3. Performance
+Coco metric is used for evaluating the performance of the model. The pre-trained model was trained and validated on data fold 0. This model achieves a mAP=0.853, mAR=0.994, AP(IoU=0.1)=0.862, AR(IoU=0.1)=1.0.
 
-This model achieves the following FROC sensitivity value on the validation data (our own split from the training dataset):
+![detection train loss](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_train_loss.png)
 
-| Methods             | 1/8   | 1/4   | 1/2   | 1     | 2     | 4     | 8     |
-| :---:               | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| [Liu et al. (2019)](https://arxiv.org/pdf/1906.03467.pdf)   | **0.848** | 0.876 | 0.905 | 0.933 | 0.943 | 0.957 | 0.970 |
-| [nnDetection (2021)](https://arxiv.org/pdf/2106.00817.pdf)  | 0.812 | **0.885** | 0.927 | 0.950 | 0.969 | 0.979 | 0.985 |
-| MONAI detection     | 0.835 | **0.885** | **0.931** | **0.957** | **0.974** | **0.983** | **0.988** |
+The validation accuracy in this curve is the mean of mAP, mAR, AP(IoU=0.1), and AR(IoU=0.1) in Coco metric.
 
-**Table 1**. The FROC sensitivity values at the predefined false positive per scan thresholds of the LUNA16 challenge.
+![detection val accuracy](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_val_acc.png)
 
-## 5. Commands example
+## 4. Commands example
 Execute training:
 ```
 python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf
@@ -71,11 +81,11 @@ Override the `train` config to execute evaluation with the trained model:
 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf
 ```
 
-Execute inference on resampled LUNA16 images (resampled following Section 3.1 of https://github.com/Project-MONAI/tutorials/tree/main/detection) by setting `"whether_raw_luna16": false` in `inference.json`:
+Execute inference on resampled LUNA16 images by setting `"whether_raw_luna16": false` in `inference.json`:
 ```
 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
 ```
-With the same command, we can execute inference on raw LUNA16 images by setting `"whether_raw_luna16": true` in `inference.json`. Remember to also set `"data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/original/dataset_fold0.json'"` and change `"data_file_base_dir"`.
+With the same command, we can execute inference on original LUNA16 images by setting `"whether_raw_luna16": true` in `inference.json`. Remember to also set `"data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/mhd_original/dataset_fold0.json'"` and change `"data_file_base_dir"`.
 
 Note that in inference.json, the transform "LoadImaged" in "preprocessing" and "AffineBoxToWorldCoordinated" in "postprocessing" has `"affine_lps_to_ras": true`.
 This depends on the input images. LUNA16 needs `"affine_lps_to_ras": true`.

configs/evaluate.json

@@ -5,6 +5,17 @@
         "data": "$@test_datalist",
         "transform": "@validate#preprocessing"
     },
+    "validate#key_metric": {
+        "val_coco": {
+            "_target_": "scripts.cocometric_ignite.IgniteCocoMetric",
+            "coco_metric_monai": "$monai.apps.detection.metrics.coco.COCOMetric(classes=['nodule'], iou_list=[0.1], max_detection=[100])",
+            "output_transform": "$monai.handlers.from_engine(['pred', 'label'])",
+            "box_key": "box",
+            "label_key": "label",
+            "pred_score_key": "label_scores",
+            "reduce_scalar": false
+        }
+    },
     "validate#handlers": [
         {
             "_target_": "CheckpointLoader",

configs/inference.json

@@ -9,7 +9,7 @@
     "ckpt_dir": "$@bundle_root + '/models'",
     "output_dir": "$@bundle_root + '/eval'",
     "data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/dataset_fold0.json'",
-    "data_file_base_dir": "/home/canz/Projects/datasets/LUNA16/93176/Images_resample",
+    "data_file_base_dir": "/datasets/LUNA16_Images_resample",
     "test_datalist": "$monai.data.load_decathlon_datalist(@data_list_file_path, is_segmentation=True, data_list_key='validation', base_dir=@data_file_base_dir)",
     "device": "$torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')",
     "amp": true,

configs/metadata.json

@@ -1,7 +1,8 @@
 {
     "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json",
-    "version": "0.4.3",
+    "version": "0.4.4",
     "changelog": {
+        "0.4.4": "add data resource to readme",
         "0.4.3": "update val patch size to avoid warning in monai 1.0.1",
         "0.4.2": "update to use monai 1.0.1",
         "0.4.1": "fix license Copyright error",
@@ -29,8 +30,10 @@
     "label_classes": "dict data, containing Nx6 box and Nx1 classification labels.",
     "pred_classes": "dict data, containing Nx6 box, Nx1 classification labels, Nx1 classification scores.",
     "eval_metrics": {
-        "val_coco": 0,
-        "froc": 0
+        "mAP_IoU_0.10_0.50_0.05_MaxDet_100": 0.853,
+        "AP_IoU_0.10_MaxDet_100": 0.862,
+        "mAR_IoU_0.10_0.50_0.05_MaxDet_100": 0.994,
+        "AR_IoU_0.10_MaxDet_100": 1.0
     },
     "intended_use": "This is an example, not to be used for diagnostic purposes",
     "references": [

configs/train.json

@@ -7,14 +7,14 @@
     "ckpt_dir": "$@bundle_root + '/models'",
     "output_dir": "$@bundle_root + '/eval'",
     "data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/dataset_fold0.json'",
-    "data_file_base_dir": "/home/canz/Projects/datasets/LUNA16/93176/Images_resample",
+    "data_file_base_dir": "/datasets/LUNA16_Images_resample",
     "train_datalist": "$monai.data.load_decathlon_datalist(@data_list_file_path, is_segmentation=True, data_list_key='training', base_dir=@data_file_base_dir)",
     "device": "$torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')",
     "epochs": 300,
     "val_interval": 10,
     "learning_rate": 0.01,
     "amp": true,
-    "batch_size": 3,
+    "batch_size": 4,
     "patch_size": [
         192,
         192,

docs/README.md

@@ -1,16 +1,13 @@
-# Description
-A pre-trained model for volumetric (3D) detection of the lung lesion from CT image.
-
 # Model Overview
-This model is trained on LUNA16 dataset (https://luna16.grand-challenge.org/Home/), using the RetinaNet (Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002).
+A pre-trained model for volumetric (3D) detection of the lung nodule from CT image.
 
-LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules, and to assign a probability for being a nodule to each location.
+This model is trained on LUNA16 dataset (https://luna16.grand-challenge.org/Home/), using the RetinaNet (Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002).
 
-Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset!
+![model workflow](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_workflow.png)
 
 ## 1. Data
 ### 1.1 Data description
-The dataset we are experimenting in this example is LUNA16 (https://luna16.grand-challenge.org/Home/), which is based on [LIDC/IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5].
+The dataset we are experimenting in this example is LUNA16 (https://luna16.grand-challenge.org/Home/), which is based on [LIDC-IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5].
 
 LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules, and to assign a probability for being a nodule to each location.
 
@@ -29,31 +26,44 @@ In this model, we resampled them into 0.703125 x 0.703125 x 1.25 mm.
 
 Please following the instruction in Section 3.1 of https://github.com/Project-MONAI/tutorials/tree/main/detection to do the resampling.
 
+### 1.4 Data download
+The mhd/raw original data can be downloaded from [LUNA16](https://luna16.grand-challenge.org/Home/). The DICOM original data can be downloaded from [LIDC-IDRI database](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI) [3,4,5]. You will need to resample the original data to start training.
+
+Alternatively, we provide [resampled nifti images](https://drive.google.com/drive/folders/1JozrufA1VIZWJIc5A1EMV3J4CNCYovKK?usp=share_link) and a copy of [original mhd/raw images](https://drive.google.com/drive/folders/1-enN4eNEnKmjltevKg3W2V-Aj0nriQWE?usp=share_link) from [LUNA16](https://luna16.grand-challenge.org/Home/) for users to download.
+
 ## 2. Training configuration
-The training was performed with at least 12GB-memory GPUs.
+The training was the following:
+
+GPU: at least 16GB GPU memory
 
 Actual Model Input: 192 x 192 x 80
 
-## 3. Input and output formats
-Input: list of 1 channel 3D CT patches
+AMP: True
+
+Optimizer: Adam
+
+Learning Rate: 1e-2
+
+Loss: BCE loss and L1 loss
+
+### Input
+list of 1 channel 3D CT patches
+
+### Output
+In training mode: dictionary of classification and box regression loss in training mode;
 
-Output: dictionary of classification and box regression loss in training mode;
-list of dictionary of predicted box, classification label, and classification score in evaluation mode.
+In evaluation mode: list of dictionary of predicted box, classification label, and classification score in evaluation mode.
 
-## 4. Results and Scores
-The script to compute FROC sensitivity value on inference results can be found in https://github.com/Project-MONAI/tutorials/tree/main/detection
+## 3. Performance
+Coco metric is used for evaluating the performance of the model. The pre-trained model was trained and validated on data fold 0. This model achieves a mAP=0.853, mAR=0.994, AP(IoU=0.1)=0.862, AR(IoU=0.1)=1.0.
 
-This model achieves the following FROC sensitivity value on the validation data (our own split from the training dataset):
+![detection train loss](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_train_loss.png)
 
-| Methods             | 1/8   | 1/4   | 1/2   | 1     | 2     | 4     | 8     |
-| :---:               | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| [Liu et al. (2019)](https://arxiv.org/pdf/1906.03467.pdf)   | **0.848** | 0.876 | 0.905 | 0.933 | 0.943 | 0.957 | 0.970 |
-| [nnDetection (2021)](https://arxiv.org/pdf/2106.00817.pdf)  | 0.812 | **0.885** | 0.927 | 0.950 | 0.969 | 0.979 | 0.985 |
-| MONAI detection     | 0.835 | **0.885** | **0.931** | **0.957** | **0.974** | **0.983** | **0.988** |
+The validation accuracy in this curve is the mean of mAP, mAR, AP(IoU=0.1), and AR(IoU=0.1) in Coco metric.
 
-**Table 1**. The FROC sensitivity values at the predefined false positive per scan thresholds of the LUNA16 challenge.
+![detection val accuracy](https://developer.download.nvidia.com/assets/Clara/Images/monai_retinanet_detection_val_acc.png)
 
-## 5. Commands example
+## 4. Commands example
 Execute training:
 ```
 python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf
@@ -64,11 +74,11 @@ Override the `train` config to execute evaluation with the trained model:
 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf
 ```
 
-Execute inference on resampled LUNA16 images (resampled following Section 3.1 of https://github.com/Project-MONAI/tutorials/tree/main/detection) by setting `"whether_raw_luna16": false` in `inference.json`:
+Execute inference on resampled LUNA16 images by setting `"whether_raw_luna16": false` in `inference.json`:
 ```
 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf
 ```
-With the same command, we can execute inference on raw LUNA16 images by setting `"whether_raw_luna16": true` in `inference.json`. Remember to also set `"data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/original/dataset_fold0.json'"` and change `"data_file_base_dir"`.
+With the same command, we can execute inference on original LUNA16 images by setting `"whether_raw_luna16": true` in `inference.json`. Remember to also set `"data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/mhd_original/dataset_fold0.json'"` and change `"data_file_base_dir"`.
 
 Note that in inference.json, the transform "LoadImaged" in "preprocessing" and "AffineBoxToWorldCoordinated" in "postprocessing" has `"affine_lps_to_ras": true`.
 This depends on the input images. LUNA16 needs `"affine_lps_to_ras": true`.