Ph.D. Defense Presentation Slides (Changhee Han) カリスの東大博論審査会（公聴会）発表スライド Pathology-Aware Generative Adversarial Networks for Medical Image Augmentation（医用画像拡張に向けた、病変部を意識した敵対的生成ネットワーク）

Pathology-Aware Generative Adversarial Networks
for Medical Image Augmentation
Ph.D. Defense
by
Changhee Han1,2,3
1
Nakayama Lab, Creative Informatics, UTokyo
2
Research Center for Medical Big Data, NII
3
Department of Radiology, NCGM Hospital January 21, 2020

Diagnosis and Diagnostic Errors*
1/91
“Life is short, and the Art long; the occasion ﬂeeting; experience
fallacious, and judgment diﬃcult. The physician must not only be
prepared to do what is right himself, but also to make the patient,
the attendants, and externals cooperate.”
Hippocrates [460-375 BC]

Goals of This Presentation
2/91
1. Ph.D. Defense
2. GCL Social Innovation Project Defense
2.1 Social Problem Solving/Value Creation Based on ICT
2.2 Ripple Eﬀect
2.3 Social Leadership

Changes from Pre-Defense*
3/91
1. Changed title to Pathology-Aware Generative Adversarial
Networks for Medical Image Augmentation
2. Introduced new chapter describing background on GAN-based
Medical Data Augmentation (DA)/Physician Training: Chapter 3
Investigated Contexts and Application
3. Integrated contents of Article 2 into Article 3: Chapter 5
GAN-based Medical Image Augmentation for 2D Classification
4. Added GAN-based DA results on different training set sizes:
Chapter 5
5. Introduced new chapter including feedback from 9 physicians
(3 project related physicians/6 project non-related radiologists)
and findings from GCL workshop C: Chapter 8 Discussions on
Developing Clinically Relevant AI-Powered Diagnosis Systems

Ph.D. Thesis Contents (Chapters)
4/91
1. Introduction
2. Background
3. Investigated Contexts and Applications
4. GAN-based Medical Image Generation
Article 1: ISBI ʼ18
5. GAN-based Medical Image Augmentation for 2D Classiﬁcation
Article 2: Springer Book Chapter/Article 3: IEEE Access
6. GAN-based Medical Image Augmentation for 2D Detection
Article 4: CIKM ʼ19
7. GAN-based Medical Image Augmentation for 3D Detection
Article 5: 3DV ʼ19
8. Discussions on Developing Clinically Relevant AI-Powered
Diagnosis Systems
9. Conclusion
Appendix: Scientiﬁc Production

Research Questions in GAN-based Medical Image Augmentation*
5/91

How to generate realistic/diverse medical images?
Collecting/annotating medical images are laborious
∴ Information conversion is essential!
Consideration: Network architecture, loss function,
training scheme
How to help AIʼs diagnosis with novel images?
Good for Computers (Data Augmentation, DA)
Interpolation/extrapolation for high diversity (former
matters as best model when given data is uncertain)
How to help physicianʼs diagnosis with novel images?
Good for Humans (Physician Training)
Interpolation/extrapolation for high realism

Contributions*
6/91
Proposed clinically-valuable novel applications,
DA/Physician Training using noise-to-image GANs,
focusing on ability to generate realistic/diverse images
For required extrapolation, in collaboration with
physicians, proposed novel 2D/3D GANs controlling
pathology (i.e., tumors/nodules) on major modalities
(i.e., brain MRI, 90% of MRI/lung CT, 85% of CT)
After detailed discussions with many physicians,
conﬁrmed clinical relevance of our pathology-aware
GANs as i) clinical decision support system and
ii) non-expert physician training tool

Background
7/91
1. Medical Image Analysis
1.1 Image Diagnosis
2. Deep Learning
2.1 Convolutional Neural Networks (CNNs)
2.2 Deep Generative Networks
2.2.1 Generative Adversarial Networks
(GANs: I. Goodfellow+, Jun ʼ14)
3. Methods to Address Data Paucity
3.1 Overcoming Diﬃculties in Optimization
3.1.1 DA
3.2 Overcoming Lack of Generalization

Medical Image Analysis
8/91
It computationally analyzes medical images to support
clinical decision making
In Image Diagnosis, sensitivity (i.e., recall) matters
more than speciﬁcity to avoid overlooking diagnosis

Generative Adversarial Networks (GANs)
9/91
https://vinodsblog.com/2018/10/15/everything-you-need-to-know-about-convolutional-neural-networks
Two-CNN combination (Generator for image generation
vs Discriminator for Synthetic image discrimination)
Very good at realistic image/video generation
Applications: Denoising/DA
(MRI → CT/Benign image → Malignant image)

GANs (Noise-to-Image GANs/Image-to-Image GANs)
10/91
Initial values either from random noise samples or certain image
Example noise-to-image GAN, BigGAN: A. Brock+, Sep ʼ18
Example image-to-image GAN, CycleGAN: J. Zhu+, Mar ʼ17

Biggest Challenge in Medical Imaging
11/91
Because of lack of reliable annotated datasets, Medical Imaging
research results tend to be dataset/task-dependent
(Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique:
H. Greenspan+, IEEE Trans. Med. imaging, May ʼ16)
1. It is costly and laborious to collect medical images,
such as MR and CT images, especially for rare disease
Available patients are usually ~10-200
2. It is time-consuming and observer-dependent, even for
expert physicians, to annotate them due to unbalanced
pathological-to-healthy ratio

Methods to Address Data Paucity
12/91
1. Overcoming Lack of Generalization
1.1 DA
1.1.1 Classic DA techniques
1.1.2 Recent DA techniques
1.1.3 GAN-based DA techniques
1.2 Semi-supervised Learning
(using pseudo labels for unlabled data)
1.3 Regularization (e.g., Dropout, Ridge, Lasso, Elastic Net)
1.4 Ensembling (e.g., Bagging, Boosting)
1.5 Data Fusion (e.g., T1w MRI + T2w MRI, MRI + CT)
2. Overcoming Diﬃculties in Optimization

Classic DA Techniques
13/91
https://blog.algorithmia.com/introduction-to-dataset-augmentation-and-expansion
Human perception is invariant to size, shape, brightness, color
We recognize objects while their such features change, so
changing such features are OK to obtain more data
1. x/y/z-axis ﬂipping and rotating
2. Zooming and scaling
3. Cropping
4. Translating (moving along x/y/z-axis)
5. Elastic deformation
6. Adding Gaussian noise (distortion of high frequency features)
7. Brightness/Contrast ﬂuctuation

Recent DA Techniques
14/91
CutMix: S. Yun+, May ʼ19
1. Mixup (H. Zhang+, Oct ʼ17)
BC Learning (Y. Tokozume+, Nov ʼ17)
Mix 2 images for regularization
2. Cutout (T. DeVries and G. W. Taylor, Aug ʼ17)
Randomly mask out square regions for regularization
3. CutMix
Mixup + Cutout
4. AutoAugment (E. D. Cubuk+, May ʼ18)
Automatically search for improved DA policies

GAN-based DA Techniques
15/91
Real
Image Distribution
GAN-Generated
Image Distribution
Distribution after
Data Augmentation
Similar to Mixup among all images within same class
∵ Many-to-many mappings ﬁll uncovered Real image
distribution by generating realistic/diverse images
Especially good for medical image augmentation since
those modalities display human bodyʼs strong
anatomical consistency with inter-subject variability
(not like dog/cat/airplane on same picture...)

GAN-based Medical Image Generation
16/91
GAN-based Synthetic Brain MR Image Generation
(Article 1, ISBI ʼ18)
Collaboration with Physician, Dr. Furukawa at Kanto
Rosai Hospital
Collaboration with Medical Imaging Researcher,
Dr. Rundo at University of Cambridge
Collaboration with Computer Vision Researchers,
Dr. Hayashi at Kyushu University and Araki
at Chubu University

17/91
Original Brain
MR Images
Synthetic Images for
Data Augmentation
T1 T1c
T2 FLAIR
Physician Training
(GAN)
Generate
Generate
(Conditional
GAN)
Realistic Tumors
with Desired Size/Location
by Adding Conditioning
Realistic Tumors
in Random Locations
Previously, noise-to-image GAN-based realistic/diverse
medical image generation was challenging due to lack
of available medical images

18/91
Original Brain
MR Images
Data Augmentation
T1 T1c
T2 FLAIR
Physician Training
(GAN)
Generate
Generate
(Conditional
GAN)
Realistic Tumors
with Desired Size/Location
by Adding Conditioning
Realistic Tumors
in Random Locations
1. Proposed clinically-valuable novel applications,
DA/Physician Training using noise-to-image GANs
2. To conﬁrm GANʼs such potentials, generated 128 × 128
multi-sequence brain MR images using Deep
Convolutional GAN (DCGAN: A. Radford+, May ʼ16) and
Wasserstein GAN (WGAN: M. Arjovsky+, Aug ʼ17)
3. Even expert physician failed to distinguish
Real/Synthetic images in Visual Turing Test

DatasetDetails
19/91
T1 (Real, 128 × 128/64 × 64)
T2 (Real, 128 × 128/64 × 64)
T1c (Real, 128 × 128/64 × 64)
FLAIR (Real, 128 × 128/64 × 64)
240 × 155 T1, T1c, T2, FLAIRw brain sagittal MRI of 220
high-grade glioma cases from BRATS 2016
Selected slices from #80 to #149 among whole 240
slices to omit initial/ﬁnal slices
Training set (220 patients/15, 400 slices per sequence)
Slices from same patients look similar
Tumor/non-tumor images are trained together
Not yet applied to DA
Images are resized to 128 × 128 from 240 × 155

Example Brain MR Images
20/91
T1 (Real, 128 × 128/64 × 64)
T2 (Real, 128 × 128/64 × 64)
T1c (Real, 128 × 128/64 × 64)
FLAIR (Real, 128 × 128/64 × 64)
Original MR images
T1 (DCGAN, 128 × 128)
T2 (DCGAN, 128 × 128)
T1c (DCGAN, 128 × 128)
FLAIR (DCGAN, 128 × 128)
Concat (DCGAN, 128 × 128) Concat (DCGAN, 64 × 64)
DCGAN-generated MR images
T1 (WGAN, 128 × 128)
T2 (WGAN, 128 × 128)
T1c (WGAN, 128 × 128)
FLAIR (WGAN, 128 × 128)
Concat (WGAN, 128 × 128) Concat (WGAN, 64 × 64)
WGAN-generated MR images

Example Brain MR Images
21/91
T1 (WGAN, 128 × 128)
T2 (WGAN, 128 × 128)
T1c (WGAN, 128 × 128)
FLAIR (WGAN, 128 × 128)
Concat (WGAN, 128 × 128) Concat (WGAN, 64 × 64)
WGAN-generated MR images

Visual Turing Test Results by Physician
22/91
To evaluate synthesized imagesʼ realism, expert
physician classiﬁes randomly-selected 50 Real vs 50
Synthetic brain MR images in random order
Accuracy (%) Real as Real (%) Real as Synt (%) Synt as Real (%) Synt as Synt (%)
T1 (DCGAN, 128 × 128) 70 52 48 12 88
T1c (DCGAN, 128 × 128) 71 48 52 6 94
T2 (DCGAN, 128 × 128) 64 44 56 16 84
FLAIR (DCGAN, 128 × 128) 54 24 76 16 84
Concat (DCGAN, 128 × 128) 77 68 32 14 86
Concat (DCGAN, 64 × 64) 54 26 74 18 82
T1 (WGAN, 128 × 128) 64 40 60 12 88
T1c (WGAN, 128 × 128) 55 26 74 16 84
T2 (WGAN, 128 × 128) 58 38 62 22 78
FLAIR (WGAN, 128 × 128) 62 32 68 8 92
Concat (WGAN, 128 × 128) 66 62 38 30 70
Concat (WGAN, 64 × 64) 53 36 64 30 70
Even expert found classifying Real/Synthetic images
hard, especially in lower resolution due to less details

Related Research (CT Lung Nodule Generation)
23/91
How to fool radiologists: M. J. M. Chuquicusma+, ISBI 2018, Apr ʼ18
Presented at same Medical Image Analysis conference
as our paper
Using DCGAN, just generated 56 × 56 Region of
Interest (RoI) images, not whole ones

Related Research (Radiology Education)*
24/91
Towards generative adversarial networks as a new paradigm for radiology education:
S. G. Finlayson+, NIPS 2018 Workshop, Dec ʼ18
Due to infrastructural/legal constraints, it is diﬃcult to
obtain pathological images with desired features for
training medical students/radiology trainees
Harvard Medical School is now designing trial to test
eﬀectiveness of GAN-generated images against Real
images in human learning

GAN-based Medical Image Augmentation for 2D Classification
25/91
1. PGGAN-based Brain MR Image Augmentation for
Tumor Classification
(Article 2, Springer Book Chapter)
2. Combining Noise-to-Image and Image-to-Image GANs
for Tumor Classification (Article 3, IEEE Access)
Collaboration with Physician, Dr. Furukawa at Kanto
Rosai Hospital
Collaboration with Computer Vision Researchers,
Dr. Hayashi at Kyushu University and Araki
at Chubu University

Combining Noise-to-Image and Image-to-Image GANs for Tumor
Classiﬁcation
26/91
OriginalBrain
MRImages
T2 FLAIRTumor Detection
(Binary Classification)
(PGGANs)
Noise-to-Image
Generation
Transformation
(Classic DA)
Geometrically-transformed
Original Images
Novel Realistic Images
with/without Tumors
Refined Images
SyntheticBrain
MRImages
RefinedBrain
MRImages
SyntheticBrain
MRImages
(ResNet-50)
Train
(UNIT/SimGAN)
Image-to-Image
Translation
1. Realistic high-resolution medical image generation was
challenging despite common CNN input size ~256 × 256
2. Generated 256 × 256 brain MR images using
Progressive Growing of GANs (PGGANs: T. Karras+, Oct
ʼ17) with WGAN-Gradient Penalty (GP: I. Gulrajani+, Apr ʼ17)
3. Further reﬁned texture/shape of PGGAN-generated
images similarly to Real ones using Multimodal
UNsupervised Image-to-image Translation
(MUNIT: X. Huang+, Apr ʼ18)

Combining Noise-to-Image and Image-to-Image GANs for Tumor
Classiﬁcation
27/91
OriginalBrain
MRImages
T2 FLAIRTumor Detection
(Binary Classification)
(PGGANs)
Noise-to-Image
Generation
Transformation
(Classic DA)
Geometrically-transformed
Original Images
Novel Realistic Images
with/without Tumors
Refined Images
SyntheticBrain
MRImages
RefinedBrain
MRImages
SyntheticBrain
MRImages
(ResNet-50)
Train
(UNIT/SimGAN)
Image-to-Image
Translation
1. DA: This two-step GAN-based DA signiﬁcantly
outperformed classic DA alone, boosting sensitivity
93.7% to 97.5%
2. Firstly analyzed how medical GAN-based DA is
associated with ImageNet pre-training/discarding
weird-looking Synthetic images for humans
3. Physician Training: PGGANs could generate realistic
256 × 256 brain MR images

DatasetDetails
28/91
240 × 240 T1c brain axial MRI of 220 high-grade glioma
cases from BRATS 2016 (slices with/without tumors)
Selected slices from #30 to #130 among whole 155
slices to omit initial/ﬁnal slices
Training set
(154 patients/4, 679 tumor/3, 750 non-tumor images)
Validation set
(44 patients/750 tumor/608 non-tumor images)
Test set
(22 patients/1, 232 tumor/1, 013 non-tumor images)
During GAN training, only use training set to be fair
(tumor/non-tumor images are trained seperately)
Training set images are zero-padded to reach power of
2: 256 × 256 from 240 × 240

PGGANswithWGAN-GP
29/91
Real MR
Images
GeneratorDiscriminator
Latent Space
4×4
4×4
Latent Space
4×4
4×4
Latent Space
4×4
Training progresses
Real MR
Images
Real MR
Images
4×4
8×8
8×8
256×256
256×256
Training method for GANs with progressively growing
generator/discriminator: starting from low resolution,
newly added layers model ﬁne-grained details as
training progresses

MUNIT
30/91
Non-tumor
PGGAN-generated
MUNIT-refined
Tumor
MUNIT is image-to-image GAN based on both
auto-encoding/translation

ResNet-50 Binary Classification Results with (without pre-training)
31/91
Only use good-looking synthetic images for humans
except (5) → Meaningless when pre-trained
DenseNet-169 and k-means++ cluster images into 200
groups and weird image clusters are manually discarded
DA Setups Accuracy(%) Sensitivity(%) Specificity(%)
(1) 8,429 real images 93.1 (86.3) 90.9 (88.9) 95.9 (83.2)
(2) + 200k classic DA 95.0 (92.2) 93.7 (89.9) 96.6 (95.0)
(3) + 400k classic DA 94.8 (93.2) 91.9 (90.9) 98.4 (96.1)
(4) + 200k PGGAN-based DA 93.9 (86.2) 92.6 (87.3) 95.6 (84.9)
(5) + 200k PGGAN-based DA w/o clustering/discarding 94.8 (80.7) 91.9 (80.2) 98.4 (81.2)
(6) + 200k classic DA & 200k PGGAN-based DA 96.2 (95.6) 94.0 (94.2) 98.8 (97.3)
(7) + 200k MUNIT-refined DA 94.3 (83.7) 93.0 (87.8) 95.8 (78.5)
(8) + 200k classic DA & 200k MUNIT-refined DA 96.7 (96.3) 95.4 (97.5) 98.2 (95.0)
Each GAN-based DA alone helps classification, when
pre-trained, due to robustness from GAN-based images
But, without pre-training, it harms classification due to
biased initialization from GAN-overwhelming data

32/91
(1) 8,429 real images 93.1 (86.3) 90.9 (88.9) 95.9 (83.2)
(2) + 200k classic DA 95.0 (92.2) 93.7 (89.9) 96.6 (95.0)
(3) + 400k classic DA 94.8 (93.2) 91.9 (90.9) 98.4 (96.1)
(4) + 200k PGGAN-based DA 93.9 (86.2) 92.6 (87.3) 95.6 (84.9)
(7) + 200k MUNIT-reﬁned DA 94.3 (83.7) 93.0 (87.8) 95.9 (78.6)
Each GAN-based DA + classic DA clearly outperforms
GAN-based DA or classic DA alone by sensitivity
The former learns non-linear manifold of Real images
to generate novel local tumor features strongly
associated with sensitivity
The latter learns geometrically-transformed manifold of
Real images to cover global features and provide
robustness on training for most cases

33/91
(1) 8,429 real images 93.1 (86.3) 90.9 (88.9) 95.9 (83.2)
(2) + 200k classic DA 95.0 (92.2) 93.7 (89.9) 96.6 (95.0)
(3) + 400k classic DA 94.8 (93.2) 91.9 (90.9) 98.4 (96.1)
(4) + 200k PGGAN-based DA 93.9 (86.2) 92.6 (87.3) 95.6 (84.9)
(7) + 200k MUNIT-refined DA 94.3 (83.7) 93.0 (87.8) 95.9 (78.6)
MUNIT refinement significantly improves sensitivity
Without pre-training, sensitivity is higher than
specificity for MUNIT-refined DA + classic DA because
refinement can fill Real tumor image distribution

Changing Training Data Sizes from 154 Patients*
34/91
Binary classiﬁcation results with pre-training
Real vs Real + Classic vs Real + Classic + PGGANs
In general, PGGAN-based DA moderately increases
both accuracy/sensitivity on top of classic DA
Sensitivity is considerably high with only 20%/50% Real
training images
＊MUNIT-reﬁned DA shows accuracy 96.4% and
sensitivity 97.5% under 100% data without pre-training

Visual Turing Test Results by Physician
35/91
Expert physician classifies 50 Real (R) vs 50 Synthetic
(S) MR images
PGGAN 79.5 73 27 14 86
MUNIT 77.0 58 42 4 96
Expert classified a few PGGAN-generated images as
Real, despite high resolution (i.e., 224 × 224)
But, expert classified less MUNIT-refined images as
Real due to slight artifacts induced during refinement

T-SNE Visualization
36/91
It captures linear/non-linear relations, preserving local distances
Images represented into 2D space while ﬁxing positions of Original
images as ﬁlled blue/red dots
Non-tumor Original
Non-tumor Classic DA
Tumor Original
Tumor Classic DA
Non-tumor Original
Non-tumor UNIT DA
Tumor Original
Tumor UNIT DA
Non-tumor Original
Non-tumor PGGAN DA
Tumor Original
Tumor PGGAN DA
(Left) Original vs Classic DA and Tumor vs Non-tumor
Original Tumor/Non-tumor image distributions largely
overlap while Non-tumor images distribute wider
Classic DA-based Tumor/Non-tumor image
distributions also often overlap, and both images
distribute wider than Original ones

T-SNE Visualization
37/91
Non-tumor Original
Tumor Original
Tumor Classic DA
Non-tumor Original
Non-tumor UNIT DA
Tumor Original
Tumor UNIT DA
Non-tumor Original
Non-tumor PGGAN DA
Tumor Original
Tumor PGGAN DA
(Center) Original vs PGGAN-based DA and Tumor vs Non-tumor
PGGAN-generated images distribute widely, while their
Tumor/Non-tumor images overlap much less than
Classic DA ones

T-SNE Visualization
38/91
Non-tumor Original
Tumor Original
Tumor Classic DA
Non-tumor Original
Non-tumor UNIT DA
Tumor Original
Tumor UNIT DA
Non-tumor Original
Non-tumor PGGAN DA
Tumor Original
Tumor PGGAN DA
(Right) Original vs UNIT-based DA and Tumor vs Non-tumor
MUNIT-refined images show even better discrimination
and more similar distribution to Original ones than
PGGAN-generated images
Thus, two-step GAN-based DA can effectively fill
distribution uncovered by Original or Classical DA
images with higher sensitivity

GAN-based Medical Image Augmentation for 2D Detection
39/91
Conditional PGGAN-based Brain MR Image
Augmentation for Tumor Detection Using Highly-Rough
Annotation (Article 4, CIKM ʼ19)
Collaboration with Physicians, Dr. Noguchi, Dr. Kawata,
and Dr. Uchiyama at NCGM Hospital
Collaboration with Medical Imaging Researchers,
Dr. Rundo at University of Cambridge and Prof. Murao
at NII

Conditional PGGAN-based Brain MR Image Augmentation for Tumor Detection
Using Highly-Rough Annotation
40/91
OriginalBrain
MRImages
FLAIRTumor Detection
(Bounding Boxes)
(Conditional
PGGAN)
Generate
Novel Images w/ Tumors
at Desired Positions/Sizes
SyntheticBrain
MRImages (YOLOv3)
Train
1. Achieving accurate diagnosis using minimium
annotated images is essential both for AI/physicians
2. Proposed Conditional PGGANs (CPGGANs), bounding
box-based pathology-aware GAN that can generate
256 × 256 realistic/diverse brain MR images with
novel tumors at desired positions/sizes

Conditional PGGAN-based Brain MR Image Augmentation for Tumor Detection
Using Highly-Rough Annotation
41/91
OriginalBrain
MRImages
FLAIRTumor Detection
(Bounding Boxes)
(Conditional
PGGAN)
Generate
Novel Images w/ Tumors
at Desired Positions/Sizes
SyntheticBrain
MRImages (YOLOv3)
Train
1. DA: Under limited data and highly-rough bounding box
annotation, this CPGGAN-based DA boosted sensitivity
83% to 91% with IoU threshold 0.25
2. Physician Training: Additional normal brain images for
CPGGAN training could generate realisitc brain images
with tumors at desired positions/sizes

DatasetDetails
42/91
256 × 256 Real tumor
256 × 256 Highly-Rough Annotation
32 × 32 Real Tumor Bbox
256 × 256 Real Non-tumor
256 × 256 T1c axial MRI of 180 metastases cases: we
(NCGM Hospital) collected it to match clinical request
(slices only with tumors)
Training set (2, 813 images/5, 963 bounding boxes)
Validation set (337 images/616 bounding boxes)
Test set (947 images/3, 094 bounding boxes)
Additional training set (16, 962 images from
193 normal subjects for comparision)
During GAN training, only use training set(s) to be fair

CPGGAN Architecture
43/91
Real MR
Images
GeneratorDiscriminator
Latent Space
4×4
4×4 4×4
Latent Space
4×4
Training progresses
Real MR
Images
Real MR
Images
4×4
8×8
8×8
256×256
256×256
Real MR
Images
4×4
16×16
16×16
Latent Space
4×44×4
Latent Space
Tumor-conditioning Generator/Discriminator
Conditioning image: Prepare 256 × 256 black image (i.e., pixel
value: 0) with white bounding boxes (i.e., pixel value: 255)
describing tumor positions/sizes for attention
Generator input: Resize conditioning image to previous
generatorʼs output resolution/channel size and concatenate them
(noise samples → ﬁrst 4 × 4 images)
Discriminator input: Concatenate conditioning image with Real or
Synthetic image

10 Tumor Detection Experimental Setups
44/91
We do not discard weird-looking images
unless tumor appearance is unclear within bounding boxes
Number of training images
2, 813 Real images
+4, 000 GAN-generated images
GAN setups for DA
CPGGANs
CPGGANs trained with additional normal brain images
(conditioning image is fully black for normal brain)
Image-to-Image GAN
(i.e., WGAN-GP with U-Net-like generator)

Example Brain Tumors
45/91
256 × 256 CPGGAN-generated Tumor w/o Normal
32 × 32 CPGGAN-generated Tumor Bbox w/o Normal
256 × 256 CPGGAN-generated Tumor w/ Normal
32 × 32 CPGGAN-generated Tumor Bbox w/ Normal
256 × 256 Image-to-Image GAN-generated Tumor w/o Normal
32 × 32 Image-to-Image GAN-generated Tumor Bbox w/o Normal

Example Brain Tumors
46/91
256 × 256 CPGGAN-generated Tumor w/o Normal
256 × 256 CPGGAN-generated Tumor w/ Normal
Tumor conditioning: Positions/sizes of training images
with random combination of horizontal/vertical flipping,
width/height shift up to 10%, and zooming up to 10%
→ Extrapolation effect via model reduction (Enforcing constraints
for interpolation and extrapolation in Generative Adversarial Networks: P. Stinis+, J. Comput. Phys., Nov ʼ19)
Theoretically infinite conditioning instances are external to
training data
We can use common/desired medical priors as conditions

YOLOv3-based Tumor Detection
47/91
YOLOv3 for real-time tumor alert
Network resolution: 416 × 416 during training and
608 × 608 during validation/testing
DA: Geometric/intensity transformations to both
Real/Synthetic images
Test: Pick model with best sensitivity on validation with
detection threshold 0.1%/IoU threshold 0.5 between
96, 000-240, 000 steps to avoid severe False Positives
(FPs) while achieving high sensitivity

Tumor Detection Results
48/91
IoU ≥ 0.5 IoU ≥ 0.25
Sensitivity (%) FPs per slice Sensitivity (%) FPs per slice
2,813 real images 67 4.11 83 3.59
+ 4,000 CPGGAN-based DA 77 7.64 91 7.18
+ 8,000 CPGGAN-based DA 71 6.36 87 5.85
+ 12,000 CPGGAN-based DA 76 11.77 91 11.29
+ 4,000 CPGGAN-based DA (+ normal) 69 7.16 86 6.60
+ 4,000 Image-to-Image GAN-based DA 72 6.21 87 5.70
Adding only 4, 000 CPGGAN-generated images
achieves best sensitivity improvement due to
Real/Synthetic training image balance
∵ GAN-based DA works because both Real/Synthetic
image distribution are biased
We cannot increase FPs to achieve such high
sensitivity without CPGGAN-based DA

Tumor Detection Results
49/91
IoU ≥ 0.5 IoU ≥ 0.25
Sensitivity (%) FPs per slice Sensitivity (%) FPs per slice
2,813 real images 67 4.11 83 3.59
+ 4,000 CPGGAN-based DA 77 7.64 91 7.18
+ 8,000 CPGGAN-based DA 71 6.36 87 5.85
+ 12,000 CPGGAN-based DA 76 11.77 91 11.29
GAN training with normal brain images helps increase
realism but not sensitivity because it decreases
attention on tumors (i.e., lower tumor diversity)

Example Tumor Detection Results
50/91
Ground Truth w/o GAN 4k GAN 8k GAN 12k GAN 4k GAN+Normal 8k GAN+Normal 12k GAN+Normal
It can alleviate risk of overlooking tumor diagnosis with
acceptable FPs
Highly-overlapping bounding boxes only require
physicianʼs single check

Visual Turing Test Results by 3 Radiologists
51/91
3 radiologists classify 50 Real vs 50 Synthetic resized
32 × 32 tumor bounding boxes
Test 1: Trained without additional normal brain images
Test 2: Trained with additional normal brain images
Test1
Physician 1 88 80 20 4 96
Physician 2 95 90 10 0 100
Physician 3 97 98 2 4 96
Test2
Physician 1 81 78 22 16 84
Physician 2 83 86 14 20 80
Physician 3 91 90 10 8 92
Tumors looked more realistic when trained with normal
tumorless brain images

Visual Turing Test Results by 3 Radiologists
52/91
3 radiologists classify 50 Real vs 50 Synthetic 256 × 256
whole brain MR images
Test 3: Trained without additional normal brain images
Test 4: Trained with additional normal brain images
Test3
Physician 1 97 94 6 0 100
Physician 2 96 92 8 0 100
Physician 3 100 100 0 0 100
Test4
Physician 1 91 82 18 0 100
Physician 2 96 96 4 4 96
Physician 3 100 100 0 0 100
Radiologists clearly classiﬁed whole brain images
except 2 Synthetic images inﬂuenced by normal brains

T-SNE Visualization
53/91
32 × 32 Resized Tumor
Bounding Box MR Images 256 × 256 MR Images
Synthetic tumors have similar distribution to Real ones,
but they also ﬁll Real image distribution uncovered by
Original dataset
Tumor (i.e., RoI) realism/diversity matter more than
whole image realism/diversity since YOLOv3 look at
image patches

Related Research (Gastric Cancer Detection)*
54/91
Normal Conditioning Pathological
Gastric cancer detection from endoscopic images using synthesis by GAN: T. Kanayama+, MICCAI 2019, Oct ʼ19
Bounding box-based image-to-image translation from
normal image to pathological one

GAN-based Medical Image Augmentation for 3D Detection
55/91
3D Multi-Conditional GAN-based Lung CT Image
Augmentation for Nodule Detection (Article 5, 3DV ʼ19)
Collaboration with Physicians, Dr. Furukawa at Jikei
University Hospital and Dr. Umemoto at Juntendo
University Hospital
Collaboration with Medical Imaging Engineers,
Dr. Kitamura, Kudo, and Ichinose at FujiFilm

3D Multi-Conditional GAN-based Lung CT Image Augmentation for
Nodule Detection
56/91
OriginalLungNodule
TrainingImages
FLAIRLung Nodule Detection
(Bounding Boxes)
(Multi-Conditional
GAN)
Generate
Novel Synthetic 32×32×32
Nodules on CT at Desired
Position/Size/Attenuation
AdditionalLungNodule
TrainingImages
(3D Faster RCNN)
Train
1. Since human body is 3D, 3D pathology-aware GAN is
essential to generate desired pathological images, but
direct 3D generation is computationally demanding
2. Proposed 3D Multi-Conditional GAN (MCGAN) that can
generate 32 × 32 × 32 realistic/diverse nodules, placed
naturally on lung CT, at desired position, size, and
attenuation

3D Multi-Conditional GAN-based Lung CT Image Augmentation for
Nodule Detection
57/91
OriginalLungNodule
TrainingImages
FLAIRLung Nodule Detection
(Bounding Boxes)
(Multi-Conditional
GAN)
Generate
Novel Synthetic 32×32×32
Nodules on CT at Desired
Position/Size/Attenuation
AdditionalLungNodule
TrainingImages
(3D Faster RCNN)
Train
1. DA: Under limited data and bounding box annotation,
this 3D MCGAN-based DA boosted sensitivity under
any nodule size/attenuation at ﬁxed FP rates
2. Firstly analyzed how medical GAN-based DA is
associated with GAN training without ℓ1 loss
3. Physician Training: GAN training with ℓ1 loss could
generate realistic nodule images, placed naturally on
lung CT, at desired position/size/attenuation

DatasetDetails
58/91
Lung CT (Real nodule w/ surroundings, 64 × 64 × 64)
Lung CT (Noise box-replaced nodule w/ surroundings, 64 × 64 × 64)
Lung CT with nodules of 745 subjects from LIDC Dataset: we
(FujiFlim) annotated attenuation to match clinical request (both
benign/malignant nodules, sometimes including thoracic walls)
Training set (632 scans/3, 727 nodules)
Validation set (37 scans/143 nodules)
Test set (76 scans/265 nodules)
During GAN training, only use training set to be fair
All nodules are resized to 32 × 32 × 32 with
64 × 64 × 64 surrounding tissues for MCGAN training
Resulting nodules are resampled to their original
resolution and mapped back onto original CT scans

3D MCGAN Architecture
59/91
Inputvolume
(w/surroundings)
(w/surroundings)
Syntheticnodule
Realnodule
64×64×64×1
64×64×64×1
32×32×32×1
64×64×64×1
32×32×32×1
64×64×64×6 for Generator
32×32×32×6 for Discriminator
Inputconditions
・Size
(Small/Medium/Large)
・Attenuation
(Solid/Part-solid/GGN)
LSGAN loss
32×32×32×64
16×16×16×128
32×32×32×128
32×32×32×64
16×16×16×1
64×64×64×2
8×8×8×256
4×4×4×512
32×32×32×64
16×16×16×128
8×8×8×512
16×16×16×256
3D Nodule Generator
Context Discriminator Nodule Discriminator
Layers
Conv 4×4×4 stride 2 + BN + LeakyReLU
Deconv 4×4×4 stride 2 + BN + ReLU
Deconv 4×4×4 stride 1 + BN + ReLU
Conv 1×1×1 stride 1 + Tanh
Conv 1×1×1 stride 1 + Sigmoid
Concat
Dropout 0.5
Fake Pair?
Real Pair?
WGAN-GP loss
4×4×4×1
32×32×32×7
16×16×16×64
8×8×8×128
(Realism/Size/Attenuation)
Fake Nodule?
Real Nodule?
(Realism/Size/Attenuation)
4×4×4×256
Nodule-conditioning Generator/Discriminators
Nodule Generator: Input noise box-replaced 32 × 32 × 32 nodule
with 64 × 64 × 64 surroundings and size/attenuation conditions
Context Discriminator: Learn to classify Real vs Synthetic
nodule/surrounding pairs
Nodule Discriminator: Learn to classify Real vs Synthetic
nodules in terms of realism/size/attenuation

7 Nodule Detection Experimental Setups
60/91
We do not discard weird-looking images
Number of training images
632 Real images
+1x GAN-generated images
GAN setups for DA
3D MCGAN with ℓ1 loss
G∗
= arg minG maxD1,D2 LLSGAN(G, D1)+LWGAN-GP(G, D2)
+100Lℓ1 (G)
3D MCGAN without ℓ1 loss
G∗
= arg minG maxD1,D2 LLSGAN(G, D1)+LWGAN-GP(G, D2)

Example Lung Nodules with Surrounding Tissues
61/91
Lung CT (Real nodule w/ surroundings, 64 × 64 × 64)
Lung CT (Noise box-replaced nodule w/ surroundings, 64 × 64 × 64)
Lung CT (Synthetic nodule w/ surroundings, 64 × 64 × 64)
Lung CT (L1 loss-added synthetic nodule w/ surroundings, 64 × 64 × 64)
Nodule conditioning: Position/size/attenuation of
training images with random combination of
width/height/depth shift up to 10% and zooming up to
10% → Extrapolation eﬀect
We can use common/desired medical priors as conditions
For Physician Training, post-processing like Poisson
image editting can blend slight shading diﬀerence

3D Faster RCNN-based Nodule Detection
62/91
3D Faster RCNN (S. Ren et al. ʼ15) for accurate detection
Network resolution: 160 × 176 × 224
DA: Geometric/intensity transformations to both
Real/Synthetic images
Test: Pick model with best sensitivity on validation with
detection threshold 50%/IoU threshold 0.25 between
30, 000-40, 000 steps to avoid severe FPs while
achieving high sensitivity

Nodule Detection Results (CPM)
63/91
Competition Performance Metric (CPM) is average
sensitivity at 1/8, 1/4, 1/2, 1, 2, 4, and 8 FPs per scan
This quantiﬁes ability to identify signiﬁcant percentage
of nodules with both very few/moderate FPs
CPM by Size (%) CPM by Attenuation (%)
CPM (%) Small Medium Large Solid Part-solid GGN
632 real images 51.8 44.7 61.8 62.4 65.5 46.4 24.2
+ 1× 3D MCGAN-based DA 55.0 45.2 68.3 66.2 69.9 52.1 24.4
+ 2× 3D MCGAN-based DA 52.7 44.7 67.4 42.9 65.5 40.7 28.9
+ 3× 3D MCGAN-based DA 51.2 41.1 64.4 66.2 61.6 57.9 27.7
+ 1× 3D MCGAN-based DA w/ ℓ1 50.8 43.0 63.3 55.6 62.6 47.1 27.1
Adding only 1× 3D MCGAN-generated images achieves
best CPM improvement due to Real/Synthetic training
image balance

Nodule Detection Results (CPM)
64/91
CPM by Size (%) CPM by Attenuation (%)
CPM (%) Small Medium Large Solid Part-solid GGN
632 real images 51.8 44.7 61.8 62.4 65.5 46.4 24.2
+ 1× 3D MCGAN-based DA 55.0 45.2 68.3 66.2 69.9 52.1 24.4
+ 2× 3D MCGAN-based DA 52.7 44.7 67.4 42.9 65.5 40.7 28.9
+ 3× 3D MCGAN-based DA 51.2 41.1 64.4 66.2 61.6 57.9 27.7
GAN training with ℓ1 loss helps increase realism but
rather decreases CPM (i.e., lower nodule diversity)

Example Nodule Detection Results
65/91
Ground Truth w/o GAN + 1× GAN + 2× GAN + 3× GAN + 1× GAN w/ L1 + 2× GAN w/ L1 + 3× GAN w/ L1
(a) (b) (c) (d) (e) (f) (g) (h)
It can alleviate risk of overlooking nodule diagnosis
with acceptable FPs
Highly-overlapping bounding boxes only require
physicianʼs single check

Visual Turing Test results by Two Physicians
66/91
2 physicians classify 50 Real vs 50 Synthetic
32 × 32 × 32 nodules
Test 1: Trained without ℓ1 loss
Test 2: Trained with ℓ1 loss
Test1
Physician 1 43 38 62 52 48
Physician 2 43 26 74 40 60
Test2
Physician 1 57 44 56 30 70
Physician 2 53 22 78 16 84
Physicians completely failed to classify Real/Synthetic
nodules

Visual Turing Test results by Two Physicians
67/91
2 physicians classify 50 Real vs 50 Synthetic
64 × 64 × 64 nodules with surrounding tissues
Test 3: Trained without ℓ1 loss
Test 4: Trained with ℓ1 loss
Test3
Physician 1 62 50 50 26 74
Physician 2 79 64 36 6 94
Test4
Physician 1 58 42 58 26 74
Physician 2 66 72 28 40 60
Physicians relatively recognized Synthetic nodules with
surroundings, due to slight shading difference between
nodules/surrounding
Synthetic images influenced by ℓ1 loss were more
difficult to classify

T-SNE plot with 500 lung nodule images
68/91
Synthetic tumors have similar distribution to Real ones,
but concentrated in left inner areas with less Real ones,
which implies eﬀective DA performance

69/91
Discussions on Developing Clinically
Relevant AI-Powered Diagnosis Systems
(Questionnaire Survey & Workshop)*

Questionaire to Physicians*
70/91
Subjects: 3 physicians committed to (at least one of)
our projects and 6 project non-related radiologists
without much AI background
Experiment: Answering questionnaire after reading 10
summary slides about general Medical Image Analysis
and our GAN projects with example synthesized images
We conducted both qualitative (i.e., comments) and
quantitative (i.e., Likert scale) evaluation
Five-point Likert scale
1. Very negative
2. Negative
3. Neutral
4. Positive
5. Very positive

Question 1*
71/91
Question: Are you keen to exploit medical AI in general
when it achieves accurate and reliable performance in
near future (scale from 1 to 5)?
Please tell us your expectations, wishes, and worries
(free comments)
Likert scale: 5 5 5 (average: 5)/5 5 3 4 5 5 (average: 4.5)
Comments
I am looking forward to its practical applications,
especially at low/zero price
As radiologists, we need AI-based diagnosis during
image interpretation ASAP
Since physicianʼs annotation is always subjective, we
cannot claim AI-based diagnosis is really correct even if
AI diagnoses similarly. Because I do not believe other
physiciansʼ diagnosis, but my own eyes, I would use AI
just to identify abnormal candidates

Question 2*
72/91
Question: What do you think about using
GAN-generated images for DA (scale from 1 to 5)?
(free comments)
Likert scale: 5 5 4 (average: 4.7)/4 5 4 4 4 4 (average: 4.2)
Comments
It would be effective, especially as rare disease training
data
If Deep Learning could be more effective, we should
introduce it
It would be effective to train AI on data-limited disease,
but which means AI is inferior to humans

Question 3*
73/91
Question: What do you think about using
GAN-generated images for Physician Training
(scale from 1 to 5)?
(free comments)
Likert scale: 3 4 3 (average: 3.3)/3 5 2 3 2 3 (average: 3)
Comments
It depends on how to construct system
Physicians should actively introduce/learn new
technology; in this sense, GAN technology should be
actively used for physician training in rare disease
It could be useful for medical student training, which
aims for 85% accuracy by covering typical cases. But
expert physician training aims for over 85% accuracy by
comparing typical/atypical cases and acquiring new
understanding̶Real atypical images are essential

Question 4*
74/91
Question: Any comments or suggestions about our
projects towards developing clinically applicable
system based on your daily diagnosis experience?
Comments
GANs can generate typical images, but not atypical
images; this would be next challenge
For now, please show small abnormal findings, such as
nodules and ground glass opacities̶it would halve
radiologistsʼ efforts. Then, we could develop accurate
diagnosis step by step
Showing abnormal findings with shapes/sizes/types
would increase diagnosis accuracy. But I also want to
know how role of diagnosticians changes after all

GCL Workshop C (Supervised by GCL Instructors)
75/91
Medical Imagingが切り開く医療の未来
――医⽤画像ならではの挑戦、そして意義,
AI Dojo Tech Talk, December 2017.
Seminar with over 40 attendees
GCL Workshop C: Medical Imagingが切り開く
医療の未来――臨床現場にフィットする
AIを創るには？, March 2019.
Workshop towards developing medical AI ﬁtting into
clinical environment in 5 years (2 Medical Imaging
experts, 2 physicains, 3 informatics/healthcare
generalists)
We realized that Why and How (Human/Capital) are as
important as What (Technology)
We released its in-depth summary on YouTube and note
Workshop attendees invited me to Research Center for
Medical Big Data, NCGM Hospital, and FujiFilm

Gap Between AI and Healthcare Sides*
76/91
AI, especially Deep Learning, does not provide clear decision
criteria, does it block clinical applications like diagnosis?
Healthcare side: AI provides minimum explanation like
heatmap, which makes persuading patients diﬃcult
So, physiciansʼ intervention is essential
Instead of replacing physicians, we expect reliable
second opinion to prevent misdiagnosis
AI side: Compared with other systems/physicians,
Deep Learningʼs explanation is not particularly poor, so
we require too severe standards for AI
Word ”AI” is promoting anxiety/perfection
If we could thoroughly verify diagnosis reliability against
physicians, such excessive explanation is optional

Developing Medical AI Fitting into Clinical Environment in 5 years*
77/91
We discussed Why (Clinical meaning/interpretation) and
How (Data acquisition, Commercial deployment, safety/feeling
safe)
Data acquisition problem: Ethical screening in Japan
is exceptionally strict, so acquiring and sharing
large-scale medical data/annotation are challenging
Moreover, whenever diagnostic criteria/equipments
and targets change, further reviews/software
modiﬁcations are requried
Solution: We could exploit complete medical checkup,
which is unique in East Asia
Additionally, we should compile database of electronic
charts and exploit AI techniques like GAN-based
DA/domain adaptation

78/91
We Submitted These Discussions to
International Journal of
Computer Assisted Radiology and Surgery!*

Conclusion*
79/91
To tackle difficulties of collecting annotated
pathological images, proposed to use noise-to-image
GANs for Medical DA/Physician Training and found
effective loss functions/training schemes for each of
them (diversity vs realism)
For these applications, proposed new 2D/3D bounding
box-based pathology-aware GANs and confirmed
clinical relevance from many physicians: i) Clinical
decision support system on MRI/CT (earning rate:
3%/16%, interpretation time: 1 min/2-20 mins);
ii) Non-expert physician training tool

Conclusion (Philosophy)*
80/91
Future of Medical Image Analysis → Information Conversion!
Interpolation: GAN-based medical image
augmentation is reliable because those modalites can
display human bodyʼs strong anatomical consistency at
ﬁxed body with inter-subject variability
Extrapolation: Pathology-aware GAN is promising
because common/desired medical priors can play key
role in conditioning
Bounding box condition can signiﬁcantly
relieve annotation burden

Future Work Towards Further GAN-based Extrapolation*
81/91
COCO-GAN: Generation by parts via conditional coordinating: C. H. Lin+, ICCV 2019, Oct ʼ19
3 next steps towards GAN-based extrapolation for better
DA/Physician Training, on top of pathology-aware conditioning:
1. Generation by parts with coordinate conditions
Due to biological restrictions, humans learn to reason spatial
relationships across parts of surroundings
Similarly, COCO-GAN produced SOTA quality full images
Since human anatomy has strong local consistency, reasoning
such spatial relationships can perform eﬀective extrapolation

Future Work Towards Further GAN-based Extrapolation*
82/91
Correlation via synthesis: end-to-end nodule image generation and
radiogenomic map learning based on generative adversarial network: Z. Xu+, arXiv, July ʼ19
2. Generation with image + gene expression conditions
Such fusing can non-invasively identify molecular properties
of particular type of disease
Similarly, Xu et al. produced realistic synthetic nodule images
Such condition fusing can largely boost extrapolation
3. Transfer learning across diﬀerent body parts/diseases
e.g., abdomen to chest/(brain) tumors to traumatic injuries
CNN Performance: Medical image volumes ≫ Natural videos
(Med3D: Transfer Learning for 3D Medical Image Analysis: S. Chen+, arXiv, Apr ʼ19)
Such transfer learning from large-scale medical datasets for
GANs can perfrorm eﬀective extrapolation

Selected Publications
83/91
K. Murao, Y. Ninomiya, C. Han, et al., Cloud platform for deep learning-based CAD via collaboration between
Japanese medical societies and institutes of informatics, In SPIE Medical Imaging (oral presentation),
Houston, The United States, February 2020.
C. Han et al., Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases
Detection Using Highly-Rough Annotation on MR Images, In ACM International Conference on Information and
Knowledge Management (CIKM, acceptance rate: 19%), Beijing, China, November 2019.
C. Han et al., Combining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor
Detection, IEEE Access (impact factor: 4.098), October 2019.
C. Han et al., Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT
Image Augmentation for Object Detection, In International Conference on 3D Vision (3DV), Québec City,
Canada, September 2019.
C. Han et al., GAN-based Multiple Adjacent Brain MRI Slice Reconstruction for Unsupervised Alzheimerʼs
Disease Diagnosis, In Computational Intelligence methods for Bioinformatics and Biostatistics (CIBB),
Bergamo, Italy, September 2019.
C. Han et al., Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection, In Neural
Approaches to Dynamics of Signal Exchanges, Springer, September 2019.
L. Rundo, C. Han, et al., CNN-based Prostate Zonal Segmentation on T2-weighted MR Images: A
Cross-dataset Study, In Neural Approaches to Dynamics of Signal Exchanges, Springer, September 2019.
L. Rundo*, C. Han*, et al., USE-Net: incorporating Squeeze-and-Excitation blocks into U-Net for prostate zonal
segmentation of multi-institutional MRI datasets, Neurocomputing (impact factor: 4.072), July 2019 (* denotes
Co-First Authors).
C. Han et al., Learning More with Less: GAN-based Medical Image Augmentation, Medical Imaging
Technology, Japanese Society of Medical Imaging Technology, June 2019.
C. Han et al., GAN-based Synthetic Brain MR Image Generation, In IEEE International Symposium on
Biomedical Imaging (ISBI), Washington, D.C., The United States, April 2018.
C. Han, Application of Learning Classifier Systems to Gene Expression Analysis in Synthetic Biology, In S.
Patnaik, X. Yang, and K. Nakamatsu (eds.) Nature Inspired Computing and Optimization: Theory and
Applications, Springer, March 2017 (Masterʼs Course).
C. Han et al., Optimization of Artificial Operon Construction by Consultation Algorithms Utilizing LCS, In IEEE
Congress on Evolutionary Computation (CEC, oral presentation), Vancouver, Canada, July 2016 (Masterʼs C).

Record of Awards
84/91
International Conference on 3D Vision (3DV) Student Travel
Grant (grant: $CAD 500), September 2019.
The University of Tokyo AI Solutions Global Competition 2019
Winner (prize: right to attend SU Global Summit 2020 including
ﬂight tickets and accommodation), August 2019.
画像の認識・理解シンポジウム (MIRU) 学⽣奨励賞 (Student
Research Encouragement Award), August 2019.
電⼦情報通信学会医⽤画像研究会 (MI Symposium) MI研究奨励賞
(Best Research Award), May 2019.
The University of Tokyo Graduate Program for Social ICT Global
Creative Leaders (GCL) Mini-Presentation Competition Winner
(prize: 0.1M Yen), December 2018.
画像の認識・理解シンポジウム (MIRU) 最優秀研究計画賞 (Best
Research Plan Award, Team Leader, prize: GPUs worth 1M Yen),
August 2017.
Vision and Sports Summer School (VS3) Best Poster Award,
August 2016.

Research Center for Medical Big Data, NII (RA)
85/91
Collaboration with Medical Imaging Researchers

Related Publications
Related Presentations
C. Han et al., Learning More with Less: Conditional PGGAN-based MRI Augmentation with Highly-Rough
Annotation for Brain Metastases Detection, 画像の認識・理解シンポジウム（MIRU）, Osaka, Japan, July 2019.
C. Han et al., Towards Annotating Less Medical Images: PGGAN-based MR Image Augmentation for Brain
Tumor Detection, 医⽤画像研究会 (MI Symposium), Okinawa, Japan, January 2019.

NCGM Hospital (Visiting Researcher)
86/91
Collaboration with Radiologists

Related Presentation
C. Han et al., Learning More with Less: Conditional PGGAN-based MRI Augmentation with Highly-Rough
Annotation for Brain Metastases Detection, 画像の認識・理解シンポジウム（MIRU）, Osaka, Japan, July 2019.

FujiFilm (Intern)
87/91
Collaboration with Medical Imaging Engineers
Related Publication
C. Han et al., Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT
Image Augmentation for Object Detection, In International Conference on 3D Vision (3DV), Québec City,
Canada, September 2019.

Visiting Periods Abroad
88/91
Collaboration with world-leading Medical Imaging Researchers
Almost all my publications are with foreign researchers,
especially Dr. Leonardo Rundo (University of Cambridge) who I
also invited to Nakayama lab during 2018.02-03
2019.08-09 Visiting Scholar @University of
Cambridge, Cambridge, The UK (Supervisor: Prof. Evis
Sala)
2018.07-08 Visiting Scholar @Università degli Studi di
Milano-Bicocca, Milan, Italy (Supervisor: Prof.
Giancarlo Mauri)
2017.09-10 Visiting Scholar @Università degli Studi di
Milano-Bicocca, Milan, Italy (Supervisor: Prof.
Giancarlo Mauri)
2016.04-09 Exchange Student @Technische
Universität München, Munich, Germany (Supervisor:
Felix Achilles)

Inter-College Collaboration in Japan
89/91
Collaboration with Kyushu/Chubu University Computer Vision
Researchers after winning research plan presentation competition
C. Han et al., Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection, In Neural
Approaches to Dynamics of Signal Exchanges, Springer, September 2019.
C. Han et al., GAN-based Synthetic Brain MR Image Generation, In IEEE International Symposium on
Biomedical Imaging (ISBI), Washington, D.C., The United States, April 2018.
Related Presentations
C. Han et al., Towards Annotating Less Medical Images: PGGAN-based MR Image Augmentation for Brain
Tumor Detection, 医⽤画像研究会 (MI Symposium), Okinawa, Japan, January 2019.
C. Han et al., Infinite Brain Tumor Images: Can GAN-based Data Augmentation Improve Tumor Detection on
MR Images?, 画像の認識・理解シンポジウム（MIRU）, Hokkaido, Japan, August 2018.
C. Han et al., Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection, In
International Computer Vision Summer School (ICVSS), Sicily, Italy, July 2018.
C. Han et al., Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection, In The Italian
Workshop on Neural Networks (WIRN), Vietri sul Mare, Italy, June 2018.
荒⽊諒介, 韓昌熙, et al., MIRU2017若⼿プログラム報告：GANによる合成脳MR画像⽣成, 情報処理学会
コンピュータビジョンとイメージメディア研究会 (CVIM), Osaka, Japan, May 2018.
C. Han et al., GAN-based Synthetic Medical Image Generation, 画像の認識・理解シンポジウム (MIRU)
若⼿プログラム, Hiroshima, Japan, August 2017.

Outreach Activities
90/91
Invited talks
Medical Imagingの研究動向と論⽂採択のコツ,
放射線治療⼈⼯知能研究会, Tokyo, Japan, February 2020.
MIRU2017若⼿プログラム報告：GANによる
合成脳MR画像⽣成, 情報処理学会コンピュータビジョンと
イメージメディア研究会（CVIM）, Osaka, Japan, May 2018.
Article writing
MRI/CTを読影したAIが⼈命を救う!?
̶医療×AI = Medical Imaging̶, August 2019.
Professional service
Reviewer, INFORMS Journal on Computing, 2019.
Track Chair, Session: Machine Learning and Computational
Intelligence in multi-omics and medical image analysis,
Computational Intelligence methods for Bioinformatics and
Biostatistics (CIBB), Bergamo, Italy, September 2019.

I Will Actually Develop Clinically Relevant AI after Graduation*
91/91
So, Please Graduate Me!

Ph.D. Defense Presentation Slides (Changhee Han) カリスの東大博論審査会（公聴会）発表スライド Pathology-Aware Generative Adversarial Networks for Medical Image Augmentation（医用画像拡張に向けた、病変部を意識した敵対的生成ネットワーク）

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Mehr von カリス東大AI博士

Mehr von カリス東大AI博士 (7)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)