Ultrasound imaging makes use of backscattering of waves during their interaction with scatterers present in biological tissues. Simulation of synthetic ultrasound images is a challenging problem on account of inability to accurately model various factors of which some include intra-/inter scanline interference, transducer to surface coupling, artifacts on transducer elements, inhomogeneous shadowing and nonlinear attenuation. Current approaches typically solve wave space equations making them computationally expensive and slow to operate. We propose a generative adversarial network (GAN) inspired approach for fast simulation of patho-realistic ultrasound images. We apply the framework to intravascular ultrasound (IVUS) simulation. A stage 0 simulation performed using pseudo B-mode ultrasound image simulator yields speckle mapping of a digitally defined phantom. The stage I GAN subsequently refines them to preserve tissue specific speckle intensities. The stage II GAN further refines them to generate high resolution images with patho-realistic speckle profiles. We evaluate patho-realism of simulated images with a visual Turing test indicating an equivocal confusion in discriminating simulated from real. We also quantify the shift in tissue specific intensity distributions of the real and simulated images to prove their similarity. Full paper: https://arxiv.org/abs/1712.07881

1. Can Generative Adversarial Networks
Model Imaging Physics?
Dr. Debdoot Sheet
Assistant Professor, Department of Electrical Engineering
Principal Investigator, Kharagpur Learning, Imaging and Visualization Group
Indian Institute of Technology Kharagpur
www.facweb.iitkgp.ernet.in/~debdoot/
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 2

2. Disclosure
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 1
• Conflicts and Interests
– SkinCurate Research Pvt. Ltd. – Founder, stock options
– Intel Technology India Pvt. Ltd. – ResearchSponsor
– Dept. of Biotechnology, Govt. of India – Research Sponsor
– Nesa Medtech Pvt. Ltd. – Research Sponsor
– Sigtuple Technologies Pvt. Ltd. – ResearchSponsor
– Samsung Inc. – Research Sponsor
– Nvidia Inc. – Lab. Resources Sponsor
– Texas Instruments India Pvt. Ltd. – Lab. Resources Sponsor
– Analog Devices India Pvt. Ltd. – Lab. Resources Sponsor
– Indian Council of Medical Research, Govt. of India – Travel Grants
– Dept. of Science and Technology, Govt. of India – Travel Grants
– GE John F. Welch Technology Center (GE-JFWTC) – Startup Incubation Grant
– Biotechnology Industry Research Assistance Council (BIRAC) – Startup Incubation Grant

3. Can Generative Adversarial Networks
Model Imaging Physics?
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 3
Some experiences with simulating patho-realistic ultrasound images
In association with Francis Tom Debarghya China DeepSIP DeepKLIV
“Simulating Patho-realistic Ultrasound Images using Deep Generative
Networks with Adversarial Learning”
Francis Tom, Debdoot Sheet
IEEE International Symposium on Biomedical Imaging (ISBI), 2018.

4. Overview of the Talk
• Ultrasound Imaging
– Physics and Instrumentation
– Signal Processing and Visualization
– Simulating Ultrasound Images
• Artificial Neural Networks (ANN)
– Convolutional Neural Network (CNN)
– Generative Modeling
– Adversarial Learning
• Simulation via Adversarial Transformation
• Take home message
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 4

5. Ultrasound Imaging
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 5

6. Physics of Ultrasound Imaging
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 6
Transducer
• M. Ali, D. Magee and U. Dasgupta, "Signal Processing Overview of Ultrasound Systems for Medical Imaging", Texas Instruments
White Paper, no. SPRAB12, Nov., 2008.
• M. A. Haidekker, "Ultrasound imaging", Medical Imaging Technology, pp. 97-110, 2013.

7. Sensing and Analog Signal Processing
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 7
Transducer
T/R Switch
HV Transmitter
Analog Front End
Tx Beamformer,
Front-end digital
controller,
Rx Beamformer
z ! " = $%&,( ), *
%+,, ), * = %-,(. ) /01 ), * /01 ), * = 234 5 ,
sin 9:
sin 9;
=
<:
<;
%&,, ), * = %+,, ), *
<; cos9: − <: cos 9;
<; cos9: + <: cos 9;
;
%&,( ), * = %&,, ), * /01 ), * A ) = B ) + CD )
B ) = B( + B: ) E
D ) = F( ) + DG )

8. Digital Signal Processing
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 8
Transducer
9
*
H *, 9 = ! " I *, 9 = J log H *, 9 + <

9. 17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 9
Visualization

10. Who is this Person?
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 10
Ian Donald was a Professor of Midwifery at Glasgow University, when he
first explored the clinical use of ultrasound in the 1950s after seeing it
being used in the Glasgow shipyards to look for flaws in metallurgy.

11. Simulating Ultrasound
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 11
Jørgen Arendt Jensen and Peter Munk, ”Computer phantoms
for simulating ultrasound B-mode and cfm images”, Acoustical
Imaging”, vol. 23, pp. 75-80, Eds.: S. Lees and L. A. Ferrari,
Plenum Press, 1997.
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A
computer simulation", Phys. Med. Biol., vol. 25, no. 3, pp. 463–
479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]

12. Pseudo B-mode US Image Simulation
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 12
0 M, N = O M, N P Q
P Q ~S 0,1 F( =
2W)(
<
ℎY M, N = sin F(M 2
3
YZ
;[
Z
ℎE M, N = 2
3
EZ
;[]
Z
^ M, N = 0 M, N ∗ ℎY M, N ∗ ℎE M, N
I M, N = `abI2H" ^ M,N
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med.
Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]

13. Where do we stand with Simulations?
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 13
Digital Phantom Pseudo B-mode US Real B-mode US
How to bridge this disparity in appearance?

14. The Challenge
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 14
0 M, N = O M, N P Q
P Q ~S 0,1
^ M, N = 0 M, N ∗ ℎY M, N ∗ ℎE M, N
I M, N = `abI2H" ^ M,N
J. C. Bambre and R. J. Dickinson, "Ultrasonic B-scanning: A computer simulation", Phys. Med.
Biol., vol. 25, no. 3, pp. 463–479, 1980. [http://dx.doi.org/10.1088/0031-9155/25/3/006]
Are&these&convolution)kernels)not&descriptive&
enough&to&model&the&signal&mixing&process&
resulting&in&image&formation?
Can&we&learn the&convolution)kernels?
F( =
2W)(
<
ℎY M, N = sin F(M 2
3
YZ
;[
Z
ℎE M, N = 2
3
EZ
;[]
Z

15. Convolution
Convolutional Neural Networks [Debdoot Sheet] 15
cd =
/ − e + 2fd
gd
+ 1
ch =
i − ℎ + 2fh
gh
+ 1
cd
ch
fd
fh
gd
gh

16. Convolution
Convolutional Neural Networks [Debdoot Sheet] 16
3×/×i
3×e×ℎ
3×e×ℎ
/
i
2× / − e + 1 × i − ℎ + 1

17. Who is this Person?
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 17
Yann LeCun is well known for his work
on optical character recognition and
computer vision using convolutional neural
networks (CNN), also famously referred to as
the LeNet, and is a founding father of
convolutional nets.
He is also one of the main creators of
the DjVu image compression technology
(together with Léon Bottou and Patrick
Haffner).

18. Learning Convolution with a Neural Network
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 18
* =
* =f
lm
n
o
p
lmq:
rs
lm
tq:
= lm
t
− u
vw l
vlm
w l = r − rs
vw l
vlm
=
vw l
vr
vr
vlmq:
vp
vo
vo
vlm
#&Training&
Samples?

19. Generative Models
Adversarial Autencoder [Debdoot Sheet] 19
Encoder Decoderz

20. Adversarial Learning
Adversarial Autencoder [Debdoot Sheet] 20
Encoder Decoderz
Discriminator Fake vs. Real

21. Adversarial Transformation of Photographs
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 21
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, & Russell Webb, “Learning from Simulated and
Unsupervised Images through Adversarial Training”, Proc. IEEE/CVF Conf. Comp. Vis. Patt. Recog. (CVPR), pp. 2107-2116, 2018.
(Best Paper @ CVPR2018)

22. Adversarial Transformation for US Simulation
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 22
Generator&Net
P 9
Discriminator&Net
x y
z{ 9
Simulated&vs.&Real
z| y
}z| y
}z{ 9
Digital&
Phantom
Simulated
Real

23. Adversarial Transformation for US Simulation
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 23
Hu, Y., Gibson, E., Lee, L. L., Xie, W., Barratt, D. C., Vercauteren, T., & Noble, J. A. (2017). Freehand Ultrasound Image Simulation
with Spatially-Conditioned Generative Adversarial Networks. In Molecular Imaging, Reconstruction and Analysis of Moving Body
Organs, and Stroke Imaging and Treatment (pp. 105-115). Springer, Cham.

24. The Challenges
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 24
Transducer
• Non&uniform&sampling&in&Cartesian&
coordinate&domain
• Signal&interpolated&for&image&
formation,&leading&to&smearing&of&
speckles

25. Our Solution
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 25
Digital&Phantom Digital&Phantom
cart2pol(&)
Pseudo&BLmode
Stage 0
Stage I
P+ 9
Stage II
P++ 9
z{~
9 z{~~
9
cart2pol(&)
Stage I
x+ y
Stage II
x++ y
z|~
y z|~~
y
Stage&I&Sim Stage&II&Sim
Real Real Simulated&vs.&Real
64&x&64 256&x&256256&x&256256&x&256256&x&256
}z{~
}z{~~
}z|~
}z|~~

26. Architecture of the Networks
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 26
Stage&I&G()
Stage&I&D()
Stage&II&G()
Stage&II&D()

27. 17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 27
Lets Quiz: Real vs. Simulated

28. Results
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 28
Intra-tissue similarity of speckle (JS div.) Inter-tissue dissimilarity of speckle (JS div.)

29. Take home message
• Full paper on US simulation:
https://arxiv.org/abs/1712.07881
• Adversarial autoencoders:
https://arxiv.org/abs/1511.05644
• Generative adversarial networks:
https://papers.nips.cc/paper/542
3-generative-adversarial-nets.pdf
17 May 2018 Can GANs Model Imaging Physics? [Debdoot Sheet] 29