Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning

Petteri Teikari, PhD
http://petteri-teikari.com/
Version “Mon 30 October 2017“
Portable Retinal
Imaging and
Medical Diagnostics
With a focus on hardware-centric
deep learning, and end-to-end
deep learning pipelines for
diagnosis including imaging
optimization
No26 Lynn Skordal
and SabineRemy Lenses

About
Presentation
Who is this for: Tech-savvy neuroscientist and clinicians,
bioimaging researchers, deep learning researchers. Some
ofthe slidesmayseemtoobviousforimagingpeople.
Presentation type: ‘Managerial’ approach providing
various research leads hopefully making easier to
collaborate cross-disciplinary multiteam research / startup
projects
Take-home message: Focus on end-to-end (systems
design, systems engineering, you name it) design of
medical diagnostics from data acquisition to data analysis
on a single deep learning pipeline. Don’t just gather dumb
and possibly suboptimal data, and expect deep learning to
dothe magic.

Deep
Learning
By now we can assume
that you as readers
are familiar with
the concept
Ornarrowartificialintelligence or
simply machinelearning,asthe
approachisnotexactly“intelligent”
yetinhumanintelligence-sense
Especially inmedicaldomains,itis
bettertothinkintermsofdata
ratherthan‘deeplearningmagic’.
Developadatastrategy.
HowtoSpota MachineLearning
Opportunity, EvenIfYou Aren’ta
DataScientist
Kathryn Hume| OCTOBER20, 2017
https://hbr.org/2017/10/how-to-spot-a-machine-learning-o
pportunity-even-if-you-arent-a-data-scientist
STRATEGY MAY–JUNE 2017 ISSUE
What’s YourData Strategy?
LeandroDalleMule, ThomasH. Davenport
https://hbr.org/2017/05/whats-your-data-strategy
STRATEGY MAY–JUNE 2017 ISSUE
TheFourCringe-WorthyMistakesToo
ManyStartups MakewithData
AmandaRichardson
http://firstround.com/review/the-four-cringe-worthy-mis
takes-too-many-startups-make-with-data/
If you think that your organization’s analysis
needs to come in the form of a dedicated
employee with a “data scientist” title,
Richardson challenges you to think broader.
“To me, data science is a collection of skills, not
a job. … Everybody on your early team needs
to be strategic. Everybody should be able to
do analysis.”
— now everyone does computations — it’s
become part of a generalized skill set. That’s
how the world evolves. And I think we’re there
with data science. More people should be
able to take responsibility for and have the
capability to work with data to make
decisions.”
http://dx.doi.org/10.1038/nature14539
DeepLearning
by Ian Goodfellow, YoshuaBengio, AaronCourville
https://www.goodreads.com/book/show/24072897-deep-learning

Case
Examples
Ophthalmology
and Optometry

Retinopathy
Screening
https://youtu.be/wa9OdaRMgO8
EyeSpy:SocialEyesUsesDeepLearningtoSpot SeriousEyeProblems
“Eye care in developing countries occurs in overflowing general hospitals and
clinics, as well as “outreach camps,” where hundreds to thousands of patients are
seen over a few days. MARVIN, running on NVIDIA GPU SHIELD Android tablets,
can bring “best practice” care to “the edge,” where medical resources of any kind—
letalonespecialtyservicessuchasophthalmology—arealwaysinshortsupply.“
https://blogs.nvidia.com/blog/2016/02/17/deep-learning-4/

Visual
Impairment
OxSight Heliossmart glassesfor thevisionimpaired
“A key aspect to a successful visual prosthetic is allowing users to
quickly identify and locate relevant objects that they encounter — to
create a visual map. This requires vast amounts of computation to
optimizethelargenumberofparametersfoundinagivensetting.“
https://blogs.nvidia.com/blog/2017/04/30/gpu-powered-glasses/

The GPU
Deep
Learning
Ecosystem
Asymmetric
computing needs
Training deep learningmodelsrequire
alotof computationpower, butwhen
deployedto cloudorto ‘edge devices’
such asmobile phones,drones,
medicaldevices,etc., the computation
requirementscanberelaxed
TheNVIDIA JetsonTX2 (Pascal)Tech Report
PostedbyDr.AdrianWong Date: March08, 2017
https://www.techarp.com/articles/nvidia-jetson-tx2-pascal-tech-report/

Training vs.
Inference
Unoptimized
inference roughly 10x
faster than training
[With high-end Volta DGX-1 GPU Server, ~$149k]
https://www.slideshare.net/daikumatan/2016-0630deeplearningarchi
https://devblogs.nvidia.com/parallelforall/inside-volta/

Training
Typically “local” GPU
clusters are used for
large-scale training as
cloud GPU instances can
get very expensive very
fast
GPU Technology Conference(GTC) ’17Highlights
https://medium.com/@vishy_punditry/gpu-technology-conference-gtc-17-highlights-6dee628961e1
Facebooktoopen-sourceAIhardwaredesign
https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/

Inference
Easier to deploy the
model to cloud when
releasing the product
from R&D stage
AWSDeepLearning AMI https://aws.amazon.com/amazon-ai/amis/
https://www.ibm.com/cloud-computing/bluemix/gpu-computing https://azuremarketplace.microsoft.com/en-us/marketplace/apps
/microsoft-ads.dsvm-deep-learning
https://cloud.google.com/gpu/

Edge
Computing
Model
http://uk.businessinsider.com/internet-of-things-cloud-computing-2016-10?r=US&IR=T

Edge with
GPUs
TheNVIDIA JetsonTX2 (Pascal)Tech Report
PostedbyDr.AdrianWong Date: March08, 2017
https://www.techarp.com/articles/nvidia-jetson-tx2-pascal-tech-report/

Retinal
Imaging
with GPUs
GPUs have been
accelerating old
school signal
processing and
visualization already
for some time
GPU Based OCT Visualization by Carl
Glittenberg with the use of OCTANE, OSIRIX, Cinema
4D, and an NVIDIA GPU (GTX 580) to render Carl Zeiss
Meditec Cirrus HD OCT data files in realtime CUDA.
https://youtu.be/c-Iv--yxtKU
MOptimMOcean3000 OCT
with NVIDIAQuadro 600

Desktop
OCTs are
essentially
desktop
computers
Thus easily
accelerated by typical
desktop PCIe GPUs
NVIDIAHome>Products>HighPerformanceComputing >IndustryApplications>MedicalImaging
http://www.nvidia.com/object/medical_imaging.html

Cloud,
Desktop, or
Local GPU
Server
Computing
Options

Precision
Recap
Floatingpoint(real)
16bit (half-precision)
32bit (single)
64bit(double)
128bit (decimal)
Fixed point (integer)
LuisR. Izquierdo and J.GaryPolhill (2006)
IsYour Model Susceptibleto Floating-Point Errors?
http://jasss.soc.surrey.ac.uk/9/4/4.htmlhttps://www.slideshare.net/tomoaki0705/cvim-half-precision-floating-point
http://www.byclb.com/TR/Tutorials/dsp_appl_spc/application_specific_processors.htm
Presented byMichael J. Bonato, atHighPerformanceEmbedded Computing Conference
2002 MIT LincolnLaboratory, http://slideplayer.com/slide/7237441/

GPUs in
‘Traditional’
Medical
Image
Analysis
Medicalimage processingonthe GPU
–Past,present andfuture
Anders Eklund, Paul Dufort, Daniel Forsberg, StephenM. LaConte
Medical Image Analysis Volume17, Issue8, December2013, Pages 1073-1094
https://doi.org/10.1016/j.media.2013.05.008 -Cited by203 articles
MedicalimagesegmentationonGPUs
–Acomprehensivereview
Erik Smistad, Thomas L. Falch, MohammadmehdiBozorgi, AnneC. Elster, Frank Lindseth
Medical Image Analysis Volume20, Issue 1, February2015, Pages 1-18
https://doi.org/10.1016/j.media.2014.10.012 -Cited by81 articles
The review concludes that most segmentation methods may
benefit from GPU processing due to the methods’ data parallel
structure and high thread count. However, factors such as
synchronization, branch divergence and memory usage can
limit the speedup.
Methodologyto Increase the Computational Speed to
ObtaintheFractalDimension Using GPUProgramming
Juan Ruiz de Miras, Jesús Jiménez Ibáñez
The Fractal Geometryofthe Brain (2016) pp 533-551
https://doi.org/10.1007/978-1-4939-3995-4_34
In this chapter, we present our experience optimizing the processing time oftheclassic
box-counting algorithm to compute the FD by means of CUDA and OpenCL GPU
programming. Speedups of up to 28× (CUDA) and 6.3× (OpenCL) against the single-
thread CPU version ofthe algorithmhavebeenobtained.
Designof aportable wide field of view GPU-accelerated
multiphoton imagingsystem for real-timeimagingof
breast surgicalspecimens
Michael G. Giacomelli;TadayukiYoshitake; Lennart Husvogt;Lucas Cahill;Osman Ahsen;
Hilde Vardeh;YurySheykin;BeverlyE. Faulkner-Jones;JoachimHornegger; Jeff Brooker;
Alex Cable;James L. Connolly;James G. Fujimoto
Proceedings Volume9712, Multiphoton Microscopy in the Biomedical Sciences XVI;
97121G (2016);doi:10.1117/12.2209241
The systemis designed to produce large field ofview images at a high frame rate, while
using GPU processing to render low latency, video-rate virtual H&E images for real-
time assessment. The imaging system and virtual H&E rendering algorithm are
demonstrated byimaging unfixed human breast tissue in aclinical setting.
GPU computinginmedicalphysics:
Areview
GuillemPratx, LeiXing
Med. Phys., 38:2685–2697. doi: 10.1118/1.3578605 - Cited by211articles

GPUs in
Deep Learning
Medical Image
Analysis
Asurveyondeep learninginmedical
image analysis
GeertLitjens, Thijs Kooi, Babak EhteshamiBejnordi, Arnaud ArindraAdiyoso Setio, Francesco Ciompi, Mohsen
Ghafoorian, Jeroen A.W.M. van der Laak, Bram van Ginneken, Clara I. Sánchez
DiagnosticImage Analysis Group;Radboud University MedicalCenter; Nijmegen, The Netherlands
(Submitted on19 Feb 2017(v1), last revised 4 Jun2017 (this version, v2))
https://arxiv.org/abs/1702.05747 - Cited by54 articles
RetrievalFromandUnderstandingof
Large-ScaleMulti-modalMedical
Datasets:AReview
Henning Müller ; Devrim Unay
IEEE Transactions on Multimedia ( Volume:19, Issue:9, Sept. 2017 )
https://doi.org/10.1109/TMM.2017.2729400
Aperspective ondeep imaging
Hayit Greenspan ; BramvanGinneken ; Ronald M. Summers
IEEE Transactions on Medical Imaging ( Volume:35, Issue:5, May2016 )
https://doi.org/10.1109/TMI.2016.2553401 -Cited by 29 articles
The combination of tomographic imaging and deep learning (or machine learning in
general) promises to empower not only image analysis but also image
reconstruction. This perspective article proposes to develop novel image
reconstruction theory and techniques in the machine learning/deep learning
framework, with an emphasis on medical imaging. This direction has a potential to
re-invent the futureofhealthcare.
Breakdown of the papers included in this survey in year of publication, task
addressed, imaging modality, and application area. The number of papers for
2017 hasbeen extrapolated fromthepaperspublished in January
Collage of some medical imaging
applications in which deep learning
has achieved state-of-the-art
results. From top-left to bottom-right:
mammographic mass classification,
segmentation of lesions in the brain
(BRATS, ISLES and MRBrains
challenges), leak detection in airway
tree segmentation, diabetic
retinopathy classification (Kaggle
Diabetic Retinopathy challenge
2015), prostate segmentation
(PROMISE12 challenge), nodule
classification (LUNA16 challenge),
breast cancer metastases detection
in lymph nodes ( CAMELYON16),
human expert performance in skin
lesion classification, and state-of-
the-art bone suppressionin x-rays.
Exampleof a
radiologist’s
work flow in a
clinical
environment.

GPUs
dominate
deep learning
hardware
GPU:the biggestkeyprocessor for AI and
parallelprocessing
ToruBaji; NVIDIA (Japan)
Proc. SPIE 10454, Photomask Japan2017:XXIVSymposiumonPhotomask and Next-
GenerationLithography Mask Technology, 1045406 (13 July2017)
http://dx.doi.org/10.1117/12.2279088
https://www.technologyreview.com/s/603917/nvidias-deep-
learning-chips-may-give-medicine-a-shot-in-the-arm/
In imageclassification, GPUpowerand model
capacity haveincreasedwhileopen-source
datasetshavelaggedbehind
–Sun etal. (2017), Google
TheCenter for Clinical DataScience (CCDS), Boston, isatthe
confluence of major technology trends driving the healthcare
industry: AI-based diagnostics of large volumes of medical images,
shared among multiple medical institutions, utilizing GPU-based
neural networks. Founded by Massachusetts General Hospital
and later joined by Brigham & Women’s Hospital, CCDS today
announced it has received what it calls a purpose-built AI
supercomputer from the portfolio of Nvidia DGX systems with
Volta, said by Nvidia to be the biggest GPU on the market.
Sept7, 2017
https://www.hpcwire.com/2017/09/07/first-volta-based-nvidia-dgx-systems-ship-bost
on-based-healthcare-providers/

CPUs
and Intel trying
to fight a bit
with Xeons
Intel already been
buying out hardware
like Altera, Nervana,
Movidius to compete
with NVIDIA
Intelplanstoshipitsfirst-
generation NeuralNetwork
Processorby theendoftheyear
BLAIR HANLEY FRANK @BELRIL OCTOBER 17, 201711:00 AM
https://venturebeat.com/2017/10/17/intel-plans-to
-ship-new-ai-chips-by-the-end-of-the-year/
Thecompanyannouncedtodaythatitsfirst-
generationNeuralNetworkProcessor,code
named“LakeCrest,”willberollingouttoasmallset
ofpartnerssoontohelpthemdrastically
acceleratehowmuchmachinelearningworkthey
cando.
Intel and Google engineers have been working together to
optimizeTensorFlow*, a flexible open-source AI
framework, for Intel® Xeon® and Intel® Xeon Phi™
processors.https://www.intelnervana.com/tensorflow/
Intel Agreesto
Buy Altera for
$16.7 Billion
On-again-off-again deal is
latest acquisition in active
semiconductorsector
https://www.wsj.com/articles/intel-agree
s-to-buy-altera-for-16-7-billion-1433162
006
See also Xilinx vsAltera
FPGA discussion
Intel is paying more than
$400 million to buy
deep-learning startup
Nervana Systems
The chip giant isbetting that
machine learningisgoing to
be a bigdeal in the data
center.
https://www.recode.net/2016/8/9/12413
600/intel-buys-nervana--350-million
Intel buyscomputer
vision startup Movidius
asit looks to build upits
RealSense platform
In theyearsahead,we’llsee
newtypesofautonomous
machineswithmoreadvanced
capabilitiesaswemake
progresson oneofthemost
difficultchallengesofAI:
gettingourdevicesnot just to
see,but alsotothink.”
https://techcrunch.com/2016/09/05/intel-buys-
computer-vision-startup-movidius-as-it-looks-
to-build-up-its-realsense-platform/

CPUs
Intel putting
more bets on
Nervana Neural
Network
Processsor
And the next-gen Xeon
‘Knights Crest’
InteltoshipnewNervanaNeural
NetworkProcessor byendof2017
Posted Oct17,2017 by John Mannes (@JohnMannes)
https://techcrunch.com/2017/10/17/intel-to-ship-new-nerva
na-neural-network-processor-by-end-of-2017/
This morning at the WSJ’s D.Live event, Intel formally
unveiled its Nervana Neural Network Processor (NNP)
family ofchipsdesigned for machine learning use cases.Intel
has previously alluded to these chips using the pre-launch
name Lake Crest. Intel appears to have every intention of
building a full product line around its Nervana NNP chips. A
subsequent Xeon processor for AI has been rumored under
thecodename“KnightsCrest.”
https://www.anandtech.com/show/11942/intel-shipping-nervana
-neural-network-processor-first-silicon-before-year-end
Bringing to mind Google'sTPUand NVIDIA'sTensor
Cores,theNNP'stensor-based architectureisanother
exampleof howoptimizationsfordeep learning
workloads arereflected in thesilicon. TheNNP also
utilizesNervana’snumerical formatcalledFlexPoint,
describedasin-between floatingpointand fixed point
precision.

Memory
as a processor?
IBM Research
demonstrates the
idea
IBMCanRunanExperimentalAI inMemory,Not onProcessors
https://www.technologyreview.com/the-download/609205/ibm-can-run-an-experimental-ai-in-memory-not-on-processors/
.. .And that’s exactly the concept that a team from IBM Research in Zurich hasnowappliedtosomeAIalgorithms. The
team has used a grid of one million memory devices, which are all based on a phase-change material called
germanium antimony telluride. The alloy’s specialtrick is that, when it’s hit by an electrical pulse, its state can be
changed—fromamorphous,likeglass,tocrystalline,likemetal, orviceversa.
By varying the size and duration of the electric pulses, it’s possible to change the amount by which that crystallization
changes. And that, in turn, can be used to represent a number of different states, not just regular 0s and 1s, which can be
used to performcalculations rather than just store data. By using that quirk and enough chunks of memory, the IBM
researchers have shown that they can perform machine-learning tasks like finding correlations in unknown data streams.
Theworkis publishedin NatureCommunications[doi:10.1038/s41467-017-01481-9].
This is, admittedly, a small, niche, lab-based study. But the team reckons it could, if scaled up, create computing
systems that perform some AI tasks 200 times faster than regular devices. Even if it can achieve just a
fractionofthatboost,in-memoryAImayhaveafuture.
Experimental results. a A million processes are mapped to the pixels of a
1000 × 1000 pixel black-and-white sketch of Alan Turing. The pixels turn on and
offin accordancewith theinstantaneousbinaryvaluesof the processes
Experimental platform and characterization
results. a Schematic illustration of the experimental
platform showingthe main components.

Retinal Image Analysis
Hardware
GPUsofcourse
the defacto
standardfrom
KaggletoGoogle
Everyone has heard of Kaggle, but have you heard of London-based
GoogleDeepMind? Their researchers build deep learning algorithms
to conquer everything from Pongand the ancientgameof go to
blindness caused by diabetic retinopathy. If the latter sounds
particularly familiar, you may be recalling the
Diabetic RetinopathyDetection competition which ran on Kaggle from
February2015toJuly2015.
In this blog post, I interview JeffreyDeFauw who came in 5th place in
this competition using convolutional neural networks and is first author
of Google DeepMind'sstudy spearheading efforts to automate
analysis of ophthalmic images using machine learning in order to help
cliniciansdiagnosesight-threateningdiseases.
JeffreyDeFauw (7 Nov 2016): “Everything was trained on
a NVIDIAGTX980 (4.6 TFLOPS) in the beginning; this was
the GPU for thedesktop Iwas alsoworking on,which wasn’t
ideal. Therefore, later I also tried using the GRIDK520 on AWS
(even thoughit wasatleast twotimesslower).“
Something elsethat I had alreadystarted testing in modelssomewhat,which
seemed to be quite critical for decent performance, was oversampling the
smaller classes. I.e., you make samples of certain classes more likely than
others to be picked as input to your network. This resulted in more stable
updates and better, quicker training in general (especially since I was using
small batch sizes of 32 or 64 samples because of GPU memory
restrictions).
http://blog.kaggle.com/2016/07/11/from-kaggle-to-google-deepmind-an-interview-with-jeffrey-de-fauw/
ARTIFACTS IN THE IMAGES WHICH ARE SOMETIMES
ONLY VISIBLE WHEN VIEWING THE IMAGES FROM BOTH
EYES SIDE BY SIDE. THESE ARTIFACTS CAN RESEMBLE
SOME OF THE PATHOLOGIES OFTHEDISEASE.
Easier to handle the artifacts
and the image quality
optimization during acquisition
process if possible rather
than trying to figure out during
the analysis which is artifact
and which is pathological
feature

Hardware
GPUsof course the defacto
standard fromKaggleto
Google
Dependingonyourbudget,
the moreGPUsthe faster
the development
Applying artificialintelligence to disease
staging: Deep learning for improved
staging of diabetic retinopathy
HidenoriTakahashi, Hironobu Tampo, Yusuke Arai, Yuji Inoue,
Hidetoshi Kawashima| Departmentof Ophthalmology, Jichi Medical University
PLOSOne(2017) https://doi.org/10.1371/journal.pone.0179790
“We propose a novel AI disease-staging system for
grading diabetic retinopathy that involves a retinal
area not typically visualized on fundoscopy
and another AI that directly suggests treatments and
determinesprognoses.”
“The original photographs were 2,720 × 2,720
pixels. The outlying 88 pixels of the margin
were deleted and the photographs shrunken by
50% to 1,272 × 1,272 pixels to fit in the
graphical processing unit memory (12 GB,
TITAN X; NVIDIA, ~11 TFLOPs, ~£1,200/
2150 SGD). Four graphical processing units
were used simultaneously.”
Cholletetal(2017),Google: All networks were
implemented using the TensorFlow framework [1] and
trained on 60 NVIDIA K80 GPUs (60*£3,800 ~ £228k
~408kSGD)each.
https://www.nextplatform.com/2016/04/22/baidu-eyes-deep-learning-strategy-wake-new-gpu-options/
What Baidu really cares about is memory capacity.
With this in mind, it is clear why they will be anxiously
waiting for Pascal, because currently they’re limited to 12
GB of memory per GPU, which is constraining. “We are
constantly running into limitations because of memory,”
Bryan Catanzaro says. The M40, which is designed for
deep learning training has 12 GB of memory, but at GTC16,
Nvidia pointed to a 24 GB version of the Quadro
M6000
NVIDIADGX-1 -Unboxingat Benevolent.AI
withold-generation P100insteadofV100s
https://youtu.be/nWBUoNPMHDs

Hardware
FPGA
Approaches
starttoemerge
Hamza Bendaoudi. (2017). Flexible
Hardware Architectures for
Retinal Image Analysis (PhD
thesis, École Polytechnique de
Montréal). Retrieved from
http://publications.polymtl.ca/2518/
Overview of theZynq-based system. Zynq-7000 AP SoCthat integratesadual-core ARMCortex–A9based processing
system (PS) and a7-seriesXilinxProgrammable Logic fabric (PL)inasingledevice.
FPGAversusCPU implementation performances
FPGA resources utilization for the Multi-Scale Line Detector (MSLD) architecture for
DRIVE database images. The FPGA implementation uses 20% of the FPGA LUTs and
50% of its DSP Blocks. The maximum clock frequency is 60.4 MHz. This
implementation achieves real-time execution of the MSLD algorithm with a throughput
of71 f/s.
Summaryof existing implementations ofRetinal BloodVesselDetection
Algorithms

The Big
Boys
Google Cloud,AWS,
IBM, MicrosoftAzure,
etc.
Docker+Kubernetize
yourcode tostayas
platform-agnosticas
possible
DataSciencefor theInternetof Things(IoT)
UniversityofOxford
Course developed byAjit Jaokar
https://www.conted.ox.ac.uk/courses/data-sci
ence-for-the-internet-of-things-iot
Deployingdeeplearningmodelswith
DockerandKubernetes
https://www.slideshare.net/PetteriTeikariPhD/deploying-deep-learn
ing-models-with-docker-and-kubernetes
Dockerwillinclude
Kubernetesinthebox
OCT17,2017
https://www.infoworld.com/article/3233133/containers/
docker-will-include-kubernetes-in-the-box.html
Both the enterprise and desktop
editions of Docker will include
Kubernetes, making the container
technology easier to use in both
development and production
environments

AWS
WithNVIDIAV100s
firstin frontofGoogle
and Azure
Cloud GPU of course more flexible than local cluster, but then
you have take into account the instance rates, and possible
privacy issues if you are processing confidential patient data
AWSbeatsGoogle
andMicrosoft to
launchinstances
withNvidiaVolta
GPUs
BLAIRHANLEYFRANK@BELRIL
OCTOBER25,20179:28PM
https://venturebeat.com/2017/10/25/aws-beats-goo
gle-and-microsoft-to-launching-instances-with-nv
idia-volta-gpus/
Amazon Web Services is the first cloud provider to
launchnewcomputeinstancesallowing developers
to build applications that tap into Nvidia’s new
generation of Volta GPUs. These GPUs are
designedtoprovidehigh-performanceacceleration
for applicationslikeAIcomputation.
https://www.tweaktown.com/news/59155/nvidia-tesla-v100-tes
ted-near-unbelievable-gpu-power/index.html
https://www.mic
roway.com/downl
oad/Microway_Te
sla_V100_P100_S
olutions.pdf

Local GPU
Server,
Deskop and
Laptop
Options

Depending
on your
needs then
+ GPUServer
+ Desktop
+ Laptop asthin client
https://www.slideshare.net/PetteriTeikariPhD/deep-learning-workstation

GPUs for
Embedded
Portable
Devices
Autonomous driving
probably the hottest
application right now
driving NVIDIA’s R&D
efforts Tegra Jetson→
TX1/2 Drive PX→
Thecompanyclaimsthe new
computer, which isaboutthe sizeof a
license plate. Drive PX Pegasuswill
purportedlybe about 10 timesmore
powerful than itspreviouscomputer
Drive PX2.
https://nvidianews.nvidia.com/news/nvi
dia-announces-world-s-first-ai-compute
r-to-make-robotaxis-a-reality
http://www.siliconhighwaydirect.co.uk/product-p/900-83310-0001-000.htm:
NVIDIA JetsonTX1and TX2Comparison
https://youtu.be/hz9000u5pw0

Alternatives
for GPUs
-ASIC
-FPGA
-MCUs
Application-specific
instruction sets can
for sure make ASICs
viable alternatives for
GPU in terms of raw
computing power
Google’s 2nd
generation Tensor Processing Unit (TPU, ASIC for machine learning) available on Google
CloudPlatformcapableforbothtrainingandinference

Alternatives
for GPUs
-ASIC
-FPGA
-MCUs
FPGAs can offer
better performance/
watt making them
good alternatives for
embedded devices

Alternatives
for GPUs
-ASIC
-FPGA
-MCUs
FPGAs on the cloud
as well
AmazonAndXilinxDeliver
NewFPGASolutions
https://www.forbes.com/sites/moorinsights/2017/
09/27/amazon-and-xilinx-deliver-new-fpga-soluti
ons/#1e5c62332370
This afternoon Microsoft announced Brainwave, an FPGA-based
system for ultra-low latency deep learning in the cloud. Early
benchmarking indicates that when using Intel Stratix 10 FPGAs,
Brainwave can sustain 39.5 Teraflops on a large gated recurrent unit
without anybatching.

Alternatives
for GPUs
-ASIC
-FPGA
-MCUs
OCT Imaging have
been accelerated also
with FPGAsin practice the
FFT inthe signal processing
pipeline
AlazarTech ATS9373 waveformdigitizer(A/Dboard)
http://www.alazartech.com/landing/oct-news-2016-09

Alternatives
for GPUs
-ASIC
-FPGA
-MCUs
Deep learning directly
on the Arduinos and
Fitbits of the world
(wearable computing)
SparseSep – a set of novel techniques for
optimizing large-scale deep learning models –
that allows deep models to function even under
the extreme system constraints presented by
wearablehardware
Sourav Bhattacharya, Nicholas Lane (2016)
https://doi.org/10.1145/2994551.2994564
DeepX toolkit (DXTK); an opensource collection of software
components for simplifying the execution of deep models on
resource-sensitiveplatforms.
NicholasLaneetal.(2017),NokiaBellLabsandUCL
https://doi.org/10.4108/eai.30-11-2016.2267463
DeepEye-AkhilMathuret al.(2017)
http://dx.doi.org/10.1145/3081333.3081359

Smartphone
AI
Built-in ‘AI
processors’ coming to
high-end mobile
phones
http://consumer.huawei.com/en/phones/mate10-pro/
(17Oct 2017)http://www.wired.co.uk/article/huawei-mate-10-pro-price-specs-release-date
Monday16October2017
https://www.theguardian.com/technology/2017/oct/16/hua
wei-launches-mate-10-pro-with-built-in-ai-to-challenge
-apple-samsung
GoogleintroducesNeural Networks
API indeveloperpreviewofAndroid8.1
https://techcrunch.com/2017/10/25/google-introduces-neur
al-networks-api-in-developer-preview-of-android-8-1/
If available, the API can make use of special AI chips on the device — or
fall back to theCPU if that’stheonlyoption. Google’snewPixel 2phones
featuresucha special chip (thePixel Visual Core) and Googlepreviously
said thatitplanned to turniton oncethe8.1 previewwent live(so today…).
Remember how Google said the Pixel 2and Pixel 2XL both have a
customimaging chip that's just laying idle? Well, you can finally use it... in a
manner of speaking. Google has released its first Developer Preview for
Android 8.1, and the highlight is arguably Pixel Visual Core support for third-
party apps. Companies will have to write support into their apps before you
notice the difference, but this should bring the Pixel 2 line's HDR+
photography to any app, not just Google's own camera software. You might
not have to jump between apps just to get the best possible picture quality
whenyou'resharing photos throughyourfavoritesocial service.

Smartphone
-based
diagnostics
#1
Smartphone as the
‘mainframe’ for tasks
also beyond image
classification
Mobile phone-basedbiosensing:
Anemerging“diagnosticand
communication”technology
Biosensorsand BioelectronicsVolume 92, 15June 2017,
Pages549-562. https://doi.org/10.1016/j.bios.2016.10.062
Herein we provide an overview of a broad range of
biosensing possibilities, from optical to
electrochemical measurements; explore the various
reported designs for adapters; and consider future
opportunities for this technology in fields such as health
diagnostics, safety & security, and environment
monitoring.
Mobile phone adapter toperform ELISA tests.
Lateral Flow (LF) Imminochromatographic Device. (a)
Schematic representation of LF adapter for mobile
phones, (b)LFcassettecomposition and
Comparison ofdifferent detection methodsapplicable on mobile phones.
Fluorescent and brightfield microscope adapted for
its use on mobile phones.
SurfacePlasmon Resonance(SPR)-based
mobilephonebiosensor. (above)Connecting the
flash cameraand thecamerawithoptical fibbers,
(below)Explanation mechanism

Smartphone
-based
diagnostics
#2
Smartphone as the
also beyond image
classification
Smartphone-powered
tooloffers point-of-care
infectiondiagnoses
Housed within a portable 3D-printed cradle, the tool uses
the smartphone and an app to interpret real-time images
of a microfluidic chip embedded on credit card-like
cartridge. Researchers say it's the only point-of-care
platform to date that is able to simultaneously perform
multiple tests for viral or other nucleic targets using just
onedroplet of body fluid.
http://www.mobihealthnews.com/content/smartphone-powered-tool-offers-p
oint-care-infection-diagnoses
Cana smartphone-enabled ultrasound
machinebecomemedicine’snext
stethoscope?
ButterflyIQ, isthefirstsolid-stateultrasound machineto reach themarketin theUS,
developed by ButterflyNetwork
https://www.technologyreview.com/s/609195/this-doctor-diagnosed-his-own-can
cer-with-an-iphone-ultrasound/

Smartphone
-based
diagnostics
#3
Smartphone as the
also beyond image
classification

Smartphone
-based
diagnostics
Mounted on
a drone
Lab-on-a-drone: towardpinpoint
deploymentofsmartphone-
enablednucleicacid-based
diagnosticsformobile healthcare
Anal Chem. 2016May3;88(9):4651–4660.
doi: 10.1021/acs.analchem.5b04153
Lab-on-a-drone. (A) Convective thermocycling enables the PCR
to be actuated isothermally using a single heater. (B) The instrument
can be assembled for $50 ($US)∼ using readily available
components (built around Arduino microcontroller). (C)
Fluorescence detection of reaction products is achieved using an
ordinary smartphone camera. (D) The entire assembly is incredibly
lightweight, enabling deployment on consumer-class quadcopter
drones. (E) Ruggedization is demonstrated by performing in-flight
PCR as a drone payload. Successful in-flight replication of two
different DNA targets is achieved (lane M, FlashGel DNA marker;
lane 1, 147 bp S. aureus target (16 min in-flight reaction time); lane 2,
237 bp target from a -phage DNA template (18 min in-flight reactionλ
time)). Tamb 23 °C.∼
Drone-basedsamplepreparation.
Quantitative smartphone-based fluorescence detection. The PCR to Go analysis
app enablessmartphone-based image acquisition, processing, and dataanalysis.

Smartphone
-based
Diagnostics
Implanted
sensors
ConfirmRx Smartphone-
connected Insertable
Cardiac Monitor (ICM)
from Abbott
Senseonics, a long-term
implantablesmartphone-
connected continuousglucose
monitor, http://www.senseonics.com/
IMDShield:SecuringImplantableMedicalDevices
Shyamnath Gollakota, Haitham Al Hassanieh, Ben Ransford,
Dina Katabi, Kevin Fu
https://groups.csail.mit.edu/netmit/IMDShield/
https://www.meddeviceonline.com/doc/long-lasting-injectable-cgm-uses-live
-cells-to-regenerate-sensor-0001
Chronically Implanted Pressure Sensors:
Challenges and State of the Field
LawrenceYu, BrianJ. Kimand Ellis Meng
Sensors 2014, 14(11),20620-20644;
doi:10.3390/s141120620
The Juvenile Diabetes
Research Fund (JDRF) has
invested in a long-lasting
continuous glucose monitor
(CGM) that uses engineered
live cells to replenish and
maintain the stability of the
implant’s optical biosensor over
time. The GluSenseGlyde,, a
product of Rainbow Medical, is
projected to last up to one year
under the patient’s skin,
disrupting the current market of
CGMs that last between seven
and 90 days.

Smartphone
-based
Therapeutics
Digitalhealth
ecosystemand
gooduseofsensor
dataforthepatient’s
good
Digital therapeuticsstartupsband
togetherto form new industry group
By Jonah ComstockOctober 25, 2017
http://www.mobihealthnews.com/content/digital-therapeutics-startups-
band-together-form-new-industry-group
“A group of health startups in the burgeoning digital therapeutics space have joined
togetherto createa new industryassociation, the Digital Therapeutics Alliance (DTA).”
Trends InDigitalHealthToKeepAnEye
OnIn2018
Answer by NitinGoyal, MD, Orthopaedic Surgeon, on Quora:
https://www.forbes.com/sites/quora/2017/09/29/4-trends-in-digital-
health-to-keep-an-eye-on-in-2018/#3a0a6b3764ac
1) DigitalHealth Interventions
2) Provider-Centric Solutions
3) Big Data &Analytics
4) NewModel InsuranceCompanies
HxRefactored 2016
The Health IoT: RemoteCare andMobile Solutions -AndrewHooge, Validic
Putting digital biomarkers and apps topharma’s clinical trials via
Clinical ResearchOrganizations (CROs)likeinVentiv Health, Parexel orQuintilesIMS.
https://www.technologyreview.com/s/604053/can-digital-therapeutics-
be-as-good-as-drugs/
Howhospitalinnovatorsaretackling
patientsatisfaction,vendorpartnerships,
anddataoverload
By Jonah ComstockOctober 13, 2017
http://www.mobihealthnews.com/content/how-hospital-innovators-are-ta
ckling-patient-satisfaction-vendor-partnerships-and-data
Day1 of last week’s Health 2.0conference in Santa Clara, California was a Provider
Symposium, whereinnovationpersonnel at someofthe nation’s largeand notable
healthsystems came together to speak about their experiences around innovation,
big data, and patient engagement, among othertopics.

Inference
Optimization
Efficient Processing of Deep Neural Networks: A Tutorial
and Survey
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel Emer
last revised 13 Aug 2017 https://arxiv.org/abs/1703.09039
Impactofenergy-awarepruning.Energyvalues
estimatedwithmethodologyin Yang etal.(2017)
[http://eyeriss.mit.edu/energy.html; https://energyestimation.mit.edu/]
Methods to reducenumerical precision forAlexNet (Krizhevskyet al.2012; Citedby12,168 articles)
. Accuracymeasured forTop-5 error onImageNet. * Notapplied to firstand/or lastlayers.

Inference
Optimization
A Survey of Model Compression and Acceleration for Deep
Neural Networks
Yu Cheng, Duo Wang, Pan Zhou, Tao Zhang (Submitted on 23 Oct 2017)
https://arxiv.org/abs/1710.09282v1

Post-training
Optimization
The simplest
approach as we
“just” optimize the
“research-grade”
networks for
deployment once we
are happy with the
results

Model
Compression
Bayesian
approach for
inducingsparsity for
hidden units
BayesianCompressionforDeep
Learning
ChristosLouizos, Karen Ullrich, MaxWelling
(Submitted on 24 May2017(v1), last revised 10 Aug 2017(this version, v3))
https://arxiv.org/abs/1705.08665
By employing sparsity inducing priors for hidden
units (and not individual weights) we can prune
neurons including all their ingoing and outgoing
weights. This avoids more complicated and inefficient
coding schemes needed for pruning or vector
quantizing individual weights. As a additional
Bayesian bonus we can use the posterior
uncertainties to assess which bits are significant
and remove the ones which fluctuate too
much under posterior sampling. From this we derive
the optimal fixed point precision per layer, which is still
practicalonchip.

Model
Compression
Sparsification via
anelegant Bayesian
interpretationto
GaussianDropout.
VariationalDropoutSparsifiesDeepNeural
Networks
Dmitry Molchanov, ArseniiAshukha, DmitryVetrov
(Submitted on 19 Jan2017 (v1), last revised 13 Jun2017 (this version, v3))
https://github.com/ars-ashuha/variational-dropout-sparsifies-dnn
https://goo.gl/GZk5FF
We showed that Variational
Dropout leads to extremely
sparse solutions both in fully-
connected and convolutional
layers. Sparse VD reduced the
number of parameters up to
280 times on LeNet
architectures and up to 68
times on VGG-like networks
with a negligible decrease of
accuracy.
This effect is similar to the
Automatic Relevance
Determination effect in
empyrical Bayes. However, in
Sparse VD the prior distribution
remaines fixed, so there is no
additionalrisk ofoverfitting.
We visualize the weights of
Sparse VD LeNet-5-Caffe
network and demonstrate
several filters of the first
convolutional layer and a piece
ofthefully-connectedlayer :)
Sparsification
StructuredBayesian PruningviaLog-Normal MultiplicativeNoise
Kirill Neklyudov, DmitryMolchanov, Arsenii Ashukha, DmitryVetrov
(Submitted on 20May2017)
https://bayesgroup.github.io/bmml_sem/2017/log_norm_pres.pdf
Fromsamegroup:

Model
Compression
Softweight-
sharing(Nowlan& Hinton,1992)
exposing the relation between
compressionand the
minimumdescriptionlength
(MDL, Rissanen1978, Citedby6292
)
principle.
SoftWeight-SharingforNeuralNetworkCompression
Karen Ullrich, Edward Meeds, Max Welling
(Submitted on 13 Feb 2017(v1), lastrevised 9 May2017 (this version, v2))
https://github.com/KarenUllrich/Tutorial-SoftWeightSharingForNNCompression
Illustration of our mixture model
compression procedure on LeNet-5-
Caffe. Left: Dynamics of Gaussian
mixture components during the learning
procedure. Initially there are 17
components, including the zero
component. During learning
components are absorbed into other
components, resulting in roughly 6
significant components. Right: A
scatter plot of initial versus final weights,
along with the Gaussian components’
uncertainties. The initial weight
distribution is roughly one broad
Gaussian, whereas the final weight
distribution matches closely the final,
learned prior which has become very
peaked, resulting in good quantization
properties.
Convolution filters from LeNet-
5-Caffe. Left: Pre-trained
filters. Right: Compressed
filters. The top filters are the
20 first layer convolution
weights; the bottom filters are
the 9½ (out of 50) first of the 20
by 50 convolution weights of
thesecond layer.

Model
Compression
SimpleReLU
tweak via skipping
zero-valued neurons
SpeedingupConvolutionalNeuralNetworks
ByExploitingtheSparsityofRectifierUnits
Shaohuai Shi, XiaowenChu
(Submitted on 25 Apr2017 (v1), lastrevised 15 May 2017(this version, v2))
We measure the speed of compared algorithms
on the Interl CPU: E5-2630v4 at the core
frequency of 2.20GHz with 128 GB memory.
The experimental resultsare showninTable3
Matrixmultiplicationforconvolution The proposed sparse convolution
algorithm achieves some speedup
improvements on CPUs compared
to the traditional matrix-matrix
multiplication algorithm for
convolution when the sparsity is
notlessthan0.9.

Model
Compression
Energy estimation
especially formobile and
embedded use,butalso
large-scale datacenters
are usinga lot of energy
DesigningEnergy-EfficientConvolutionalNeural
NetworksusingEnergy-AwarePrunings
Tien-Ju Yang, Yu-Hsin Chen, VivienneSze
(Submitted on 16 Nov 2016 (v1), last revised 18 Apr 2017 (this version, v4))
http://eyeriss.mit.edu/energy.html | https://energyestimation.mit.edu/
[8] S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep
Neural Networks with Pruning, Trained Quantization and Huffman Coding,”
in ICLR, 2016. https://arxiv.org/abs/1510.00149
Energy
consumption
breakdown of
differentAlexNetsin
termsof the
computationand
thedata movement
of inputfeature
maps, output
featuremapsand
filterweights. From
lefttoright: original
AlexNet,AlexNet
pruned by[8],
AlexNetpruned by
theproposed
energy-aware
pruning.

FPGA/ASIC
Low-precision inference
optimization
See
ShiftCNN:GeneralizedLow-Precision
ArchitectureforInferenceof
ConvolutionalNeuralNetworks
Denis A. Gudovskiy, LucaRigazio | Panasonic SiliconValleyLab
(Submitted on 7Jun2017)
The proposed architecture targets custom inference
accelerators and can be realized on FPGAs or ASICs.
Extensive evaluation on ImageNet shows that the state-
of-the-art CNNs can be converted without retraining into
ShiftCNN with less than 1% drop in accuracy when the
proposed quantization algorithm is employed
Conventional CNN implementations on
GPUs use a power-inefficient floating-
point format. For example, (Horowitz,2014
) estimated that 32-bit floating-point
multiplier consumes 19× more power
than8-bitintegermultiplier
Table 3 presents FPGA utilization and
power consumption estimates of the
implemented computational pipeline in
terms of number of lookup tables (LUTs),
flip-flops (FFs), digital signal processing
blocks (DSPs) and dynamic power
measuredbyXilinxVivadotool.
Automotive-gradeXilinxZynq XA7Z030 devicerunning at200 MHz clock rate

Optimizing for
Embedded
NVIDIA GPUs
Th

Change-
based
Processing
Especiallyforsurveillance
(security,traffic,animal,baby,
etc.)applications,nopointof
streamingalot ofvideodata
overthe internetifnothing
relevant happensonthe
otherside.
And again, what would be
better way to robustify
such change detection
than deep learning on the
edge device
CBinfer:Change-BasedInferenceforConvolutional
NeuralNetworksonVideoData
Lukas Cavigelli, PhilippeDegen, LucaBenini
(Submitted on 14 Apr 2017(v1), lastrevised 21Jun 2017(this version, v2))
Extracting per-frame features using
convolutional neural networks for real-
time processing of video data is currently
mainly performed on powerful GPU-
accelerated workstations and compute
clusters. However, there are many
applications such as smart surveillance
cameras that require or would benefit
fromon-siteprocessing

Mobile DNNs
Google workswith
depthwise separable
convolutions
MobileNets:EfficientConvolutionalNeural
NetworksforMobileVisionApplications
Andrew G. Howard, Menglong Zhu, Bo Chen, DmitryKalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, Hartwig Adam | GoogleInc.
(Submitted on 17Apr 2017)
Xception:DeepLearningwithDepthwise
SeparableConvolutions
François Chollet | GoogleInc.
(Submitted on7 Oct2016 (v1), lastrevised 4Apr 2017(this version, v3))
https://github.com/fchollet/deep-learning-models
The Xception architecture: the data first goes through the entry flow, then through
the middle flow which is repeated eight times, and finally through the exit flow.
Note that all Convolution and SeparableConvolution layers are followed by batch
normalization (not included in the diagram). All SeparableConvolution layers use a
depth multiplier of1 (no depthexpansion).
23,626,728
22,855,952
Parameter count
Xception significantly
outperformsInception V3 on a
larger image classification
dataset comprising 350 million
images and 17,000classes
(JFTdataset). Since the Xception
architecture has the same
number ofparameters as
Inception V3,the performance
gains are not due to increased
capacitybut rather to amore
efficient use of model
parameters.
Left: Standard convolutional layer with batchnorm and
ReLU. Right: Depthwise Separable convolutions with
Depthwise and Pointwise layers followed by batchnorm and
ReLU.

Mobile DNNs
ShuffleNetslightly
lowerclassification
errorwithslightly
lowercomplexity
(MFLOPS)
ShuffleNet:AnExtremelyEfficientConvolutional
NeuralNetworkforMobileDevices
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, JianSun | MegviiInc(Face++)
(Submitted on 4 Jul 2017)https://arxiv.org/abs/1707.01083
The new architecture utilizes
two proposed operations,
pointwise group convolution and
channel shuffle, to greatly
reduce computation cost while
maintaining accuracy.
Channel shuffle with two stacked group convolutions. GConv stands for group convolution. a) two stacked convolution layers with the same
number of groups. Each output channel onlyrelates tothe input channels within the group. Nocross talk; b) input and output channelsare fully
related whenGConv2 takesdatafrom different groupsafter GConv1;c) an equivalent implementationtob) usingchannel shuffle.
Dueto memory access and otheroverheads, wefind every 4× theoretical complexity
reduction usually resultsin ~2.6× actual speedup in ourimplementation.

Mobile DNNs
Internet-of-Things(IoT)/8-bit
ArduinoUno-level
computingcapability
This paper develops a novel tree-
based algorithm, called Bonsai, which
can be trained on a laptop, or the
cloud, and can then be shipped onto
severely resource constrained
Internet of Things (IoT) devices.
The Arduino Uno board has an 8 bit
ATmega328P microcontroller operating
at 16 MHz with 2 KB SRAM and 32 KB
read-only flash memory. The BBC
Micro:Bit has a 32 bit ARM Cortex M0
microcontroller operating at 16 MHz
with 16 KB SRAM and 256 KB read-only
flash. Neither provides hardware
support for floating point
operations.
MachineIntelligenceonResource-ConstrainedIoT
Devices:TheCaseofThreadGranularityOptimization
forCNNInference
Mohammad Motamedi, Daniel Fong, Soheil Ghiasi
ACM TransactionsonEmbedded ComputingSystems(TECS), 2017
https://doi.org/10.1145/3126555
The problem of selecting the best thread granularity for
mobile SoCs is complicated because the target platform
has a heterogeneous architecture. This allows threads to
be mapped to the available CPU cores, GPUs, and DSPs.
In addition, most SoCs share the same memory among
all processors which leads to a more non-deterministic
memory access time. The problem of selecting an
optimal thread granularity for a mobile SoC has yet to be
addressed.

Mobile DNNs
Example inclinical/health
use toillustrate thisabit
ARM,ASIC,FPGA,
Qualcomms,you name it,on
mobilephones,drones,
robots
withoutbeingGPUsorlow-
endCortexARMsor
Arduino-level cheapos
The Monte Jade Science and Technology Association of New England (
MJNE) will hold its annual meeting in MIT Sloan School of Management on
Saturday November 4th, 2017. This year, the theme of our conference is
“Innovation beyond boundaries- Internet ofMedical Things (IoMT)". We
would like to address this theme from various perspectives including digital
health, patient monitoring, healthcare management, consumer wearable
device design, IoTsecurity, IP and venture capital investment, etc.
Speakers: ProfessorHTKung ,Professor ofComputerScience andElectricalEngineering, HarvardUniversity; Professor
Chung-Kang Peng, Nonlinear DynamicsinMedicineattheBethIsraelDeaconessMedicalCenter /Harvard Medical
School;StevenMo, PhD, Vice Presidentof ofOperationsandMarket Strategy,UnitedHealth Group'sOptum Division; Zen
Chu, Senior LecturerinHealthcareInnovation at MITSloanManagementandHarvard-MITHealthS&Tprogram; Matt
Templeton, Director ofDesign Innovation,BluecrossBlueshield ofMassachusetts; Jili Chung,JD,PhD, MBA,Consultant,
PHYCOS InternationalCo., Ltd.Taiwan
ImecandHolst Centre willpresenta
wirelessEEGheadset
https://medicalxpress.com/news/2014-11-breakthrough-qua
litative-reliable-eeg.html
Problem:Highdataratefrom
rawtelehealthEEG/ECG
OneSolution:Inferencein situ,
andtransmitonly classlabels
SLEEPNET: Automated Sleep Staging
SystemviaDeep Learning
New algorithmsfor processing time-
series big EEGdatawithin mobile health
monitoring systems
doi: 10.1016/j.cmpb.2017.07.007
Cloud-based deep learning of big EEG
datafor epileptic seizure prediction
doi: 10.1109/GlobalSIP.2016.7906022
Cardiologist-level arrhythmiadetection
with convolutional neural networks
0
1
2
3
Awake (W)
Stage 1Stage 1
Stage 2
Stage 3 & 4
REM Sleep
4
Data rate (each 30 second epoch labeled): 0.1 bits/second
reduction in data rate is 800,000 times
5 different states, requires
thus 3 bits to encode (23
bits
= 8 bits)
Sleep EEG e.g. 500 Hz, 6+2+1+1
(EEG+ground+EOG+ECG) channels, 16bit
ADC (500*10*16 bits/s = 80 kilobits/s = 10
kilobytes/s)
Alternativelyonecould preprocessand filterto
20 Hz at6 channelsasused for machine
learning developmentin https://arxiv.org/abs/1707.08262
https://doi.org/10.1016/j.future.2017.02.014

Mobile DNNs
When youwould like to
deploymedicalmodels
directlyonedge devices
withoutcloudinference
It is nice to be identify
prescription pills
correctly even without
internet connection, both
in developed and emerging
economies.
At worst, someone’s life
might depend on correct
medication, and internet
connection gone down at
that moment is not the
nicest thing to happen
MobileDeepPill:ASmall-FootprintMobileDeepLearning
SystemforRecognizingUnconstrained PillImages
XiaoZeng, KaiCao, MiZhang
MobiSys'17 Proceedingsofthe 15thAnnual International Conference on Mobile
Systems, Applications, and Services
https://doi.org/10.1145/3081333.3081336
Correctidentificationofprescriptionpillsbasedontheir visual
appearance is a key step required to assure patient safety and
facilitate more effective patient care. With the availability of high-
quality cameras and computational power on smartphones, it is
possible and helpful to identify unknown prescription pills using
smartphones. Towards this goal, in 2016, the U.S. National Library of
Medicine (NLM) of the National Institutes of Health (NIH)
announced a nationwide competition, calling for the creation of a
mobile vision system that can recognize pills
automatically from a mobile phone picture under unconstrained
real-worldsettings.

Mobile DNNs
Whetheroncustomplatform
built ontopofNvidiaJetson
TX1oronasmartphone
(SamsungGalaxyS7)
MobileDeepPill:ASmall-FootprintMobileDeepLearning
SystemforRecognizingUnconstrained PillImages
XiaoZeng, KaiCao, MiZhang
MobiSys'17 Proceedingsofthe 15thAnnual International Conference on Mobile
Systems, Applications, and Services
https://doi.org/10.1145/3081333.3081336
MobileDeepPill only requires 34MB runtime memory to
run the multi-CNNs model and is able to perform low-power,
near real-time pill image recognition on commodity
smartphoneswithoutcloudoffloading.

Optimizing for
Embedded
use
ArduinoUno-level
computingcapability
on£1-10 IoTMCUs,wireless
sensornetworks,etc.

Embedded
DNNs
ArduinoUno-level
computingcapability
This paper develops a novel tree-based algorithm,
called Bonsai, which can be trained on a laptop,
or the cloud, and can then be shipped onto
severely resource constrained Internet of Things
(IoT) devices.
The Arduino Uno board has an 8 bit ATmega328P
microcontroller operating at 16 MHz with 2 KB SRAM
and 32 KB read-only flash memory. The BBC
Micro:Bit has a 32 bit ARM Cortex M0
microcontroller operating at 16 MHz with 16 KB
SRAM and 256 KB read-only flash. Neither provides
hardware support for floating point operations.
e.g. Femtoduino has a PCB footprint of 20.7 x 15.2
mm (0.81" × 0.6"), DFRobot Beetle 20 x 22mm
(ATmega32u4), and duinoμ ($18) just 12 x 12 mm
(ATMEGA32U4)
Resource-efficientMachineLearningin2KB
RAMfortheInternetofThings
AshishKumar, Saurabh Goyal, Manik Varma
Proceedings of the34th International Conference on Machine
Learning, PMLR70:1935-1944, 2017.
http://proceedings.mlr.press/v70/kumar17a.html
http://www.manikvarma.org/code/Bonsai/download.html and is
part ofMicrosoft’sELL machinelearningcompiler for IoTdevices
Prediction costs per test point on the Arduino Uno with the highest accuracy model of
sizelessthan 2KB– The BonsaiOptmodelwasamoreefficientimplementationof the
chosen Bonsai model. BonsaiOpt was significantly more accurate, faster and energy-
efficient than all other methods. Transmitting the test feature vector to thecloud, whenever
possible, and running uncompressed GBDT might sometimes yield higher accuracies
but would also consume 47x-497x more energy which might not be feasible in many
IoTapplications.
This paper proposed an alternative IoT
paradigm, centric to the device rather than the
cloud, where ML models run on tiny IoT
devices without necessarily connecting to the
cloud thereby engendering local decision making
capabilities. The Bonsai tree learner was developed
towards this end and demonstrated to be fast,
accurate, compact and energy-efficient at
predictiontime.
Bonsai was deployed on the Arduino Uno board
(ATmega328P MCU) as it could fit in a few KB of flash,
required only 70 bytes of writable memory for
binary classification and 500 bytes for a 62
class problem, handled streaming features and
made predictions in milliseconds taking only
milliJoulesofenergy.
Bonsai is also shown to generalize to other
resource constrained settings beyond IoT
by generating significantly better search results as
compared to Bing’s L3 ranker when the model size
isrestrictedto300bytes.
duinoμ

Embedded
DNNs
’Intelligent’sensingwithin-
device datasimplificationto
reduce databandwidth?
Unlesstransmittingalldatais
more energyefficientthan
simplisticedge computing Arduino-based“CircadianLightLogger”
witha Bluetooth Low Energy 4.0(LE)radio link, byPetteri Teikari
https://www.slideshare.net/PetteriTeikariPhD/arduinobased-circadian-light-logger
Bluetooth5Technologyfor theInternet of Things
ByEuropeanEditors Contributed ByDigi-Key's European Editors 2017-03-16
https://www.digikey.com/en/articles/techzone/2017/mar/bluetooth-5-technology-f
or-the-internet-of-things?dclid=COOcisn0-NYCFecx0wod0IgLnQ
nRF52development board
from NordicSemiconductor
How Willthe Internet of MedicalThingsChange Healthcare?
BluetoothLow Energyfor WirelessHealth, Oct 20, 2016
http://www.electronicdesign.com/iot/how-will-internet-medical-things
-change-healthcare
WearableDevicesinMedicalInternet of Things:Scientific
ResearchandCommerciallyAvailableDevices
MostafaHaghi, Kerstin Thurow, and ReginaStoll
Healthc Inform Res. 2017 Jan;23(1):4–15. doi: 10.4258/hir.2017.23.1.4
Smartclothingin communication with outside world.
Designandimplementationof astandardised framework
forthemanagement of a wirelessbodynetworkinan
MobileHealthenvironment
Alejandro Talaminos ; David Naranjo ; Gerardo Barbarov ; LauraM. Roa ; JavierReina-Tosina
Healthcare Technology Letters(Volume:4, Issue:3, 62017 )
https://doi.org/10.1049/htl.2016.0101

Raspberry Pi-
type DNNs
Low-powered and
low-cost ARM-based
“credit card”
computers
Raspberry Pi: Deep learning objectdetection with OpenCV
by AdrianRosebrock on October16, 2017 in DeepLearning, OpenCV3, RaspberryPi
https://www.pyimagesearch.com/2017/10/16/raspberry-pi-deep-learning-object-detection-with-opencv/
https://youtu.be/O7fjwRQn-Zg
Even when applying our optimizedOpenCV +
Raspberry Pi install the Pi is only capable
of getting up to ~0.9 frames per
second when applying deep learning forobject
detection withPython andOpenCV.
Isthatfastenough?
Well,thatdependson yourapplication.
VC4CL(VideoCore IV OpenCL)
https://www.raspberrypi.org/forums/viewtopic.php?t=194952
https://github.com/doe300/VC4CL
The VC4C compiler supports compilation of OpenCL C source-code,
LLVM-IR intermediate code as well as SPIR-V via the corresponding
front-end and can use standard LLVM as well as KhronosSPIRV-LLVM
as front-end compiler. The VC4CL library can also be used with the
KhronosICD loader.

Raspberry Pi
as an independent
imaging device
With intelligent edge
computing before
cloud transmission,
or for totally
independent operation
Layingthe foundation to use RaspberryPi3 V2 camera
module imagery for scientific andengineeringpurposes
Mary A.Pagnutti; RobertE.Ryan; GeorgeJ.Cazenavette; MaxwellJ.Gold; Ryan Harlan; EdwardLeggett; JamesF.Pagnutti;
InnovativeImaging&ResearchCorp.(UnitedStates)
J.ofElectronicImaging,26(1),013014(2017).doi:10.1117/1.JEI.26.1.013014
The Raspberry Pi credit-card-sized
computer supports several
accessories, including a camera
module containing the Sony IMX219
sensor. This computer and camera
configuration is of particular interest
since it can provide raw-data
format imagery thatcan beusedfora
multitude of applications, including
computer vision, biophotonics,
medical testing, patient-centric
healthcare with IoT, remote sensing,
astronomy, improved image quality,
high dynamic range (HDR) imaging,
and security monitoring. This paper
evaluates the characteristics of the
Raspberry Pi V2.1 camera based on
the Sony IMX219 sensor and the
radiometric performance of its raw-
data format imagery, so the system
can be effectively used for scientific
imagingandengineeringpurposes.
Turning the camera on and taking images can cause changes in output due to sensor
warming. The plots show that after approximately a 200 frame warm-up period, data values
reachsteadystate.
RaspberryPi camera mean-variancecurvesfor
ISO100, 200, and 400.DN = digital number (as
in pixel intensity)
RaspberryPiCameraV2
spectral response.

Miniature
Sensor
Revolution
Smallerand cheaper
sensorsfor mobile platforms
coupled withdeeplearning
MEMS Micro-Electro-
MechanicalSystems

MEMS
Introduction
Microelectromechanicalsystems(MEMS): fabrication,designandapplications
Jack W Judy(26 November 2001)Smart Materialsand Structures, Volume 10, Number 6
https://doi.org/10.1088/0964-1726/10/6/301 |Cited by628
Micro-opticaldesign ofathree-dimensional microlensscanner for vertically
integratedmicro-opto-electro-mechanicalsystems MaciejBaranski et al.(2015)
Applied OpticsVol. 54, Issue 22,pp.6924-6934(2015)
https://doi.org/10.1364/AO.54.006924
MEMS-basedhandheldscanning probewithpre-shapedinputsignalsfor
distortion-freeimagesinGabor-domain optical coherence microscopy Andrea
Cogliati etal. (2016)
Optics Express Vol. 24, Issue 12, pp. 13365-13374 (2016)
https://doi.org/10.1364/OE.24.013365
Design, simulation and3Dprinting ofcomplex micro-opticsfor imaging Thiele et al.
(2016)
Optical MEMS and Nanophotonics(OMN), 2016
https://doi.org/10.1109/OMN.2016.7565887
MEMS-tunable VCSELsusing 2D high-contrastgratings Qiao et al. (2017) | Optics LettersVol.
42, Issue 4, pp. 823-826(2017)
https://doi.org/10.1364/OL.42.000823
In-Plane OpticalBeamCollimation Usinga Three-Dimensional CurvedMEMS
MirrorYasser M. Sabry et al. (2017)
Micromachines 2017, 8(5), 134; doi: 10.3390/mi8050134
Two novel MEMSactuator systemsfor self-alignedintegrated3Doptical
coherenttomography scannersJovic et al. (2017)
Micro Electro Mechanical Systems(MEMS), 2017IEEE
https://doi.org/10.1109/MEMSYS.2017.7863528
Wide-angle structuredlightwithascanningMEMSmirrorin liquid
Xiaoyang Zhang et al. (2016) |OpticsExpressVol.24, Issue 4, pp. 3479-3487(2016)
https://doi.org/10.1364/OE.24.003479
ITO-free3DMEMS photodetectorfor point-of-care biosensingdevicess Mirzaet al.
(2017) |https://doi.org/10.1109/HIC.2016.7797729
Healthcare Innovation Point-Of-Care TechnologiesConference (HI-POCT), 2016 IEEE
Handheldultrahighspeedsweptsource optical coherence tomography
instrumentusing aMEMS scanningmirror Lu et al.(2014)
Biomedical OpticsExpressVol. 5, Issue 1, pp. 293-311 (2014) https://doi.org/10.1364/BOE.5.000293
Optical MEMS: FromMicromirrorsto Complex Systems
Olav Solgaard et al. (2014) |Journal ofMicroelectromechanical Systems( Volume: 23,Issue: 3, June 2014 )
https://doi.org/10.1109/JMEMS.2014.2319266
Cogliati et al. (2016)
Thiele et al. (2016)
(a) Optical design example with
ZEMAX of a two-lens system with
30° field of view. (b) Structural design.
(c) Light microscope image of the 3D
printed device.
Jovic et al. (2017)
Opticalmicro iris Opticalmicro shutter Four-colorreal-time
sequencing platform
Reconfigurable
microfluidicliquid
lens
Micromachined
confocal scanning
microscope
Solgaard et al. (2014)

MEMS mirrors
for Optical
Coherence
Tomography
#1
Compact OCT endomicroscopiccatheter
usingflip-chip bondedLissajous scanned
electrothermalMEMSfiberscanner
Yeong-HyeonSeo; Kyungmin Hwang;Ki-HunJeong
MicroElectroMechanicalSystems(MEMS), 2017IEEE
DOI: 10.1109/MEMSYS.2017.7863457
(a) Flip-chip mounted MEMS fiber
scanner and endomicroscopic catheter
assembly parts. A 20 mm long optical fiber is
attached to the flip-chip mounted
microactuator. (b) Fully packaged
endomicroscopic catheter. The inner
diameter of the flip-chip bonded MEMS fiber
scanner is only 1.3 mm and the OCT
endomicroscopic catheter was precisely
assembled with a 1.65 mm diameter
stainless tube and 1 mm diameter GRINlens
(scale bar: 5mm).
(a) Spectral-domain OCT system diagram
with compact OCT endomicroscopic catheter.
The FPGA system was also implemented to
achieve real-time 2D OCT imaging. (b) Two-
dimensional SD-OCT image of a finger nail.
Two-dimensional OCT images are
successfully obtained within 16 ܸVpp
operation
voltage.
(a) Compact OCT endomicroscopic catheter
with Lissajous scanned electrothermal MEMS
fiber scanner. The MEMS fiber scanner was
designed for flip-chip bonding, and integrated
with printed-circuit board. MEMS fiber scanner
was fully packaged with 1.65 mm diameter. (b)
Working principle of the Lissajous scanned
electrothermal MEMS fiber scanner.
Handheldultrahighspeedsweptsource
opticalcoherencetomography instrument
usingaMEMSscanningmirror
Chen D. Luet al.(2014)
Biomedical OpticsExpressVol. 5, Issue 1, pp. 293-311 (2014)
https://doi.org/10.1364/BOE.5.000293
Handheld OCT instrument
internaloptical layoutand
unfolded optical components
showing the(A)OCT 1060nm
optical path, (B)iriscamera
visibleoptical path, and (C)
fixation targetvisibleoptical
path.
Recentadvances inMEMS-VCSELs for
highperformancestructuraland
functionalSS-OCTimaging
Jayaramanet al. (2014)
MIT, Dept of Electrical Engineeringand Computer Science
http://hdl.handle.net/1721.1/100204
MEMS-VCSELdevicestructuresat 1300nm
(A) and 1060nm(B). Bothdevicesemploya
fullyoxidizedGaAs/AlxOybottom mirror, multi-
quantum well gain region andsuspended top
mirror.
Whole eye image obtained with 1060nm MEMS-VCSEL,
illustrating imaging of the anterior eye and retina in a
single acquisition (~40mm imaging range), enabled by the
long coherence length of the VCSEL3 . Axial eye length
measurements using this long-range OCT have compared
favorablywithclinicalopticalandultrasound biometers.

MEMS mirrors
for Optical
Coherence
Tomography
#2
25MostInteresting
MedicalMEMSand
SensorsProjects
MEMSJournal,Inc.,MikePinelis
https://www.slideshare.net/MikePinelisPhD/
140804-25-most-interesting-medical-mems-se
nsors
https://www.slideshare.net/gayathripv1995/mems-based-opti
cal-coherence-tomography-imaging An in-vivo cancer diagnostic procedure using a MEMS based OCT optical probe is much faster
and less traumatic than a conventional white light endoscopic biopsy. It offers significant
improvement in monitoring, screening, and remote digital diagnosis of patients impacting clinical
management.
IME researchers are developing a miniature MEMS optical probe.
Integration of this probe with Optical Coherence Tomography (OCT)
provides optical biopsy at the location of malignant tissue. The 3D
scanning MEMS micromirror makes it possible to scan a large localized
area and adds much needed multiple optical biopsy capability to
minimize sampling errors.
www.a-star.edu.sg

Systems-on-
Chip (SoC)
with MEMS
and ICs
integrated
IntegratingMEMSandICs
AndreasC. Fischeretal.(2015)Microsystems&
Nanoengineering1,Articlenumber: 15005 (2015)
doi: 10.1038/micronano.2015.5
Wafer-LevelVacuumPackagingofSmart
Sensors
AllanHiltonandDorotaS.Temple
Sensors2016,16(11),1819; doi: 10.3390/s16111819
(a) Examples of multi-chip architecture in which vacuum-packaged
microelectromechanical systems (MEMS) sensors are connected to
complementary metal-oxide semiconductor (CMOS) integrated circuits
(ICs) through the package substrate. Includes both two-dimensional (2D)
and three-dimensional (3D) integration examples; (b) Example of
system-on-chip (SoC) architecture in which wafer-level vacuum
packaging (WLVP) is used to place MEMS and CMOS in the same
package. Photograph of 2D multi-chip package reprinted with
permissionfrom ColibrysLtd, (Yverdon-les Bains, Switzerland).
Systemonachip usingintegrated
MEMS and CMOSdevices US9440846B2
OriginalAssignee: Mcube, Inc.
https://www.google.com/patents/US9440846

MEMS for
neuroscience and
neuromorphic
computation
State-of-the-art MEMSandmicrosystemtoolsfor
brainresearch
JohnP.Seymour, FanWu,KensallD. Wise&EuisikYoon
Microsystems&Nanoengineering3,Articlenumber: 16066(2017)
doi: 10.1038/micronano.2016.66
Recording and stimulating technologies vary across scale and
degrees of invasiveness. (a) Illustration of the rodent brain and a
variety of technologies from electroencephalogram (EEG) to intracortical
microelectrodes. (b) High-density systems will increasingly require built-in
active electronics to serialize large data streams and reducethesize of the
connectors. Sample electrical signals show the amplitudes of various
signal sources. The intracortical arrays are often microelectrodes but may
also include chemical and optical sensors. (c) Polyimide
electrocorticogram (ECoG) for large area mapping67. (d) A “Utah array”
with 400 μm shank spacing and 100 channels has been used in human
studies50. (e) Close-packed recording sites with 9×9 μm area and a pitch
of 11 μm178. (f) MicroLED optoelectrode made from GaN on silicon176. (g)
Parylene ECoG with greatly improved resolution over EEG and even
single-cell capabilities23. (h) CMOS integration on probe shaft and
backend40. (i) Fluidic probe for drug delivery45. (j) Active 3D silicon
recording systemwith flexibleparyleneinterconnect182.
Example optoelectrodes with integrated waveguides: (a–c) Laser diode coupled waveguide
probe demonstrating diode directly mounted on neural probe165; (d) a digital micromirror
directing multi-color light into waveguides terminated with metal-coated corner mirrors171; (e)
single waveguide on a silicon recording array162; (f and g) schematic of multi-color laser
diodes coupled from a PCB using graded-index lenses and mixed on the neural probe and
micrograph of an actual device166; and (h) a 4x4 ZnO arraydemonstratinga very similar form
factor as the Utah array with the added capability of optical stimulation through the ZnO tine
and ITO-coated tip109.
Fiberless optical stimulation using μLEDs. (a) GaN μLEDs grown on sapphire wafers and
transferred onto a polymer substrate by laser-liftoff achieved 50×50 μm2 μLEDs175. (b) First
demonstration of monolithic integration of multiple GaN μLEDs on silicon neural probes and
capable of a 50 μm pitch. Scale=15 μm. (c) In vivo demonstration of same optoelectrode
controlling pyramidal cells (PYR)indistinctparts of the CA1 pyramidal cell layer176.

MEMS
Market
MEMSMarkettoTop$22billionby2018
11/8/2013https://www.eetimes.com/document.asp?doc_id=1320035
Micro-Electro-MechanicalSystems
(MEMS)MarketGrowingataCAGRof
9.8% During2017to2022
May23, 2017| www.prnewswire.com/news-releases
The MEMS market for the healthcare vertical is expected to grow at the
highest CAGR during the forecast period. This growth is attributed to the high
adoption of MEMS for clinical monitoring applications such as ECG patient
monitoring and EEG measurement; and imaging applications such as CT-
imaging and digital X-ray.
Moreover, MEMS are being used for diagnostic and treatment equipment
positioning applications, which include high precision positioning of equipment
such as surgery tables, and prostheses and patient monitoring applications such
as movementandposition monitoring.
BioMEMS versusConsumerMEMS – consumerhealthcareand connected healthcare Yolé
Développement (Villeurbanne, France) https://www.linkedin.com/company/417127/
MedicalMEMSandSensors2017
Themedical
MEMSand
sensor market
sizeiscurrently
approximately
$3.2billion

MEMS
Commercial
Devices
Si-Ware's complete handheld spectrometer module is
sold under the brand name Neospectra and takes the
placeof a bulkyAC-powered bench top device.
Spectrophotometers
1st MEMSSpectrometer Debuts
https://www.eetimes.com/document.asp?doc_id=1325630
Smartphone Spectrophotometers
NeoSpectraMicroin Action
https://youtu.be/N_Pj5M25_Ws
ShowcasedatSPIEPhotonicsWest2017,hereis
NeoSpectraMicrodesignedintotheXPNDBLS(
http://www.goxti.com)iPhonecasewithafood
materialstestingappdevelopedbyGreenTropism(
http://www.greentropism.com).Together theybring
youthefirstever spectroscopyenabled
smartphone.

MEMS
Commercial
Devices
Hyperspectral Filters https://www.eetimes.com/document.asp?doc_id=1325630
Eyetracking Theintuitive interface for Human-Computer Interaction in headsets
http://www.adhawkmicrosystems.com/eye-tracking/
Eye tracking MEMS from Adhawk Microsystems
Enablingfoveatedrendering;andallowing alsofinger andheadtrackingalongwithprojectedkeyboard,at1,000fps
https://youtu.be/P0uucoUV_fI
VTT'sMEMS hyperspectral iPhone demo
https://www.youtube.com/watch?v=1qWKPQMUEFM
VTTFabry-Perotinterferometertechnologies
https://youtu.be/JS6JECf72QY

MEMS →
NEMS
MicrotoNano-scale
Moreon researchlabs
atthismomentwith
lesscommercialization
yet
MicrotubularNEMSforon-andoff-
chipbiosensingand-medical
applicationsOliver G.Schmidt
NanoBioSensorsConference2017,PDF
Alarge-scaleNEMSlight-emitting
arraybasedonCVDgraphenes
Hyungsik Kim et al. (2017) | ProceedingsVolume
10126, Advancesin DisplayTechnologiesVII;
101260G(2017)doi: 10.1117/12.2251017
Mechanicalandoptical
nanodevicesinsingle-crystal
quartzYoung-IkSohn, Rachel Miller, Vivek
Venkataraman, MarkoLončar (Submitted on 10
Oct 2017)https://arxiv.org/abs/1710.03372
Ultra-sensitivephotonsensor
based onself-assembled
nanoparticleplasmonicmembrane
resonatorXinghuaWang ;  KaeJyeSi ;  Jiong Yang ;
Xuezhong Wu ;  Qinghua Qin ;  Wenlong Cheng ;  YueruiLu
Micro Electro MechanicalSystems(MEMS), 2016 IEEE
https://doi.org/10.1109/MEMSYS.2016.7421816
ControllingStictioninNano-
Electro-MechanicalSystemsUsing
LiquidCrystalsOleksandr Buchnevetal.
ACS Nano,2016,10 (12), pp 11519–11524
DOI: 10.1021/acsnano.6b07495
NanoelectromechanicalSystems
H.G.Craighead
School of Applied and Engineering Physics, Cornell University
Science  24Nov2000:Vol.290, Issue5496,pp. 1532-1535
DOI: 10.1126/science.290.5496.1532 | Cited by1593 articles
Nanoelectromechanicalsystems: Nanodevicemotionat
microwavefrequenciesXueMing Henry Huang etal. (2003)
Nature 421,496(30 January2003)| doi: 10.1038/421496a| Cited by691
https://www.photonics.com/Article.aspx?AID=17533

Retinal
Imaging
Basics and
image
Enhancement
Enhancement typically
refers to ‘enhancement’
for human visual system
(i.e. the nicest for the
clinician to see)

Image enhancement
with ‘old school’
computer vision
techniques
Typicallythegreenchannelischosenfor
analysis(vasculatureshowsclearly),andred
channel isquitelow-contrastwithopticdisk
saturated. In thiscaseonecouldarguethat
theopticdisklooksthebest foranalysisat
least on bluechannel.
Balazs Thesis (2015):
“A well-established agreementisthat the green channel
in the RGB color space providesmore blood vessel
structural information and islesssubjecttonon-uniform
illumination, whilethe HSVcolor spacedoesnot preserve
the fidelityofretinal images. Becausegreen light is
absorbed bythe blood and reflected bythe retinal
pigment epithelium, providingagood contrast for
visualizingretinal vascular network, bleedingand
exudation, we routinely extract and enhancethe green
channel (in the grayrange of [0, 1]) from aRGB color
fundusphotograph”
Normal fundus photographs of the right eye https://en.wikipedia.org/wiki/Fundus_photography
“Raw”RGB Image RedChannel Green Channel Blue Channel
DenoisedGreen (BM3D)
Normalized for display
Residualnoiseremoved
Normalized for display
CLAHE for rawGreen
channel NotethatCLAHE
accentuatesJPEGartifacts
(and noiseingeneral even
thoughthegivenfundus
imagewasquitegood
quality)
CLAHE for denoised
Greenchannel Nowwith
denoising aspre-processing
step, theCLAHE produces
visually morepleasing result.
And wecan hopethat the
denoising wasgood,and did
notremoveany(pathological)
featuresand onlynoiseand
artifacts

Color Space
CIELab makes senseat least
when processing for human
observers (e.g.NamandKim2017;Dengetal.2017;
Zhuetal.2017)
, also for computers?
The international standards agency has formulated
an alternative color space (CIELab and CIELuv)
with the explicit goal of making distances in the
color space correspond to human judgments of
color similarity. CIELab, roughly, is formed by an oval
with two axes: a and b, which correspond to the
"opponent" colors of Yellow-Blue and Red-
Green. The third dimension of L*a*b* space is
lightness, which isapproximatelyself-describing.
OriginalRGBFundusImage CIELABimage
Rajput and Patil (2014), https://doi.org/10.1109/ICSIP.2014.25
Systematicevaluation ofconvolutionneural
networkadvancesontheImagenet
DmytroMishkin; NikolaySergievskiy; Jiri Matas
ComputerVision andImageUnderstanding
Volume161,August2017,Pages11-19
https://doi.org/10.1016/j.cviu.2017.05.007
Performanceofusing
variouscolorspaces
(RGB,HSV,YcrCb)
andpre-processing.
NotethatCIELabwas
notinthecomparison
Vision-based,real-timeretinal
imagequalityassessmentHerbert
Daviset al. (2009)
Proceedings of the 22nd IEEE International Symposiumon Computer-Based
MedicalSystems(CBMS), pp. 1–6
https://doi.org/10.1109/CBMS.2009.5255437 - Cited by 47
https://github.com/Kavitha-G/Retinal-Image-Quality-Assessment
This paper presents a method that is based on computationally simple
features. For each color channel in the RGB and CIELab image, a total
of N = 17 features were calculated. The features were coded for the
CIELabspace.
← Note that the “b” or yellow contrast appears five times in the top ten
weighted features and contributes to 48% of the overall weight. “L” or
luminescencecontributes17%and“a”or magentacontrastis35%.

Retinal Image Enhancement
Using Robust Inverse Diffusion
Equation and Self-Similarity
Filtering
Lu Wanget al. (2016)
https://doi.org/10.1371/journal.pone.0158480
Image enhancement
with ‘old school’
computer vision
techniques
Many different methods have been put forth for retinal image denoising and enhancement [Patton et al. 2006, Abràmoff et al. 2010, Quellec et al.2011]
,
such as the Gamma transformation, histogram equalization [Reza 2004, Sepasian et al.2008]
, sharpening by the Laplacian operation [
Xue et al.2014]
, filtering methods in transformation fields [Quellec et al. 2011]
, variational methods and partial differential equations (PDEs) [
Yu et al. 2012, Fu et al. 2011, Rampal et al. 2013]
. However, one of major challenges faced by these methods is, how to avoid enhancing noise,
producing overshoot artifactsaroundedges, and erasingfinedetails in enhancedimages [Fu et al.2011,Chang et al.2015]
.
In order to accurately detect and evaluate diabetic retinopathy as soon as possible, it is very crucial to properly enhance
possible retinal pathological features such as microaneurysm, bleeding and exudating spots. In this paper, a
robust inverse diffusion equation is presented, which combines a powerful self-similarity filtering [Buades et al.2005, Dabov et al.2007]
for
detail preserving image denoising. A flux corrected transport (FCT) technique [Fu et al. 2010]
is used to control diffusion flux
adaptively,whicheffectivelyeliminatesovershootsinherentin theLaplacian operation..
original image resultsbyGamma manipulation
self-similarity filtering
NL-meansBuadeset al.2005
robustinversediffusion
original image resultsby Gammamanipulation
self-similarity filtering
NL-meansBuadeset al.2005
robustinverse diffusion
Retinalimageenhancementfordetectionof
microaneurysms
Retinalimageenhancementfordetection
ofsoftexudationsindiabeticretinopathy

DeepBilateralLearning
forReal-TimeImage
Enhancement
MichaëlGharbi, JiawenChen,
JonathanT.Barron, SamuelW.Hasinoff,
FrédoDurandMITCSAIL, Google Research, MIT
CSAIL / Inria, Université Côted’Azur
(Submittedon10Jul2017)
https://github.com/mgharbi/hdrnet
https://groups.csail.mit.edu/graphics/hdrnet/
Image enhancement
with deep learning
better in this
application as well
Our novel neural network architecture can reproduce
sophisticated image enhancements with inference running in
real time at full HD resolution on mobile devices. It can not only
be used to dramatically accelerate reference implementations,
but can also learn subjective effects from human retouching
(“copycat”filter).
By performing most of its computation within a bilateral grid
andbypredicting localaffinecolor transforms,our modelisable
to strike the right balance between expressivity and speed. To
build this model we have introduced two new layers: a data-
dependentlookupthatenablesslicingintothebilateralgrid,and
a multiplicative operation for affine transformation. By
training in an end-to-end fashion and optimizing our loss
function at full resolution (despite most of our network being at
a heavily reduced resolution), our model is capable of learning
full-resolutionandnon-scale-invarianteffects. https://youtu.be/GAe0qKKQY_I

Computing
task
examples
on the edge

Image
Quality
HerbertDavisetal. (2009): “Real-time medical image quality is a critical requirement in a number of
healthcare environments, including ophthalmology where studies suffer loss of data due to unusable
(ungradeable) retinal images. Several published reports indicate that from 10% to 15% of images are
rejected fromstudiesduetoimagequality [e.g.for fundusZimmer-GallerandZeimer (2006)].“
Retinalimagequalityassessment,
erroridentificationandautomatic
qualitycorrection
UnitedStatesPatent9779492,10/03/2017
http://www.freepatentsonline.com/9779492.html
PrejaasTewarieetal. (2012): [OSCAR-IB] “The total
number of rejected OCT scans from the pooled
prospective validation set was high (42%–43%) in
each of the readers. The proportion of rejected OCT
scans for each of the “OSCAR IB” criteria showed that the
rejected scans frequently failed on more than one single
criterion.“
Schipplingieetal. (2014): [OSCAR-IB for multiple sclerosis] “An expert task force convened with the aim to
provide guidance on the use of validated quality control (QC) criteria for the use of OCT in MS research and
clinical trials. Substantial agreement for QC assessment was achieved with aid of the OSCAR-IB criteria. The
task force has developed a website for free online training and QC certification. The criteria may prove
useful for future research and trials in MS using OCT as a secondary outcome measure in a multi-centre
setting.”
BiancaS.Gerendasetal.(2017): “Of 629 patients, eight were excluded due to poor image quality(majority of
B-scans missing or very low signal to noise ratio) at baseline. All remaining OCT scans (621 patients, 1863 OCT
volumes)wereprocessedandanalyzedusing automatedsegmentation.

Retinal
Imaging
Automatic
qualitycontrol
Sothatthe anyonecan
take agood picture with
your system, anddo not
need specialtraining for
it. Gettingthe User
Experiencerightfor the
operator as well
COLOR
FOCUS
CONTRAST
ILLUMINATION
CAMERA Artifacts
Common fundus quality problems (JoãoDias' Master'sthesis)
Common OCT quality problems
Poorfocusing Retinal pathology Signal strength
BeamplacementDecentrationBadillumination
Segmentation Algorithm failure
The OSCAR-IB ConsensusCriteria forRetinal OCT Quality
Assessment http://dx.doi.org/10.1371/journal.pone.0034823

Retinal
Imaging
Commercial
playershave
alreadyadopted
automatic
qualitycontrol
BrunoL.B.Esporcatteetal.(2017): Subjects underwent ocular
imaging with dilated pupils using an optical coherence tomograph,
the Spectralis™ OCT (software version 5.1.3; Heidelberg
Engineering). The examiner was required to manually center the
scan on the optic disc. To increase the image quality, the device
included an automatic real-time function that gathered
multipleframes,andimageswereaveragedtoreducenoise.
http://www.medelsis.com.tr/en/heidelberg_spect_oct.html
Image Quality Assessment of FundusImages
Using Deep Convolutional Neural Networks
with Extremely FewParameters
Christian Wojek; Keyur Ranipa;abhishekrawat;ThomasMilde;Alexander
Freytag| Corporate Research, Carl ZeissAG
ARVOAnnual MeetingAbstract | June 2017
http://iovs.arvojournals.org/article.aspx?articleid=2637978
We present a solution for automated IQ assessment of fundus
images taken by a hand-held fundus camera VISUSCOUT100. Our
results surpass 99.8% AUC even with a tiny CNN that directly allows
forin-fieldapplicationswithlimitedhardware.

Mechanical
Considerations
Reducemotion
artifacts,andmake
the acquisitionas
comfortableas
possible forthe
subject
Focus-tunableand Monovision
Near-eyeDisplays| SIGCHI 2016
StanfordComputational ImagingLab
http://www.computationalimaging.org/publications
We develop an optical system based on focus-tunable optics and evaluate
three display modes (conventional, dynamic-focus, and monovision) with
respectto thevergence-accommodation conflict.
pupil labs
Pupil Headsets are plug and play USB devices
carefully designed to be lightweight, unobtrusive, and
easy to use. The hardware is strong and flexible. The 3d
printed frame holds research grade cameras that
record your field of view and your eye movements. Our
versatile and modular system is ready to be adapted to
the specificities of your research or application
requirements. https://pupil-labs.com/pupil/
HTCViveEyeTrackingAddOn
Microsoft HololensBinocular Add-on
Epson Moverio BT-300 Add-on
Oculus RiftDK2add-on
Pupil in Google Cardboard
Connect yourPupil headsetto your Android deviceand streamvideo
over WiFi and record video on yourAndroid
https://pupil-labs.com/blog/2017-08/introducing-pupil-mobile/
2016Aug09

Image
Quality
Beyondretinal
imaging
MRIQC: Advancing the automaticprediction
of image quality in MRIfrom unseen sites
Oscar Esteban et al. | Departmentof Psychology, Stanford University
PLOS One (Sept 25, 2017) https://doi.org/10.1371/journal.pone.0184661
Visual reports MRIQC generates one individual report per subject in the input folder and one
group report including all subjects. To visually assess MRI samples, the first step (1) is opening
the group report. This report shows boxplots and strip-plots for each of the IQMs. Looking at the
distribution, it is possible to find images that potentially show low-quality as they are generally
reflected as outliers in one ormore strip-plots.
Summary table ofthe trainandtestdatasets.
Image-quality metrics(IQMs)
Later efforts to develop image-quality metrics (IQMs) appropriate for MRI
include the Quality Assessment Protocol (QAP), and the UK Biobank [
Alfaro-Almagro etal. 2017]. MRIQC extends the list of IQMs from the QAP, which
was constructed from a careful review of the MRI and medical imaging literature [
Shehzad etal.2015].

Retinal
Imaging
Sharpness
Videosofthe Interior oftheEye Capturedby theD EYEPortable
Retinal ImagingSystemhttps://youtu.be/CRncyRPH3kc?t=17s
t1
t1+25
t1+50
Extremely difficult and frustrating to
take a sharp image
Solution: Reconstruct automatically the
sharpest fundus image from video frames
withsome ofthemsharp andsome blurry.
Even harder for miotic pupils (non-dilated pupils,
opposedtomydriaticpupilsthat givebetter images)

Retinal
Imaging
Multiframe
reconstructionalso
fortheverylow-cost
instruments?
Combinewith
medicaleducation?
Low-CostIndirectOphthalmoscope
AUG24,2017 By AaronWang, MD,
RengarajVenkatesh, MBBS, Pavan Kumar GurudattaSr ,
John MAvallone MD, David L Guyton MD
https://www.aao.org/1-minute-video/low-cost-indirect-
ophthalmoscope
Aninexpensivemethodofindirect
ophthalmoscopy
K.R.Bishai| BritishJournal of Ophthalmology, 1989, 73,
235-236. http://dx.doi.org/10.1136/bjo.73.3.235
Usefulnessof StructuredVideoIndirect
Ophthalmoscope—GuidedEducation
inImprovingResidentOphthalmologist
ConfidenceandAbility
OphthalmologyRetinaVolume 1, Issue 4, July–August
2017, Pages282-287
https://doi.org/10.1016/j.oret.2016.12.010
ComparisonStudyof FunduscopicExamof
PediatricPatientsUsingtheD-EYEMethod
andConventionalIndirectOphthalmoscopic
Methods
OpenJournal ofOphthalmology, 2017, 7,145-152
https://doi.org/10.4236/ojoph.2017.73020
Note! This journal is on the “flakyacademicjournals”list (by DH Kaye )as
being published by SCIRP. Also on the Predatory Journal list, so in practice
thisisa promotional articleon D-EYE?

Retinal
Imaging
Nextgeneration
depthsensing
smartphones
PierreCambou, GuillaumeGirardin, Dr. Yohann Tschudi| Yole
Développement
https://www.i-micronews.com/imaging/10000-apple-iphone-
x-unlocking-the-next-decade-with-a-revolution-3.html
https://arstechnica.com/gadgets/2016/12/google-tango-review-promising-google-
tech-debuts-on-crappy-lenovo-hardware/
Next-gen QualcommSpectra ISPsbring moredepth to mobiledevices
http://techreport.com/review/32397/next-gen-qualcomm-spectra-isps-bring-more-
depth-to-mobile-devices
Google Tango and iPhone bringing depth sensing to new generation
smartphones. At least useful for eye tracking and pupillometry applications to track the
distancebetweentheeyeandthecameratocorrectfor z-direction movements

Autofocus method for
automated microscopy
using embedded GPUs
J. M. Castillo-Secilla etal. (2017)
Biomed Opt Express. 2017 Mar 1;8(3):1731–1740.
https://dx.doi.org/10.1364/BOE.8.001731
GPUacceleration usefulfor
embedded imaging even
without thedeep learning
And putting the NVIDIA Jetson
(Tegra SoC) would for sure able
future-proofing your embedded
camera module
With hand-held imaging you
could provide also guidance
where to move the camera
to get the reconstruction
done the quickest
Secondly, to propose an efficient low-cost implementation based on the limited resources of an embedded GPU
System on Chip architecture, able to fulfill the time constraints of an automated optical microscope. To the best of our
knowledge, there is no proposal of an autofocus system for microscopy using multiple images with
acceleration on embedded GPUs. In our work the method was implemented using a commercial SoC NVIDIA Tegra
that includes a quad-core multiprocessor along with just 256 GPU cores and shared memory. By means of parallelizing
acquisitionandprocessing,ourimplementationguaranteevirtuallyzerodelayatmaximumcameraframerate.
Example of a 8-Stack of
images obtained by a
microscope (converted to 8-
bit grayscale) with a stepsize
of 3µm (1-8) and the final
result of multifocus (9). Scale
barsrepresent 30µm.
The results show that our approach
outperforms the three best autofocus
methods in the case of TB identification,
even considering the ideal observer metric
provided by the No-reference Anisotropic
Quality Index (AQI) index. (A MATLAB
toolbox for computing AQI for arbitrary
grayscale or color images is available from
DOI: 10.6084/m9.figshare.4276382.v1).

Auxiliary
Sensing
For motion-
induced artifacts
Correctionformotion
artifacts likeheartbeat
and respitatory
artifacts,likee.g.inMRI
and intwo-photon
microscopy?
Reduction of motion artifactsduring in
vivo two-photon imaging ofbrain through
heartbeat triggered scanning
MartinPaukert, Dwight E. Bergles| JohnsHopkinsUniversity
TheJournal of Physiology(2012), 590: 2955–2963.
doi: 10.1113/jphysiol.2012.228114
ECG-triggeredmotion correction reduces artifacts when imaging throughthethinned skull
Motion artifact and background noise
suppressionon opticalmicroangiography
framesusing anaïve Bayes mask
RobertoReif, Utku Baran, and RuikangK. Wang|Universityof Washington
Applied OpticsVol. 53, Issue 19, pp. 4164-4171 (2014)
https://doi.org/10.1364/AO.53.004164
Optical microangiography(OMAG)isamethodof processing OCTdata
4D respiratory motion-compensated image
reconstruction of free-breathing radial MR
datawith very high undersampling
Christopher M. Rank, ThorstenHeußer, Maria T. A. Buzan, Andreas Wetscherek, Martin T. Freitag,
JulienDinkel, Marc Kachelrieß | Medical PhysicsinRadiology,German CancerResearchCenter(DKFZ),Heidelberg,Germany
Magn. Reson. Med., 77:1170–1183. doi: 10.1002/mrm.26206
To consider sliding
lung motion, the
registration methods
shall be improved.
Moreover, we plan to
further validate the
4D joint MoCo-
HDTV algorithm by
evaluating a larger
number of subjects
within the scope of a
clinical study with
specificpathology.

Auxiliary
Sensing
For motion-
induced artifacts
Low-costeye
movement
tracking
AdHawk has created eye-tracking sensors that are small
chips made from microelectrical mechanical
systems (MEMS), which are commonly used in gyro
chips. AdHawk Microsystems said that its smaller, faster,
more power-efficient motion-tracking solutions will
rendercamera-basedeyetrackingobsolete.
AdHawkMicrosystems (Kitchener, Ontario, Canada)
can capture thousands of data points per second,
enabling a system based on the chips to be able to
predict where a user will look next, leading to
moreimmersiveAR/VRexperiences.
In health care, by measuring the smallest
movements in the eye, the AdHawk system could be
used for early detection of conditions such as
Parkinson’s disease or to understand the
emotionalstateof theuser.
And in training, observers can understand when
you’re tired and not taking in information effectively by
tracking blinkfrequency and eyemovement. This is
especially relevant in use cases like pilot or driver
training.
AdHawk’stiny sensorscould enablemuch smaller VRheadsets
and ARglasses venturebeat.com/2017/10/19|uploadvr.com| https://youtu.be/EHCwIfmNPFo
~1,000 fps

Auxiliary
Sensing
For motion-
induced artifacts
Eyemovement
trackingifyou
haveanadaptive
opticssystem
DynamicPupilTrackingfor an
AdaptiveOpticsSystem
Fan Yi , Michael Collins, Brett Davisand Cael Degnian
QueenslandUniversityofTechnology,Australia
https://eprints.qut.edu.au/105597/
The dynamic pupil tracking function worked at a sampling rate of
30~60 Hz and the image resolution across the pupil varied from
30~60μm/pixel.
Pupil offset is monitored and fed to the AO system in real-time. Running in a close-
loop mode, the spatial light modulator (SLM) of the AO system dynamically tracks
thepupil position and delivers the optical designs to the eye.
Movements of a real eye during a 30 mins viewing task of a fixed Maltese cross target.
Sampling frequency was 10 Hz. Range of movements were within ±0.6 mm in horizontal
and vertical directions. The peaks indicated when blinks happened.
Adaptiveopticsretinalimaging
withautomaticdetectionofthe
pupiland itsboundaryinrealtime
usingShack–Hartmannimages
Albertode Castro, Lucie Sawides, XiaofengQi, and Stephen A.
Burns| Applied OpticsVol. 56, Issue 24, pp. 6748-6754(2017)
https://doi.org/10.1364/AO.56.006748
A spot quality metric combined with neighborhood rules for detecting the
pupil boundary allows detection of the pupil using the Shack-Hartmann
(SH) images and improves the practical control of AO retinal imaging in real
time. … In principle, this may allow the experimenter, and the system
designer, to relax constraints on maintaining head position during
imaging, making images more convenient to obtain and increasing the
subject’scomfort.
Error budget studies show that the limiting factor on the AO systems used
in ophthalmology is not usually the number of lenslets [Evansetal. 2009],
and technological changes are allowing the construction of deformable
mirrors(DMs)with higher numbersofactuators.
Example of the automatic pupil detection algorithm in a 6-mm diameter pupil diabetic
patient with a localized posterior subcapsular cataract. (a) SH image, (b) depending on
its metric, rejected spots are marked in red, boundary spots in blue, and accepted
lenslets withits local slope are marked ingreen;(c) retinal imaging showing arterywalls.

Eye
Movement
Beyondjust
correction for
visualfunction
analysis
Thepupilisfasterthanthecornealreflection(CR):
Arevideobasedpupil-CReyetrackerssuitablefor
studyingdetaileddynamicsofeyemovements?
Ignace Hooge,Kenneth Holmqvist, MarcusNyström
Vision Research Volume 128, November 2016, Pages6-18
https://doi.org/10.1016/j.visres.2016.09.002
We conclude that the pupil-CR technique (with SMI Hi-Speed withPsychoPy at 500
Hz) is not suitable for studying detailed dynamics of eye movements (e.g. saccade
dynamics).
Usingmachinelearningtodetecteventsineye-
trackingdata
Raimondas Zemblys, Diederick C. Niehorster, Oleg Komogortsev, Kenneth Holmqvist
Behavior Research Methodspp 1–22 (23February 2017)
https://doi.org/10.3758/s13428-017-0860-3
We conclude that machine-learning (Random Forest) techniques lead to
superior detection compared to current state-of-the-art event detection
algorithmsand can reachtheperformanceof manual coding.
Holmqvist et al. (2015)
show that for most of the
eye trackers, noise in
the corners of the
screen is up to three
times higher than in
the middle of the
screen.
Eyemovementtestinginclinicalexamination
Harold E. Bedell and Scott B. Stevenson
Vision Research Volume 90, 20 September 2013, Pages32-37
https://doi.org/10.1016/j.visres.2013.02.001
Here we review the clinical uses of eye tracking, with both an historical and
contemporary view. We also consider several new imaging technologies that are
becoming available in clinics and include inbuilt eye-tracking capability. These highly
sensitive eye trackers should be useful for evaluating a variety of subtle, but important,
oculomotor signsand disorders.
At present, the extraction of eye-movement information from both standard-
and AO-SLO devices requires special purpose, custom software. Therefore,
at this point these technologies remain essentially a research tool. The importance of
precise eye tracking for optimizing SLO and OCT image quality suggests that the
requisite software will be bundled with these devices in the future as they
become deployed more widely in clinics. The signals of eye movement that are
extracted as part of the image-registration process therefore are potentially available
tousers.
The current rapid rate of technological improvement all but ensures that sophisticated
retinal imaging and refractive surgical systems, with sophisticated integrated
eye-tracking capabilities, will be deployed more and more commonly in the clinic. It is
worth bearing in mind that, in addition to their principal role as an imaging or surgical
device, these instruments have the capability to be used as a clinical eye
tracker. The manufacturers of these instruments should be encouraged to provide
appropriate interfaces to allow for eye-movement signals to be sampled, stored, and
visualized so that this additional functional capability can be used to the best
clinical advantage.

Retinal
Imaging
Noise &Super-
resolution
Especially OCTs can be very noisy
Solution: Multiframe reconstruction for
reduced imaging noise, and higher
resolution via super-resolution either with
orwithoutwavefront sensor

Statistical model for OCT
image denoising
Muxingzi Li et al. (2017)
https://doi.org/10.1364/BOE.8.003903
https://github.com/vccimaging/OCTDenoising
”The statistical analysis
presented above reveals three
interesting properties of the
speckle noise present in OCT
images:(1) Thisnoiseisstationary
(spatiallyindependent) overtheimage, (2)
thestandarddeviationofthenoiseislinearly
proportional toitsmean,and (3)aftera
square-root transformation,thenoise
followsaGaussiandistribution.Moreover,
theleadingcoefficient dependsonlyonα
thedevice.Itisconstant fordifferent
datasetsobtainedwith thesamedevicebut
takeson differentvaluesfordifferent
devices.”

Denoising methods for
improving automatic
segmentation in OCT
images of human eye
A.Stankiewicz,T.Marciniak,A.Dąbrowski,
M.Stopa, P. Rakowicz,E.Marciniak
TheJournalofPolish AcademyofSciences
Volume65,Issue1(Feb2017)
https://doi.org/10.1515/bpasts-2017-0009
”Precision of the denoising process
was evaluated based on the results
of automated retina layers
segmentation, since this stage (vital
for ophthalmic diagnosis) is strongly
dependent on the image quality.”
Note!The“best”methodofwaveletdenoising
hereisnotreallythe“best”,andthe
comparisoncouldhaveincludedoldschool de
factostandardBM3D/BM4D,andrecentdeep
learningstate-of-theartdenoisers.
Example of 3D OCT scan of human retina with annotated
most important layers. Inner limiting membrane (ILM), nerve
fiber layer (NFL), ganglion cell layer (GCL), inner plexiform
layer (IPL), inner nuclear layer (INL), outer plexiform layer
(OPL), outer nuclear layer (ONL), inner segments of
photoreceptors (IS), outer segments of photoreceptors
(OS), and retinal pigment epithelium (RPE) [
Abràmoffetal. 2010].
Example of 3D OCT retina cross-section (B-scan): (A) B-scan with
expert’s manual segmentation, (B) B-scan with erroneous automatic
segmentation of retina layers (places with erroneous segmentation are
indicated byarrows)
Illustration of manually segmented layers (A) and results of
automatic analysis after noise reduction with selected methods:
(B) average filtering (3x3 mask), (C) median filtering (3x3 mask),
(D) anisotropic diffusion (κ = 20), (E) soft wavelet thresholding (τ
= 10), (F) multiframe wavelet thresholding (k = 1). Data is
presented in a section of the size 220x220 pixels cropped from
the center of the volume.

Matching 3D OCT retina
images into super-
resolution dataset
AgnieszkaStankiewicz ;  Tomasz Marciniak ;
AdamDąbrowski ;  MarcinStopa ;
ElżbietaMarciniak ;  Andrzej Michalski
SignalProcessing:Algorithms,
Architectures,Arrangements,and
Applications(SPA),2016
https://doi.org/10.1109/SPA.2016.7763600
”Our approach is based on multiframe
super-resolution method applied to
several 3D standard resolution OCT
scans. Presented experiments where
performed on volumetric data acquired
from adult patients with the use of
Avanti RTvue device. Each OCT cross-
section (B-scan) was subjected to
image denoising and retinal layers
segmentation. The generated 3D super-
resolution scans have significantly
improved quality of the vertical cross-
sections.”
Reconstruction step (Combining of images) As was
reported before, there are many possible algorithms for
performing the reconstruction step, of which the majority is
based on some variation of the nearest neighbor interpolation.
Following this course of intuitive approach, we investigated the
feasibility of utilizing three most promising k-NN methods for our
purpose.
Replacing the sensor in the OCT device may be very expensive. Thus the
software-based super-resolution technique it is possible to obtain high
resolution data in a fast and inexpensive way. One high-resolution scan is
difficult to acquire for old and pathological eyes, as opposed to several
small ones that take short time and can be acquired in a time period of
several minutes.
The proposed research has a potential to significantly impact the clinicians’
approach to analyze retinal pathologies. The purposefulness of the
described approach was confirmed by a group of ophthalmology
experts. In the future work, we plan to conduct an experimental research
aimed at enhancement of computational efficiency by implementing parallel
computing techniques.

Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning

Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (15)

Ähnlich wie Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning

Ähnlich wie Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning (20)

Mehr von PetteriTeikariPhD

Mehr von PetteriTeikariPhD (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Portable Retinal Imaging and Medical Diagnostics With a Focus on Deep Learning