Clinical decision making with Machine Learning In this document, the author discusses how machine learning can be applied to optimize clinical decision making in in vitro fertilization (IVF). Specifically, the author presents how their clinic collects vast amounts of IVF data and is working to build predictive models using machine learning techniques to personalize treatment protocols and embryo selection. Key areas discussed include using time-lapse imaging, preimplantation genetic testing, and endometrial receptivity analysis to select the best embryos, as well as collecting extensive clinical factors on over 918 single embryo transfer cycles to build a model that can predict IVF outcome with 73% accuracy.

1. Clinical decision making with Machine Learning
Oleksii Barash, Ph.D.
Reproductive Science Center of San Francisco Bay Area

2. Disclosure
We have no financial relationship with any
commercial interest related to the content
of this activity

3. Reproductive Science Center of the SF Bay Area
• Founded in 1983
• In top 30 largest IVF (In Vitro
Fertilization) clinics in USA*
• In top 20 clinics with the best clinical
outcomes*
• Over 2000 treatment cycles (fresh and
frozen) in 2017
* - CDC Report 2015

4. What is infertility?
WHO - Infertility definitions and terminology
• Failure to conceive within 12 months of
regular unprotected intercourse.
• Primary or secondary.
• 84% of couples will conceive within 1 year and
92% within 2 years.

5. Scope of the problem
• Infertility affects 12% of the reproductive age population in
the US (≈12 million people)
• Infertility affects men and women equally
• More than 50% of infertility patients will have a baby with
treatment
• Over 1.5M IVF cycles per year worldwide (≈ 200,000 in USA)
in 2014
• Cost of one IVF cycle in US: 10K – 100K

6. Global fertility Market
Equity Research Reports, 2012
Key growth drivers:
1. Aging and Infertility
2. Increasing prevalence of Obesity
3. Cultural shifts (“Celebrities” and LGBTQ)

7. Unreasonable expectations…
• 59% of childless women aged 35‐39 still planned to
have a baby
• 30% aged 40‐45 did too!
(Sobotka, Austrian survey data)
• 58% said they wanted 2 children (aged 21‐23)
• Only 36% had achieved that by age 36‐38
(Smallwood and Jeffries,UK Population Trends)

8. Biological clock
Speroff, 2004

9. IVF treatment overview

10. IVF is essentially manufacturing
• Complex multidimensional process;
• Constant intake flow of the patients;
• Cutting edge labor and equipment;
• Hundreds of contributing factors (Lab + Clinical);
• Every patient is unique – limited standardization
Ultimate goal –
single healthy baby

11. Manufacturing outcome prediction

12. IVF produces a lot of data?
• Main shareholders are open to
cutting edge technologies
• Wide Electronic Medical Records
adoption;
• IoT devices – sensors, incubators,
microscopes, lasers
• Morpho-kinetics (time-lapse)
• Preimplantation Genetic Testing
• “Omics”

13. Transforming data into knowledge
• Increasing number
of publications
• Retrospective and
small
• Rare RCTs

14. Evidence based medicine
Conscientious, explicit
and judicious use of
current best evidence in
making decisions about
the care of an
individual patient.*
* - Sackett. BMJ 1996;312:311-2

15. Meta-Analysis
Fertility and Sterility 2010; 94:936-945
• Small number of
samples
• Diverse experimental
conditions

16. Personalized decisions to be made in each
IVF cycle
• Hormonal Stimulation protocol / dosage / duration
• Lutheal support
• How many embryos to transfer (1, 2 or 3)
• Embryo selection for the transfer (morphological and
genetic)
• Financial products (risk sharing programs, money back)

17. Do You Know Your Embryo Biology?

18. Time-lapse and Machine learning

19. Embryo selection for the transfer
• From 1 to 30+ embryos per IVF cycle
• Many morphological and kinetic features per embryo
• Critical choice – no second chance

20. Traditional embryo evaluation
M. Montag, 2014

21. Time-lapse monitoring
M. Montag, 2014

22. Non-invasive imaging and predictions

23. EEVA (Early Embryo Viability Assessment)
• Xtend algorithm:
– over 1,000 combinations of potential parameters
– includes egg age, cell count and Post P3 analysis – which measures cell activity after the four
cell stage
– Post P3 is the result of a proprietary analysis based on 74 computer-based attributes that
are combined into one parameter
– each embryo gets a developmental potential score ranging from 1 (highest) to 5 (lowest).
– 84% specificity vs 52% by traditional assessment
– The odds ratio of predicting blastocyst formation is 2.57 vs 1.67 by traditional assessment

24. EEVA (Early Embryo Viability Assessment)

25. Unusual cleavage patterns

26. EEVA Xtend algorithm

27. EEVA Xtend algorithm

28. Preimplantation Genetic Testing

29. Oocyte aneuploidy and maternal age
Handyside, 2015
• All primary oocytes are
formed before baby-
girl is born.

30. Preimplantation Genetic Testing
Handyside, 2015
DNA sequencing
DNA flow cell

31. Preimplantation Genetic screening
National Human Genome Research Institute, 2014
• Log scale!

32. Preimplantation Genetic Testing
National Human Genome Research Institute, 2017
• Log scale!

33. Single Nucleotide Polymorphism (SNP)
algorithm
• 300,000 probs per embryo
• Per chromosome confidence
• Highly accurate and comprehensive results
• Parental genomic information
• Cumulative distribution function (cdf) curves

34. Cumulative live birth rate after SET,
PGS, N=1024
# Cycles Live births Total ETs 1-l/n S(t)
1 178 313 0.43131 0.56869
2 22 59 0.627119 0.72952
3 7 15 0.533333 0.85574
4 1 2 0.5 0.92787
5 1 1 0 1
Presented by RSC team at ASRM 2016

35. Gene expression, stage & multinucleation

36. ML-based solutions for IVF

37. Univfy
Univfy algorithm:
• Takes patient data
• Predictive model
based on 13,000 IVF
cycles;
• Chances for positive
outcome
• Chances of twins if 2
embryos were
transferred

38. Celmatix
Celmatix algorithm:
• Incorporated in our EMR
(ARTworks)
• Software as a service
(SaaS model)
• Data analytics platform to
help optimize patient
management and
counseling

39. Celmatix - Fertilome
Celmatix algorithm:
• 25,000 peer-reviewed studies
• 1,713 genes
• 427 variant/diagnosis combinations
• 201 gene-diagnosis combinations
• 32 target genes in the kit

40. Endometrial Receptivity Analysis (ERA)
by Igenomix
Patented in 2009: PCT/ES 2009/000386
Customized microarray (238 genes)
Bioinformatic analysis of data obtained by the customized microarray
Classification and prediction from gene expression.

41. Endometrial Receptivity Analysis (ERA)
Receptive
Model Classifies the Molecular Receptivity
Status of the Endometrium
Post-ReceptivePre-Receptive

42. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2010 2011 2012 2013 2014 2015 2016
%ofcycles
ETx1
ETx2
ETx3
ETx4
ETx5
～ Average age – 36.0 ± 5.5 y.o.
～ 39.3% of all patients are over 38 y.o.
SET rate in non-PGT cycles
(2010-2016), fresh D5 ET, N=3925

43. Preimplantation Genetic Testing (PGT) at RSC
～ SET frequency in PGS IVF cycles (average age – 37.5 ± 4.29 y.o. ) – 89.9%
FISH SNP – aCGH - NGS
661
1387
4
735
0
200
400
600
800
1000
1200
1400
0
200
400
600
800
1000
1200
1400
1600
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
NumberofIVFcycles
Total volume
PGT cases
78

44. Live
birth
rate
Maternal age
Number of
embryos for
biopsy
Morphology
of the
embryos
SET vs eSET
D5 vs D6
Biopsy
Total
gonadotropin
dosage
Number of
previous
failed cycles
Number of
normal
embryos per
cycle
Number of
eggs
Euploidy rate
Presented by RSC team: ASRM 2016, 2015, 2014; ESHRE 2015, 2016;
PCRS 2014, 2015, 2016; PGDIS 2015, 2017
Factors affecting PGT outcomes

45. Live birth rate
Embryo
_Age
Blastula
tion_rat
e
Donor_
eggs Euploid
y_rate Number
_of_nor
mal
d5_to_t
otal_rat
io Total_d
ay_5_bx
Total_d
ay_6_bx
Total_fo
r_biosy
Bx_Day
Emb_Ex
pansion
ICM
TE
Gender
Best_E
mbryo_
For_ET
Elective
_SET
Cycle_n
umber
Number
_of_Foll
icles
Zygotes
Fert_rat
e
Unfert
M2
M1
GV
ATR
Multi_P
N
PN_1
Degene
rated
Cleaved
Cleavag
e_rate
Number
_ext_cu
ltureGood_e
xt_cultu
reNumber
_to_blNumber
_CryoGood_d
3_rateTVA_M
D
Number
_of_tar
nsfers_t
o_deliv
ery
Semen_
Source
Fresh_F
rosen_s
p
BMI
PATIEN
TTYPET
EXT
NO_OF
_DAYS
SUMSTI
M
ASPIRA
TED_O
OCYTES
HCG_D
RUG
TOTAL2
PN
GRAVID
ITY
PREM
TERM
SAB
BIOCHE
MICAL
LIFETIM
E_SMO
KED
PRIORIV
F
PRIORF
ET
PRIORI
UI
HEIGHT
WEIGHT
PRIMAR
YDIAGN
OSIS
SEMENS
OURCE
FSHLEV
EL
NEARES
T_AMH
MED1
Peak_E
2
TOTALI
US
FOLLICL
ES_BIG
GER_TH
AN_14
ASPIRA
TED_O
OCYTES
NO_FR
OZEN
NO_VIT
INITIAL
CONSUL
T_PREM
INITIAL
CONSUL
T_GRAV
IDITY
INITIAL
CONSUL
T_SAB
INITIAL
CONSUL
T_TERM
INITIAL
CONSUL
T_BIOC
HEMICA
L
Stim
protoco
l
Factors affecting PGT outcomes
More factors?
Bias?
Reproducibility of the
results?

46. Factors affecting PGT outcomes
What if we can evaluate ALL available factors?

47. What if we can assess ALL available factors?
20 factors:
202 = 400 plots
381 factors
3812 = 145,161
plots
20 x 20
Machine Learning

48. Algorithm
Timeframe: Jan 2013 – Jul 2017
Retrospective analysis
Number of PGS transfers: 918
Average age: 35.6 ± 4.8
ONLY Single embryo transfers
Machine learning methods:
• GLM (Generalized Linear Models)
• RPART (Classification and Regression Trees)
• GBM (Generalized Boosted Regression Models)
IVF lab
Embryo_Age
Blastulation_rate
Donor_eggs
Euploidy_rate
Number_of_normal
d5_to_total_ratio
Total_day_5_bx
Total_day_6_bx
Total_for_biopsy
Bx_Day
Embryo_Morphology
Expansion
ICM
TE
Gender
Clinical_Outcome
BEST_ EMBRYO_FOR_ET
ELECTIVE_SET
Number_of_tarnsfers_to_delivery
Biopsy tech
CYCLE #
PEAK E2
TVA MD
TVA TECH
# Follicles >12 mm
# EGGS
# INSEM
# 2PN
% FERT
# UNFERT
#M2 or mature
# INT
# IMM
# ATR
# > 2PN
# 1PN
# DEG
FERT CK TECH
ICSI TECH
SEMEN SOURCE
FRESH/FROZEN SP
CLEAVED
% CLEAVED
HATCH TECH
# EXT CULTURE
# GOOD EXT CULT
# TO BLAST
# CRYO
% OF GOOD QUALITY EMBRYOS
…
clinical
BMI
PRIMARY_DX
PATIENTTYPETEXT
LUPRON
STIM
GNRHA
MED1
SUMSTIM
TRANSFER_DATE
HCG_DRUG
GRAVIDITY
PREM
TERM
SAB
BIOCHEMICAL
PATIENTRACE
LIFETIME_SMOKED
SMOKING_FREQ
PRIORIVF
PRIORFET
PRIORIUI
HEIGHT
WEIGHT
STIMPROTOCOL
LUPRONPROTOCOL
PRIMARYDIAGNOSIS
SECONDARYDIAGNOSIS
TERTIARYDIAGNOSIS
SEMENSOURCE
PATIENTTYPE
FSHLEVEL
E2LEVEL
NEAREST_AMH
AFC
MED1
MED2
MED3
MED4
MAX_E2
TOTALIUS
FERT_METHOD_ICSI
FERT_METHOD_IVF
INITIALCONSULT_PREM
INITIALCONSULT_GRAVIDITY
INITIALCONSULT_SAB
INITIALCONSULT_TERM
INITIALCONSULT_BIOCHEMICAL
Stim protocol
…
381 variables per SET:

49. Lab factors, 918 SETs
Pregnant, %Non-Pregnant, %
% of total SETs
Yes No

50. Lab + Clinical, 918 SETs

51. Relevant feature selection algorithm* (Lab factors)
*Number of CART trees = 100

52. Relevant feature selection algorithm* (Lab + Clinical)
*Number of CART trees = 100

53. Building the model to predict IVF outcome
Only weak predictors are present
Relatively small sample size (10K)
A lot of features (>300)
Accuracy of predictions = 0.73
AUC = 0.76
(Sensitivity/specificity balance)

54. Building the model to predict IVF outcome
(PGT only)
• Benchmark AUC – Starting point
• Feature engineering
• Feature importance
• Feature transformations
• Non-important features
• Model interpretation

55. Building the model to predict IVF outcome
(FETs only)
Relative
Importance
Feature Description
0.95784
403_NumCatTE_Prior full
term_Prior pre-term_TE_0
Out-of-fold mean of the response grouped by: ['Prior full term',
'Prior pre-term', 'TE'] using 5 folds (numeric columns are
bucketed into 25 equally populated bins)
0.55907
164_CV_TE_# EXT
CULTURE_FACNAME_LUPRON_
PGD.1_Retrieval MD_Retrieval
technician_Thawing
technician_0
Out-of-fold mean of the response grouped by: ['# EXT
CULTURE', 'FACNAME', 'LUPRON', 'PGD.1', 'Retrieval MD',
'Retrieval technician', 'Thawing technician'] using 5 folds
0.35233 217_BIOCHEMICAL BIOCHEMICAL (original)

56. Ongoing PR after SET with different blastocyst
morphology (918 SETs)
Blastocyst morphology
AA AB BA BB B-/-B p-Value
Total SETs 266 292 33 232 95 n/a
Positive hCG 222 240 26 178 61 n/a
Negative hCG 44 52 7 54 34 n/a
Biochemical 25 23 1 36 16 n/a
Miscarriages 18 17 3 14 6 n/a
Ongoing PR per ET, % 67.3 68.5 66.7 55.2 41.1 p<0.05
Birth outcomes
(2013-2015)
107 135 8 117 61 n/a
Live births 66 82 4 56 26 n/a
Live birth rate, % 61.7 60.7 50.0 47.9 42.6 p<0.05
http://www.ivfbigdata.com/pgt-calculator/

57. eSET FUTURE
SET vs DET in PGS cycles (2013-2016)
ETx1 ETx2 P-value
Total FETs 569 89
Positive HCG 442 78
Negative HCG 127 11
Ongoing pregnancies 335 66
Ongoing PR, % 58.9% 74.2% p<0.00599
Live birth rate,% 53.5% 71.6% p<0.00523
Twins 3 33 (1 triplet)
Twin rate 0.9% 50.0% p<0.00001
Presented by RSC team at ASRM 2016

58. The 5 Steps Towards Evidence Based Practice
1. Ask the right clinical question:
Formulate a searchable question
2. Collect the most relevant publications:
Efficient Literature Searching
Select the appropriate & relevant studies
3. Critically appraise and synthesize the evidence.
4. Integrate best evidence with personal clinical expertise, patient preferences and
values:
Applying the result to your clinical practice and patient.
5. Evaluate the practice decision or change:
Evaluating the outcomes of the applied evidence in your practice or patient.

59. The 5 Steps Towards Evidence Based Practice
1. Ask the right clinical question:
Formulate a searchable question
2. Collect the most relevant DATA:
Efficient Literature Searching
Select the appropriate & relevant studies
3. Critically appraise and synthesize the evidence.
4. Integrate best evidence with personal clinical expertise, patient preferences and
values:
Applying the result to your clinical practice and patient.
5. Evaluate the practice decision or change:
Evaluating the outcomes of the applied evidence in your practice or patient.

60. The current problem with the models: A vs B

61. Conclusion
1. Machine learning is not yet widely used in clinical practice
2. Augmented decision making with machine learning
3. Auto ML for rapid experimentation knowledge discovery

62. Thank you!
Lab:
K. A. Ivani, Ph.D.
O. O. Barash, Ph.D.
N. Huen
S. C. Lefko
C. MacKenzie
J. Ciolkosz
E. Homen
E. Jaramillo
MDs:
L. N. Weckstein
S. P. Willman
M. R. Hinckley
D. S. Wachs
E. M. Rosenbluth
S. P. Reid
M. V. Homer
E. I. Lewis