The spine is visualized on many CT and MR exams, including thorax and abdomen scans that were originally not intended for spine imaging. Because these often cover several but not all vertebrae, it is difficult to make strong assumptions for automatic analysis. Challenges are therefore the unknown number of target structures (vertebrae) in the image, their anatomical identification (which vertebrae are visible? must not assign the same label to two vertebrae) and that some biomarkers are related only to part of the vertebrae, often the vertebral body. This talk covers an instance segmentation approach for vertebra detection, segmentation, and anatomical identification, and a partitioning approach to separate vertebral body and arch based on thin-plate spline surfaces positioned by a convolutional neural network.

1. Learning-based vertebra segmentation,
identification and partitioning
DDH Open, 3 April 2019
Nikolas Lessmann

2. Extrapulmonary diseases in lung cancer screening
Cardiovascular disease
Osteoporosis
Lung cancer

3. Osteoporosis

4. Potential new biomarkers
• Height loss compared to
previous screening round?
• Density loss over time?
• …
CT-biomarkers for osteoporosis
Bone mineral density
(usually done with dual energy
X-ray absorptiometry or QCT)
Relative height loss
(compared with
neighboring vertebrae)

5. • Visible in many different types of scans (different FOV, context)
• Difficult to distinguish (anatomical label)
• Difficult to separate from each other in low-resolution scans
• How reliable are density measures in low-SNR scans?
Challenges for automatic vertebra analysis in CT

6. Part 1
Vertebra segmentation

7. Vertebrae often need to be separated not only
from the background, but also from each other
Generic algorithm should not make
assumptions about:
• How many vertebrae are visible
• Which vertebrae are visible
• Their location within the image
= instance segmentation
An instance segmentation problem

8. Dedicated loss functions, e.g., arXiv:1708.02551
Detection & segmentation pipelines
(e.g., Mask RCNN, based on region proposals)
Recurrent attention
(instance by instance segm.)
Instance segmentation with CNNs
arXiv:1703.06870
arXiv:1605.09410

9. • Only one class of objects (vertebra)
• Size and shape are relatively consistent
• Instances are located in a column, one next to the other
A restricted instance segmentation problem

10. Combined detection and segmentation
• Segment one vertebra
• Look for the next in the neighborhood
The network needs a memory, like
recurrent networks, to remember which
vertebrae have already been found
Iterative instance segmentation

11. ROI (field of view of CNN) moves
through the image volume

12. Patch-based iterative instance segmentation
Image Memory Segmentation
1. Iteration

13. Patch-based iterative instance segmentation
Image Memory Segmentation
2. Iteration

14. Patch-based iterative instance segmentation
Image Memory Segmentation
3. Iteration

15. • Use prior knowledge that vertebrae are located next to each other
• Search for spine, then follow the spine
Traversal along the spine

16. I = Image patch
M = Memory patch
S = Segmentation mask
L = Anatomical label
Prediction of label 1-24, global
probabilistic model to find most
plausible labeling
C = Completeness score
Vertebrae may be visible only
partially in the image, relevant for
following automatic steps
Network architecture

17. Label regression with probabilistic interpretation
Prediction: 14.8
Interpretation:
• 80% probability label 15
• 20% probability label 14
Maximum-likelihood approach
Anatomical labeling
Mean certainty per labeling sequence
0.24 0.56 0.89

18. • Some vertebrae are often only partially
contained in the scan
• For some applications, that is important to
know – biomarkers computed based on
incompletely visible vertebrae can be
unreliable
Completeness score

19. Sum of three error terms
• Segmentation: Minimize FP and FN voxels
• Labeling: L1 norm
• Completeness: Binary crossentropy
Loss function
Weight map to improve
segmentation accuracy at
the boundary between
vertebrae

20. • Adam optimizer with fixed learning rate + high momentum (0.999)
• Single patch per iteration (no mini-batches)
 each patch has ~2,000,000 voxels
• Also patches without vertebral bone are presented to the network
• Memory state derived from reference segmentation
• Data augmentation: elastic deformations, random Gaussian noise and smoothing, random
cropping
Training details

21. • Manual and semi-automatic annotation of NLST (low-dose chest CT) scans
• Several datasets from vertebra segmentation challenges
Evaluation
Lumbar spine CT Chest CT Lumbar spine
MR (VB only)

22. Evaluation

23. Part 2
Vertebra partitioning

24. • We can do vertebra segmentation in CT
• We can do vertebral body segmentation in MR
• We cannot do vertebral body segmentation in CT… (New training set? Transfer learning? GANs?)
Motivation

25. Alternative idea: Let a CNN fit a continuous surface that represents the boundary
Surface fitting

26. • Splines are defined by a set of control points
• Splines are smooth between control points
Think of a stick or a thin metal sheet
that is fixated at several points and
resists bending due to its rigidity
 CNN could predict location of control points
Fitting a thin-plate spline with a CNN

27. How to backpropagate through this?
Flowchart

28. Differentiable TPS surface?
Surface “height” across grid
Configuration of grid points w.r.t. control points
Configuration of control points w.r.t. control points
Control point “heights” (network predicition)

29. If we had vertebra and vertebral body segmentations of the same subjects (and maybe even the
same scans), we could train a simpler approach …
We have segmentation masks from
• Different subjects
• Different scans
• Different modalities even
How to train without matching segmentation pairs?

30. Segmentation masks do not match, but we can
learn something from this data – statistical
approach, capture plausible shapes
• Do not use image intensities at all
• Train convolutional autoencoder on vertebral
body segmentation masks and use as
out-of-distribution classifier
Shape encoding with autoencoders

31. Flowchart
Trained with vertebra segmentation masks from CT
scans (around 600 NLST scans, > 6000 vertebrae)
Previously trained with vertebral body segmentation
masks from MR scans (around 300 UMCU scans,
>1000 vertebral bodies)

32. Reconstruction error + several regularizing terms
• Volume penalty: Volume Vertebra – Volume Vertebral Body
• “Fights” the reconstruction loss by pushing the surface to the right (posterior)
• Empty segmentation masks are perfectly reconstructed by the AE…
• Curvature penalty: abs(det(Hessian))
• Prevents that surface is very wobbly (seems not really necessary)
Loss function

33. Loss function

34. Manual annotation of vertebrae and vertebral
bodies in 5 chest CT scans
Results

35. Measurements of the height of the vertebral
bodies and correlation with extracted volume
Average of 3 measurements in 389 scans
Results

36. We can…
• Find and segment the vertebrae
• Label them anatomically
• Determine if they are completely or partially visible
• Partition them into body and posterior elements
Next step: Application (fracture prediction, fracture classification, …)
Also possible: use partitioning results to train direct vertebral body segmentation
Conclusions and future work

37. Thank you!