1. Image Registration Methods for Reconstructing a Gene Expression Atlas of Early Zebrafish Embryogenesis Department Of Electronic Engineering Technical School Of Telecommunications Engineering Technical University Of Madrid Evangelia Balanou Master Thesis European Postgraduate Program On Biomedical Engineering University of Patras – National Technical University of Athens

2. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Outline Introduction Motivation Problem Goal Image Registration Components Design and Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results and Evaluation Comparison of Registration Methods Atlas Construction Conclusions and Future Work

3. Introduction Motivation Problem to be solved

5. Study embryonic development of vertebrates:

6. Vertebrate developmental disorders

7. Human hereditary disease

8. Vertebrate model: zebrafish

10. Short generation timeEarly development of a zebrafish embryo

11. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Problem Quantitative spatio-temporal data at cellular level about gene expression required Provided by Fluorescence In Situ Hybridization techniques and Laser Scanning Microscopy Second gene expression pattern x One gene expression pattern y z

13. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Goal Design and implement an image processing framework able to register different datasets with different gene expression patterns to a common template at a given developmental stage “Registration is the process of determining a geometrical transformation that aligns points in one view of an object with corresponding points in another view of that object or another object.” Template One dataset Template + registered image

14. Image Registration Fundamental task in image processing Various techniques (data, application)

15. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Image Registration Intensity-based : Calculates the transformation using voxel values alone Input: 2 images – fixed, moving Output: geometrical transformation Optimization problem Decomposed into a set of basic elements (defining different methods) Registration Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Initial Parameters

16. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Transformation Registration Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Initial Parameters Defines the type of parameters whose values align the two images (search space) Spatial mapping of points from the fixed image space to points in the moving image space (inverse mapping)

17. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Interpolation Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Initial Parameters Evaluate moving image intensities at the mapped, non-grid positions

18. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Similarity Measure Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Initial Parameters A measure of “how well” fixed and transformed moving match each other Provides a quantitative criterion to be optimized over the search space (similarity measure function, S(T) ) The desired optimum may be one of the local ones

20. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Optimization Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Parameters Transformation Initial Parameters Most complex component Starting from an initial set of parameters, iteratively searches the optimal solution of the similarity measure function over the parameter space defined by the transformation Stops when stopping criterion is met

22. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Cost Function

23. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Local optimization Start

24. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Local optimization End

25. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Global optimization Start

26. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Global optimization End Capture range of correct optimum (initial parameter range or initialization)

27. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Resampling Similarity measure Fixedimage TransformationParameters Movingimage Interpolation Optimization Resampling Registeredimage Transformation Initial Parameters Once a stopping criterion is met or iteration number has reached, the last transformation parameters are used to produce the registered image

28. Design & Implementation Implemented framework’s concept Different steps it is composed of

29. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Concept Goal: Design and implement an image processing framework able to register different datasets with different gene expression patterns to a common template at a given developmental stage Partial views Template embryo Nuclei channel Reference gene channel (goosecoid) Another gene channel *All images are 3D and grayscale *Colourmap just for visualization

30. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Concept Partial view of another embryo Partial views Template embryo Nuclei channel Nuclei channel Registration Reference gene channel Reference gene channel Another gene channel *All images are 3D and grayscale *Reference gene (position): goosecoid (gsc) *Colourmap just for visualization Gene expression atlas

31. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Overview registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Gravity centres Registration pipeline initialization Partial embryo view, third channel preprocessing transformation Third channel mapped Atlas construction pipeline

32. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Registration Pipeline registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Gravity centres Registration pipeline initialization Partial embryo view, third channel preprocessing transformation Third channel mapped Atlas construction pipeline Purpose: Determine the transformation parameters that bring into spatial alignment the template and one partial view

33. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Preprocessing & Addition Step registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Registration pipeline Purpose: Remove noise, blur, downsample, threshold Combine information from nuclei and gsc channels into a single image

35. Weighted Addition addition preprocessing preprocessing 0 255 Combined image Original channels Preprocessed channels Resolution: 512 x 512 x 465 Voxel size: 1.517 x 1.517 x 1,509μm Resolution: 128 x 128 x 116 Voxel size: 6.068 x 6.068 x 6.036μm

36. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Initialization Step registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Registration pipeline Purpose: Initial positioning of moving to fixed image’s space (no initial parameters in registration) If NOT sufficient overlapping, registration fails

37. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Initialization Step Preprocessed partial embryo view, nuclei channel (binary) Preprocessed partial embryo view, gsc channel initialization Moving image Initialized Moving image Rotation centre Fixedimage Preprocessed whole embryo view, nuclei channel (binary) Preprocessed whole embryo view, gsc channel Based on nature of data (nuclei and gsc channel) For both views one gravity centre from each channel The resulting four points define a spatial transformation that is applied on the moving image

38. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Initialization Step nmoving gscmoving y gscmoving Rotation axis nfixed vF nmoving translation Moving (partial view) Rotation angle gscfixed Translated nmoving vM Translated nmoving nfixed x z gscfixed *Blue/Orange-nuclei Green/Yellow-gsc expression pattern Fixed (template view)

39. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Initialization Step Before initialization Fixed Image Initialized Moving After Initialization Fixed (template) + Initialized Moving (partial) Partial view before and after initialization *Blue/Orange-nuclei Green/Yellow-gsc expression pattern

40. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Registration Step registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Registration pipeline Purpose: Find the transformation parameters that register the initialized moving image to the fixed

42. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Atlas Construction Pipeline registration addition initialization Partial embryo view, nuclei channel Initialized Moving image preprocessing Moving image Partial embryo view, gsc channel preprocessing Registered image Rotation centre Transformation Parameters Whole embryo view, nuclei channel addition preprocessing Fixedimage Fixedimage Whole embryo view, gsc channel preprocessing Gravity centres Registration pipeline initialization Partial embryo view, third channel preprocessing transformation Third channel mapped Atlas construction pipeline Purpose: Transformation of the third channel of the partial view Only transformation step is implemented as a new program

43. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Atlas Construction Pipeline Atlas Construction Pipeline -> Apply Transformation Parameters Partial view White-nuclei Red-gsc expression pattern Green-snail expression pattern Template Orange-nuclei Yellow-gsc expression pattern Registration Pipeline -> Transformation Parameters

45. Available at http://amide.sourceforge.net/

46. Amira

48. Results & Evaluation

49. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Data Developmental stage: Shield (6 hpf) Framework tested with six datasets (six embryos) One template, one whole embryo view Partial views of five different embryos Animal Dorsal Ventral Vegetal Template embryo Partial view nuclei channel gsc channel co-stained gene expression pattern e.g. snail * Images provided by: DEPSN , France

50. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Questions Does the implemented framework succeed in registering our data? What is the combination of similarity measure and optimization algorithm that results in a successful registration? In other words… What is the most appropriate registration method for our application?

51. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Preprocessing & Addition Original channels Preprocessed channels Combined image preprocessing addition Template preprocessing Slice Volume rendering Fixed image addition preprocessing One partial View preprocessing Slice Volume rendering One moving image Framework works with 2 datasets each time Preprocessing: smoothed, downsampled, nuclei channel turned to binary Addition: nuclei and gsc channels combined into a single image 5 partial -> 5 iterations (6 images in total – 1 fixed, 5 moving)

52. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Initialization Template Image Template Image Template Image Initialized Partial 2 Initialized Partial 3 Initialized Partial 1 Template + Initialized partial 1 Template + Initialized partial 2 Template + Initialized partial 3 Template Image Template Image Initialized Partial4 Initialized Partial 5 Initialization looks promising… Template + Initialized partial 5 Template + Initialized partial 4 *Blue/Orange-nuclei Green/Yellow-gsc expression pattern

53. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Method Evaluation Four different methods implemented Evaluation only by visual inspection of the results Optimization algorithms not comparable unless running with optimized parameters Lack of golden standard Point-to-point correspondence does not exist (different embryos)

55. Global optimization algorithm is still computing (D.E. not suitable)

57. Conclusions & Future work

58. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Summary - Conclusions Goal achieved Designed and implemented an image processing framework able to map different gene expression patterns on a common template (for a given developmental stage) Key points Addition: Combine information from two channels Initialization: Solves the problem of capture range for optimization Registration Method: Correlation Coefficient + Gradient Descent

59. Outline Introduction Motivation Problem Goal ImageRegistration Transformation Interpolation Similarity Measure Optimization Resampling Design & Implementation Concept Overview Registration Pipeline Atlas Construction Pipeline Tools Implementation Results & Evaluation Comparison of Registration Methods Atlas Conclusions & Future Work Conclusions-Future work Advantages Modularity Configurability Semi-automated Future work More datasets -> more gene expression patterns Other developmental stages Validated with known gene regulatory networks

60. Thanks to… Biomedical Image Technologies Laboratory (BIT) Technical School Of Telecommunications Engineering (ETSIT) Technical University of Madrid (UPM)

61. Thank you for your attention