This document describes a new method for tracking cells that collide in microscope videos. It presents 3 key challenges: 1) cells' motion and appearance can change abruptly during collisions, 2) previous methods used single hypotheses that cannot handle collisions well, 3) manual tracking is tedious and expensive. The proposed method addresses these by using multiple hypotheses about collision states, Kalman filtering for reliability, and detecting/tracking cells in videos in a step-by-step process. Experiments show it achieves an 88% coverage of colliding cell positions, outperforming other methods.

1. Tracking Colliding Cells Nhat ‘Rich’ Nguyen Future Computing Lab

2. Flu Health Center Blood Test

3. White Count is a blood test to measure the number of white blood cells.

4. In a drop of blood… Number of white cells blood cancer 50,000 stress, viral infection, drug intake 25,000 healthy 5,000 flu, poisoning 0

5. It isimportant to keep track of white blood cells.

6. Challenges Methods Experiments

7. Challenges Methods Experiments

8. Video of white blood cells via a microscope

10. Manual Automatic Tedious Expensive Subjective Little Effort Economical Objective

11. As many cells move at a wide range of speeds… Collisions

12. abrupt change

13. Smoothness Constraints Region A broken tracks Region A robust tracks

14. Challenges Methods Experiments

15. Challenges Methods Experiments

16. Smoothness Constraint Region A broken tracks Region A Region A robust tracks Our Method

17. The first tracking method for colliding cells.

18. Training 100 cell samples 100background samples

19. Detection Classify each pixel as a Cell or Background

20. Tracking time

21. Kalman Filter Popular Extensively used for tracking. Optimal Estimate the most probable state. Simple Two steps: predict and correct.

22. No Collision Collision smooth smooth & abrupt reliability flexibility ? Kalman filter

23. Multiple Hypotheses H2 Non- Colliding Colliding H1 H3 H4

24. No Collision Non- Colliding Colliding H1

25. Collision H2 Non- Colliding Colliding

26. During Collision Non- Colliding Colliding H3

27. After Collision Non- Colliding Colliding H4

28. H2 Non- Colliding Colliding H1 H3 H4

29. The reliability of the Kalman filter, the flexibility of multiple hypotheses.

30. Tracking Steps

31. 1 2 3

32. 1 2 3

33. 1 2 3

34. 1 2 3

35. 1 2 3

36. 1 2 3

37. 1 colliding cells 2 3 non-colliding cell

38. stay colliding 1 2 3 split away keep moving

39. Region B Our method

40. Challenges Methods Experiments

41. Challenges Methods Experiments

42. Data 8 300 ~6K image sequences cells tracks cell positions

43. 112 188 colliding cells non-colliding cells

44. Compared Methods SC Smoothness Constraints Single Hypothesis Multiple Hypotheses SH MH

45. Comparisons MH SC SH

46. Percentage of Tracked Positions MH SH SC

47. Colliding vs. Non-colliding MH SH SC

48. Impact of detection MH SH SC

49. Given adequate detection results, our method covers 88% of colliding cell positions.

50. Challenges Methods Experiments

51. Conclusion The first tracking method for colliding cells. The reliability of the Kalman filter, the flexibility of multiple hypotheses. Excellent cell positions coverage. Non- Colliding Colliding 88%

52. Thank you.

53. Questions ?

55. Bonus Slides

56. S. J. Schmugge, S. Keller, N. Nguyen, R. Souvenir, T. H. Huynh, M. Clemens, M. C. Shin. "Segmentation of Vessels Cluttered with Cells using a Physics based Model". 11th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), New York, September 6-10 2008. N. Nguyen, S. Keller, T. Huynh, M. Shin. “Tracking Colliding Cells”. IEEE Workshop on Applications of Computer Vision (WACV), Snowbird, UT December 07-09 2009. [to be submitted to IEEE Transactions on Medical Imaging] N. Nguyen, S. Keller, Eric Norris, T. Huynh, M. Shin. “Tracking Colliding Cells in In-Vivo Intravital Microscopy Images”. Publications

57. Example of Multiple cell tracking

59. White Blood Cells Circulate in your blood Defend you against bacteria Protect you from disease

61. Previous Automatic Methods Ray et al. [2002] Active Contour Cui et al. [2005] Monte Carlo Mukherjee et al. [2004] Level Set Analysis

62. Previous Automatic Methods Eden et al. [2005] SmoothnessConstraints Li et al. [2005] Lineage Construction Smith et al. [2008] Probabilistic Formalization

63. Variation in cell appearances within an image time Varied appearance of a cell over time

64. Qualitative Comparison

65. Challenges In a collision, cell motion and appearance 1. could be different 2. change abruptly

66. Approach To improve tracking accuracy of colliding cells by: having separate collision states to describe cells inside and outside of collisions testing multiple hypotheses of cell motion and appearance as transitions between abrupt motion patterns.

67. AdaBoost Idea: combine many “rules of thumb” to a highly accurate prediction rule. Input: visual features from training samples. Schema: maintain a strategy to determine “rules of thumb” using weight distribution. Output: a single strong classifier which is a linear combination of the set of weak classifiers.

68. Detection Procedure Scan each pixel p in the image Compute image feature vector V from a window centered around p Classify p as a Cell pixel if the feature score in V satisfies the learned decision rule; otherwise classify p as a Background pixel. Cluster groups of Cell pixels into cell observation.

69. Motion and Appearance Model Collision States: State Transition: Hypotheses: to predict the state in the next frame control input vector State Vector*: Motion and Appearance Models: (for ) state transition matrix control input matrix process noise vector ~N(0,Qs) Observation Vector:

70. Multiple Hypotheses No Collision Collision

71. Performance RMSE RMSE : Root mean squared errors of position (pixel) -0.03 -0.17 +0.36 -0.21 +0.33 -0.20 SH introduces additional error in positions. MH does not introduce any additional error. Estimating colliding cells’ positions is more difficult.

72. Collision Duration RMSE The effect of collision duration on RMSE

73. Impact Detection RMSE The impact of detection on RMSE -0.17 -0.13 -1.05 -1.09 Different improvement between dataset. Different improvement between methods.

74. Future Work 1. Add more features to improve detection. 5 7 6 8

75. Future Work 2. Incorporate a probabilistic approach to transition between collision states. 72 73 75 76

76. Future Work 3. Expand to track cells with more complex motions and behaviors. 49 50 51 52

77. Detection Performance Recall : TP / (TP + FN) 75% Precision: TP / (TP + FP) 77%

78. Collision Duration The effect of collision duration on tracking 6 112 Exclude SC from being considered for collision. Classify colliding positions into bins based on the number of frames of the collision. colliding cells bins of collision duration

79. Detection Impact The impact of detection on tracking 38 596 Data with good detection results before and after collision (+/- 2 frames) cell positions treated colliding cells

80. Performance Table PTP: Percentage of Tracked Positions (%) +27 +9 +23 +3 +4 +24 +28

81. Detection Impact Table +9 +7 +16 +18

82. More Results Eden et al. [2005] Our Method

83. Tracking Steps Predict motion Predict collision Get measurements Get errors in position & area Match with minimal error

84. Collision States: Hypotheses: to predict the state in the next frame control input vector State Vector*: Motion and Appearance Models: (for ) state transition matrix control input matrix process noise vector ~N(0,Qs) Observation Vector: Measurement Model: measurement noise vector ~N(0,R) measurement transition matrix

85. State Vector of cell i : Predicted State Vector: Zero Matrix Zero Matrix Zero Vector Zero Vector Predicted Covariance:

86. Predicted State Vector: Hypothesized Measurement Vector: measurement transition matrix Error of hypothesis : observation from the detector weight vector Rule 1: Rule 2: error threshold of Unlikely (i, k) pair Stop corresponding condition:

88. Training 100 Cell Samples 100 Background Samples Features Mean Intensity Standard Dev. of Intensity Radial Mean  Decision Rules on feature scores

89. Collision Duration The effect of collision duration on PTP

90. H01 H00 No Collision (s = 0) Collision (s = 1) H11 H10

91. Measurements Cell matches Cell Detection Correspondence Update Cell image Finished tracks Tracks Predictions Multiple Hypotheses H00: No Collision – No Collision H01: No Collision –Collision H11: Collision –Collision H10: Collision – No Collision