Weitere ähnliche Inhalte
Mehr von Francisco (Paco) Florez-Revuelta (17)
Kürzlich hochgeladen (20)
Fusion of Skeletal and Silhouette-based Features for Human Action Recognition with RGB-D Devices
- 2. 1. Introduction
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
2
Motivation:
Use of both skeleton and silhouette in previous works
Problems with skeleton: lack of precision or noisy
caused by occlusion caused by body parts or objects
Pick-up and Throw
- 3. 1. Introduction
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
3
Motivation:
Use of both skeleton and silhouette in previous works
Problems with silhouettes: the only available
viewpoint is unfavourable for recognition
Tennis Serve
Forward Punch
Hammer
- 4. 1. Introduction
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
4
Solution:
Fusing different features that complement each other:
skeleton, RGB colour, silhouette (2D), volume (3D)…
In this work, we fuse skeleton and silhouette
- 5. 2. Fusion of skeleton and
silhouette
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
5
Concatenation of skeleton and silhouette
features
Skeleton:
Silhouette:
3D coordinates of the
joints
20
3
1
2
4
8
9
7
5
6
10
12
11
13
14
15
16
17
19
18
Radial summary
- 6. 3. Classification method based on
a bag of key poses
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
6
[1] A.A. Chaaraoui, P. Climent-Pérez and F. Flórez-Revuelta. Silhouette-based human action
recognition using sequences of key poses. Pattern Recognition Letters, 34(15):1799-1807,
- 7. 3. Classification method based on
a bag of key poses
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
7
[1] A.A. Chaaraoui, P. Climent-Pérez and F. Flórez-Revuelta. Silhouette-based human action
recognition using sequences of key poses. Pattern Recognition Letters, 34(15):1799-
- 8. 3. Classification method based on
a bag of key poses
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
8
Sequence recognition
Transform a sequence into a sequences of key poses
using the bag of key poses
Sequence matching using dynamic time warping
[1] A.A. Chaaraoui, P. Climent-Pérez and F. Flórez-Revuelta. Silhouette-based human action
recognition using sequences of key poses. Pattern Recognition Letters, 34(15):1799-1807,
- 9. © A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
4. Experimentation
9
Evaluation with the MSR Action3D dataset
- 10. 4. Experimentation
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
10
Cross-subject validation as in [2]:
Training: actors 1, 3, 5, 7 and 9
Testing: actors 2, 4, 6, 8 and 10
[2] W. Li, Z. Zhang, and Z. Liu. Action recognition based on a bag of 3D points. In 2010 IEEE
Computer Society Conference on Computer Vision and Pattern Recognition Workshops,
pp. 9-14, 2010.
- 11. 4. Experimentation
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
11
Cross-subject validation as in [2]:
[2] W. Li, Z. Zhang, and Z. Liu. Action recognition based on a bag of 3D points. In 2010 IEEE
Computer Society Conference on Computer Vision and Pattern Recognition Workshops,
pp. 9-14, 2010.
- 12. 4. Experimentation
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
12
Confusion matrices for AS1:
a02 a03 a05 a06 a10 a13 a18 a20
a02 0,92
0,08
a03
1,00
a05
0,91
0,09
a06
0,09
0,73
0,18
a10
1,00
a13
1,00
a18
1,00
a20
0,14
0,07
0,29
0,50
a02 a03 a05 a06 a10 a13 a18 a20
a02 0,67
0,25
0,08
a03
0,58 0,42
a05
0,18 0,73
0,09
a06
0,18 0,82
a10
1,00
a13
0,07
0,93
a18
0,33 0,20
0,07
0,40
a20
0,07 0,14 0,07 0,14
0,57
Skeleton
Silhouette
a02 a03 a05 a06 a10 a13 a18 a20
a02 1,00
a03
1,00
a05
0,09 0,91
a06
0,18
0,73
0,09
a10
1,00
a13
1,00
a18
1,00
a20
0,29
0,71
Fusion
- 13. © A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
4. Experimentation
13
Leave-one-actor-out:
- 14. 5. Conclusions and future work
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
14
Straightforward fusion of skeleton and silhouette
Improvement in the recognition rate
Include also side and top projected silhouettes
Select the weight for each feature vector
Feature subset selection
- 15. 5. Conclusions and future work
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
15
We have already applied the approach in [3] for
feature selection to the fusion of skeleton and
silhouette
Cross-Subject
LOAO
[3] A.A. Chaaraoui, J.R. Padilla-López, P. Climent-Pérez, and F. Flórez-Revuelta. Evolutionary
joint selection to improve human action recognition with RGB-D devices. Expert Systems with
Applications, 41(3):786-794,2014.
- 16. 5. Conclusions and future work
© A.A. Chaaraoui, J.R. Padilla-López and F. Flórez-Revuelta (CDC4CV’13)
16
Straightforward fusion of skeleton and silhouette
Improvement in the recognition rate
Include also side and top projected silhouettes
Select the weight for each feature vector
Feature subset selection
Should we create a large bank of features and
select them appropriately?