A short history of video coding
•
29 likes•9,961 views
Video coding is an essential component of video streaming, digital TV, video chat and many other technologies. This presentation, an invited lecture to the US Patent and Trade Mark Office, describes some of the key developments in the history of video coding. Many of the components of present-day video codecs were originally developed before 1990. From 1990 onwards, developments in video coding were closely associated with industry standards such as MPEG-2, H.264 and H.265/HEVC. The presentation covers: - Basic concepts of video coding - Fundamental inventions prior to 1990 - Industry standards from 1990 to 2014 - Video coding patents and patent pools.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A short history of video coding
Similar to A short history of video coding (20)
Recently uploaded
Recently uploaded (20)
A short history of video coding
- 1. A History of Video Coding Invited Talk, United States Patent and Trademark Office Iain Richardson December 2014 Slide 1 vcodex.com
- 2. What this talk is about Slide 2
- 3. Slide 3 From To
- 4. Video coding : the basics Slide 4
- 5. A video codec (1) Slide 5 Video Source Decompress (Decode) Compress (Encode) Video Display Coded video enCOder / DECoder
- 6. A video codec (2) Slide 6 video source partition transform entropy encode predict (add) inverse transform entropy decode compressed video video output VIDEO ENCODER VIDEO DECODER predict (subtract) reconstruct
- 7. 1. Partitioning Slide 7 frame or picture slice or tile macroblock (MB) or coding tree unit (CTU) sequence
- 8. 2. Prediction Slide 8 Original Frame Prediction Frame Residual
- 9. 3. Transform + Quantize Block of samples After transform After quantization Slide 9
- 10. 4. Entropy coding Slide 10 1 1 0 1 1 1 1 1 0 0 0 1 1 coded components (syntax elements) bitstream
- 11. source frames ........ ........ coded bitstream decoded frames ENCODE DECODE Slide 11
- 12. Video Encoder Form prediction Transform + quantize Inverse transform + quantize Entropy encoder Current MB Prediction MB Residual MB Decoded Residual MB Intra Inter .... Current frame or field Previously coded frames or fields + Coded bitstream - + + Slide 12
- 13. Video Decoder Coded bitstream .... Entropy decoder Inverse transform + quantize Decoded Residual MB Current decoded frame or field Inter Previously decoded frames or fields Prediction MB Form prediction Intra Decoded MB + + Slide 13
- 14. Ancient history Slide 14
- 15. Slide 15 1851
- 16. 1910 Slide 16
- 17. Slide 17 1964
- 18. The foundations Slide 18
- 19. Slide 19 1952 Variable length coding 1972 Frame differencing 1974 Discrete Cosine Transform 1981 Motion compensated video codec 1990 Bidirectional prediction Variable block size motion compensation
- 20. Huffman variable length coding, 1952 Slide 20 D A Huffman, Minimum redundancy codes
- 21. Frame differencing, 1972 Slide 21 US 3,679,821, Schroeder, Bell Labs
- 22. Discrete Cosine Transform, 1974 Slide 22 The Discrete Cosine Transform Ahmed, Natarajan and Rao
- 23. Video codec with motion compensation, 1981 Slide 23 Displacement measurement and its application in interframe video coding, Jain and Jain
- 24. Bidirectional + variable block size motion prediction, circa 1990 Slide 24 MPEG-1, LeGall, 1991 Chan et al, 1990
- 25. By 1991, the basic codec was complete Slide 25 MPEG-1 encoder, 1991
- 26. Video Encoder Form prediction Transform + quantize Inverse transform + quantize Entropy encoder Current MB Prediction MB Residual MB Decoded Residual MB Intra Inter .... Current frame or field Previously coded frames or fields + Coded bitstream - + + Slide 26
- 27. The video coding standards Slide 27
- 28. Slide 28 Moving Picture Experts Group - MPEG standards - Broadcast / entertainment Joint Video Team / Joint Collaborative Team Video Coding Experts Group - H.26x standards - Communications
- 29. Slide 29
- 30. Slide 30 1990 1993 1995 1996 1999 2003 2013 ITU-T JOINT ISO MPEG MPEG-1 H.263 H.262/MPEG-2 H.264/AVC H.265/HEVC MPEG-4 H.261 ….
- 31. ITU-T H.261 (1990) Slide 31 2-way video telephony ISDN lines (p x 64kbps)
- 32. ITU-T H.261 (1990) Slide 32 16x16 blocks 8x8 DCT I P P P Forward prediction …011011000100… Variable length coding
- 33. MPEG-1 Video (1991-93) Slide 33 Video CD playback
- 34. MPEG-1 Video (1991-93) Slide 34 I B B P B B P B B P B B I 0 1 2 3 4 5 6 7 8 9 10 11 12 Intra, forward and bidirectional prediction Half-pixel motion vectors
- 35. MPEG-2 Video (1995) Slide 35 Digital TV broadcasting, DVD
- 36. MPEG-2 Video (1995) Slide 36 16x16 and Interlaced video 16x8 block sizes
- 37. H.263 (1996), MPEG-4 Part 2 (1999) Slide 37
- 38. H.263 (1996), MPEG-4 Part 2 (1999) Slide 38 16x16, 8x8 blocks Arithmetic coding Error resilience Advanced prediction Deblocking filter Object coding Mesh coding Face animation Many optional coding tools
- 39. H.264 / Advanced Video Coding (2003) Slide 39
- 40. H.264 / Advanced Video Coding (2003) Slide 40 16x16 down to 4x4 blocks Intra prediction 0 1 2 3 4 5 6 Multiple reference frames + context adaptive coding (CABAC / CAVLC), ¼ pixel motion vectors, loop filter….
- 41. H.265 / High Efficiency Video Coding (2013) Slide 41
- 42. H.265 / High Efficiency Video Coding (2013) Courtesy of Parabola Research Slide 42 64x64 down to 4x4 blocks Parallel processing support + Lots of “H.264 like” optimizations….
- 43. HEVC analyzer screenshot Courtesy of Parabola Research Slide 43
- 45. Compression performance Slide 45 MPEG-2 ~2x better H.264/AVC ~2x better H.265/HEVC
- 46. Performance examples Slide 46
- 47. Slide 47 MPEG-1 MPEG-2 H.264 HEVC
- 48. Slide 48 Difference image (MPEG-1) Difference image (HEVC)
- 49. Patents and patent pools Slide 49
- 50. Example Slide 50 September 2001 Patent application is filed September 2001 Proposal document is submitted to Joint Video Team May 2003 H.264 is published, including the concept of the application & proposal 2005 Patent is granted
- 51. Slide 51 Published Standard Patents Implementors Patent pool
- 52. Patents and patent pools: the early standards Slide 52 H.261, MPEG-1, H.263 : - no formal patent pool MPEG-2 : - first patent pool, managed by MPEG-LA MPEG-4 Part 2 : - patent pool, managed by MPEG-LA
- 53. Patents and patent pools: H.264 / AVC Slide 53 2001-2003 Royalty Free Baseline intention announced by 32 patent holders May 2003 First edition of H.264 published Late 2003 MPEG-LA and Via Licensing announce patent pools. Royalty Free Baseline abandoned. 2004 onwards Via Licensing pool “folds”. MPEG-LA pool dominates.
- 54. Patents and patent pools: recent developments Slide 54 Open source VP8 format: MPEG-LA attempts to create patent pool, Google and MPEG-LA reach agreement and pool is abandoned. HEVC: Patent pool created by MPEG-LA (2014).
- 55. Conclusion Slide 55 Many of the key components of modern video codecs originated before 1990. In the last 25 years, developments in video coding have been strongly influenced by standardisation. Performance continues to improve, driven by optimization of compression tools.
- 56. Useful information sources Slide 56 Low bit rate / Video Coding Experts Group archives, 1996-present: http://wftp3.itu.int/av-arch/video-site/ Joint Video Team archives (H.264 development): http://wftp3.itu.int/av-arch/jvt-site/ JCT-VC archives (HEVC development): http://wftp3.itu.int/av-arch/jctvc-site/ Video coding resources: http://vcodex.com/
Editor's Notes
- On the left is a typical block of pixel samples, 4x4 in this case. In any region of the picture, we’re going to have a range of brightnesses or levels. Each one of those numbers is important to the video image and each one takes up space in the video file. The idea of a video coding transform is to convert a block of samples into a frequency representation, such as the example in the middle. In a video encoder, we follow the transform with quantization, which removes small or insignificant values. So instead of the 16 image samples on the left, we have a transformed and quantized block on the right. Most of the numbers are zero, which makes this transformed and quantized block very easy to store in a compressed form. And if we do this in the right way, it’s possible to reverse the process and get a decoded image block that looks almost the same as the original.