- The document discusses Kurento, an open-source media server platform that allows building multimedia applications through simple APIs. - Kurento provides a media server called Kurento Media Server (KMS) that handles media processing and streaming, along with JavaScript, Java and REST APIs called the Kurento Media API to access KMS capabilities. - The document provides examples of using Kurento to build WebRTC applications including a basic file player, WebRTC loopback, adding a face overlay filter to WebRTC, and recording WebRTC streams. It also discusses promises and generators as ways to simplify asynchronous JavaScript code for Kurento applications.

1. www.kurento.org
Developing rich multimedia applications with Kurento
Developing rich multimedia applications with
Kurento: a tutorial for JavaScript developers
lulop@kurento.org
http://www.kurento.org
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu

2. www.kurento.org
WebRTC technologies from the
perspective of a developer
2
Developing
the client side
Developing the
infrastructure
side
Before WebRTC
First wave of
WebRTC technologies
Second wave of
WebRTC technologies
Begin End Begin End
• APIs
• Standards
• FOSS
• Coherent APIs
• Standards
• FOSS
• Multiplatform
Begin End
• APIs
• Standards
• FOSS
• Platforms
• Out of the box
capabilities
2012 2014

3. www.kurento.org
What’s a WebRTC infrastructure
3
Peer-to-Peer WebRTC Application (without media infrastructure)
WebRTC video stream
WebRTC Application with media infrastructure
Media infrastructure

4. www.kurento.org
Some WebRTC media infrastructure
providers
4

5. www.kurento.org
Media infrastructure common
capabilities
5
Transcoding media infrastructure
VP8 H.264
MCU media infrastructures
Recording media infrastructure

6. www.kurento.org
Can’t we go beyond?
Media is
here
Media got
there
Media got
there
Analyze
Transform
Store
Transcoding
MCU
Recording
Enrich
Augment
Adapt
Sensors
Context
Events
Media is
here
What WebRTC
infrastructures are
• Transcoding
• MCU
• Recording
What WebRTC
infrastructures should be
• Flexible processing
• Augmented reality
• Blending
• Mixing
• Analyzing
• Etc.
6

7. www.kurento.org
•Interoperable media exchange (multiplatform/multiprotocol)
•WebRTC, RTP, HTTP (video tag), etc.
•Standard capabilities Capabilities
•Transcoding, MCU, recording
•Advanced capabilities
•Computer vision, augmented reality, mixing, blending, etc.
Multimedia infrastructure
• REST API
• JavaScript API
• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
7

8. www.kurento.org
Future
Internet
Multimedia
Infrastructure
Simple
Development
APIs
Kurento: the equation
8

9. www.kurento.org
Kurento Media Server (KMS): the
nucleus of Kurento
• KMS is a middleware for media streams
– Receives the stream
– Process the stream
– Issues the stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
9

10. www.kurento.org
The Media API: The API for accessing
KMS capabilities
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
Java
Media API
JavaScript
Media API
REST
API
Applications define the processing
of streams getting through KMS
10

11. www.kurento.org
Media API: Media Elements and Media
Pipelines
SinkSRC
Sink
SRC
SRCSink
Sink
Media Element
• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements
implementing the desired media
logic.
• The Media API provides the
capability of creating media
pipelines by joining media
elements of the toolbox
Media Element
Sink
SRC
11

12. www.kurento.org
The Lego Game: Developers create
applications connecting media elements
Protocols
& Codecs
Computer
Vision
Augmented
Reality
Mul sensory
Mul media
Media
Repository
IPTV
Integra on
Group
Communica ons
Sink%
H pEndpoint
Sink%SRC%
RtpEndpoint
SRC$
DataChannelEndpointSink%SRC%
WebRtcEndpoint
Sink%
SRC%
BarCodeReader
Sink%
SRC%
PointerTracker
Sink%
SRC%
FaceDetector
Sink%
SRC%
FaceOverlay
Sink%
SRC%
ChromaFilter
Sink%
SRC%
HeartRateBlender
Sink%
RecorderEndpoint
SRC$
PlayerEndpoint
Sink%
CdnUpload
Sink%
IPTVConnector Mixer
SinkSRC
Sink
SRC
Sink
SRCSink
Sink
Application 2
Sink
SRC
Sink
SRCSink
SinkSRC
Sink
Application 3
SinkSRC
SRCSink
SRC Sink
SinkSRC
Application 1
Toolbox of media elements
12

13. www.kurento.org
Media API: trivial example (JavaScript)
kwsMedia.create("MediaPipeline", function(error, pipeline){
pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){
pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){
playerEndpoint.connect(httpGetEndpoint, function(error){
httpGetEndpoint.getUrl(function(error, url){
videoInput.src = url;
});
playerEndpoint.play(function(error){
});
});
});
});
});
Media Pipeline (inside KMS)
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
13

14. www.kurento.org
Kurento Architecture: putting it all
together
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video Playing
and Recording
Kurento
WS JSON-RPC API
Media
Media
Kurento Media Connector (KMC)
14
SIP
Servlet
SOAP
HTTP
Servlet
REST
Browser
Developer code with
specific application
logic (pipelines)
Media capabilities of
the browser
Kurento Media API
Specific
logic

15. www.kurento.org
Application execution flow
Client
Code
Media Connector
(KMC)
Media
Server (KMS)
Crate pipeline/element…
(JSON-RPC)
Commands requesting
the creation of a pipeline
Reference to media object …
(JSON-RPC)
Pipeline
cereation
phase
Media
exchange
phase
1
2
Specific logic
(e.g. AAA)
Media
pipeline
creation
Media exchange between client and server
15

16. www.kurento.org
What about Node.js?
16
Kurento Media Server (KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video Playing
and Recording
Kurento
WS JSON-RPC API
Media
Media
Kurento Media Connector (KMC)
SIP
Servlet
SOAP
HTTP
Servlet
REST
Node.js Pipeline creation
+
AAA
Kurento Media API
Browser
Browser-side code

17. www.kurento.org
Let’s develop with Kurento
• What you need
– A Kurento instance
• You can install your own Kurento instance
– http://www.kurento.org/docs/current/
• You can launch a Kurento instance at the FI-LAB (FI-WARE project)
– http://lab.fi-ware.org
– FI-LAB is a cloud infrastructure making possible to execute instances from images or blueprints
• kws-media-api
– https://github.com/kurento/kws-media-api
– http://github.com/kurento/kws-utils
» Node.js
• npm install kws-media-api
• npm install kws-utils
» Browser
• bower install kws-media-api
• bower install kws-utils
• Getting help
– Kurento web site
• http://www.kurento.org
– Kurento mailing list
• https://groups.google.com/forum/#!forum/kurento
– Twitter
• @kurentoms
17

18. www.kurento.org
Kurento Hello World: Playing a file
with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media from
file or URI
HTTP media
streaming
Sink
SRC
PlayerEndpoint
Kurento WS JSON-RPC API
Create / excec …
Ref / answer …
18
Kurento Media Connector (KMC)

19. www.kurento.org
Playing a file with an HTML5 client:
JavaScript code
const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this
const file_uri = “http://uri.to.video.file/”; //change this
var videoInput = document.getElementById("videoInput"); //video tag element
KwsMedia(ws_uri, function(kwsMedia){ //instantiate kws-media-api manager
kwsMedia.create("MediaPipeline", function(error, pipeline){ //instantiate pipeline
pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){
pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){
playerEndpoint.connect(httpGetEndpoint, function(error){
httpGetEndpoint.getUrl(function(error, url){
videoInput.src = url;
});
playerEndpoint.play();
});
});
});
});
});
Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer
19

20. www.kurento.org
Playing a file with an HTML5 client:
HTML code
<!DOCTYPE html>
<html>
<head>
<meta charset="US-ASCII">
<title>Insert title here</title>
<script type="text/javascript" src="../api/kws-media-api.js"></script>
<script type="text/javascript" src="demo.js"></script>
</head>
<body>
<button id="playButton">Click to start playing</button><br>
<video id="videoInput" autoplay></video>
</body>
</html>
Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer
20

21. www.kurento.org
Playing a file with an HTML5 client:
See the example working
https://www.youtube.com/watch?v=Mp6UEL9U-v4
21

22. www.kurento.org
WebRTC loopback
Kurento Media Server
WebRTC
Streaming
SinkSRC
22
WebRtcEndpoint

23. www.kurento.org
WebRTC loopback: JavaScript code
23
Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback
const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this
var videoInput = document.getElementById("videoInput");
var videoOutput = document.getElementById("videoOutput");
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer){
KwsMedia(ws_uri, function(kwsMedia){
kwsMedia.create("MediaPipeline", function(error, pipeline){
pipeline.create("WebRtcEndpoint", function(error, webRtc){
webRtc.processOffer(offer, function(error, answer){
webRtcPeer.processSdpAnswer(answer);
});
webRtc.connect(webRtc);
});
});
});
};
23

24. www.kurento.org
WebRTC loopback: HTML code
24
<!DOCTYPE html>
<html>
<head>
<meta charset="US-ASCII">
<title>Insert title here</title>
<script type="text/javascript" src="../api/adapter.js"></script>
<script type="text/javascript" src="../api/kws-media-api.js"></script>
<script type="text/javascript" src="../api/kws-utils.js"></script>
<script type="text/javascript" src="demo.js"></script>
</head>
<body>
<button id="launchButton">Click me ...</button><br>
<div style="float: left; border: 2px solid black; margin: 5px;">
<h1>Local stream</h1>
<video id="videoInput" autoplay></video>
</div>
<div style="float:left; border: 2px solid black; margin: 5px;">
<h1>Remote stream</h1>
<video id="videoOutput" autoplay></video>
</div>
</body>
</html>
Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback
24

25. www.kurento.org
WebRTC loopback: see the example
working
25
https://www.youtube.com/watch?v=hUChVutQ2og
25

26. www.kurento.org
WebRTC with FaceOverlay
Media Pipeline
WebRTC
Streaming
SinkSRC
26
Sink
SRC
WebRtcEndpoint
FaceOverlayFilter

27. www.kurento.org
WebRTC with FaceOverlay: JavaScript
code
27
Source: https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlay
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput , onOffer, onError);
function onOffer(offer) {
KwsMedia(ws_uri, function(kwsMedia) {
kwsMedia.create("MediaPipeline", function(error, pipeline) {
pipeline.create("WebRtcEndpoint", function(error, webRtc) {
pipeline.create("FaceOverlayFilter”, function(error, filter) {
//hat_uri is the uri pointing to the overlay image to draw on top of detected faces
//These percent variables need to be defined to adjust overlay image to face box
filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent);
webRtc.connect(filter, function(error) {
filter.connect(webRtc, function(error) {
webRtc.processOffer(offer, function(error, answer) {
webRtcPeer.processSdpAnswer(answer);
});
…//many }); omited here for simplicity
});
27

28. www.kurento.org
WebRTC recorder
28
28
Media Pipeline
WebRTC
Streaming
SinkSRC
Sink
Media
to
file or
URI
Media Pipeline
WebRTC
Streaming
SinkSRC
Media
from
file or
URI
SRC
WebRtcEndpoint
WebRtcEndpoint
RecorderEndpoint
PlayerEndpoint

29. www.kurento.org
WebRTC Recorder: JavaScript code
webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer) {
KwsMedia(ws_uri, function(kwsMedia) {
kwsMedia.create('MediaPipeline', function(error, pipeline) {
pipeline.create('RecorderEndpoint', {uri : file_uri}, function(error, recorder) {
pipeline.create('WebRtcEndpoint', function(error, webRtv) {
webRtc.connect(recorder, function(error) {
recorder.record();
webRtc.connect(webRtc, function(error) {
webRtc.processOffer(offer, function(error, answer) {
webRtcPeer.processSdpAnswer(answer);
});
... }); //many }); omited here for simplicity
}
29 29
Source: https://github.com/Kurento/kws-tutorial/tree/develop/Recorder

30. www.kurento.org
WebRTC with Recorder: see the
example working
30
https://www.youtube.com/watch?v=wp5A4NZ-kd4
30

31. www.kurento.org
But … is this code beautiful?
31
31
Developers love spaghetti for eating,
but not for code
From that code structure
to this code structure

32. www.kurento.org
Avoiding the callback hell
• The hidden secret of the content API
– Every call to a kws-media-api returns a promise
• var p1= kwsMedia.create(“MediaPipeline”);
• p1.then(function(pipeline){…})
– Promises
• A proxy to a value not knows when the proxy is created
• Allows associating handlers to
– Value successfully obtained
– Error
• https://developer.mozilla.org/en-
US/docs/Web/JavaScript/Reference/Global_Objects/Promise
• http://www.html5rocks.com/en/tutorials/es6/promises/
– Example
32 32

33. www.kurento.org
Are promises really useful for our
objectives?
• Developing with promises
– WebRTC loopack with FaceOverlay
• Source
– https://github.com/Kurento/kws-media-
api/tree/develop/example/WebRtcEndpoint-FaceOverlayFilter_promises_2
• Does structure improve?
– Not much
– Problems
• Nesting avoiding, but chaining isn’t much beautiful
• Nesting eliminated => closures eliminated
– Scope of variables (pipelines, elements, etc.) lost.
– Need to pass variables along on each call
• What’s the point of using promises then?
– Continue reading if you are the kind of guy of like living on the
edge …
33 33

34. www.kurento.org
Generators+co: the callback hell killer
• Generators
– Cooperative (non preemtive) simulation of synchronous
behavior on iterator functions by maintaining execution
state
• https://developer.mozilla.org/en-
US/docs/Web/JavaScript/Guide/Iterators_and_Generators
– Allow multiple entry points for suspending and resuming
execution at certain locations.
– Available on EC6 (Harmony)
• Co
– Combines promises and generators
– Uses promises to “connect” generator exit and reentrance
– https://github.com/visionmedia/co
34

35. www.kurento.org
Revisiting the WebRTC with loopback
example
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer){
co(function*(){
var kwsMedia = yield KwsMedia(ws_uri);
var pipeline = yield kwsMedia.create("MediaPipeline");
var webRtc = yield pipeline.create("WebRtcEndpoint");
var filter = yield pipeline.create("FaceOverlayFilter");
//you need to define the hat_uri and percents variables
yield filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent);
var answer = yield webRtc.processOffer(offer);
webRtcPeer.processSdpAnswer(answer);
yield webRtc.connect(filter);
yield filter.connect(webRtc);
})();
}
35
Source code of previous examples using generators:
https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback
https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlayGenerator
https://github.com/Kurento/kws-tutorial/tree/develop/RecorderGenerator

36. www.kurento.org
Isn’t it beautiful ?
36

37. www.kurento.org
Let’s do some more complex stuff: a
WebRTC game
Media Pipeline
SinkSRC
Sink
SRC
Sink
SRC
37
Sink
SRC
Sink
WebRtcEndpoint
MirrorFilter PointerDetectorFilter
FaceOverlayFilter
RecorderEndpoint

38. www.kurento.org
WebRTC game: code
38
Source:
https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator
38

39. www.kurento.org
WebRTC game: see the example
working
39
https://www.youtube.com/watch?v=5eJRnwKxgbY
39

40. www.kurento.org
Summary and conclusions
• Kurento
– Open Source Softare
• LGPL v2.1
– Multimedia infrastructure for HTML5
• WebRTC
• HTML5
– Advanced media processing capabilities
• Recording, MCUs, Transcoding, Computer Vision, Augmented Reality, Mixing, Blending.
• kws-media-api
– Access Kurento capabilities from JavaScript
• Media elements and media pipelines
• Like a “Lego game”
– Beautiful code
• Promises
• Generators
• CO
40

41. www.kurento.org
Thank you
41
Suggestions, comments and complains:
lulop@kurento.org
http://www.kurento.org
http://www.github.com/kurento
Twitter: @Kurentoms
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu