Presentation by Juan A. Martinez Navarro Representative, Odin Solutions S.L. FIWARE Tech Summit 28-29 November, 2017 Malaga, Spain

1. Stream processing with Kurento Media Server
Juan Navarro Moreno
Kurento Software Developer
jnavarro@naevatec.com
github.com/j1elo
www.kurento.org

2. Overview

3. 3
Overview – What is Kurento?
●
A server that simplifies creation of streaming applications.
●
Focused on processing of Audio/Video media streams.
●
Supporting WebRTC communications.
●
Based on modules that perform input → processing → output.
Kurento is an Open Source project.
Hosted on GitHub: https://github.com/Kurento

4. 4
Overview – What can be done with Kurento?
●
Abstracts compatibility issues between sender and receiver.
●
Manipulates or redistributes the streams.
●
Extracts information from the streams.
Input stream
Kurento
Media
Server
Output stream
Product-specific data
Eg.: FIWARE data-store

5. 5
Overview – What can be done with Kurento?
Examples:
●
Abstraction of incompatible
video codecs
●
Multi-point distribution of media
●
Processing / Storing media

6. 6
Overview – What can be done with Kurento?

7. 7
Overview – How does Kurento work?
A server with two main components:
●
Endpoints: where data flows In/Out.
●
Filters: composable modules where data is processed
or transformed.
Media stream
Src
Sink Media stream
Input
Endpoint
Output
Endpoint

8. 8
Overview – How does Kurento work?
Endpoints and Filters are linked together, mix-and-matched to form a
Pipeline.
Input
Endpoint
Src
Sink
Output
Endpoint
Src
Sink
Src
Sink
Src
Sink

9. 9
Overview – How does Kurento work?
The server is controlled with an RPC API.
●
Client applications manipulate Endpoints and Filters through this API.
●
Ready-made client SDKs for Java, Node.js and in-browser JavaScript.
Components of a Real-World application:
●
Kurento Media Server
●
Client Application – usually a server too.
●
User Interface – common case is a webpage.

10. 10
Overview – How does Kurento work?

11. Getting technical

12. 12
Getting technical – Component overview
Kurento Media Server (aka. “KMS”)
●
Core: Basic functionality and RPC API.
●
Elements: Endpoints used for input and
output of streams.
●
Filters: Implementation of all stream-
processing modules.

13. 13
Getting technical – KMS Elements
Protocols and Codecs Media Repository
WebRtcEndpoint
RtpEndpoint
HttpEndpoint
PlayerEndpoint
RecorderEndpoint

14. 14
Getting technical – KMS Elements
Different Endpoints for different needs:
●
WebRtcEndpoint – Full support of WebRTC standard.
•
Currently compatible with Chrome and Firefox (Safari and Edge are
work-in-progress).
•
All WebRTC lingo: SDP, (Trickle-)ICE, STUN, TURN, Google REMB.
●
RtpEndpoint – For RTP and SRTP streams.
•
Supports port auto-discovery as an alternative to ICE.
●
HttpEndpoint – Accepts GET/POST requests (eg. file uploading to KMS).

15. 15
Getting technical – KMS Elements
Special Endpoints:
●
PlayerEndpoint – Retrieves content from either of the file system, HTTP
servers, or RTSP sources. Input-only.
●
RecorderEndpoint – Redirects streams to storage. Output-only.

16. 16
Getting technical – KMS Filters
Process or transform data as it flows through filters.
●
Could be a simple transformation. Eg: Convert video to black and white.
●
Could be a complex task, involving external libraries. Eg: Apply
Computer Vision algorithms and extract features from the video.
●
Imagination is the limit.

17. 17
Getting technical – KMS Filters
Computer Vision Generic filter
Zbar
FaceDetector
PlateDetector
GStreamerFilter

18. 18
Getting technical – Pipelines
●
All elements get created and linked through specific RPC commands
from the Client Application.
●
As seen before, complex topologies are possible.

19. 19
Getting technical – Pipelines
SinkSink
SRCSRC
SinkSink
SRCSRCSinkSink
SinkSinkSRCSRC
WebRtcEndpoint
AR Filter
RecorderEndpoint
HttpGetEndpoint
RtpEndpoint
MP4 fle stored
in media
repository
Web application
using HTML5
<video> tag
RTP full duplex
client video
phone
WebRTC full
duplex client
video application
SinkSink

20. 20
Getting technical – Client Application
●
Implements or includes an RPC client.
●
Orchestrates the creation of the Pipeline, with Elements and Filters.
●
WebRtcEndpoint follows the standard way of configuration via SDP
Negotiation (aka. SDP Offer/Answer Model).
•
The Client Application is in charge of passing around ICE Candidates.
●
RtpEndpoint also uses SDP for configuration, but no ICE.
●
Other Endpoints have custom RPC methods for configuration.

21. 21
Getting technical – Client Application
●
Provides access to external sources such as FIWARE data-stores.
●
Proceeds with the rest of the business logic.
●
Kurento provides a FIWARE integration package for Java applications.
●
Also there are multiple example applications made available.

22. 22
Getting technical – Client Application
Application
Server
Application
Server
Media
Server
Media
ServerClientClient
Media
Negotiation
phase
11
Media channel
preparation
phase
(optional)
22
Media
exchange
phase
33
Specifc app logic
Specifc
pipeline
building
Pipeline
managed
media

23. A closer look

24. 24
A closer look – GStreamer media library
Kurento is powered under the hood by the GStreamer project:
https://gstreamer.freedesktop.org
GStreamer is in charge of all media handling:
●
Decoding / encoding of input / output video & audio.
●
Establishment of communication streams (eg. RTP Sessions).
●
Special interest: Agnostic bin.

25. 25
A closer look – Agnostic bin
Kurento abstracts codec compatibility issues between Endpoints.
●
Achieved through an “invisible” intermediate filter – ‘agnosticbin’.
●
Applies on-the-fly transcoding iff the codecs are not compatible.
Media Element Media Element
Agnostic media
adapter, “hidden”
behind every
connection
Agnostic media
adapter, “hidden”
behind every
connection
H.264VP8
Src
Sink
Src
Sink

26. 26
A closer look – SDP Offer/Answer
●
The standard signaling format for WebRTC.
●
Used to configure codecs & advanced settings in WebRtcEndpoint and
RtpEndpoint.
●
SDP is a mess and we all hate it (if you don’t yet, you should).
(The problem is, the IETF RTCWEB working group was/is dominated by
players from the SIP world who wanted to re-use their existing code.)
See: https://webrtchacks.com/sdp-anatomy/

27. 27
A closer look – SDP Offer/Answer
v=0
o=- 0 0 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=audio 9 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=sendonly
a=direction:active
a=ssrc:445566 cname:user@example.com
m=video 9 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=sendonly
a=direction:active
a=ssrc:112233 cname:user@example.com
v=0
o=- 372 372 IN IP4 192.168.1.15
s=Kurento Media Server
c=IN IP4 192.168.1.15
t=0 0
m=audio 41654 RTP/AVPF 96
a=rtpmap:96 opus/48000/2
a=recvonly
a=direction:passive
a=ssrc:3224 cname:user225@host-9b2
m=video 61134 RTP/AVPF 103
a=rtpmap:103 H264/90000
a=recvonly
a=direction:passive
a=ssrc:2717 cname:user225@host-9b2
SDP Offer to RtpEndpoint: SDP Answer from RtpEndpoint:

28. Application examples

29. 29
A closer look – Application example: RTP Player
Kurento RTP Player is a new application example, not yet published.
●
Receives an audio/video stream.
●
Redirects it and sends to a browser through WebRTC.
●
The RTP can originate from any source. A sample tool is provided.
External
RTP
producer

30. 30
A closer look – Application example: RTP Player
void start(String browserSdpOffer)
{
// Create and link Endpoints
MediaPipeline pipeline = kurento.createMediaPipeline();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
rtpEndpoint.connect(webRtcEndpoint);
// Configure the RtpEndpoint
String fakeSdpOffer = "...";
String kmsSdpAnswer = rtpEndpoint.processOffer(fakeSdpOffer);
String browserSdpAnswer = webRtcEndpoint.processOffer(browserSdpOffer);
webRtcEndpoint.gatherCandidates();
}

31. 31
A closer look – Application example: Magic Mirror
●
Receives a video stream from a browser.
●
Applies Computer Vision to detect a face and overlays a picture.
●
Sends back the modified video to the same Endpoint it came from.

32. 32
A closer look – Application example: Magic Mirror
void start()
{
// Create and link Endpoints, Filters
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
FaceOverlayFilter faceOverlayFilter =
new FaceOverlayFilter.Builder(pipeline).build();
faceOverlayFilter.setOverlayedImage("mario-hat.png");
webRtcEndpoint.connect(faceOverlayFilter);
faceOverlayFilter.connect(webRtcEndpoint);
webRtcEndpoint.gatherCandidates();
}

33. Closing

34. Thank you!
http://fiware.org
Follow @FIWARE on Twitter
Juan Navarro Moreno
Kurento Software Developer
jnavarro@naevatec.com
github.com/j1elo
www.kurento.org