Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Developing applications with Kurento

3.949 Aufrufe

Veröffentlicht am

This presentation was used in the context of a FI-WARE webminar for introducing Kurento. Kurento is a framework for building multimedia and streaming applications based on predefined blocks. Send and receive median through RTP, WebRTC, HTTP and RTSP. Use processing for making face detection, plate recognition or object tracking. Use augmented reality, group communications or media mixing and blending among others. During the webminar, we used Kurento APIs for showing how to create media applications for videoconferencing or video streaming in a simple and seamless manner. We also demonstrated how these applications can be enriched with Kurento's advanced processing capabilities.

Veröffentlicht in: Internet, Technologie, Bildung
  • Als Erste(r) kommentieren

Developing applications with Kurento

  1. 1. Open APIs for Open Minds Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento
  2. 2. Mul$media  infrastructures  for  the   Future  Internet   2 Media  is   here   Media  got   there   Media  got   there   Analyze   Transform   Store   Transport   Enrich   Augment   Adapt   Sensors   Context   Events   Media  is   here  
  3. 3. Enrich,  augment,  adapt,  analyze,   transform,  store:  what’s  the  problem?     3 Complexity  
  4. 4. Future   Internet   Mul$media   Infrastructure   Simple   Development   APIs   The  FI-­‐WARE  Stream-­‐oriented  Generic   Enabler  Implementa$on   4
  5. 5. 5 • Interoperable  media  exchange  (mul$plaNorm/mul$protocol)   • WebRTC,  RTP,  HTTP  (video  tag),  etc.   • Process  media  (Computer  vision,  augmented  reality,  media  indexing,  etc.)   • Media  and  metadata  recording  and  recovery   • Transform  and  adapt  media  (H.264,  H.263,  VP8,  Ogg,  and  others)   • Media  rou$ng  and  mixing   • Etc.   Provides  mul$media  capabili$es  to  the  FI-­‐WARE  infrastructure   •  REST  API   •  JavaScript  API   •  Java  API   Exposes  those  capabili$es  through  a  simple  to  use  APIs   •  LGPL  2.1   Is  distributed  through  a  flexible  FOSS  license   Kurento  
  6. 6. Kurento  Media  Server  (KMS):  the   nucleus  of  Kurento   6 •  KMS  is  a  middleware  for  media  streams   – Receives  the  stream   – Process  the  stream   – Issues  the  stream   Send   Receive   Analyze   Augment   Enrich   Transform   Transcode   Record   Process   Replicate   Media   Source   Media   Sink   KMS  
  7. 7. The  Media  API:  The  API  for  accessing   KMS  capabili$es   7 Send   Receive   Analyze   Augment   Enrich   Transform   Transcode   Record   Process   Replicate   Media   Source   Media   Sink   KMS   Java   Media  API   JavaScript   Media  API   Other   languages   Applica$ons  define  the  processing   of  streams  geang  through  KMS  
  8. 8. Media  API:  Media  Elements  and  Media   Pipelines   8 Sink%SRC% Sink% SRC% SRC%Sink% Sink% § Media Element • Provides a specific media functionality › Send/receive media › Process media › Transform media • Exchange media through › Sources › Sinks § Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media  Element   Sink   SRC  
  9. 9. Developers  create  applica$ons  just   connec$ng  Media  Elements   9 Protocols( &(Codecs( Computer(( Vision( Augmented(( Reality( Mul9sensory(( Mul9media( Media(( Repository( IPTV( Integra9on( Group( Communica9ons( Sink% H>pEndpoint( Sink%SRC% RtpEndpoint( SRC$ DataChannelEndpoint( Sink%SRC% WebRtcEndpoint( Sink% SRC% BarCodeReader( Sink% SRC% PointerTracker( Sink% SRC% FaceDetector( Sink% SRC% FaceOverlay( Sink% SRC% ChromaFilter( Sink% SRC% HeartRateBlender( Sink% RecorderEndpoint( SRC$ PlayerEndpoint( Sink% CdnUpload( Sink%IPTVConnector( Mixer( Sink%SRC% Sink% SRC% Sink% SRC%Sink% Sink% Applica$on  2   Sink% SRC% Sink% SRC%Sink% Sink%SRC% Sink% Applica$on  3   Sink%SRC% SRC%Sink% SRC% Sink% Sink%SRC% Applica$on  1   Toolbox  of  media  elements  
  10. 10. Media  API:  trivial  example  (Java)   MediaPipeline  mp  =  contentSession.getMediaPipelineFactory().create();     PlayerEndpoint  playerEndpoint  =  mp.newPlayerEndpoint(file:///myFile.webm).build();     H2pGetEndpoint  hepEndpoint  =  mp.newHepGetEndpoint().terminateOnEOS().build();     playerEndpoint.connect(hepEndpoint);     hepEndpoint.getUrl();  //URL  where  the  media  is  made  available   10 Media  Pipeline   HepGetEndpoint   Media  from   file  or  URI   HTTP  media   streaming   Sink   SRC   PlayerEndpoint  
  11. 11. Is  that  enough?  Think  about  the   WWW  development  model   11 Process  WWW  request   -­‐  DDBB  access   -­‐  Authen$ca$on   -­‐  XML  processing   -­‐  Etc.   HTTP  request:     I  want  this  resource   HTTP  response:   The  resource   •  Intui$on  behind  the  WWW  model     – Client  asks  what  it  wants   – Server  side  APIs  execute  the  associated  processing  
  12. 12. We  need  an  equivalent  model:  the   role  of  the  Signaling  Plane   •  The  API  must  provide  nego$a$on  capabili$es   –  I  want  “this  media”  …   •  Iden$fica$on  of  the  media  to  exchange   –  File  in  hard-­‐drive,  IP  camera,  user,  etc.   –  in  “this  way”  …   •  Iden$fica$on  of  the  processing  of  media   –  Augmented,  analyzed,  etc.   –  with  “this  format”  …   •  Quality   –  Codec,  screen-­‐size,  frame-­‐rate,  etc.   –  at  “this  moment”   •  Stream  control   –  Play,  stop,  start,  pause,  etc.   12
  13. 13. Don’t  get  it?  think  about  WWW   development  again  …   13 Process  WWW  request   -­‐  DDBB  access   -­‐  Authen$ca$on   -­‐  XML  processing   -­‐  Etc.   HTTP  request:     I  want  this  resource   HTTP  response:   The  resource   Process  media  request   -­‐  Media  API   -­‐  DDBB  access   -­‐  Authen$ca$on   -­‐  XML  processing   -­‐  Etc.   Signaling  request:     I  want  this  media   Signaling  response:   The  media  is  here   Intui$on  behind     tradi$onal  WWW     Applica$ons     (Servlets,  ASP,  PHP,     Rails,  etc.)   Intui$on  behind     Kurento     development  APIs:   Mul$media  RTC  is  just   another  feature  of  your   applica$on  
  14. 14. Dealing  with  the  signaling:  The   Content  Handler   14 KMS   Sink  SRC   Sink   SRC   Sink   SRC  Sink   Media  API   REST  API  (Open  API  protocol)   The  Content  Handler   Equivalent  to  a  Servlet/ASP/PHP  script   -­‐  When  receiving  “this  request”…   -­‐  execute  “this  logic”   Developer  can  use  the  media  API   Code  building  the  media  pipeline  and   execu$ng  the  applica$on  logic  the   developer  wants  
  15. 15. Kurento  Architecture   15 Kurento  Media  Server  (KMS)   Receive   Video   Augmented   Reality   Send   Video   Computer     Vision   Video  Playing   and  Recording   Java  EE  compaJble  container   HTTP     Servlet   SIP   Servlet   Web   services   Kurento     REST  API   Specific  handler  implementa$ons   Signaling  and     WWW  traffic   Media   Media   Signaling  and     WWW  traffic   Media  API   DD.BB.   Kurento  ApplicaJon  Server  (KAS)   Other  java   APIs.  
  16. 16. Applica$on  execu$on  flow   Client   Code   Applica$on   Server  (KAS)   Media   Server  (KMS)   I  want  this  media  in  this  way  …   (JSON)   Commands  reques$ng   the  crea$on  of  a  pipeline   What  you  want  is  here  …   (JSON)   Media     negoJaJon   phase   Media     exchange   phase   1   2   Specific  applica$on   logic  at  the     server-­‐side   (Content  Handler)   Media   pipeline   creaJon   Media  exchange  between  client  and  server  
  17. 17. Content  Handler:  trivial  example   @H2pPlayerService(path  =  "/player”)   public  class  MyPlayerHandler  extends  HepPlayerHandler  {       @Override   public  void  onContentRequest(HepPlayerSession  contentSession)  {    //Create  the  pipeline  for  providing  media  through  HTTP   }       @Override   public  void  onContentStarted(HepPlayerSession  contentSession)  {    //Media  started  flowing,  you  can  execute  addi$onal  ac$ons   }     @Override   Public  void  onSessionTerminated(HepPlayerSession  contentSenssion){    //Media  exchange  termianted,  you  can  collect  your  resources   }    
  18. 18. Let’s  develop  with  Kurento   •  What  you  need   –  A  Kurento  instance   •  You  can  install  your  own  Kurento  instance   •  You  can  launch  a  Kurento  instance  at  the  FI-­‐LAB   –  hep://lab.fi-­‐ware.org     •  Geang  help   –  FI-­‐WARE  catalog  entry   •  hep://catalogue.fi-­‐ware.org/enablers/stream-­‐oriented-­‐kurento     –  Installa$on  guide   •  heps://forge.fi-­‐ware.org/plugins/mediawiki/wiki/fiware/index.php/ StreamOriented_-­‐_Installa$on_and_Administra$on_Guide     –  Developer  guide   •  heps://forge.fi-­‐ware.org/plugins/mediawiki/wiki/fiware/index.php/ StreamOriented_-­‐_User_and_Programmers_Guide     –  Kurento  web  site   •  hep://www.kurento.org     18
  19. 19. Kurento  Hello  World:  Playing  a  file   19 Media  Pipeline   HepGetEndpoint   Media  from   file  or  URI   HTTP  media   streaming   Sink   SRC   PlayerEndpoint   Media  API   REST  API  (Open  API  protocol)   Create  the  pipeline  connec$ng:   HepGetEndpoint   PlayerEndpoint   I  want     “this  media”  
  20. 20. Playing  a  file:  Handler  code   @HepPlayerService(path  =  "/player”)   public  class  MyPlayerHandler  extends  HepPlayerHandler  {     @Override   public  void  onContentRequest(HepPlayerSession  contentSession)  throws  Excep$on  {   MediaPipeline  mp  =  contentSession.getMediaPipelineFactory().create();   contentSession.releaseOnTerminate(mp);   PlayerEndpoint  playerEndpoint  =  mp.newPlayerEndpoint(   "h2p://media.w3.org/2010/05/sintel/trailer.webm").build();   contentSession.setAeribute("player",  playerEndpoint);   H2pGetEndpoint  h2pEndpoint  =  mp.newH2pGetEndpoint().terminateOnEOS().build();   playerEndpoint.connect(h2pEndpoint);   contentSession.start(hepEndpoint);   }     @Override   public  void  onContentStarted(HepPlayerSession  contentSession)  {   PlayerEndpoint  playerEndpoint  =  (PlayerEndpoint)  contentSession.getAeribute("player");   playerEndpoint.play();   }   }     20 Source:  heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerHandler.java    
  21. 21. Playing  a  file:  client  code   21 <!DOCTYPE  html>   <html>   <head>   <script  src="./js/kws-­‐content-­‐api.js"></script>   <script>   var  conn;   func$on  start()  {   var  op$ons  =  {    remoteVideoTag  :  "remoteVideo"   };   conn  =  new  kwsContentApi.KwsContentPlayer("./player",  op$ons);   }   func$on  terminate()  {    conn.terminate();    }   </script>   </head>   <body>   <bueon  onclick="start();">Start</bueon>   <bueon  onclick="terminate();">Terminate</bueon>   <br  />   <video  id="remoteVideo"  autoplay></video>   </body>   </html>   Source:  heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/webapp/player.html    
  22. 22. Media  Pipeline   Adding  Computer  Vision   22 HepGetEndpoint   Media   from   file  or     URI   HTTP  media   streaming   Sink   SRC   PlayerEndpoint   SRC   Sink   JackVaderFilter   Media  API   REST  API  (Open  API  protocol)   Create  the  pipeline  connec$ng:   HepGetEndpoint,  Filter  and   PlayerEndpoint   I  want     “this  media”  
  23. 23. Adding  Computer  Vision:  Handler  code   23 @HepPlayerService(path  =  "/playerWithFilter”)   public  class  MyPlayerHandler  extends  HepPlayerHandler  {     @Override   public  void  onContentRequest(HepPlayerSession  contentSession)  throws  Excep$on  {   MediaPipeline  mp  =  contentSession.getMediaPipelineFactory().create();   contentSession.releaseOnTerminate(mp);   PlayerEndpoint  playerEndpoint  =  mp.newPlayerEndpoint(   "h2p://media.w3.org/2010/05/sintel/trailer.webm").build();   contentSession.setAeribute("player",  playerEndpoint);   JackVaderFilter  filter  =  mp.newJackVaderFilter().build();   H2pGetEndpoint  hepEndpoint  =  mp.newHepGetEndpoint().terminateOnEOS().build();   filter.connect(hepEndpoint);   playerEndpoint.connect(filter);   contentSession.start(hepEndpoint);   }     @Override   public  void  onContentStarted(HepPlayerSession  contentSession)  {   PlayerEndpoint  playerEndpoint  =  (PlayerEndpoint)  contentSession.getAeribute("player");   playerEndpoint.play();   }   }     Source:   heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java    
  24. 24. Adding  Computer  Vision:  Client  code   24 <!DOCTYPE  html>   <html>   <head>   <script  src="./js/kws-­‐content-­‐api.js"></script>   <script>   var  conn;   func$on  start()  {   var  op$ons  =  {    remoteVideoTag  :  "remoteVideo"   };   conn  =  new  kwsContentApi.KwsContentPlayer("./playerWithFilter",  op$ons);   }   func$on  terminate()  {    conn.terminate();    }   </script>   </head>   <body>   <bueon  onclick="start();">Start</bueon>   <bueon  onclick="terminate();">Terminate</bueon>   <br  />   <video  id="remoteVideo"  autoplay></video>   </body>   </html>   Source:  heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/webapp/playerFilter.html      
  25. 25. Adding  Computer  Vision:  Client  code   25 <!DOCTYPE  html>   <html>   <head>   <script  src="./js/kws-­‐content-­‐api.js"></script>   <script>   var  conn;   func$on  start()  {   var  op$ons  =  {    remoteVideoTag  :  "remoteVideo"   };   conn  =  new  kwsContentApi.KwsContentPlayer("./playerWithFilter",  op$ons);   }   func$on  terminate()  {    conn.terminate();    }   </script>   </head>   <body>   <bueon  onclick="start();">Start</bueon>   <bueon  onclick="terminate();">Terminate</bueon>   <br  />   <video  id="remoteVideo"  autoplay></video>   </body>   </html>   Source:  heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/webapp/playerFilter.html      
  26. 26. 26 Source:  heps://github.com/Kurento/kmf-­‐tutorial/blob/master/src/main/webapp/playerFilter.html       Thanks  !   •  hep://fi-­‐ppp.eu   •  hep://fi-­‐ware.org     •  hep://ww.kurento.org   •  Follow  @Fiware  on  Twieer  !   •  Follow  @Kurentoms  on  Twieer  !