Suggestion for a generic architecture of teleoperation system (remote control by video) for autonomous cars or drones. such a system requires transport of several video streams in extremely low latency

1. Teleportation
system
architecture

2. Requirements
• Multi (Camera) video streams*, multi resolution
• stream prioritization
• Adapt video streams to wireless channel/s*
– dynamic channel mapping
– Bitrate control per channel/camera (optional)
• Predict future bitrate
• Protect video
• Low latency
• *Terminology: Video stream, wireless channel Not 1:1 mapping2

3. Teleportation schematics
3

4. Camera
• Camera may provide different input to
main processing unit: Analog, SDI, USB
4

5. Processing unit
• Capture video
– Nano limited to 4K30fps capture on USB
– One MIPI interface
• Rescale and reorder to:
– 1 X 4K30fps stream (Jetson Nano)
– 2 X 4K60fps streams (Jetson Xavier)
• HEVC encoding according to streams
– Tile based
– Accept rate control commands in RT
5

6. Channel handler
• Dynamically map high priority camera to
low jitter channels
• Add per stream/channel FEC protection
• FEC amount varies dynamically
6

7. Channel controller
• Receives wireless channel feedback from
wireless channels:
– Packet drop
– Jitter
– Cellular params like RSSI
• Dynamically maps video streams to cellular
channels according to prioritization
• Decide on the amount of FEC per channel
• Predict future bitrate per channel and assign
bitrates per stream encoder
7

8. Player
• Controls jitter buffer
• Low latency decoding
• Fast marker rendering (webGL)
8

9. API overview
• Use Video-Conferenceing WebRTC-like API:
MySession = CreateSession(config)
Mapper = MySession.Map(Vidstreams,WirelessChannels)
Mapper.SetPriority(Config) #which channel has priority
Channel = MySession.AddChannel(config)
Channel.AddFEC(config) #static/dynamic scheme
Det1 = Stream.AddDetector(DL_Detector)