Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Tom Watzka & Satwant Singh
Lattice Semiconductor
Integrating Image, Radar, IR
and TOF Sensors:
Developing Vision Systems
w...
© 2018 MIPI Alliance, Inc. 2
Sensors are Proliferating
Lattice Semiconductor
SemicoResearchCorp.
0
100
200
300
2015 2016 2...
© 2018 MIPI Alliance, Inc. 3
Sensors in the Automotive Environment
Lattice Semiconductor
• Imaging cameras: surround view,...
© 2018 MIPI Alliance, Inc. 4
Sensors in the Drone Environment
Lattice Semiconductor
• Stereo Vision
• Ultrasonic (Sonar)
•...
© 2018 MIPI Alliance, Inc. 5
Sensors in the AR/VR Environment
Environmental Understanding Cameras
Depth Cameras
• Structur...
© 2018 MIPI Alliance, Inc. 6
• Region of Interest
• Down sampling
• Virtual Channels
VC1
VC2
Lattice Semiconductor
Sensors...
© 2018 MIPI Alliance, Inc. 7
Combining Identical Sensors
• Sync Sensors
• Buffer Lines
• Concatenate both lines at 2x freq...
© 2018 MIPI Alliance, Inc. 8
• Sensors not synchronized
• Different Clocks
• Output Frequency >= Input 1 + Input 2
• Lines...
© 2018 MIPI Alliance, Inc. 9
MIPI CSI-2SM
Virtual Channels (VC)
Lattice Semiconductor
© 2018 MIPI Alliance, Inc. 10
MIPI CSI-2SM v1.2/v1.3
MIPI CSI-2SM v2.0
KEY issue: difference in supported MIPI CSI-2SM
ver...
© 2018 MIPI Alliance, Inc. 11
Challenges – VC Fields
If there are no more than 4 input channels:
• Bridge can perform simp...
© 2018 MIPI Alliance, Inc. 12
Continuous Clock Mode:
• Simple Clocking Design
• High power
consumption
Non-Continuous Mode...
© 2018 MIPI Alliance, Inc. 13
Word Clock Domain N
Internal
Clock
Domain
Internal Clock Domain
D-PHY Clock Domain 0
Challen...
© 2018 MIPI Alliance, Inc. 14
Lattice Device
Down
Sample
MIPI CCISM
Manager
VC
Expand
VC
Combine
+
Frame ID
MIPI CCISM
ROI...
© 2018 MIPI Alliance, Inc. 15
Why FPGAs?
Lattice Semiconductor
Design RTL
Simulate
Synthesize
Debug
FPGA
Days to
Weeks
Dev...
© 2018 MIPI Alliance, Inc. 16
• MIPI Camera WG:
https://members.mipi.org/workgroup/join/154
• MIPI CSI-2SM
V2.1: https://m...
MIPI DevCon Seoul 2018: Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors
Nächste SlideShare
Wird geladen in …5
×

MIPI DevCon Seoul 2018: Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors

71 Aufrufe

Veröffentlicht am

In this presentation, Tom Watzka of Lattice Semiconductor Corp. provides an overview of the architectures and tradeoffs in mission-critical vision applications and the details of combining and tagging multiple data streams, including Camera Control Interface (CCI) integration.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

MIPI DevCon Seoul 2018: Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors

  1. 1. Tom Watzka & Satwant Singh Lattice Semiconductor Integrating Image, Radar, IR and TOF Sensors: Developing Vision Systems with Dissimilar Sensors
  2. 2. © 2018 MIPI Alliance, Inc. 2 Sensors are Proliferating Lattice Semiconductor SemicoResearchCorp. 0 100 200 300 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units 0 500 1000 1500 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units 0 10 20 30 2015 2016 2017 2018 2019 2020 2021 2022 Millionsof Units Automotive 27% CAGR Drone 27% CAGR VR/AR/MR 166% CAGR
  3. 3. © 2018 MIPI Alliance, Inc. 3 Sensors in the Automotive Environment Lattice Semiconductor • Imaging cameras: surround view, parking assistance and traffic sign recognition. • Lidar: Emergency braking and collision avoidance. • Short and medium range radar: Traffic alert and blind spot detection. • Longer range radar: Cruise control and early forward vehicle detection. Lots of Dissimilar Sensors
  4. 4. © 2018 MIPI Alliance, Inc. 4 Sensors in the Drone Environment Lattice Semiconductor • Stereo Vision • Ultrasonic (Sonar) • Time-of-Flight • Lidar • Infrared • Monocular Vision Lots of Dissimilar Sensors
  5. 5. © 2018 MIPI Alliance, Inc. 5 Sensors in the AR/VR Environment Environmental Understanding Cameras Depth Cameras • Structured Infrared light projector/scanner • RGB Depth cameras • Time-of-flight camera Lattice Semiconductor Lots of Dissimilar Sensors
  6. 6. © 2018 MIPI Alliance, Inc. 6 • Region of Interest • Down sampling • Virtual Channels VC1 VC2 Lattice Semiconductor Sensors in the Industrial Environment Cameras with Virtual Channel Capability
  7. 7. © 2018 MIPI Alliance, Inc. 7 Combining Identical Sensors • Sync Sensors • Buffer Lines • Concatenate both lines at 2x frequency Lattice Semiconductor Same “nominal” clock rate
  8. 8. © 2018 MIPI Alliance, Inc. 8 • Sensors not synchronized • Different Clocks • Output Frequency >= Input 1 + Input 2 • Lines sent out FIFO via Virtual Channels Virtual Channel Output Lattice Semiconductor Combining Dissimilar Sensors Different Data Rates
  9. 9. © 2018 MIPI Alliance, Inc. 9 MIPI CSI-2SM Virtual Channels (VC) Lattice Semiconductor
  10. 10. © 2018 MIPI Alliance, Inc. 10 MIPI CSI-2SM v1.2/v1.3 MIPI CSI-2SM v2.0 KEY issue: difference in supported MIPI CSI-2SM version • MIPI CSI-2SM v1.2/v1.3 - only supports up to 4 slaves (2-bit VC) • MIPI CSI-2SM v2.0 - can support up to 16 or 32 slaves (4-bit VC for MIPI CSI-2SM over D- PHY , 5-bit VC for MIPI CSI-2SM over C-PHY) Lattice Semiconductor Challenges – Virtual Channel (VC) Fields Different VC Mappings between Different MIPI CSI-2SM Versions
  11. 11. © 2018 MIPI Alliance, Inc. 11 Challenges – VC Fields If there are no more than 4 input channels: • Bridge can perform simple mapping to MIPI CSI-2SM V1.2. If there are more than 4 input channels: • Bridge can perform simple mapping to MIPI CSI-2SM V2.0. • If processor cannot support MIPI CSI-2SM V2.0: • Must merge similar sensors into one stream MIPI CSI-2SM V1.2 VC 1&2 Bridge AP/ISP Bridge AP/ISP Lattice Semiconductor MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1&2 MIPI CSI-2SM V1.2 VC 1,2,3,4 MIPI CSI-2SM V1.2 VC 1,2,3,4,5,6
  12. 12. © 2018 MIPI Alliance, Inc. 12 Continuous Clock Mode: • Simple Clocking Design • High power consumption Non-Continuous Mode: • Reference Clock Needed • Lower Power Consumption MIPI D-PHYSM Rx MIPI CSI-2SM MIPI CSI-2SM RefClk Continuous Mode Lattice Semiconductor Challenges – Clocking Continuous vs. Non Continuous Modes for MIPI CSI-2SM MIPI D-PHYSM Tx MIPI D-PHYSM Rx MIPI CSI-2SM MIPI CSI-2SM MIPI D-PHYSM Tx RefClk MIPI CSI-2SM Non-Continuous Mode MIPI CSI-2SM
  13. 13. © 2018 MIPI Alliance, Inc. 13 Word Clock Domain N Internal Clock Domain Internal Clock Domain D-PHY Clock Domain 0 Challenges – Multiple Clock Domains Lattice Semiconductor Input Buffer 0 FIFO : D-PHY Clock Domain N Input Buffer N FIFO Identical Sensors • Can use common reference clock • Separate Clock Domains • FIFO Resolves Meta-Stability • Internal Circuitry employs simple buffering adjusting for clock phasing Word Clock Domain 0D-PHY Clock Domain 0 Input Buffer 0 FIFO Sensor 0 Logic : D-PHY Clock Domain N Input Buffer N FIFO Internal Buffer 0 FIFO Internal Buffer N FIFO Sensor N Logic : Sensor 0 Logic Sensor N Logic Dissimilar Sensors (or cameras with different reference clocks) • First clock domain referenced to source clock • Second clock domain referenced to word clock. • Third clock domain referenced to internal clock
  14. 14. © 2018 MIPI Alliance, Inc. 14 Lattice Device Down Sample MIPI CCISM Manager VC Expand VC Combine + Frame ID MIPI CCISM ROI MIPI CSI-2SM Auto Image Adjust Optical VC1&2 IR Radar Lattice Semiconductor Virtual Channel Pre-Processing in Hardware Balancing Power and Performance MIPI CSI-2SM MIPI CSI-2SM MIPI CSI-2SM
  15. 15. © 2018 MIPI Alliance, Inc. 15 Why FPGAs? Lattice Semiconductor Design RTL Simulate Synthesize Debug FPGA Days to Weeks Dev Cycle Masks Fabrication Testing Qual Production ASIC FPGA ASIC Months to Years Dev Cycle • Product Development Cycle • Faster time to Market • Lower NRE • Enable Innovation • Size • Pin constrained – same as ASIC • Power • Largely a function of Interfaces Fewer and fewer designs are using ASICs
  16. 16. © 2018 MIPI Alliance, Inc. 16 • MIPI Camera WG: https://members.mipi.org/workgroup/join/154 • MIPI CSI-2SM V2.1: https://members.mipi.org/wg/All- Members/document/folder/11134 ADDITIONAL RESOURCES Lattice Semiconductor

×