Autonomous Vehicle Development with Unity

•

3 likes•1,360 views

In this intermediate-level session, Jose De Oliveira, who heads up Unity's Autonomous Vehicles Strategy, will reveal our roadmap to leverage the leading 3D real-time rendering engine for creating a simulation and visualization framework for automotive projects. In particular, he will focus on how Unity is ideal for supporting the development of autonomous vehicle technologies and processes. Jose De Oliveira - Unity Technologies

Automotive

“Autonomous Vehicles is a very easy technology to
develop. Self-driving cars are, in fact, a given.”
-- Nobody Ever

How can we increase the brainpower and
optimize the efforts applied to one of the
hardest problems in tech?

Jose De Oliveira
Lead Engineer for Autonomous Vehicles
Unity

Synthetic Environments for
Autonomous Vehicles
Development

Agenda
•  Context
•  Why simulate
•  Unity SimViz
•  SimViz Roadmap?
•  SimViz Announcement

Product Design
Sales & Marketing
Service & Training
HMI
Autonomous Vehicles
Automotive Areas

•  ML is core for AV and is
data-centric

•  Collecting data is expensive
•  Annotating data is both
expensive and error prone
•  Real-world data can only
prepare for what has already
happened
The autonomous vehicles data problem

Simulation can get autonomous
systems ready for unforeseen
scenarios and edge conditions in a
fast and cost effective way.
Why Simulate?

Sim.Viz (/simviZ/), noun
1.  Concatenation of the words “simulation” and "visualization" (plus
some alliteration for flair).

SimViz: !
•  Unity’s initiatives, features and products to support
simulation and visualization applications in the ﬁelds
of autonomous vehicles (AV) and beyond.!
!

External Data!
(HD Maps,V2X)!
Autonomous System
(Real or Simulated)!
Simulation
Environment!
Control!
Planning!
Perception!
Coordination!
Vehicle
Dynamics!
Environment!
Sensor
Model!
Parameters
(weather, time)!
Dynamic
Elements!
Static
Elements!
Scenarios!
Simulation systems

How can Unity help AV development
•  Incentivize development community: environments and
tools for simulations
•  Build AV and ML support into the platform
•  Promote an ecosystem of companies that extend the
platform and create simulation solutions

Licensed image from Adobe Stock

Asset Store Auto Packages
EasyRoads3D HD Pack (Andasoft)!
Enviro - Sky and Weather (Hendrik Haupt)!
https://assetstore.unity.com/lists/real-world-simulation-54066!

Sensors!
Content! Integration!
SimViz Solution Templates Wording
Sensor SDK!
Reference models!
Leverage 2018+ features!
Communication layer!
Performance gains!
Linux improvements!
ML Agents!
Support to open formats!
Pipelines for data ingestion!
Dynamic elements models!
Starter content!

Unity ML Agents Workflow
Set Up Game
for Training!
Train
Agents!
Embed
Agents!

Q4! Q1! Q2! Q3! Q4!
SimViz Reference
Package!
•  Sensor template!
•  Content template!
ROS
Integration!
Platform Integration!
•  DxM (Data Exchange
Module)!
•  OpenDrive Support!
•  Sensor Ready Materials!
Scalable Deployment!
•  Headless runtime!
•  AI based pedestrian
model!
•  Sensor Validation
Standards!
•  ML agent integration!
SimViz Solution Roadmap
2018! 2019!

SimViz Solution Template: Environments
Artificial Scenes included: !
o  City streets
o  Open Highway
o  Residential streets

SimViz Solution Template: Sensors
Sensor Models included:!
o  RGB-D Camera
o  LiDAR
o  Radar
o  IMU
o  GPS
SynCity Lite

Integration of Scenes and Sensor Templates

What's hot

Deep Learningによる超解像の進歩

Hiroto Honda

20210711 deepI2P

Takuya Minagawa

【DL輪読会】Vision-Centric BEV Perception: A Survey

Deep Learning JP

3D CNNによる人物行動認識の動向

Kensho Hara

SOTAのvision modelは35億のインスタの弱ラベル付き画像を使用しているが，ラベル無し画像は使われていない．そこで，ラベル無し画像を追加で使用して精度の向上を図る．学習は以下の手順で行う． 1. 教師モデルをラベル付きの画像で学習 2. 教師モデルを使ってラベルなし画像の疑似ラベルを生成 3. 生徒モデルをラベル付き画像と擬似ラベル付き画像を用いて学習 4. 3で学習した生徒モデルを教師モデルとして2,3を数回繰り返すこの時，疑似ラベルの生成時にはノイズを加えず、生徒モデルの学習時にはノイズを加える．結果として，ImageNetのSOTAを1%更新．ImageNet-A,C,Pでロバスト性の向上を確認した．

Self training with noisy student

harmonylab

Masked Autoencoders Are Scalable Vision Learners

GuoqingLiu9

物体検出の歴史（R-CNNからSSD・YOLOまで）

HironoriKanazawa

【チュートリアル】動的な人物・物体認識技術 -Dense Trajectories-

Hirokatsu Kataoka

【DL輪読会】"Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware"

Deep Learning JP

【メタサーベイ】Vision and Language のトップ研究室/研究者

cvpaper. challenge

動作認識の最前線：手法，タスク，データセット

Toru Tamaki

Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry...

Masaya Kaneko

Face Quality Assessment 顔画像品質評価について

Plot Hong

文献紹介：VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Toru Tamaki

Transformer 動向調査 in 画像認識(修正版)

Kazuki Maeno

HELLO AI WORLD - MEET JETSON NANO

NVIDIA Japan

スタディサプリを支えるデータ分析基盤～設計の勘所と利活用事例～

Tetsuo Yamabe

NVIDIA Jetson導入事例ご紹介

NVIDIA Japan

Super resolution-review

Woojin Jeong

ngx_mrubyを用いた柔軟なABテスト基盤

Recruit Lifestyle Co., Ltd.

What's hot (20)

Deep Learningによる超解像の進歩

20210711 deepI2P

【DL輪読会】Vision-Centric BEV Perception: A Survey

3D CNNによる人物行動認識の動向

Self training with noisy student

Masked Autoencoders Are Scalable Vision Learners

物体検出の歴史（R-CNNからSSD・YOLOまで）

【チュートリアル】動的な人物・物体認識技術 -Dense Trajectories-

【DL輪読会】"Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware"

【メタサーベイ】Vision and Language のトップ研究室/研究者

動作認識の最前線：手法，タスク，データセット

Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry...

Face Quality Assessment 顔画像品質評価について

文献紹介：VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Transformer 動向調査 in 画像認識(修正版)

HELLO AI WORLD - MEET JETSON NANO

スタディサプリを支えるデータ分析基盤～設計の勘所と利活用事例～

NVIDIA Jetson導入事例ご紹介

Super resolution-review

ngx_mrubyを用いた柔軟なABテスト基盤

Similar to Autonomous Vehicle Development with Unity

Join us to see how Public-sector organizations and AWS Partners are combining Smart Devices and Artificial Intelligence to create flexible, secure and cost-effective solutions. Applying machine learning models to live video/audio, cameras can be transformed into flexible IoT devices that perform critical functions around public safety, security, property management, smart parking & environmental management. Learn how these solutions are architected using AWS services such as AWS IoT Core, AWS GreenGrass, AWS DeepLens, Amazon SageMaker and Amazon Alexa.

Machine Learning at the Edge

Amazon Web Services

VIA Mobile360 Surround View: Smart Transportation

VIA Embedded

Retaam_ThingWorx

Khaled Shaikh

Digital Transformation. Examples from Automotive Industry

Boost40

Over the next decade, accelerating autonomous driving technology—including advances in artificial intelligence, sensors, cameras, radar and data analytics—are set to transform how we commute. In this session, you learn how to use Amazon AI for a highly productive, on demand, and scalable autonomous driving development environment. We compare the most popular AI frameworks including TensorFlow and MXNet for use in autonomous driving workloads. You learn about the AWS optimizations on MXNet that yield near linear scalability for training deep neural networks and convolutional neural networks. We demonstrate the ease of getting started on AWS AI by using a sample training dataset for building an object detection model on AWS. This session is intended for audiences who have some exposure to the underlying concepts for AI-based autonomous driving development. After attending the session, you can get started with AI development on AWS by using a sample dataset for building an object detection model.

AMF305_Autonomous Driving Algorithm Development on Amazon AI

Amazon Web Services

CloudsourceIT Company Profile 180912

Jorge Fco. Suárez Basáñez

Iron yun Video Analytics VMS & AI Solutions

Niranjan Gadag

Citrix application infrastructures are very performance sensitive. “Citrix is slow” or “Citrix is not working” is a common complaint heard at the help desk from frustrated users. These complaints could be caused from a small problem anywhere in your infrastructure and you have to spend hours finding out where the real problem lies before you can resolve it and restore a positive user experience. Is it really a Citrix issue, or is the issue actually originating somewhere else in the infrastructure – i.e., the network, application, virtual platform, storage, etc.? View these slides and discover how you can extend Microsoft System Operations Center (SCOM) – using the Microsoft SCOM Citrix Universal Management Pack - to monitor and manage Citrix infrastructures end to end, so when a user complains that Citrix is slow, you can pinpoint exactly where the cause of the problem lies — in just one click. Learn how to: • Monitor all of your Citrix tiers – XenApp, XenDesktop, XenServer, NetScaler, XenMobile, etc. directly from the Microsoft SCOM console • Get deep visibility into every aspect of Citrix performance • Pinpoint in just one click where the real cause of a problem lies • Proactively detect and fix performance issues before users complain • Publish real-time dashboards in Microsoft SCOM to provide key insights for the different stakeholders in your organization • Generate powerful, end-to-end historical and trend reports that help you optimize and right-size your Citrix infrastructure for maximum ROI

Monitor & Manage Citrix App Performance Using Microsoft SCOM

eG Innovations

Platinum Keynote - The Expanding Role of the OSGi Alliance - C Hayman

mfrancis

This session provides an overview of the AWS Digital Media Partner ecosystem and explains how APN partners, including ISVs, SIs and solution providers, leverage the AWS Cloud to enable media workloads to be executed in the cloud. Included is an overview of the AWS media workflow categories and a discussion of the alignment of these partners to these categories as well as to the sub-vertical industries that comprise Media and Entertainment.

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Amazon Web Services

Today, in 2022, Apache Kafka is the central nervous system of many applications in various areas related to the automotive and manufacturing industry for processing analytical and transactional data in motion across edge, hybrid, and multi-cloud deployments. This presentation explores the automotive event streaming landscape, including connected vehicles, smart manufacturing, supply chain optimization, aftersales, mobility services, and innovative new business models. Afterwards, many real-world examples are shown from companies such as Audi, BMW, Porsche, Tesla, Uber, Grab, and FREENOW. More detail in the blog post: https://www.kai-waehner.de/blog/2022/01/12/apache-kafka-landscape-for-automotive-and-manufacturing/

Apache Kafka Landscape for Automotive and Manufacturing

Kai Wähner

Functional AI and Pervasive Networking in Automotive

Alison Chaiken

Software defined vehicles,automotive standards (safety, security), agile cont...

Dr. Anish Cheriyan (PhD)

Jari P. Ängeslevä - Ai the new kid in the block

Eficode

Enterprise Integration Patterns Revisited (again) for the Era of Big Data, In...

Kai Wähner

The 4 d space of future automotive mobility; klaus schaaf @ yotr

Year of the X

IoT and mobile computing are at the heart of many organizations’ digital transformation initiatives. Together, these technologies are enabling enterprises to innovate new business models and solutions that enhance customer and partner interactions, optimize supply chain logistics, and improve operational efficiency. Implementing a scalable and extensible IoT architecture is key for this transformation. WSO2 IoT Server offers an extensible framework for device manufacturers to write plugins for various types of devices they wish to integrate with. Data gathered from devices can be processed with batch and real-time analytics and this data can even be fed into a prediction engine. Devices can be exposed as managed APIs for integration. On top these, it offers a complete enterprise mobility management (EMM/MDM/MAM) solution for enterprises looking to manage employees’ mobile devices. This talk will give you an introduction to the key capabilities of WSO2 IoT Server and discusses how it can be extended to build a scalable IoT architecture.

WSO2Con USA 2017: Building Enterprise Grade IoT Architectures for Digital Tra...

WSO2

Brochure_210x280_Corporate_2016_PRINT

Sunil Agrawal

It's high time for augmented reality to be brought to a wider audience. In ABB, we know that it is not just a gimmick any more. However, with every innovative technology comes new challenges. In these slides, we show how to overcome them and deliver valuable products with Hololens and Unity. Speakers: Maciej Włodarczyk - ABB Rafał Kielar - ABB Watch the session on YouTube: https://youtu.be/QFsj8Pi_3Ho

How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...

Unity Technologies

Microsoft Automotive Business Unit - J Za

mfrancis

Similar to Autonomous Vehicle Development with Unity (20)

Machine Learning at the Edge

VIA Mobile360 Surround View: Smart Transportation

Retaam_ThingWorx

Digital Transformation. Examples from Automotive Industry

AMF305_Autonomous Driving Algorithm Development on Amazon AI

CloudsourceIT Company Profile 180912

Iron yun Video Analytics VMS & AI Solutions

Monitor & Manage Citrix App Performance Using Microsoft SCOM

Platinum Keynote - The Expanding Role of the OSGi Alliance - C Hayman

[AWS LA Media & Entertainment Event 2015]: M&E Ecosystem Update Q4 2015

Apache Kafka Landscape for Automotive and Manufacturing

Functional AI and Pervasive Networking in Automotive

Software defined vehicles,automotive standards (safety, security), agile cont...

Jari P. Ängeslevä - Ai the new kid in the block

Enterprise Integration Patterns Revisited (again) for the Era of Big Data, In...

The 4 d space of future automotive mobility; klaus schaaf @ yotr

WSO2Con USA 2017: Building Enterprise Grade IoT Architectures for Digital Tra...

Brochure_210x280_Corporate_2016_PRINT

How ABB shapes the future of industry with Microsoft HoloLens and Unity - Uni...

Microsoft Automotive Business Unit - J Za

More from Unity Technologies

Build Immersive Worlds in Virtual Reality

Unity Technologies

Augmenting reality: Bring digital objects into the real world

Unity Technologies

Let’s get real: An introduction to AR, VR, MR, XR and more

Unity Technologies

Using synthetic data for computer vision model training

Unity Technologies

When it comes to emerging technology, Forrester found that “94% of those who have implemented real-time 3D are expanding their investment.” Wonder why? Learn more in this webinar featuring guest speaker Paul Miller, a principal analyst at Forrester. He covers the key findings of a commissioned study conducted by Forrester Consulting on behalf of Unity, published in March 2020. Learn more: https://on.unity.com/2Yz49kg Watch the webinar: https://on.unity.com/3aYGlsF

The Tipping Point: How Virtual Experiences Are Transforming Global Industries

Unity Technologies

Unity Roadmap 2020: Live games

Unity Technologies

Unity Roadmap 2020: Core Engine & Creator Tools

Unity Technologies

Unity developed a new architecture that improves the support for existing and future augmented reality (AR) and virtual reality (VR) platforms. Learn about the technology under the hood, the consequent benefits, and improvements to the platform, and how it impacts your workflows in creating AR/VR experiences. Speakers: Mike Durand, Matt Fuad - Unity Watch the session on Youtube: https://youtu.be/Stqk1GxlSK0

Unity XR platform has a new architecture – Unite Copenhagen 2019

Unity Technologies

Autodesk and Unity announced a collaboration last year to streamline workflows and enable seamless development across the AEC design, build and operate lifecycles. This fall, Unity Reflect launches, giving designers, architects, and engineers the ability to seamlessly federate their Revit models for real-time 3D. Andrew Sullivan - Digital Delivery Manager, SHoP Architects will provide an overview of how they are using the product to enable real-time decision making, reduce the time between revisions and meetings, and ultimately improve design review and construction planning processes. Recording available here: https://youtu.be/qe0yxHA0fHI

Turn Revit Models into real-time 3D experiences

Unity Technologies

Daimler Protics implemented mixed and augmented reality on mobile devices and used the Microsoft HoloLens for automotive production, training, and marketing. Discover the challenges Daimler Protics faced and the Unity solutions that eased the mixed reality implementation. Speakers: Daniel Keßelheim - Daimler Protics Sebastian Rigling - Daimler Protics Session available here: https://youtu.be/fTc1c8iTGqU

How Daimler uses mobile mixed realities for training and sales - Unite Copenh...

Unity Technologies

Hear from Volvo's lead Unity developer, Timmy Ghiurau, about how he broke new ground by bringing technology forged in gaming into one of the leading brands in the automotive industry. Timmy will share how he used his gaming background to inspire people across his large organization to adopt Unity and embrace real-time 3D as a way of working. Timmy Ghiurau - Volvo Session available here: https://youtu.be/CD4Go3Uv5Uc

How Volvo embraced real-time 3D and shook up the auto industry- Unite Copenha...

Unity Technologies

Are you involved in testing or QA on projects in Unity? In these slides, you'll get an overview of the state of Unity for all things testing-related, and have the opportunity to share your stories of success, failure, pain, and glory. Learn from your fellow developers and give feedback on how Unity could help you hold your projects to a higher standard of quality. You will also get an introduction to the newest features in the Test Framework. Speakers: Christian Warnecke - Unity Richard Fine - Unity Watch the session on YouTube: https://youtu.be/wTiF2D0_vKA

QA your code: The new Unity Test Framework – Unite Copenhagen 2019

Unity Technologies

From buildings and infrastructure to industrial machinery and factories, digital twins are becoming integral across the industrial sector. In this webinar, first shown on Engineering.com, leaders from Unity and Unit040, provider of digital twin platform Prespective, share how digital twins add value at all stages of the project and product lifecycle, from the early stages of design to predictive maintenance using IoT data. Watch the webinar here: create.unity3d.com/real-time-3d-and-digital-twins

Engineering.com webinar: Real-time 3D and digital twins: The power of a virtu...

Unity Technologies

Major automotive brands like Volkswagen are leveraging the power of virtual reality to create immersive training programs that can be delivered across multiple global locations at the same time. Learn how to scale the production and distribution of real-time VR training in enterprise. Speakers: Thomas Wimmer - Innoactive Andreea Raducan - Innoactive Watch the session on YouTube: https://youtu.be/5DNFUTfyOEc

Supplying scalable VR training applications with Innoactive - Unite Copenhage...

Unity Technologies

Augmented reality (AR), virtual reality (VR), and mixed reality (MR) – collectively known as XR – are making inroads in the automotive industry. Join this session led by Visionaries 777, which works with major auto brands like INFINITI, to learn about the range of immersive experiences you can build with Unity to create a better customer experience that results in more engagement and sales. Speakers: David Castañeda - Visionaries 777 Frantz Lasorne - Visionaries 777 Session available here: https://youtu.be/WJpeWHGXyms

XR and real-time 3D in automotive digital marketing strategies | Visionaries ...

Unity Technologies

Real-time CG animation in Unity: unpacking the Sherman project - Unite Copenh...

Unity Technologies

The developers of Varjo VR-1 learned a lot about human eye resolution and the demands it puts on virtual reality (VR) content. In these slides, you'll explore what next-generation VR can mean for your VR experiences. Learn about what matters the most when it comes to visual quality, the possible caveats, and the role performance requirements play in this equation. Speaker: Mikko Strandborg - Varjo

Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1 - Unite Cop...

Unity Technologies

Unity is enabling film and animation studios to revolutionize their pipelines with features developed specifically to empower storytellers who are creating linear and interactive content. Learn more about features such as Python, Shotgun, the Arbitrary Output Variables (AOV) used in Recorder for export, Alembic, and Universal Scene Description (USD). Speaker: Mathieu Muller - Unity Watch the session on YouTube: https://youtu.be/wrc3R-BoDGs

What's ahead for film and animation with Unity 2020 - Unite Copenhagen 2019

Unity Technologies

VR allows for an entirely new level of immersion, leading to more thrilling and engaging content to be delivered and is growing rapidly. Despite this, VR, especially on mobile, currently contains a number of limitations, which can make it an unrealistic, unconvincing and, sometimes, an uncomfortable experience. Virtual reality (VR) is a new way to deliver thrilling and engaging content and allows for a deep level of immersion. Despite this, VR, especially on mobile, currently has several limitations, which can make it an unrealistic, unconvincing and, sometimes, an uncomfortable experience. To achieve the true potential of VR, these limitations must be either solved or mitigated. Ways of mitigating these limitations include optimal alpha compositing approaches, texture filtering techniques and bump mapping methods for use with VR content. In these slides, technology company Arm will outline how to improve the rendering quality of your VR content, describing the most common pitfalls and bad practices, before providing clear examples and mitigation solutions of how to best overcome them. Speaker: Ryan O'Shea - ARM

How to Improve Visual Rendering Quality in VR - Unite Copenhagen 2019

Unity Technologies

From buildings and infrastructure to industrial machinery and factories, digital twins are becoming integral revisualization tools across the industrial sector. Learn how Unit040, a company specializing in visualization and simulation, creates digital twins that combine real-time 3D technology with BIM, CAD and CAE systems to add value at all stages of the building and product lifecycle, from the early design phase to predictive maintenance using Internet of Things (IoT) data. Speakers: Pieter Weterings - Unit040 Guido van Gageldonk - Unit040 Watch the session on YouTube: https://youtu.be/j4i14p89h_s

Digital twins: the power of a virtual visual copy - Unite Copenhagen 2019

Unity Technologies

More from Unity Technologies (20)