[SP02] Developing autonomous vehicles with AirSim

1. Part I: AI and Simulation Ashish Kapoor

2. Introducing Microsoft Research

3. Aerial Robotics at MSR • Dedicated Drones Lab • Hardware Lab • 333 Exemption • Indoor OptiTrack • Part 107 • Approved Test Sites

4. Aerial Robotics at MSR

5. Many Recent Successes of Machine Learning Peng et al. 2016 Mnih et al. 2013Silver et al. 2016 1. Close-world systems (also only in software) 2. Luxury to obtain near-infinite data via simulation etc.

6. Can we use such ML methods to build systems that operate in real-world?

7. Successes of Machine Intelligence Real World Flying Systems Peng et al. 2016 Mnih et al. 2013Silver et al. 2016

8. Fast Aerobatic Flights with Safety Constraints 11

9. Infinite Soaring Machine

10. Optimal Scanning: What Trajectory to Fly?

11. Racing FPV Quadrotors

12. Three Fundamental Challenges Lack of large amount of data • High Sample Complexity of ML Methods Computational Constraints • Real-time Performance, Limited Memory and Compute Power Operating in the Open World • Safety, Uncertainty, Mixed- Initiative Autonomy

13. Part II: Getting to Know AirSim Shital Shah

14. A Day at Office Ultrasonic sensors Stereo vision Companion computer

15. Current State of UAV Simulators

16. AirSim: The Next Generation Simulator

17. What Does Simulator Enables? • Wide variety of environment, day of time, weather patterns • No legal hassle, much cheaper and safe Generate lots of training data • Run exact same code that would be run onboard • Slow down or accelerate simulated time Develop autonomy algorithms • If it doesn’t work in simulator then it won’t work in real world • Use sensors with varies parameters Test perception algorithms • Can fail thousands of time to learn patterns • Run in cloud for distributed learning Reinforcement Learning techniques

18. Why Use Unreal Engine? Actual footage captured from simulated drone in AirSim with Open World Environment

19. The Heart of the Vehicle Sensor Input Motor Output Sensors generate information about the world around Motor voltages drives the propeller Humans provide desired state Sensor Input Desired State

20. How Simulator Works? Sensor Input Motor Output Desired State Humans provide desired state Generate all the sensor data using ground truth Run physics engine to get Lin & Ang • Position • Velocity • Acceleration

21. What Does Physics Engine Do? Physics Engine Force Torque Position Velocity Acceleration Linear & Angular Flavors Total of 3 + 3 = 6 vectors

22. The Physics Loop Initialize Get all the forces and torques Find 𝑥𝑥, ̇𝑥𝑥, ̇̇𝑥𝑥 𝛼𝛼, ̇𝛼𝛼, ̇̇𝛼𝛼 Time elapsed dt

23. May the Force & Torque be with You 𝐹𝐹 = 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ∗ 𝐾𝐾𝐹𝐹 𝑇𝑇 = 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ∗ 𝐾𝐾𝑇𝑇 𝐾𝐾𝐹𝐹 ∝ 𝜌𝜌𝐷𝐷4 𝐾𝐾𝑇𝑇 ∝ 𝜌𝜌𝐷𝐷5

24. Physics - Linear Dynamics 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 *In real physics engine you would use better integrator than Euler

25. Physics - Angular Dynamics 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 − 𝜔𝜔 × (𝐼𝐼 𝜔𝜔) 𝐼𝐼 𝜔𝜔𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 𝑞𝑞𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑞𝑞𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ∗ 𝑄𝑄(𝜔𝜔 ∗ 𝑑𝑑𝑑𝑑) *In real physics engine you would use better integrator than Euler *Calculations are in body frame

26. Simulating IMU Physics engine tells us, • Angular velocity • Linear acceleration So just add noise, bias and drift! 𝐼𝐼 𝐼𝐼 𝐼𝐼 = 𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺 + 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴

27. Simulating Barometer

28. Simulating Magnetometer

29. AirSim is Open Source Unreal Engine 4 • Designed as UE4 plugin • Just drop in to 100s of realistic environments APIs for Dozens of Languages • Get camera images • Send commands to the vehicle C++ Header- Only Library • Eigen library as only dependency • Cross-platform Highly Extensible • Add new vehicles models • Add new modes

30. Experiment: Simulation vs. Reality Simulated trajectory in purple, real trajectory in red

31. AirSim Has APIs from PythonClient import * import cv2 import sys client = AirSimClient('127.0.0.1') # get depth image result = client.setImageTypeForCamera(0, AirSimImageType.Scene) # show image in opencv rawImage = np.fromstring(result, np.int8) png = cv2.imdecode(rawImage, cv2.IMREAD_UNCHANGED) cv2.imshow("Camera Image", png) Few lines of Python code can get you FPV image from drone! Scene from Unreal Boy with a Kite environment

32. Make Drone Move in AirSim Using APIs from PythonClient import * import sys client = AirSimClient('127.0.0.1') # Stay 5 meters above ground z = -5 # Fly! client.moveOnPath([(0,-253,z),(125,-253,z),(125,0,z),(0,0,z)], 15, 0, DrivetrainType.ForwardOnly, YawMode(False,0), 20, 1) Same code can be ran from offboard computer on real drone! Other languages available: C++, C#, Java and many more!

33. Make Drone Move in AirSim Using APIs

34. AirSim Extensibility New Vehicle Types Car Apollo Moon Lander New Sensor Types Ultrasonics LIDAR New Physics Engines PhysX Bullet New Sim Modes Computer Vision Just Physics You can contribute on GitHub: https://github.com/microsoft/airsim

35. Part III: Applications & Future Ashish Kapoor

36. AgIoT: Precision Agriculture Scan farm using drone to capture low level details on daily basis, analyze differences each day and fuse with sensor information to identify areas that needs specific work

37. AgIoT: Vision 40 Spatio-temporal view of the farm Sensors & UAVs Yield estimation Precision Irrigation Pest Infection Fertilizer application …Ag Services

38. AgIoT:System Architecture 41 Azure ML Analytics • Monsanto • Cargill • DuPont • Syngenta • Sakata • Fertilizer • … Cameras, Drones, sensors Weather Data (Rain, Wind, Pollen) Recommendations (daily best practices) Farm Aggregator (IOT Edge) Farm Aggregator (IOT Edge)

39. Autonomous 3D Scanning of Large Structures 3D reconstruction in simulator using a simulated drone flight 3D reconstruction in real world using actual drone

40. Autonomous 3D Scanning of Large Structures

41. Thank You! In Collaboration With: Debadeepta Dey, Neel Joshi, Andrey Kolobov, Chris Lovett, Jim Piavis, Gireeja Ranade, Shital Shah, Sudipta Sinha Interns: Sanjiban Choudhury, Mike Roberts, Niteesh Sood, Wen Sun

42. セッションアンケートにご協力ください  専用アプリからご回答いただけます。 decode 2017  スケジュールビルダーで受講セッションを登録後、アンケート画面からご回答ください。  アンケートの回答時間はたったの 15 秒です!

43. Ask the Speaker のご案内本セッションの詳細は『Ask the Speaker Room』各コーナーカウンタにてご説明させていただきます。是非、お立ち寄りください。

44. Appendix

45. Architecture

46. Evaluation: Simulation Vs. Reality

47. Optimal Scanning: What Trajectory to Fly? • Safety: How risky are the flight maneuvers? • Efficiency: How long does the process take? • Performance: How good is the 3D model?

[SP02] Developing autonomous vehicles with AirSim

Du liest eine Vorschau.

Hier ansehen

[SP02] Developing autonomous vehicles with AirSim

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie [SP02] Developing autonomous vehicles with AirSim (20)

Weitere von de:code 2017 (20)

Aktuellste (20)