2. Augmented Reality
a live direct or indirect view of a physical,
real-world environment whose elements
are augmented (or supplemented) by
computer-generated sensory input such
as sound, video, graphics or GPS data
3.
4.
5. Why Augmented Reality?
• Presents data in a way that we are
accustomed to
• Can present a lot of data in multiple
ways
Useful
• Remember the Terminator, he was fun
right?
• Extends our senses
• Physical games are more fun
Fun
• Takes advantage of distinguishing
platform elements
Mobile Only
9. Video Camera
■
Many resolutions (width and height) and
framerates
■
Image filtering controls
■
Field of View
■
High-performance code - be careful!
■
8 x 106 pixels * 30 Hz =
240,000,000 pixels / second
It all starts here
10. Directly Accessing the Camera (iOS)
session = new AVCaptureSession ();
session.SessionPreset = AVCaptureSession.PresetMedium;
device = AVCaptureDevice.DefaultDeviceWithMediaType (AVMediaType.Video);
device.LockForConfiguration(out error);
device.ActiveVideoMinFrameDuration = new CMTime (1, 30);
device.UnlockForConfiguration();
input = AVCaptureDeviceInput.FromDevice (device, out error);
session.AddInput (input);
11. Getting Images (iOS)
output = new AVCaptureVideoDataOutput ();
output.VideoSettings = new AVVideoSettings (CVPixelFormatType.CV32BGRA);
queue = new DispatchQueue ("VideoCameraQueue");
output.SetSampleBufferDelegateAndQueue (new VideoCameraDelegate { Camera = this } , queue);
session.AddOutput (output);
class VideoCameraDelegate : AVCaptureVideoDataOutputSampleBufferDelegate
{
public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput,
CMSampleBuffer sampleBuffer,
AVCaptureConnection connection)
{
try {
var frame = ImageFromSampleBuffer (sampleBuffer);
Camera.OnFrameCaptured (frame);
sampleBuffer.Dispose ();
} catch (Exception ex) {
Debug.WriteLine (ex);
}
}
}
12. Field of View
public float FieldOfView {
get {
return device.ActiveFormat.VideoFieldOfView;
}
}
Field of View (FOV)
is the Visible Angle
14. Location Sensor
■
Our friends Latitude & Longitude
■
Altitude above sea level (sometimes)
■
10m resolution, on a good day
■
“Better” with Wi-fi
■
“Not great” indoors
■
(There are alternatives such as
triangulation from known points)
Global Spherical Coordinates
16. Orientation Sensor
■
Pitch, Roll, Yaw (3 numbers)
■
Quaternion (4 numbers)
■
4x4 Transformation Matrix (16
numbers)
A Variety of Formats
Pitch
Yaw
Roll
■
Need a “reference orientation” or known
orientation from which to measure
It’s relative…
20. Augmentation Techniques
Overlay 2D Annotations
• Heads Up Display
• Terminator
• Use OpenGL, SceneKit, MonoGame, …
• Mixing virtual reality with augmented
Show 3D Virtual Objects
Distort Camera Image
• Enhance recognized and correlated objects
• Hot / Cold Maps
• Pixel perfect
• No lag
21. Aligning the Augmented View to the Video
Camera
A Pinhole Camera Model
• Simple
• Used by 3D renderers
• Only depends on Field of View
• “Close enough” to the video camera
In a 3D Cartesian World
• Use 3D X, Y, Z coordinates instead of
Latitude, Longitude, Altitude
• Orient the camera by applying the
orientation transform
Projection
Matrix
ModelView
Matrix
24. ModelView Matrix - Two Techniques
• Very easy
• Not accurate
• High latency
• Best for outdoor unpredictable
environments
Location & Orientation
Sensors
• Very difficult
• Very accurate
• No latency
• Best for indoor or controlled
environments
Image Recognition
25. ModelView Matrix from Location & Orientation
public static Matrix4d GetModelView (Location location, Matrix4d orientation)
{
// 1. Calculate position in 3D cartesian world
// 2. Find “up"
// 3. Orient to face the north pole
// 4. Apply the device orientation
}
26. ModelView Matrix: Location to 3D Cartesian
• Center of earth is at (0, 0, 0)
• Does not rotate, the universe around it does
• Physically accurate
• X, Y, Z are generally unintelligible
Earth-centered, Earth-fixed
• Flatten the earth like a map
• X axis is East/West
• Y axis is North/South
• Z axis is Altitude
North, East, Up
27. ModelView Matrix: Location to 3D Cartesian (ECEF)
//
// 1. Calculate position in 3D cartesian world
//
var pos = location.Position;
public Vector3d Position {
get {
var omega = ToRad * Longitude;
var phi = ToRad * Latitude;
var r = RadiusOfEarth + Altitude;
return new Vector3d (
r * Math.Cos(phi) * Math.Cos(omega),
r * Math.Cos(phi) * Math.Sin(omega),
r * Math.Sin(phi));
}
}
• (X, Y, Z)
28. ModelView Matrix: Up
//
// 2. Find "up"
//
var up = location.Position;
up.Normalize ();
• (0, 0, 0)
• (X, Y, Z)
Up
29. ModelView Matrix: Look at the North Pole
//
// 3. Orient to face the north pole
//
var northPos = Location.NorthPole.Position;
var northZAxis = (pos - northPos);
northZAxis.Normalize ();
var northYAxis = up;
var northXAxis = Vector3d.Cross (northYAxis, northZAxis);
northXAxis.Normalize ();
northZAxis = Vector3d.Cross (northXAxis, northYAxis);
northZAxis.Normalize ();
var lookNorthI = new Matrix4d (
new Vector4d(northXAxis),
new Vector4d(northYAxis),
new Vector4d(northZAxis),
Vector4d.UnitW);
Location NorthPole = new Location (90, 0, 0);
True North
• (X, Y, Z)
30. ModelView Matrix: Orient
//
// 4. Apply the device orientation
//
var newOrient = new Matrix4d (
-orientation.Column1,
orientation.Column2,
-orientation.Column0,
Vector4d.UnitW);
var newOrientI = newOrient;
newOrientI.Transpose ();
var modelViewI = (newOrientI * lookNorthI);
modelViewI.Row3 = new Vector4d (pos.X, pos.Y, pos.Z, 1);
var modelView = modelViewI;
modelView.Invert ();
return modelView;
31. Location to 2D View
Projection
Matrix
ModelView
Matrix
3D
Worl
d
Point
2D
View
Point
x x =
(after a
perspective
divide)
32. Location to 2D View
public PointF LocationToView (Location location)
{
// Move location to 3D earth
var pos3d = new Vector4d (location.Position, 1);
// Camera model
var m = Matrix4d.Mult (modelViewMatrix, projectionMatrix);
// Project into homogeneous 2D point
var pos2h = Vector4d.Transform (pos3d, m);
// Perform the perspective divide
var pos2 = pos2h / pos2h.W;
// Stretch into our view
return new PointF (
(float)((pos2.X + 1) * 0.5) * viewSize.Width,
(float)((-pos2.Y + 1) * 0.5) * viewSize.Height
);
}
34. Same Math Can be Used to Make 3D
void SetOpenGLCamera ()
{
GL.MatrixMode (All.Projection);
GL.LoadMatrix (ref projectionMatrix.Row0.X);
GL.MatrixMode (All.Modelview);
GL.LoadMatrix (ref modelViewMatrix.Row0.X);
}