iOS OpenGL

Elastic Image Software LLC
iOS OpenGL
Douglass Turner
Elastic Image Software
email: douglass.turner@gmail.com
mobile: 781 775 3708

2
A Talk in Three Acts

• Introduction to Computer Graphics
• Computer Graphics in OpenGL
• OpenGL and CocoaTouch Playing Nice Together
3

4
Introduction to Computer Graphics
A Trip Down the Graphics Pipeline

Copyright Steve Jubinville5
Build and Tesselate Model into Facets

Copyright Steve Jubinville6
Pose Model

Aim Camera
7

8
Design the Look

Design Appearance. Light Scene.
Copyright © 2006 Holger Schömann9

Copyright © 2006 Florian Wild
10

Copyright © 2006 Jack Qiao
11

12
Render Scene

13
Clip scene to the bounds of the viewing frustrum.

Rasterize Scene
14

15
for each object in the scene
! determine its corresponding pixels
! for each corresponding pixel
! ! calculate depth of object at pixel
! ! if depth is closer than any object yet seen
! ! ! put this object in the pixel
! ! endif
! end
end
The GPU implements a z-buffer algorithm in hardware

In OpenGL, the Look is evaluated at the same time that
depth comparison is performed.
Together these are the core activities of computer graphics.
They are the raison d'être of the GPU. Compute intensive,
Highly optimized and parallelized.
16

17
Update Model State. This could entail geometric
transformation or perhaps taking a time step in
the physic engine.

18
Rinse, repeat.

19
Example.
Beautiful Panoramas iPad app. In the App Store: http://bit.ly/QVIDgg

20
Computer Graphics in OpenGL

OpenGL Cocoa
21

OpenGL is Old School
22

You are ﬂipping levers on a large state machine.
23

OpenGL is brutally spare. Just the essentials.
float oldSchool[]
glPushMatrix()
glPopMatrix()!
24

Keep in mind OpenGL is a dragster, not a Prius. So, just buckle up ...
25

... and enjoy the ride, baby!
26

27
Posing models. Positioning light sources.
Aiming the camera. All these task require
and understanding of transformation
matrices.

28
Transformation matrix. Coordinate frame
World space, Screen space, etc.
All are different ways of thinking about
the same underlying concept.

29
When we move an object we apply a
transformation via a transformation
matrix. This transformation is deﬁned in
terms of a coordinate frame.
A speciﬁc location on the transformed
object can be interpreted in terms of any
coordinate frame we choose.

30
The motivation for a coordinate frame
transformation is not only motivated by the need
to pose a model or aim a camera as in the
physical world.
In computer graphics land it often has to do with
the convenience of performing a particular
calculation in one space rather then another.

31
We refer to the transformation pipeline: the successive
application of a series of transformations as a 3D model wends
its way from application land to GPU land.

Hierarchical modeling is based on the hierarchical
organization of transformations matrices. Many objects
- trees, robots, cars - form natural hierarchies.
32

33
The UIView class supports all the key concepts
of transformations and hierarchy. It is a nice,
friendly way to gain intuition about facility with
this powerful concept.

34
Hierarchical modeling in Cocoa Touch

35
struct CGAffineTransform {
CGFloat a, b, c, d;
CGFloat tx, ty;
};
struct CATransform3D {
CGFloat m11, m12, m13, m14;
};
Transforms

36
UIView properties
// Coordinate frame in parent space
@property(nonatomic) CGRect frame;
// Dimensions of view in object space
@property(nonatomic) CGRect bounds;
// View center in parent space
@property(nonatomic) CGPoint center;
// View transformation matrix
@property(nonatomic) CGAffineTransform transform;
// Parent view
@property(nonatomic,readonly) UIView *superview;
// Child views
@property(nonatomic,readonly,copy) NSArray *subviews;

37
UIView methods
// Transform 2D point from/to another view’s coordinate frame
- (CGPoint)convertPoint:(CGPoint)point toView:(UIView *)view;
- (CGPoint)convertPoint:(CGPoint)point fromView:(UIView *)view;
// Transform rectangle from/to another view’s coordinate frame
- (CGRect)convertRect:(CGRect)rect toView:(UIView *)view;
- (CGRect)convertRect:(CGRect)rect fromView:(UIView *)view;
// Manipulate view hierarchy
- (void)removeFromSuperview;
- (void)insertSubview:(UIView *)view atIndex:(NSInteger)index;
- (void)addSubview:(UIView *)view;
- (void)insertSubview:(UIView *)view belowSubview:(UIView *)siblingSubview;
- (void)insertSubview:(UIView *)view aboveSubview:(UIView *)siblingSubview;

38
- (BOOL)application:(UIApplication *)application
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
[window addSubview:viewController.view];
[window makeKeyAndVisible];

return YES;
}
View hierarchy in iOS

39
Transformation matrices in iOS
- (void)setTransform:(CGAffineTransform)newValue {
// Scale along the y-axis only
CGAffineTransform constrainedTransform =
CGAffineTransformScale(CGAffineTransformIdentity, 1.0, newValue.a);

newValue.a = constrainedTransform.a;
newValue.b = constrainedTransform.b;
newValue.c = constrainedTransform.c;
newValue.d = constrainedTransform.d;
[super setTransform:newValue];

}

HelloTeapot Demo and Code Walkthrough
http://github.com/turner/HelloTeapot
40

In what follows, we will be referring to the ES1
“ﬁxed-function” transformation pipeline provided
by the GPU.
In ES2 (GLSL) the transformation pipeline must
be handled entirely in application space.
41

43
A camera with a spotlight attached to it orbits
around a stationary teapot model lit by three
stationary colored point light sources.

In iOS we never explicitly say “render scene”. Rather we
inform the system that now would be a good time to
redisplay the scene:
[myView setNeedsDisplay] and [myView drawRect]
44

- (void)startAnimation {
animationTimer =
[NSTimer scheduledTimerWithTimeInterval:animationInterval
target:self
selector:@selector(drawView)
userInfo:nil
repeats:YES];
}
45
In OpenGL we manage rendering ourselves

46
In OpenGL we manage rendering ourselves
!
if (!self.isAnimating) {
! !
self.displayLink =
[NSClassFromString(@"CADisplayLink") displayLinkWithTarget:self
selector:@selector(drawView:)];
[self.displayLink setFrameInterval:animationFrameInterval];
[self.displayLink addToRunLoop:[NSRunLoop currentRunLoop]
forMode:NSDefaultRunLoopMode];
! ! !
self.animating = YES;
! !
} // if (!self.isAnimating)
!
}

! Position the camera
! glMatrixMode(GL_MODELVIEW);!
! glLoadIdentity();
! glLoadMatrixf(_openGLCameraInverseTransform);
47
http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.m

Position spotlight at camera location
glPushMatrix();
! glMultMatrixf(_cameraTransform);!
!
! // A white spotlight for a camera headlight!
GLfloat spotLightColor[] = { 1.0, 1.0, 1.0, 1.0 };
GLfloat spotLightPosition[] = { 0.0, 0.0, 2.0, 1.0 };
GLfloat spotLightDirection[] = { 0.0, 0.0, -1.0, 1.0 };
GLfloat spotCutOff!! ! = 3.5;
!
! glEnable(GL_LIGHT3);
! glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLightColor);!
! glLightfv(GL_LIGHT3, GL_POSITION, spotLightPosition);
! glLightfv(GL_LIGHT3, GL_SPOT_DIRECTION, spotLightDirection);
! glLightf(GL_LIGHT3, GL_SPOT_CUTOFF, spotCutOff);
!
! glPopMatrix();!
48

49
glPushMatrix & glPopMatrix
These functions push/pop a transformation matrix on/off the
transformation stack.
This stack is the key data structure for presenting a
hierarchical model to the GPU in fixed-function OpenGL.

50
glPushMatrix - Create a new matrix and push
it on the stack. Subsequent matrix concatenation
is done with stack.top.
stack.top is the current coordinate frame.
glPopMatrix - Pop the stack. The pre-existing
matrix is now stack.top.
We revert to the coordinate frame the existed
prior to the push.
! glPushMatrix();
! glMultMatrixf(_cameraTransform);
!
! glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);
! glPopMatrix();!

51
Position the red stationary light source
! const GLfloat redLightColor[]! ! = { 1.0f, 0.0f, 0.0f, 1.0f };
! const GLfloat redLightPosition[]! = { 10.0f, 0.0f, 0.0f, 1.0f };
! glLightfv(GL_LIGHT0, GL_DIFFUSE, redLightColor);!
! glLightfv(GL_LIGHT0, GL_POSITION, redLightPosition);

52
Position the green stationary light source
! const GLfloat greenLightColor[]! = { 0.0f, 1.0f, 0.0f, 1.0f };
! const GLfloat greenLightPosition[]!= { 0.0f, 10.0f, 0.0f, 1.0f };
! glLightfv(GL_LIGHT0, GL_DIFFUSE, greenLightColor);!
! glLightfv(GL_LIGHT0, GL_POSITION, greenLightPosition);

53
Position the blue stationary light source
! const GLfloat blueLightColor[]!! = { 0.0f, 0.0f, 1.0f, 1.0f };
! const GLfloat blueLightPosition[]! = { 0.0f, 0.0f, 10.0f, 1.0f };
! glLightfv(GL_LIGHT0, GL_DIFFUSE, blueLightColor);!
! glLightfv(GL_LIGHT0, GL_POSITION, blueLightPosition);

54
Position and draw the teapot
! glPushMatrix();
! JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
! JLMMatrix3DSetScaling(scale, sx, sy, sz);
! JLMMatrix3DMultiply(rotation, scale, concatenation);
! glMultMatrixf(concatenation);
! for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {
! ! glDrawElements(
GL_TRIANGLE_STRIP, indices[i],
GL_UNSIGNED_SHORT, &indices[i+1]
);
! } // for (num_teapot_indices)
! glPopMatrix();

55
! glPushMatrix();
! ! glDrawElements(
);
! glPopMatrix();

56
! glPushMatrix();
! ! glDrawElements(
);
! glPopMatrix();

57
Notice the absence of GL_CAMERA?
You may be surprised to learn there is no concept of
camera in OpenGL. Camera posing is equivalent to
inverting the camera transform and applying it to the
scene observed by the camera.

camera teapot
world
wTtwTc
58

camera teapot
glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(_openGLCameraInverseTransform);
?
cTt
59

camera teapot
world
wTt(wTc)
cTt
cTt wTtcTw *=
-1
60

cTt wTtcTw *=
61

- (void)placeCameraAtLocation:(M3DVector3f)location target:(M3DVector3f)target up:(M3DVector3f)up;
62

The vertices comprising the 3D geometry of the teapot
float teapot_vertices [] = {
0.0663, 0.1178, 0.0,
0.0672, 0.1152, 0.0,
0.0639, 0.1178, 0.0178043,
...
};
The surface normals associated with each vertex
// The list of surface normals corresponding to the vertices
float teapot_normals[] = {
-0.987635, -0.156768, 0,
-0.902861, -0.429933, 0,
-0.953562, -0.156989, -0.257047,
...
};
63

The teapot model is defined as a collection of triangle stripes
that combine to form the teapot surface.
short indicies[] = {
// how many vertices in vertex strip
26,
// vertex strip indices
1122, 1243, 1272, 1242, ... ,1283, 1199,
...
};
64

glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(3 ,GL_FLOAT, 0, teapot_vertices);
glEnableClientState(GL_NORMAL_ARRAY);
glNormalPointer(GL_FLOAT, 0, teapot_normals);
glEnable(GL_NORMALIZE);!
65
! ! glDrawElements(
);
! }

Design the Look
66

67
We now begin leaving behind the ﬁxed-
function pipeline of ES1 and focus on
ES2, GLSL, and visual coolness in
general.

The Look: the lighting, illumination model,
and surface shaders that in combination
achieve the desired look of a model in a
scene.
68

Lights
69

70
Point Light Directional Light
Spot Light Area Light

Illumination Model
72

Ambient Diffuse
Specular
Diffuse
+
Specular
73

Diffuse(NormalVector, LightVector)Ambient(ConstantReﬂectance)
Specular(LightReﬂectionVector, EyeVector) Shiny(Specular, Diffuse)74

Surface Shaders
75

Surface shaders were introduced to the ﬁeld of computer graphics
by Pixar with its rendering API and shading language called
RenderMan.
A shader is a small functions evaluated at every location on a
being rendered.
GLSL borrows heavily from the RenderMan model.
76

Copyright © Douglass Turner77
I created this gallery of shaders using a rendering/shading
system I wrote that was inspired by RenderMan.

78
Copyright © Douglass Turner

Elastic Image Software LLC ZBrush Modeling Daniel Lieske
Material Design Ralf Stumpf
79

Copyright: Stephen Molyneaux
80

Hello Shader Demo and Code Walkthrough
https://github.com/turner/HelloShader
81

82
Before we dive into shaders lets get a lay of the land. There are two
ﬂavors of shaders in GLSL.
A vertex shader is evaluated at each vertex.
A fragment shader is evaluated at each screen space pixel
corresponding to a sampled on the facet being rasterized.

83
Vertex and fragment shaders work together in a pipeline fashion. Vertex
attributes - color, surface normal, texture coordinate - are evaluated in
the vertex shader then passed on to the fragment shader where those
values are interpolated across the surface.

84
ShowST. A shader that visualizes the texture
coordinate parameterization of a surface.

85
https://github.com/turner/HelloShader/blob/master/HelloShader/Shaders/ShowST.fsh
// ShowST.vsh
attribute highp vec4 myVertexXYZ;
attribute highp vec2 myVertexST;
// P * V * M - Projection space
uniform mediump mat4 myProjectionViewModelMatrix;
varying vec2 v_st;
void main() {
gl_Position = myProjectionViewModelMatrix * myVertexXYZ;

v_st = myVertexST;
}
ShowST - vertex shader

86
attribute vec2 myVertexST - Vertex attribute
varying vec2 v_st - Varies across the surface
uniform sampler2D myTexture_0 - Invariant throughout rendering cycle
Shader variables come in different ﬂavors

87
Vertex attribute myVertexST is a texture coordinate
passed to the fragment shader via varying variable v_st.
// ShowST.vsh
attribute highp vec2 myVertexST;
varying vec2 v_st;
void main() {

v_st = myVertexST;
}

88
// ShowST.fsh
varying vec2 v_st;
void main() {

// Visualize the s-t parameterization of the underlying surface
gl_FragColor.r = v_st.s;
gl_FragColor.g = v_st.t;
gl_FragColor.b = 0.0;
gl_FragColor.a = 1.0;

}
ShowST - fragment shader

89
http://github.com/turner/HelloiPadGLSL/blob/master/Shaders/ShowST.fsh
// ShowST.fsh
varying vec2 v_st;
void main() {

// Visualize the s-t parameterization of the underlying surface
gl_FragColor.r = v_st.s;
gl_FragColor.g = v_st.t;
gl_FragColor.b = 0.0;
gl_FragColor.a = 1.0;

}
The texture coordinate values v_st.s and v_st.t
are for the red and green channels of the shader color.

90
TexturePairShader. A shader that mixes the
colors of a pair of textures using the tweaked
alpha channel of one of the pair.
Fail Whale over Mandril

91
TexturePairShader - vertex shader
https://github.com/turner/HelloShader/blob/master/HelloShader/Shaders/TEITexturePairShader.vsh
// TEITexturePairShader.vsh
attribute mediump vec2 myVertexST;
varying mediump vec2 v_st;
void main() {
v_st = myVertexST;
}

92
TexturePairShader - vertex shader
// TEITexturePairShader.vsh
attribute mediump vec2 myVertexST;
void main() {
v_st = myVertexST;
}

93
TexturePairShader - fragment shader
// TEITexturePairShader.fsh
uniform sampler2D myTexture_0;
uniform sampler2D myTexture_1;
void main() {
vec4 rgba_0 = texture2D(myTexture_0, v_st);
vec4 rgba_1 = texture2D(myTexture_1, v_st);
// Interpolate between the two textures using the alpha
// channel of texture 0 as the interpolant.
float interpolate = rgba_0.a;
gl_FragColor.r = rgba_0.r + (1.0 - interpolate) * rgba_1.r;
gl_FragColor.g = rgba_0.g + (1.0 - interpolate) * rgba_1.g;
gl_FragColor.b = rgba_0.b + (1.0 - interpolate) * rgba_1.b;
gl_FragColor.a = rgba_0.a + (1.0 - interpolate) * rgba_1.a;

}

94
Shader variables come in different ﬂavors

95
An application communicates with its shaders in the follow ways:
• attributes - geometry, color, normal, texture coordinate
• uniforms - texture sampler, matrix, eye vector, light vector
• textures - texture channels - r, g, b, a - are not just for color

96
Lets look at how we wire our iOS app together
with our shaders. We will use a texture shading
example.
https://github.com/turner/HelloShader/blob/master/HelloShader/Classes/Renderer/GLRenderer.m

97
First make a texture object ...
TEITexture *t =
[[ [TEITexture alloc] initWithImageFile:@"twitter_fail_whale_red_channnel_knockout"
extension:@"png"
mipmap:YES ] autorelease];

[glView.renderer.rendererHelper.renderables setObject:t forKey:@"texture_0"];

98
Load shaders, Bind shader attribute to application handle.
// Vertex shader
NSString *vertShaderPathname =
[[NSBundle mainBundle] pathForResource:@"TEITextureShader" ofType:@"vsh"];
[self compileShader:&vertShader type:GL_VERTEX_SHADER file:vertShaderPathname];
!
// Fragment shader
NSString *fragShaderPathname =
[[NSBundle mainBundle] pathForResource:@"TEITextureShader" ofType:@"fsh"];
[self compileShader:&fragShader type:GL_FRAGMENT_SHADER file:fragShaderPathname];
!
// Bind attribute
glBindAttribLocation(m_program, VertexSTAttributeHandle,! "myVertexST");

99
Associate shader uniform with application object instance ...
TEITexture *t =
(TEITexture *)[self.rendererHelper.renderables objectForKey:@"texture_0"];
t.location = glGetUniformLocation(m_program, "myTexture_0");
glActiveTexture( GL_TEXTURE0 );
glBindTexture(GL_TEXTURE_2D, t.name);
glUniform1i(t.location, 0);
Activate a texture unit, bind it to a texture object, and assign a number to the
corresponding texture sampler used in the fragment shader

100
We now have a linkage between our application texture object and a texture
sampler in our fragment shader. Rock.

101

102

103
Lets Talk Textures

104
Texture mapping is the fundamental tool for creating visual complexity

105
By establishing a mapping between a surface and a texture we can “attach”
the texture to the surface.
The interpretation of the a can go far beyond that of a decal to be applied to
a surface. Bumps, opacity, displacement, and much more can be designed
with a texture.

106

107
Shaders can have it both ways: Compute surface
color (or surface roughness, or opacity) algorithmically
or look it up in a texture. Or a bit of both.

OpenGL and iOS
Playing Nice Together
108

• OpenGL is mostly M
• Keep V dumb
• Lots of chatter between M and C
MVC
109

110
Lets take a look at how iOS and OpenGL interact. We will us
Hello iPad OpenGL as our guide.
https://github.com/turner/HelloShader

111
-(id)initializeEAGL {
!
! CAEAGLLayer *eaglLayer = (CAEAGLLayer *)self.layer;
!
! eaglLayer.opaque = TRUE;
! eaglLayer.drawableProperties =
! [NSDictionary dictionaryWithObjectsAndKeys:
! [NSNumber numberWithBool:FALSE], kEAGLDrawablePropertyRetainedBacking,
! kEAGLColorFormatRGBA8, kEAGLDrawablePropertyColorFormat,
! nil];
!
! self.renderer = [[[ES2Renderer alloc] init] autorelease];
!
! m_animating = FALSE;
! animationFrameInterval = 1;
! m_displayLink = nil;
!
! return self;
!
}
The view’s layer is used as the rendering surface

112
Rendering frame rate is sync-ed to the screen refresh rate.
The actual rendering is handled via a selector.

if (!self.isAnimating) {

self.displayLink =
[NSClassFromString(@"CADisplayLink") displayLinkWithTarget:self
selector:@selector(drawView:)];

[self.displayLink setFrameInterval:animationFrameInterval];

[self.displayLink addToRunLoop:[NSRunLoop currentRunLoop]
forMode:NSDefaultRunLoopMode];

self.animating = YES;

} // if (!self.isAnimating)

}

113
// framebuffer
glGenFramebuffers(1, &m_framebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffer);

// rgb buffer
glGenRenderbuffers(1, &m_colorbuffer);
glBindRenderbuffer(GL_RENDERBUFFER, m_colorbuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, m_colorbuffer);
[m_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:layer];
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &m_backingWidth);
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &m_backingHeight);

// z-buffer
glGenRenderbuffers(1, &m_depthbuffer);
glBindRenderbuffer(GL_RENDERBUFFER, m_depthbuffer);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, m_backingWidth, m_backingHeight);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, m_depthbuffer);
A framebuffer object is created. A color and depth buffer
are attached.

114
- (void) render {
! !
[EAGLContext setCurrentContext:m_context];
glBindFramebuffer(GL_FRAMEBUFFER, m_framebuffer);
// transform, light, shade, etc.
glBindRenderbuffer(GL_RENDERBUFFER, m_colorbuffer);
[m_context presentRenderbuffer:GL_RENDERBUFFER];
}
The render loop. Draw to the colorbuffer then present to
the display. This is the classic “ping pong” between back
buffer and front buffer.

Thank You
Douglass Turner
Elastic Image Software
email: douglass.turner@gmail.com
mobile: 781 775 3708
115

iOS OpenGL

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (14)

Ähnlich wie iOS OpenGL

Ähnlich wie iOS OpenGL (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

iOS OpenGL