Hidden Camera 3 APIs in Android 4.4 (KitKat)

Camera 3
The New Camera Subsystem in Android KitKat

Balwinder Kaur,

Senior Member, Technical Staff, Aptina Imaging

AnDevCon, Burlingame, CA
11.15.13
Slide Deck v 1.2 last updated 12.27.13

© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or
specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to
scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners.

1

|

© 2013 Aptina Imaging Corporation

Agenda
I.

Camera 3
‣ Introduction
‣ History

‣ Computational Photography Use Cases

II.

The APIs
‣ {@hide} – Sneak Peek into coming Android Pro Camera APIs

III. The Platform
‣ Architecture
‣ 3A

IV. Emerging Trends

V.

Demo

VI. Q&A
2

|


Camera 3 – The New Camera
Framework

3

|


Limitations of Camera 1
• 3 Operating Modes
‣ Preview
‣ Capture
‣ Video

• Burst Mode Photography – hard to
implement
‣ Zero Shutter Lag
‣ Multi-Shot HDR
‣ Panoramic Stitch

• Metadata: Practically, only face
detection data could be returned
to the application
4

|


Camera 3
• Android 4.3 (Jelly Bean MR2) and Android 4.4 (KitKat)
have shipped with a new Camera Hardware Interface
‣ Android 4.3 => Camera 3 Hardware Interface
‣ Android 4.4 => Camera 3.1 Hardware Interface
What
happened
to
Camera2
?

5

|


Overview of Camera 3
•

Camera Framework till Android 4.0 was modeled after a point & shoot camera.

•

Expanded Camera Subsystem is modeled after a Professional Camera
‣

•

6

But it is more than that

It is the basis of a potential paradigm shift in photography

|


A Bit of History
FCAM

F2 Frankencamera
Project

Lead - Stanford Professor Marc Levoy
Main architects - Eino-Ville (Eddy) Talvala and Andrew Adams
Eddy Talvala is the Tech Lead for Camera Framework on the
Android Team
Industry Partner - Nokia

Computational Photography
‣ Out of scope for this talk
‣ Good starting point - A white paper on Computational Photography Platforms
by Marc Levoy
‣ Use and relevance of software increases in photography
‣ Processing happens in near real time instead of post-processing
‣ Open platforms facilitate innovation
7

|


Definition
• Computational photography refers to computational image
capture, processing, and manipulation techniques that
enhance or extend the capabilities of digital photography.
• Good examples of image capture and manipulation include
panoramas, and high-dynamic-range imaging which is the use
of differently exposed pictures of the same scene to extend
dynamic range beyond even that of analog film-based media.
Source: Wikipedia

8

|


Multi Shot HDR
• HDR = High Dynamic Range
• Multiple shots at different exposures and then blended together

9

|


Panoramic Stitching

Source: commons.wikimedia.org

10

|


Multi-Flash Photography

Source: Stanford Graphics Research
11

|


Flash No-Flash Photography

Source: Microsoft Research

12

|


Flash No-Flash Photography

Source: Microsoft Research
13

|


Focus Bracketing

Focus Stacking for macro photography

Source: Wikipedia
14

|


View Finder Editing
Today’s smartphones have features like

•

‣

Computational Photography can take it to the next level

•

‣

•

15

Touch to Expose & Touch to Focus

Edit the frame in the view finder by selecting areas to sharpen, blur, brighten
and maintain these during capture mode

Source: SIGGRAPH 2012

|


Open Platform Requirements
for Computation Photography
• Fine grained control of the Sensor, Flash, Lens and Image Signal
Processing pipeline
• Ability to program
‣ exposure time, auto white balance, focus, frame-rate etc.
‣ metering algorithm, autofocus algorithm etc.

• Meta data
‣ Every frame is returned with the settings that it was taken with (verses
settings take effect sometime in future)

• Platform should have enough compute and memory power.
‣ With multi core devices and GPU capability increasing, Android becomes a
very attractive platform as an open computational photography platform.

16

|


Other use cases
• Using Imagers for Information
‣ Gesture Recognition
‣ Face Recognition
‣ Augmented Reality

• Requirements differ for this class of applications
‣ High and steady Frame Rate under all light conditions is
Important.
‣ Of less importance
•
•

17

|

High Resolution
Image Quality & Color Accuracy


The APIs

18

|


Camera API
Camera APIs
Package - android.hardware.camera
Android Version

Additions to
android.hardware.camera

Kit Kat (4.4)

API level 19

None

Jelly Bean MR2 (4.3)

API level 18

None

Jelly Bean MR1 (4.2)

API level 17

Minor (HDR, shutterSound)

Jelly Bean (4.1)

API level 16

Minor (AutoFocus Move
Callback)

Ice-cream Sandwich

API level 15

Minor (Video Stabilization)

Ice-cream Sandwich
19

API Level

API level 14

Major (Face Detection)

|


A Sneak Peek of the New Camera APIs
• Kitkat shipped with a set of Experimental Java camera
APIs with the {@hide} tag.
‣ https://android.googlesource.com/platform/frameworks/base
/+/kitkat-release/core/java/android/hardware/camera2/

• Package name is android.hardware.camera2
‣ But in other parts of AOSP many references to ProCamera
(Professional Camera) are found.

20

|


Pro Camera APIs ?
• They are experimental and not part of the SDK, but
available in the platform.

• Likely to change before released in the SDK.
• Camera 3 Hardware Interface is fairly stable now.
• It’s a good time to start thinking about new Camera
applications

21

|


android.hardware.camera2
Interface
Camera Device
Exception
CameraAccessException

22

|


Classes
CameraCharacteristics
CameraDevice.CaptureListener
CameraDevice.StateListener
CameraManager
CameraManager.AvailabilityListener
CameraMetadata
CameraMetadata.Key<t>
CaptureFailure
CaptureRequest
CaptureRequest.Builder
CaptureResult
Face
Rational
Size

Camera Manager
Interface for iterating, listing and accessing Camera Devices
CameraManager manager = (CameraManager)
getSystemService(Context.CAMERA_SERVICE);
//Iterating through Camera Devices
manager.getCameraIdList();
// Query a device for Capabilities
manager.getCameraCharacteristics(cameraId);
// Open a Camera
manager.openCamera ( cameraId,...);
// add an Availability Listener as Cameras become available or unavailable
manager.addAvailabilityListener(CameraManager.AvailabilityListener);

23

|


Camera Device
•

Device is modeled as a pipeline
‣

Input : CaptureRequest for a single Frame

‣

Output: CaptureResult Metadata packet + output image buffers for the request.
These buffers can go to one or multiple output targets
•

SurfaceView

•

SurfaceTexture

•

MediaCodec

•

MediaRecorder

•

Renderscript Allocation et al

•

ImageReader (Use this for Still Images)

‣

‣
24

Requests are processed in order, multiple requests are in flight to maintain full
frame rate ( generally means 30 frames per second )

Android 4.4 added the flush() command to drop all inflight requests.
|


Capture Request
// Step 1 : Configure Output Surfaces
camera.configureOutputs (List<Surfaces> outputs)
// Set the size and format.
// Step 2: Build a CaptureRequest
// Use default Templates – Manual, Preview, Still_Capture,
// Record, Video_Snapshot, Zero_Shutter_Lag
CaptureRequest.Builder bldr =
camera.createCaptureRequest(CameraDevice.TEMPLATE_VIDEO_SNAPSHOT);
//no chaining of calls available
bldr.addTarget();
bldr.setTag(); // to identify the result when it is returned
bldr.set (Metadata.Key …);
CaptureRequest req = bldr.build();

25

|


Capture Modes
//Step 3: Submit a request
//Pick 1 of the 4
// capture, captureBurst, setRepeatingRequest, setRepeatingBurst

camera.capture(CaptureRequest, CameraDevice.CaptureListener) //one time capture
camera.captureBurst(…) // Bracketing Modes. Guaranteed that no other request will be
interspersed in this queue
camera.setRepeatingRequest (…) // Preview. Lower priority. Cancelled with stopRepeating() or
flush() call. videoRates – needs clarification

camera.setRepeatingBurst (…) // same as setRepeatingBurst but at full frame rates

26

|


CaptureListener
//Step 4: Process the progress of the capture
//4 callback methods available
onCaptureStarted(CameraDevice, CaptureRequest, long timestamp);
onCaptureCompleted(CameraDevice, CaptureRequest, CaptureResult);
//or
onCaptureFailed(CameraDevice, CaptureRequest, CaptureFailure);
// once all the capture requests are processed
onCaptureSequenceCompleted(CameraDevice, int seqid, int framenumber)

27

|


Capture Result
//Step 5: Investigate the CaptureResult or CaptureFailure Object Received
// The results of a single image capture from the
//image sensor. Contains the final configuration for
//the capture hardware (sensor, lens, flash), the
//processing pipeline, the control algorithms,
//and the output buffers.
get(CameraMetadata.Key<T> key) ;
getRequest() ;

28

|


ImageReader
•

Introduced in KitKat API Level 19

•

Android.media.ImageReader

•

Allows direct application access to image data rendered into a Surface

•

maxImages determines the maximum number of Image objects that can be be
acquired from the ImageReader simultaneously. Use cautiously !

•

ImageFormat is defined by ImageFormat or PixelFormat

•

ImageReader.OnImageAvailableListener provides callback for when the next
Image is available

•

Use acquireLatestImage, rather than acquireNextImage. It removes old images

ImageReader imgreader = ImageReader.newInstance(width, height, format,
maxImages);
imgreader.getSurface() ; // Pass this into the CameraRequest Class

29

|


Camera Metadata
• Base class for Camera Controls and Information
• Metadata is of 2 types
‣ Static and Dynamic

• Sensor Control
• Flash Control
• Lens Control
• Pipeline Control
• 3 Subclasses
‣ CameraCharacteristics

‣ CaptureRequest
‣ CaptureResult

30

|


Camera Metadata – Static
• Characteristics for an Individual Camera Device
• Static – does not change for a device, so no need to
open a Camera to get this information.
‣ Can be queried from Camera Manager or Camera Device

• Keys are of the format - “Available” or “Info”
• Some interesting Metadata Keys
‣ INFO_SUPPORTED_HARDWARE_LEVEL : Limited or Full
‣ 3A : CONTROL_AE_*, CONTROL_AWB_*, CONTROL_AF_*
‣ LENS_INFO_*, FLASH_INFO_*, SENSOR_INFO_*

31

|


Camera Metadata - Dynamic
• Dynamic Metadata is “per frame”
• Set in CaptureRequest via the Builder to specify
settings while capturing a frame.
• Returned as part of the CaptureResult
• One has to be a “pro” with Image Sensors and Image
Sensing Pipelines to use this.
• Very long list of options available

32

|


How do we use these APIs?
•

To query whether the APIs are available or not on a platform, use
reflection.

•

{@hide} only means not available in SDK.

•

Steps to setup development environment
‣

Connect KitKat Device or start KitKat AVD
adb pull /system/framework/core.jar .
adb pull /system/framework/framework.jar .

‣

Convert from Dex to Jar using tool such as dex2jar
dex2jar core.jar
dex2jar framework.jar

‣

In Eclipse, Project → Properties → Java Build Path → Libraries → Add External
JARs... → core-dex2jar.jar & framework-dex2jar.jar
Source : stackoverflow

•
33

APIs don’t work on AVD because the HAL is a Version 1 Implementation
|


• https://github.com/lbk003/Cam3HiddenAPIs

34

|


The Platform

35

|


AOSP Architecture
Application
Framework/SDK

Java Classes Exposed to the
application for interacting with
the hardware

frameworks/base/core/java/and
roid/hardware

JNI

Glue Code between Java Classes
and Native Classes

frameworks/base/core/jni/andro
id_hardware_Camera.cpp

Native
Framework

Native counterpart of the Java
Camera Classes. Manages all
Binder Interactions

frameworks/av/camera/Camera.
cpp

IPC Binder

3 Binder Interfaces
ICameraService and Icamera
from Application to Framework.
ICameraClient for callbacks into
the Application

frameworks/av/camera

Camera Service

The Camera Service that
manages permissions and
lifecycle of the Camera Devices

frameworks/av/services/camera
/libcameraservice/CameraServic
e.cpp

HAL Interface

Hardware Interface. Multiple
Versions

platform/hardware/libhardware/
include

HAL
Implementation

Hardware specific
implementations. Depends on
host processor, ISP and Sensor

platform/hardware/<vendor>/<pl
atform>

Kernel drivers

http://source.android.com/devices/camera/camera.html
36

|


Camera Version Saga
Camera
HAL
Version

Description

1.0

Android 4.0 (Icecream Sandwich)

Converted from C++
CameraHardwareInterface abstraction
layer to camera.h

2.0

Android 4.2 (Jelly
Bean MR1)

Initial Release of expanded capability HAL. camera2.h
Allowed ZSL in the service layer.

3.0

Android 4.3 (Jelly
Bean MR2)

First revision of expanded-capability
HAL camera3.h. Operational mode
same as Camera 2, but ABI different,
and optimized to support multiple
host architectures

3.1
37

Android Version

Android 4.4 (Kit
Kat)

Minor Revision. Added flush() call to
drop all inflight requests immediately

|


Camera Version Saga (contd)
•

platform/hardware/libhardware/include/camera_common.h

•

Camera HAL version = CAMERA_DEVICE_API_VERSION

•

CAMERA_MODULE_API_VERSION is different
CAMERA_MODULE_API_
VERSION

CAMERA_DEVICE_A
PI_VERSION

Android Version

(Camera HAL
Version)

1.0

1.0

Android 4.0 (Ice-cream Sandwich)

2.0

2.0

Android 4.2 (Jelly Bean MR1)

2.1

3.0

Android 4.3 (Jelly Bean MR2)

2.2

3.1

Android 4.4 (Kit Kat)

• CAMERA_DEVICE_API_VERSION 2.x is outdated.

38

|


Camera 3 HAL
• Supports 2 modes – Limited and Full
• Limited just means Camera version 1

• Camera HAL version 3 and 1 can co-exist. For example,
a front facing camera may be a version 1 device,
whereas the rear facing one could be a version 3
device.

39

|


Camera 1 HAL

40

|


Camera 3 Core Operation Model

41

|


Camera 3 HAL Model

42

|


Camera Pipeline

43

|


3A
•

3A = Auto Focus, Auto Exposure, Auto White Balance

•

Along with demosaicing and denoising, forms a crucial part of the
image pipeline.

•

The 3A algorithms are typically the “secret sauce” of the ISP

•

The State Machine for each of the 3A is defined by the HAL.

•

When a device is opened the 3A is INACTIVE

•

3A algorithms at the top level are controlled by
ANDROID_CONTROL_MODE key
1.

ANDROID_CONTROL_MODE_OFF

2.

ANDROID_CONTROL_MODE_AUTO

3.

ANDROID_CONTROL_USE_SCENE_MODE
•

44

|

SCENE_MODE_FACE_PRIORITY -the AE/AWB/AFMODE controls work as in
ANDROID_CONTROL_MODE_AUTO, but the 3A routines must bias toward metering and focusing
on any detected faces in the scene.


Open Source Implementation of HAL3
Nexus 5
• https://android.googlesource.com/platform/hardware/
qcom/camera/+/kitkat-release/QCamera2/HAL3/
• Released for the Qualcomm MSM8974 Snapdragon 800
based device

45

|


Implementing Camera3 HALs
• Overall with Camera3 specification, the Camera HAL
implementations have become simpler, since most of the
functionality like stream management, etc. are defined and have
moved into the Service Layer
• Most Camera HAL implementations have had required considerable
amount of cleanup and refactoring.
• Most of the work is getting the Hardware vendors (AP, ISPs) ready
to support these features
• Devices with Camera 3 implementations - Nexus 7 (4.3) and Nexus
5
• Major progress but there are areas of investigations - like
processing of non-Bayer sensors, multi-stream, multi – array
sensors etc.

46

|


A Peek into the Future

47

|


Emerging Trends
• Photography is going to become a lot about the software as
processing from offline computers to smart devices
• Imagers for information
‣ Computer Vision, Gesture Recognition, Augmented Reality

• Non-Bayer Sensors
‣ Clarity+

• High Dynamic Range
‣ HDR in Preview, Video

• Multi Sensor Imagers – Array imagers, Stereo

48

|


DEMO
49

|


Q&A
50

|


References
•

‣
‣

http://graphics.stanford.edu/projects/lightfield/

‣

http://fcam.garage.maemo.org/

‣

http://graphics.stanford.edu/courses/cs178/applets/

‣

https://graphics.stanford.edu/papers/fcam/fcam-sigtalk.pdf

‣

•

http://graphics.stanford.edu/projects/camera-2.0/

http://graphics.stanford.edu/papers/camera20/levoy-platforms-cga10.pdf

Android
‣
‣


‣

•

http://source.android.com
http://android.googlesource.com

Clarity+
‣

51

http://www.aptina.com/Aptina_ClarityPlus_WhitePaper.pdf

|


About
Aptina is a global provider of CMOS imaging solutions that enable
Imaging Everywhere™. Using performance enhancing technologies like
Aptina A-Pix™, Aptina A-PixHS™ , DR-Pix™ and Clarity+™ technology,
Aptina has created a market-leading portfolio of image sensor
products found in leading consumer electronics like smart phones,
tablets, laptops, digital and video cameras, as well as applications in
surveillance, automotive, medical, video conferencing, and gaming.
Clarity+ is Aptina’s breakthrough clear pixel technology that
provides 2x improvement in image quality under low light conditions.

Contact Information
Balwinder Kaur bkaur@aptina.com
52

|


Hidden Camera 3 APIs in Android 4.4 (KitKat)

Hidden Camera 3 APIs in Android 4.4 (KitKat)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Hidden Camera 3 APIs in Android 4.4 (KitKat)

Similar to Hidden Camera 3 APIs in Android 4.4 (KitKat) (20)

Recently uploaded

Recently uploaded (20)

Hidden Camera 3 APIs in Android 4.4 (KitKat)