Fast-paced Introduction to Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.
Optio is a subsidiary of Allied Minds, an innovative U.S. science and technology development and commercialization company. Operating since 2006, Allied Minds forms, funds, manages and builds products and businesses based on
innovative technologies developed at leading U.S. universities and federal research institutions. Allied Minds serves as a diversified holding company that supports its businesses and product development with capital, central
Hamilton Turner
Oct 2015
1
Fast-paced Introduction to
Android Internals

CONFIDENTIAL. ALL RIGHTS RESERVED.Oct 2015
$ whoami
2
@hamiltont
@hamiltont
• Director of Malware Research, OptioLabs
– Machine Learning, Data Science, Penetration
Testing, and Malware Prevention
• Doctorate in Computer Engineering
– Mobile & Cloud Computing, Distributed
Software Engineering, Security & Privacy
• Heavy contributor to open source software
• Android framework hacker for ~4 years

$ who…are we
3
and now a brief message from our sponsors
OptioLabs
– Mobile security company founded in 2011
– Caters to Government, Enterprise, and Mobile Carriers/Manufacturers
– Founders Dr. Charles Clancey, Dr. Jules White, Dr. Brian Dougherty
– Offices in Baltimore & Nashville
– Our OptioCore product increases Android security and enables legal
compliance with numerous regulations
– Recent news:
• AMD and OptioLabs partner to increase enterprise security
• Rugged Meets Secure: OptioCore Powers Sonim XP7
www.optiolabs.com

$ who…are we
4
We are hiring!
Mobile Developer (Application & Framework)
Server Developer (Backend & Frontend)
DevOps
Send resumes to our VP of Engineering
www.optiolabs.com
Andy Meister
andy.meister@optiolabs.com

Slide Set EULA
• Android is…..
– A capital-B Big project
– A complex project
– Changing rapidly
– Different on each physical product
• This presentation is…..
– not comprehensive
– probably not fully correct
– already outdated
• Go explore yourself!
• Please let me know when you find something wrong with this
slide deck
5

What does it mean to be ‘Android’?
• Android is a (loosely) protected term!
– AOSP is all the open source code
– Stock ROM is what manufacturer ships
– Vanilla is very close to AOSP
– Android is ROM that passed Android Compatibility
• Android Compatibility
– Different for each version of Android
– 3 main parts
• Compatibility Test Suite (automated tests)
• Compatibility Verifier (human tests)
• Compatibility Definition Doc (manual)
• Android + Google Mobile Services (GMS)
– Google Play! Maps! YouTube! Gmail!
– All these are not open source and require a license agreement with Google if you want to sell a device
that comes with these
https://source.android.com/compatibility/index.html
6

Is Android really Open Source?
~90% of development is behind closed doors
– Only Google&partners see source code
– Only Google&partners accept (or ignore/deny) patches
• Google is both the KeyMaster and the Architect
– Very few public discussions occur on AOSP design principles
– No public roadmap of features
– However….
• Google accepts huge responsibility for standardization & long term stability of platform
• Major contributions to upstream Linux kernel
• Huge money influx for PR releases, OEM standardization efforts, etc
“Open source is different than a community-driven project. Android is light on the community-driven
side and heavy on the open source. Everything we do ends up in the open source repository.”
- Andy Rubin, Cofounder of Android Inc, SVP@Google until 2013
See https://source.android.com/source/code-lines.html#about-private-code-lines
7

Super basic legality of Android
• Pre-note: I am not a lawyer. This slide is extremely simplified for explanation purposes and there is therefore incorrect
by design
• Google is very careful to minimize “threats” to parties interested in modifying AOSP
– Actively avoids using components that require ‘derived works’ to be open sourced
• E.g. heavy modders are very familiar with the busybox install process ;-)
• Why Android doesn’t come with the well-tested, POSIX correct glibc?
– Actively avoids using components that *might, possibly* require licensing fees or agreements
• Apache Harmony vs Oracle (sun) Java – see https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google,_Inc.
• When modifying Android, you should take a strong interest in the software licenses!
– Smartphone IP is extremely valuable and heavily protected – see https://en.wikipedia.org/wiki/Smartphone_patent_wars
8
Component License
Android applications Up to developer
GSM App suite
AOSP External libraries Apache 2 BSD, misc
Android Kernel GPLv2
Bootloader ???

Android
• Often explained using the stack
on the right
9

Android
• Or these…
10
http://blogs.freescale.com/mcus/2010/05/android-makes-the-move-to-power-architecture-technology/
http://www.techdesignforums.com/edasource/images/68/esd1004_mentor1_large.jpg

Android, Simplified
11
http://blogs.freescale.com/mcus/2010/05/android-makes-the-move-to-power-architecture-technology/
http://www.techdesignforums.com/edasource/images/68/esd1004_mentor1_large.jpg

Android
• The running software stack is critically important to understand
– However, the build system is quite obtuse
– And documentation is sparse & scattered
– One of your big challenges will be mapping running code to
• Source code
• Android filesystem
• Always be on the lookout for the ‘bigger picture’
– All 5 of these are representations of ‘Android’
– Of the docs you can find, 90% will be on the software stack
– 10% will be on the source code
– The rest is up to you

Filesystem of Android device
• Roughly derived from standard linux filesystem
– See https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
– Uses ramdisk, /sbin, /sys, /dev (sometimes)
• Android adds multiple root folders
– /data, /system, /cache
– These are basically an Android filesystem Hierarchy
• Build system enforces this consistent hierarchy
– Many of the FHS folders are symlinks
• E.g. /etc  /system/etc
– Just enough of a linux filesystem to run a kernel and some helpful functions
• Heavy use of partitions
– Reliable update mechanisms are easier with multiple bootable partitions
– Updates can be made smaller
13

Android Filesystem Partitions
• System
– Android OS
– Preinstalled APKs
– Shared libraries
– Executables
• Userdata
– User data, user apps
– App-specific data folders
– Backups
– Dalvik caches
– Crash information
– Encrypted and/or persistent system data
• Cache
– Simple writable, always trashable partition
• Recovery, Boot
– Not first-class citizens while AOSP is running!
14

• Adding main symlinks
– /etc, /tombstones,
/vendor
• Note: Some phones have
distinct vendor.img
partition
15

• Interactions with Kernel
• /dev is tmpfs
– Android uses udev (ueventd) to
fill /dev
• /acct is used by cgroups
– CPU, Memory, DiskIO accounting
and limiting
• /proc and /sys are standard
linux mechanisms for userspace
and kernel interaction
16

Useful tooling: Android shell
• Android uses MirBSD Korn shell
(since 4.0 ICS)
• Utilities
– Gnu core utilities? GPL!
– Busybox? GPL!
• Android toolbox? BSD
– See `ls –l /system/bin`
– Emulates ~75 standard tools (ls, rm,
rmdir, top, sleep)
– See AOSP_ROOT/system/core/toolbox
for source
17

Physical Android Architecture
• Most Android phones have two processors
– Mobile app processor
– Mobile baseband processor
– Communication using interrupts
– Which one ‘leads’?
– Frequently on same Soc
• Sensors/USB/Screen/modem
– Sometimes on SoC, sometimes connected externally
18

Physical Android Hardware: Sensors
• General problem: sensors normally require proprietary code
– To be business-friendly, AOSP tries very hard to avoid GPL-ing someone’s
sensor code
– This unfortunately means that hardware access is *not* done using one
standard mechanism
• Access mechanisms:
– /dev nodes (display, lights)
– Shared library objects (touch, audio)
– Sockets
– Android HAL
• Android specifies header files, manufacturer provides *.so files
– D-Bus (until recently)
19

Acquiring Android Source Code
• Google has comprehensive docs
– https://source.android.com/source/downloading.html
• AOSP uses git for version control
– Git repositories become huge given too much time
– AOSP has a lot of pieces (~450 different git repos)
– Working with ~450 git repositories is a nightmare….
• Repo
– Google tool to automate working across many git repos
– Enable branch/fetch/status/etc in multiple git repositories with one command
$ repo init -u https://android.googlesource.com/platform/manifest -b android-4.0.1_r1
See https://source.android.com/source/build-numbers.html#source-code-tags-and-builds
20

Acquiring Android Source Code
• In repo terms, each git repo is called a project
• A project has
– A remote (e.g. url)
– A path
– A name
• Repo will clone your projects into ‘detached
HEAD’
– aka anonymous branch
– You should use `repo start <project name>` to
do a git branch
• Repo has a concept of project groups
– Note the ‘notdefault’ group
See https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt
21
<manifest>
<remote name="aosp“
fetch="https://android.
googlesource.com" />
<project path="packages/apps/Mms“
name="platform/packages/apps/Mms" />
…
…
<project path="prebuilts/eclipse-build-deps"
name="platform/prebuilts/eclipse-build-deps"
groups="notdefault,eclipse" />
</manifest>
$ cat .repo/manifest.xml

Uploading Android Source Code
• Doing code reviews across ~450 git repos is a
pain!
• Enter ‘gerrit’
– Google developed a code review website that is
specific to AOSP modifications
– Extremely structured process
– Can be self-hosted
– Integrated with ‘repo upload’
• Note: Your dev team does not have to use gerrit
– You can use git directly to push code
– Not quite as painless as ‘repo upload’, but you don’t
have to setup/maintain/use gerrit
22

Navigating Android Source Code
23
abi C++ support for ABI
art Android ART runtime, compiler, dex2oat, etc
bionic Android C standard library
bootable Android Recovery image (no bootloader)
build Makefile-based AOSP build system
dalvik Dalvik VM source code, dex format spec, libdex
developers Source code examples
development Resources for setting up dev environments. Config files, small scripts, USB drivers
device Device-specific files and binaries
docs Doxyfile for source.android.com
external External projects used by AOSP
frameworks Android core components
hardware Android Hardware Abstraction Layer and hardware support
kernel Precompiled kernel images with Android modifications
libcore All java.* packages from Object.java down, as supplied by Apache Harmony
libnativehelper Android wrapper to make JNI easier to work with
ndk Native Development Kit source code & tools (for using C/C++ from APKs)
out Build output directory. make clean ~= rm -rf out
packages Stock Android apps, Input methods, content providers, etc
pdk Platform Development Kit for OEMs to test upcoming AOSP versions
prebuilts Binary (vs source) tools. SDK, qemu for emulator, clang, etc
sdk Software Development Kit – eclipse, ADT, SdkManager, etc
system Native components of Android that run on top of Linux
tools Compilation and IDE tools – gradle files, Eclipse add-ons, Studio add-ons
vendor OEM chipset binaries and device definitions

24
You want to work with AOSP in any capacity

25
You want to port AOSP to a really-small IoT kind of device

26
You want to modify what it means to be ‘Dalvik’

27
You want to add a new system service to Android

28
You want to build a custom ROM to is AOSP with different images

29
You want to use a different Bluetooth stack

• godir is your friend!
– It’s find and cd rolled into one command
30

Building Android Source Code
• Use make!
• I don’t know why that needed an entire slide…
31

1. Setup computer
See https://source.android.com/source/building.html
2. Setup your shell
$ source build/envsetup.sh
3. Choose compile target
$ lunch aosp_arm-eng
4. Compile – full compile takes hours
$ make
5. Run emulator (or skip this and run on device instead)
$ emulator
6. Flash device
$ adb reboot-bootloader
$ fastboot –w flashall
32

1. Setup computer
2. Setup your shell
$ make
$ emulator
6. Flash device
33
What is this?
How does this work?

Android Build System – High Level
• AOSP has ~40k lines of makefiles under /build and /device
– Important to grok the main points before diving in there!
• Comparisons to standard projects using make
– Uses ‘Android.mk’, not ‘Makefile’
– Heavy usage of ANDROID_* environment variables
– All output (even intermediate) goes into ‘out’ folder
• rm –rf out/
– Smallest buildable unit is a ‘module’
• Similar to standard make targets, but there are 1000s of these
• While many ‘modules’ are named statically, even more are created dynamically!
– For example, every module “foo” automatically gets an accompanying module “foo-clean”
– Non-recursive build system
• See http://aegis.sourceforge.net/auug97.pdf
• See build/core/build-system.html
– You never call gcc yourself! Multiple ‘build templates’ exist
• Android application, shared library, static library, standalone executable, java library, prebuilt
• Build templates enforce the Android filesystem layout
• Accept “parameters” in the form of make variables (will default parameters if these variables are not set)
34

To build a module:
• Setup expected variables:
LOCAL_PATH := $(call my-dir)
LOCAL_MODULE_NAME := some_example
• Then include the proper build template:
include $(BUILD_JAVA_LIBRARY)
The build template will read the variables you set, compile as expected
(in this case, as expected for a java library), and place the intermediates
and final output in the correct locations inside the out/ folder
35

• Build system includes lots of ‘dark magic’
• Not recommended to just start writing an
Android.mk
– Find one that does what you want and modify it
– Or you’ll have forgotten a variable
• Look into build/core/ to find makefiles for each build
template
– All options listed at build/core/config.mk line ~75
– Missing variables are often set to default values
36

What is lunch?
• AOSP has many compile options
– Set options with env variables or
buildspec.mk
– envsetup.sh defines functions to help
set options using env variables
– One helper is ‘lunch’
– “I’m having JellyBeans for lunch”
• Just calling lunch presents a menu of
the form:
$TARGET_PRODUCT-$TARGET_BUILD_VARIANT
37

Android Build Targets
• Target product ~= device + configuration
– E.g. nexus 6 (aka shamu)
– E.g. en keyboard/apps/tts
– Device name is arbitrary
• Example products
– aosp_arm: AOSP default locale on Emulator
– full_fugu: All locales on Nexus Player
– aosp_shamu: AOSP default on Nexus 6
– See https://source.android.com/source/build-
numbers.html
• To build your own, take a look at the different
AndroidProducts.mk and vendorsetup.sh scripts
– See https://source.android.com/source/add-
device.html
38

Android Build Variants & Type
• Target_build_variant determines the default apps and settings
– user: Include stuff plan user would want
– userdebug: Include user stuff, and a few items/settings to make
debugging easier
– eng: Include userdebug stuff, user stuff, and item/settings an
engineer might use
• Target_build_type
– You can set this to release or debug to change the settings of
ConfigBuildFlags.DEBUG
– This is used to enable/disable some extra logging in core system
services
39

1. Setup computer
2. Setup your shell
$ make
$ emulator
6. Flash device
40

Modifying a Single Module
1. Modify code
$ gedit frameworks/base/core/java/com/android/internal/os/ZygoteInit.java
2. Compile that module
Note: findmakefile function is helpful to discover the LOCAL_MODULE_NAME
$ make framework
3. Rebuild the system.img file using special snod module
$ make snod
4. Reflash
41

Android Bootup (Roughly) Explained
42

Android init.rc
• Android uses it’s own language for the init files
– Easy to understand
– I recommend you read init.rc
• Main primitives
– service: names a command string execution. Distinct from
other uses of word ‘service’
• Start with ‘start <servicename>’
• Can be grouped into ‘classes’
• Start class with ‘class_start <classname>’
– action: on some trigger, do this
• Predefined actions
– early-init, init, early-fs, fs, early-boot, boot
• User-defined action: property setting changed
– trigger: something happened
• ‘on <trigger>’ defines an action to react
• ‘trigger <triggername>’ fires the trigger
• Location of init.rc files
– In running system (e.g. in ramdisk) : /init.*rc
– In build system: system/core/rootdir/*.rc and /device
43

Android Properties
• OS-wide key-value properties
– Not the same as Java properties
– Not environment variables
– Follow a hierarchy of naming
• Use getprop and setprop tools
– Note: There is actually a socket file, but its
tricky to access
• Setprop can be used to trigger actions in
init.rc!
– Start/stop native code
– Set logging levels
– Change properties that persist across boot
44

Android Properties
• Where do properties originally get set?!
– Filesystem (/default.prop, /system/build.prop)
– Init.rc files
– Java code – SystemProperties.set
– C++ code – property_set
• Some properties are controlled by UID/GID permissions
– E.g. ril.* properties can only be set by RADIO user
• Special cases
– ro.* is read-only. These can only be set once (normally at boot)!
– persist.* properties are written to the filesystem and persist across reboots
– ctl.* properties are not actually stored into the properties map. They enable you to start or stop an init.rc
‘service’ using the properties system
• E.g. ‘ctl.start foobar’ would start service ‘foobar’
45

What’s a zygote?
• At a native level, binary is called
app_process
• Encapsulates starting the Dalvik VM
• Eases the use of JNI
• Loads all Java libraries into itself
• Is capable of being forked
• Drastically speeds up creation of new,
ready-to-go processes pre-loaded with
the Dalvik VM and java libraries
46

Android System Server
• Note that only one bit of Android talks
to the Hardware & Kernel
• This is system server
– Multiplexes access to single-user systems
like the display
– Does most of the complex book-keeping
– Accepts remote messages from apps and
acts on their behalf (e.g. send SMS)
• Most ‘Android System Services’ run as a
thread inside the system_server
process
– Although a number run as standalone
apps with the persistent=true flag
47

Main Android Services
• Some biggies:
– WifiService
– ActivityManagerService
– PowerService
– PackageManagerService
– LocationManagerService
• There are about ~90 more
– Individually they are not too bad
– Look in source code under frameworks/base/services
– Don’t start with ActivityManager ;-)
• Developers access these using Context.getSystemService
• How does this all happen?
48

Android IPC Architecture
“In the Android platform, the binder is used for
nearly everything thathappens across processes
in the core platform.” - Dianne Hackborn,
Android Framework Engineer @ Google -
https://lkml.org/lkml/2009/6/25/3
I highly recommend:
https://newcircle.com/s/post/1340/deep_dive_into_android_ipc_binder_framework_at_andevcon_iv
49

Android IPC Basics
• Android app calls into
android.jar
• In many places, android.jar
wants to call some core
framework service
– Proxy software design pattern!
• To send data through the
Binder driver, you have to write
data into a string
– Marshalling/unmarshalling
50
https://i.stack.imgur.com
/EkzyV.png

Android IPC Basics
• Android app calls into
android.jar
• In many places, android.jar
wants to call some core
framework service
– Proxy software design pattern!
• To send data through the
Binder driver, you have to write
data into a string
– Marshalling/unmarshalling
51
https://i.stack.imgur.com/EkzyV.png

Android IPC Basics
• AIDL files are used to
generate the Java source
code for Proxy
– Exactly the same as you can
do in an Android application!
• Very useful to read
generated Proxy & Stub files
52
https://i.stack.imgur.com/EkzyV.png

Android: Files to Read
• Some source code files are important enough
to actually read…
• Zygote, ZygoteInit
• SystemServer
• Init.rc
57

How is Android not Linux?
• ueventd vs udev
– Check for devices showing up and populate /dev
• Kernel changes!
– Android Out of Memory (OOM) killer
– Wakelocks
– Binder RPC / IPC
– Android’s Anonymous Shared Memory (ashmem)
– Android Alarm (on top oc Linux’s Real time clock RTC)
– Android’s ring logger mechamisms
– Custom init.rc language
– Active work to upstream most of these changes
• https://events.linuxfoundation.org/images/stories/slides/abs2013_stultz.pdf
• Android Upstreaming group - https://wiki.linaro.org/WorkingGroups/Kernel/AndroidUpstreaming
• C standard library
– Bionic vs glibc. Seems this is mainly for licensing reasons
• User management (e.g. uid/gid mapping)
– Linux uses UID/GID. Android uses these for apps, and manages users in the AOSP code
• Hardware approach
– Linux mainly uses /dev for hardware exposure
– Android uses mostly shared object files
60

Resources
• Embedded Android
– Really excellent for everything below the AOSP java code
• New Android Book –
– Good for Android AOSP code
– E.g. ‘How does the PowerManager code work’
• Exploring SDK Add-ons
– Very interesting talk covering some of the interaction between Google and OEMs
– https://www.youtube.com/watch?v=1TY5rqyAyp4
• Deep Dive into Android IPC/Binder Framework at AnDevCon IV
– Binder is critical to Android, more understanding == better!
– https://newcircle.com/s/post/1349/andevcon_iv_video_presentations_android_devel
opers
– https://newcircle.com/s/post/1340/deep_dive_into_android_ipc_binder_framework_a
t_andevcon_iv
• Binder IPC talk
– http://www.dre.vanderbilt.edu/~schmidt/cs282/PDFs/android-binder-ipc.pdf
61

$ who…are we
62
We are hiring!
Mobile Developer (Application & Framework)
Server Developer (Backend & Frontend)
DevOps
Send resumes to our VP of Engineering
www.optiolabs.com
Andy Meister
andy.meister@optiolabs.com

Fast-paced Introduction to Android Internals

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (15)

Ähnlich wie Fast-paced Introduction to Android Internals

Ähnlich wie Fast-paced Introduction to Android Internals (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Fast-paced Introduction to Android Internals