20150410 作業系統-以IP Camera探討thread/system call/cpu,io bound的案例分享!

1. Chung Yuan Christian University
Department of Information and Computer Engineering
(1032 CS208G Operation System)
Teacher: Bin Shyan Jong
Student: Wen Chih Ho
Student ID: 9877606

2.  Usage Scenarios
 IP Camera Software Stack
 Linux Kernel
 Thread in User Space Application
Web Server
IP Cam Main Program
Linux Thread Experiment
 Android Mobile APP
 I/O- versus CPU-Bound
Linux Boot Time
Video Decode Performance

3. Internet
Ethernet
3G/4G
Ethernet
(Cloud Server/ Big Data)
(IOT)

4. Linux Kernel
File System
Driver(FAT)
Camera
Driver
WiFi
Driver
GPIO
Driver
System Libries
C/C++ Library
Applications
Thread
Library
Crypt
Library
…
IP Camera
Application
Web Server
Application
…
System Call
API (Application Programming Interface)

6. System Call Interface
ALSA
core
ALSA
driver
V4L
core
V4L
driver
TTY
core
TTY
driver
serial
core
Serial
driver
Block Layer
Block
driver
SCSI core
libata
USB
storage
SATA
driver
Network
Stack
Net
driver
Virtual
File System
FAT
driver
ext3
driver

7. Trace system call and signals
Allocate new space to load the information of programmer
sets the end of the data segment
File containing an ordered list of libraries found in the directories
specified in /etc/ld.so.conf. This file is not in human readable format, and
is not intended to be edited. This file is created by ldconfig command.
Shared libraries stores in /lib, /usr/lib, /usr/lib64, /lib64,
/usr/local/lib directories.
The arch_prctl() function sets architecture-specific process or thread state.
ARCH_SET_FS
Set the 64-bit base for the FS register to addr.
the 386 architecture introduced two new general segment registers FS,GS.

9. #include <stdio.h>
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <sys/resource.h>
#include <sys/reg.h>
int main(){
puts("Parent started");
pid_t pid;
pid=fork();
if (pid<0){
puts("fork() failed");
return(-1);
}
if (pid==0){
ptrace(PTRACE_TRACEME,0,0,0);
puts("Child sleeping...");
sleep(1);
puts("Child exec...");
execlp("./hello","hello",NULL);
}else{
printf("Child PiD == %dn",pid);
int sta=0;
struct rusage ru;
wait4(pid,&sta,0,&ru);
long rax_rt=ptrace(PTRACE_PEEKUSER,pid,8*RAX,0);
printf("Child execve() returned with %ldn",rax_rt);
ptrace(PTRACE_SYSCALL,pid,0,0);
int intocall=1;
while(1){
wait4(pid,&sta,0,&ru);
if (WIFEXITED(sta)){
puts("Child Exited");
break;
}
long
_ORIG_RAX=ptrace(PTRACE_PEEKUSER,pid,8*ORIG_RAX,0);
long _RAX=ptrace(PTRACE_PEEKUSER,pid,8*RAX,0);
if (intocall){
printf("Entering SYSCALL %ld .... ",_ORIG_RAX);
intocall=0;
}else{
printf("Exited with %ldn",_RAX);
intocall=1;
}
ptrace(PTRACE_SYSCALL,pid,0,0);
}
}
}
Process can be traced
Ref: Programming with PTRACE, Part2

10. read()=0
write()=1
open()=2
close()=3
stat()=4
fstat(5
mmap=9
mprotect()=10
munmap()=11
brk()=12
access()=21
arch_prctl()=158

11. 32bit x86 32bit x86 32bit x86
Anatomy of a Program in Memory
Linux From Scratch for Cubietruck -- C1 : toolchain introduction

12. Android 筆記-Linux Kernel SMP開機流程解析

14. IP Camera

15. Web
Server
Web
Server
daemon
Open Sockets for server
Accept connection
pthread_create for client
fork()
 Client thread provide function
 GET /?action=snapshot
 GET /?action=stream
 GET /input
 GET /output
 GET /?action=command
 GET /command.cgi?action=…
 File Download

16.  IP Camera Program
 Grab Video Frame
 Send Video to client
 Send Sensor
information to client
 Save snapshot or
MP4 File
Main
Program
Grab Camera
Frame Thread
Client Connection
Thread
Save MP4 File
Thread
input
output
output
Sensor Data
input

18. Ref: Small example of pthreads
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
/* A task that takes some time to complete. The id identifies distinct
tasks for printed messages. */
void *task(int id) {
printf("Task%d startedn", id);
int i;
double result = 0.0;
for (i = 0; i < 100000000; i++) {
result = result + sin(i) * tan(i);
}
printf("Task%d completed with result %en", id, result);
}
/* Same as 'task', but meant to be called from different threads. */
void *threaded_task(void *t) {
long id = (long) t;
printf("Thread %ld startedn", id);
task(id);
printf("Thread %ld donen", id);
pthread_exit(0);
}
/* Run 'task' num_tasks times serially. */
void *serial(int num_tasks) {
int i;
for (i = 0; i < num_tasks; i++) {
task(i);
}
}
/* Run 'task' num_tasks times, creating a separate thread for each call to 'task'. */
void *parallel(int num_tasks){
int num_threads = num_tasks;
pthread_t thread[num_threads];
int rc;
long t;
for (t = 0; t < num_threads; t++) {
printf("Creating thread %ldn", t);
rc = pthread_create(&thread[t], NULL,threaded_task, (void *)t);
if (rc) {
printf("ERROR: return code from pthread_create() is %dn", rc);
exit(-1);
}
}
}
void *print_usage(int argc, char *argv[]) {
printf("Usage: %s serial|parallel num_tasksn", argv[0]);
exit(1);
}
int main(int argc, char *argv[]) {
if (argc != 3) {print_usage(argc, argv);}
int num_tasks = atoi(argv[2]);
if (!strcmp(argv[1], "serial")) {
serial(num_tasks);
} else if (!strcmp(argv[1], "parallel")) {
parallel(num_tasks);
}
else {
print_usage(argc, argv);
}
printf("Main completedn");
pthread_exit(NULL);
}

19. Experiment Environment:
Intel Core i7 CPU, 4 core, support Hyper Threading @2.80GHz
OS : Ubuntu 14.04 LTS trusty
./pthreads_test parallel 8 &
ps -Lf; ps -T; ps -Lm;
UID PID PPID LWP C NLWP STIME TTY TIME CMD
stenly 16711 16709 16711 0 1 12:04 pts/12 00:00:00 -bash
stenly 16849 16711 16849 0 9 12:04 pts/12 00:00:00 [pthreads_test] <defunct>
stenly 16849 16711 16850 82 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16851 79 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16852 85 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16853 86 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16854 84 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16855 88 9 12:04 pts/12 00:00:07 [pthreads_test] <defunct>
stenly 16849 16711 16856 85 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16849 16711 16857 85 9 12:04 pts/12 00:00:06 [pthreads_test] <defunct>
stenly 16869 16711 16869 0 1 12:04 pts/12 00:00:00 ps -Lf
PID SPID TTY TIME CMD
16711 16711 pts/12 00:00:00 bash
16849 16849 pts/12 00:00:00 pthreads_test <defunct>
16849 16850 pts/12 00:00:06 pthreads_test
16849 16851 pts/12 00:00:06 pthreads_test
16849 16852 pts/12 00:00:06 pthreads_test
16849 16853 pts/12 00:00:06 pthreads_test
16849 16854 pts/12 00:00:06 pthreads_test
16849 16855 pts/12 00:00:07 pthreads_test
16849 16856 pts/12 00:00:06 pthreads_test
16849 16857 pts/12 00:00:06 pthreads_test
16870 16870 pts/12 00:00:00 ps
PID LWP TTY TIME CMD
16711 - pts/12 00:00:00 bash
- 16711 - 00:00:00 -
16849 - pts/12 00:00:54 pthreads_test <defunct>
- 16849 - 00:00:00 -
- 16850 - 00:00:06 -
- 16851 - 00:00:06 -
- 16852 - 00:00:06 -
- 16853 - 00:00:07 -
- 16854 - 00:00:06 -
- 16855 - 00:00:07 -
- 16856 - 00:00:06 -
- 16857 - 00:00:06 -
16871 - pts/12 00:00:00 ps
- 16871 - 00:00:00 -
PID : Process ID
PPID : Parent process ID
LWP : Light Weight Process(thread)ID(alias spid, tid)
SPID : Session ID
TID : Thread ID
NLWP : number of lwps(threads) in the process
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
stenly 16709 0.0 0.0 109784 2072 ? S 12:04 0:00 sshd: stenly@pts/12
stenly 16709 16612 16709 0 1 12:04 ? 00:00:00 sshd: stenly@pts/12
root 16612 1029 16612 0 1 12:04 ? 00:00:00 sshd: stenly [priv]
root 1029 1 1029 0 1 2月16 ? 00:00:17 /usr/sbin/sshd –D
root 1 0 1 0 1 2月16 ? 00:00:10 /sbin/init

21. IP Camera

22.  Android Mobile App
 Connect to IP Camera
 Receive Video/Audio
Data
 Play Video/Audio Data
Main
Program
Network
video/audio/
command transfer
OpenGL Video
Render Thread
Video Decode
Thread
Input/
output
output
output
UI Thread
input

23.  All application threads are based on the native
pthreads in Linux with a Thread representation in
Java, but the platform still assigns special properties
to threads that make them differ. From an application
perspective, the thread types are UI, binder, and back
ground threads.
 UI Thread
 The UI thread is started on application start and stays alive during the lifetime of the Linux process. The UI thread
is the main thread of the application, used for executing Android components and updating the UI elements on the
screen.
 Binder Thread
 Binder threads are used for communicating between threads in different processes.Each process maintains a set
of threads, called a thread pool, that is never terminated or recreated, but can run tasks at the request of another
thread in the process.
 Background Threads
 All the threads that an application explicitly creates are background threads. This means that they have no
predefined purpose, but are empty execution environments waiting to execute any task.
Ref Book: OReilly Efficient Android Threading

24. USER PID PPID VSIZE RSS WCHAN PC NAME
root 1 0 480 348 ffffffff 00000000 S /init
root 174 1 649496 38368 ffffffff 00000000 S zygote
u0_a77 12895 174 682124 51656 ffffffff 00000000 S com.example.mythread
Garbage collection
com.example.mythead
Java debug wire protocol
Android Just-In-Time
Android IPC Binder

25.  Example
 OS Layer
 System Boot Time
 Application Layer
 Video Decode Performance

26. System Boot Time

27. Ref: PATHPARTNER Android Jelly
Bean Bootup Time Optimization

28. Ref: PATHPARTNER Android JB Bootup Time Optimization

29. 1: kernel init 2: zygote class preloading 3: PackageManager package scanning 4: System Services starting
Ref: Sony Improving Android Bootup Time - eLinux.org

30. Video Performance

32. import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.app.Activity;
import android.view.Menu;
import android.widget.VideoView;
public class MainActivity extends Activity {
VideoView videoView ;
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
videoView = (VideoView)this.findViewById(R.id.videoView1);
Uri uri = Uri.parse(Environment.getExternalStorageDirectory().getPath()+"/Movies/videoview.3gp");
videoView.setVideoURI(uri);
videoView.start();
}
}

33. import android.media.MediaPlayer;
import android.os.Bundle;
import android.os.Environment;
import android.app.Activity;
import android.view.Menu;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.SurfaceHolder.Callback;
public class SurfaceMediaPlayer extends Activity {
SurfaceView surfaceView;
MediaPlayer mediaPlayer;
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_surface_media_player);
mediaPlayer=new MediaPlayer();
surfaceView=(SurfaceView)
this.findViewById(R.id.surfaceView1);
surfaceView.getHolder().addCallback(new Callback() {
@Override
public void surfaceCreated(SurfaceHolder holder) {
play();
}
});
}
private void play() {
try {
String path =
Environment.getExternalStorageDirectory().getPath()+"/Mo
vies/mediaplayer.3gp";
mediaPlayer.setDataSource(path);
mediaPlayer.setDisplay(surfaceView.getHolder());
mediaPlayer.prepare();
mediaPlayer.start();
} catch (Exception e) {
}
}
}

34. So, we are using ffmpeg software decode
library in our IP Camera App.
But that will cause two big issue otherwise the video will lag
(1)We only can decode one VGA video stream
(VGA=640x480)
(2)We can not decode HD or higher video stream
(HD=1280x720)
But, now everybody are using HD video and even has the 4K TV.
What we can do?

35. Receive H264
video raw data
from network
ffmpeg library
decode and output
RGB pixel data
OpenGL draw RGB
pixel data
Why we can see the video play is smoothly?
FPS : frame per second( 30 fps or 25 fps)
If 30 fps that is means you have finished all process in 33ms for each frame
If 25 fps that is means you have finished all process in 40ms for each frame
avcode_decode_video2()
sws_getContext()
sws_scale()
We need to profiling those function to find out what happen on our problem.

36. One Frame Decode YUV to RGB OpenGL Draw Total
XIAOMI 18~33ms 33ms 4.2~4.8ms 51~70ms
GPlus 27ms 64ms 17ms 108ms
RockChip 29ms 74ms 28ms 131ms
Our Target?
15 FPS => 67ms
25 FPS => 40ms
30 FPS => 33ms

37. 1) ARM/Thumb instruction set
2) Concurrent process video data
3) Ask GPU for help (CPU vs GPU)
4) Use video hardware decode component
5) …
Requirement:
Android/Apple mobile and tablet must all support

38. One Frame Decode YUV to RGB OpenGL Draw Total
XIAOMI 18~33ms 33ms 4.2~4.8ms 51~70ms
GPlus 27ms 64ms 17ms 108ms
RockChip 29ms 74ms 28ms 131ms
4~2xms 3x~5xms 3x~7xms
H/W Decode GPU Shader support
So this is a CPU bound issue.

40. Back