DTrace talk at Oracle Open World

1

Thursday, September 23, 2010

<Insert Picture Here>

Oracle Performance Measurement and Tuning with Solaris DTrace
Jeff Savit
Principal Sales Consultant

Angelo Rajadurai
Principal Software Engineer


The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

3


Agenda

• Basic Concepts of Performance Tuning <Insert Picture Here>

• Performance Tuning Features in Solaris
• DTrace - Dynamically Observing the Entire
Application Stack
• Dynamic Tracing Oracle WebLogic Server with
DTrace
• Further Reading

4


Basic Concepts for Performance Management

Measure
Performance

Analyze
Define Performance
Model &
Goals Model Deduce

Take Action

5


Start with the Best Platform
Solaris - Designed for Performance & Scale

Dynamic, multithreaded, preemptive kernel and process model
Tightly integrated file system and virtual memory
64-bit kernel, supporting both 32-bit and 64-bit applications
Resource management
Optimized, SMP-hot network stack
CPU and memory scalability from low to high. Today, up to:
256 CPUs on T5440, 512 on M9000; 4TB RAM on M9000
Optimized for modern multicore, multithread, NUMA systems
Premier x86 support optimized for Intel Nehalem and AMD Opteron
Solaris Containers (Zones): negligible-overhead, built in virtualization

6


Define the Problem

• High system time is not a problem
• You may be asking the system to do stuff
• Slow I/O is not a problem
• You may be streaming bulk I/O
• 100% utilization is not a problem
• You are using all the box you paid for :-)
• Modern CPUs, NICs, disks, can do more than 1 thing at a
time without adding delay
• Fine even if saturated unless some work unable to get
enough cycles to meet SLA
• None of these are problems unless they relate to
business metrics

7

Monitor: Right Location & Tools
ruby
Frameworks
python
Presentation php

C++
C
JSP/Javascript/Ajax/JavaFX
perl
Container WebLogic Server shell

Database Apps run Native
in VM Apps
Oracle 11g

Operating System Solaris/OEL/Windows

8


Solaris Tool chest
Tracing & Debugging
Proc tools
abitrace - trace ABI interfaces
cputrack - per-processor hw counters mdb – debug process/kernel
pargs – process arguments truss – trace function and system calls
pflags – process flags
pcred – process credentials
pldd – process's library dependencies System Statistics
plockstat – process lock statistics
psig – process signal disposition acctcom – process accounting
pstack – process stack dump busstat – Bus hardware counters
pmap – process memory map cpustat – CPU hardware counters
iostat – IO & NFS statistics
pfiles – open files and names
kstat – display kernel statistics
prstat – process statistics
mpstat – processor statistics
ptree – process tree
netstat – network statistics
ptime – process microstate times
nfsstat – nfs server stats
pwdx – process working directory
sar – kitchen sink utility
pgrep – grep for processes
vmstat – virtual memory stats
pkill – kill processes list
pstop – stop processes
prun – start processes Toolkits
prctl – view/set process resources
pwait – wait for process DTraceToolkit – DTrace Tools
preap – reap a zombie process K9Toolkit – perl perf tools
nicstat – network stats

9

perfbar
see the big picture from across the room

Don't be shy about starting with a
simple graphical tool!

http://blogs.sun.com/partnertech/
resource/tools/perfbar.{sparc,i386}

See:LDOM Booth for use of perfbar

10


Fenxi
Reading a 1000 lines of text does not show you the big picture

Open source tool written at Sun
for aggregating and displaying
vmstat, mpstat, iostat data

https://fenxi.dev.java.net/

11


So why do we need another tool?

12


Observing the Stack is hard
ruby
Frameworks
python
Presentation php

C++
C
JSP/Javascript/Ajax/JavaFX
perl
Container WebLogic Server shell

Database Apps run Native
in VM Apps
Oracle 11g

Operating System Solaris/OEL/Windows

13


“M
ay
m in” D
be ata
: ad th bas
er se e e
e lop aba w i m Ad
ev dat ll idd mi
D kn le n:
o w wa
t he ” re
k gu
as ys
’d
“I
The Blame Game

“I :
do Sy
n’t sa e rt in”
xp adm
De kno dmi e
E s
ve w! n ar e Sy
lop A
er sk d lew th
” th id sk
e M a
s
et
“L
14

Static instrumentation is time consuming

Restart
Stop App Add Code Compile
w/ flags

Prebuilt Instruments Custom Instruments

Instrument Validate

Run in
production

Process Collect info

15

Debugging Transient Problems
In production. Impossible!

Please let me
observe a live
system.

16

If Only We Can ...

• turn instrumentation ON and OFF
dynamically
• probe any arbitrary location
• not just pre-defined probe points
• collect arbitrary data
• have low probing overhead
• have no overhead for disabled probes

17

Dynamic Tracing

-> foo() Application Logic
-> bar()
<- bar()
<- foo() Frameworks c
Presentation c+
ph
12::SELECT id, name, symbol, conversion_rate FROM currencies WHERE
JSP/Javascript/Ajax

pytho
status = 'Active' and deleted = 0

rub
Container
12::SELECT category, name, value FROM config
Glassﬁsh/
perl
12::SELECT id FROM outbound_email WHERE type = 'system'

Database shel
Native
Operating System Solaris/Linux/OS X/

18

D - Scripts
probe
provider : module : probe : name where

/ predicate / when
{

actions do what

}

19

DTrace examples

nthreads
{ Thread
printf("new thread created by %s(%d)n", execname,pid); monitor
}

pid$1::malloc:entry pid$1:::entry
{ {
@=quantize(arg0); @[probefunc]=count();
} }

malloc size hot method
observer finder

20

More DTrace examples
profile-101
/execname==”java”/
{
@[tid]=count();
}

How many Java threads are active?

pid$1::sleep:entry io:::done
{ {
@[ustack()]=count(); @=quantize(args[0]->b_bcount);
} }

Who went to sleep? Size distribution of IO

21

DTrace and Java

• DTrace can observe the following in Java
• When VM starts and ends
• When thread starts and ends
• When class gets loaded and unloaded
• When object allocated and freed
• When GC starts and ends
• JNI calls
• When a method is called and method returns

22

DTrace and Java - Example

hotspot$1:::class-loaded
{
Class Loading
printf("%s loadedn", copyinstr(arg0,arg1));
}

hotspot$1:::method-entry
{
@[copyinstr(arg1,arg2), copyinstr(arg3,arg4)]=count(); Java Methods
}

JVM needs to be started with -XX:+ExtendedDTraceProbes

23

DTrace and Java - More Example
#!/usr/sbin/dtrace -qs
hotspot$1:::gc-begin
{
printf("GC started at %Yn",walltimestamp);
printf("%20s | %-10sn", "pool", "time(ms)");
printf(" ------------------------------------n");
self->ts = timestamp;
}

hotspot$1:::mem-pool-gc-begin
GC
{
self->pool_ts[copyinstr(arg2,arg3)] = timestamp;
}
hotspot$1:::mem-pool-gc-end
{
self->pool = copyinstr(arg2,arg3);
printf("%20s | %-10dn", self->pool, (timestamp - self->pool_ts[self->pool])/1000);
}
hotspot$1:::gc-end
/self->ts/
{
printf(" ------------------------------------n");
printf("%20s | %-10dnn","Total GC time", (timestamp - self->ts)/1000);
}

24

DTrace providers
Other
Providers

DTrace
ip core provider cpc
tcp
mib FC
udp proc DTrace
core PID sh
fbt
java iSCSI IO vminfo nfsv4
perl syscall sysinfo
ruby USDT
python profile nfsv3
javascript plockstat mysql
lockstat postgres
kerberos

25

DTrace Eco-Systems

• DTraceToolkit
• Collection of useful D-scripts

• Chime
• DTrace Data Visualization

• D-Light
• Sun Studio based tool to observe application using DTrace

• DTrace analytics
• DTrace based front end for Sun ZFS storage

26

DTrace & Oracle WebLogic Server

27


Instrumenting JVM vs Byte Code
Byte Code

gc-begin class-loaded method-entry
method-compile-begin
gc-end class-unloaded method-return
method-compile-end

Garbage Class JIT Execution
Collector Loader Compiler Engine
JVM

28

Instrumenting JVM vs Byte Code
Byte Code @OnMethod
{
clazz=”myclass”,
method=”mymeth”
}

Garbage Class JIT Execution
Collector Loader Compiler Engine
JVM

BTrace Supports other annotations
@OnTimer, @OnError, @OnEvent & @OnLowMemory

29

BTrace Code Structure

import com.sun.btrace.annotations.*;
import statuc com.sun.btrace.BTraceUtil.*;
@BTrace
public class BTraceSample {
clazz = “java.lang.thread”,
@OnMethod ( method = “start” ) where
public static void func()

{
action do what

}
30

Sample B-script

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

@BTrace public class ThreadStart {
@OnMethod( New
clazz="java.lang.Thread", Java Thread
method="start" Tracer
)
public static void onnewThread(@Self Thread t) {
println(strcat("starting ", name(t)));
}
}

31

More B-scripts
@BTrace public class AllMethods {
private static Aggregation count = newAggregation(AggregationFunction.COUNT);

@OnMethod(
clazz="/javax.swing..*/",
method="/.*/"
)
public static void m(@ProbeClassName String probeClass,
@ProbeMethodName String probeMethod) {
AggregationKey key = newAggregationKey(strcat(probeClass,strcat(".",probeMethod)));
addToAggregation(count,key,1);
}

@OnEvent public static void onEvent() {
truncateAggregation(count,10);
printAggregation("Count", count);
}
}

Count all javax.swing methods

32

DTrace Vs BTrace
DTrace for JVM BTrace

Dynamic tracing entire
Only within a Java Process
system

Instrument JVM Instrument Java Byte Code

D-script Java like instrumentation Lang

Link Java to entire stack Just within Java

High overhead for method Low overhead for method
tracing tracing

33

Combining DTrace & BTrace

Best

DFrame
of
DTrace Both BTrace
Worlds

34

Dynamic Tracing Framework for
Oracle WebLogic Server

35

The DFrame config file
• The Framework is controlled using the config file
• Two types of probes
• mapped
• config file format
class:method:entry/exit:logical name
• probe format
weblogic*:::event(logical name, class.method)
• unmapped
• config file format
class:method
• probe format
weblogic*:::method-entry(class, method)
weblogic*:::method-return(class,method)

36

How to use the Framework
• Only one command to remember
embedProbes
• Usage
embedProbes <pid>
embedProbes -f configFile <pid>
embedProbes -d [debug on]
embedProbes -B [save B-script]
• The command reads config file, creates the b-script
and embedded dprobes into weblogic

37

Demonstration

38


Reducing the Observer Effect

• Use low overhead tools
✓ dframe overhead for most probes very low (<3%)
• Ability to dynamically enable and disable probes
✓ dframe can be turned on and off dynamically
• Measure the probe effect before running in production
✓ ex: for medrec application on a Sun T5140
for processing 2016 transactions,
this is 487,240 events
the overhead us just %3.8

39


Learn more

• White paper on DFrame
– http://bit.ly/DFrame

• Need pre-release access?
– email dframe-interest_us@oracle.com

40


DTrace Resources

• DTrace Community Page
• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/

• DTrace Documentation
• http://wikis.sun.com/display/DTrace/Documentation

• DTraceToolkit
• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit

• DTrace Chime
• http://hub.opensolaris.org/bin/view/Project+dtrace-chime/

• DTrace Hands on Lab
• http://dtracehol.com/ - AMI ami-22fd164b (dtrace-hol)

41


42


DTrace talk at Oracle Open World

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie DTrace talk at Oracle Open World

Ähnlich wie DTrace talk at Oracle Open World (20)

DTrace talk at Oracle Open World