Slides for my talk at SkyCon'12 in Limerick.
Here I've squeezed four talks into one, covering a lot of ground quickly, so I've included links to more detailed presentations and other resources.
8. Running NYTProf
perl -d:NYTProf ...
perl -MDevel::NYTProf ...
Configure profiler via the NYTPROF env var
perldoc Devel::NYTProf for the details
To profile code that’s invoked elsewhere:
PERL5OPT=-d:NYTProf
NYTPROF=file=/tmp/nytprof.out:addpid=1:...
9. Reporting: KCachegrind
• KCachegrind call graph - new and cool
- contributed by C. L. Kao.
- requires KCachegrind
$ nytprofcg # generates nytprof.callgraph
$ kcachegrind # load the file via the gui
11. Reporting: HTML
• HTML report
- page per source file, annotated with times and links
- subroutine index table with sortable columns
- interactive Treemap of subroutine times
- generates Graphviz dot file of call graph
- -m (--minimal) faster generation but less detailed
$ nytprofhtml # writes HTML report in ./nytprof/...
$ nytprofhtml --file=/tmp/nytprof.out.793 --open
12.
13. Summary
Links to annotated
source code
Link to sortable table
of all subs
Timings for perl builtins
14.
15. Color coding based on Overall time spent in and below this sub
Median Average Deviation
relative to rest of this file (in + below)
Time between starting this perl
statement and starting the next.
So includes overhead of calls to
perl subs. Timings for each location calling into,
or out of, the subroutine
16.
17. Treemap showing relative
proportions of exclusive time
Boxes represent subroutines
Colors only used to show
packages (and aren’t pretty yet)
Hover over box to see details
Click to drill-down one level
in package hierarchy
20. Questions?
For more details see
Slides: http://www.slideshare.net/Tim.Bunce/develnytprof-v4-at-yapceu-201008-4906467
Screencast: http://blip.tv/timbunce/devel-nytprof-yapc-asia-2012-6376582
22. Ouch!
$ perl some_script.pl
Out of memory!
$
$ perl some_script.pl
Killed.
$
$ perl some_script.pl
$
Someone shouts: "Hey! My process has been killed!"
$ perl some_script.pl
[...later...] "Umm, what's taking so long?"
24. $ perl -e 'system("cat /proc/$$/stat")' # $$ = pid
4752 (perl) S 4686 4752 4686 34816 4752 4202496 536 0 0 0 0 0 0 0 20 0 1 0 62673440 123121664 440
18446744073709551615 4194304 4198212 140735314078128 140735314077056 140645336670206 0 0 134 0
18446744071579305831 0 0 17 10 0 0 0 0 0 0 0 0 0 0 4752 111 111 111
$ perl -e 'system("cat /proc/$$/statm")'
30059 441 346 1 0 160 0
$ perl -e 'system("ps -p $$ -o vsz,rsz,sz,size")'
VSZ RSZ SZ SZ
120236 1764 30059 640
$ perl -e 'system("top -b -n1 -p $$")'
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13063 tim 20 0 117m 1764 1384 S 0.0 0.1 0:00.00 perl
$ perl -e 'system("cat /proc/$$/status")'
...
VmPeak:! 120236 kB
VmSize:! 120236 kB <- total (code, libs, stack, heap etc.)
VmHWM:! 1760 kB
VmRSS:! 1760 kB <- how much of the total is resident in physical memory
VmData:! 548 kB <- data (heap)
VmStk:! 92 kB <- stack
VmExe:! 4 kB <- code
VmLib:! 4220 kB <- libs, including libperl.so
VmPTE:! 84 kB
VmPTD:! 28 kB
VmSwap:! 0 kB
...
Further info on unix.stackexchange.com
25. C Program Code int main(...) { ... }
Read-only Data eg “String constants”
Read-write Data un/initialized variables
Heap
(not to scale!)
Shared Lib Code
Shared Lib R/O Data repeated for each lib
Shared Lib R/W Data //
C Stack (not the perl stack)
System
27. $ perl -e 'system("cat /proc/$$/smaps")' # note ‘smaps’ not ‘maps’
address perms ... pathname
...
7fb00fbc1000-7fb00fd22000 r-xp ... /.../5.10.1/x86_64-linux/CORE/libperl.so
Size: 1412 kB <- size of executable code in libperl.so
Rss: 720 kB <- amount that's currently in physical memory
Pss: 364 kB
Shared_Clean: 712 kB
Shared_Dirty: 0 kB
Private_Clean: 8 kB
Private_Dirty: 0 kB
Referenced: 720 kB
Anonymous: 0 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
... repeated for every segment ...
... repeated for every segment ...
28. Memory Pages
✦ Process view:
✦ Single large memory space. Simple.
✦ Operating System view:
✦ Memory is divided into pages
✦ Pages are loaded to physical memory on demand
✦ Mapping can change without the process knowing
29. C Program Code
Read-only Data Memory is divided into pages
Page size is typically 4KB
Read-write Data
Heap
← Page ‘resident’ in physical
memory
← Page not resident
RSS “Resident Set Size”
is how much process memory is
Shared Lib Code currently in physical memory
Shared Lib R/O Data
Shared Lib R/W Data
C Stack
System
30. Key Point
✦ Don’t use Resident Set Size (RSS)
✦ It can shrink even while the process size grows.
✦ Heap size or Total memory size is a good indicator.
32. Heap
← Your perl stuff goes here
• Heap is managed by malloc()
• Memory freed is rarely returned to
the operating system
• Heap grows but rarely shrinks
33. Perl Data Anatomy
Integer
(IV)
String
(PV)
Number
with a
string
Head Body Data Illustrations from illguts
35. Glob (GV) Symbol Table (Stash)
Sub (CV)
lots of tiny chunks!
36. Devel::Peek
• Gives you a textual view of data
$ perl -MDevel::Peek -e '%a = (42 => "Hello World!"); Dump(%a)'
SV = IV(0x1332fd0) at 0x1332fe0
REFCNT = 1
FLAGS = (TEMP,ROK)
RV = 0x1346730
SV = PVHV(0x1339090) at 0x1346730
REFCNT = 2
FLAGS = (SHAREKEYS)
ARRAY = 0x1378750 (0:7, 1:1)
KEYS = 1
FILL = 1
MAX = 7
Elt "42" HASH = 0x73caace8
SV = PV(0x1331090) at 0x1332de8
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x133f960 "Hello World!"0
CUR = 12 <= length in use
LEN = 16 <= amount allocated
37. Devel::Size
• Gives you a measure of the size of a data structure
$ perl -MDevel::Size=total_size -le 'print total_size( 0 )'
24
$ perl -MDevel::Size=total_size -le 'print total_size( [] )'
64
$ perl -MDevel::Size=total_size -le 'print total_size( {} )'
120
$ perl -MDevel::Size=total_size -le 'print total_size( [ 1..100 ] )'
3264
• Is very fast, and accurate for most simple data types.
• Has limitations and bugs, but is the best tool we have.
39. What?
✦ Track memory size over time?
✦ See where memory is allocated and freed?
✦ Experiments with Devel::NYTProf
✦ Turned out to not seem useful
40. Space in Hiding
✦ Perl tends to consume extra memory to save time
✦ This can lead to surprises, for example:
✦ sub foo {
my $var = "X" x 10_000_000;
}
foo(); # ~20MB still used after return!
✦ sub bar{
my $var = "X" x 10_000_000;
bar($_[0]-1) if $_[0]; # recurse
}
bar(50); # ~1GB still used after return!
42. The Plan
✦ Extend Devel::Size
✦ Add a C-level callback hook
✦ Add some kind of "data path name" for the callback to use
✦ Add a function to Devel::Size to return the size of everything
✦ Stream the data to disk
✦ Write tools to visualize the data
✦ Add multi-phase scan
1. scan symbol tables, skip where ref count > 1
2. process the skipped items
3. scan arenas for other values (e.g. leaks)
✦ Write tool to compare two sets of data
43. The Status
✓ Add a C-level callback hook
✓ Add some kind of "data path name" for the callback to use
✓ Add a function to Devel::Size to return the size of everything.
✓ Stream the data to disk
✓ Write tools to visualize the data
• Will become a separate distribution
• “Devel::SizeMeGraph”?
• Source repo available by Sunday
• Ready to demonstrate
45. Questions?
For more details see
Full slides: http://www.slideshare.net/Tim.Bunce/perl-memory-use-yapcasia2012
Screencast: http://blip.tv/timbunce/perl-memory-use-and-devel-sizeme-at-yapc-
asia-2012-6381282
Tim.Bunce@pobox.com
http://blog.timbunce.org
@timbunce
50. - Perl is 25 years old
- Perl 5 is 18 years old
- Perl 6 was conceived 12 years ago
- Perl 6 “Rakudo Star” is 2 years old
51. “CPAN is my language,
Perl is my VM”
- Vast library of free code modules on “CPAN”
- Over 5,100 active authors (making releases)
- Over 26,000 distributions (110,000 modules)
- ~1,000 uploads per month (by ~500 authors)
- ~250 new distributions per month
- Automated smoke testing for all uploads
55. Perl 5 Development
- 5.10 – 2007
- 5.12 – 2010
- 5.14 – 2011
- 5.16 – 2012
- 5.17.4 latest monthly development release
- 5.18 – due May 2013
56. Perl 5 New Features
- Refactored internals
many fixes, more speed, less memory
- New language features (state, say, //, autodie, ...)
- Language feature management
- Unicode 6.1 and many new Unicode features
- Many powerful new regex features
See http://www.slideshare.net/rjbs/whats-new-in-perl-v510-v516
58. Demo
Regexp::Debuger
See http://www.youtube.com/watch?v=zcSFIUiMgAs
59. A Culture of Testing
-2002: Perl 5.8.0 had 26,725 core tests
+41,666 more for bundled libraries etc.
-2007: Perl 5.10.0 has 78,883 core tests
+109,427
-2010: Perl 5.11.5 has 191,008 core tests
+167,015
-2012: Perl 5.16.0 has 262,370 core tests
+261,776
61. Learn it once, use it many times. Learn as
you go. Many acceptable levels of
competence. Multiple ways to say the same
thing. No shame in borrowing. Indeterminate
dimensionality. Local ambiguity is okay.
Natural Language
Punctuation by prosody and inflection.
Principles in Perl
Disambiguation by number, case and word
order. Topicalization. Discourse structure.
Pronominalization. No theoretical axes to
grind. Style not enforced except by peer
pressure. Cooperative design. “Inevitable”
Divergence. http://www.wall.org/~larry/natural.html
62. Timeline
2000: Perl 6 conceived (after a smashed coffee mug)
2001-2004: Initial design docs
2005: First prototype, pugs (in Haskell)
2005+ Continual radical evolution (the whirlpool)
2010: 1st Rakudo Star release
2012: 17th Rakudo Star release
Mostly implemented in Perl6 (>60% and rising)
63. “If we'd done Perl 6 on a schedule, you'd have it by
now. And it would be crap.”
—Larry Wall, 2002
“Do it right.” and “It's ready when it's ready.”
Freedom to explore deeply and change
“Truly radical and far-reaching improvements over
the past few years.”
64. “We've spent a decade stealing the very best ideas
from the best programming languages, and making
them simple and practical for mortal developers to
use.”
—Damian Conway
65. “Should be called Perl 8 or Perl 9”
“feedback at many levels from multiple implementations”
66. Multiple Implementations
Two main implementations:
Rakudo - built on Parrot Compiler Toolchain
Niecza - targeting the CLR (.NET and Mono)
Plus: STD, viv, Perlito, Pugs and others
Work on a JVM implementation is starting
All sharing a common test suite
See http://perl6.org/compilers
67. Some Perl 6 Features
Rich set of operators and meta-operators
Rich type system
Rich object system, including roles/traits
A full Meta Object Protocol
Multiple dispatch using types and expressive signatures
Gradual typing
Lazy lists and iterators, with controllable eagerness
Deep introspection
Native Call Interface
Representational polymorphism
Powerful matching and parsing with subclassable grammars
68. Series and Reduction
‣ say 1, 2, 4 ... 1024;
1 2 4 8 16 32 64 128 256 512 1024
‣ my @fib = 1, 1, *+* ... *; # infinite
say @fib[^10]; # 0..9
1 1 2 3 5 8 13 21 34 55 89
‣ say [*] 1..10; # reduction, use any operator
3628800
‣ sub postfix:<!>($n) { [*] 1..$n }
say 10!
3628800
69. Multiple Dispatch
‣ multi fact(0) { 1 }
multi fact($n) { $n * fact($n – 1) }
‣ multi fib(0) { 0 }
multi fib(1) { 1 }
multi fib($n) { fib($n – 1) + fib($n – 2) }
‣ multi quicksort([]) { () }
multi quicksort([$pivot, *@rest]) {
my @before = @rest.grep(* < $pivot);
my @after = @rest.grep(* >= $pivot);
(quicksort(@before), $pivot, quicksort(@after))
}
70. Native Call Interface
sub PQexecPrepared(
- OpaquePointer $conn,
Str $statement_name,
Int $n_params,
CArray[Str] $param_values,
CArray[int] $param_length,
CArray[int] $param_formats,
Int $resultFormat)
- returns OpaquePointer
is native('libpq')
{ ... }
Also supports structures and callbacks.
71. my @suits = < ♣ ♢ ♡ ♠ >;
my @ranks = 2..10, < J Q K A >;
# concatenate each rank with each suit
my @deck = @ranks X~ @suits;
# create hash of card to points value
my %points = @deck Z ( (2..10, 10, 10, 10, 11)
X+ (0,0,0,0) );
# grab five cards from the deck
my @hand = @deck.pick(5);
# display my hand
say ~@hand;
# tell me how many points it's worth
say [+] %points{@hand};
72. @xyz»++ # increment all elements of @xyz
@x = @a »min« @b # @x is smallest of @a and @b
$mean = ([+] @a) / @a # calculate mean of @a
$sumsq = [+] (@x »**» 2) # sum of squares of @x
$fact = [*] 1..$n # $n factorial
for %hash.kv -> $k, $v { say "$k: $v" }
73. Example Module
JSON::Tiny
https://github.com/moritz/json
74. module JSON::Tiny;
proto to-json($) is export {*}
multi to-json(Real:D $d) { ~$d }
multi to-json(Bool:D $d) { $d ?? 'true' !! 'false'; }
multi to-json(Str:D $d) {
'"'
~ $d.trans(['"', '', "b", "f", "n", "r", "t"]
=> ['"', '', 'b', 'f', 'n', 'r', 't'])
.subst(/<-[c32..c126]>/, { ord(~$_).fmt('u%04x') }, :g)
~ '"'
}
multi to-json(Positional:D $d) {
return '[ ' ~ $d.map(&to-json).join(', ') ~ ' ]';
}
multi to-json(Hash:D $d) {
return '{ '~ $d.map({ to-json(.key)~' : '~to-json(.value) }).join(', ')~ ' }';
}
multi to-json(Any:U $) { 'null' }
multi to-json(Any:D $s) { die "Can't serialize an object of type "~$s.WHAT.perl}
use JSON::Tiny::Actions;
use JSON::Tiny::Grammar;
sub from-json($text) is export {
my $a = JSON::Tiny::Actions.new();
my $o = JSON::Tiny::Grammar.parse($text, :actions($a));
return $o.ast;
}
76. class JSON::Tiny::Actions;
method TOP($/) { make $/.values.[0].ast };
method object($/) { make $<pairlist>.ast.hash }
method pairlist($/) { make $<pair>>>.ast.flat }
method pair($/) { make $<string>.ast => $<value>.ast }
method array($/) { make $<arraylist>.ast }
method arraylist($/) { make [$<value>>>.ast] }
method string($/) {
make $0.elems == 1
?? ($0[0].<str> || $0[0].<str_escape>).ast
!! join '', $0.list.map({ (.<str> || .<str_escape>).ast });
}
method value:sym<number>($/) { make +$/.Str }
method value:sym<string>($/) { make $<string>.ast }
method value:sym<true>($/) { make Bool::True }
method value:sym<false>($/) { make Bool::False }
method value:sym<null>($/) { make Any }
method value:sym<object>($/) { make $<object>.ast }
method value:sym<array>($/) { make $<array>.ast }
method str($/) { make ~$/ }
method str_escape($/) {
if $<xdigit> {
make chr(:16($<xdigit>.join));
} else {
my %h = '' => "", '/' => "/", 'b' => "b", 'n' => "n",
't' => "t", 'f' => "f", 'r' => "r", '"' => """;
make %h{~$/};
}
}
77. Perl 6
Already full of awesome
Hundreds of examples on http://rosettacode.org
Developers happy to stay in stealth mode for now
Get ahead of the revolution: http://perl6.org
79. Perl
has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
for the last 25 years, and the next 25!