program without magic comment ruby Ruby 1.9

Ruby 1.9: It works! ihower@handlino.com 2009/4/18 OSDC.TW

Ruby 1.9: It (must) works! ihower@handlino.com 2009/4/18 OSDC.TW

? • a.k.a. ihower • http://ihower.idv.tw/blog/ • http://twitter.com/ihower • 2006 Ruby • ( ) Rails Developer • http://handlino.com • http://registrano.com

Agenda • Overview • What’s changed? • What’s New? (External Interater, M17N, The Threading Model) • Ruby 1.9 on Rails

1. Overview

Ruby 1.8.6 • Ruby 1.8.6, release at 2007/3/12 • Stable • Performance sucks • Memory leaks • NewRelic Report 69.4% http://railslab.newrelic.com/2009/04/01/the-state-of-the-stack-a-ruby-on-rails-benchmarking-report

Ruby 1.8.7 • Ruby 1.8.7 release at 2008/5/31 • 1.9 • gems, library ( 1.8.6 ) • JRuby 1.8.7 • NewRelic Report 14.5%

Ruby 1.9 • Ruby 1.9.0 release development version in 2007/12/25 • Ruby 1.9.1 release stable/production version in 2009/1/30 • Better performance • Better encoding support

Ruby Implementations production • Ruby 1.8.6, 1.8.7 (MRI, Matz’ Ruby Interpreter) • EngineYard 1.8.6 MBARI GC patch http://www.infoq.com/news/2009/01/ruby-patches-ﬁx-leaks • Ruby 1.9.1 (YARV) • JRuby • Ruby1.8.6 1.9 • Ruby Enterprise Edition(REE) • Ruby 1.8.6 GC memory leak

Ruby Implementations production • MacRuby (based on Objective-C) • Rubinius (Engine yard project) • MagLev (based on smalltalk) • IronRuby (based on Microsoft .NET) • Cardinal (based on Parrot VM)

Performance ( 1.8 2~2.5 ) http://antoniocangiano.com/category/ruby-benchmark-suite/ http://github.com/acangiano/ruby-benchmark-suite/

Ubuntu : Intel® Q6600® quad-core Computer Language Benchmarks Game http://shootout.alioth.debian.org/

2. What’s changed?

Ordered Hash { :a => 1, :b=> 2, :c => 3 }.merge( :d => 4 ) 1.8 {:a=>1, :d=>4, :b=>2, :c=>3} 1.9 {:a=>1, :b=>2, :c=>3, :d=>4}

new Hash literal config = { :foo => 1234, :bar => 6789 } 1.8,1.9 Person.find(:all, :conditions => { :name => 'ihower' } ) config = { foo: 1234, bar: 6789 } 1.9 Person.find(:all, :conditions => { name: 'ihower' } )

Hash syntax => {1,2,3,4} 1.8 => {1=>2, 3=>4} {1,2,3,4} 1.9 => syntax error, unexpected ',', expecting tASSOC

Hash#select hash = { :a => 1, :b => 2, :c => 3 } hash.select{ |k, v| v > 1 } [[:b, 2], [:c, 3]] 1.8 {:b=>2, :c=>3} 1.9

Array#to_s, Hash#to_s 1.8 1.9 [1,2,3,4].to_s [1,2,3,4].to_s => quot;1234quot; => quot;[1, 2, 3, 4]quot; {1=>2,3=>4}.to_s {1=>2,3=>4}.to_s => quot;1234quot; => quot;{1=>2, 3=>4}quot;

case syntax name = case name = case when x == 1 : quot;onequot; when x == 1 1.8 when x == 2 : quot;twoquot; quot;onequot; 1.8,1.9 else quot;manyquot; when x == 2 end quot;twoquot; else quot;manyquot; end name = case when x == 1 then quot;onequot; when x == 2 then quot;twoquot; 1.8,1.9 else quot;manyquot; end

block’s parameter always local local x = quot;fooquot; y = quot;barquot; [1,2,3].each do |x| y=x+1 end 1.8 [x,y] => [3, 4] 1.9 [x,y] => [quot;fooquot;, 4]

block-local variable local bar = quot;ihowerquot; 1.9 bar = quot;ihowerquot; 1.8,1.9 [1,2,3].each do |val; bar| [1,2,3].each do |val| bar = val foo = val end bar = val end bar => quot;ihowerquot; foo # NameError: undefined local variable or method `foo’ bar => 3

block-local variable local bar = quot;ihowerquot; 1.9 bar = quot;ihowerquot; 1.8,1.9 [1,2,3].each do |val; bar| [1,2,3].each do |val| bar = val foo = val end bar = val end bar => quot;ihowerquot; foo # NameError: undefined local variable or method `foo’ bar bar => 3

block can accept block argrments proc1 = lambda do |a, *b, &block| pa pb pc block.call end proc1.call(1,2,3,4) { puts quot;in block1quot; } # 1 [2,3,4] quot;in block1quot;

new lambda literal lambda lambda { |a| a*3 }.call(4) 1.8,1.9 # => 12 lambda { |a| a*3 }[4] 1.8,1.9 # => 12 ->(a) { a*3 }.(4) 1.9 # => 12

def my_if(condition, then_clause, else_clause) if condition then_clause.call else else_clause.call end end 5.times do |val| my_if (val < 3), -> { puts quot;#{val} is smallquot; }, -> { puts quot;#{val} is bigquot; } end # 0 is small 1 is small 2 is small 3 is big 4 is big

def my_while(cond, &body) while cond.call body.call end end a=0 my_while -> { a < 3 } do puts a a += 1 end # 0 1 2

Kernel.proc , do not use it! • proc lambda 1.8 • 1.9 proc Proc.new

BasicObject root Class.superclass.superclass => Object 1.8,1.9 1.8 Class.superclass.superclass.superclass => nil Class.superclass.superclass.superclass => BasicObject 1.9 BasicObject.superclass => nil BasicObject.instance_methods => [:==, :equal?, :!, :!=, :instance_eval, :instance_exec, :__send__] metaprogramming blank canvas

standard library changes • + Rubygems • no longer needed require ‘rubygems’ • + Rake • no longer gem install rake • ~ Test::Unit replaced by MiniTest • - soap, jcode...etc, some rarely used, old libraries

3. What’s new?

External Iterator

Iterators to_enum Internal External 1.9 1.8,1.9 arr = [ 1, 2, quot;ihowerquot; ] arr = [ 1, 2, quot;ihowerquot; ] arr.each do |a| enum_a = arr.to_enum puts a # Enumerators ( each) end enum_a.next # => 1 # => 1 enum_a.next # => 2 # => 2 enum_a.next # => quot;ihowerquot; # => quot;ihowerquot;

Enumerators 1.9 1.9 arr = [ 1, 2, quot;ihowerquot; ] h = { foo: quot;567quot;, bar: quot;890quot; } enum_a = enum_h = h.to_enum arr.to_enum(:each_with_index) enum_h.next # => [:foo, quot;567quot;] enum_a.next # => [1, 0] enum_h.next enum_a.next # => [2, 1] # => [:bar, quot;890quot;] enum_a.next # => [quot;ihowerquot;, 2]

StopIteration exception 1.9 short_enum = [1, 2, 3].to_enum long_enum = ('a'..'z').to_enum loop do puts quot;#{short_enum.next} #{long_enum.next}quot; end # => 1-a # => 2-b # => 3-c

StopIteration exception 1.9 short_enum = [1, 2, 3].to_enum long_enum = ('a'..'z').to_enum loop do puts quot;#{short_enum.next} #{long_enum.next}quot; end # => 1-a StopIteration # => 2-b # => 3-c

Enumerator from iterator iterator method 1.9 arr = [ 1, 2, quot;ihowerquot; ] enum_a = arr.each # Enumerator enum_a.next # => 1 enum_a.next # => 2 enum_a.next # => quot;ihowerquot;

Enumerators.new code block 1.9 seq = Enumerator.new do |yielder| n1 = 0 n2 = 1 loop do n3 = n1 + n2 yielder.yield n3 n1 = n2 n2 = n3 end end seq.next # 1 seq.next # 2 2.times { puts seq.next } #3 #5

Enumerator objects are also enumerable 1.9 seq.first(10) # => [1, 2, 3, 5, 8, 13, 21, 34, 55, 89] seq.to_a # Don’t do this if your Enumerator has infinite elements

M17N (Multilingualization)

In Ruby 1.8, a String is always just a collection of bytes

Ruby 1.8 String fails >> quot; quot;.reverse 1.8 => quot;207226 344quot; >> quot; quot;.size => 6

Ruby 1.8 Regexp UTF-8 • Support None (n or N), EUC (e or E), Shift_JIS (s or S), UTF-8 (u or U) >> quot; quot;.scan(/./u) 1.8,1.9 => [quot; quot;, quot; quot;]

Use Regexp to handle String Regexp >> quot; quot;.scan(/./u).reverse.join 1.8,1.9 => quot; quot; >> quot; quot;.scan(/./u).size => 2

$KCODE = “U” or ruby -KU • regular expressions default encoding • $KCODE = quot;Uquot; terminal >> quot; quot; 1.8 => quot;344270255346226207quot; >> $KCODE='u' >> quot; quot; => quot; quot;

Unicode codepoint Ruby 1.8 UTF-8 >> quot; quot;.unpack(quot;U*quot;) 1.8,1.9 => [20013, 25991] >> quot; quot;.unpack(quot;U*quot;).reverse.pack(quot;U*quot;) => quot; quot; >> quot; quot;.unpack(quot;U*quot;).size => 2

Unicode codepoint(cont.) Rails ActiveSupport >> quot; quot;.chars # in Rails 1.8,1.9 => #<ActiveSupport::Multibyte::Chars:0x25aebbc @wrapped_string=quot; quot;> >> quot; quot;.chars.size => 2

iconv library UTF-8 data • C Library, it’s fast. • convert your encoding to UTF-8 • handle it • export back to your encoding

jcode library too simple, not useful $KCODE = 'UTF8' 1.8 require 'jcode' str = quot; quot; str.size # 9 str.jsize # 2

No enough encodings supported in Ruby 1.8 • String Encoding • Regexp • encoding Regexp letter • #String.upcase encoded data valid? • $KCODE • String, IO object, program source

pick one encoding, likely Unicode, and works all data in one format? No. Ruby 1.9 make it possible to work with data with 83 encoding.

>> Encoding.name_list => [quot;ASCII-8BITquot;, quot;UTF-8quot;, quot;US-ASCIIquot;, quot;Big5quot;, quot;CP949quot;, quot;Emacs-Mulequot;, quot;EUC-JPquot;, quot;EUC-KRquot;, quot;EUC-TWquot;, quot;GB18030quot;, quot;GBKquot;, quot;ISO-8859-1quot;, quot;ISO-8859-2quot;, quot;ISO-8859-3quot;, quot;ISO-8859-4quot;, quot;ISO-8859-5quot;, quot;ISO-8859-6quot;, quot;ISO-8859-7quot;, quot;ISO-8859-8quot;, quot;ISO-8859-9quot;, quot;ISO-8859-10quot;, quot;ISO-8859-11quot;, quot;ISO-8859-13quot;, quot;ISO-8859-14quot;, quot;ISO-8859-15quot;, quot;ISO-8859-16quot;, quot;KOI8-Rquot;, quot;KOI8-Uquot;, quot;Shift_JISquot;, quot;UTF-16BEquot;, quot;UTF-16LEquot;, quot;UTF-32BEquot;, quot;UTF-32LEquot;, quot;Windows-1251quot;, quot;BINARYquot;, quot;IBM437quot;, quot;CP437quot;, quot;IBM737quot;, quot;CP737quot;, quot;IBM775quot;, quot;CP775quot;, quot;CP850quot;, quot;IBM850quot;, quot;IBM852quot;, quot;CP852quot;, quot;IBM855quot;, quot;CP855quot;, quot;IBM857quot;, quot;CP857quot;, quot;IBM860quot;, quot;CP860quot;, quot;IBM861quot;, quot;CP861quot;, quot;IBM862quot;, quot;CP862quot;, quot;IBM863quot;, quot;CP863quot;, quot;IBM864quot;, quot;CP864quot;, quot;IBM865quot;, quot;CP865quot;, quot;IBM866quot;, quot;CP866quot;, quot;IBM869quot;, quot;CP869quot;, quot;Windows-1258quot;, quot;CP1258quot;, quot;GB1988quot;, quot;macCentEuroquot;, quot;macCroatianquot;, quot;macCyrillicquot;, quot;macGreekquot;, quot;macIcelandquot;, quot;macRomanquot;, quot;macRomaniaquot;, quot;macThaiquot;, quot;macTurkishquot;, quot;macUkrainequot;, quot;CP950quot;, quot;stateless-ISO-2022-JPquot;, quot;eucJPquot;, quot;eucJP-msquot;, quot;euc-jp-msquot;, quot;CP51932quot;, quot;eucKRquot;, quot;eucTWquot;, quot;GB2312quot;, quot;EUC-CNquot;, quot;eucCNquot;, quot;GB12345quot;, quot;CP936quot;, quot;ISO-2022- JPquot;, quot;ISO2022-JPquot;, quot;ISO-2022-JP-2quot;, quot;ISO2022-JP2quot;, quot;ISO8859-1quot;, quot;Windows-1252quot;, quot;CP1252quot;, quot;ISO8859-2quot;, quot;Windows-1250quot;, quot;CP1250quot;, quot;ISO8859-3quot;, quot;ISO8859-4quot;, quot;ISO8859-5quot;, quot;ISO8859-6quot;, quot;Windows-1256quot;, quot;CP1256quot;, quot;ISO8859-7quot;, quot;Windows-1253quot;, quot;CP1253quot;, quot;ISO8859-8quot;, quot;Windows-1255quot;, quot;CP1255quot;, quot;ISO8859-9quot;, quot;Windows-1254quot;, quot;CP1254quot;, quot;ISO8859-10quot;, quot;ISO8859-11quot;, quot;TIS-620quot;, quot;Windows-874quot;, quot;CP874quot;, quot;ISO8859-13quot;, quot;Windows-1257quot;, quot;CP1257quot;, quot;ISO8859-14quot;, quot;ISO8859-15quot;, quot;ISO8859-16quot;, quot;CP878quot;, quot;SJISquot;, quot;Windows-31Jquot;, quot;CP932quot;, quot;csWindows31Jquot;, quot;MacJapanesequot;, quot;MacJapanquot;, quot;ASCIIquot;, quot;ANSI_X3.4-1968quot;, quot;646quot;, quot;UTF-7quot;, quot;CP65000quot;, quot;CP65001quot;, quot;UTF8-MACquot;, quot;UTF-8-MACquot;, quot;UCS-2BEquot;, quot;UCS-4BEquot;, quot;UCS-4LEquot;, quot;CP1251quot;, quot;localequot;, quot;externalquot;, quot;internalquot;]

All String are Encoded • In Ruby 1.9 a String is a collection of encoded characters. raw bytes Encoding >> quot; quot;.encoding.name 1.9 => quot;UTF-8quot; Encoding object

String works in characters 1.9 >> quot; quot;.reverse => quot; quot; >> quot; quot;.size => 2 >> quot; quot;.bytesize => 6

String Indexing 1.9 1.8 >> quot;abcquot;[0] >> quot;abcquot;[0] => 97 => quot;aquot; >> quot; quot;[1] >> quot; quot;[1] => 184 => quot; quot; >> quot;abcquot;[0].ord => 97

! In Ruby 1.9, String has attached Encoding object, and works in characters.

Transcoding utf8 = quot; quot; 1.9 utf8.bytesize # 6 utf8.bytes.to_a # [230, 184, 172, 232, 169, 166] big5 = utf8.encode(quot;big5quot;) big5.encoding.name # ”Big5” big5.bytesize # 4 big5.bytes.to_a # [180, 250, 184, 213]

Transcoding fails str = quot;Résuméquot; 1.9 str.encode(quot;big5quot;) => Encoding::UndefinedConversionError: quot;xC3xA9quot; from UTF-8 to Big5 from (irb):2:in `encode' from (irb):2 from /usr/local/bin/irb19:12:in `<main>'

Force Transcoding byte data utf8 = quot; quot; 1.9 big5 = utf8.encode(quot;big5quot;) big5.valid_encoding? => true big5.force_encoding(quot;utf-8quot;) big5.valid_encoding? => false

Force Transcoding fails big5.valid_encoding? # false 1.9 big5 =~ /123456/ => ArgumentError: invalid byte sequence in UTF-8 from (irb):11 from /usr/local/bin/irb19:12:in `<main>'

Encoding.compatible? : ASCII with a bigger Encoding ascii = quot;my quot;.force_encoding(quot;asciiquot;) 1.9 utf8 = quot;Résuméquot; # Encoding.compatible?(ascii, utf8) #<Encoding:UTF-8> # my_resume = ascii + utf8 puts my_resume # quot;My Résuméquot; puts my_resume.encoding.name # UTF-8

Encoding.compatible? big5 = quot; quot;.encode(quot;big5quot;) 1.9 utf8 = quot;Résuméquot; # Encoding.compatible?(big5, utf8) # nil # big5 + utf8 => Encoding::CompatibilityError: incompatible character encodings: Big5 and UTF-8 from (irb):25 from /usr/local/bin/irb19:12:in `<main>'

String Iteration each • Strings are no longer enumerable • Ruby 1.8 each() has been removed • 1.9 Ruby 1.9 use explicit iterator • each_line • each_byte • each_char • each_codepoint

What’s default encoding? ? encoding ?

program source encoding magic comment # encoding: UTF-8 #!/usr/bin/env ruby -w # encoding: UTF-8 # coding: UTF-8 # -*- coding: UTF-8 -*-

program without magic comment ruby foobar.rb # US-ASCII ruby -e foobar.rb irb # OS ($LC_CTYPE, $LANG) ruby -KU foobar.rb # UTF-8

program source encoding error invalid multibyte char (US-ASCII)

IO Object 1.9 # encoding: utf-8 f = File.open(quot;big5.txtquot;,quot;r:big5quot;) puts f.external_encoding # Big5 puts f.gets.encoding # Big5

IO Object 1.9 # encoding: utf-8 f = File.open(quot;big5.txtquot;,quot;r:big5:utf-8quot;) puts f.external_encoding # Big5 puts f.internal_encoding # UTF-8 puts f.gets.encoding # UTF-8

IO Object 1.9 # encoding: utf-8 f = File.open(quot;another_big5.txtquot;, quot;w:big5quot;) puts f.external_encoding # Big5 data = quot; quot; puts data.encoding.name # UTF-8 f << data

Regexp has encoding too • Oniguruma engine 1.9 http://www.geocities.jp/kosako3/oniguruma/ /w/.encoding • Same basic API => #<Encoding:US-ASCII> • better performance / /.encoding => #<Encoding:UTF-8> • support for a lot of encoding • extended syntax

Default Encoding Overview • String literal program source • Symbols Regexp 7-bit US-ASCII program source • IO external_encoding OS • ruby -E • Encoding.external_encoding

The Threading model

Fibers (semi-coroutine) code block

f = Fiber.new { 1.9 2.times do puts quot;Fiber say hiquot; Fiber.yield # puts quot;Fiber say byequot; end }# >> f.resume # Fiber say hi >> f.resume # Fiber say bye # Fiber say hi >> f.resume # Fiber say bye >> f.resume FiberError: dead fiber called from (irb):78:in `resume' from (irb):78 from /usr/local/bin/irb19:12:in `<main>'

Fibers • Enumerator • ﬁber library ﬁber object • (Ruby 1.8 continuation continuation kernel library) • ... zzz

Native Threads • Green threads 1.8 Ruby thread native thread • Native (operation system) thread 1.9 Ruby thread native thread

GIL Giant Interpreter Lock • 1.9 C extension libraries thread safe Thread ( Ruby Threads JRuby)

4. Ruby on Rails

Ruby 1.9 on Rails • Rails 2.3.2 or edge • mysql-ruby driver for Ruby 1.9 • Passenger(mod_rails) or Thin

Upgrade issue • program source encoding problem • invalid multibyte char # encoding: UTF-8 • encoding compatibility problem • ACSII-8BIT (binary) UTF-8 String • http://github.com/hectoregm/mysql-ruby (UTF-8 friendly) • some gem, plugins are not Ruby 1.9 compatible • http://isitruby19.com/

Benchmark • Rails 2.3.2 production mode • No DB, just render :text => 'hello' • My MacBook Pro 2.2G • httperf

Performance Reply rate [replies/s] min avg max stddev Ruby 1.8.6 241.8 258.3 267.4 14.4 Mongrel 1.1.5 Ruby 1.8.6 244.6 260.9 271.2 11.4 mod_rails Ruby 1.9.1 264.0 269.5 273.4 4.9 mod_rails Ruby 1.8.6 244.5 273.2 299.6 22.7 Thin 1.0.0 Ruby 1.9.1 348.4 363.2 371.2 12.8 Thin 1.0.0

Performance Ruby 1.9 Rails Stack !! 400 300 Ruby 1.8.6/Mongrel 1.1.5 Ruby 1.8.6/mod_rails edge 200 Ruby 1.9.1/mod_rails edge Ruby 1.8.6/Thin 1.0.0 Ruby 1.9.1/Thin 1.0.0 100 0 Reply rate [replies/s]

5. Conclusion

TIOBE Programming Community Index for April 2009

Ruby challenge • Ruby 1.9 change a lot and 1.8 is slow and should be deprecated. • Migrate to Ruby 1.9 is the big challenge in 2009.

Reference • Understanding M17n http://blog.grayproductions.net/articles/understanding_m17n • Books • Programming Ruby 1.9 (Pragmatic) • The Ruby Programming Language (O’Reilly) • Ruby Best Practices (O’Reilly) • The Well-Grounded Rubyist (Manning) • Slide • MigrationToRuby 1.9 • Ruby 1.9: What to Expect http://slideshow.rubyforge.org/ruby19.html • Matz on Ruby 1.9

Other reference links • http://svn.ruby-lang.org/repos/ruby/tags/v1_9_1_0/NEWS • http://blog.nuclearsquid.com/writings/ruby-1-9-what-s-new-what-s-changed • http://blog.grayproductions.net/articles/getting_code_ready_for_ruby_19 • http://dablog.rubypal.com/2009/1/14/10-things-to-be-aware-of-in-moving-to- ruby-1-9 • http://dablog.rubypal.com/2009/1/16/son-of-10-things-to-be-aware-of-in- ruby-1-9 • http://blog.kineticweb.com/articles/2009/01/30/ruby-1-9-1-the-other-features

Thank you.

Bonus

Object#tap puts quot;dogquot;.reverse .tap{ |o| puts quot;reversed: #{o}quot; } .upcase # reversed: god GOD

program without magic comment ruby Ruby 1.9

Wen-Tien Chang

program without magic comment ruby