Treetop - I'd rather have one problem

Some people, when faced with a problem think,
“I know, I’ll use regular expressions”.
Now they have two problems.

I’d rather have one problem.

Treetop • Roland Swingler • LRUG May 2009

Tuesday, 19 May 2009

This quotation is used a lot in presentations, normally before the presenter delves into some
gnarly regexps. I’m looking for a better way.

Example 1



I run a ﬁlm listing site: http://ﬁlmli.st. All the data is scraped from other sites - getting the
data is easy with net/http or httparty or similar and then parsing the html with nokogiri or
hpricot, but...

<span>
Fri/Sun-Tue 10.45 12.30 (Tue) 12.40 (not Tue)
4.00 7.00 9.30; Wed 3.00 7.30 9.00
</span>


... you still need to turn a text string like this into a list of Times so you can do interesting
things with it. Regexps? No. That way lies madness.

Example 2



Chatroom bots need to be able to distinguish between messages that they should take
actions on and those which they should ignore. How should we deﬁne what messages they
should listen out for?

/^s*whereiss+(.+?)(?:s+(?:ons+)?(.+?))?s*$/


Regular expressions? Pretty confusing.

whereis <person> [[on] <day>]


Much nicer to have a simpler language.

Example 3


Scenario: producing human-readable tests
Given I have non-technical stakeholders
When I write some integration tests
Then they should be understandable by everyone


Wouldn’t it be great if someone had written a library like this?


They have! Cucumber. Cucumber’s implementation got me started looking into...


Treetop. A ruby Parsing Expression Grammar. Basically a parser generator, but really simple.

What is a parser?


A parser determines whether strings are syntactically valid according to a set of rules known
as a grammar.

Yes / No


From a theoretical viewpoint, parsers just say true or false, depending on whether the string
is valid or not.

Syntax Tree


Not so useful, so instead we get back a syntax tree we can do useful things with.

whereis <person> [on <day>]


Lets try building a tree for this example. You can consider a string to be a list of characters,
but to start getting meaning from it, you need a tree.

words words



We have some words...

words variable words variable



variables...

optional part




an optional part of an expression (enclosed with square brackets)

expression

optional part




and a root node for the whole expression

grammar Message
end


lets build that up in treetop. Each of those four types of node in the tree is going to have a
rule. We write these rules in a grammar - you think of it like a ruby module.

grammar Message
rule expression
(words / variable / optional_part)+
end
end


The ﬁrst rule for the whole expression. Lots of things should be familiar from regular
expressions - ‘+’ for one or more, brackets for grouping, and ‘/’ is like the regexp ‘|’ for
alternation. So this says an expression is one or more words, variables or optional parts, in
any order.

grammar Message
rule expression
end

rule words
[^><[]]+
end
end


words - character classes, just like regexps

grammar Message
rule expression
end

rule words
[^><[]]+
end

rule variable
'<' identifier:( [a-zA-Z_] [a-zA-Z_0-9 ]* ) '>'
end
end


variables are enclosed with angle brackets, can be any valid ruby identiﬁer string, and are
labeled so we can use part of the text later.

grammar Message
rule expression
end

rule words
[^><[]]+
end

rule variable
'<' identifier:( [a-zA-Z_] [a-zA-Z_0-9 ]* ) '>'
end

rule optional_part
quot;[quot; expression quot;]quot;
end
end


optional parts are enclosed with square brackets. Here we see that rules can be recursive -
which makes the parser signiﬁcantly more powerful than regular expressions.

$ tt message.treetop


We compile the grammar with the command line tt command - you can also load grammars
dynamicaly

require ‘message’

parser = MessageParser.new
tree = parser.parse(“whereis <person>...”)


this gives us a parser we can call from ruby code

require ‘message’

parser = MessageParser.new
tree = parser.parse(“whereis <person>...”)

tree.elements[0].text_value
#=> “whereis ”

tree.elements[1].identifier.text_value
#=> “person”


each node knows about its children and its text_value. The label we deﬁned earlier provides
sugar methods to access particular subnodes.

Fri/Sun-Tue 4.00 7.00


Another example. This time we’ll think about the tree in a top down fashion rather than
bottom up. This is closer to how treetop will actually evaluate an expression.

expression



expression

days times



expression

days times

day day range time time

Fri / Sun-Tue 4.00 7.00


expression

days times

day day range time time

day day hrs mins hrs mins

Fri / Sun - Tue 4 . 00 7 . 00


rule expression
days “ ” times
end


rule times
time (“ ” time)+
end

rule time
hours “.” minutes
end

rule hours
1 [0-2] / [0-9]
end

rule minutes
[0-5] [0-9]
end


rule days
(day !“-” / day_range) (“/” days)?
end

rule day_range
day “-” day
end

rule day
“Mon”/“Tue”/“Wed”/“Thu”/“Fri”/“Sat”/“Sun”
end


The bit highlighted in red is a negative lookahead assertion. We need this because treetop
evaluates alternatives from left to right - if we didn’t have the assertion then Sun-Tue would
match Sun as a Day, not a DayRange, and we’d be left with “-Tue” which isn’t valid.

Enriching Nodes


Adding in some semantics

rule time
hours “.” minutes
end

irb> aTimeNode.text_value #=> “9.00”
irb> aTimeNode.elements.size #=> 3
irb> aTimeNode.hours.text_value #=> “9”


rule time
hours “.” minutes {
def to_seconds
hours.to_i * 60 * 60 + minutes.to_i * 60
end
}
end

irb> aTimeNode.text_value #=> “9.00”
irb> aTimeNode.to_seconds #=> 32400


We can add in methods inline in the grammar. This is just like a module scope, and we can
do any ruby we like in here.

# in film_time.treetop
rule time
hours “.” minutes <TimeNode>
end

# in another .rb file
class TimeNode < Treetop::Runtime::SyntaxNode
def to_seconds
hours.to_i * 60 * 60 + minutes.to_i * 60
end
end


Cleaner in my mind to split these out into actual subclasses of SyntaxNode - keeps the
grammar more readable. In some cases you need to have modules rather than subclasses.

Interpretation &
Compilation


We’re going to build up a regular expression for the bot example. Each node will be
reponsible for building a different part of the regexp.

expression

optional part


/^whereis (.+?)(?:s+on (.+?))?$/


Interpreter Pattern


This is confusing - it comes from GoF. Actually we’re doing compilation here. Each node gets
an interpret method - you treat the syntax tree as a composite.

# expression
def interpret
children = elements.map {|node| node.interpret }
RegExp.compile(“^” + children.join + “$”)
end


# words
def interpret
Regexp.escape(text_value)
end


# variable
def interpret
“(.+?)”
end


# optional_part
def interpret
children = elements.map {|node| node.interpret }
“(?:s+” + children.join + “)?”
end


Adding context


For anything more than a simple language, you’ll need to pass around context as you
interpret the tree.

# expression
def interpret(context=[])
children = elements.map do |node|
node.interpret(context)
end
matcher = RegExp.new(“^” + children.join + “$”)
...


In our case we just want to record the list of variable names, so an Array will suffice. Each
interpret method now needs to take this context.

# variable
def interpret(context)
context << identifier.text_value.to_sym
“(.+?)”
end


# expression
def interpret(context=[])
children = elements.map do |node|
node.interpret(context)
end
matcher = RegExp.new(“^” + children.join + “$”)

class << matcher
send(:define_method, :variables) do
context
end
end
matcher
end


we decorate the regular expression with a list of the variables. In the real code, the returned
match objects are also decorated so you have methods for each variable and don’t have to
remember the captured groups by position

Other Options


You can also build external interpreters / compilers that use the tree

Complications?


# We want to write:
hello [world]

# We actually mean:
hello[ world]


whitespace shuffling. In the reall code, grammar is more complicated - most of the
complication comes from dealing with edge cases here

# We should optimize:
hello [[[world]]]

# To this:
hello [world]


This isn’t done in the real code, but should be.

# Left recursion without consuming input BAD:
rule infinity_and_beyond
infinity_and_beyond / “foo”
end


Problems?


Slow.

Other libraries


Racc - accepts yacc grammars. Racc runtime is part of the ruby std dist. so once you’ve built
your parser there is no dependency. Ragel - used by mongrel/thin.

Thanks!

Twitter: @knaveofdiamonds

XMPP bot:
http://github.com/knaveofdiamonds/harken

Film listings for London’s indie cinemas:
http://filmli.st

Treetop:
http://github.com/nathansobo/treetop
http://treetop.rubyforge.org


Treetop - I'd rather have one problem

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Empfohlen

Empfohlen (20)

Treetop - I'd rather have one problem