Presentation of the Dolnośląska Ruby User Group, which took place on 20.02.2023.
Compared to the previous version, it contains 1 regular expression fix and Ruby 3.2 improvements.
Regex is used for Google Analytics and much more!
We explore Ruby's default regexp class, but go beyond (so it's for advanced developers):
- Explaining how backtracking and deterministic finite automaton work.
- Why get the RE2 library from Google? or regular expressions at all?
- What Ruby regex improvements ship with Ruby 3.2?
3. Who am I?
Piotr Wasiak
Ruby, Rails developer
Current PRUG organiser
3
Interests:
● climbing, hiking, squash
● contract bridge, chess
● ruby, programming, crypto
4. Regular Expression
is a character sequence, that defines a search pattern
The purpose is:
● validate the string by the pattern
● get parts of the content (e.g. find or find_and_replace in text editors)
4
5. RegEx history
● Concept of language arose in the 1950s
● Different syntaxes (1980+):
○ POSIX (Basic - or Extended Regular Expressions)
○ Perl (influenced/imported to other languages as PCRE 1997, PCRE2 2015)
5
6. RegEx as a state machine
6
Statement validation: /(?<name>ADAM|PIOTR)s?[=><]{1,2}s*"(?:PIENIĄDZ|KUKU)"/g
15. original_regexp = %r{ # there is no heredoc for regexp
(?: # strings with some special chars, but not ending with .
[a-z0-9!#$%&'*+/=?^_`{|}~-]+
(?:
.[a-z0-9!#$%&'*+/=?^_`{|}~-]+
)*
|
"
(?: # special chars enquoted
[x01-x08x0bx0cx0e-x1f!#-x5b]-x7f]
|
# prepended with backslash, here escaped
[x01-x09x0bx0cx0e-x7f] # more special chars
)*
" # closing quote
)
@ # the most crucial ampersand
(?: # domain regexp
(?: # at least one subdomain joined and finished with .
[[:alnum:]]
(?:
[a-z0-9-]* # subdomain can have many alphanumeric or - inside
[[:alnum:]] # subdomain have to finish with alphanumeric char
)?
. # dot separator
)+
[[:alnum:]] # domain have to start with alphanumeric char
(?:
[a-z0-9-]* # domain can have many alphanumeric or - inside
[[:alnum:]] # domain have to finish with alphanumeric char
)? 15
/x comments mode
| # or direct ip implementation or 3 numbers
with . suffix and some special usecases
[ # enquoted with square brackets
(?:
(?: # numbers are quite complex in RegEx
25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]? #
0-255
). # . suffix
){3} # 3 times
(?:
25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]? # 0-255
| # or 3 numbers with . suffix and some
special usecases
[a-z0-9-]* # alnums also starting with -
[[:alnum:]] # finishing without -
:
(?:
[x01-x08x0bx0cx0e-x1f!-Z]-x7f] #
many chars
|
# more ansii chars prefixed with
backslash
[x01-x09x0bx0cx0e-x7f]
)+
)
] # closing square bracket
)
}x # switch to treat spaces/new lines and `# `
suffix as comments
16. Ruby simply string methods are faster and more meaningful:
● .start_with? / .end_with?
● .include?(‘some substring’)
● .chomp
● .strip
● .lines
● .split(‘ ’) # without regexp
● .tr(‘ !?‘, ‘1-9’)
16
Do not overuse regular expression (1/2)
17. Libraries and gems for common concepts:
● URI(url)
+ .host / .path / .query / .fragment
● File(path_to_file)
+ .dirname / .basename / .extname
● Nokogiri::HTML(
open('https://nokogiri.org/’)
)
17
Do not overuse regular expression (2/2)
18. Do not use REGEX as language parser
Programming languages depend more on language nodes/tree.
There will be always a problem with some exceptions, different coding
styles
In Ruby we need to use Ripper or other tools to decompose Ruby code
into pieces
Markup languages can be parsed by e.g. Nokogiri, Ox, Oj gems easier
and more secure
18
19. Clear RegEx
● extract common parts in alternation
● put more likely to appear words in the front of alternation
● use comments and whitespace with /x modifier
● give a name for captured groups, use also non-captured
● split code to smaller logical pieces
● lint code with ruby -w for warnings
19
24. - in general "dot matches at line breaks mode" is turn on with s flag
instead of ruby m flag
- In Ruby, ^ and $ always match on every line.
If you want to specify the beginning of the string, use A.
For the very end of the string, use z (or Z including final line break).
Quirks in Ruby RegEx engine (1/3)
24
25. Quirks in Ruby RegEx engine (2/3)
Ruby does not allow
● look-ahead
● negative look-behind
inside a look-behind, such as:
25
26. - Intersection […&&[…]]
- Subtraction […&&[^…]]
26
Quirks in Ruby RegEx engine (3/3)
Character classes operators
31. Tools / Websites
● regex101.com/
nicest editor, explanation on hover, cheatset, performance analysis
● www.debuggex.com/ visualized graphs with cheat-set
● Visualization plugins for Visual Studio Code
● rubocop and rubocop-performance have some rules for regex
● rubular.com/ check if RegEx works in Ruby 2.5. Other with 2.1
● rubyapi.org/3.1/o/regexp good Ruby docs
31
36. “Most modern engines are regex-directed because this is the only way to
implement useful features such as lazy quantifiers and backreferences;
and atomic grouping and possessive quantifiers that give extra control
to backtracking.”
PCRE like solutions
36
45. 6. Ruby 3.2 RE changes
45
Regexp improvements against ReDoS
It is known that Regexp matching may take unexpectedly long.
If your code attempts to match a possibly inefficient Regexp against an
untrusted input, an attacker may exploit it for efficient Denial of Service