The document discusses PHP's lexer and parser. It describes how PHP code is converted to an abstract syntax tree (AST) representation by the lexer and parser. The lexer tokenizes code into tokens using a lexer definition file called zend_language_scanner.l. The parser then converts the tokenized code into an AST using grammar rules defined in zend_language_parser.y. Examples of if statement parsing rules and an if statement AST representation are provided.
81. @asgrim
Order tokens by operator precedence
if ($token->isOperator()) {
$tokenPrecedence = self::$operatorPrecedence[$token->getToken()];
while (
count($operators)
&& self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()]
> $tokenPrecedence
) {
$higherOp = array_pop($operators);
$stack[] = $higherOp;
}
$operators[] = $token;
next($tokens);
continue;
}
82. @asgrim
Order tokens by operator precedence
if ($token->isOperator()) {
$tokenPrecedence = self::$operatorPrecedence[$token->getToken()];
while (
count($operators)
&& self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()]
> $tokenPrecedence
) {
$higherOp = array_pop($operators);
$stack[] = $higherOp;
}
$operators[] = $token;
next($tokens);
continue;
}
83. @asgrim
Order tokens by operator precedence
if ($token->isOperator()) {
$tokenPrecedence = self::$operatorPrecedence[$token->getToken()];
while (
count($operators)
&& self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()]
> $tokenPrecedence
) {
$higherOp = array_pop($operators);
$stack[] = $higherOp;
}
$operators[] = $token;
next($tokens);
continue;
}
84. @asgrim
Order tokens by operator precedence
if ($token->isOperator()) {
$tokenPrecedence = self::$operatorPrecedence[$token->getToken()];
while (
count($operators)
&& self::$operatorPrecedence[$operators[count($operators) - 1]->getToken()]
> $tokenPrecedence
) {
$higherOp = array_pop($operators);
$stack[] = $higherOp;
}
$operators[] = $token;
next($tokens);
continue;
}
85. @asgrim
Order tokens by operator precedence
// Clean up by moving any remaining operators onto the token stack
while (count($operators)) {
$stack[] = array_pop($operators);
}
return $stack;
108. @asgrim
astkit example usage
$if = AstKit::parseString(<<<EOD
if (true) {
echo "This is a triumph.n";
} else {
echo "The cake is a lie.n";
}
EOD
);
$if->execute(); // First run, program is as-seen above
$const = $if->getChild(0)->getChild(0);
// Replace the "true" constant in the condition with false
$const->graft(0, false);
// Can also graft other AstKit nodes, instead of constants
$if->execute(); // Second run now takes the else path
115. @asgrim
Class BetterReflection
public function sourceLocator() : SourceLocator
{
$astLocator = $this->astLocator();
return $this->sourceLocator
?? $this->sourceLocator = new MemoizingSourceLocator(new AggregateSourceLocator([
new PhpInternalSourceLocator($astLocator),
new EvaledCodeSourceLocator($astLocator),
new AutoloadSourceLocator($astLocator),
]));
}
public function classReflector() : ClassReflector
{
return $this->classReflector
?? $this->classReflector = new ClassReflector($this->sourceLocator());
}
116. @asgrim
Given a class structure...
<?php
class Foo
{
private $bar;
public function thing()
{
}
}
117. @asgrim
⊠we get the AST!
Class, name Foo
|-- Statements
| |-- Property, name bar
| | |-- Type [private]
| | `-- Attributes [start line: 7, end line: 9]
| `-- Method, name thing
| |-- Type [public]
| |-- Parameters [...]
| |-- Statements [...]
| `-- Attributes [start line: 7, end line: 9]
`-- Attributes [start line: 3, end line: 10]
121. @asgrim
Monkey patching example
use RoaveBetterReflectionReflectorClassReflector;
use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator;
use RoaveBetterReflectionUtilAutoloadClassLoader;
use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader;
$loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__));
// Create the reflection first (without loading)
$classInfo = (new ClassReflector(
new SingleFileSourceLocator(
__DIR__ . '/MyClass.php',
(new BetterReflection())->astLocator()
)
))->reflect('MyClass');
$loader->addClass($classInfo);
122. @asgrim
Monkey patching example
use RoaveBetterReflectionReflectorClassReflector;
use RoaveBetterReflectionSourceLocatorTypeSingleFileSourceLocator;
use RoaveBetterReflectionUtilAutoloadClassLoader;
use RoaveBetterReflectionUtilAutoloadClassLoaderMethodFileCacheLoader;
$loader = new ClassLoader(FileCacheLoader::defaultFileCacheLoader(__DIR__));
// Create the reflection first (without loading)
$classInfo = (new ClassReflector(
new SingleFileSourceLocator(
__DIR__ . '/MyClass.php',
(new BetterReflection())->astLocator()
)
))->reflect('MyClass');
$loader->addClass($classInfo);
123. @asgrim
Monkey patching example
// Override the body...!
$classInfo->getMethod('foo')->setBodyFromClosure(
function () {
return 4;
}
);
$c = new MyClass();
echo $c->foo() . "n";
124. @asgrim
Monkey patching example
// Override the body...!
$classInfo->getMethod('foo')->setBodyFromClosure(
function () {
return 4;
}
);
$c = new MyClass();
echo $c->foo() . "n"; // returns 4, not 5
125. @asgrim
To summarise
â For PHP engine:
â AST is an efficient data structure to represent code
â AST means faster compilation (ignoring opcache)
â Separation in PHP engine for parser and compiler
â https://wiki.php.net/rfc/abstract_syntax_tree
â Concepts can be used in userland
â PHP Parser library - https://github.com/nikic/php-parser
â Better Reflection - https://github.com/Roave/BetterReflection
â Reflect on not-yet-loaded files
â Monkey patching in userland code (!)
â Static analysis opportunities
â Better Reflection
â phpstan
â Exakat static analysis (uses own AST)
â Phan (uses php-ast extension)