02. chapter 3 lexical analysis

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

The role of lexical analyzer Lexical Analyzer Parser Source program token getNextToken Symbol table To semantic analysis

Why to separate Lexical analysis and parsing ,[object Object],[object Object],[object Object]

Tokens, Patterns and Lexemes ,[object Object],[object Object],[object Object]

Example Token Informal description Sample lexemes if else comparison id number literal Characters i, f Characters e, l, s, e < or > or <= or >= or == or != Letter followed by letter and digits Any numeric constant Anything but “ sorrounded by “ if else <=, != pi, score, D2 3.14159, 0, 6.02e23 “ core dumped” printf(“total = %d”, score);

Attributes for tokens ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Lexical errors ,[object Object],[object Object],[object Object],[object Object],[object Object]

Error recovery ,[object Object],[object Object],[object Object],[object Object],[object Object]

Input buffering ,[object Object],[object Object],[object Object],[object Object],E = M * C * * 2 eof

Sentinels ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],E = M eof * C * * 2 eof eof

Specification of tokens ,[object Object],[object Object],[object Object],[object Object],[object Object]

Regular expressions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Regular definitions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extensions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recognition of tokens ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recognition of tokens (cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Transition diagrams ,[object Object]

Transition diagrams (cont.) ,[object Object]

Architecture of a transition-diagram-based lexical analyzer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Lexical Analyzer Generator - Lex Lexical Compiler Lex Source program lex.l lex.yy.c C compiler lex.yy.c a.out a.out Input stream Sequence of tokens

Structure of Lex programs declarations %% translation rules %% auxiliary functions Pattern {Action}

Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Int installID() {/* funtion to install the lexeme, whose first character is pointed to by yytext, and whose length is yyleng, into the symbol table and return a pointer thereto */ } Int installNum() { /* similar to installID, but puts numerical constants into a separate table */ }

Finite Automata ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Finite Automata ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Finite Automata State Graphs ,[object Object],[object Object],[object Object],[object Object],a

A Simple Example ,[object Object],[object Object],1

Another Simple Example ,[object Object],[object Object],[object Object],0 1

And Another Example ,[object Object],[object Object],0 1 0 1 0 1

And Another Example ,[object Object],[object Object],[object Object],1 1

Epsilon Moves ,[object Object],[object Object],A B 

Deterministic and Nondeterministic Automata ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Execution of Finite Automata ,[object Object],[object Object],[object Object],[object Object],[object Object]

Acceptance of NFAs ,[object Object],[object Object],1 0 1 ,[object Object],0 1 1 0

NFA vs. DFA (1) ,[object Object],[object Object],[object Object]

NFA vs. DFA (2) ,[object Object],NFA DFA ,[object Object],0 1 0 0 0 1 0 1 0 1

Regular Expressions to Finite Automata ,[object Object],Regular expressions NFA DFA Lexical Specification Table-driven Implementation of DFA

Regular Expressions to NFA (1) ,[object Object],[object Object],[object Object],[object Object],A  a

Regular Expressions to NFA (2) ,[object Object],[object Object],A B  A B    

Regular Expressions to NFA (3) ,[object Object],A   

Example of RegExp -> NFA conversion ,[object Object],[object Object],[object Object], 1 C E 0 D F   B   G    A H 1 I J

Next Regular expressions NFA DFA Lexical Specification Table-driven Implementation of DFA

NFA to DFA. The Trick ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

NFA -> DFA Example 1 0 1         A B C D E F G H I J ABCDHI FGABCDHI EJGABCDHI 0 1 0 1 0 1

NFA to DFA. Remark ,[object Object],[object Object],[object Object],[object Object],[object Object]

Implementation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Table Implementation of a DFA S T U 0 1 0 1 0 1 0 1 S T U T T U U T U

Implementation (Cont.) ,[object Object],[object Object],[object Object]

02. chapter 3 lexical analysis

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie 02. chapter 3 lexical analysis

Ähnlich wie 02. chapter 3 lexical analysis (20)

02. chapter 3 lexical analysis