The document discusses a new approach to constrained text generation aimed at assessing reading performance by creating standardized sentences using various methods including constraint programming and machine learning models. It outlines the challenges in generating a sufficient number of sentences that adhere to specific linguistic constraints, as well as the development of multi-valued decision diagrams (MDDs) to efficiently represent and manipulate these constraints. The findings illustrate the efficacy of using LLMs (like GPT-2) for ranking generated sentences and highlight potential applications for similar tasks in different languages.

1. Constrained text generation to measure reading performance:
A new approach based on constraint programming
JOINT WORK WITH ALEXANDRE BONLARRON(1,3), AURÉLIE CALABRÈSE(2), PIERRE KORNPROBST(1).
1
Jean-Charles Régin(3)
(1) Université Côte d’Azur, Inria, France
(2) Aix Marseille Université, CNRS, LPC, Marseille, France
(3) Université Côte d’Azur, I3S, France

2. • Standardized text: sentences read at the same speed
• Usability: to assess reading performance
Standardized Text
(Mansfield et al., 1993)
3

3. Constrained text generation
This is a rules (constraints) dominated problem
MNREAD Chart
=
4

4. MNREAD Rules
Display Rules
Lexical Rules
Grammatical Rules
Length Rules
E.g. Entering the rectangle
E.g. 3000 words from CE2 textbooks
E.g. No Punctuation
E.g. 60 characters, between 9 and 15 words
Example of a MNREAD sentence
There are 38 MNREAD sentences in French
5

5. Are there enough sentences?
No, a few thousand sentences are needed to detect and
monitor visual pathology throughout life.
Is it really difficult to have more sentences that
respect the rules ?
Questions :
There are 38 MNREAD phrases in French.
6

6. Naive method
Search for MNREAD type sentences in a corpus.
Problem : This method does not scale up
Solution : We have to generate them, but how ?
Display
Lexicon
Grammar
Length
1 000 000 sentences
10 000 sentences
6 sentences
2300 books 10 000 000 sentences 3 sentences
7

7. How to generate standardized sentences?
8
LLM-based approach (GPT, BERT) + SEARCH := good text quality, but unlikely to find in an
instance that satisfies the constraints.
Generates word-by-word sentences and selects the next word as the most suitable (use token
instead of words)
prompt (ChatGPT 3.5): give me a sentence of sixty characters with spaces included
“Elephants march majestically through the savannah at sunset, their presence captivating”
prompt (ChatGPT 3.5): give me a sentence of sixty characters
“The cat sat on the mat and purred softly”

8. How to generate standardized sentences?
• One "good" sentence out of 8000 (only); remembering bias
• A semi-automatic method for the English language
• Non-trivial extension to Latin languages!
mon ami est beau; mon amie est belle
Ad hoc method: a recent method proposed by the creators of MNREAD and based on hand-
defined models (Mansfield et al., 2019).
9

9. How to generate standardized sentences?
n-gram based methods (Papadopoulos et al., 2015)
Corpus n-gram Génération
When using with a random walk it produces sentences in the style of an author
Problems:
How to integrate constraints?
How to manage the meaning of the sentences?

10. • A generalization of Binary Decision Diagrams (BDD)
• Each layer represents a variable
• Each path between root and tt is a valid assignment of the variables
• An MDD models all tuples that satisfy a constraint
Multi-Valued Decision Diagram (MDD)
11
MDD having 3 solutions :
(a,b) (a,a) (b,b)
• Data structures for calculating and storing problem solutions in a
compressed form using an acyclic directed graph
• Advantage: Powerful modeling tools. With one billion of arcs we can
represent 10^90 solutions!

11. MDD and compression
● Sum of 3 variables
● Corresponds to an automata
● The last layer is the sum value

12. Reduction
● Operation which merges equivalent
nodes
● Two nodes are equivalent if they
have
○ the same outgoing edges
(same destination + same labels)
root
a
0
b
1
c
0 1
d
2
e
0 1
0 1 1 0 1
● Minimization of finite automata
tt

13. Reduction
● Operation which merges equivalent
nodes
● Two nodes are equivalent if they
have
○ the same outgoing edges
(same destination + same labels)
root
a
0
b
1
ce
d
2 0 1 1 0
1 1 0
● Minimization of finite automata
tt

14. Reduction
● Reduction may gain an exponential factor
● Consequence:
○ MDD can be exponentially smaller than an equivalent automaton

15. Compression gain
● Compression may gain an exponential factor
● It often does!
● Example
○ MDD requiring 600,000 edges for representing 10^90 solutions, that is a
compression factor of 10^86.
● Sometimes it can be subtle

16. Alldiff constraint
● #node = 2^n
● #solutions = n!
● n!/2^n ???
● is exponential

17. MDD: creation
● MDD can be created without enumerating the solution set
● Can be created from Dynamic Programming
● Kind of Search compression
● So what?
● Operations!

18. MDD: operations
● Intersection, union, difference, negation etc…
● Operations are performed without decompression
● Intersection of 2 MDDs is equivalent to make the conjunction of the 2
constraints represented by the MDDs
● Relation between MDD operations and constraints combination
○ Intersection : conjunction
○ Union : disjunction
○ Negation : negation

19. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
1.20

20. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
1.21

21. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
1.22

22. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
1.23

23. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
1.24

24. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
1.25

25. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
1.26

26. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
1.27

27. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
1.28

28. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
tt
1
1.29

29. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
tt
1
1.30

30. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
tt
1 1
1.31

31. Intersection
root1
a
0
b
1
ce
d
2 0 1 1 0
tt1
1 1 0
root2
k
0 2
l
0 2
tt2
1 2
root
ak'
0
cel'
0
dl'
2
tt
1 1
1.32

32. Intersection
● Be careful, do not think that the number of nodes/edges of the intersection
will be reduced
● The resulting MDD can be exponentially larger! Because it can be locally
decompressed

33. Operations
● In-place
● On-the-fly (i.e avoid having to define the MDD ina dvance and proceeds by
level)

34. 3 IDEAS
35

35. First idea
store and retrieve n-grams efficiently
36

36. Successions Constraint
• Assuming all n-grams are inserted in
the MDD as solutions.
• MDD as a TRIE
• To store and reTRIEve n-grams.
What is the next word of The white cat ? 37

37. Second idea
Integrate constraints on-the-fly
38

38. - Using the first MDD (successions)
- We compile the second one
-Constraint are checked on-the-fly
The girl and the boy walked through the forest under t he majestic oak trees
39
MDD Unfolding (top-down)

39. The modeling properties of MDD leads to solve
the problem by representing each rule by an
MDD and by intersecting them.
40

40. Modeling
From Rules to MDD
9 to 15 words Language Restrictions (3000 lemma) 59 characters corpus
An arc is a word
An arc is a lemma
An arc is the number of characters of a word A state is the sum
An arc is a word A state is a k-gram
MDD Universel MDD Lexique MDD Size MDD Corpus
41

41. Intersection
From MDD to sentences
9 to 15 words Language Restrictions (3000 lemma) 59 characters corpus
MDD Taille MDD Corpus
MDD Universel MDD Lexique
# Le
Le sac
42
The intersections of MDD gives: Le sac noir

42. Third idea
Use an LLM to select best sentences
43

43. • Transformers (very large context window):
• Perplexity is derived from Shannon entropy.
• It quantify the uncertainty of a model with respect to a sample
• Lower the better, range is [1 ; + inf[
LLM sentences scoring : Perplexity
44

44. • Input : 443 books belonging to the youth category (FR)
• Input : 75 books belonging to the fiction category (EN)
• Evaluation :
• MNREAD candidate sentence set (syntax and meaning correct)
• Ineligible set of sentences (syntax and/or meaning problems)
• Software & Hardware :
• The model is implemented in Java 17 in an MDD solver (MDDLib) @I3S.
• The LLM use to rank sentences is GPT-2
• Machine: Ubuntu 18.04 using an Intel(R) Xeon(R) Gold 5222 @ 3.80GHz CPU and 256 GB RAM.
45
Experimental conditions

45. • With 1% of the corpus, i.e. 63000 sentences, in 3-grams we obtain 9899 sentences
•J'aimerais bien que le soleil commence à se rendre au salon
Phrases generated in 3-grams
Do we have sentences?
•Ils sont morts et les yeux sur le nom de ce que vous croyez
•Mes yeux se posent sur le nom qui lui a dit que vous croyez
•Ses mains n'étaient pas de sa mère dans ses bras et le même
•Aucun de ses pieds nus sur les yeux de ce qui ne se passera
•L'expression a pris un coup de poing et de leur sort demain
•Y en a pas de nous préparer à tout bout de sa petite bouche
•
•
J'en ai dit que si je vous en emparez et vous ne pouvez pas
Entrez là et tu as de ma part de sa main dans le monde voit
Bien que je ne veux pas que les yeux de ce que ça me plaira
I wish the sun would start coming into the living room
• Those sentences are not admissible.
• In 3-grams we produce a large majority of sentences with problems of meaning and syntax
• None of his bare feet on the eyes of what will happen
46

46. Are MNREAD sentences generated?
• YES ! In 5-grams , with 443 books , we generate hundreds of sentences (7028).
47

47. Are MNREAD sentences generated?
• YES ! In 5-grams , with 75 books , we generate hundreds of sentences (204).
48

48. Performances analysis
FR 443 3Go 72s 7028
EN 75 <<1Go 3s 204
MNREAD sentence generation
49
Scoring takes roughly 1 hours for 7000 sentences. GPT-2 (pylia)
Scoring takes roughly 30 mins for 7000 sentences. GPT-3 (OpenAI cloud)
Recent benchmark :
569.77 ms / 15 tokens ( 37.98 ms per token) ~~ 1 sentence, llama.cpp (comparable to
GPT-3) (pylia)

49. • Select sentences by using GPT2 or similar generative model.
• Examples (everybody can have a personal opinion about the score!):
• Very good : The two men looked at each other in a state of stupefaction (10)
• Moderatley good: The wolves had for the most part wholly ignorant of warfare (270)
• Bad: The farmer sat down on the Museum steps except the nice one (930)
• Poetic (medium): Il est tombé dans le vide avec une sorte de douceur absente (100)
• Complex:
• The aircraft will be as common as I can to hinder their way (380)
• The depth was very great and it seemed to me to do as I did (97)
Discussion

50. • Score are related to frequency of occurrence
• The wolves had for the most part wholly ignorant of warfare (272)
• We change words by using more frequent ones
• The wolves had for the most part completely ignored the war (90)
Discussion

51. English Ranking
52
The two men looked at each other in a state of stupefaction ,10.47
The wolves had for the most part wholly ignorant of warfare ,272

53. • Generate sentences having only ten words (or 12, 11…): no problem
• Changing the level of vocabulary: no problem
• Modifying the size: no problem
• Other constraints: be careful with the combinatorics. If main constraints are relaxed then number of
solutions explodes!
New Constraints

54. • Promising method: more suitable than generic methods for handling constraints (e.g., GPT, Bert) and
more flexible than the ad-hoc method of Mansfield et al [3].
• Advantages: modularity (easy to add and/or remove rules), constraints taken into account at generation,
potentially applicable to other languages
• Perspectives: a perplexity constraint
55
Conclusion

55. Thanks.
56