GraphQL is a flexible and expressive query language. With the objective to replace the flawed and inefficient REST architectural style, GraphQL has been adopted by numerous online APIs and services. Despite its popularity, testing theimplementation of a GraphQL schema is a crucial and still an open problem. We found that classical techniques of test generation may be efficiently applied to GraphQL server. We propose a simple but expressive technique called deviation testing that automatically searches for anomalies in the way a schema is served. We demonstrate the feasibility of our approach using an implementation of GraphQL for Pharo and VisualWorks. Running our technique on the popular Yelp and Apollo GraphQL server uncovered several anomalies in the way the schema is served.

1. Deviation Testing
A Test Case Generation
Technique for GraphQL APIs
Daniela Meneses Vargas
Alison Fernández Blanco
Andreina Cota Vidaurre
Juan Pablo Sandoval Alcocer
Milton Mamani Torres
Alexandre Bergel
Stéphane Ducasse

2. What is
GraphQL?

3. REST (Representational State Transfer)
● REST has become the standard for designing web APIS
● Offers some great ideas, such as stateless server and structured access to
resources
● It has shown to be too inflexible to keep up with the rapidly changing
requirements of the clients that access them

4. REST - Overfetching
{
"film": {
"id" : 1,
"name" : "Schindler's List",
"imageUrl" : "https://someimageurl.com/uxs.png",
"rating" : 5,
"gender" : "ACTION",
"director" : { "name" : "James Cameron" }
}
}
HTTP GET: /film/<id>
Schindler's
List

5. REST - Underfetching
HTTP GET:
/director/<id>
/director/<id>/films
/director/<id>/roleDirector:
Steven Spielberg
Films:
Jurassic Park, War
House, Indiana
Jones
Role:
Film Director

6. GraphQL
Query language and the execution engine of queries by using a type system you
define for your data.
● GraphQL is not dealing with dedicated resources
● In GraphQL the server declares what resources are available and the client
asks for what it needs

7. GraphQL - Schema
type Query {
director(id: Int): Director
allFilms: [Film]
film(id: Int): Film
}
type Director {
name: String!
id: Int!
films: [Film]
role: String
imgUrl: String
}
type Film {
name: String!
director: Director
rating: Int
gender: String
imgUrl: String
}
Schema Query
All the types that are exposed
in an API are written down in a
schema using the SDL.
Schema server as a contract
between the client and the
server
{
director(id: 1){
name
films {
name
}
role
imgUrl
}
}

8. GraphQL
HTTP POSTDirector:
Steven Spielberg
Films:
Jurassic Park, War
House, Indiana
Jones
Role:
Film Director
{
director(id: 1){
name
films {
name
}
role
imgUrl
}
}
{
"data": {
"director": {
"name": "Steven Spielberg",
"films": [ { "name": "Jurassic Park", ...} ],
"role": "Film Director",
"imgUrl": "https://asd.com/qwerte.png"
}
}
}

9. SGraphQL - GraphQL in Smalltalk
https://github.com/OBJECTSEMANTICS/GraphQL

10. GraphQL
AutoTest

11. Deviation Testing
Technique that measures the difference between a test case and its automatic
generated variations (deviations).
The goal is increase the test coverage and help to find potential bugs in any
GraphQL implementation.

12. Deviation Testing Workflow

13. Deviation Rules

14. Fields Selection Deviation
{
allFilms {
name
rating
gender
director {
name
}
}
}
{
allFilms {
name
rating
gender
}
}

15. Fields Deviation Result Comparison
{
allFilms {
name
rating
gender
director {
name
}
}
}
{
"allFilms" : [
{
"name" : "terminator",
"rating" : 5,
"gender" : "ACTION",
"director" : {
"name" : "James Cameron"
}
},
{
"name" : "Star wars: New Hope",
"rating" : 9,
"gender" : "SCIENCE_FICTION",
"director" : {
"name" : "George Lucas"
}
}
]
}

16. Fields Deviation Result Comparison
{
allFilms {
name
rating
gender
}
}
{
"allFilms" : [
{
"name" : "terminator",
"rating" : 5,
"gender" : "ACTION"
},
{
"name" : "Star wars: New Hope",
"rating" : 9,
"gender" : "SCIENCE_FICTION"
}
]
}

17. {
"allFilms" : [
{
"name" : "terminator",
"rating" : 5,
"gender" : "ACTION",
"director" : {
"name" : "James Cameron"
}
},
{
"name" : "Star wars: New Hope",
"rating" : 9,
"gender" : "SCIENCE_FICTION",
"director" : {
"name" : "George Lucas"
}
}
]
}
Fields Deviation Result Comparison
{
"allFilms" : [
{
"name" : "terminator",
"rating" : 5,
"gender" : "ACTION"
},
{
"name" : "Star wars: New Hope",
"rating" : 9,
"gender" : "SCIENCE_FICTION"
}
]
}

18. Empty Field Deviation
{
film(id: 1) {
name
rating
gender
director {
name
}
}
}
{
film(id: 1) {
}
}

19. Empty Field Deviation
{
film(id: 1) {
}
}
{
"errors": [
{
"message": "Syntax Error: Expected Name, found }",
"locations": [
{
"line": 4,
"column": 3
}
]
}
]
}

20. Non Null Deviation
{
film(id: 1) {
name
rating
gender
director {
name
}
}
}
{
film(id: null) {
name
rating
gender
director {
name
}
}
}

21. Non Null Deviation
{
film(id: null) {
name
rating
gender
director {
name
}
}
}
{
"errors": [
{
"message": "must provide an id",
"locations": [
{
"line": 2,
"column": 12
}
],
"path": [
"film"
]
}
],
"data": {
"film": null
}
}

22. Type Deviation
{
film(id: “bad id”) {
name
rating
gender
director {
name
}
}
}
{
"errors": [
{
"message": "No valid ID extracted from 1",
"locations": [
{
"line": 2,
"column": 12
}
],
"path": [
"film"
]
}
],
"data": {
"film": null
}
}

23. Type Deviation
{
film(id: 1) {
name
rating
gender
director {
name
}
}
}
{
film(id: “bad id”) {
name
rating
gender
director {
name
}
}
}

24. Running the test generator
1
2
3
4

25. Case Studies
Application Passed Failed Total Deviations Failed
Smalltalk 38 10 48 Argument Types
Empty Field Selections
Yelp 90 5 95 Empty Field Selections
Apollo 134 0 134 None

26. Future Work
● Add more deviation rules
● Support for multiple initial queries

27. Conclusions
● We created a tool to test a GraphQL standard compliance by creating
deviated queries from the original and comparing the responses.
● We run the tool against SGraphQL Demo API, Yelp and Apollo Demo API;
finding on two of them problems in the implementation of GraphQL standard.

28. Thanks!
Semantics
www.semantics.bo
semantics.bolivia@gmail.com
Daniela Meneses
daniela11290@gmail.com
Andreina Cota
andycotvy@gmail.com

29. Implementation - Iterators

30. Implementation - Mutators