1. The document summarizes the agenda and discussion from a Quality Standards Working Group meeting. They discussed progress on two papers about developing a tool to assess the quality of asthma comparative effectiveness research. 2. They provided an update on a study to identify high-quality asthma comparative effectiveness evidence for guidelines. They have drafted two papers on developing their quality assessment tool and applying it to literature. 3. The group discussed priorities for future research, publications, and implementation of quality standards. They considered whether proposed studies were still relevant and priorities. They also discussed the next REG summit.

1. ERS 2017
Milan, 9th September 2017
Quality Standards Working Group Meeting

2. Agenda
• Working group progress update
o Discussion of the two main papers
• Priorities for future research, actions, publications, promotion and
implementation
o Review potential studies
– Are they still relevant, feasible, valid and a priority
o Other research ideas
o Setting priorities
• Additional items

3. Attendees
• N Roche (chair)

4. Active study update
• REG_P005:
o Phase I: Assess the quality of the asthma comparative effectiveness
literature to identify evidence for consideration by future guideline
developers
o Phase II: Develop (implementable) tools to facilitate quality appraisal of
published real-world evidence

5. Assess the quality of the asthma comparative effectiveness
literature to identify evidence for consideration by future
guideline developers
Paper Focus Journal Lead
author
Status
1 Overview of the taskforce
work
Allergy Nicolas First draft due to be submitted to
EAACI and REG
2 Development of the tool
and link to (and
differentiation from) other
work in this area
ERJ? Jon First draft almost complete
3 Results of literature review Clinical and
experimental
allergy?
Nicolas On hold – determine whether
required following publication of
paper 1.

6. REG_P005: Paper 1
Objective
Provide an overview of how this Task Force developed a quality assessment tool and applied it to some selected
PICOT questions for which RCTs provide only limited evidence.
Working title
Quality standards in respiratory real-life effectiveness research: a Respiratory Effectiveness Group –
European Academy of Allergy and Clinical Immunology Task Force report
Rationale
• In observational research as well as in RCTs, ensuring high-quality methodology is crucial to avoid biases that
would compromise the reliability of results.
• While quality standards for RCTs are well-defined and extensively disseminated, e.g., by the CONSORT
statements, this is less so in the field of observational comparative effectiveness research.
• Several tools assess the quality of evidence provided by observational research, they have not been specifically
designed for effectiveness research evaluation

7. Methodology
• Development and testing of the quality assessment tool
• Systematic review of tools proposed in the literature.
• Results discussed during Taskforce meetings and used to build a first version of the REG/EAACI tool.
• Tested by the taskforce
• Refined and formatted
• Tested again on a larger basis using all papers retrieved by the literature search
• Selection of topics
• A first set of 21 PICOT questions was identified during a Taskforce meeting, selected based on their relevance for asthma
management, limited ability of available RCTs to provide robust answers and potential of observational research to
address them.
• Nine were prioritised, and then ranked and feasibility discussed.
• Four final questions were selected.
• Assessment of the literature
• Identify relevant articles corresponding to the PICOT questions
• Papers considered of sufficient quality were sent to at least two reviewers
• Summary tables were built, mentioning the main conclusion of the articles, the type of study, the GRADE assessment,
the possible impact on clinical practice and whether similar evidence is available from RCTs.
REG_P005: Paper 1

8. Results
REG_P005: Paper 1
PRIMARY CRITERIA
1. Background 1.1. Clearly stated research question
2. Design
2.1 Population defined
2.2. Comparison groups defined and justified
3. Measures
3.1. (If relevant), exposure (e.g. treatment) is
clearly defined
3.2. Primary outcomes defined
4. Analysis
4.1. Potential confounders are addressed
4.2. Study groups are compared at baseline
5. Results
5.1. Results are clearly presented for all
primary and secondary endpoints as well as
confounders
6. Discussion/Interpretation
6.1. Results consistent with known information
or if not, an explanation is provided
6.2 The clinical relevance of the results is
discussed
7. Conflict of interests
7.1. Potential conflicts of interest, including
study funding, are stated
SECONDARY CRITERIA
1. Background
1.1. The research is based on a review of the background literature
(ideal standard is a systematic review)
2. Design
2.1. Evidence of a priori design, e.g. protocol registration in a dedicated
website
2.2 Population justified
2.3 The datasource (or database), as described, contains adequate
exposures (if relevant) and outcome variables to answer the research
question
2.4 Setting justified
3. Measures 3.1 Sample size / Power pre-specified
4. Analysis NO SECONDARY CRITERIA
5. Results
5.1. Flow chart explaining all exclusions and individuals screened or
selected at each stage of defining the final sample
5.2. The authors describe the statistical uncertainty of their findings
(e.g. p-values, confidence intervals)
5.3. The extent of missing data is reported
6. Discussion/Interpretation 6.1. Possible biases and/or confounding factors described
7. Conflict of interests NO SECONDARY CRITERIA
*For guideline development, secondary criteria only appear if all
primary criteria are met

9. Results
REG_P005: Paper 1
• PICOT questions dealt with:
• The influence of adherence/persistence on asthma outcomes
• The influence of smoking on asthma outcomes and treatment effectiveness
• The impact of the device on outcomes
• The relation between particle size of maintenance therapy and asthma outcomes
• 46 papers were selected
• The main limitation of currently available database studies was the lack of detailed clinical data regarding:
• control as assessed by validated questionnaires,
• severity as assessed by symptoms, and lung function,
• symptoms suggestive of comorbidities such as rhinitis or gastro-oesophageal reflux disease.
• As a consequence, authors used proxies.
• In most retained studies, these factors were used to match and/or adjust analyses, and so the level of evidence could be
raised to moderate.
• The level could not be raised to high, since residual confounding by hidden factors could not be ruled out. For instance,
data on behaviours are frequently lacking, while smoking history or BMI are absent from several databases.

10. Conclusions
REG_P005: Paper 1
• The final tool was considered easy to use by members of the REG and EAACI networks who were
involved in the testing phase (n=66)
• Fair levels of agreement were obtained in the implementation phase
• ≥73% for primary items and ≥69% for secondary ones.
• The high number of PICOT questions (n=21) identified, outlines the evidence gaps that are still
present regarding asthma treatment.
• This contrasts with the high number of RCTs assessing asthma care, illustrating that classical RCTs
are not sufficient to answer all questions regarding real-life effectiveness of therapeutic strategies.
• The number of published articles available for each of the PICOT questions was surprisingly low,
illustrates that observational effectiveness research in asthma is still infrequently performed.
• 46% of retrieved papers were considered of sufficient quality to be analysed further in terms of their
results and how these could inform current knowledge and recommendations.

11. REG_P005: Paper 2
Objective
To outline the methodology and process that lead to the development of the Real Life Evidence AssessmeNt Tool
(RELEVANT). The tool was created to assist the Task Force in discriminating flawed CER studies from those of
sufficiently high quality to warrant consideration by asthma guideline bodies
Working title
The Real Life Evidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset

12. Methodology
• Development and testing of the quality assessment tool was across 6 key phases
• Phase I: Review of the literature
• Phase II: Initial tool creation
• Phase III: Taskforce review and feedback
• Phase IV: Pilot application of the quality assessment tool
• Phase V: Extended pilot
• Phase VI: Development of an online tool
REG_P005: Paper 2

13. Results
REG_P005: Paper 2
• Initial tool creation (Phase I-III)
• A 43-item checklist was created from primary reference papers, and was reduced to 25 quality
criteria by the task force members’ discussions.
• Tool appraisal and item reduction (Phase IV & V)
• Initial pilot:
• Evaluation of inter-rater agreement of the initial tool indicated wide variation (33-100%) in concordance
across individual criteria
• The qualitative feedback suggested this variation was largely driven by differences in semantic
interpretation of some of the criteria
• This was addressed by rephrasing, removal of any redundancies and eliminating subjective words
• Extended pilot:
• The inter-rater agreement of the revised tool found concordance in excess of 70% for 94% of primary
criteria and 93% of secondary criteria
• Qualitative feedback from the pilot raters identified some remaining criteria in the tool that could be
further simplified to reduce potential inter-rater variability.

14. Results
REG_P005: Paper 2
• RELEVANT (Phase VI)
• Guides systematic appraisal of the quality of published observational CER papers across seven
domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, Conflicts
of Interest
• Raters must indicate fulfillment (Yes / No / NA) of 11 quality criteria across these seven domains
• RELEVANT quality” is defined as fulfillment of all 11 primary criteria.
• Failure to meet any one primary criterion reflects a potential “fatal flaw” in a study’s design or, if
failure reflects absence of the necessary detail, a lack of necessary transparency in reporting
• If all primary criteria are fulfilled, assessment of ten additional, secondary, parameters is prompted
to enable further characterization of the relative strengths and weaknesses of the paper

15. Conclusions
REG_P005: Paper 2
• RELEVANT is a user-friendly quality appraisal tool comprising 21 quality criteria (11 primary; 10
secondary) across seven core quality domains.
• It is the first of its kind to support quality appraisal of published research and to have been developed
through iterative feedback derived from pilot implementation and inter-rater agreement evaluation.

16. Next steps
• Publication of papers
• The quality standards in respiratory real-life effectiveness
research tool:
o Finalise tool
o Make it available – where?
o Promote it

17. Other research ideas?

18. Setting priorities
• Are these projects:
o Relevant?
o Feasible?
o Valid?
o A priority?
• How do we set priorities in quality standards research?
• How to we ensure these priorities are pursued?
• What are the two most important projects to push forwards?

19. REG Summit 2018: “Working together to make an
impact”
• Who do we partner with for greatest impact
• How do we increase acceptability/validity of real-life research

20. Any other business?