Università Cattolica del Sacro Cuore

Promoting methods for evidence use in scientific assessments - issue#2

A recent report sheds new light on how the European Food Safety Authority develops its scientific outputs

By European Food Safety Authority

The European Food Safety Authority (EFSA) has recently launched a number of activities to further develop the excellence of its scientific assessments and enhance their transparency and openness. A growing number of scientific and governmental institutions have started to implement assessment processes characterised by e.g. upfront development of protocols; clear rules to collect, validate, analyse and integrate evidence; and rigorous methods for documenting processes and results.

In this context, EFSA started the PROMETHEUS project (PROmoting METHods for Evidence Use in Scientific) (in 2014 with its completion scheduled for 2016), PROMETHEUS aims at further improving EFSA’s methods for “dealing with data and evidence” (i.e. for collecting/extracting, validating/appraising, analysing and integrating data and evidence) and increasing their consistency. The project also highlights and reinforces the need for transparency and openness and represents an important component for bringing forward EFSA’s transparency initiative.

With this approach we are improving methodological rigour and enhancing consistency across EFSA of the steps we take to select evidence and to show how we decide which evidence is used, or not used – and why. This will make it easier to follow how evidence contributes to the final assessment and how we report the entire process and results,” stated Dr Marta Hugas, Acting Head of Risk Assessment and Scientific Assistance at EFSA. “Ultimately, this will improve the quality of our scientific work and communication of the outcomes to decision-makers, other scientists and EFSA’s stakeholders,” added Dr Hugas.

The project envisages two scientific reports. The first, already published on 3 June 2015, is entitled "Principles and process for dealing with data and evidence in scientific assessments", and illustrates the principles and process for evidence use. The second one is scheduled in late 2016 and will contain an analysis of the methods for dealing with evidence applied by the Authority and an impact analysis of the implementation of the PROMETHEUS process in at EFSA across all the scientific areas in which it works.

The principles and process for evidence use defined in this first scientific report apply to assessments performed by generating data ex novo and by using existing data, and to all types of scientific assessments, regardless of their objective (e.g. efficacy, safety), scope (i.e. either full risk assessments or parts of risk assessment) and authors (e.g. EFSA in the case of non-applications-related assessments or, in the case of regulated products, applicants preparing dossiers and Member States and EFSA assessing them).

The principles also apply when scientific information for performing an assessment is obtained via expert knowledge elicitation.

The principles for dealing with data and evidence are those of:

  • Impartiality;
  • Excellence in scientific assessments;
  • Transparency and openness; and
  • Responsiveness.

There is a clear interdependence and even a conflict between the principles (e.g. between excellence in scientific assessments and responsiveness). Finding a compromise between these principles remains a responsibility of the assessors, in dialogue with the requester, and depends on the objectives of the assessment, the available resources and the timeframe. The principles are defined along with a series of considerations that promote them. For instance, an important element for fostering impartiality of the assessment is the upfront definition of the strategy for the assessment. Modifications to the planned strategy are acceptable as long as they are justified and documented. As for excellence in scientific assessments, the focus is on the methodological quality of the process for dealing with evidence, which influences the reliability of the results. Reliability refers to precision and accuracy. Transparency and openness are promoted by the thorough and systematic documentation of all steps of the assessment; accessibility to the data, results and all supplementary information assessed as appropriate and possible, without violating confidentiality issues; and by sharing the assessment strategy with relevant parties (e.g. requester, scientific community, general public) to seek feedback and input on the methodologies and evidence needs for performing the assessment before hand. Two fundamental aspects for fostering responsiveness are well-formulated requests and answers that are “fit for purpose”. The latter concerns the extent to which an assessment meets the expectations of the requesters and of those who will use it.

In light of the above principles, the process for dealing with data and evidence in a scientific assessment is defined in four fundamental phases:

  1. Planning a strategy for the assessment upfront, before starting the assessment (the strategy can be modified if justified);
  2. Conducting the assessment in line with the strategy;
  3. Verifying the process;
  4. Documenting and reporting the process, modifications to the strategy, results and conclusions, and ensuring accessibility of methods and data.

Expert judgment is needed for making all decisions in these steps. Indeed even if the process is structured in a way to minimise subjectivity and maximise transparency, the value of expert judgment in all phases of the process remains fundamental.

EFSA’s Executive Director Bernhard Url welcomed the approach as an important part of the Authority’s drive to be a more open EFSA: “By clearly stating these principles and defining this process, EFSA is further enhancing the quality and the transparency of its scientific assessments.”

Thorough upfront planning followed by coherent implementation will “make it easier to follow our decision-making and increase trust in the Agency’s scientific advice,” he said.

While primarily aimed at EFSA’s expert Panels and scientific staff, the principles and the process mapped out in this report should be applied by scientific organisations carrying out work on EFSA’s behalf. This framework could also be applied by risk assessors across Europe and beyond.

An easy-to-read infographic shows simply how elements in the ‘Promoting methods for evidence use’ approach works fit together in practice:

Download printable version   (0.7 Mb)