The accuracy and accessibility of cited evidence: a study examining mental health policy documents

Abstract: Purpose: Evidence-based policy making is increasingly being advocated by governments and scholars. To show that policies are informed by evidence, policy-related documents that cite external sources should ideally provide direct access to, and accurately represent, the referenced source and the evidence it provides. Our aim was to find a way to systematically assess the prevalence of referencing accuracy and accessibility issues in referenced statements selected from a sample of mental health-related policy documents. Method: 236 referenced statements were selected from 10 mental health-related policy documents published between 2013 and 2018. Policy documents were chosen as the focus of this investigation because of their relative accessibility and impact on clinical practice. Statements were rated against their referenced sources in terms of the (i) content accuracy in relation to the information provided by the referenced source and (ii) degree of accessibility of the source and the required evidence from the references provided. Results: Of the 236 statements, 141 (59.7%) accurately represented the referenced source, 45 (19.1%) contained major errors and 50 (21.2%) contained minor errors in accuracy. For accessibility, 126 (53.4%) directly referenced primary sources of evidence that supported the claims made, 36 (15.3%) contained indirect references, 18 (7.6%) provided 'dead-end' references, and 11 (4.7%) references were completely inaccessible. Conclusions: With only slightly over half of all statements assessed providing fully accessible references and accurately representing the referenced source, these components of referencing quality deserve further attention if evidence-informed policy goals are to be achieved. The rating framework used in the current study proved to be a simple and straightforward method to assess these components and can provide a baseline against which interventions can be designed to improve referencing quality.