The newDemocracy Foundation’s efforts ‘to improve public trust in how we take public decisions’ include a research project which sought to discover whether a meaningful, widely-accepted standard for evidence-based policy making might be achievable. They did this by asking two think-tanks known to have very different views to ‘stress test’ 20 major policies and see if they could find common ground on adherence to a ‘standard’.
The Evidence Based Policy Research Project commissioned the Institute of Public Affairs, and Per Capita Australia, to analyse 20 public policies using the ten criteria developed by Prof Kenneth Wiltshire AO as a test for good policymaking. These were featured in the Institute of Public Administration Australia’s (IPAA) 2012 paper Public Policy Drift – also well worth a read.
The project was commissioned ‘to coax more evidence-based policy decisions by all tiers of Government by reviewing and rating 20 high profile government decisions against the Wiltshire business case criteria’. These policies were assessed purely for good process, rather than either the outcome or intention of the policies.
The IPA, using at least six criteria out of 10 as their pass rate, found that just eight of the 20 policies passed. “The other twelve policies failed the test. No policy was found to have met all ten criteria. Three policies, criminal justice reforms in New South Wales, the legalisation of Uber in Queensland and voluntary assisted dying in Victoria, were found to have met nine criteria.”
They also noted that “the most common failures to apply the best practice policy making process related to cost-benefit analysis – a basic tenant of good policymaking but rare in Australia – followed by a failure to produce a green and then a white paper or outline a policy design framework. This was followed by a failure to undertake further consultation, outline alternatives, or a legislate with parliamentary debate“.
Per Capita found that “of the 20 policies we analysed, 11 were found to have met the Wiltshire criteria, while 9 failed. This shows that although there is high-quality policymaking in Australia, especially at the state level, policymaking still often falls short of the best practice the public should expect. Notably, all of the Victorian state government policies we assessed met the Wiltshire criteria.”
Per Capita noted that the project sought to make the assessment of policies against the Wiltshire criteria more objective, by the provision of a set of guiding questions, where a ‘Yes’ answer would indicate the policy had met the corresponding criterion, and a ‘No’ answer would mean it had not. In their view, this lowered the threshold for a policy to meet the criteria, and so their ratings were likely to have been more generous than they would have been without those questions.
As for whether the Wiltshire criteria provide a basis for a public policy business case standard, it would seem they do offer a good starting point, as there was broad agreement between the two groups of analysts. There were some notable discrepancies, however, that warrant closer scrutiny, and this could result in refinement of the current criteria. This comparative table illustrates both the broad agreement and areas of disparity between scores in the two sets of failed policy lists:
There are only three policies on which complete agreement was achieved, six where the difference was plus or minus one, and three where the discrepancy was plus or minus two. Of this last group, two were ‘passed’ by one organisation and ‘failed’ by the other.
Policy analysts and advocates across the spectrum will welcome the publication of these analyses, and applaud the further promotion of evidence-based policy. Hopefully, they will also offer constructive suggestions for further refinement of the public policy business case criteria drafted by Prof Wiltshire, and help to establish an agreed objective ‘standard’.
