By Dr Nick Pulman, Chief Clinical Information Officer, Elesh Mistry, Population Health Management Analytics Lead, and Graham Smith, Population Health Management Data Visualisation Lead at NHS Arden & GEM CSU.
Primary care is under constant pressure to innovate. In moving from reactive treatment to proactive prevention, commissioners and providers need to be able to quickly identify and scale interventions that work.
ICB and PCN-level interventions rarely have the time or budget for traditional evaluation methods. While some outcomes can be measured – for example, a reduction in A&E attendances or better wellbeing scores – without a control group, we cannot confidently assess the true impact of new ways of working. But we also don't have the capacity or resources to waste time on ineffective interventions. This has been one of the most frustrating challenges for clinicians and commissioners alike – and a priority for analytics teams to resolve.
Addressing the evaluation challenge
NHS Arden & GEM CSU has developed a new 'Rapid Evaluation' module within our Athena data and analytics platform which enables clinical and operational managers to quickly evaluate the impact of an intervention delivered to patients. Crucially, the tool analyses results against a 'control group', providing a more robust assessment of the intervention when compared with similar patients.
Athena Rapid Evaluation combines two statistical methods:
- 'Propensity score matching’ which creates a fair comparison group of people who are very similar to those receiving the intervention. Users are prompted to specify why certain patients were chosen for treatment. For example, age may be less important to the selection than past history of a certain condition. These same variables enable the tool to closely match a control group of patients for accurate comparison.
- The ‘differences-in-differences’ technique assesses how healthcare use has changed, for both the patient cohort and the comparator group, before and after the intervention. This comparison enables a more precise assessment of the actual impact of the intervention rather than coincidental changes that may skew the results. Evaluation outputs include changes in A&E attendances, emergency admissions and primary care appointments, and the cost implications of these changes.
The tool has been developed as a self-use tool with input from GPs, nurses and social prescribers to ensure it meets the needs of health professionals, is easy to use and the data is clinically validated. All analysis uses pseudonymised data enabling comparisons to be made with comparator groups with appropriate data sharing agreements in place.
There is no doubt this is an exciting step forward in allowing us to quickly and cost effectively assess interventions, build on successes and rapidly tweak or terminate interventions that are not delivering as expected. However, it is not a panacea and some challenges remain. Evaluation is currently reliant on NHS data, for example, and doesn't provide consistent information about wider determinants of health. This will become more holistic as we look to link data sets from other sectors, including social care and the voluntary sector, but improvements in data quality will be required first.
It's also important to remember this rapid evaluation approach looks only at quantitative health outcome measures. There is still an important role for traditional evaluation methods, particularly in providing qualitative insights and for large scale or complex projects. But when faced with the need to prove impact quickly, this new approach will enable those working in and with primary care to demonstrate initial outcomes within a short period, even if the true significance of an intervention may take years to fully realise.
Enabling strategic primary care
The 10 Year Health Plan's three shifts – from sickness to prevention, analogue to digital and hospital to community – are ambitious and present both challenges and opportunities for primary care. We need to be able to innovate, test, evaluate, learn and improve continuously to respond to the changing demands of our patient populations within the resources available.
Primary care has already become adept at using risk stratification to identify priority patient cohorts and develop targeted support to improve outcomes and reduce cost. Although population health approaches can be traced back to 1815 in England, it is only now that we're able to make a good attempt at control comparison. The measures we can evaluate now are relatively simple, but it's a positive step in the right direction that can be built on over time.
Combining risk stratification and rapid evaluation gives primary care organisations the tools to adopt a more strategic approach to commissioning and care delivery. Maximising the benefits will require a shift in mindset, and the discipline to act on findings, particularly where interventions are not delivering the expected benefits. But for PCNs, ICBs and new neighbourhood pilots, embedding use of these tools and techniques will accelerate learning and improvement as we all look to deliver effective, efficient improvements in patient care.
This article was originally published in Healthcare Leader.