Computer Science

Evaluating Information Technology systems in the NHS

Summary:

A team led by a Manchester Metropolitan University researcher developed the PROBE (Project Review, Objective Evaluation) methodology for evaluating healthcare information technology systems. This became a core part of the National Health Service (NHS) information strategy; in 1996, PROBE was accepted by the NHS Executive and HM Treasury as a central method for project review. PROBE has therefore brought significant benefits to clinical users, patients and NHS decision makers. Impacts described here relate to the use of PROBE to evaluate an electronic blood transfusion system. Specifically, through enabling the effective evaluation of the system, decisions by a health authority have been informed by research, clinical practices have improved, and levels of patient safety and confidentiality have risen.

Underpinning Research

From the late 1980's onwards, healthcare providers, commercial system developers and academic researchers began to focus on the potential benefits of IT systems in health. There followed a period of intense activity, during which many hundreds of health-related systems were developed and piloted in the UK, notably decision support systems (DSS) and electronic patient record systems (EPRS). However, systems often failed to provide the anticipated benefits, and, in some cases, were actively obstructive. Common problems included clinicians shunning systems in favour of paper-based practice, or systems actually introducing new errors into clinical practice.

By the early 1990's, Dr Heathfield Heathfield had been developing DSS and EPRS for some years, firstly as part of her Ph.D. studies at the University of Brighton and later as an IBM Research Fellow. Frustrated by the lack of success of healthcare IT systems, Dr Heathfield turned her attention to evaluation methodologies. She found that existing methodologies were not suited to healthcare IT systems intended to operate (a) in complex working environments, and (b) to high standards of safety and security.

Dr Heathfield subsequently began to study the implementation of healthcare IT systems, identifying the key barriers, incentives and lessons learned for successful systems. This research led to several high profile publications [1-4]. The National Health Service Information Authority (NHSIA) subsequently commissioned Dr Heathfield and colleagues to conduct specific research in evaluation methodologies for EPRS, followed by a project to develop a practical toolkit (PROBE: Project Review and Objective Evaluation) for evaluating EPRS [5-6]. The PROBE methodology makes knowledge and best practice in evaluation widely available in an easily accessible and usable format. It supports evaluations that are systematic, rigorous and consistent across projects, and therefore enables comparison and further learning. PROBE was recommended by the NHSIA for all IT projects, particularly those involving EPR and EHRS, and therefore formed a central part of NHS information strategy and policy [5].

Dr Heathfield undertook the research leading to the development of PROBE during her time at as a Senior Lecturer at MMU (September 1994 - 2002). She led the work in collaboration with colleagues from the Yorkshire Institute for Health Informatics (YICHI) at the University of Leeds, and Secta Ltd, a healthcare consultancy company. In 1996 she was funded by Cambridge University to spend a year as a Senior Visiting Fellow, directing the evaluation of a large-scale pilot of a General Practice DSS for a large pharmaceutical company.

Between 1998 and 2005, the NHS ran several large-scale national pilot projects of healthcare IT systems. Foremost among these was the Electronic Record Development and Implementation Project (ERDIP), a multi-million pound project which piloted EPRS in 12 NHS communities, and involved many thousands of clinicians and patients. Dr Heathfield and colleagues implemented several of the evaluation work packages using the PROBE methodology. The evaluation outputs played a fundamental role in the further development of EPRS, both as part of ERDIP and more widely. The resulting outputs further underlined the importance of good evaluation in healthcare IT, and (more importantly, for the purposes of this case study) emphasised the significant and central role of PROBE as an NHS-endorsed methodology.

References:

Work underpinning, motivating and complementing the development of PROBE is available as:

  1. Heathfield, HA, Buchan, IE (1996). Current evaluations of information technology are often inadequate. British Medical Journal 313: 1008. doi: 10.1136/bmj.313.7063.1008
    (29 citations).
  2. Heathfield, HA, Peel, V, Hudson, P, Kay, S, Mackay, L, Marley, T, Nicholson, L, Roberts, R, Williams, J (1997) Evaluating large scale health information systems: from practice towards theory. Proceedings of the American Medical Informatics Annual Fall Symposium, p.p. 116-120.  PubMed link (42 citations).
  3. Heathfield, HA, Pitty, D, Hanka, R. (1998) Evaluating information technology in health care: barriers and challenges. British Medical Journal 316: 1959. doi: 10.1136/bmj.316.7149.1959 (223 citations).
  4. Heathfield, HA, Hudson, P, Kay, S, Mackay, L, Marley, T, Nicholson, L, Peel, V, Roberts, R, Williams, J (1999) Issues in the multi-disciplinary assessment of healthcare information systems. Information Technology & People 12 (3): 253-275. doi: 10.1108/09593849910278277
    (33 citations).

The NHS Executive Information Management group give an early summary of PROBE in:

  1. Gronlund, TA (1996) Success through Evaluation: PROBE. In Brender, J, et al. (Eds.), Medical Informatics Europe ’96, p.p. 354-358, IOS Press. doi: 10.3233/978-1-60750-878-6-354.

The final PROBE methodology is described in the formal release document:

  1. Heathfield, HA, Clamp, S, Felton D (2001) PROBE. Project Review and Objective Evaluation for Electronic Patient Record Projects. Prepared by the UK Institute of Health Informatics for The NHS Information Authority (March 2001). Available at
    http://www.ychi.leeds.ac.uk/ychi/files/pub/Probe_Final_Version.pdf

Details of the Impact:

Since the publication of PROBE in 2001, the NHS has included evaluation as a central element of major IT projects. Furthermore, as recommended in PROBE, evaluation is undertaken formatively in parallel with system implementation, rather than as afterthought when a system fails to deliver benefits. PROBE is seen as a popular evaluation methodology, and is widely used in the UK. A recent example of its application (which forms the basis of the claimed impact) is the Blood Safety Tracking System.

The PROBE methodology was used between 2007-2010 to design the evaluation of a pilot project intended to reduce the number of deaths due to inappropriate blood transfusions. Between 1996 and 2004, five patients died as a result of receiving incompatible blood during transfusions, and incompatibility contributed to the deaths of a further nine patients, and caused major illness in 54 patients [C]. In November 2006, the National Patient Safety Agency (NPSA) issued a national specification, giving guidance on how to ensure that patients receive blood that is correctly cross-matched for them, and titled  “Electronic Clinical Transfusion Management System (ECTMS)”.
The NPSA set a goal of reducing the number of incompatible transfusions by 50% over 3-5 years (from January 1 2005) [A].

The NHS Connecting for Health Evaluation Programme (CFHEP) funded a pilot study at Croydon Health Services (Mayday) Healthcare NHS Trust, the main objective of which was the deployment of an Electronic Blood Tracking System that complied with the ECTMS specification.

PROBE was used (in conjunction with another methodology) to design the evaluation of this pilot study [B], which ran from 2007 until 2010. This evaluation process covered aspects such as the implementation and installation of the system, project management, changes in clinical and other working practices required to implement the system, the response of clinical and other staff to the system, patient perception, the efficiency, effectiveness and reliability of the system (particularly in terms of ensuring the correct matching of patients and blood) and any problems identified with the system or its use.

PROBE was used in [C] to “explore, guide and provide context to the quantitative analysis.” This gave a detailed picture of stakeholder views and organisational processes involved in the pilot, and highlighted important factors such as the need for effective change management (almost 2000 people were involved in the new system), and the need for adequate supplier representation at high-level project meetings.

The pilot study overall final report, “Right Patient Right Blood”, was published in September 2011 [D]. Specific benefits of the system identified in the report included improved patient safety (through identification during bedside checks), better monitoring of blood movement, and better management of blood stocks. A significant finding of the formal evaluation process [C] was that the time required to resolve “blood alerts” (corresponding to possible problems with transfusion) fell from 123 hours in the initial period of live system usage to around 30 minutes per alert by August 2010.

Head of Patient Safety at the NPSA, stated that  “The NPSA welcomes this report on the pilot of the ECTMS ... The lessons learned will be useful for all NHS organisations implementing systems to improve the safety of blood transfusion.” [E]

The precise impacts we claim for PROBE, therefore, include changes to the delivery of a public service, benefits from public service improvements, adoption of new processes, and improvements to patient health outcomes. The direct policy impacts of the work are still ongoing, as evidenced below:

Testimonial by The Head of Patient and Public Partnerships, NHS Connecting for Health (supplied in Autumn 2012)

“Evaluation of public sector Information and Communication Technology projects is often overlooked or done without due recognition of the complexity of the scheme. The work of Heather Heathfield developed the multifactorial evaluation approach PROBE (Project Review, Objective Evaluation) in two ways. Firstly, by focusing upon evaluation questions which are important to electronic health record projects. Secondly, by providing more detailed information about how to evaluate, including a review of the various tools and techniques available, with information to assist the reader in applying these appropriately. This greatly improved understanding of ICT projects. The approach is easily understood by lay people, and ensures that more equitable comparisons can be made. PROBE was used systematically by most of the 17 different projects that were part of the Electronic Record Development and Implementation Programme. This evaluation data is still being used, informing the development of the current Information Strategy (The Power of Information)[F]. This information strategy from the Department of Health, published in 2012, sets a ten-year framework for transforming information for the NHS, public health and social care.

Sources to corroborate the impact

All public sources are linked from http://www.scmdt.mmu.ac.uk/cir/REF

  1. National Patient Safety Agency (November 2006). Electronic Clinical Transfusion Management System: Right Patient Right Blood.
  2. Information on the use of PROBE as part of the Blood Safety Tracking Pilot evaluation.
  3. Kay, J. and Roudsari, A. (2011) NHS CFHEP 003 - Evaluation of the pilot implementation of an IT specification for a blood tracking systems.
  4. Croydon Health Services NHS Trust (2011) Right Patient Right Blood Pilot. Final Report. Report available at: http://www.scmdt.mmu.ac.uk/cir/REF
  5. National Patient Safety Agency response to ECTMS pilot.
  6. Department of Health (2012). The power of information: putting all of us in control of the health and care information we need. Published May 21 2012.

Personal statement available to view from:

  1. The Head of Patient and Public Partnerships, NHS Connecting for Health corroborating ongoing impacts of research on improvements to the NHS, including the development of ICT evaluations and the creation of information policy. 

Synthetic Biology and Citizen Science

Summary:

This case study is based on research at the interface of computer science and biology, undertaken at MMU.  Subsequent inter-disciplinary work was partly supported by the EPSRC Bridging the Gaps: NanoInfoBio project, and led to the creation of a new “citizen science” organization, which is now one of the leading groups of its type in the world. The specific impacts are (1) generation of revenue for a new business operation created as a result of the project, (2) the stimulation of and influence on policy debate, and (3) the stimulation of public interest and engagement in science and engineering.

Underpinning research

Members of the Novel Computation Group (School of Computing, Mathematics and Digital Technology, MMU) study alternatives to “traditional” forms of computing, including new theoretical models and non-silicon physical substrates [1]. This often requires an inherently inter-disciplinary approach. One well-established research theme is synthetic biology: the application of engineering principles to the (re)design of living biological systems. The Group leader, Prof. Amos, is a world- leading researcher in the field of molecular computing; his 1997 Ph.D. thesis was the first in the field of DNA computation. In 2006, Amos published a book [2] describing this emerging field of biological computation; a review in Nature (March 15 2007, p. 263-264) stated that “This is an enjoyable book… I recommend it to anyone interested in computation writ large who is not afraid to cross disciplinary boundaries that once seem impassable.” Amos contributed to a paper published around the same time, which called for closer links to be established between computer science, engineering and the life sciences in order to fulfill the true potential of bio-inspired computing [3]. To help further develop such links, two projects were funded at MMU from 2009 onwards. The Bridging the Gaps: NanoInfoBio (NIB) project at MMU was funded by the EPSRC in 2009 (EP/H000291/1, £289,095), and ran for 30 months [4]. The core aim of the project was to bring together researchers from different disciplines to focus on problems at the intersection of nanotechnology, informatics and biology. In total, the project supported 21 different projects, with funding ranging from small student bursaries to £25K grants. BACTOCOM was a European Commission Collaborative Project (248919, €1,949,997, 2010-2013) to investigate the creation of an “evolvable” synthetic biology platform [1]. Amos was the PI on both grants, and, together, they led to a significant increase in the amount of work at the intersection of biology and computer science, both at MMU and beyond.

The original NIB proposal stated “An important aspect of the programme’s activities will be public engagement”, and public outreach was assigned a specific work package task within BACTOCOM. One of the public engagement events we organized was a panel discussion on synthetic biology, which was attended by Asa Calow, a Director of the Manchester Digital Laboratory (MadLab). MadLab is an independent open learning and experimentation environment, and Calow suggested that we collaborate to develop a “DIY bio” laboratory space in Manchester. This meeting was a direct result of the public engagement event, as there had been no prior contact. The resulting proposal was funded by the Wellcome Trust’s People Awards scheme (WT095313MA, £29,705), creating the DIYbio Manchester (DIYBIOMCR) project [5]. “Do-it-yourself” biology (DIYbio) is a growing international movement, the aim of which is to democratise and widen access to biotechnology (which, of course, is of direct relevance to synthetic biology). The fundamental philosophy of DIYbio is to bring together interested “amateurs’ with professional scientists, with the aim of stimulating discussion, widening public understanding, and ensuring the safe and ethical management of amateur experiments. With this project, we aimed to engage the public with underpinning research in the unit, as well as with the full range of new ideas and societal issues at the bio/info interface. When it was set up, DIYBIOMCR was only the second such group in the UK, and one of only a handful worldwide. The creation of this organization (and its resulting impact) can therefore be traced directly back (and attributed) to the NIB project, which, in turn, arose as a result of research in the unit taking place at the intersection of computer science and biology.

Key Staff:

Prof. Martyn Amos, Chair of Novel Computation; appointed SL, 2006; Reader 2010; Professor 2012.

References to the research

  1. Goñi-Moreno, A., Amos, M. & de la Cruz, F. (2013) Multicellular computing using conjugation for wiring. PLOS ONE 8(6), e65986. (Recent exemplar output of the BACTOCOM project.) doi: 10.1371/journal.pone.0065986 (1 Google Scholar citation).
  2. Amos, M. (2006) Genesis Machines: The New Science of Biocomputing. Published November 16 2006, ISBN 1843542242. Atlantic Books, London. Full details and list of reviews at http://www2.docm.mmu.ac.uk/STAFF/M.Amos/gm.html. Nature review at http://www.scribd.com/doc/12922/Biological-programming (23 citations for the book).
  3. Timmis, J., Amos, M., Banzhaf, W. & Tyrrell, A. (2006) "Going back to our roots": Second generation biocomputing. International Journal of Unconventional Computing 2:4, p.p. 349-378. (20 citations)
  4. Jacobs, N. & Amos, M. (2012) NanoInfoBio: A case-study in interdisciplinary research. In Kettunen, J., Hyrkkanen, U. & Lehto, A. (Eds.) Applied Research and Professional Education, p.p. 289-309. Turku University of Applied Sciences. Available at http://arxiv.org/abs/1211.5508. (An earlier version of this paper, made available in 2010 as an Arxiv pre-print, has 5 citations).
  5. Amos, M., Calow, A., Jacobs, N., Jung, H.Y., Linton, T. & Verran, J. (2012) Manchester DIYbio. In Bowater, L. & Yeoman, K., Science Communication: A Practical Guide for Scientists, p.p. 250- 251, Wiley-Blackwell.

Grants

Funder Project Value to MMU Dates PI
EPSRC Bridging the Gaps: NanoInfoBio £289,095, plus additional £50K from EPSRC. Total: £339,095 Sept. 2009-Nov. 2011 Martyn Amos
European Commission BACTOCOM:
Bacterial
Computing with Engineering Populations
€328,426
(Grant total€1,949,997).
Feb. 2010-Jul. 2013 Martyn Amos
Wellcome Trust DIYbio Manchester £29,705 Mar. 2011-May 2012 Martyn Amos

Details of the impact

NIB directly led to the creation of an entirely new organisation. DIYBIO Manchester (DIYBIOMCR) is now an established independent entity, the UK’s largest “do it yourself” biology group, and is an active participant in the global DIYbio movement. For a full list of relevant DIYBIOMCR activities, please see [A,B]. Relevant projects included the Manchester Microbe Map (a bacterial “atlas” of bus stops in Manchester), and the construction of “homebrew” DNA replication equipment.
As a result of the project, the turnover of MadLab (the parent organization) has grown significantly,

and, since 2011, an average of 80% of MadLab’s income has been due to DIYBIOMCR.  The MadLab finance manager, provided the following figures on 14/11/13:

Period Total Income DIYBIO-attributable Unrelated to DIYBIO DIYBIO-attributable %
2013-14 (pred.) £330,000 £228,000 £102,000 69
2012-13 £160,606 £140,919 £19,687 88
2011-12 £85,185 £72,011 £13,174 85

These show income that is directly attributable to the DIYBIOMCR project, including income from the Wellcome Trust, consultancies, residencies, paid workshops and other collaborative projects that arose as a result of the work done on DIYBIOMCR.

A deposition from MadLab Director states that “DIYBIOMCR has brought much positive attention to MadLab, and exposed the organization to a wide range of new audiences. We have become much more outward-facing, as opposed to focusing on community groups and in-house training. At the time, the Wellcome Trust funding was MadLab’s largest single block of external money, and represented its most significant external relationship with another organization. This seed-corn money, in turn, gave access to more funding, and allowed MadLab to present itself in a different light to external bodies. To put it bluntly, the Wellcome Trust “seal of approval” gave MadLab a significant “cachet”, on which we have since built. MadLab is now in the process of converting into a Community Interest Company, and we have separately raised £160K of the £250K we need to refurbish our space. Much of the money raised has come from local and national sources that have been developed and nurtured as a direct result of DIYBIOMCR.” [C]

In recent years the US Federal Bureau of Investigation (FBI) has developed a DIY bio program that aims to share information and make connections between amateur scientists and local law enforcement officers. The objective is to address any concerns (on either side of the regulatory “fence”), and to provide a framework for ongoing discussions. The Bureau’s local Weapons of
Mass Destruction (WMD) coordinators are the focal point of such conversations, and ensure that local authorities (e.g., fire and police service, environmental health) are informed of the nature and
location of cooperating DIY biology laboratories. The FBI also works closely with the Department of
State on biosecurity, and the Department of Health and Human services on biosafety.

As a result of our high-profile activities, the FBI invited DIYBIOMCR participants Asa Calow and Rachel Turner to attend a workshop held in San Francisco on 12-14/6/2012; Calow and Turner were the only UK-based participants in attendance, and they were specifically invited as a direct consequence of the outcomes of DIYBIOMCR. The purpose of the workshop was to “educate the FBI about biology” (in the context of the growing DIY biology movement), and to “foster the positive relationship among FBI, DIYbio, amateur biology and local stakeholder communities to ensure safe conduct in science.” [D] The FBI explicitly stated that “we want to model this for the international community...” (that is, they want to help to develop policy and practice, both within the US and beyond). Their explicit aim is to help to avoid extra regulation of DIYbio scientists “that hinders their abilities to push scientific progress. Ultimately, it will be up to the communities themselves how  they wish to operate, and the FBI will continue to work with them to further their efforts.” [D].

A subsequent email (27/7/2012) to Calow and Turner from an FBI supervisory special agent with the FBI Weapons of Mass Destruction (WMD) Directorate in Washington DC stated that “I wanted to thank you again for agreeing to come and present at the event ... Your perspective on media issues was delivered so well that I believe that a great impact was left [on] the attendees ... It really left a great impression on me that some direction and reinforcement is being given to this community in proactively engaging the media, to their ultimate benefit.” [D]

A later message (10/6/2013) to Prof. Amos from the FBI Special Agent (about the DIYBIOMCR contribution) stated that “their insight, engagement within their community and energy makes them, in my eyes, a leader in being a proactive member of the amateur biology community… These steps take a long time to climb, but it seems as if they have been very successful in developing a working model that serves their community.  Additionally, Rachel’s input regarding interactions with the media have kept this issue in high standing for our future events with this community.” [D]

The total immediate (in-person) audience reached was around 700 for DIYBIOMCR [B]. Results were disseminated widely, and though high-profile channels such as pieces (both 28/3/2012) on the main BBC Ten-o-clock News (estimated viewing figure: 4.75M) and Radio 4’s Today programme (average weekly audience of around 7M) [E,F].

Sources to corroborate the impact

All public sources are linked from http://www.scmdt.mmu.ac.uk/cir/REF

  1. DIYBIOMCR activities (during and after the grant):

    Full list of projects/activities at http://www.scmdt.mmu.ac.uk/cir/REF

    Manchester Microbe Map online at http://www.danhett.com/projects/microbe/.

    Nature SpotOn blog article on DIYBIOMCR at http://www.nature.com/spoton/2012/12/spoton-nyc- diy-science-manchester-digital-laboratory-madlab/.

    Wellcome Trust Public Engagement Adviser and DIYBIOMCR contact (details uploaded into the REF submission system)

  2. DIYBIOMCR evaluation:
    Final Evaluation Report, Sally Fort. Available at
    https://dl.dropboxusercontent.com/u/3105497/DIYBioMcrEvaluationReport.pdf.
  3. Impact of DIYbio on MadLab organization:
    Deposition from Director, MadLab (full details uploaded into the submission system).
  4. [D] FBI/DIYBIOMCR consultation:
    Information on FBI workshop available from link at top. Emails from FBI Special Agent (full details uploaded into the REF submission system) (27/6/12 and 10/6/13), plus Guardian Northerner blog article, 18/6/12 at http://bit.ly/189MVcQ
  5. Media coverage (TV):
    28/3/2012: DIYBIOMCR featured on BBC Ten O’Clock News; write-up available at BBC Online at http://www.bbc.co.uk/news/science-environment-17511710. Viewing figures for BBC1 from www.barb.co.uk for week ending April 1 2012.
  6. Media coverage (radio):
    28/3/2012: DIYBIOMCR featured on BBC Radio 4 Today Programme. Recording available at http://news.bbc.co.uk/today/hi/today/newsid_9709000/9709494.stm. Today programme figures from http://www.guardian.co.uk/media/2012/feb/02/radio-4-today-audience-figures-boost.