|The new training landscape|
The first cohort of surgical trainees using the Intercollegiate Surgical Curriculum Programme (ISCP) came to the end of specialty training during 2012, with more to finish in 2013. For this group, becoming a surgeon has been not only about meeting the challenges of the day-to-day working environment but they have also pioneered ways of learning to train in a setting where progress, accountability, assessment and competence have become much more systematic.
The world of training within ISCP is as complex and challenging for trainees and trainers as any surgical setting. Throughout their training they will have experienced a different approach to development and their trainers have experienced new ways of overseeing their progress within competence-based programmes.
Although the ISCP has been evaluated previously, this was done during the pilot phases with the aim of gathering understanding and knowledge of how the various specialist curricula were to be implemented. In addition, the previous evaluation was undertaken at a time of great change with external pressures including the European Working Time Regulation (EWTR), Modernising Medical Careers (MMC) and Medical Training Application Service (MTAS), which threatened to overwhelm the training landscape. In 2011, the ISCP Management Group decided it was time to undertake a further evaluation with two underlying themes. First, to determine whether planned learning opportunities, programmes, courses and activities actually produce the desired result. Second, how could the curricula be improved?
This article summarises the outcomes of the evaluation and highlights the challenges for ISCP together with a strategy for the next phase of its evolution. It opens a discussion in which we hope that as many of you as possible will participate.
The evaluation was based on a case study approach, which included all stakeholders and crossed regional, organisational, specialty and curricular boundaries. It used a qualitative methodology widely used in learning systems research (Kirkpatrick D. . Revising Kirkpatrick’s four-level model. Training and Development, 1: 54-7). The model comprises four levels and each measures a different and complementary aspect of ISCP development and implementation.
The first level is the reaction to ISCP, which includes perceptions of training overall and of the role of ISCP as a curriculum. Second, there are the issues related to learning, which include syllabus content, a definition of the quality of assessment and its contribution to ISCP objectives and an understanding of the benefit of Workplace-based Assessments (WBAs) as processes for learning. Third, there are the practicalities of the performance of ISCP, particularly with respect to its effect on the trainer-trainee relationship and how effective it is in supporting practical training. Finally, there is the impact of ISCP on stimulating change and innovation to enhance future training.
During the period of the evaluation, changes have occurred to ISCP – most notably the inclusion of the Orthopaedic Competence Assessment Programme (OCAP), which had been established in Trauma and Orthopaedic Surgery before ISCP became fully integrated. This has provided additional detail and has enabled comparisons with different and complementary approaches.
Findings of the Evaluation
Reaction: The ISCP has had a major impact on surgical training over the last few years. It has become part of the infrastructure of surgical education, providing a web-based system describing the comprehensive syllabus content of all surgical specialties, defining the approach to WBAs and providing evidence of training progression. However, the balance between the technical performance of the system from a central management perspective and the emphasis of trainees and trainers on practical surgical training has polarised opinion. This polarisation is consistent with individual reactions to a value system for training that is perceived to be either imposed on and created by organisations external to the training environment or helpful to those seeking a rationalisation and systematic approach to training that demystifies and delineates the process.
Learning: A fundamental component of the ISCP has been the recognition of the achievement of competence across clinical and technical skills. As has always been the case, assessment of progress within the workplace is key for the acquisition of craft skills. The assessment processes have proven a major limitation, however. WBAs have detracted and diminished the more informal formative relationship between trainer and trainee. System-based assessments have generated a “tick-box” culture and WBAs have become instruments of validation and quality control of training programmes rather than a true reflection of trainee progression. As a result, the assessment process is set at the lowest common denominator and does not allow due credit for innovation and excellence. There is also a perception that the ARCP process has become adversarial rather than a mechanism to assess training progression and personal / career trajectory and highlight good performance. A balance needs to be re-established between the needs of the system and the needs of the trainee surgeon to allow for constructive criticism and formative management of personal development yet ensure appropriate standards are maintained.
Performance: The ISCP management systems and the assessment process have been criticised as too inflexible and adversely affecting the trainer-trainee relationship. The ISCP is an evolutionary process, and with evolution comes change. Such change has created problems, however, and the ISCP has proven to be an unpopular training management system despite efforts to update the website regularly to improve its functionality. There have been limited opportunities for feedback, and indeed comments to the ISCP help-desk have really been the only developmental dialogue between users and providers and these discussions have been largely functional in nature and rarely educational. The vocabulary of ISCP is dominated by functionalism and production metaphors and it lacks the language of a learning community.
Impact: These practical criticisms, however, need to be put in the context of conscious and unconscious changes in behaviour that the evaluation has identified. The ISCP as the received wisdom and reference system for support of practical workplace training has challenged users to think about what they do and what they value. Users recognise that it is an effective adjunct to formalise established training principles, providing both a good structure and culture for training. This reflects not only the comprehensive content of each specialty curriculum but also the benefits of constructive feedback to trainees on their performance and progress. The ISCP provides a set of values, which trainers and trainees can use to determine what is to be achieved in individual posts and throughout training programmes.
Challenges of the Evaluation
The evaluation has posed a number of key challenges, which need to be addressed as part of the ISCP’s further development. The sense of a centrist approach, which of necessity originated when ISCP was initially implemented, has adversely affected local approaches to development and implementation. The ISCP is effectively a product, which can be used to suit local and regional practises. There are examples within the Schools of Surgery where modifications have been introduced very effectively. Increased flexibility within the use of ISCP, as well as easier feedback for users, would help spread best practise across programmes and specialties.
The web-based system will continue to evolve to facilitate the recording of evidence of activity and progress. These changes in functionality need to include user feedback, but in turn users need to appreciate that certain aspects are determined by the regulatory framework and need to be included without modification. Nevertheless, website changes should be communicated more effectively, with possible piloting before full implementation.
The incorporation of the ISCP into the training environment is a particularly important challenge. The impression of the ISCP has been of a didactic and inflexible process. It needs to evolve to allow users flexibility and selectivity to train and be trained in their own way. This requires responsiveness by ISCP to reflect differing approaches. The approach to WBAs should have greater emphasis on formative feedback rather than the perceived summative pass / fail attitudes. Assessments should be tailored according to trainee needs – more prescriptive for development and monitoring of those with difficulties and a light touch for those making good progress. This would facilitate appreciation of excellence and achievement. Clearly, evidence still needs to be recorded and the outcomes of formative feedback should be documented to demonstrate progression.
The perceived negative effect on the trainer-trainee relationship reflects a number of influences, including the EWTR, service pressures and an inflexible assessment process within the curriculum. Surgery is a craft specialty, in which training needs to allow learners to develop and progress individually. An initial stimulus to the development of the ISCP was a desire to set standards in a way that was perceived as more difficult under the previous apprenticeship style of training. However, development of competence and, with experience, mastery of a craft is based on learning as an apprentice. The evaluation suggests that this vital relationship between trainer and trainee needs to evolve to a mentor and apprentice approach. Changes in team structure and working practices have a potential adverse effect on this approach but experience has shown that trainee assessment is much improved by several assessors carefully deciding their views on an individual trainee.
In summary, the principal challenge is to shift the emphasis, which has developed around ISCP, of a system for recording experience through training to a process which enhances and facilitates learning in the workplace, using assessments for learning rather than of learning and with trainees achieving the highest standards in their chosen specialty and training programme.
Strategy for ISCP development
This description of the evaluation findings and challenges summarises a much longer report. This report has a number of recurring themes and these have been carefully considered by the ISCP Management Group. The Group believes that the only way to address the development of ISCP is to stimulate dialogue among all those involved with training, including Schools of Surgery / Deaneries, TPDs, Specialty Training Committees, Educational and Clinical Supervisors and, of course, trainees themselves.
We therefore propose over the next six months to invite stakeholders to engage in discussion of the recurring themes and to help us to identify where changes and new applications can be made to ISCP locally, regionally and nationally and within specialties and programmes. The themes are listed in the table. The RCS Regional Coordinators will be helping to facilitate these discussions, and we shall also be using regional meetings and social media.
We plan to collate all feedback and conclusions and present the findings at the Regional Representatives meetings of RCSEng, RCSEd and RCPSG in Summer 2013. We hope and expect that this will drive changes to the functionality and role of ISCP to the advantage of surgical training in the UK and we urge all those reading this to support the process and encourage your surgical colleagues to do likewise.
Surgical Director, ISCP
A version of this article is also published in the March edition of the Bulletin of the Royal College of Surgeons of England.