Editorial |
Evidence and selection processes
E. G. Jessop
Journal of Public Health
So Sir John Tooke's report1 has been released. It ended up reviewing the entire process of junior doctor training in the United Kingdom, but the trigger was a failure of the nation-wide Medical Training Application System (MTAS).
MTAS did of course contain some classic organizational pathologies. The Department of Health got locked-in to a failing course of action. Good project management requires clear identification at the outset of decision points when ‘go’/‘no go' decisions can be made. At the top responsibility was divided and then fragmented further by turnover in Deputy Chief Medical Officers (three in four years according to the inquiry). In public health, we know that if you want to make something happen, you have to identify one single individual who will lose sleep if it is not happening. And finally civil servants have again failed to get the best out of the developers in a complex information technology project. Some obvious mistakes were made (such as late changes to the specification) but one cannot help feeling that the underlying problem is that none of us know enough about information technology to specify the requirement properly and then manage the developers rigorously.
But we need to go forward. Let us consider a few propositions dispassionately.
First, a centralized application system must preferable to tens of thousands of doctors applying severally for tens of thousands of jobs. Central applications systems work well in other countries varying in size from New Zealand to the United States. They also work in other spheres in the United Kingdom—witness the higher education application scheme UCAS. Nation-wide all-speciality recruitment may, however, be too ambitious for medicine. It is interesting that general practice (GP) recruitment went well. ‘A national evaluation of MTAS GP applicants has shown overwhelming confidence in the selection centre process... The national short-listing system enabled applicants to be allocated in rank order to their highest available Deanery of preference. The scores of unplaced applicants were cascaded into Deaneries where the applicant was prepared to train and which still had vacancies’ (pp. 146–7).
Second, even with all its flaws MTAS did what it was intended to do. As Pashayan, Duff and Mason show in a paper2 published in this issue of the Journal (and cited by the Inquiry), MTAS short-listing discriminated well between appointable and not appointable applicants, at least in respect of selection into public health training. This is the first evidence in an emotional debate. We await reports from other specialties.
Third, interviews are not essential for good selection. Many candidates were distressed that they were not allowed to present their achievements at interview. Many consultants felt disempowered that they were not allowed to interview the applicants face-to-face. When the scandal broke, all candidates were guaranteed an interview. But objective scoring systems are defensible, at least in principle. Forty years ago Dawes3 showed the simple addition of grade point average to Graduate Record Examination score gave a better predictor of students' final performance than the considered judgement of their tutors—and bear in mind that tutors have a lot more contact with their students than the 30 min or so of a standard interview. Tamblyn has shown that exam scores predict the quality of primary care,4 and the likelihood of complaints against doctors.5 Of course you do have to choose the right numbers for your scoring system: Dawes' tutors knew that ability and achievement were the right things to measure—hence the selection of grade score and GRE as the two numerical measures; the exams in Tamblyn's studies were thorough and objective tests of knowledge and clinical skills.
Fourth, anonymizing the applications is good practice. Names reveal sex and ethnic origin, opening the way to unlawful discrimination. One assumes, though it was never stated explicitly, that medical school was removed from view in MTAS for the same reason, though this is flawed logic because it assumes that medical school is not a relevant predictor, that all medical schools are the same.
So MTAS was, and is, defensible: evidence-based selection. But what happened in 2007 was not defensible and resulted in huge distress for junior doctors. Incompetence on this scale is cruelty.
 |
References |
|---|
 TOP References |
|---|
- Tooke J. Aspiring to excellence. Finding and recommendations of the independent inquiry into Modernising Medical Careers. http://www.mmcinquiry.org.uk/.
- Pashayan N, Duff C, Mason BW. Selection into specialty training in public health: performance of the Medical Training Application Service shortlisting. J Public Health (2007) 29:331–7.
- Dawes R. The robust beauty of improper linear models. Judgement Under Uncertainty—Kahneman D, Slovic P, Tversky A, eds. (1979) 34. New York: Cambridge University Press. 571–82. Am Psychol 1982, 391–407.
- Tamblyn R, Abrahamowicz M, Brailovsky P, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA (1998) 280:989–96.
[Abstract/Free Full Text] - Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA (2007) 298:993–1001.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||