There are many debates about the best way to select people into medical school, but one of the questions I ponder is how do we judge the success of such selection processes? There are a few problems in doing this.
Firstly, there’s the ‘restriction of range’ problem. The people who don’t get into medical school are often lost to follow-up. We cannot easily set up an experiment to see how they might have got on in medical school, had they been admitted. This means the students we follow are only those who were successful in the selection process. So, simple correlations between selection measures and intra-medical school measures can usually only distinguish the successful from the very successful. The simplest correlations are between selection measures and assessment outcomes – many have done that, including us. It’s helpful but not the whole answer.
Another measure could relate to our social responsibilities – the extent to which our selection processes match the health needs of our societies in which we practise. For example, Otago medical school has adopted the ‘mirror on society’ principle where we aim for the demographic makeup of the medical school class to match the demographic makeup of our country.
These areas generally try to predict success, but a third area to research is to predict difficulty. This area was the focus of our article published this week in BMC Medical Education. Even here though, there are challenges for researchers as, in general, medical students are motivated and capable – so they mostly achieve well in medical school. This means the outcome of interest in our research (students who have difficulty in the course) are relatively infrequent –this is a desirable educational outcome, but it is a hindrance in research design.
As part of a more programmatic approach to assessment at the University of Otago, we keep a fairly close eye on our students and like to have sensitive measures of difficulty – we’d rather find a few who might be struggling but turn out to be doing well, than to miss a few who later get into bother. Our system is more sensitive than specific.
…we’d rather find a few who might be struggling but turn out to be doing well, than to miss a few who later get into bother.
We’ve found that students who are new to the country and have had less time to acculturate are at greater risk of developing problems, as are those with lower academic grades at entry and those with lower scores on the component of an aptitude test that measures understanding people. Men ran into more problems than women but no ethnic group was at greater risk.
Much as it pains me to say it, men are at greater risk of running into difficulty during their course but we’re not about to use gender as a selection measure! Although we keep trying to deny it, whichever way we look at it, academic grades still seem to be a good measure of success and difficulty – I guess doctors need to be smart. However, they also need to understand the society in which they will practise and understand people – these are measures on which we should place a greater focus.