Hunting of foxes with dogs is (for the moment) banned in the UK. However, hunting of careers advisers with questionable research is still apparently legal. There have been a number of instances over the last few months of careers-adviser bashing by various bodies.
- ‘Throughout our work we have barely heard a good word about the careers work of the current Connexions service.’ – from the Unleashing Aspirations report by the Panel on Fair Access to the Professions (Summary and Recommendations, section 5.3, p. 34). [See Let the right one in for more comment on this report.]
- ‘The girls told us that, in their view, the quality of careers advice from Connexions is poor.’ – from the Shaping a Fairer Future report by the Women and Work Commission (p. 13).
- ‘Our research found that one in five people has needed to retrain or reskill as a result of unsatisfactory careers advice.’ – quote from Chris Jones, Director of City & Guilds in The Times, 9 September 2009.
Because these are not published in peer-reviewed journals they don’t have to explain exactly how they conducted their research and obtained their ‘evidence’. It’s very easy to produce dodgy statistics to support an argument which pushes your own predetermined agenda.
For a much better discussion about how you can really measure the effectiveness of careers interventions see the research report Evidence and Impact: Careers and guidance-related interventions produced on behalf of the CfBT Education Trust. In this report Deirdre Hughes and Geoff Gration surveyed a wide range of research on the effectiveness of careers guidance activities (have a look at the separate Literature Review). They explain very clearly the challenges of measuring such a complex interaction and lay out five levels of evidence quality (see p. 12 of the Synthesis Report). Most of the ‘research’ on which the criticism and policy recommendations made above are based would probably just about make it into the lowest level of evidence.
- How do you know that what you do is effective?
- What do you measure and how useful is it?
- What can you do when it’s not just your clients who are making important decisions based on unreliable evidence?
- Malcolm Maguire (2004) Measuring the outcomes of career guidance. International Journal for Educational and Vocational Guidance, 4(2-3) 179-192.
This covers much of the same ground as the CfBT report in looking at the complexities of measuring effectivness.
- Gunnar Schedin and Kerstin Armelius (2008) Does self-image matter? Client’s self-image, behaviour and evaluation of a career counselling session: an exploratory study. International Journal for the Advancement of Counselling, 30(3) 189-201.
Although this is a limited study it does indicate that opinion surveys such as the ones used above could be influenced by the self-image of the client as much as by the effectivenes of the guidance.
- Aminder Nijjar, Stop and measure the roses: How university careers services measure their effectiveness and success. HECSU, 2009.
Aminder has conducted a survey of what higher education careers services currently measure.