A new blood sport

Tally ho! The hounds have picked up a scent! It's a careers adviser.

Tally ho! The hounds have picked up a scent! It's a careers adviser.

Hunting of foxes with dogs is (for the moment) banned in the UK. However, hunting of careers advisers with questionable research is still apparently legal. There have been a number of instances over the last few months of careers-adviser bashing by various bodies.

Because these are not published in peer-reviewed journals they don’t have to explain exactly how they conducted their research and obtained their ‘evidence’. It’s very easy to produce dodgy statistics to support an argument which pushes your own predetermined agenda.

For a much better discussion about how you can really measure the effectiveness of careers interventions see the research report Evidence and Impact: Careers and guidance-related interventions produced on behalf of the CfBT Education Trust. In this report Deirdre Hughes and Geoff Gration surveyed a wide range of research on the effectiveness of careers guidance activities (have a look at the separate Literature Review). They explain very clearly the challenges of measuring such a complex interaction and lay out five levels of evidence quality  (see p. 12 of the Synthesis Report). Most of the ‘research’ on which the criticism and policy recommendations made above are based would probably just about make it into the lowest level of evidence.

  • How do you know that what you do is effective?
  • What do you measure and how useful is it?
  • What can you do when it’s not just your clients who are making important decisions based on unreliable evidence?

Further reading

, , , , , ,

  1. #1 by helencurry on 22 September 2009 - 16:28

    Connexions? I didn’t think they were real careers advisers these days – thought they were more personal counsellors 😉

    I always wonder whether the people bashing careers advisers ever actually experienced one-to-one careers guidance with a fully trained adviser. I know a lot of people write the whole thing off as ridiculous after a quiz at school told them to join the army (boys) or become hairdresser (girls). Of course this is going to be an unsatisfactory basis for a major life decision, but it hardly deserves to be called ‘careers guidance’ either. This is what I find most frustrating – that true careers services get tarred with the same brush.

    • #2 by David Winter on 22 September 2009 - 17:36

      ‘Personal advisers’ – yes. That’s part of the problem recently – too wide a remit and resources spread too thinly. To be fair, the Women and Work Commission report did point out that one of the main problems was under-resourcing. And they did complain about the broad responsibilities and the focus on NEETs – both of which are the result of government policies in the first place!

      I suspect one of the problems is that people are more likely to moan about a mildly negative experience than to cheer about a mildly positive experience. And most interactions will be either side of average, instead of being really dreadful or really brilliant.

      Having been involved in continuing professional development training and mentoring for Connexions (and other) advisers, I know that there are a lot of really dedicated people out there, struggling to do a good job with limited support and seemingly unattainable targets.

      There are some really rubbish careers advisers out there, I’m sure. But I think they are probably in the minority, just like the really fantastic ones.

  2. #3 by John King on 26 September 2009 - 19:38

    Skating around the real issue here – the media is reflecting a genuine attitude that students have towards careers guidance. The standard response being ‘we are good even if no-one thinks we are, only we can’t prove it because its difficult to measure’.

    With modern technology it is simple to measure the attitudes of clients coming in, simple to track progress, simple to measure outcomes in 6 months, simple to measure outcomes in 2 years or more – any drop in response would be more than outweighed by the sheer amounts of data produced lending significance to any result.

    Systems that rely over much on the performance of a single individual – ie the talent of a careers adviser – are inherently risky, and put pressure on that individual. Other professions, such as psychology, have developed systems (such as the NHS IAPT http://www.iapt.nhs.uk/about/iapt-pathfinder-programme/ ) that reduce this reliance on a single point of failure.

    Furthermore, for many clients walking in to an average Connexions, or even a HE CAS, the first impressions may be:

    – Shoddy building, poorly decorated
    – Jumble of books and cardboard boxes
    – Non-businesslike surroundings
    – Amateur design of posters
    – Unintelligible NOTICES IN CAPITALS covering any available wall surface

    Careers services advertise themselves as being of poor quality – why are they then surprised when people think they are of poor quality?

    • #4 by David Winter on 30 September 2009 - 18:50

      Hi John. The media may be reflecting an attitude that some students/adults are expressing about careers advisers. However, the ‘evidence’ is often anecdotal. If it is researched, it is usually opinion research rather than outcome research, which has limited usefulness.
      I don’t have a problem with people complaining if they feel they have not had a good service. What I do have a problem with is lazy journalists peddling a stereotype and lazy policy makers basing decisions on poor evidence.

      I take issue with your claim that it is ‘simple’ to measure all these things. There may be things you can measure easily, but do they really tell you what you need to know? We had an awayday at QM during which we had a discussion about what we ought to measure versus what we could realistically measure. What I liked about the Evidence and Impact report was that it tackled this complexity in a thoughtful way.
      One thing you have to think about is the ratio of the time you spend helping a client to the time you spend assessing the outcome. For psychological therapies, where the length of interaction may be over several hours, it’s OK to spend 30 minutes evaluating outcomes. For our interactions, which may be no more than 15 minutes, how much time can we expect clients to spend filling in evaluations and giving us progress reports?

  3. #5 by John King on 28 September 2009 - 00:21

    On the other hand, if we accept that the converse is true – ie., that views on careers services are shaped by inappropriate expectations rather than performance, what does this mean?

    Part of a careers interview is ‘contracting’ – a fundamental part of which is expectation setting.

    However, the contract does not really begin at this point. It begins when a decision is made to engage with the careers service, again on entry to the service, again at the reception desk…

    It may be that when services reach out in marketing, they are setting a contract at this point. We need to be careful what we promise.

    Can we learn again from CBT, where there is often little distinction between theory and practice? Practice is merely simplified and applied theory, an educational process of making the client understand what the theory is so that he or she might apply it his or herself.

    What would this mean in careers terms?

    Perhaps we could simplify items from this blog and include them in careers welcome guides, alongside a couple of exercises to try at home. Or we could have wall displays, similar to a museum exhibit, explaining different theories and differing views on them, with takeaway leaflets on each one that might appeal. Isn’t this more important than walls of posters and job ads which are not only duplicated online, but are also enforcing a contract we didn’t intend to set – that we will provide a job.

    Demystifying careers advisers might also make us far less appealing targets to hunt…

    • #6 by David Winter on 30 September 2009 - 18:57

      I completely agree with you that expectation management should start before you ever see a client. It is in your marketing and your first contact with careers staff. Demystifying the process and making it more transparent to new and potential users would probably help a great deal. I’m not sure about having wall displays about theories is the answer, but perhaps Youtube clips of examples of good careers interviews (probably in the form of a reconstruction rather than the real thing) might be interesting.

      What other ideas do people have?

      • #7 by John King on 26 October 2009 - 23:49

        Youtube clips would demystify the process – but perhaps I chose the wrong word, because although this would surely help, it wasn’t what I meant.

        I mean that we should teach students to advise themselves; that we should take our role as ‘experts’ out of the equation, because it hasn’t really worked. Some advisers certainly do qualify as experts; but not many, and not nearly enough when we look across the country. For every positive comment I hear about careers advice, I hear five negatives. Houston, we really do have a problem. Closing the airlock door won’t make it go away. We’re still losing oxygen.

        Far better to teach what we know of theory to clients; to let them choose which theory they prefer, and then to apply it themselves. Let us have an honest dialogue with clients: here’s one theory, here’s another; we don’t know which is best, we’re not going to tell you which to use. Choose for yourself, but here’s a few different exercises to try at home, if you want; and here’s a bit of the background theory which led to the exercises. See if you agree with it.

        We can do this in groups, individually, remotely, in person or via a website. What we need are the materials; the lesson plans; the digested reads; the worksheets; the one-pagers.

        If anyone has any of the above, suitable for lay readers, please post your links!

      • #8 by David Winter on 30 October 2009 - 16:11

        Don’t you already teach clients to advise themselves? One of my key aims is to prevent them coming back by making them self sufficient.

        I don’t know that teaching the theories directly is the answer. It’s hard enough trying to get people to engage with thinking about career choice (I usually end up trying to sneak it into sessions on CVs and interviews). It’s easier to sneak in key concepts rather than whole theories.

        By the way, have you visited the Resources page? Keep visiting, it will grow.

  4. #9 by helencurry on 12 October 2009 - 16:44

    Some more careers service hating – http://www.gradplus.com/graduate-news/careers-services-need-to-improve-to-help-graduate-job-hunters-19402674.aspx This time instigated, oh, by an independent CV writing consultancy.

    And it neatly avoids the issue of exactly how far universities’ responsibilites to alumni should extend. I assume they mean within a few years post-graduation, but the way it is written it could mean anything, even free help to all graduates throughout their careers… And why is it universities’ responsibility? Why shouldn’t the government support graduates as they do other jobseekers?

    • #10 by David Winter on 12 October 2009 - 23:11

      We know that non-usage rates are high even for current students so it’s not surprising that they are high for graduates.

      We do have a responsibility to look very carefully at our marketing and brand management. But we’re not helped by so many people slagging off the careers service on the basis of dodgy data.

      Hmm. I wonder what research has been done on the effectiveness of CV writing services…

      By the way, I can’t seem to track down the 63% figure on the HECSU website, so I don’t know where they got that from. It could be a very bad misreading of something in the report How are Higher Education Career Services Experiencing the Recession?.

  5. #11 by John King on 30 October 2009 - 13:38

    I think there is an appetite for this – although it needs to be taught in a modern way, which means a side of A4, not a book to read. Although self-interest may be playing a part here (I find careers theories really interesting, yet books about career theory really turn me off).

    And whilst we may find it hard to get people to engage with thinking about ‘career choice’, I haven’t yet found anyone who isn’t interested in their own future. It’s part of being human; it takes training to NOT think about your future. Maybe Gottfredson’s blinkered theory explains why: career choice is about compromise; we don’t want to compromise; so we don’t want to think about career choice.

    Many careers theories seem to be more a collection of concepts anyway – so why not break them down as such? Your resources page is excellent (thank you!) and does move some way in this direction. It also points the way forward: why not add a cover page explaining what the benefits of comparing two against one actually are to the start of that document? A simplified explanation of personal construct theory (or the part of it that relates to that exercise) would provide additional motivation for the client to actually complete the exercise. Tell them what you are going to do to them, and why; offer some evidence of its effectiveness, if any exists; then do it (or get them to do it); then seek feedback.

    • #12 by David Winter on 30 October 2009 - 16:16

      I guess this blog and the discussions around it are part of the process of getting these concepts out there. It would be a big job for one person to produce all of this stuff. So, I’m trying to stimulate thoughts and ideas in the hope that everyone who reads this blog will be inspired to incorporate something into their practice and produce materials that can then propogate these useful ideas to clients.

      So, if you feel like contributing such a resource…

      By the way, I’m the same about reading whole books – I’m a dipper. However, I have just forced myself to read through the whole of Kerr Inkson’s book on theory (see the Books page) and I actually quite enjoyed it. It is very well constructed and readable – for a theory book!

      P.S. I’ve just worked out how to add more reply levels to the comments so that threads can be kept together more.

  6. #13 by Aminder K Nijjar on 30 October 2009 - 14:32

    Thank you for all this discussion, how interesting!
    Whilst I agree with David about the issues around ‘measuring’ outcomes, I also agree with David that we do need to do more (in some cases quite a lot more) on the outcomes and impact of our work, as well as our branding etc etc.
    If you are interested in investigating this further and especially in making actual improvements, please let me know!

  1. Is guidance past it’s sell-by date? « Careers – in Theory

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: