Project Epsom: How Valid is Your Questionnaire? Management Summary

This management summary contains some content which was originally presented in Professor Peter Saville’s Keynote Speeches at: The British Psychological Society Division of Occupational Psychology Conference, Stratford-upon-Avon, January 2008: “Personality Questionnaires – Valid Inferences, False Prophecies” The Psychological Society of South Africa Annual Conference, Johannesburg, August 2008: “Does Your Test Work?” A&DC Conference, Institute of Directors, London, November 2008: “A Comparison of Leadership in Business and Elite Athletes” © 2008 Saville Consulting. All rights reserved. 1 Project Epsom: How Valid Is Your Questionnaire? Management Summary: A New Comparative Study of the Major Personality Questionnaires in Predicting Job Performance Professor Peter Saville, Rab MacIver, Dr Rainer Kurz & Tom Hopton Overview ● The Saville Consulting Wave® Professional Styles questionnaire was the most valid assessment for measuring work performance, in a major study of 308 participants who completed a range of popular personality questionnaires ● Wave Focus Styles and the newly-developed Saville Personality Questionnaire (Saville PQ™), comparable with the Occupational Personality Questionnaire (OPQ®), were found to be at least as valid, if not more so, than the OPQ32i ● Compared to the OPQ32i, the Saville PQ uses more modern technology, takes about 13 rather than 60 minutes to complete, gives both normative and ipsative scores and measures both workplace motives and talents ● Using more valid questionnaires can dramatically reduce selection errors with great impact on company profitability The Study ● Questionnaires were compared against the same independent work performance criteria. These were: - A global performance measure of Accomplishing Objectives, Applying Specialist Knowledge and Demonstrating potential - The SHL Great Eight work competencies ● Measures of participants’ work performance were established by asking third-parties to independently rate how effectively the participants performed in the work competencies covered by the Great Eight and global performance criteria ● The more accurately a personality questionnaire predicts how independent raters have judged the work performance of the participant in a completely separate rating form, the more valid the personality questionnaire © 2008 Saville Consulting. All rights reserved. 2 ● The most important aspect of a questionnaire is its validity for measuring and predicting work performance ● Validity is shown as a correlation coefficient between -1 and +1. A validity of 0 indicates chance measurement and is as effective as flipping a coin to predict a persons job performance. A validity of +1 is perfect measurement of job performance but is impossible to achieve in practice. A validity of +0.3, although seemingly low, can offer high financial gain and can cut selection errors by half or more ● Studies generally indicate that a good personality questionnaire can have a validity of +0.3. Validities above +0.7 for any technique are virtually unknown in the literature ● Higher validities significantly reduce selection errors ● Ability tests have validities of around +0.5; unstructured interviews around +0.2. Educational qualifications are surprisingly poor predictors of performance, at around +0.1 ● For the purposes of this summary, we report validity results from seven key questionnaires: - Saville Consulting Wave® Professional Styles (Wave Professional) - Saville Consulting Wave® Focus Styles (Wave Focus) - Saville Personality Questionnaire (Saville PQ) - Occupational Personality Questionnaire® (OPQ32i) - 16 Personality Factor Questionnaire (16PF5) - Hogan Personality Inventory (Hogan PI) - NEO Personality Inventory (NEO) ● These seven key questionnaires were compared against the external ratings of global work performance ● They were then compared against the external ratings of the SHL Great Eight work performance competencies © 2008 Saville Consulting. All rights reserved. 3 Results Figure 1. The validity of seven key questionnaires in measuring global work performance. Matched Sample of N=308 against external ratings on Global Performance (Applying Specialist Expertise, Accomplishing Objectives, Demonstrating Potential) 0.6 High Validity 0.5 0.4 Moderate Validity 0.3 0.2 0.1 Wave Professional Wave Focus NEO Saville PQ Hogan PI 16PF5 0 OPQ32i Chance ● All of the seven questionnaires show at least a moderate level of validity in predicting work performance according to the global work performance criteria ● Saville Consulting Wave Professional Styles comprehensively outperforms all other questionnaires in terms of validity ● Wave Focus Styles and the Saville PQ take under 15 minutes each to complete yet compare favourably in terms of validity with much longer questionnaires such as the OPQ32i, the Hogan Personality Inventory and the 16PF5 © 2008 Saville Consulting. All rights reserved. 4 Figure 2. The average validity of seven key questionnaires in measuring the Great Eight competencies. The average validity in predicting SHL’s Great Eight. Matched Sample of 308. 0.6 High Validity 0.5 0.4 Moderate 0.3 Validity 0.2 0.1 Wave Professional Wave Focus Saville PQ NEO OPQ32i Hogan PI 0 16PF5 Chance ● The Saville Consulting questionnaires are the most valid questionnaires for measuring work performance in Project Epsom, even when validity is defined by the independent SHL Great Eight measures of work performance ● Saville Consulting questionnaires are strong in terms of validity compared with the OPQ, even against SHL’s own model of work © 2008 Saville Consulting. All rights reserved. 5 Figure 3. The power of seven key questionnaires in terms of their delivery of validity in 15 minutes. Matched Sample of 308 0.6 Valid Power 0.5 0.4 0.3 0.2 0.1 Wave Focus Saville PQ Wave Professional 16PF5 Hogan PI NEO OPQ32i 0 ● “Power” relates to measuring effectiveness or output in a given unit of time. In terms of personality questionnaires, it can relate to the questionnaire which provides the greatest validity per unit of time. Figure 3 compares the power of the tests in terms of how much validity can be achieved by each in 15 minutes ● The Wave Focus Styles and Saville PQ questionnaires are the most powerful, offering high validity in the shortest completion times ● Other questionnaires in Project Epsom are the subject of future reports, academic papers and presentations, but briefly: - With the DISC questionnaire, no evidence could be found for the validities of +0.75-0.95 which were claimed on the Thomas International Website - The Hogan Development Survey, which assesses the “Dark Side” of personality, failed to relate positively or negatively to work performance. Only 8% of participants retained their exact Dark Side profile when they re-completed the same questionnaire after just one week © 2008 Saville Consulting. All rights reserved. 6 Increasing Validity Increases Return on Investment ● Questionnaires of high validity increase the chance of selecting better work performers; considerably reducing selection errors ● An example of a serious selection error is selecting a candidate from the bottom 20% of performers when you mean to select from the top 20% of performers - If a questionnaire has a validity of +0.3, one person in every ten that you select will prove to be in the bottom 20% of performers - If a questionnaire has a validity of +0.6, one person in every fifty that you select will prove to be in the bottom 20% of performers ● Moving from recruiting using a questionnaire with a validity of +0.2 to using a questionnaire with a validity of +0.4 can double the costbenefit to an organization ● It can reduce the number of serious selection errors five-fold, remarkably improving the accuracy of the selection process Conclusions ● The Saville Consulting questionnaires maximise the measurement of performance at work, assessing work-specific competencies which are relevant to modern business life ● Some other questionnaires, perhaps developed half a century ago: - measure out of date aspects of work performance - have work-irrelevant clinical scales of poor validity - use obscure and badly-written questions ● Cost savings from using better tests can be “startlingly large”. Neither employer nor employee gains from poor selection ● When the participants were followed up after six months, essentially the same questionnaire rank order of validity was maintained with Saville Consulting Professional Styles and Focus Styles outperforming the other questionnaires © 2008 Saville Consulting. All rights reserved. 7 The Full Project Epsom Phase One Paper This document offers a brief management summary of the major Saville Consulting research programme, Project Epsom. A full version of this research paper is available online and in hardcopy and further details can be found at www.savilleconsulting.com, or by calling +44(0)1534 726820 or +44(0)1372 475700. References to accompany this Management Summary are in the full version of this paper, “How Valid is Your Questionnaire?”, available from www.savilleconsulting.com. Copyright Permissions © 2008 Saville Consulting. All rights reserved. This summary is available in hard-copy format and online at www.savilleconsulting.com. The authors give permission to copy, print, or distribute this research paper provided that: 1. each copy makes clear who the research paper’s authors are; 2. each copy acknowledges that this paper was produced through research endeavour by Saville Consulting; 3. copies are not altered without the consent of the senior author. © 2008 Saville Consulting. All rights reserved. 8 www.savilleconsulting.com/howvalidisyourquestionnaire For more information please call +44 (0)1372 475700 Saville Consulting UK Ltd Harley House 94 Hare Lane Claygate Surrey KT10 0RB United Kingdom Saville Consulting British Channel Islands 1st Floor Anley House, Anley Street St Helier Jersey JE2 3QE Tel: +44 ((0)1372 475700 ) [email protected] Tel: +44(0)1534 726820 ( ) [email protected] Saville Consulting Asia Pacific Saville Consulting North America P O Box 1855 North Sydney NSW 2060 Australia P.O. Box 1446 Hightstown, NJ 08520 USA Tel: +612 8004 2941 [email protected] Tel: +1 609 918 9009 [email protected] © 2008 Saville Consulting. All rights reserved. 9