For test developers and test users, validity is the most fundamental concept in psychological assessment. It is also a surprisingly vexed notion. A review of the literature on validity composed by the “great minds” (e.g., Lee Cronbach, Jane Loevinger, Paul Meehl) will give you a case of vertigo. The definition of validity found in the AERA Standards for Educational and Psychological Testing is a statement by a committee—in the same way that a camel is a horse designed by a committee. The confusion about validity is the result of the way psychological measurement was conceptualized at the outset, beginning with Charles Spearman’s research on intelligence. Spearman taught at a private boys’ school; he noticed that his students’ scores on their various academic examinations were correlated. Based on this, he derived two conclusions. First, he proposed that there was a single, general factor underlying performance on all the exams. And second, he proposed that that factor was (or reflected) “intelligence”. Spearman set the framework and the terms of the discussion for all subsequent assessment research. The framework consists of two assumptions that follow from his two conclusions. Since Spearman’s time, virtually all psychological assessment has been based on these assumptions, neither of which is necessary or necessarily true. The first assumption is that individual differences in human performance depend on, or are related to, or reflect individual differences in the strength or magnitude of a corresponding underlying trait or propensity. The second assumption is that the goal of assessment is to measure individual differences in the strength or magnitude of the underlying trait. In this view, the goal of assessment is to measure traits. This view is never questioned, and it has implications for understanding validity. In this standard model, a test or measure is valid to the degree that it accurately measures the underlying trait. But this view of validity makes it impossible ever to determine validity. The problem is that the actual existence of traits is questionable on genetic and neuropsychological grounds. There are no anatomical or neurological structures corresponding to any of the many traits that have been proposed. Consequently, it is by definition impossible to determine whether a test accurately measures an underlying trait—because the existence of trait is in doubt. The confusion about the meaning of validity can be resolved fairly easily with a simplifying assumption. If we assume that the goal of assessment is to predict outcomes, then validity can be defined in terms of the ability to predict those outcomes. A measure of sales performance is valid if it predicts sales performance; a measure of customer service potential is valid if it predicts ratings of customer service, and so on. This view makes no assumptions about the existence of underlying traits the strength of which causes performance. It simply stipulates that assessment has a job to do, and that job is to predict non-test outcomes. This definition of validity satisfies the requirement of Occam’s razor, which states that one ought not multiply causal entities unnecessarily; the definition also satisfies the aesthetic of the Bauhaus movement which states that less is more.
Get the latest content delivered straight to your inbox every week.
We've enjoyed a lot of positive feedback lately regarding our Hogan Development Survey marketing campaign, which features a number of different animals that "personify" the 11 derailer scales measured by the assessment. These potentially-derailing characteristics aren't necessarily bad (in fact, they can be very *good* depending on your line of work), so we developed a method of presenting them that makes the concept a bit more accessible, and actually injects some fun into the realm of "strategic self awareness." From the print ads associated with this campaign: Every leader has personality traits that threaten his or her success. The Hogan Development Survey is the only business-related assessment that measures performance risks that impede work relationships, hinder productivity and limit overall career potential. The HDS provides valuable feedback for strategic self-awareness, which is the key to overcoming these tendencies and achieving success in the workplace. We discuss the HDS in depth on our website, so be sure to read more about it and tell us... Which animal are you?
Why is it that bad management seems such a universal issue? It seems that anyone who's knocked around any number of "underling" positions in any industry can relate to tales of incompetent, inefficient and often sadistic management. And if this is indeed the case, why does poor management continue to be such a pervasive problem in modern business organizations? Granted, it's fascinating to observe high-profile cases of epic managerial failure (Ken Lay, John Z. DeLorean, "Chainsaw Al" Dunlap), but what made these guys into the forces of destruction they became? And what are the implications of flawed personality in regard to tangible factors like corporate efficiency levels and profit margins? The following piece puts some hard numbers to some difficult questions. The vast body of leadership research that has accumulated over the past 100 years leads to two very different conclusions. On the one hand, many people believe that there is little consensus in the research literature regarding the principles of good management. For example, Hamel (2008) argues that the modern study of management is stagnant and out of date, and Khurana (2008) argues that efforts to create a science of management have failed. In an effort to provide guidance to practitioners, Kramer (2008) reviews the leadership literature and concludes that it is “…a strange mixture of alchemy, romantic idealism, and reason”, and concludes that the lack of consistent, actionable findings disposes some business people “…to wash their hands of the whole subject, talent shortage or no talent shortage.” On the other hand, Bloom and Van Reenen (2007) studied the performance of 732 manufacturing firms in the United States, Great Britain, France, and Germany and found that their financial performance was related to the extent to which the companies followed “well-established management practices” in the areas of shop floor operations, performance management, and talent management. The more profitable companies enhanced operations through continuous improvement, set clear performance goals, monitored, reviewed, and discussed performance, and aligned incentives with performance. Bloom and Van Reenen subsequently replicated their findings using an additional 3268 firms, including a large sample from Asia. Four of their conclusions are worth noting. First, there are in fact well established principles of management. Second, companies that use these principles make more money. Third, senior leadership determines the degree to which established management practices are used. And finally, the best run companies are multi-nationals, the worst run companies are government agencies, non-profits, and companies managed by second generation family members. The economic literature clearly shows that: (a) there are principles of good management that enhance organizational performance; and (b) some managers use these principles while others do not. However, Kramer (2008) and other critics are also right—there is little consensus in the psychological literature regarding the characteristics of good managers—cf. Hogan, 2007, pp. 106-109. In contrast, however, psychological research on managerial incompetence does converge, and psychologists have a lot to say about the characteristics of bad managers. Across studies, organizational level, and industry sector, the data show that failed managers have poor interpersonal skills, don’t learn from their mistakes, adapt poorly to change, and lack self-awareness—among other problems. This research is important for both economic and moral reasons. When managers fail, it costs time and resources to recruit, select, and on-board new ones. There are also hidden costs associated with "golden parachutes", missed business objectives, and destroyed employee morale. Lombardo reports that two Fortune 500 organizations approached the Center for Creative Leadership in 1985 for advice on preventing leadership failure (cited in Lombardo, Ruderman, & McCauley, 1988). These organizations estimated the cost of a failed executive was $500,000. Adjusted for inflation, that figure is almost one million dollars in 2008. Similarly, a poll of senior Human Resource executives estimated the cost of derailment to be between $750,000 and $1,500,000 per senior manager (DeVries & Kaiser, 2003). These costs will only grow as the talent pool shrinks (Frank, Finnegan, & Taylor, 2004) Moral considerations also underscore the importance of managerial incompetence. Specifically, bad leadership causes great misery to those subject to it (Hogan & Kaiser, 2005). The National Institute for Occupational Safety and Health (NIOSH), a division of the Centers for Disease Control, itself a division of the National Institutes of Health, published a report in 1999 (NIOSH, 1999) containing some alarming data. For example, 40% of workers nationwide report that their jobs are either very or extremely stressful, and NIOSH concludes that problems at work are more strongly associated with health complaints than any other life stressor, including finances and family problems. Consider next that organizational climate surveys routinely show that about 75 percent of working adults report that the most stressful aspect of their job is their immediate boss (Hogan, 2007, p. 106). Bad managers are a major health hazard that imposes enormous medical costs on society, and the problem of bad management should be taken seriously for moral as well as financial reasons. What is the base rate of managerial incompetence? In an informal internet survey of 245 employed adults, Curphy asked: (1) how many bosses have you worked for? And (2) how many of those bosses would you be willing to work for again? Respondents were only willing to work for 34% of their former bosses (Curphy, 2008). These results parallel those reported by Shipper and Wilson (1992) using actual organizational data. Table 1 contains ten published estimates of the base rate of managerial failure, which range from 30 to 67 percent with an average of about 50 percent. We suggest that two thirds of existing managers are insufferable and that 50% will ultimately fail.
Today's post is excerpted from the essay "Personality Theory and Positive Psychology" by Robert Hogan and Michael J. Benson. Read More »
Ronald Reagan was fond of saying, "He governs best who governs least." I am a lifetime liberal but on this point, I largely agree with the former President and conservative icon. Several years ago, on a commercial flight to somewhere, I was sitting next to a talkative retired aeronautical engineer. In the course of his monologue, he noted that he had been Ronald Reagan's boss in World War II. They ran a logistic operation for the Army in the western Pacific (Reagan had bad eyes and had received a deferment from combat assignments). I asked him about Reagan's performance, and he said Reagan was the best natural manager he had ever seen. I then pointed out that, inside the Washington beltway, Reagan was frequently criticized for his "hands off" leadership style, and the engineer snapped, "That's because he knows what he is doing." He then pointed out that everyone agreed that Reagan's transition into the White House had been the smoothest in recorded memory—more so even that of the compulsively organized Navy Nuclear Engineer Jimmy Carter—and this testified to Reagan's skill as a manager. Read More »