Blog

For test developers and test users, validity is the most fundamental concept in psychological assessment. It is also a surprisingly vexed notion. A review of the literature on validity composed by the “great minds” (e.g., Lee Cronbach, Jane Loevinger, Paul Meehl) will give you a case of vertigo. The definition of validity found in the AERA Standards for Educational and Psychological Testing is a statement by a committee—in the same way that a camel is a horse designed by a committee. The confusion about validity is the result of the way psychological measurement was conceptualized at the outset, beginning with Charles Spearman’s research on intelligence. Spearman taught at a private boys’ school; he noticed that his students’ scores on their various academic examinations were correlated. Based on this, he derived two conclusions. First, he proposed that there was a single, general factor underlying performance on all the exams. And second, he proposed that that factor was (or reflected) “intelligence”. Spearman set the framework and the terms of the discussion for all subsequent assessment research. The framework consists of two assumptions that follow from his two conclusions. Since Spearman’s time, virtually all psychological assessment has been based on these assumptions, neither of which is necessary or necessarily true. The first assumption is that individual differences in human performance depend on, or are related to, or reflect individual differences in the strength or magnitude of a corresponding underlying trait or propensity. The second assumption is that the goal of assessment is to measure individual differences in the strength or magnitude of the underlying trait. In this view, the goal of assessment is to measure traits. This view is never questioned, and it has implications for understanding validity. In this standard model, a test or measure is valid to the degree that it accurately measures the underlying trait. But this view of validity makes it impossible ever to determine validity. The problem is that the actual existence of traits is questionable on genetic and neuropsychological grounds. There are no anatomical or neurological structures corresponding to any of the many traits that have been proposed. Consequently, it is by definition impossible to determine whether a test accurately measures an underlying trait—because the existence of trait is in doubt. The confusion about the meaning of validity can be resolved fairly easily with a simplifying assumption. If we assume that the goal of assessment is to predict outcomes, then validity can be defined in terms of the ability to predict those outcomes. A measure of sales performance is valid if it predicts sales performance; a measure of customer service potential is valid if it predicts ratings of customer service, and so on. This view makes no assumptions about the existence of underlying traits the strength of which causes performance. It simply stipulates that assessment has a job to do, and that job is to predict non-test outcomes. This definition of validity satisfies the requirement of Occam’s razor, which states that one ought not multiply causal entities unnecessarily; the definition also satisfies the aesthetic of the Bauhaus movement which states that less is more.

  Read More »

Business literature typically characterizes the people at the top of organizations in remarkably positive terms. Senior executives are described as visionary, acute, charismatic, decisive, creative, etc.—i.e., as being substantially more talented than the average person. They are also usually described as being well motivated, as caring about their staff, their colleagues, their organizations, and their shareholders. The reality is somewhat less flattering.With regard to the talent level of senior executives, there are no reputable data to support the claim that they are more capable than a group of younger but equally well educated managers. The data do indicate, however, that, as one ascends the corporate hierarchy, the demands of jobs change, and the data show that derailment is a function of an inability to make the needed adjustments to changing job demands. We can conclude that the senior people in most organizations differ from their younger colleagues primarily in terms of age and political experience, and little else. But what about the claim that the senior people are better motivated than the rest of us? The example of Sprint Nextel, a telecommunications company headquartered in Overland Park, Kansas is instructive.Daniel R. Hesse, who formerly ran the wireless business at AT & T, became CEO of Sprint Nextel in December, 2007. Consider the situation he faced and with which he is valiantly trying to cope. First, the management team that he inherited had participated in the merger of Sprint with Nextel, a merger that is charitably described as “ill fated” because the two corporate cultures were so poorly aligned. Second, the same management team presided over a massive defection of customers; at 2.5 percent, Sprint’s client churn is the highest in the industry; in the first quarter of 2008, 1.1 million of Sprint Nextel’s 59.3 million customers defected to other companies. They left because they were tired of surly customer service and lackluster cell phones.At his first meeting with his senior management team, Mr. Hesse asked the group who was responsible for customer service and no one raised a hand. At his next meeting he asked senior managers how they had arrived at their earnings projections and the response was blank stares. Mr. Hesse has moved quickly to put some accountability in place for his senior managers, but that is not the point. The situation he encountered is actually quite common, the reason the situation is common is important, and has to do with the secret life of organizations.The fundamental dynamic in every organization is the individual search for power. It is a Darwinian process, which means some people are more successful at it than others, and they rise to the top. Perhaps the most important single challenge for top managers is to try to persuade their staff to spend less time trying to advance their careers and more time trying to enhance the performance of the organization that employs them. This is a challenge for two reasons. On the one hand, American culture, with its symbiotic ties to capitalism, is inherently individualistic and typically misreads Adam Smith’s concept of the “invisible hand”. Many people (who should actually know better) believe Adam Smith argued that if each person pursues his/her self-interest, the general welfare will take care of itself. Smith had no illusions about the rapacious tendencies of capitalism; he understood that, left to their own devices, most people will behave badly. Smith also understood how hard it is to get people to understand that their welfare is tied to the welfare of the collective.On the other hand, managers are usually evaluated and promoted based on how much their boss likes them. They are rarely evaluated and promoted based on their contribution to the maintenance of the entire organization, or to the team for whose performance they are responsible. And this was exactly what was going on at Sprint Nextel before Daniel Hesse arrived. His challenge is to damp down the individualistic excesses of his managers and find leaders who are willing to think about and work for the good of the organization rather than trying to enhance their personal careers. His biggest asset in this process is his own stature as a role model.

We've enjoyed a lot of positive feedback lately regarding our Hogan Development Survey marketing campaign, which features a number of different animals that "personify" the 11 derailer scales measured by the assessment. These potentially-derailing characteristics aren't necessarily bad (in fact, they can be very *good* depending on your line of work), so we developed a method of presenting them that makes the concept a bit more accessible, and actually injects some fun into the realm of "strategic self awareness." From the print ads associated with this campaign: Every leader has personality traits that threaten his or her success. The Hogan Development Survey is the only business-related assessment that measures performance risks that impede work relationships, hinder productivity and limit overall career potential. The HDS provides valuable feedback for strategic self-awareness, which is the key to overcoming these tendencies and achieving success in the workplace. We discuss the HDS in depth on our website, so be sure to read more about it and tell us... Which animal are you?

  Why is it that bad management seems such a universal issue? It seems that anyone who's knocked around any number of "underling" positions in any industry can relate to tales of incompetent, inefficient and often sadistic management. And if this is indeed the case, why does poor management continue to be such a pervasive problem in modern business organizations? Granted, it's fascinating to observe high-profile cases of epic managerial failure (Ken Lay, John Z. DeLorean, "Chainsaw Al" Dunlap), but what made these guys into the forces of destruction they became? And what are the implications of flawed personality in regard to tangible factors like corporate efficiency levels and profit margins? The following piece puts some hard numbers to some difficult questions. The vast body of leadership research that has accumulated over the past 100 years leads to two very different conclusions. On the one hand, many people believe that there is little consensus in the research literature regarding the principles of good management. For example, Hamel (2008) argues that the modern study of management is stagnant and out of date, and Khurana (2008) argues that efforts to create a science of management have failed. In an effort to provide guidance to practitioners, Kramer (2008) reviews the leadership literature and concludes that it is “…a strange mixture of alchemy, romantic idealism, and reason”, and concludes that the lack of consistent, actionable findings disposes some business people “…to wash their hands of the whole subject, talent shortage or no talent shortage.” On the other hand, Bloom and Van Reenen (2007) studied the performance of 732 manufacturing firms in the United States, Great Britain, France, and Germany and found that their financial performance was related to the extent to which the companies followed “well-established management practices” in the areas of shop floor operations, performance management, and talent management. The more profitable companies enhanced operations through continuous improvement, set clear performance goals, monitored, reviewed, and discussed performance, and aligned incentives with performance. Bloom and Van Reenen subsequently replicated their findings using an additional 3268 firms, including a large sample from Asia. Four of their conclusions are worth noting. First, there are in fact well established principles of management. Second, companies that use these principles make more money. Third, senior leadership determines the degree to which established management practices are used. And finally, the best run companies are multi-nationals, the worst run companies are government agencies, non-profits, and companies managed by second generation family members. The economic literature clearly shows that: (a) there are principles of good management that enhance organizational performance; and (b) some managers use these principles while others do not. However, Kramer (2008) and other critics are also right—there is little consensus in the psychological literature regarding the characteristics of good managers—cf. Hogan, 2007, pp. 106-109. In contrast, however, psychological research on managerial incompetence does converge, and psychologists have a lot to say about the characteristics of bad managers. Across studies, organizational level, and industry sector, the data show that failed managers have poor interpersonal skills, don’t learn from their mistakes, adapt poorly to change, and lack self-awareness—among other problems. This research is important for both economic and moral reasons. When managers fail, it costs time and resources to recruit, select, and on-board new ones. There are also hidden costs associated with "golden parachutes", missed business objectives, and destroyed employee morale. Lombardo reports that two Fortune 500 organizations approached the Center for Creative Leadership in 1985 for advice on preventing leadership failure (cited in Lombardo, Ruderman, & McCauley, 1988). These organizations estimated the cost of a failed executive was $500,000. Adjusted for inflation, that figure is almost one million dollars in 2008. Similarly, a poll of senior Human Resource executives estimated the cost of derailment to be between $750,000 and $1,500,000 per senior manager (DeVries & Kaiser, 2003). These costs will only grow as the talent pool shrinks (Frank, Finnegan, & Taylor, 2004) Moral considerations also underscore the importance of managerial incompetence. Specifically, bad leadership causes great misery to those subject to it (Hogan & Kaiser, 2005). The National Institute for Occupational Safety and Health (NIOSH), a division of the Centers for Disease Control, itself a division of the National Institutes of Health, published a report in 1999 (NIOSH, 1999) containing some alarming data. For example, 40% of workers nationwide report that their jobs are either very or extremely stressful, and NIOSH concludes that problems at work are more strongly associated with health complaints than any other life stressor, including finances and family problems. Consider next that organizational climate surveys routinely show that about 75 percent of working adults report that the most stressful aspect of their job is their immediate boss (Hogan, 2007, p. 106). Bad managers are a major health hazard that imposes enormous medical costs on society, and the problem of bad management should be taken seriously for moral as well as financial reasons. What is the base rate of managerial incompetence? In an informal internet survey of 245 employed adults, Curphy asked: (1) how many bosses have you worked for? And (2) how many of those bosses would you be willing to work for again? Respondents were only willing to work for 34% of their former bosses (Curphy, 2008). These results parallel those reported by Shipper and Wilson (1992) using actual organizational data. Table 1 contains ten published estimates of the base rate of managerial failure, which range from 30 to 67 percent with an average of about 50 percent. We suggest that two thirds of existing managers are insufferable and that 50% will ultimately fail.  

The following Robert Hogan commentary appeared on the Hogan Assessments Website in March 2007. It's a good summary of of why we (Hogan) do what we do, and I'd be curious to know the perception of the relationship between personality and career success from people *outside* the realm of I/O psychology. If you're currently searching for employees (or even an employer), how much does personality matter to you?

Read More »

Ronald Reagan was fond of saying, "He governs best who governs least." I am a lifetime liberal but on this point, I largely agree with the former President and conservative icon. Several years ago, on a commercial flight to somewhere, I was sitting next to a talkative retired aeronautical engineer. In the course of his monologue, he noted that he had been Ronald Reagan's boss in World War II. They ran a logistic operation for the Army in the western Pacific (Reagan had bad eyes and had received a deferment from combat assignments). I asked him about Reagan's performance, and he said Reagan was the best natural manager he had ever seen. I then pointed out that, inside the Washington beltway, Reagan was frequently criticized for his "hands off" leadership style, and the engineer snapped, "That's because he knows what he is doing." He then pointed out that everyone agreed that Reagan's transition into the White House had been the smoothest in recorded memory—more so even that of the compulsively organized Navy Nuclear Engineer Jimmy Carter—and this testified to Reagan's skill as a manager. Read More »

Welcome to "The Science of Personality," an informal repository of information and commentary related to industrial/occupational psychology.Much of the content is the work of Dr. Robert Hogan, founder of Hogan Assessment Systems and an international authority on personality assessment, leadership, and organizational effectiveness.As such, Dr. Hogan offers keen insight into the goings-on within the the I/O psych community, as well as a host of other related topics in his pursuit of strategic self-awareness and "educating the whole person."We invite you to explore a range of topics in the days to come, and to engage in ongoing dialogue related to personality and its role in the world of business and corporate society.