I have known many smart guys in my life, but John Holland, who died last month, was the man who most influenced my career. John was part of a small group of famous psychologists (John Flannagan, J. P. Guilford, Robert Thorndike, Harrison Gough, Hans Eysenck, Alphonse Chapanis, and B.F. Skinner) whose interest in applied psychology and assessment was sparked by their experiences in World War II. If you have ever seen Warren Buffett interviewed on TV, you will have a good sense of John, who was also from Omaha. He was mild mannered, unpretentious, overtly modest, wickedly funny, ferociously competitive, and very hard working—he was one of the five most published scientists of the 20th century. He was a keen football fan (the Kansas City Chiefs), a fine pianist, and catnip for women—his wife Elsie (also from Omaha) could have had a career in the movies. And he created intense admiration and loyalty in everyone who worked with him because he was so smart, so perceptive, so principled, and so very funny. The same seems to be true for Warren Buffett.
I learned five lessons from John that are worth repeating. The first concerns the importance of assessment for guiding peoples’ lives. As Freud said, the two biggest problems in life concern choosing a mate and choosing an occupation and people never do either for rational reasons—and he was right. The process of choosing a mate—male/female relations—is bound up in biology and hormones and is utterly irrational. The process of choosing an occupation is bound up with one’s relationship with one’s parents; nonetheless, it is in principle possible to make rational career choices and the key to doing that is valid assessment and competent feedback.
The second lesson that I learned is utter disdain for the rules of the psychometric (or academic testing) establishment—e.g., what we learned in graduate school. One of John’s best lines was, “Forget everything you learned in graduate school”; he meant it and he was right. Mainstream psychometrics concerns measuring entities (i.e., determining “true scores”). But applied assessment has a job to do, and that is to predict outcomes. The psychometric establishment is only concerned in principle with how their methods apply to real world; in reality, they don’t care. Real test development is an intellectual and scientific activity that requires careful thought and some creativity; there are no formulae for good test development, there are no cook book recipes for developing meaningful assessment, and the best tests have been developed by mavericks.
The third lesson I learned was that it is essential to have a sound, well thought out, conceptual basis for your measurement model. John was an avid student of philosophy of science. He was greatly concerned about, and immensely pleased with, the way in which his measurement model was theory-based and his research was consistent with the best precepts of real science: have an idea, test the idea, refine the idea based on the test, then start over. This lesson is quite unique and none of our competitors understand it.
The fourth lesson I learned is that if you develop your assessments correctly, they will sell. John made a lot of money from his Vocational Preference Inventory and his Self-Directed Search. The same is true for Raymond Cattell and Harrison Gough. At some point, the market can distinguish between valid assessment and psychometric garbage.
Finally, I learned from all of these guys (Holland, Cattell, Gough, etc.) that you have to retain control of your intellectual property. They all found it difficult to manage the test sales by themselves (John’s garage was stacked full of paper products), they sold their rights to “real” business people, and spent the rest of their lives bitterly regretting their decision. In 1982, Joyce and I sold the rights of the HPI to National Computer Systems. They were as incompetent as all big public companies seem to be; it was a terrible decision, and when we got the rights to the HPI back in 1992, we began making money immediately, and we vowed never to make that mistake again.
It is important to remember John’s approach to assessment because, after the wind blows away all the psychometric schlock, his work will still be standing. There is a strong tendency in our discipline to forget the past, to assume that the research that is going on today is qualitatively better and has somehow superseded the earlier research. That, of course, has to do with the vanity and arrogance of youth. Virtually all of us could profitably reread the early work of Binet, Spearman, Strong, Allport, Murray, and of course, John Holland. And when we do, we will be surprised to discover how little progress has been made in the fundamentals of assessment over the past 100 years.