Research

At Hogan, we are committed to continuously developing new assessments, reports, and other tools to meet emerging market needs. More importantly, HRD is heavily involved in driving new product development based on the latest academic findings and through the use of sound research practices. As a result, Hogan’s clients and partners can rest assured that newly introduced Hogan products come with the same reliability, validity, and practical utility as our more established and well-known products.

In recent years, Hogan has focused product development efforts on helping our clients meet several key objectives. Specifically, these efforts assisted our clients in (a) developing off-the-shelf selection systems, (b) evaluating employee leadership potential, (c) reducing accidents and other safety-related incidents, and (d) predicting organization-specific competencies. Below, we briefly describe new Hogan products designed to meet these needs:

Hogan Safety Report

To address the ever-increasing number of on-the-job accidents and injuries and their associated costs to organizations, Hogan developed the Safety Report. Organizations can use this report to identify individuals who are likely to engage in safe and productive behaviors. In developing this new product, we identified facets of personality most predictive of safety-related outcomes and developed algorithms using these dimensions. To accompany this report, we also completed the Hogan Safety technical manual detailing the development of the report and providing strong validation evidence through case studies using real-world safety-related outcomes.

Hogan Express Report

To address our clients’ needs to efficiently select employees into various job families, Hogan developed the Hogan Express Report. This product uses personality assessment and validity generalization techniques to predict job performance within seven different job families: (a) Managers & Executives, (b) Professionals, (c) Technicians & Specialists, (d) Sales & Customer Support, (e) Administrative & Clerical, (f) Operations & Trades, and (g) Service & Support. To accompany this new product, we also completed the Hogan Express technical manual, which details how we developed the report and provides strong validity evidence.

Hogan Advantage Report

After receiving requests from partners and distributors for a report designed to compete in the entry-level selection market, Hogan identified a list of competencies that would apply to a wide range of entry-level occupations. Three such competencies – Dependability, Composure, and Customer Focus – emerged from this research. Hogan then developed algorithms to predict each competency. To accompany this report, we also completed the Hogan Advantage manual detailing the development of the report and providing strong validity evidence using correlations with other tests, adjective descriptor ratings, and other cross-validation research.

High Potential Report

Another growing client need focused on the identification of employees with significant leadership potential. This lead to the development of the Hogan High Potential Report. Organizations can use this report to predict competency-based requirements associated with future job performance for management recruits. To develop this report, we identified key requirements for performance in management-level positions, accumulated validity evidence for these key requirements, and selected scales to predict management-related competencies. To accompany this report, we also completed the High Potential technical manual, which details the development of the report and provides validation evidence.

Customized Competency Reports

In addition to the off-the-shelf reports described above, Hogan also engages in efforts to develop reports using personality data to predict competencies specific to our client organizations. In these efforts, we leverage data contained in the Hogan archive to identify scales and facets most predictive of each competency. We then design algorithms to predict these competencies in the client organization, and package these competency-based results in customized reports available to our clients. As with other new Hogan products, we pair these reports with technical documentation detailing the development of the new report and providing validation evidence to support its use in the client organization.

All of the technical manuals mentioned above can be purchased in the Hogan bookstore.

Hogan has partnered with numerous companies, including over a hundred Fortune 500 companies, to assess selection and development needs. The domestic research branch of HRD is primarily responsible for designing and implementing client- related research studies. These studies most often take one of the following forms: (a) job profiling, (b) validity generalization (VG), (c) criterion-related validity, (d) competency-based, (e) customized research, or (f) other services.

Job Profiling Studies. Job profiling includes a review of job descriptions, identification of subject matter experts (i.e. SMEs - job incumbents, supervisors, those intimately familiar with the job), the collection of job analysis data, and analyses of these ratings. We also provide qualitative feedback based on the analyses to the client. Job profiling studies result in a desktop reference clients can use when interpreting Hogan assessment data.

VG Studies. With VG studies, HRD works with clients to obtain information allowing us to match worker requirements of a target job with similar jobs in the extensive Hogan archive. This information allows us to identify the assessment scales that reliably predict job performance. In short, we create new selection profiles based on criterion-related validity data from similar jobs and industries. This process consists of first conducting a job analysis using SMEs as well as conducting a focus group, when possible. The next step involves using VG methods (e.g. meta-analytic job family validity generalization, transportability of validity, and synthetic/job component validity) based on archival criterion-related validity studies to identify Hogan scale (HPI, HDS, MVPI) recommendations that best predict performance. We conduct adverse impact analyses on the recommended profile and provide the client with full technical documentation based on the Uniform Guidelines and other professional standards. VG studies provide organizations with specific screening/development guidelines for the target job. For more information, please see our current handbook chapter on conducting validity generalization studies.

Criterion-Related Validity Studies. With criterion-related validity studies, we establish profiles by collecting assessment scores and performance data from a large sample of job applicants and/or incumbents at the client organization. We analyze the data and, in combination with validity generalization evidence, create a profile consisting of the Hogan scales (HPI, HDS, MVPI) that best predict job performance. This process provides the most customized solution and helps demonstrate utility and return on investment for the client. We conduct adverse impact analyses on the recommended profile using the incumbent sample and provide the client with full technical documentation based on the Uniform Guidelines and other professional standards. Criterion-related studies offer organization specific candidate screening/development guidelines using custom Hogan profiles to predict job performance with an increase in precision from using incumbent data.

Competency-Based Studies. Competency-based studies can take one of three forms, competency mapping, competency profiles, and competency algorithms. Competency mapping studies consist of mapping the client’s competency model to the Hogan competency model and using synthetic/job component validity evidence to provide the client with the Hogan scales that best predict job performance. This type of study is best used as a desktop reference aligning Hogan scales to the client’s competency model. Competency profiling studies include competency mapping followed by the creation of a client specific profile, providing suggested ranges on the Hogan scales. These studies provide the client with a customized competency-based profile for screening or development purposes. Competency algorithm studies are the most customized option. These studies also include competency mapping as well as using Hogan archival data to build custom competency algorithms. Assessment and criterion data are collected from a large incumbent sample, when available, and analyzed to help provide further predictive validity for the new competency algorithms. These competency algorithms then drive a customized client report. More information on our competency model.

Customized Research. Clients may present unique research requests that fall outside of our standard research study processes. As a result, we develop customized solutions to meet these needs. For example, a worldwide leader in sportswear manufacturing contacted Hogan to design a talent management program. They needed a system to assess internal talent, compare results against position requirements, and facilitate leadership development programs. Hogan designed, developed, and implemented a global talent management process and system to meet these needs. This system provided the client with a cumulative fit rating based on assessment, job performance and experiential data. Two years later, the client continues to use this system with great success. Read the full case study.

Other Services

In addition to conducting these research studies, HRD also works with clients to maintain existing profiles and help clients comply with legal guidelines. This includes: (a) revalidation of existing profiles, (b) pass rate and adverse impact analyses, and (c) dissemination of our work.

Revalidation Studies. Test validation experts recommend that professionals review and/or update findings obtained in an initial validation study after five years. Adhering to these standards, HRD works with clients to revalidate their existing selection profiles. This process consists of (a) collecting new job analysis information, (b) collecting new performance data, or (c) using existing objective and subjective performance data, when available. Revalidation studies help clients ensure that their selection profiles continue to accurately reflect the job and identify the best applicants.

Pass Rate and Adverse Impact Analyses. HRD conducts follow-up pass rate and adverse impact analyses to examine how the profile is functioning after implementation. This allows Hogan and the client to ensure that the profile is meeting necessary legal guidelines and successfully screening in/out the necessary applicants.

Dissemination of our Work.

SIOP. The Society for Industrial Organizational Psychology (SIOP) is an APA Division that enhances human well-being and performance in organizational and work settings by promoting the science, practice, and teaching of industrial-organizational psychology. HRD shares its findings in personality research at the SIOP Annual Conference each year. Sample SIOP presentations.

Journal Articles. As Industrial-Organizational psychologists, HRD strives to contribute to the knowledge and development of the field of personality and psychology. Each year, members of HRD publish articles in various well-known journals, such as the Journal of Applied Psychology, Personnel Psychology, Leadership Quarterly, and Journal of Personality Assessment, among others. Recent Hogan journal publications.

Along with the design and implementation of client research studies, HRD also provides research-based support to Hogan’s worldwide network of distributors, partners, and clients. This support includes conducting validation research to support the implementation of Hogan solutions for selection, development, coaching, and other human resource management initiatives. Another goal of the HRD international team concerns adapting Hogan products and services to new international markets. To achieve this goal, HRD works with local experts to translate Hogan materials to local languages, maintaining the integrity of the Hogan instruments in new cultural adaptations. Based on these translations, the international research team conducts equivalence analyses and develops local norms to maximize the utility of Hogan’s products and services in local markets abroad.

Translation Process:

For Hogan, the goal of translation is to maintain the integrity and content of the original assessment while ensuring cultural sensitivity and relevance. To accomplish this, we use a standardized translation process that ensures congruence with the original English forms. This process, based upon a review of the research literature and standards from professional psychological organizations, conforms to the International Test Commission’s Test Translation and Adaptation Guidelines. Hogan’s translation process follows a four-stage process, consisting of (a) initial translation, (b) translation review, (c) pilot-testing, and (d) implementation.

Initial Translation. The first step to create a translation is identifying qualified translators. We work with our international partners to identify qualified individuals based on language fluency, psychological background, and familiarity with the target culture. Once we identify translators, we provide them with detailed translation guidelines and the original U.S. English items. We use adaptation to attain congruence with the original assessment while also ensuring cultural relevance. We use this method because, in practice, language and cultural nuances may make it impossible to create exact and meaningful translations of some items. Therefore, we allow minor adjustments where absolutely necessary, although such changes must be documented and justified.

Translation Review. During this phase, initial translations are reviewed through a process that combines forward and back translation. After initial translations are complete, a second set of translators provides back translations and evaluates the translated items against the original English items. These reviewers judge the acceptability of the initial translations and propose revisions when necessary. Once the initial translators and reviewers reach consensus on final translations, the adapted assessments move to pilot-testing.

Pilot-Testing. The new translation is pilot tested on individuals fluent in the target language. These individuals provide input about item wording and the test-taking experience. After these individuals complete the assessments, our partners conduct focus groups to discuss any problems or concerns with the new items. Any issues identified through these focus groups are sent for correction before final implementation of the translated assessments.

Implementation. Once all parties are confident that the translation is sound, we upload the new assessment to Hogan’s online assess¬ment management system for administration. We then monitor assessment results to determine if any items or scales fall out of acceptable ranges of performance, using our equivalence process.

Assessment Equivalence Process:

Assessment equivalence refers to the comparability of measures used in different cultural groups. The use of different translations of the same assessment within global organizations often relies on the ability to compare scores across languages and cultures. Therefore, before launching translated assessments, it is the test publisher’s responsibility to examine each form’s psychometric properties. At Hogan, our extensive equivalence process, founded in Classical Test Theory (CTT) and Procrustes Analysis, helps ensure the quality of all assessments we release globally.

Classical Test Theory. Based in CTT, we review the adapted assessments’ scale- and item-level functioning by first gathering a large dataset comprised of test cases completed in the target language/culture. Next, we build a benchmark sample from our U.S. English normative dataset that mirrors the composition of the translation sample, matching the samples on relevant characteristics where possible (e.g., job type). We then compare scale psychometrics, such as item means and item-total correlations, to identify adapted items that may need revision.

Procrustes Analysis. In multi-dimensional personality assessments (such as the HPI, HDS, and MVPI), it is common for test authors to propose a theoretical structure of the concepts measured on their assessments. That structure is typically evaluated and confirmed during the construction of the original assessment. However, once an assessment is translated into another language, it is important to establish evidence for the hypothesized factor structure within the adapted measure. At Hogan, we establish this evidence using Procrustes Analysis. With this technique, Hogan researchers compare factor loadings from the translated assessment (i.e., Target Language Factor Structure) to factor loadings from the original assessment (i.e., U.S. English Factor Structure). We then compute congruence coefficients to determine the degree of similarity between the two structures. This technique allows us to determine the degree of similarity between the factor structures of the two assessments.

Norms Process:

Norms provide a context for interpreting scores because they allow us to compare individual scores with those of a relevant group. When people use normative scores to make important decisions, such as whether or not an individual gets a job, those norms impact peoples’ lives in very important ways. As such, Hogan follows strict guidelines in creating international norms for translated assessments. We base our guidelines on twenty years of research, our experience in translating our tools into over thirty languages, and a review of the literature regarding best practices.

Considerations. Despite the importance of norms, there is little consensus regarding how to create them. Although many articles describe how to create a single norm, few provide guidance on choosing suitable normative samples or issues to consider when creating a norm. The primary source on norm development is the Standards for Educational and Psychological Testing, which suggests considering the composition and relevance of normative samples to determine their appropriateness. Composition of a norm involves the sample size and the degree to which the norm represents a target population. Norm relevance addresses alignment between the comparison group and the purpose(s) for which the scores are to be used.

Types of Norms. For our translated assessments, we create local norms that represent working populations in the target culture. Hogan publishes two types of local norms: itinerant and general. We create itinerant norms as precursors to general norms. Itinerant norms provide local normative information until we collect sufficient data to calculate general norms. These general norms closely resemble the composition of a given country’s workforce. To ensure these approximations, we rely on labor statistics from the target country to create general norms that match the distribution of jobs in the workforce as closely as possible. In addition, we stratify general norms by job categories, ethnicity, and gender. When necessary, Hogan also publishes subgroup norms to provide useful supplemental information by representing specific subsets of local populations.

Implementing & Updating Norms. One last consideration concerns how and when professionals should update norms. When updating norms, it is important to consider the potential effects of the change. For developmental applications, feedback may change as a result of norm updates. Such changes have even greater implications for personnel selection, where an applicant’s scores may cross minimum cutoff scores as a result of the norm change. As such, Hogan closely examines score distributions and other factors over time to determine when a norm may require maintenance. More importantly, Hogan investigates the impact of any proposed norm changes on existing profiles and chooses appropriate times to update norms so that active projects are not adversely affected. Because updating norms can be difficult, many test publishers avoid it, insisting on the use of sometimes outdated interpretive frameworks. However, the Standards obligate test publishers to maintain norms to permit continued accurate and appropriate score interpretation. Therefore, despite the difficulties, Hogan updates norms when appropriate for a given language or country.

Each year, HRD collaborates with graduate students and professors around the world on research activities involving our assessments. Generally, HRD offers three types of academic collaborations: (a) students or professors collect data using the Hogan tools, (b) Hogan provides academics with data from the Hogan archive, and (c) professors use the Hogan tools as part of a program or class.

Our Academic Research program benefits both parties. First, these relationships provide objective validation evidence for our assessments and promote Hogan’s name recognition. Second, students and professors increase the authority and generalizability of their research by using valid, widely used tools (e.g., HPI, HDS, MVPI). Given our long-standing ties with the academic community, Hogan remains committed to providing ongoing support for student researchers as they work toward completing their graduate degrees.

How to Apply: We welcome inquires about both academic collaborations and student projects. If you are interested in working with Hogan, we have a short application process requesting information about your project and timeline. There is a separate application for academic collaboration (universities, faculty) and student collaboration efforts.

To participate in our research program, academic researchers first complete an application that provides Hogan with background information on the proposed study (e.g., hypotheses, research design, analyses). Next, we review the merits of the proposal and approve the project after considering several factors (e.g., is it a sound, ethical research question). The academic researchers and Hogan then sign a service agreement, which outlines the obligations of both parties during the research study. Specifically, Hogan agrees to provide access to our assessments or archival data and the student/professor agrees to share his or her data and final paper with Hogan. Studies usually begin soon after the service agreement is finalized.

Publications: Based on the work completed through our research program, some students and researchers publish their research in peer-reviewed journals. Below are some examples that resulted from these collaboration efforts.

HPI

Kisamore, J. L., Stone, T. H., & Jawahar, J. (2007). Academic integrity: The relationship between individual and situational factors on misconduct contemplations. Journal of Business Ethics, 75, 381-394.

HDS

Benson, M. J., & Campbell, J. P. (2007). To be, or not to be, linear: An expanded representation of personality and its relationship to leadership performance. International Journal of Selection and Assessment, 15, 232-249.

MVPI

Thomas, J. L., Dickson, M. W., & Bliese, P. D. (2001). Values predicting leader performance in the U. S. army reserve officer training corps assessment center: Evidence for a personality-mediated model. The Leadership Quarterly, 12, 181-196.

Universities Using Hogan Tools

Several universities use Hogan assessments in their coursework. For example, executive MBA programs use our assessments to develop managers within their leadership development classes or as a decision point in an assessment center to develop leadership competencies. In addition, students who take our assessments as part of a course often receive developmental feedback and interact with coaches from our extensive Hogan coaching network. Below is a list of Academic programs that use Hogan and employ our tools as part of the classroom experience.

• Auburn University – Center for Business and Economic Development
• Columbia University - Columbia Business School
• Georgia State University – Robinson School of Business
• Indiana University – Kelley School of Business
• Northwestern University – Center for Learning and Organizational Change
• University of Maryland – University College School of Business
• University of California, Berkeley - Haas School of Business
• Rutgers University – Rutgers Business School
• Vanderbilt University – Owen School of Management
• Washington University in St. Louis – Olin School of Business

Academic Case Studies

The following case studies illustrate how the Hogan assessments have been used in different academic settings.

Harvard Entrepreneurs

Identifying "At-Risk" Students in a High Achieving Population

Hogan Research Network

Hogan has a vast research network that includes both U.S. and international universities which have collaborated with Hogan on various research projects. Domestic programs include:

• Appalachian State University
• Bradley University
• Colorado Technical University
• Elmhurst College
• Georgia State University
• Harvard University
• Middle Tennessee State University
• Minnesota State University - Mankato
• Missouri State University
• Missouri University of Science and Technology
• Northern Illinois University
• Northwestern University
• Oklahoma School of Science and Mathematics
• Oklahoma State University
• University of Maryland
• University of Missouri - St. Louis
• University of Oklahoma
• University of Texas at Tyler
• University of Tulsa
• Vanderbilt University
• Washington University in St. Louis
• Western Washington University

Below is sampling of our global academic schools that have used Hogan tools in the past.

• Monash University (Australia)
• University of Kent (UK)
• University of Nottingham (UK)
• Université de Montréal (Canada)
• Université du Québec à Montréal (Canada)
• Erasmus University Rotterdam (NL)
• Saint Mary’s University (Nova Scotia)
• Soongsil University (South Korea)