Validation Study on New Nurse Testing Program Concludes That Combined Assessments Are Predictors of Job Performance

Share Article

Three-year validation study shows that the combined Prophecy Assessments (Behavioral, Clinical, and Situational) help hospitals improve nursing programs by guiding them in the selection of their nursing job applicants. The use of valid and reliable assessment tools is the key that allows hospitals to reduce costs, build an effective team, and improve patient care.

Validated, On-line Pre-Employment Healthcare Assessments

Ensuring a commitment to provide exceptional patient care.

While the effects of implementing an assessment tool with this degree of validity are dramatic and immediate, the even-greater benefit occurs over a longer term as the workforce turns over.

Between 2008 and 2010, three testing firms partnered with Saint Francis Medical Center and Frederick Memorial Hospital to conduct one of the largest and most comprehensive test validation research projects in the history of the nursing profession. Over 850 nurses were involved, including 492 from Saint Francis Medical Center and 384 from Frederick Memorial Hospital.

The study investigated the effectiveness of three different types of pre-employment tests used for hiring nurses:

  •     CLINICAL: Written tests designed to measure job knowledge relevant to specific clinical practice areas (e.g. Labor & Deliver, Medical-Surgical).
  •     SITUATIONAL: Video-based, situational judgment tests designed to measure interpersonal competence in hospital-related situations.
  •     BEHAVIORAL: Personality/behaviorally-based tests designed to measure Conscientiousness, Tough-Mindedness, Conventional, Extroversion, Stability, Teamwork, and Good Impression.

The purpose of this study was to find the degree to which the results of the Prophecy Assessments were correlated to job performance and the degree to which they could explain job performance variance in nursing professionals. Using Structural Equation Modeling techniques a set of weights was derived that could be applied in the scoring process when all three assessments are administered in hiring situations. This research revealed that the combined set of tests produces a (corrected) theoretical correlation (or validity coefficient) to job performance of .40, which explains 16% of job performance variance.

The percentage of job performance variance is determined by squaring the correlation (.40 x .40 = .16). While 16% may not seem like a significant factor, consider that the U.S. Department of Labor guidelines classify a validity coefficient value above .35 as “very beneficial” in its ability to predict job performance. For the sake of comparison, a test with a validity coefficient of .35 would be expected to predict 12.3% of variance in job performance.

Because personnel tests typically provide only limited correlations to job performance (often ranging in the .15 to .25 range for single tests), this level of validity (.40) explaining 16% of job performance (derived by squaring the .40 correlation) can be viewed as outstanding.

Practical Implications for Hospitals Using the Prophecy Assessments

Using a test that has shown significantly high correlations to job performance practically guarantees an improved workforce. The validity coefficient of a test can be used for modeling hiring scenarios that will have different outcomes with the resulting workforce. Modeling such scenarios requires making various assumptions regarding two factors, the Base Rate and the Selection Ratio.

  •     The Base Rate refers to the percentage of the applicant pool that is qualified at the desired level.
  •     The Selection Ratio refers to the percentage of nurse applicants who will be tested and subsequently hired.

The Base Rate and Selection Ratio can be combined with the validity coefficient from this study (.40) and used to model expected hiring outcomes that will emerge from using the test as a pre-employment screening device. If a hospital desired to only screen in the best possible nurses and the assumption is made that about half (50%) of the nurse applicants are minimally qualified (the base rate), and only the highest scoring 5% of those who take the Prophecy Assessments are hired (the selection ratio), 81% of the nurses hired will be expected to perform at an “above average” level as opposed to the 50% that would be expected by hiring at random (resulting in a 31% benefit).

Because of the strength of the combined Prophecy Assessment and realistic base rates and selection rates that must be used, medical facilities that use the combined set of Prophecy Assessments are likely to improve the effectiveness of their nurse hiring program by between 13% and 27% (given a base rate of 50%), and between 9% and 15% (given a base rate of 80%) when compared to using no testing program, or a selection process with invalid tests.

While the effects of implementing an assessment tool with this degree of validity are dramatic and immediate, the even-greater benefit occurs over a longer term as the workforce turns over. The consistent hiring of better-performing workers compounds, over time, like interest, resulting in a workforce that is changed for the better. The use of valid and reliable assessment tools, such as Prophecy Assessments, is the key that allows hospitals to reduce costs, build an effective team, and improve patient care.

For more information, or to request a copy of the Prophecy Validation Study Executive Summary or the entire validation study, go to http://prophecyhealth.com/Contact.

Amistaff Healthcare, Inc. (Amistaff) is a private organization that provides assessment services to the healthcare industry. One of the divisions of Amistaff is NurseTesting, which was founded in 2004 to create a standardized system for testing nurses and healthcare professionals seeking employment at healthcare facilities and staffing agencies. Amistaff exclusively provided the Clinical (written) Assessment content included in this study, whereas the Situational Assessment content through partnership with Biddle Consulting Group, Inc.

Biddle Consulting Group, Inc. (BCG) specializes in Equal Employment Opportunity (EEO) consulting, litigation support, personnel testing software development, and Affirmative Action Play (AAP) outsourcing and software. Since 1974, when known under the name Biddle & Associates, BCG has worked with thousands of employers in these areas, as well as providing litigation support as consultants or experts in over 200 state, federal, and circuit court of appeal EEO cases involving statistics and/or job-relatedness (test validity) analyses. BCG has developed and validated personnel tests in hundreds of situations that are used by thousands of employers. BCG’s role in the partnership was to oversee the validation study and jointly develop the Situational Assessments.

PeopleClues is an international test publishing firm that provides modern, validated behavioral assessments through an online platform specifically designed for the commercial market. PeopleClues Assessments are used by thousands of companies to hire, train, and promote applicants and incumbents. PeopleClues multi-faceted personality test item bank (“Clues”) including 70 personality/behaviorally based personality items and a timed, seven-minute cognitive ability test were included as part of this study.

Saint Francis Medical Center (located in Cape Girardeau, Missouri) is a 258-bed facility serving more than 650,000 people throughout Missouri, Illinois, Kentucky, Tennessee, and Arkansas.

Frederick Memorial Hospital (located in Frederick, Maryland) is a large medical facility that employees 2,700 medical professionals, including 375 doctors and several hundred nurses.

###

Share article on social media or email:

View article via:

Pdf Print

Contact Author

Melissa Spangler
Visit website