Have you ever witnessed someone trying to tell a spooky ghost story, only to be silenced by someone matter-of-factly stating that they don’t believe in ghosts? This situation has always been a somewhat annoying one to me because, while I’m personally very skeptical of the existence of ghosts, I like to hear a good ghost story nonetheless.
That is why it was both annoying and amusing to me when I was recently attending a friend’s wedding reception and, upon being asked by a new acquaintance (a distant relation of the groom, I believe) what I do for work, I told him about The Predictive Index (PI) and its Behavioral Assessment, to which he replied, “Oh, I don’t believe in those kinds of assessments.”
The statement of disbelief in workplace assessments struck me as funny because, unlike ghosts, workplace assessments such as the PI Behavioral and Cognitive Assessments are backed with scientific evidence of their validity. When it comes to ghosts, whether you believe the stories might be a matter of opinion, unless you’ve come face-to-face with one of those little buggers. And if you haven’t, there is little valid evidence to support their existence (although the producers of “Ghost Hunters” might disagree).
But personality and/or behavioral assessments aren’t ghost stories. Such assessments have stories that are solid, and they have been proven to correlate with workplace performance time and again. We know this because we’ve applied a variety of research methods, and the one we are discussing today is a specific type of research called criterion-related validity studies.
Why Conduct a Study
Imagine you own a company that sells cars. You have about a hundred salespeople across a few similar locations. Across all of these locations, you have some salespeople who sell cars at the rate that Starbucks is currently selling pumpkin spice lattes—really fast. At the same time, you have plenty of salespeople who do “well enough,” and of course there are a few who couldn’t sell hot chocolate to children on a cold winters day—in other words, they stink.
Now imagine one day you get to the dealership and you see more customers than salespeople. In fact, customers are just hanging around and eventually leaving because there are not enough salespeople there to help them. You are losing potential business! You decide that it’s time to hire some more sales employees.
Naturally, you want to find more people who are just like the top performers who sell the most cars, but replicating them may not be so easy. These top performers likely come from all kinds of backgrounds, and the only thing that they seem to have in common is that they are terrific at selling cars. You could bring in lots of people, interview them all, and spend gobs of time guessing at the things that differentiate the star salespeople from the less amazing performers. This process would take a lot of time and energy, and there’s no guarantee that you are going to get to the heart of the great sales employee story. In fact, you’d probably fail in pinpointing the magic formula for identifying top performers. Thankfully, there is an easier way to predict job performance: you could conduct a validity study.
How Validity Studies Work
You’ve been tracking the sales numbers of all your employees—the good, the bad, and the ghostly bad. You also have access to the PI Behavioral Assessment. All of your sales employees will complete the assessment, and this will tell you about their Behavioral Drives – primarily, you will know how Dominant, Extraverted, Patient, and Formal each of these employees is.
Now, here’s the fun part: researchers can combine your employees’ sales numbers with their Behavioral Assessment results to see if there is a relationship between the two. They can (and do) use all of others performance measures, but for this example, we are going to stick to sales. Using statistical analyses, researchers can determine which of the Behavioral Drives matter when it comes to selling cars, and you can use these results to hire more people with the same Behavioral drives. For most validity studies of this kind, the researchers will calculate two main things: The correlation coefficient (this value is called “r”) and the statistical significance of the correlations (this value is called “p”). R indicates the strength and direction of the relationships that exist between the study variables, and p indicates the likelihood that the correlation occurred by chance. By examining r and p, researchers can determine which variables matter the most when it comes to the performance data and, depending on the results, can make recommendations for who would be the most likely to achieve high performance in the future.
Common Complications
It’s important to consider what factors beyond Behavioral Drives may be impacting your sales data. Are zombies chasing your employees around and causing them to miss work? Are the employees accidentally selling witches’ brooms instead of cars? These things could affect your data!
More seriously though, there are all kinds of outside factors that could impact your data. For instance, if one of your car dealerships sells economy vehicles and another sells luxury vehicles, the luxury vehicles might sell less frequently but bring in more cash than the economy vehicles. Combining the sales data from the employees at both locations without accounting for the difference between the two might yield misleading results.
Perhaps you have two different managers at two different store locations, under whom certain behaviors prosper. For example, manager A is like Beetlejuice. He is quite the talker and tends to show up even when he is not needed. The more extraverted employees tend to get caught up in conversations with him while customers are around, distracting them from the customers and resulting in fewer sales. Manager B is more like Dumbledore—he stays shut up in his office most of the time, but is always willing to help those who ask for it—and the ones who do are more extraverted. Extraverted people would have lower sales on Beetlejuice’s team and higher sales on Dumbledore’s team, but if the data were combined, perhaps it would average out and it would seem that there were no interesting results at all. This is why it is important to consider outside variables when conducting validity studies. This is also why you might not want to hire people who resemble fictional characters.
The Results
The results from a validity study can take a variety of forms. Two elements that most validity studies include are descriptive statistics and correlation tables. Descriptive statistics show things like the average level of each Behavioral Factor and performance metric (like average car sales per employee per day). It also shows standard deviation, which is a way of showing the variance in the data. It’s important to include the standard deviation because this tell you if the data of your group is too similar. For instance, if everyone sold about the same number of cars in October, and you use only data from the month of October for your validity study, then your study will not have any significant findings because everyone has the same performance data, regardless of their Behavioral Drives. For this reason, it’s important to look the at descriptive statistics for each variable in a study to see if there are any factors that may limit the results.
Correlation tables are the part of a study where things get good. These tables are where the relationships between the variables (Behavioral Drives & Performance Data) are shown. Each statistically significant (i.e., most likely did not happen by accident) correlation is shown in these tables. There can be positive correlations, which mean that as one variable increases, so does the other (for example, as Dominance increases, so does sales volume) or negative correlations, which mean that as one variable increases, the other tends to decrease (for example, as Dominance increases, sales volume decreases).
These correlations are what people use to determine who is the most likely to be a top-performer. If your study found that salespeople who are more Dominant sell the most cars, then you would have all of your future applicants take the Behavioral Assessment and try to hire more Dominant salespeople. But the study results don’t have to be limited to just the four primary Behavioral Drives; the combination of these drives (called Factor Combinations) might also predict performance as well. For instance, a person who is more Dominant than they are Extraverted is a Task-Oriented person, while a person who is the opposite – more Extraverted than they are Dominant – is a people-oriented person. researchers can measure the difference between the Primary Factors (Dominance and Extraversion) and examine that data in comparison with the performance data (like sales) to see if people who are more Task-Oriented or who are more People-Oriented tend to have higher performance.
By the way, this is something that has been done in the real-world time and again. Here at PI, we’ve conducted nearly 500 validity studies that examine all kinds of performance data from a wide variety of clients across ten different industries. 127 of these studies were on sales roles—such roles are popular for validity studies because of the clear financial impact that they can have on a business, not to mention the fact that sales performance is an objective measure that is often readily available. Findings from such studies tend to vary, even within the same role, due to the fact that each company is unique in what it takes to be successful there. At PI, we do tend to see some trends between our sales studies – for instance, sales employees who are Dominant and/or Extraverted tend to have higher sales numbers, but this is by no means the rule. There are even a handful of sales roles for which studies found that lower Dominance and lower Extraversion leads to better performance.
How to Use the Results
It’s important to remember that validity studies are not the end-all solution to hiring (as much as we’d like them to be!) What if you hired only people who were Dominant because higher Dominance correlated with higher sales, and all of your uber-dominant salespeople started butting heads and arguing with each other in front of customers? This hasn’t happened in the real world (that I know of) but it’s an example of how other factors that may impact job performance (such as getting along with your peers) should always be taken into account, regardless of the study results. The best way to use the study results is as a single data point among others, such as match to a job assessment, background, experience, cognitive ability, etc.
Finally, today’s car sales example was just that—an example. While PI has conducted 11 validity studies on automotive salespeople, these studies were each conducted for a unique client and had unique findings. There are other reasons for conducting validity studies and other methods that can be used to examine and analyze the data. Beyond figuring out the right stuff for making hires, validity studies can help you to have a more legally-defensible hiring process, because you have established the job relevancy of an assessment. Also, they can be a powerful tool for getting stakeholder buy-in because you can show others how hiring the right people will impact your sales (or other important metrics).
For this post, we focused on studies that include the PI Behavioral Assessment, but PI also conducts validity studies on its Cognitive Assessment to show the relationship between cognitive ability and job performance. So, whether you use the PI Behavioral Assessment, the Cognitive Assessment, or both, rest assured that you are interacting with a tool that is backed by strong validation research.
Happy Hiring!
Commentaires