Last week we took a look at how extensive and complicated surveys can impede the recruiting process and ultimately degrade the quality of a project’s recruitment. This week we turn to algorithms to see what impact they can have on qualitative studies and if there are any possible solutions to mitigate the negative influences they cause in some instances.
As mentioned in last week’s blog, a meeting with a colleague/friend of INGATHER’s few weeks ago generated a conversation about the challenges facing facilities and recruiting departments. He indicated his company is now offering discounts to clients who let his staff write the screeners for certain projects. In addition to facility providers talking about screeners, algorithms are being discussed by large numbers of market research professionals from all corners of our professional industry. Those involved seem to take two basic positions when considering the question, “Are algorithms choking off the recruiting funnel?” The answer seems to depend upon who is responding. The advocates of algorithms say the math allows the research team (provider and/or client) a better and far fuller gateway to identify and recruit ideal, respondents for the qualitative research. These algorithms are often generated from quantitative segments in order to produce potential qualitative study subjects – unicorns. Algorithm advocates believe the math is invaluable. While the algorithm advocates extol the virtues of the tool, others have a different perspective. Some facility and recruiting providers think algorithmic math may not always be the best application in the recruiting process. Why not? Because there are so many unknown variables built into many algorithms that the best respondents may be left by the side of the numerical road. One hidden instigator is self-perception by the respondent. We have found may prospects’ opinions and responses change by the time they get into the next round of qualifying. In some studies there are four different opportunities for participants to offer their opinions: the initial pre-screen survey, the recruiting call during rescreening, a pre-focus group rescreening and the final response during the group session.
If these responses were based on objective facts, it would be one thing, and generally valid. However, in most cases respondents are offering subjective opinions that are based on attitudinal perceptions. The result: potential revisions across a full scale of possible variables leading to changing algorithmic segmentation.
“Alogrithms are much in vogue these days – maybe too much in vogue. The step-by-step procedure they employ for guaranteering a result, unfortunately, comes with no actual guarantee.” Robert Mankoff, The New Yorker
How can we improve the use of algorithms? Isn’t that what we all wish to do with our industry, improve it? What first comes to mind is, are we over qualifying? Is it possible to find the perfect (or as close as possible) respondent with a screener and algorithm? Why use both when one question in an algorithm can disqualify a great respondent without knowing which one knocked the unicorn out? How significant was that knockout question in the overall study? How weighted was that question? In addition, when you have a low incidence algorithm combined with a low incidence screener, you’re exponentially limiting who can pass through the funnel. It is possible for researchers to occasionally include so many criteria that they exclude highly qualified respondents.
- If your research requires segmentation via algorithmic computations we recommend your screener be limited to demographics as well as five to ten key questions, keeping the screener concise to increase qualified participants and mitigate respondent fatigue.
- Don’t use an algorithm. Replace it with attitudinal questions in your screener. Algorithms hide what you are seeking and can impede what you’re looking for. What’s the sense in that? Recruiters are here to assist in providing the most qualified participants. Give them the opportunity to deliver the best respondents for your study.
Everyone is on a budget and all companies are looking to control costs. Algorithms almost always require more recruiting effort. To meet algorithm requirements, facilities and their recruiting departments must devote more labor and more time to the project. The additional resources can nearly double the recruiting costs, not to mention the possibility of not being able to recruit those unicorns in time for the session.
INGATHER Research & Sensory is always looking for the best way to meet the challenges of our industry. We are always interested in hearing from you and others who want to improve the value of qualitative, sensory and quantitative research. We believe the path to doing so is built upon strategic implementation of all elements, from origination of a project through the analysis of the findings. We’re here to assist you throughout this process.
By Bob Chapin, CMO and Kelli Hammock, Project Director