“What’s the most important ingredient for a perfect user research study, Watson?”, the little inner Sherlock voice asks.
“Since when do you care about user research, Sherlock?”
“Because gathering true stories from the users helps us arrive at insights to devise the right strategy for the product/business. Now answer the question, Watson.”
“Well isn’t it obvious? Any strategy fails if you don’t know the people you’re making it for. So, the right participants are the most important for any project's success. Hold on, do we have a case?”
"Yes, we seem to have a case of ‘The Imposter Participant!'"
The experiences encountered by our team of researchers sometimes look like a new Sherlock story. Participants determine the impact of any research and it is important not to lose sight of this fact. Without them, the research can be a complete waste of time and resources. Our due diligence ensures that we have the right people. However, this process is not always straightforward, and we face unexpected challenges along the way.
Recently, we encountered one such challenge that required us to don detective hats in our user research process. The project involved conducting remote IDIs with users from tier-1 cities across India. Just like any user research project, we followed the standard protocols of creating and sharing the screeners for participant recruitment for the multiple user segments.
The project began as scheduled, we conducted a couple of IDIs with the users. However, towards the middle of the IDIs phase, the team was holding a routine sync-up session to consolidate the key learnings and observations and identify any patterns and behaviors so far, which is pretty much a routine activity in any user research project at UCC. As the team delved deeper into the participants' stories, reviewing and consolidating the stories from the IDIs, certain discrepancies started to arise with a few participants. Their profiles and responses seemed inconsistent at a deeper level. It's important to note that we had already completed approximately 60% of the IDIs for the study. However, in order to address our initial doubts, we decided to engage in additional investigative efforts to gain clarity. Well, you just can't ignore a researcher's hunch, can you? (And tbh, we’re glad we followed it)
As a practice in our research, we ask the participant recruitment teams to share the participant IDs and employment proofs for us to verify the genuineness of the participants (we maintain the utmost confidentiality in collecting the individuals’ personal IDs to make sure their privacy is not compromised). Well, we had to start somewhere, so why not with the first red flag- mismatching accent? We rolled up our sleeves, pushed our glasses into place, and started examining the IDs closely.
We began with a participant who had a fairly smooth Hindi accent, which was highly unlikely for someone who’s native to a non-Hindi speaking region, and et voilà! An investigative eye helped us identify the mismatch in the font size, colors of the DOB, and the rest of the text in the IDs (in this case- the Aadhar card).
This discovery led us down a rabbit hole of investigation. We were shocked and frustrated by these findings. How could we trust the stories and feedback we gathered via these IDIs when our participants were not who they claimed to be? We realized that it was necessary for us to take matters into our own hands in order to uncover the truth.
We began with verifying the IDs of all the participants from the Indian government portals to eliminate any doubts and approve genuine profiles. As an additional element in our Sherlock toolkit, we resorted to LinkedIn to cross-verify the professional and educational information of the participants, and to our surprise, we found a whole different story there. On deeper investigation, we found that the participant was not even remotely close to what they claimed to be a classic case of an imposter participant. Maybe we should have flagged this participant during the IDI as they sounded professionally well-versed, with excellent communication skills which is unlikely for the segment that we met. Sigh!
However, as researchers, it is crucial for us to enter each discussion with an open mind and curiosity to delve into people’s stories. Being skeptical can hinder that opportunity. Thus flagging the participant too early in the conversation wasn't our approach as we saw it as an opportunity to gain a unique perspective and learn more about their journey.
As we kept pursuing this further, we uncovered different layers of misinformation. Now as researchers we are habitual to throwing ourselves into the projects that we do with the highest level of commitment, it was disheartening to see this level of absurdity being played out so casually by a (then) trusted recruitment partner.
Another problem we identified was forged employment IDs. Some of the employment IDs shared with us were 100% fake while some others were poorly doctored (Poorly because- Hello, have you heard of Photoshop? Works better than Windows paint). Also as part of the study, we asked the participant to share their phone screen during the remote IDIs and noticed that they were part of multiple WhatsApp groups named after different areas of Mumbai, despite claiming to belong to a region miles away. From doctored employment and personal IDs to deceiving participant stories, we found ourselves in a pile of mess that we had to clear up and move forward.
We confronted the recruitment partner team with these discrepancies and quickly came up with ideas to overcome this challenge. We reached out to multiple sources and found the right set of participants for completing the study, and based on the learnings from our experience, deployed new tricks and hacks to authenticate the participants.
Once we had all the necessary information, we proactively approached our client and shared the incident with complete honesty and integrity. Their response was positive and encouraging, as they not only appreciated our vigilance in identifying the discrepancies but also agreed to redo all the IDIs with a fresh set of participants. They were happy and appreciative of our attitude that we did not simply chase the number to finish the study but were determined to curate genuine true stories that will provide them the actionable insights they require to build user-centric products/designs.
This collaborative decision reiterated that transparency and trust are paramount, further strengthening the foundation of our partnership. It served as a powerful testament to the profound impact that a culture of integrity and trust can have on achieving business success.
This experience reaffirmed our belief in the importance of trust and transparency among teams and partners involved in research. As the entire research ecosystem consists of multiple partners working in sync and depending on each other, trust and integrity become fundamental necessities for the ecosystem to thrive. Without these essential elements, the quality of the research may be compromised, leading to skewed research outcomes.
It's not enough to simply recruit participants and assume they are who they claim to be. We need to be vigilant and take steps to verify their identities and backgrounds to ensure the insights we gather are accurate and actionable. So, if you ever find yourself in a similar situation, don't be afraid to channel your inner Sherlock Holmes and do some investigating of your own. The truth is out there - it's up to us to uncover it.
Reflections from the UCC Research team - Nabila Ahmed and Reena Sharma
Comments