Over the past decade, software developers and growth marketers have automated most qualitative user feedback and testing. And yet, what about testing with communities like patients or senior citizens who may be more challenging to reach?
It was 2:00 a.m. at the Marriott Hotel in Singapore and I just wanted to get to bed after a 16-hour flight. As co-founder of a digital health company, I was in the process of building a community of test patients. Because of security and privacy concerns, I had to approach this process unconventionally; manually recruiting prospective testers online through administered groups and forums.
One of our test users had placed two urgent calls to me. I immediately called her back.
“One of our group members needs a new doctor. She is not doing well and needs a better specialist. I know you have a doctor on staff and I know it’s not his job but…umm, but…”
I interjected immediately.
“Don’t worry. You don’t need to say anything. We’ll do everything we can.”
Immediately, I dropped everything and called our company’s Chief Medical Officer to start a referral process. For the next few days, we fired off introductions to new doctors and assisted even though these tasks were not at all related to our company’s product. We were engaging with a non-conventional community which sometimes required going above and beyond the call of duty.
In recent years, product managers have fundamentally altered and automated usability testing for new products. Employing distributed labor marketplaces like Rainforest QA, Usabilia and Juicy Studio, growth-minded product managers have accelerated UI, UX, and backend product testing to ship faster and faster.
One of the most important subsets of usability testing, qualitative user feedback, has also faced an onslaught of automation.
And yet, there are numerous organizations who operate in spaces like healthcare, politics, or even eldercare where obtaining qualitative feedback is not that easily automated. Often, these are fields where security, privacy, and other restrictions necessitate a manual recruiting strategy focused on partnership and community development. A good example of this is a digital health company looking to test the first iteration of its product with patients where protected health information may be shared. Yet another example is an application focused on First Amendment violations targeting journalists or other at-risk groups where identity disclosure may be prohibitive. One needs to look no further than recent news of Google’s Nightingale project with Ascension Health to underscore the importance of the right policies and controls in these spaces.
I’ve learned the lessons in this space first-hand. Over the past two years, I have built a user community of patients who suffer from cardiovascular disease. For no monetary compensation, they are testing our company’s digital health application because they believe in its potential to make a difference in their lives and those of others. The most remarkable and fulfilling experience of my professional career, I have learned that to test your product with non-conventional users, you have to approach the process non-conventionally as well. In the words of Y Combinator Founder Paul Graham, you are going to have to “do things that don’t scale” and not be afraid of digging right in.
Specifically, you have to look for and recruit users in unexpected places; some of which resist automated growth marketing efforts. Second, you need to understand the value of partnership as these groups tend to resist more transactional relationships. And finally, you need to ask for permission and be honest and forthright with your intentions as to the testing process and the eventual product that you hope will hit the market.
Recruit in unexpected places
If recruiting test users in a challenging space like healthcare, law and order, or even eldercare, you need to seek them out in non-traditional and unexpected places. While you may think that online discussion forums like that are centered around user testing is the first place to go, there are other channels in which you can find more engaged and eager communities.
In medicine, community platforms including Patients Like Me and Care Opinion provide a key outlet to reach potential participants in a constructive and open way. In the political space, sites like Democratic Gain and Hill Zoo act similarly. Uniquely, these platforms have built-in security and approval features that protect users’ identities and allow them to only enter into conversations with their express and full consent. This is a key consideration for sensitive groups.
Facebook Groups allows for even more long tail recruiting but with the obvious and attendant risks that recruiting on an open platform like Facebook carries. Due to the closed nature of the Groups product as well as many built-in security features, Facebook Groups has escaped many of the information integrity issues and as a result, is one of the healthy components of the platform. Start by searching for a group in your space. Proceed by asking the group’s administrator for permission to engage with members by explaining your purpose and focus honestly. Often, in areas like medicine or politics, group members are eager to participate in testing new products where they can offer feedback in real-time and make an impact.
Even more fascinating is Quora. Quora’s platform emphasizes long tail discussions on a range of topics that even Facebook Groups cannot be narrow enough to encapsulate or cover. So, if you are looking for users with an extremely narrow focus, say those who are interested in testing a mobile app for tourism in the historic center of Hvar, Croatia, Quora may be your best opportunity to connect with like-minded individuals. Quora’s discussions can go quite deep, drive substantial value, and be generative of new product features.
It’s a partnership, not a transaction
Usability testing solutions are often quite transactional in nature. Product managers offer prospective users a specific set of instructions from start to finish. Upon completion, users are rewarded with some sort of incentive. Other than for close friends and family of the founders or the management team, this is purely transactional in nature.
By contrast, engaging in qualitative testing with non-traditional communities requires entrepreneurs to treat the process of product testing as more of a partnership than a transaction.
Entrepreneurs must first understand the incentive shift that occurs here. For instance, cardiovascular or oncology patients are much more interested in testing groundbreaking and novel digital health technologies that will improve their condition(s). Similarly, members of a political action group focused on the First Amendment will be much more interested in the ability of an application to scale their message of freedom of the press, speech, and assembly than any potential monetary reward. Entrepreneurs need to tailor the value of their product and create a pitch to prospective testers than encapsulates the real end benefits of their products.
Second, entrepreneurs must be prepared for the type of feedback these communities will provide and the type of feedback they will not be able to provide. Members of non-traditional communities are rarely trained in traditional product feedback and testing cycles. And unfortunately, they may not care. The last thing many heart failure patients are worried about is font placement or cross-device usability and interoperability.
Rather, what they do care about, as stated earlier, is the ability of the product to achieve its stated objective. And when they do provide feedback, it is often delivered asynchronously through multiple channels. To engage with these communities, entrepreneurs need to embrace the power of qualitative feedback and communicate on the channels easiest for their users.
As an example, A Place for Mom, a marketplace platform for eldercare homes, did not limit its feedback to structured survey responses and set up a live, phone-call based, feedback system that automated response taking rather than user interactions. Other innovative approaches can be a chatbot that directly engages testers at a time and format that is comfortable for them, video conferencing sessions, or even in-person advocacy interviews. The goal in all of these scenarios is to make testers as comfortable as possible in providing feedback in a format that works for them.
Ask for permission, but be honest and equitable
A few months ago, I was sitting down with a former tester of mine over a cup of coffee. Lamenting how many people try to push “affiliate CBD products” in her cardiovascular health group, she immediately thanked me for doing something no one else has done: ask for permission before engaging with anyone.
“You did not just think that you could post, recruit, and get responses in our Facebook Group. You were realistic and asked for permission to be here. These are our lives.”
Silicon Valley is rife with over-ambitious entrepreneurs who do not ask for permission and follow the edict of “move fast and break things.” While this strategy may still work for general market consumer and mobile applications, for members of unconventional communities or even protected groups, it is impossible to test or let alone introduce a new product with this focus.
When recruiting testers, the first thing you must do is simple: ask for permission. Whether it is the administrator of a Facebook Group, the leader of an in-person support organization, or the head of the appropriate government or non-profit agency, make sure you ask clearly and concisely with the appropriate leader of the subject community. This process will likely involve a clear explanation of the product, the benefits it provides to the community or class of individuals, as well as any compensation, either monetary or in-kind. Depending on the community, this process may take weeks, months, or even years so do not despair if you are off to a slow start. It will be worth it when you have access to insights from real-world users.
When you ask for permission, it is also important to be forthright and honest with your intentions regarding product feedback, commercialization, and data usage. In some instances, Federal agencies like the FDA and the FTC will regulate how you can recruit, engage with, and compensate test users of products. Additionally, laws like HIPAA or even California’s new Privacy law, the CCPA, will require you to be forthright and obtain unambiguous opt-in consent, in a specific and detailed format, before the inception of any testing.
And yet, for other groups of users where there is no specific law protecting data and information rights, common sense dictates being honest and forthright with intention and use. This means a clear description of what type of data you are collecting, easy methodologies and means for the users to gain access, and on-demand destruction, if requested. Unfortunately, certain segments of users are often bombarded with testing and feedback offers from nefarious marketplace actors. Although these scams are often easy to uncover as what happened with American Consumer Panels, this makes both recruitment more challenging and the necessity of being honest and forthright more paramount. In sum, if you are honest, you will reap the benefits significantly.