4: Research Tools
Contents
4: Research Tools#
A survey (har har!) of methods
Sampling methods
Census
Probability sample
Convenience sample
Measurement methods
Surveys
((62d9d01a-50e2-4b12-877a-a99b2eb04946))
Sam Savage, in “The Flaw of Averages, does a nice job of pointing out how most of us crave a single number – an average – that promises insight or security, or makes a difficult decision for us (ease). A certain approach to survey design makes it easy to generate such averages, with an accompanying massive loss in fidelity and resolution.
Realtime interview
Self-administered
Interviews
Semi-Structured
Direct observation
Merely collecting and organizing extant data (meta-measurement)
Analysis of extant quant/qual data
Different from the previous because you don’t collect the data yourself.
Recruitment
Reasons people will participate
Goodwill/Interest (they want to)
Desire to “be heard”
They’re forced to
They’re externally incentivized (gift card, etc.)
Some surveys are institutionalized to the extent that recruitment is pretty easy (ex: “Best of Taos”)
Recruitment methods
Outbound
Email
Social/LInkedIn
Can be 1:1 or 1:many
Gabby’s “lead with a low-effort survey” then following up with direct outreach
Paid advertising
Appeal from a platform (an email list, an institutional partner’s platform, etc.)
These are convenience sampling methods, though you could apply some limited statistical correction for bias
Inbound
The survey itself isn’t something anybody would seek out, but…
You can inline it with existing inbound marketing-ish stuff like popular+relevant articles, email list content, and so on.
These are also convenience sampling methods, of course
Outsourced
Pay-per-completion research companies can recruit for you
These companies may have a relevant panel they can put the survey in front of. If they don’t they owe it to you to be honest, but since the model usually is pay-per-completion, they should be honest. :)
I don’t have any particular negative experience with these companies, but there’s a lot of garbage out there, so be careful, and we generally are doing research on groups we should be present with anyway, so involving a third part intermediary is often unnecessary. That’s why you’re probably getting the vibe that I’m a bit down on using outsourced recruitment.
You probably don’t need a… probability sample :) … but if you did these options would become more attractive. But again, you probably don’t need a probability sample. You just contextualize your findings within the less rigorous convenience sampling methods we tend to use with TEI.
For sake of completeness:
We’d probably never use these because they require insider status within a semi-closed system, but for completeness…
Intercept
Census
Convenience
About survey questions
Screener questions
Closed questions
Likert scale/rating
Yes/No, or other forced binary choice
Ranking
Open questions
One word/number
Multi-word/sentence/paragraph response
Demographics
Avoiding bias
Difficult to do, but worth trying!
Outside eyes on your questions can help
100% of questions delivered every time or conditionals/branching
@TODO: I need to speak in some useful way to the “toy” set of methods. I’m not sure what to say right now, but I don’t want to skip over this.
Mixing methods
Quant/Qual
Surveys can do both (big open-ended questions with paragraph-length answers), but with limitations in terms of qual richness/nuance/interpretation
Inductive/Deductive
About interviews
Range from Indi Young-style listening sessions to facilitated surveys
the latter is technically different than an interview, but still there are “interviews” that are almost as prescribed and rote as a survey
Agenda and prompts
I think this might be the biggest determinant of how an interview unfolds: how detailed is your agenda?
Secondarily: how do you prompt interviewees? Just the question? Do you cave under the pressure of silence and start leading them inappropriately?
Setting and context
Also important is setting and context. How much time is available? How squeezed is the interviewee? How squeezed are YOU?
There’s not one “best” approach to interviews.
Even with lots of mistakes, you’ll get nuanced, useful information that helps you build context and add shades of meaning and nuance to quant data.
It’s hard to recommend a starting point RE: length because uber-short interviews require more skill while longer interviews require stamina and experience.
That said, consider asking for 40m interviews, which are less than an hour by a meaningful amount (good for busy people) and long enough for you to screw up and still walk away with something useful.
Record or take notes during or after?
Potential chilling effect of recording vs. potential distraction of notetaking.
I don’t have a recipe for you here. Depnds on your preferences and strengths.
Neither drawback (the chilling effect or the distraction) is so bad you must avoid it no matter what, and avoiding your personal weakness might more than compensate for the weakness of the choice you’ve made.
Do enough interviews and you will eventually fail spectacularly. Don’t worry; it’s all in the game, yo. These extreme outliers teach you very little that you can use to improve your technique, so you just dust yourself off and move on.
Analysis
The “data soak”
Coding
Pulling out vivid data points either of a qual or quant nature
Seeking patterns
Correlations
Instructive outliers
Another “data soak” to absorb the vibe of the data/patterns/outliers/”odors” that feel interesting or unusual or unexpected
In other words, TIME can really help you notice these patterns
Some recommendations
Most questions are likely to benefit from mixed methods with a convenience opt-in sampling method
Don’t rule out other approaches, just know that within TEI (again, you might think of this research project as a prototype or first sprint) the mixed method + convenience sampling tends to be most useful and applicable.
Surveys
Semi-structured interviews/listening sessions
Measuring through data-scraping
Digital toys
Pattern libraries (not a method per-se, but if the output is going to be a pattern library, then the method will follow suit, and because the pattern library is such a great starting point SSR output, I should mention it here.)