In this mini-course so far, we have noted that qualitative research can be cool and insightful, learning can be more fun, and there are tons of things that you might want to know and you want your students to know.
Today, we will consider how to frame questions to get the most valid possible information in interviews. You may want to assign students to read this blog post to help them do the best interviews they can.
Of course, the first thing you need to do is decide what you want to know such as answering questions identified in the last post.
When planning questions, it’s important to consider subjects’ mental processes in interpreting and answering questions, and recognize common cognitive limitations. Having people interview subjects can reduce some of the biases and produce more valid accounts than purely first-person self-reports.
Let’s begin considering how someone might respond to questions about a particular case. People generally remember recent events better than events that took place a long time ago. Also, people generally remember unusual events better than routine events. So to get a realistic assessment of a typical case, you might have students interview subjects about the most recent case within the parameters of your assignment (e.g., involving a family dispute or one with serious problems of distrust).
On the other hand, you may want to ask about unusual (“deviant”) cases as a way to distinguish typical and atypical cases. These might be particularly successful, unsuccessful, or surprising cases. When asking about deviant cases, it is important to probe how and why a case was different from more typical cases.
Problems with Asking About Multiple Cases
It may be tempting to ask subjects about multiple cases but these questions can be hard for people to answer well. To do so, subjects need to scan their memories to identify all the cases that fit in the applicable category and then consider which cases have the feature in question. For example, if you ask about cases within the last year in which parties had unrealistic expectations, subjects need to think about all their cases that year and then identify those cases in which parties had unrealistic expectations. This is a very difficult cognitive task which can take more time than normally expected, and so subjects are likely to guess.
It also may be tempting to ask about general strategies or important factors for a set of cases, but this is also hard to answer properly. Subjects may give what they believe are accurate responses, but one should view them with healthy skepticism because of the numerous cognitive biases described in this post.
Some subjects will respond with the old standbys – “it depends” or they decide on a “case-by-case basis.” These may be accurate but generally are not very helpful for interviewers. I sometime try to finesse this problem by asking about frequencies in broad categories, such as a five-point scale where 1 means virtually never, 3 means about half the time, and 5 means virtually always. Subjects may be able to go through the cognitive process of identifying and categorizing cases well enough to fit in these response options, though the answers obviously are very general.
Sometimes people ask subjects to rank a series of factors in terms of their importance or other criteria. This generally is a very difficult cognitive task, especially if subjects are asked to assess multiple cases and there are a lot of factors to rank. Generally, it is better to ask people to discuss each factor separately and then compare the responses. Even so, it can be hard for subjects to make good generalizations, which may reflect their beliefs or aspirations but not necessarily their behaviors.
One example is that practitioners may use ADR techniques less frequently than they say they do or aspire to do, as Milton Heumann and Jonathan Hyman found in their study, Negotiation Methods and Litigation Settlement Methods in New Jersey: “You Can’t Always Get What You Want.” For example, lawyers in that study reported that 61% said that problem-solving negotiation should be used more frequently. On average, however, subjects said that they used a problem-solving process in up to 33% of their civil cases, though the researchers “seldom” heard “stories about the interests of the parties” when observing actual settlement negotiations.
Getting Good Information About Specific Cases and Events
People generally have an easier time recalling specific cases and events than making valid generalizations.
Even when asking about specific cases, however, people may have a hard time providing accurate information because of numerous cognitive biases such as the following. Memory can fade quickly, so people can forget or mistake particular details. Hindsight bias can distort memories so that they “remember” things to fit into a neat narrative as if the events were inevitable. Social desirability bias reflects people’s desire to appear appropriate and may lead them to distort accounts that would be unflattering. Reactive devaluation, where people discount ideas advanced by their adversaries, is particularly common in conflict situations.
There is a special risk of bias for students conducting interviews. There is likely to be a great temptation to simply “find” that a case fits the concepts in the course. Our theoretical concepts can be lenses helping us to understand things more clearly. But they also can be blinders focusing attention only on what we expect to see and causing us to ignore things that don’t fit the theory.
So you should encourage students not to assume that the case neatly fits the theory. Comparing theory with actual practice is a major purpose of the assignment. So you should tell students to particularly look for anything that seems to deviate from theory – as well as to analyze things that seem to fit. In other words, they should try to be truly open-minded about whether the subjects’ accounts fit the theory or not.
Good Interviewing Techniques
Below are some useful techniques of interviewing to counteract such biases and get the best possible information. In addition, be sure to assign students to read the guidance about conducting and summarizing interviews.
It helps to put subjects at ease by using good active listening techniques. Interviewers should convey empathetic understanding, demonstrating verbally and non-verbally that they are trying to see the world as if they were looking through the subjects’ eyes. Interviewers should periodically check to confirm if their understanding is accurate. Interviewers should not express opinions about the case, even agreement with the subjects’ perspective.
Generally, it is best to start with open questions and avoid leading questions that may bias subjects to agree with the implication of the questions. Interviewers should ask follow-up questions to get more detail about relevant issues and elicit other possible perspectives about the events. Interviewers can probe by asking about the perspectives of others involved in the case or mentioning plausible interpretations and asking if they are accurate. When asking such questions, interviewers should demonstrate genuine curiosity rather than seem to challenge the subjects’ account. Obviously, responses about others’ perspectives may be biased and interviewers should consider this possibility in evaluating the responses.
In asking about a case, it helps to develop a chronology, starting from the beginning and following up with questions about each succeeding event: “what happened after that”?
I generally recommend getting accounts of full cases because I think that this is likely to produce the best understanding of cases, including contextual factors that may be very significant. For example, some important contextual factors could be the history of interactions, relationships between various people involved, and power dynamics, among many others.
Considering the small risk that interviewers could be subpoenaed about a complete case, we noted an option to reduce this risk by asking only about particular parts of cases illustrating certain issues. Even if one asks only about specific parts of cases, it is important to ask appropriate follow-up questions to get the most complete and accurate possible accounts.
Interviewers should not necessarily take subjects’ statements at face value, so after completing their interviews, students should evaluate various statements. Do particular statements seem plausible and persuasive? Are they internally consistent or contradictory? Are there reasons why statements may not be fully accurate? Are there alternative explanations for some events than the subjects’ accounts? Interviewers should include assessments of such issues in their reports.
