Effective Use and Presentation of Social Science Evidence

This article provides an overview of social science expert evidence and its proper uses in labor and employment cases, discusses important recent rulings and developments in the area of social science expert evidence, and provides some practical tips on the effective management of social science evidence.

Successful employment litigation often requires the effective use of social science evidence. Both plaintiffs and defendants in discrimination lawsuits frequently utilize psychologists or sociologists to address questions of commonality in class cases, and questions of liability and damages in individual cases. Wage and hour litigation increasingly involves social science experts who opine on a company's compliance culture and practices or who use surveys to gather evidence on commonality issues, classification issues, or the scope of alleged violations. This article provides an overview of social science expert evidence and its proper uses in labor and employment cases, discusses important recent rulings and developments in the area of social science expert evidence, and provides some practical tips on the effective marshalling of social science evidence.1

LEVELS OF EXPERT ANALYSIS AND CONCERNS AT EACH LEVEL

Social science experts typically offer opinions at one of three levels of analysis:

1. Some experts summarize social science research to provide background information for the judge and jury as they consider the facts of a specific case but do not themselves attempt to apply the research to the facts of a particular case. This "social framework" evidence, as it is sometimes called because the general social science research provides a framework for interpreting case facts,2 may be admitted if the underlying research summarized by the expert is reliable and the court deems the background information provided by the research to be helpful to the fact finder. A common form of social framework evidence in employment cases involves expert testimony that summarizes research on racial or gender stereotypes,3 with the party offering the testimony claiming that jurors are likely to have mistaken beliefs about the operation of stereotypes.4

2. Some experts summarize social science research and then opine on whether the facts of the case are "consistent with" the research findings or opine that there are indications in the factual record that certain social scientific findings apply to the case at hand.5 That is, these experts go beyond simple social framework evidence by linking the general social science research to the specific case, but instead For instance, the plaintiffs expert in a recent sex discrimination case first summarized research on gender bias and then opined that "the allegations made in this case are highly consistent with scientific research on the form and content of sex discrimination, as well as the circumstances in which sex discrimination typically takes place."6 Experts working at this level of analysis typically conduct no independent study of the parties involved in the case, but instead base their opinions on "expert judgments" about the match between deposition testimony, company documents, and findings from social science studies, such as whether language used by agents of the defendant was consistent with the operation of stereotypes.

3. Some experts use social science findings or social science methods as the basis for case-specific descriptive or causal claims that go beyond the consistency opinions found at the second level of analysis. These experts use the same method as experts operating at the second level of analysis, but they go even farther in terms of their case-specific claims and do not limit their opinions to "consistency" judgments - they affirmatively assert that certain facts or causal relations exist within the case.7 Sometimes these case-specific opinions are formed using reliable scientific methods. For example, in a dispute over the classification of assistant managers as exempt from overtime and break requirements, an expert for the defendant conducted an observational study of the work performed by a sample of the assistant managers to establish the time they spent in exempt versus nonexempt activities.8 The expert conducted an observational study to arrive at a reliable descriptive summary of the work behaviors in question. In other cases, experts operating at this level of analysis base their opinions on nothing more than their personal judgments and conjectures. For example, an expert for the plaintiff in a recent race discrimination case opined, on the basis of her interpretations of an incomplete set of depositions, that the defendant's organizational climate was tolerant of racial harassment and that the plaintiff "was discriminated against and subjected to a hostile work environment."9

In analyzing these three types of social science evidence, we will assume that Federal Rule of Evidence (FRE) 702 and the Daubert line of cases apply regardless of the litigation phase at which expert challenges arise. The argument for full Daubert scrutiny is straightforward: The Federal Rules of Evidence apply in any proceeding not specifically excluded from the scope of the rules, and class proceedings are not specifically excluded,10 and FRE 702 contains no exceptions to the reliability requirement based on the perceived difficulty of using reliable methods, the complexity of a case, or whether the judge or jury is the fact finder. The premise behind FRE 702 - that only reliable expert evidence merits consideration by a fact finder - indicates that the rule sets an admissibility threshold and does not simply authorize weighing of the evidence by its placement somewhere on a reliability continuum. Furthermore, in Wal-Mart Stores, Inc. v. Dukes, the Supreme Court strongly suggested that full Daubert scrutiny should apply at the class certification stage.11 Nonetheless, even if the court applies some alternative to Daubert to scrutinize expert evidence, the expert's method and opinions on class certification questions should still be reviewed for reliability.12

Social Framework Opinions

Three basic questions should be asked with respect to social framework evidence and expert opinions based on research summaries:

1 . Are the research findings that the expert summarized reliable?

2. Did the expert provide a reliable summary of the research?

3. Will this research summary be helpful to the fact finder?

Are the Research Findings Reliable?

The first question is the classic "junk science" question: Is research underlying the expert's opinions the product of sound, reliable methods? Rarely will social science that has been published in peer-reviewed journals be ruled unreliable when the expert purports only to summarize that research for the jury, but some areas of social science research are so underdeveloped or contested that challenges to the reliability of the research may be justified. As a general rule, whenever only a handful of studies support an important proposition in the expert's report or the only data underlying an opinion is unpublished, it may be very beneficial to seek the data underlying the expert's opinions. In one recent case, opinions of the plaintiff's expert were based in part on unpublished data obtained from a police department that had agreed to give the expert access to confidential information for consulting purposes. After the defendant sought the underlying data, opinions based on these data were withdrawn because the expert could not produce the confidential data.13

An example of contested research often cited by plaintiff experts to support claims of racial, ethnic, sex, and age discrimination is research on unconscious or implicit bias, which is most often associated with the Implicit Association Test or the IAT.14 This research is the subject of vigorous debate within psychology, but experts citing IAT research often mischaracterize the findings from this body of work and omit important limitations on the research. Simply put, despite the assertions of expert witnesses to the contrary, there is little evidence that implicit bias as measured by the IAT is associated with discriminatory behavior.15 But the foundational problems run much deeper: An effort to audit the IAT research by requesting the original data behind more than 50 studies found widespread violations of the data-sharing norm and, for the relatively few studies for which data were shared, the audit found many incomplete and erroneously reported results and one case of admitted data fabrication.16 As will be discussed, expert reliance on IAT research is likely to become even more common following the decision in Dukes. Defendants confronting experts who rely on implicit bias research should consider challenging the reliability of the underlying research and not merely the application of that research to the case at hand.

Did the Expert Provide a Reliable Summary of the Research?

The second basic question for social framework evidence is whether the expert has provided a reliable summary of the social science research. Scientific norms govern research reviews, including the requirement that the research being reviewed is reported accurately with its complexities, contradictions, and limitations disclosed.17 Demonstrating that an expert has provided a tendentious and misleading review of the underlying research should be grounds for excluding the social framework opinion under FRE 702 because a research summary that misrepresents findings is neither reliable nor helpful. Such tendentious reviews are, unfortunately, more common than one may expect, with some experts subtly misrepresenting the research and others doing so more blatantly. An example from the latter category is found in a recent case where the rebuttal expert enumerated more than 25 clear misrepresentations and errors found in the plaintiff expert's summary of social science research on gender and racial bias.18

Attempting to analyze the research cited by an expert and then challenging the expert's portrayal of that research at a deposition is discouraged. Rarely is deposition time well spent by contesting in detail the social science studies cited by an expert. Experienced experts will ask for time to review the studies in question and will often point to additional research to support their opinions if there are legitimate questions raised about specific studies. Without the aid of an expert consultant, it is very difficult (and rarely efficient) for practicing attorneys to investigate and master the intricacies of a body of social science research, and even with the aid of such a consultant it is often difficult to obtain meaningful concessions at an expert's deposition. Typically, the more effective response to a misleading review of social science research is the submission of an expert's report in opposition, followed by a Daubert motion challenging the opposing expert's opinion.

Will the Research Summary Be Helpful to the Fact Finder?

The third basic question for social framework evidence is whether a summary of the research will be helpful to the fact finder. A number of courts have found that an expert summary of stereotyping research will not be helpful to the fact finder in circumstances where the general public understands that stereotypes exist and may influence judgments about others:

Information about and commentary on gender issues is so abundant in our society that it has become a common stereotype that women receive disparate and often unfairly discriminatory treatment in the workplace. . . . Gender stereotypes are the stuff of countless television situation comedies and are the focus of numerous media treatments on nearly a daily basis. It is unarguable that virtually all adults in our society know about gender stereotypes.19

Some courts also have ruled that social framework evidence will be misleading and unhelpful because the testimony is framed at such an abstract level of analysis that it does not provide specific guidance on the issues in the case.20 Rulings of the latter variety no doubt prompt some counsel to encourage their experts to go beyond social framework testimony and connect the general social science findings to the facts of the case.21

Case-Specific "Consistent with" Opinions

One variety of such linkage testimony comes in the form of opinions that the case facts are "consistent with" social science findings. Opinions of this variety raise the same three basic questions presented by social framework testimony, because the expert first provides a summary of social science research before seeking to apply that research to the case. If the research predicate is faulty, then the applications to the case will be as well. But these opinions that explicitly link the research to the case raise additional questions about how the expert determined that the facts of the case were supposedly consistent with social science research findings, how the expert evaluated contested evidence, and how fact finders will interpret such opinions.

General social science research cannot reliably be assumed to support any case-specific claims, regardless of whether those claims are causal or descriptive. The fit between general research findings and the particulars of a case must be established empirically, using reliable scientific methods and principles.22 The method used by experts to determine whether case facts are "consistent with" social science findings on the circumstances under which race or gender influence decisions, consists solely of examining the policies and practices in a workplace as described in depositions and other discovery materials and then making "expert judgments" about the consistency of those policies and practices with research on gender or race bias research. That approach is not a scientifically accepted or reliable method because it involves nothing more than personal, subjective judgments about the significance and meaning of the evidence rather than scientific tests or measurements.

The Supreme Court, in its recent decision in Wal-Mart v. Dukes, came very close to declaring this "read the file" method unreliable and inappropriate; however, there were other sufficient reasons to reject the plaintiffs' expert's opinions, rendering such a definitive ruling on the question of methodology unnecessary. In Dukes, the plaintiffs' expert, the sociologist Dr. William Bielby, opined that the degree of subjectivity in Wal-Mart's personnel practices, the strength and uniformity of WalMart's organizational culture, and the deficiencies of its diversity measures across locations, made Wal-Mart vulnerable to gender bias in its personnel decisions and that these factors allowed gender to contribute to supposed statistical disparities in outcomes between male and female employees.23

The basis for these opinions was Dr. Bielby's review of portions of the discovery materials in light of general social science findings on gender bias, with Dr. Bielby making subjective judgments about whether the case facts matched conditions that have been found in the research to be conducive to bias. He provided no evidence that he used objective, scientific measurements or assessment tools to reach his opinions. The Court cited a recent article by Professors Monahan, Walker, and Mitchell for the proposition that the method used by Dr. Bielby was unreliable,24 and stated that it "rejected" Dr. Bielby's "social framework analysis."25 Ultimately, however, the Court rejected Dr. Bielby's testimony on vagueness grounds: whether his method was reliable or not, he could not say with any specificity what percentage of personnel decisions were affected by gender stereotypes and thus he could not say that discrimination was a common policy across Wal-Mart stores.26

There are good scientific reasons for rejecting the testimony of experts who use this "read the file" approach,27 and the Dukes opinion provides strong support for attacks on this method as a basis for expert opinions. A typical response to this challenge is that the expert is basing his or her opinion on experience as a social scientist, but this response should be rejected.28 First, the experts have been qualified as experts because of their scientific training and research and knowledge of general social science research, not because they conducted studies in which they accepted a set of litigation documents assembled by others and then, after reading those documents, made unaided, subjective judgments about the likelihood individuals within a company were biased or had discriminated on the basis of race or sex. It is a fundamental premise of federal expert witness law, posi-Daubert, that experts must employ the same level of rigor inside courtrooms that they do in their professional work outside courtrooms.29

Second, invoking experience as the foundation for expertise does not excuse an expert from explaining in detail how she reached reliable conclusions:

While it might seem a daunting prospect for trial judges to regulate the work of experts, the task is not really so difficult. The trial judge must simply assess whether the expert came to her conclusion by employing the same methodology that the expert would employ in her professional life. This requires more of an explication by the expert than simply a statement that "I relied on my vast experience."30

After Daubert, subjective belief in the rectitude of one's subjective judgments is not sufficient for an expert's opinions to be admitted.31

Another concern with experts who issue "consistent with" opinions is that they often will have implicitly or explicitly made credibility determinations or resolved factual ambiguities in favor of their client. These experts will often cite vivid testimony or anecdotes that supposedly illustrate gender or race bias at work and then opine broadly that the materials they have reviewed are "consistent" with the conclusion that gender or race bias was at work,32 while simultaneously ignoring evidence that is not consistent with their client's theory of the case. Courts often reject expert testimony that smuggles in credibility judgments or engages in a biased review of the facts, on grounds that it intrudes on the fact finder's role and is unscientific.33 Training as a social scientist and experience conducting research on gender bias does not provide any special expertise to review evidence in a contested case and make "better" judgments about that evidence than could be made by the fact finder in the case.

The very nature of "consistent with" opinions is also problematic. By using this vague "consistency" language, the expert simultaneously avoids having to state her claims in definite, falsifiable terms while encouraging the judge or jury to assume the conclusion has a solid scientific basis, which it lacks. With this approach, if the expert can find just a handful of facts in a record that she can interpret as evidence of some phenomenon (e.g., an organizational climate tolerant of harassment), then she can opine that the evidence in the case is consistent with the presence or operation of that phenomenon - regardless of how small the ratio of supposedly "consistent" to "inconsistent" facts there may be. With this approach, the expert cannot estimate with any scientific reliability which explanations may be true or false or which explanations have more support (i.e., the evidence may simultaneously be "consistent with" social science findings that can support both the plaintiff and the defendant's theory of the case). Yet persons without knowledge of the scientific method and social science research may fail to appreciate that "consistent with" opinions are nothing more than speculative statements about what might be possible and that they have no scientific credibility.

At the deposition of an expert offering an opinion of the "consistent with" variety, counsel should press the expert to provide a measure of how such consistency was judged and explain the methodology, as well as to estimate the percentage of personnel decisions that were and were not consistent with the operation of gender, racial, or age bias. The expert then must either be candid - and admit that her method was subjective and does not allow her to estimate scientifically the prevalence of bias within a company - or alternatively, engage in speculation to provide a substantive response. If the expert chooses the second path, the examining attorney should press the expert to explain the basis for the opinion to gain an admission that nothing other than intuition, "expert judgment," or experience provides the basis for the client-friendly speculation.34 As noted above, the expert's admission in Wal-Mart v. Dukes that he could not specify the degree to which gender bias affected personnel decisions at Wal-Mart proved fatal to his testimony that gender bias adversely affected between 0.5 percent and 95 percent of the personnel decisions made across stores, and provided no proof that discrimination was a common practice across stores. As a consequence, the Supreme Court concluded that "[i]t is worlds away from 'significant proof that Wal-Mart Operated under a general policy of discrimination.'"35

Prior to Dukes, social science experts had played a key role in supplying evidence of commonality. Dr. Bielby and other social science experts who reviewed a company's practices, and then opined that these practices permitted bias to enter the decision process frequently offered testimony that served as the glue holding the class theory together. With the Court's rejection of Dr. Bielby's testimony in the Dukes case, plaintiffs can no longer rely on general and vague claims about the operation of bias and its likely presence within some unspecified portion of a company's personnel decisions. They must now come forward with specific and significant proof that discrimination on the basis of race, sex, age, or some other impermissible ground was the company's standard operating procedure.

To meet this new challenge, two changes in plaintiff strategies are likely. First, implicit bias research is likely to become a more important foundation for the opinions of plaintiffs' experts because some social scientists have been willing to testify that this body of research provides evidence of widespread implicit biases. Accordingly, these social scientists will invoke this body of research and then argue that it establishes that every company will have a high percentage of biased managers who, as a result of this bias, will engage in numerous mostly subtle acts of discrimination against women or minorities. In other words, when pressed to pick a percentage of personnel decisions between 0.5 percent and 95 percent affected by bias, these experts are likely to pick a high percentage. In fact, one implicit bias researcher, Dr. Anthony Greenwald, has already offered testimony of this kind in a race discrimination class action filed against all Iowa executive agencies in an Iowa state court. Dr. Greenwald first described general research involving the Implicit Association Test (IAT) aimed at measuring implicit racial bias and then sought to connect that research to the case at hand:

The Race IAT has been described as measuring "automatic preference" for White vs. African American. In completing the measure, respondents are asked, for one of two rapid response tasks, to press the same computer key in response to (a) words with pleasant meaning and (b) images of African American faces. More than 75% of the several million Americans who have taken this test on the Internet have discovered that they respond noticeably more slowly to that task than to the task with which this is contrasted - one that requires pressing the same computer key in response to (a) words with pleasant meaning and (b) images of European American (i.e., White) faces.

The 75% figure just mentioned is an approximation of the proportion of Americans who are implicitly race-biased. This large proportion was a surprise to the researchers who first discovered it in 1998. The large proportion suggested that (implicit) race biases were considerably more pervasive in American society than most scientists had previously suspected.

It might be of little concern that a large proportion of Americans reveal implicit race bias on the IAT, if there were no evidence that this IAT measure had a relation to discriminatory behavior. However, as noted previously, in peer-reviewed research studies, the IAT race attitude measure has consistently been found to predict racially discriminatory judgments and actions. Because the percentage of White Americans who possess implicit biases is large (on the order of 75%), it is more likely than not that discrimination resulting from implicit biases is more s ocie tally consequential than is discrimination produced by the smaller proportion (on the order of 10%) who are explicitly biased. For workplace administrators who wish to avoid discrimination in their organizations, phenomena of implicit bias pose a severe challenge. Managers must find ways to avoid discrimination that occurs in difficult-to-detect forms.

There is no reason to believe that implicit bias is any less pervasive in the State of Iowa than in other states. Race IATs taken at the IAT demonstration site (https://implicit.harvard.edu/implicit/) by 1,296 Iowans showed levels of implicit race bias that were slightly higher than the average of all other Race IATs taken at that site.36

As noted above, there are a number of problems with the underlying implicit bias research, and with attempts to link such general research to a specific case.37 When confronted with claims of this sort based on implicit bias research, defendants need to present a vigorous challenge because the mere fact that an expert's testimony was admitted in a prior case may be cited as a reason for similar testimony being admitted again regardless of whether the testimony was previously challenged or rigorously scrutinized by the prior judge. Sometimes the most persuasive argument in favor of admissibility is not the reliability of a method but rather the fact that prior courts admitted the expert evidence.38

Second, plaintiffs may ask the court to compel managers of the defendant to submit to testing for the presence of implicit bias pursuant to Federal Rule of Civil Procedure 35 (Rule 35). So far, it appears that only one plaintiff has attempted to compel employees of a defendant to take the IAT,39 but new attempts may be expected as part of an effort to obtain case-specific evidence on bias levels to respond to the claim that it is inappropriate to infer bias levels within a company from general social science research conducted with persons outside the company.

Under Rule 35, any party whose mental condition is in controversy may be compelled to submit to a mental examination by an expert.40 But to compel a party to submit to a mental examination, that party's mental state must be "in controversy," and there must be "good cause" for the examination.41 One party cannot put another party's mental state in controversy,42 and whether a mental condition is actually in controversy may be difficult to determine at times. If a defense expert disputes the underlying science and application of the science to the case by a plaintiff's expert who asserts that "implicit bias" was likely at work in a company, does this count as an affirmative denial that puts the condition into dispute? In such cases, the trial judge must make a discretionary determination of whether the "in controversy" and "good cause" requirements have been met.43

Even if an issue is deemed to be in controversy and there is good cause to compel an examination, the person to be examined must be a party to the action under Rule 35. 44 Few courts have addressed the question whether agents of a party are covered by Rule 35, but language in the leading Supreme Court case on Rule 35 suggests that agents of parties are not covered.45 However, in Beach v. Beach, the court ruled that "[o]ne who is not a party in form may be, for various purposes, a party in substance,"46 and, in Dinsel v. Pennsylvania Railroad Co., the court relied on its inherent power to order the examination of an employee of a party.47 In the only reported case in which an employment plaintiff moved under Rule 35 to compel employees of the defendant to take the IAT to support her claim, the magistrate judge denied the motion on several grounds, including that the employees of the defendant were not parties covered by Rule 35. The district court, in ruling on objections to the magistrate's order, did not specifically address this issue but ruled instead that the magistrate's ruling should stand because it was not clearly erroneous or contrary to law.48

Case-Specific Descriptive and Causal Opinions

Many experts do not stop with opinions that the facts of the case are "consistent with" the conditions found in social science studies in which racial or gender bias were found to be at work. Instead, these experts opine that certain descriptive or causal conclusions hold for a particular case, as in the following cases:

From Dr. Barbara Reskin's report for the plaintiffs in Puffer v. Allstate:49

Allstate Protection . . . has a uniform culture of paternalism across the company. The fact that almost all of Allstate Protection's (AP 's) top leaders are male reflects and heightens the impact of this culture. Its policies and practices with respect to assessing, compensating, developing, transferring, and promoting personnel in salary grades 63 and above are uniform nationwide. The primary causes of the systematic gender disparities at Allstate Protection are its use of discretion in personnel decisions affecting managers at grade 63 and higher and its failure to check the biases that discretion permits - especially ingroup favoritism and sex stereotyping - through a system of monitoring and accountability.

From Dr. Susan Fiske's report for the plaintiffs in Butler v. Home Depot:50

(X) Gender stereotyping plays a major role in Home Depot's hiring, placement, and promotion patterns. (II) Much of this stereotyping is automatic and not fully conscious at the individual level, (III) but it is convenient for individual decisionmakers, so they do not examine it. (IV) Organizations can control these effects of stereotyping, through proper information and proper motivation, (V) and organizations can reduce bias by how they structure themselves, but Home Depot does not take adequate steps to control these biased individual practices.

From Dr. Eugene Borgida's report in EEOC v. Bloomberg:51

In summary, the stereotypes about employees who are mothers and/or pregnant more likely than not influenced the perceptions, evaluations, and decisions about them at Bloomberg. The cultural and organizational context at Bloomberg more likely than not activated the gender stereotype about mothers as less competent and as less agentic and less committed to their careers. Given the subjectivity, discretion, and lack of accountability in the Bloomberg decision making process, stereotypic perceptions more likely than not influenced employment decisions about employees who are mothers and/ or pregnant.

From Professor Deborah Rhode's report in Velez v. Novartis:52

Taken as a whole, the record demonstrates a corporate culture that has tolerated and condoned pervasive gender bias. Management has been at best indifferent and at worst openly resistant to women's equal employment opportunities. Female employees have paid a substantial price and fundamental changes will be necessary to achieve an equal playing field for women.

None of these experts conducted an empirical study of the company and its managers' decisions or a statistical analysis of company records. They simply reviewed some of the depositions and discovery materials and made subjective or intuitive judgments about what might be happening in the case. In other words, opinions at this level of analysis are often based on the same unreliable, "read the file" method that experts offering "consistent with" opinions use. As such, the same challenges we discussed for "consistent with" opinions will apply to these opinions.

However, not every case-specific descriptive or causal claim is founded on unreliable methods. In some cases, experts utilize social science methods to formulate reliable case-specific opinions. For instance, the third author (Mitchell) conducted an experiment on the influence of race on personnel decisions using standard social scientific methods and employees of the defendant organization as participants.53 The experiment found no evidence of racial bias and was used by the defendant to rebut the plaintiffs' social science expert and defend its practices. And in two recent cases, the industrial-organizational psychologist, Dr. James Outtz,, conducted structured reviews of the defendant organizations to provide evidence that class certification was not appropriate. In Gutierrez v. Johnson & Johnson, Dr. Outtz conducted a review of the personnel policies and practices of various Johnson & Johnson operating companies that demonstrated that the different operating companies did not share common practices and operated in different labor markets.54 In McReynolds v. Merrill Lynch, Dr. Outtz conducted structured interviews of financial analysts to gather information on the determinants of success, and this information was used to show that factors other than race led to disparities in outcomes among financial analysts.55

Perhaps the most common form of case-specific opinions based on social science methods in employment litigation (other than statistical evidence) comes in the form of opinions based on a survey of employees or putative class members. Such survey evidence may be reliable depending on how the survey is formulated and implemented, but conducting a reliable survey in the midst of litigation requires care.

The most obvious concern with surveys is that the on-going litigation will contaminate responses or alter the behavior of those being surveyed. This problem doomed the EEOCs survey evidence in EEOCv. Dial Corp., where a researcher retained by the EEOC administered a questionnaire to assess whether a hostile work environment existed within the defendant corporation.56 Potential respondents included a number of plaintiff class members, and respondents were notified of the study's purpose but were told that their responses would be confidential.57 The defendant moved to exclude expert testimony based on the questionnaire, and the court ruled that, among other problems with the study, apparent bias in responses made the questionnaire results unreliable.58

One option for avoiding this contamination problem is to conduct the study in such a way as to conceal the fact of the study or at least the study's purpose, and if possible, to use persons who are blind to the study's purpose to administer the study.59 For instance, email experiments can be conducted where the apparent race, ethnicity, or gender of the correspondent is systematically varied and responses to requests within the emails are measured. Or, an experiment can be embedded in an observational study, where the race or sex of an interacting partner is systematically varied and the interactions are recorded unobtrusively to test for disparate treatment. Another option is to conduct the study with similarly situated persons who are not involved in the lawsuit. This approach was employed in Whitewayv. FedEx Kinko's Office and Print Services, Inc., a wage-and-hour class action covering center managers employed in California.60 Because agents of the defendant were not supposed to have contact with class members, an expert for the defendant conducted a study of the exempt and nonexempt duties performed by a sample of branch managers in other western states.61 This approach may be possible in any large organization where similarly situated teams, units, or branches can be observed or assigned to different conditions of a study.62

The use of survey evidence appears to be becoming more frequent in employment litigation, especially wage and hour litigation, making it likely that employment counsel will eventually encounter these issues. The use of survey evidence in wage and hour cases may receive a big boost, or big setback, depending on how the California Supreme Court rules in the Brinker Restaurant Corp. v. Superior Court case.63 In that case, the plaintiff proposed to survey class members to prove violations of meal, break, and off-the-clock requirements, without survey respondents/class members being subject to individualized hearings or cross-examination. The party and amicus briefs filed in the Brinker Restaurant case provide useful discussions of the issues presented by survey evidence and apposite case law.64

CONCLUSION

Social science evidence may exert considerable influence on the judge at class certification or on the jury at trial. In almost every case, the first goal should be to prevent the admission of unreliable expert evidence, with winning a battle of experts as the back-up strategy. By taking advantage of the arguments and strategies discussed, counsel can more effectively manage social science evidence and prevent the admission of unreliable and misleading expert opinions.

Footnote

NOTES

1. Excluded from this discussion are experts who present statistical evidence, even though these experts are often social scientists applying social scientific techniques, because statistical evidence raises somewhat different issues. Also, much of the discussion is framed in terms of making challenges to another parties' experts, but counsel offering their own experts should, of course, be prepared to address these challenges made on their own experts.

2. See Mitchell, G, Walker, L, Monahan, J, "Beyond Context: Social Facts as Case-Specific Evidence," Emory LJ., 60, 1109, 1116-1117 (2011).

3. See, e.g., Expert Report of Louise Marie Roth, Jenkins v. BellSouth Corp., 2007 WL 4286233 (W.D. Wash. Jan. 23, 2007).

4. See, e.g., Tyus v. Urban Search Management, 102 F. 3d 256, 263 (7th Cir. 1997) ("Social scientists in particular may be able to show that commonly accepted explanations for behavior are, when studied more closely, inaccurate. These results sometimes fly in the face of conventional wisdom. ... In this case, Dr. Tarini was prepared to testify about the way an advertising campaign sends a message to its target market and how an all- White campaign affects African- Americans. This kind of social research, which would demonstrate the way one of the most important industries in our country actually operates, would have given the jury a view of the evidence well beyond their everyday experience.").

5. Experts operating at this level of analysis often use variants on the "consistent with" language. For instance, Dr. William Bielby, who was involved in the landmark Dukes v. Wal-Mart case, has often testified that a company's policies and practices were "vulnerable" to racial or gender bias. E.g., Expert Report of William T. Bielby, Cremin v. Merrill Lynch, Case No. 96-cv-3773 (N.D. 111.); Expert Report of William T. Bielby, Satchell v. FedEx Express, Case No. 03-cv-2659 (N.D. CaL).

6. Expert Report of Peter Glick, Tuli v. Brigham & Women's Hospital, Inc., Case No. 1:07cv- 12338 (D. Mass. Sept. 22, 2008). For an example from the wage and hour domain, see Declaration of William T. Bielby, Jimenez v. Allstate Insurance Co., Case No. 10-cv-08486 (CD. CaL Aug. 8, 2011).

7. This method of deriving case-specific opinions from a subjective application of social science research to the facts of a case was popularized by Dr. William Bielby, one of the leading plaintiff's experts, and is sometimes called "social framework analysis." See Mitchell et al., supra ?. 2, at 1112. Courts and commentators sometimes confuse "social framework" evidence with "social framework analysis." As originally conceived by Professors Monahan and Walker, experts offering social framework evidence do not attempt to link the research to the case; indeed, they originally proposed that reliable social science propositions be provided to jurors in the form of jury instructions rather than expert testimony.

8. Sepulveda v. Wal-Mart Stores, Inc., 237 F.R.D. 229, 236 (CD. CaL 2006), affdinpart, rev'dinpart, 275 F. App'x 672 (9th Cir. 2008).

9. Preliminary Expert Report of Caren B. Goldberg, Bennett v. CSX Transportation, Inc., Case No. 4:10-cv-0l4l7 (D.S.C. May 25, 2011). For an example from the wage and hour domain, see the expert report of Dr. Richard Martell, who used the same method to conclude that the defendant company maintained common policies and a common culture that enabled off-the-clock work to occur. See Wren v. RGIS Inventory Specialists, 256 F.R.D. 180, 187 (N.D. CaL 2009) ("The expert testimony of Plaintiffs' Industrial Organizational Psychologist, Dr. Richard Martell, indicates that RGIS has created and maintained common corporate policies regarding compensation of RGIS auditors throughout the United States, as well as a common culture, both of which are deficient with respect to ensuring that auditors are paid for all time worked."). Dr. Martell' s report is available through PACER (Expert Report of Richard F. Martell in the matter of Wren v. RGIS Inventory Specialists, LLC, et al., Case No. 06-cv-05778 (N.D. CaL July 10, 208)).

10. See Fed. R. Evid. 101, 1101.

11. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2553-2554 (2011).

12. Less than a month after the Supreme Court issued its opinion in the Dukes case, the Eighth Circuit held that Daubert does not apply at the class certification stage because decisions on trial admissibility are not appropriate at the class certification stage, and it approved what it described as the district court's "focused Daubert analysis which scrutinized the reliability of the expert testimony in light of the criteria for class certification and the current state of the evidence." In re Zurn Pex Plumbing Products Liability Litigation, 2011 WL 2623342, at *6 (8th Cir. July 6, 2011). It is unclear how this standard departs from Daubert, but the Eighth Circuit stated that it involved a '"rigorous analysis' of the parties' claims to determine 'whether the defendant's liability to all plaintiffs may be established with common evidence.'" Id. Whether endorsing full Daubert scrutiny or not, presumably courts will examine whether the expert's opinions on class certification questions were based on a reliable methodology. See Joseph M. McLaughlin, McLaughlin on Class Actions § 3.07, at 3-76 (2d ed. 2006).

13. Of course, if the underlying data are provided, then an expert consultant should be retained to evaluate the data and determine whether they are being reported fully and faithfully by the opposing expert.

14. As discussed at www.projectimplicit.org.

15. See Oswald, F, Mitchell, G, Blanton, H, Jaccard, J, Tetlock, PE, "Reassessing the Predictive Power of the Race IAT: A New Meta-analysis of Criterion Studies," unpublished manuscript (2011); see also Mitchell, G, Tetlock, PE, "Antidiscrimination Law and the Perils of Mindreading," Ohio St. LJ., 67, 1023 (2006); Mitchell, G, Tetlock, PE, "Facts Do Matter: A Reply to Bagenstos," Hofstra L. Rev., 37, 737 (2009).

16. See Blanton, H, Jaccard, J, Klick, J, Meilers, B, Mitchell, G, Tetlock, PE, "Strong Claims and Weak Evidence: Reassessing the Predictive Validity of the IAT," /. Applied Psychol., 94, 567 (2009); Blanton, H, Mitchell, G, "Reassessing the Predictive Validity of the IAT II: Reanalysis of Heider & Skowronski (2007J, "North Am. J. Psychol, 13, 99 (2011).

17. See, e.g., Arlene Fink, Conducting Research Literature Reviews (3d ed. 2010); Christopher Winship, Introduction to Special Issue on Sociological Expert Evidence, Sociological Methods & Research (forthcoming 2011).

18. See Expert Report of Philip E. Tetlock, Holloway v. Best Buy, Inc., Case No. 05-cv5056 (N.D. CaL).

19. Ray v. Miller Meester Advertising, Inc., 664 N.W.2d 355, 365-366 (Minn. Ct. App. 2003). This passage was recently cited by the court in EEOC v. Bloomberg, 2010 WL 3466370, at *17 (S. D. N. Y. Aug. 31, 2010), as one of a number of reasons for excluding the EEOCs social science expert.

20. Id. at *18 ("Dr. Borgida's opinion focuses on factors that give rise to gender stereotyping and would serve merely to distract the jury's attention from considering the evidence as it applies to EEOCs pattern or practice claim and other causes of action alleged in the Second Amended Complaint.").

21. When social framework evidence purports to summarize how people usually act with respect to certain conditions or stimuli, in order to suggest how parties in the case likely behaved (e.g. , testimony that males tend to hold gender stereotypes to suggest that male managers in a case were likely to hold gender stereotypes), an additional challenge to the testimony may be made on grounds that it is propensity evidence proscribed by Federal Rule of Evidence 404(a). See King, AG, Amin, SS, "Social Framework Evidence as Inadmissible 'Character' Evidence, " Law & Psychol. Rev., 32, 1 (2008).

22. See, e.g., Borsboom, D, Measuring the Mind: Conceptual Issues in Contemporary Psychometrics, Cambridge, UK: Cambridge University Press, 73 (2005) ("It is known that if a model fits in a given population, that does not entail the fit of the same model for any given element from a population, or even for the majority of elements from that population." (citations omitted)).

23. Declaration of William T. Bielby, Ph.D. in Support of Plaintiffs' Motion for Class Certification, Dukes v. Wal-Mart Stores, Inc., Case No. 01-2252 (N.D. CaL 2003), available at http: //www. wa Ima rtclass. com/sta ticdata/reports/rß * html.

24. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2553 n.8 (2011).

25. Id. at 2555. Dr. Bielby described his method for linking general social science research to the case as "social framework analysis."

26. Id. at 2554 ("[E] ven if properly considered, Bielby 's testimony does nothing to advance respondents' case. '[Wlhether 0.5 percent or 95 percent of the employment decisions at Wal-Mart might be determined by stereotyped thinking' is the essential question on which respondents' theory of commonality depends. If Bielby admittedly has no answer to that question, we can safely disregard what he has to say. It is worlds away from "significant proof" that Wal-Mart "operated under a general policy of discrimination.").

27. For much more detailed discussions of the problems with this method, see Mitchell, G (2010); Monahan, J, Walker, L, Mitchell, G, "Contextual Evidence of Gender Discrimination: The Ascendance of "Social Frameworks," Va. L. Rev., 94, 1705, 1742-1748 (2008); Monahan, J, Walker, L, Mitchell, G, "The Limits of Social Framework Evidence," 8 Law, Probability & Risk, 307, 311-314 (2009).

28. Another common response is to invoke the Supreme Court's opinion in Price Waterhouse, where the testimony of Dr. Susan Fiske, a psychologist who has conducted research on gender stereotypes, was considered. However, invoking Dr. Fiske 's testimony in the Price Waterhouse case should be unavailing for multiple reasons. First, as noted in the text, the Supreme Court in Dukes strongly suggested this method was inappropriate. Second, the Supreme Court issued no majority opinion in Price Waterhouse, see Price Waterhouse v. Hopkins, 490 U.S. 228, 228 (1989), and Dr. Fiske's testimony did not provide a unifying rationale among the different opinions for purposes oí Marks analysis, see Marks v. United States, 430 U.S. 188, 193 (1977). Indeed, none of the individual opinions in Price Waterhouse endorsed Dr. Fiske's method. Third, Price Waterhouse predates the Daubert line of cases. Fourth, the plurality opinion authored by Justice Brennan notes that the defendant failed to object to the plaintiff expert's testimony at trial and that its appellate argument opposing her testimony "comes too late." Price Waterhouse, 490 U.S. at 255. Justice Brennan's opinion never expressly endorsed Dr. Fiske's method or conclusions but rather stated that "we are tempted to say that Dr. Fiske's expert testimony was merely icing on Hopkins' cake," going on to say that it took no expertise in psychology to see that the statements at issue in this case reflected stereotypes. See id. at 256. Fifth, Justices White and O'Connor concurred in the judgment but did not join the plurality's opinion, with both writing separate opinions in which neither endorsed the expert's opinions or method, see id. at 258-279, and with Justice O'Connor writing that the expert's testimony would not alone be sufficient to shift the burden of persuasion to a defendant, see id. at 277. Finally, the dissent's opinion, authored by Justice Kennedy, noted that the Court was constrained by the defendant's failure to object to the plaintiff's expert testimony and stated that "[tloday's opinions cannot be read as requiring fact-finders to credit testimony based on [the expert's] type of analysis." Id. at 294 n.5.

29. See, e.g., Paris v. Ford Motor Co., 2007 WL 4967217, at *7 (D.N.M. May 25, 2007) ('"Regardless of the specific factors at issue, the purpose of the Daubert inquiry is always the same: '[t]o make certain that an expert, whether basing testimony upon professional studies or personal experience employed in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.'" (citations omitted)).

30. Capra, DJ, "The Daubert Puzzle," Ga. L. Rev., 32, 699, 781 (1998).

31. See, e.g., In re Accutane Products Liability, 511 F. Supp. 2d 1288, 1290-1291 (M. D. FIa. June 15, 2007) ("An expert's methodology must be consistent with the 'methods and procedures of science' rather than being founded on 'subjective belief or unsupported speculation.' When an expert relies on the studies of others, he must not exceed the limitations the authors themselves place on the study. That is, he must not draw oveneaching conclusions." (citations omitted)); Estate of Groff v. Aquila, Inc., 2007 WL 4644707, at *10 (S.D. Iowa Sept. 28, 2007) Qf Daubert s admissibility standard requires Dr. Hall to do more than support his position with an educated guess based on general principles; he needs objective evidence to support his theories.").

32. See, e.g., Declaration of William T. Bielby, Ph.D. in Support of Plaintiffs' Motion for Class Certification at 9, 20-21, 29, 38, Dukes v. Wal-Mart Stores, Inc., Case No. 01-cv-2252 (N.D. CaI. 2003); Report of Professor Susan T. Fiske on Behalf of Plaintiffs at 5, 7, 8, 9, 21, Butler v. Home Depot, Inc., Civil Action No. 3:94-cv-4335 (N.D. Cai. Mar. 21, 1997).

33. See, e.g., EEOC v. Bloomberg, 2010 WL 3466370, at *16 (S.D.N.Y. Aug. 31, 2010) ("In addition to finding Dr. Borgida's method to be unreliable, the Court also finds that Dr. Borgida did not apply his social framework analysis reliably to the facts here. The Court is troubled by Dr. Borgida's decision to ignore completely what he referred to as 'disconforming' examples in his report. Specifically, Dr. Borgida engaged in credibility determinations, crediting testimony that supported his position while rejecting testimony that contradicted his opinion. ... An expert "should not be permitted to 'supplant the role of counsel in making argument at trial, and the role of the jury in interpreting the evidence.'" (citations omitted)).

34. For a recent example, see EEOC v. Bloomberg, 2010 WL 3466370, at *16-17 (S.D.N.Y. Aug. 31, 2010) ("Applying his methodology to the facts, Dr. Borgida reached the conclusion that 'stereotypic perceptions more likely than not influenced employment decisions about employees who are mothers and/or pregnant.' Dr. Borgida effectively intuited this conclusion: '[I]n knowing the research as I do, and in seeing and reading what I read and saw, and in looking at the relationship between what I expected and what I saw, I attached a more likely than not expression to ... that.' . . . Dr. Borgida was unable to determine how many decisions at Bloomberg were, as he believed, affected by gender stereotypic thinking.").

35. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2554 (2011).

36. Expert Report of Anthony G. Greenwald, Pippen v. State of Iowa, Case No. CL 107038 (Dist. Ct., Polk County, Iowa). Dr. Greenwald recently testified at the trial of this case. See Jeff Eckhoff, "Psychologist: Racial Bias Seen in State Hiring," The Record Herald, Sept. 24, 2011, available at http://indianolarecordherald.desmoinesregister.com/ article/201 10925/NEWS01/309250046/Psychologist-Racial-bias-seen-state-hiring (last visited Oct. 6, 2011).

37. In addition to objections under Federal Rule of Evidence 702 or a state law equivalent, Dr. Greenwald' s testimony also presents the question raised in note 20 above, namely, whether this testimony violates the ban on character evidence by asserting that all managers at Iowa agencies have the propensity to exhibit racial bias and engage in discrimination.

38. For example, the decision to admit Dr. Bielby's opinions by the district court in Dukes appears to have been positively influenced by prior acceptance of testimony by Dr. Bielby. See Dukes, 222 F. R. D. at 192 ("Dr. Bielby's testimony on sex stereotyping also has been admitted in prior cases in this district." (citation omitted)).

39. Palgut v. City of Colorado Springs, 2008 U.S. Dist. LEXIS 123115 (D. Colo. July 3, 2008).

40. Fed. R. Civ. P 35(a)(1).

41. Id.

42. Koch v. Cox, 489 F.3d 384, 391 (D.C. Cir. 2007).

43. Schlagenhauf v. Holder, 379 U.S. 104, 119 (1964).

44. Fed. R. Civ. P 35(a)(1) ("The court ... may order a party ..... " (emphasis added)).

45- See Schlagenhauf, 379 U.S. at 115 n.12 ("Although petitioner was an agent of [the defendant], he was himself a party to the action. He is to be distinguished from one who is not a party but is, for example, merely the agent of a party."); Kropp v. Gen. Dynamics Corp., 202 F. Supp. 207, 208 (E.D. Mich. 1962) (holding that the court lacked jurisdiction to compel a truck driver, a nonparty and agent of corporate defendant, to submit to a physical examination under Rule 35(a)).

46. 114 F.2d 479, 481 (D.C. Cir. 1940).

47. 144 F. Supp. 880, 882 (W. D. Pa. 1950.

48. Palgut, 2008 U.S. Dist. LEXIS 123115, at *12.

49. Case No. 04-05764 (N.D. 111.).

50. Case No. 94-4335 (N.D. CaL).

51. Case No. 07-08383 (S.D.N.Y.).

52. Case No. 04-09194 (S.D.N.Y.).

53. Report of Gregory Mitchell, Ph.D., Bridge water v. Northrop Grumman Ship Systems, Inc., 2007 WL 4267340 (S.D. Miss. 2007).

54. Report of Dr. James Outtz, Gutienez v. Johnson & Johnson, Case No. 01-cv-5302 (D.N.J). For the court's opinion denying class certification, in part on grounds that the companies were occupationally diverse and not following common practices, see Gutierrez v. Johnson & Johnson, 269 F.R.D. 430 (D.N.J. 2010).

55. Report of Dr. James Outtz, McReynolds v. Merrill Lynch, Case No. 05-cv-6583 (N.D. 111). For the court's opinion denying class certification, see McReynolds v. Merrill Lynch, 2010 WL 3184179 (N.D. 111.).

56. No. CIV.A. 99-C-3356, 2002 WL 31061088, at *l-3 (N.D. 111. Sept. 17, 2002).

57. Id. at *4-5.

58. Id. at *9 ("[T]he inclusion of a large number of class members in the survey appears to have strongly influenced the overall results, which further supports the defendant's position that the survey data do not reliably reflect the views or experiences of the overall population of relevant employees.").

59. See, e.g., Vita-Mix Corp. v. Basic Holding, Inc., 581 F.3d 1317, 1325 (Fed. Cir. 2009) (double-blind study of blend users with respect to their manner of use of stir stick in patent infringement case); Mario v. UPS, Inc., No. CV 03-04336DDP(RZX), 2005 WL 6197774, at *10 (CD. CaI. Mar. 1, 2005) (double-blind survey of employees regarding their duties in wage-and-hour case).

60. No. C 05-2320 SBA, 2007 WL 2408872, at *l-2 (N.D. CaI. Aug. 21, 2007), rev'd, 319 F. App'x 688 (9th Cir. 2009).

61. Id. at *8. The plaintiff challenged the study on grounds that it did not examine the activities of the actual class members (which is an external validity challenge on the basis of participants' characteristics, a topic we address below), but the court rejected this challenge: "FedEx argues, and White way does not effectively rebut, that there is no operational/functional difference between the centers in California and the centers in other western states surveyed." Id. (citation omitted).

62. See, e.g., id. at *9 ("[T]here remains no evidence^ that . . . the job duties/responsibilities of any Center Manager . . . are any different than another.").

63. Brinker Restaurant Corp. v. Superior Court, 165 CaI. App. 4th 25 (2008), review granted, S16Ó350 (Oct. 22, 2008).

64. The briefs filed in the Brinker case are collected at http://www.uclpractitioner.com/ Brinker. html (last visited Oct. 6, 2011). Survey issues receive considerable attention in the parties' main briefs and in amicus briefs submitted by the National Retail Federation, the California Employment Lawyers Association, and Rogelio Hernandez.

Effective Use and Presentation of Social Science Evidence

This article provides an overview of social science expert evidence and its proper uses in labor and employment cases, discusses important recent rulings and developments in the area of social science expert evidence, and provides some practical tips on the effective management of social science evidence.

Successful employment litigation often requires the effective use of social science evidence. Both plaintiffs and defendants in discrimination lawsuits frequently utilize psychologists or sociologists to address questions of commonality in class cases, and questions of liability and damages in individual cases. Wage and hour litigation increasingly involves social science experts who opine on a company's compliance culture and practices or who use surveys to gather evidence on commonality issues, classification issues, or the scope of alleged violations. This article provides an overview of social science expert evidence and its proper uses in labor and employment cases, discusses important recent rulings and developments in the area of social science expert evidence, and provides some practical tips on the effective marshalling of social science evidence.1

LEVELS OF EXPERT ANALYSIS AND CONCERNS AT EACH LEVEL

Social science experts typically offer opinions at one of three levels of analysis:

1. Some experts summarize social science research to provide background information for the judge and jury as they consider the facts of a specific case but do not themselves attempt to apply the research to the facts of a particular case. This "social framework" evidence, as it is sometimes called because the general social science research provides a framework for interpreting case facts,2 may be admitted if the underlying research summarized by the expert is reliable and the court deems the background information provided by the research to be helpful to the fact finder. A common form of social framework evidence in employment cases involves expert testimony that summarizes research on racial or gender stereotypes,3 with the party offering the testimony claiming that jurors are likely to have mistaken beliefs about the operation of stereotypes.4

2. Some experts summarize social science research and then opine on whether the facts of the case are "consistent with" the research findings or opine that there are indications in the factual record that certain social scientific findings apply to the case at hand.5 That is, these experts go beyond simple social framework evidence by linking the general social science research to the specific case, but instead For instance, the plaintiffs expert in a recent sex discrimination case first summarized research on gender bias and then opined that "the allegations made in this case are highly consistent with scientific research on the form and content of sex discrimination, as well as the circumstances in which sex discrimination typically takes place."6 Experts working at this level of analysis typically conduct no independent study of the parties involved in the case, but instead base their opinions on "expert judgments" about the match between deposition testimony, company documents, and findings from social science studies, such as whether language used by agents of the defendant was consistent with the operation of stereotypes.

3. Some experts use social science findings or social science methods as the basis for case-specific descriptive or causal claims that go beyond the consistency opinions found at the second level of analysis. These experts use the same method as experts operating at the second level of analysis, but they go even farther in terms of their case-specific claims and do not limit their opinions to "consistency" judgments - they affirmatively assert that certain facts or causal relations exist within the case.7 Sometimes these case-specific opinions are formed using reliable scientific methods. For example, in a dispute over the classification of assistant managers as exempt from overtime and break requirements, an expert for the defendant conducted an observational study of the work performed by a sample of the assistant managers to establish the time they spent in exempt versus nonexempt activities.8 The expert conducted an observational study to arrive at a reliable descriptive summary of the work behaviors in question. In other cases, experts operating at this level of analysis base their opinions on nothing more than their personal judgments and conjectures. For example, an expert for the plaintiff in a recent race discrimination case opined, on the basis of her interpretations of an incomplete set of depositions, that the defendant's organizational climate was tolerant of racial harassment and that the plaintiff "was discriminated against and subjected to a hostile work environment."9

In analyzing these three types of social science evidence, we will assume that Federal Rule of Evidence (FRE) 702 and the Daubert line of cases apply regardless of the litigation phase at which expert challenges arise. The argument for full Daubert scrutiny is straightforward: The Federal Rules of Evidence apply in any proceeding not specifically excluded from the scope of the rules, and class proceedings are not specifically excluded,10 and FRE 702 contains no exceptions to the reliability requirement based on the perceived difficulty of using reliable methods, the complexity of a case, or whether the judge or jury is the fact finder. The premise behind FRE 702 - that only reliable expert evidence merits consideration by a fact finder - indicates that the rule sets an admissibility threshold and does not simply authorize weighing of the evidence by its placement somewhere on a reliability continuum. Furthermore, in Wal-Mart Stores, Inc. v. Dukes, the Supreme Court strongly suggested that full Daubert scrutiny should apply at the class certification stage.11 Nonetheless, even if the court applies some alternative to Daubert to scrutinize expert evidence, the expert's method and opinions on class certification questions should still be reviewed for reliability.12

Social Framework Opinions

Three basic questions should be asked with respect to social framework evidence and expert opinions based on research summaries:

1 . Are the research findings that the expert summarized reliable?

2. Did the expert provide a reliable summary of the research?

3. Will this research summary be helpful to the fact finder?

Are the Research Findings Reliable?

The first question is the classic "junk science" question: Is research underlying the expert's opinions the product of sound, reliable methods? Rarely will social science that has been published in peer-reviewed journals be ruled unreliable when the expert purports only to summarize that research for the jury, but some areas of social science research are so underdeveloped or contested that challenges to the reliability of the research may be justified. As a general rule, whenever only a handful of studies support an important proposition in the expert's report or the only data underlying an opinion is unpublished, it may be very beneficial to seek the data underlying the expert's opinions. In one recent case, opinions of the plaintiff's expert were based in part on unpublished data obtained from a police department that had agreed to give the expert access to confidential information for consulting purposes. After the defendant sought the underlying data, opinions based on these data were withdrawn because the expert could not produce the confidential data.13

An example of contested research often cited by plaintiff experts to support claims of racial, ethnic, sex, and age discrimination is research on unconscious or implicit bias, which is most often associated with the Implicit Association Test or the IAT.14 This research is the subject of vigorous debate within psychology, but experts citing IAT research often mischaracterize the findings from this body of work and omit important limitations on the research. Simply put, despite the assertions of expert witnesses to the contrary, there is little evidence that implicit bias as measured by the IAT is associated with discriminatory behavior.15 But the foundational problems run much deeper: An effort to audit the IAT research by requesting the original data behind more than 50 studies found widespread violations of the data-sharing norm and, for the relatively few studies for which data were shared, the audit found many incomplete and erroneously reported results and one case of admitted data fabrication.16 As will be discussed, expert reliance on IAT research is likely to become even more common following the decision in Dukes. Defendants confronting experts who rely on implicit bias research should consider challenging the reliability of the underlying research and not merely the application of that research to the case at hand.

Did the Expert Provide a Reliable Summary of the Research?

The second basic question for social framework evidence is whether the expert has provided a reliable summary of the social science research. Scientific norms govern research reviews, including the requirement that the research being reviewed is reported accurately with its complexities, contradictions, and limitations disclosed.17 Demonstrating that an expert has provided a tendentious and misleading review of the underlying research should be grounds for excluding the social framework opinion under FRE 702 because a research summary that misrepresents findings is neither reliable nor helpful. Such tendentious reviews are, unfortunately, more common than one may expect, with some experts subtly misrepresenting the research and others doing so more blatantly. An example from the latter category is found in a recent case where the rebuttal expert enumerated more than 25 clear misrepresentations and errors found in the plaintiff expert's summary of social science research on gender and racial bias.18

Attempting to analyze the research cited by an expert and then challenging the expert's portrayal of that research at a deposition is discouraged. Rarely is deposition time well spent by contesting in detail the social science studies cited by an expert. Experienced experts will ask for time to review the studies in question and will often point to additional research to support their opinions if there are legitimate questions raised about specific studies. Without the aid of an expert consultant, it is very difficult (and rarely efficient) for practicing attorneys to investigate and master the intricacies of a body of social science research, and even with the aid of such a consultant it is often difficult to obtain meaningful concessions at an expert's deposition. Typically, the more effective response to a misleading review of social science research is the submission of an expert's report in opposition, followed by a Daubert motion challenging the opposing expert's opinion.

Will the Research Summary Be Helpful to the Fact Finder?

The third basic question for social framework evidence is whether a summary of the research will be helpful to the fact finder. A number of courts have found that an expert summary of stereotyping research will not be helpful to the fact finder in circumstances where the general public understands that stereotypes exist and may influence judgments about others:

Information about and commentary on gender issues is so abundant in our society that it has become a common stereotype that women receive disparate and often unfairly discriminatory treatment in the workplace. . . . Gender stereotypes are the stuff of countless television situation comedies and are the focus of numerous media treatments on nearly a daily basis. It is unarguable that virtually all adults in our society know about gender stereotypes.19

Some courts also have ruled that social framework evidence will be misleading and unhelpful because the testimony is framed at such an abstract level of analysis that it does not provide specific guidance on the issues in the case.20 Rulings of the latter variety no doubt prompt some counsel to encourage their experts to go beyond social framework testimony and connect the general social science findings to the facts of the case.21

Case-Specific "Consistent with" Opinions

One variety of such linkage testimony comes in the form of opinions that the case facts are "consistent with" social science findings. Opinions of this variety raise the same three basic questions presented by social framework testimony, because the expert first provides a summary of social science research before seeking to apply that research to the case. If the research predicate is faulty, then the applications to the case will be as well. But these opinions that explicitly link the research to the case raise additional questions about how the expert determined that the facts of the case were supposedly consistent with social science research findings, how the expert evaluated contested evidence, and how fact finders will interpret such opinions.

General social science research cannot reliably be assumed to support any case-specific claims, regardless of whether those claims are causal or descriptive. The fit between general research findings and the particulars of a case must be established empirically, using reliable scientific methods and principles.22 The method used by experts to determine whether case facts are "consistent with" social science findings on the circumstances under which race or gender influence decisions, consists solely of examining the policies and practices in a workplace as described in depositions and other discovery materials and then making "expert judgments" about the consistency of those policies and practices with research on gender or race bias research. That approach is not a scientifically accepted or reliable method because it involves nothing more than personal, subjective judgments about the significance and meaning of the evidence rather than scientific tests or measurements.

The Supreme Court, in its recent decision in Wal-Mart v. Dukes, came very close to declaring this "read the file" method unreliable and inappropriate; however, there were other sufficient reasons to reject the plaintiffs' expert's opinions, rendering such a definitive ruling on the question of methodology unnecessary. In Dukes, the plaintiffs' expert, the sociologist Dr. William Bielby, opined that the degree of subjectivity in Wal-Mart's personnel practices, the strength and uniformity of WalMart's organizational culture, and the deficiencies of its diversity measures across locations, made Wal-Mart vulnerable to gender bias in its personnel decisions and that these factors allowed gender to contribute to supposed statistical disparities in outcomes between male and female employees.23

The basis for these opinions was Dr. Bielby's review of portions of the discovery materials in light of general social science findings on gender bias, with Dr. Bielby making subjective judgments about whether the case facts matched conditions that have been found in the research to be conducive to bias. He provided no evidence that he used objective, scientific measurements or assessment tools to reach his opinions. The Court cited a recent article by Professors Monahan, Walker, and Mitchell for the proposition that the method used by Dr. Bielby was unreliable,24 and stated that it "rejected" Dr. Bielby's "social framework analysis."25 Ultimately, however, the Court rejected Dr. Bielby's testimony on vagueness grounds: whether his method was reliable or not, he could not say with any specificity what percentage of personnel decisions were affected by gender stereotypes and thus he could not say that discrimination was a common policy across Wal-Mart stores.26

There are good scientific reasons for rejecting the testimony of experts who use this "read the file" approach,27 and the Dukes opinion provides strong support for attacks on this method as a basis for expert opinions. A typical response to this challenge is that the expert is basing his or her opinion on experience as a social scientist, but this response should be rejected.28 First, the experts have been qualified as experts because of their scientific training and research and knowledge of general social science research, not because they conducted studies in which they accepted a set of litigation documents assembled by others and then, after reading those documents, made unaided, subjective judgments about the likelihood individuals within a company were biased or had discriminated on the basis of race or sex. It is a fundamental premise of federal expert witness law, posi-Daubert, that experts must employ the same level of rigor inside courtrooms that they do in their professional work outside courtrooms.29

Second, invoking experience as the foundation for expertise does not excuse an expert from explaining in detail how she reached reliable conclusions:

While it might seem a daunting prospect for trial judges to regulate the work of experts, the task is not really so difficult. The trial judge must simply assess whether the expert came to her conclusion by employing the same methodology that the expert would employ in her professional life. This requires more of an explication by the expert than simply a statement that "I relied on my vast experience."30

After Daubert, subjective belief in the rectitude of one's subjective judgments is not sufficient for an expert's opinions to be admitted.31

Another concern with experts who issue "consistent with" opinions is that they often will have implicitly or explicitly made credibility determinations or resolved factual ambiguities in favor of their client. These experts will often cite vivid testimony or anecdotes that supposedly illustrate gender or race bias at work and then opine broadly that the materials they have reviewed are "consistent" with the conclusion that gender or race bias was at work,32 while simultaneously ignoring evidence that is not consistent with their client's theory of the case. Courts often reject expert testimony that smuggles in credibility judgments or engages in a biased review of the facts, on grounds that it intrudes on the fact finder's role and is unscientific.33 Training as a social scientist and experience conducting research on gender bias does not provide any special expertise to review evidence in a contested case and make "better" judgments about that evidence than could be made by the fact finder in the case.

The very nature of "consistent with" opinions is also problematic. By using this vague "consistency" language, the expert simultaneously avoids having to state her claims in definite, falsifiable terms while encouraging the judge or jury to assume the conclusion has a solid scientific basis, which it lacks. With this approach, if the expert can find just a handful of facts in a record that she can interpret as evidence of some phenomenon (e.g., an organizational climate tolerant of harassment), then she can opine that the evidence in the case is consistent with the presence or operation of that phenomenon - regardless of how small the ratio of supposedly "consistent" to "inconsistent" facts there may be. With this approach, the expert cannot estimate with any scientific reliability which explanations may be true or false or which explanations have more support (i.e., the evidence may simultaneously be "consistent with" social science findings that can support both the plaintiff and the defendant's theory of the case). Yet persons without knowledge of the scientific method and social science research may fail to appreciate that "consistent with" opinions are nothing more than speculative statements about what might be possible and that they have no scientific credibility.

At the deposition of an expert offering an opinion of the "consistent with" variety, counsel should press the expert to provide a measure of how such consistency was judged and explain the methodology, as well as to estimate the percentage of personnel decisions that were and were not consistent with the operation of gender, racial, or age bias. The expert then must either be candid - and admit that her method was subjective and does not allow her to estimate scientifically the prevalence of bias within a company - or alternatively, engage in speculation to provide a substantive response. If the expert chooses the second path, the examining attorney should press the expert to explain the basis for the opinion to gain an admission that nothing other than intuition, "expert judgment," or experience provides the basis for the client-friendly speculation.34 As noted above, the expert's admission in Wal-Mart v. Dukes that he could not specify the degree to which gender bias affected personnel decisions at Wal-Mart proved fatal to his testimony that gender bias adversely affected between 0.5 percent and 95 percent of the personnel decisions made across stores, and provided no proof that discrimination was a common practice across stores. As a consequence, the Supreme Court concluded that "[i]t is worlds away from 'significant proof that Wal-Mart Operated under a general policy of discrimination.'"35

Prior to Dukes, social science experts had played a key role in supplying evidence of commonality. Dr. Bielby and other social science experts who reviewed a company's practices, and then opined that these practices permitted bias to enter the decision process frequently offered testimony that served as the glue holding the class theory together. With the Court's rejection of Dr. Bielby's testimony in the Dukes case, plaintiffs can no longer rely on general and vague claims about the operation of bias and its likely presence within some unspecified portion of a company's personnel decisions. They must now come forward with specific and significant proof that discrimination on the basis of race, sex, age, or some other impermissible ground was the company's standard operating procedure.

To meet this new challenge, two changes in plaintiff strategies are likely. First, implicit bias research is likely to become a more important foundation for the opinions of plaintiffs' experts because some social scientists have been willing to testify that this body of research provides evidence of widespread implicit biases. Accordingly, these social scientists will invoke this body of research and then argue that it establishes that every company will have a high percentage of biased managers who, as a result of this bias, will engage in numerous mostly subtle acts of discrimination against women or minorities. In other words, when pressed to pick a percentage of personnel decisions between 0.5 percent and 95 percent affected by bias, these experts are likely to pick a high percentage. In fact, one implicit bias researcher, Dr. Anthony Greenwald, has already offered testimony of this kind in a race discrimination class action filed against all Iowa executive agencies in an Iowa state court. Dr. Greenwald first described general research involving the Implicit Association Test (IAT) aimed at measuring implicit racial bias and then sought to connect that research to the case at hand:

The Race IAT has been described as measuring "automatic preference" for White vs. African American. In completing the measure, respondents are asked, for one of two rapid response tasks, to press the same computer key in response to (a) words with pleasant meaning and (b) images of African American faces. More than 75% of the several million Americans who have taken this test on the Internet have discovered that they respond noticeably more slowly to that task than to the task with which this is contrasted - one that requires pressing the same computer key in response to (a) words with pleasant meaning and (b) images of European American (i.e., White) faces.

The 75% figure just mentioned is an approximation of the proportion of Americans who are implicitly race-biased. This large proportion was a surprise to the researchers who first discovered it in 1998. The large proportion suggested that (implicit) race biases were considerably more pervasive in American society than most scientists had previously suspected.

It might be of little concern that a large proportion of Americans reveal implicit race bias on the IAT, if there were no evidence that this IAT measure had a relation to discriminatory behavior. However, as noted previously, in peer-reviewed research studies, the IAT race attitude measure has consistently been found to predict racially discriminatory judgments and actions. Because the percentage of White Americans who possess implicit biases is large (on the order of 75%), it is more likely than not that discrimination resulting from implicit biases is more s ocie tally consequential than is discrimination produced by the smaller proportion (on the order of 10%) who are explicitly biased. For workplace administrators who wish to avoid discrimination in their organizations, phenomena of implicit bias pose a severe challenge. Managers must find ways to avoid discrimination that occurs in difficult-to-detect forms.

There is no reason to believe that implicit bias is any less pervasive in the State of Iowa than in other states. Race IATs taken at the IAT demonstration site (https://implicit.harvard.edu/implicit/) by 1,296 Iowans showed levels of implicit race bias that were slightly higher than the average of all other Race IATs taken at that site.36

As noted above, there are a number of problems with the underlying implicit bias research, and with attempts to link such general research to a specific case.37 When confronted with claims of this sort based on implicit bias research, defendants need to present a vigorous challenge because the mere fact that an expert's testimony was admitted in a prior case may be cited as a reason for similar testimony being admitted again regardless of whether the testimony was previously challenged or rigorously scrutinized by the prior judge. Sometimes the most persuasive argument in favor of admissibility is not the reliability of a method but rather the fact that prior courts admitted the expert evidence.38

Second, plaintiffs may ask the court to compel managers of the defendant to submit to testing for the presence of implicit bias pursuant to Federal Rule of Civil Procedure 35 (Rule 35). So far, it appears that only one plaintiff has attempted to compel employees of a defendant to take the IAT,39 but new attempts may be expected as part of an effort to obtain case-specific evidence on bias levels to respond to the claim that it is inappropriate to infer bias levels within a company from general social science research conducted with persons outside the company.

Under Rule 35, any party whose mental condition is in controversy may be compelled to submit to a mental examination by an expert.40 But to compel a party to submit to a mental examination, that party's mental state must be "in controversy," and there must be "good cause" for the examination.41 One party cannot put another party's mental state in controversy,42 and whether a mental condition is actually in controversy may be difficult to determine at times. If a defense expert disputes the underlying science and application of the science to the case by a plaintiff's expert who asserts that "implicit bias" was likely at work in a company, does this count as an affirmative denial that puts the condition into dispute? In such cases, the trial judge must make a discretionary determination of whether the "in controversy" and "good cause" requirements have been met.43

Even if an issue is deemed to be in controversy and there is good cause to compel an examination, the person to be examined must be a party to the action under Rule 35. 44 Few courts have addressed the question whether agents of a party are covered by Rule 35, but language in the leading Supreme Court case on Rule 35 suggests that agents of parties are not covered.45 However, in Beach v. Beach, the court ruled that "[o]ne who is not a party in form may be, for various purposes, a party in substance,"46 and, in Dinsel v. Pennsylvania Railroad Co., the court relied on its inherent power to order the examination of an employee of a party.47 In the only reported case in which an employment plaintiff moved under Rule 35 to compel employees of the defendant to take the IAT to support her claim, the magistrate judge denied the motion on several grounds, including that the employees of the defendant were not parties covered by Rule 35. The district court, in ruling on objections to the magistrate's order, did not specifically address this issue but ruled instead that the magistrate's ruling should stand because it was not clearly erroneous or contrary to law.48

Case-Specific Descriptive and Causal Opinions

Many experts do not stop with opinions that the facts of the case are "consistent with" the conditions found in social science studies in which racial or gender bias were found to be at work. Instead, these experts opine that certain descriptive or causal conclusions hold for a particular case, as in the following cases:

From Dr. Barbara Reskin's report for the plaintiffs in Puffer v. Allstate:49

Allstate Protection . . . has a uniform culture of paternalism across the company. The fact that almost all of Allstate Protection's (AP 's) top leaders are male reflects and heightens the impact of this culture. Its policies and practices with respect to assessing, compensating, developing, transferring, and promoting personnel in salary grades 63 and above are uniform nationwide. The primary causes of the systematic gender disparities at Allstate Protection are its use of discretion in personnel decisions affecting managers at grade 63 and higher and its failure to check the biases that discretion permits - especially ingroup favoritism and sex stereotyping - through a system of monitoring and accountability.

From Dr. Susan Fiske's report for the plaintiffs in Butler v. Home Depot:50

(X) Gender stereotyping plays a major role in Home Depot's hiring, placement, and promotion patterns. (II) Much of this stereotyping is automatic and not fully conscious at the individual level, (III) but it is convenient for individual decisionmakers, so they do not examine it. (IV) Organizations can control these effects of stereotyping, through proper information and proper motivation, (V) and organizations can reduce bias by how they structure themselves, but Home Depot does not take adequate steps to control these biased individual practices.

From Dr. Eugene Borgida's report in EEOC v. Bloomberg:51

In summary, the stereotypes about employees who are mothers and/or pregnant more likely than not influenced the perceptions, evaluations, and decisions about them at Bloomberg. The cultural and organizational context at Bloomberg more likely than not activated the gender stereotype about mothers as less competent and as less agentic and less committed to their careers. Given the subjectivity, discretion, and lack of accountability in the Bloomberg decision making process, stereotypic perceptions more likely than not influenced employment decisions about employees who are mothers and/ or pregnant.

From Professor Deborah Rhode's report in Velez v. Novartis:52

Taken as a whole, the record demonstrates a corporate culture that has tolerated and condoned pervasive gender bias. Management has been at best indifferent and at worst openly resistant to women's equal employment opportunities. Female employees have paid a substantial price and fundamental changes will be necessary to achieve an equal playing field for women.

None of these experts conducted an empirical study of the company and its managers' decisions or a statistical analysis of company records. They simply reviewed some of the depositions and discovery materials and made subjective or intuitive judgments about what might be happening in the case. In other words, opinions at this level of analysis are often based on the same unreliable, "read the file" method that experts offering "consistent with" opinions use. As such, the same challenges we discussed for "consistent with" opinions will apply to these opinions.

However, not every case-specific descriptive or causal claim is founded on unreliable methods. In some cases, experts utilize social science methods to formulate reliable case-specific opinions. For instance, the third author (Mitchell) conducted an experiment on the influence of race on personnel decisions using standard social scientific methods and employees of the defendant organization as participants.53 The experiment found no evidence of racial bias and was used by the defendant to rebut the plaintiffs' social science expert and defend its practices. And in two recent cases, the industrial-organizational psychologist, Dr. James Outtz,, conducted structured reviews of the defendant organizations to provide evidence that class certification was not appropriate. In Gutierrez v. Johnson & Johnson, Dr. Outtz conducted a review of the personnel policies and practices of various Johnson & Johnson operating companies that demonstrated that the different operating companies did not share common practices and operated in different labor markets.54 In McReynolds v. Merrill Lynch, Dr. Outtz conducted structured interviews of financial analysts to gather information on the determinants of success, and this information was used to show that factors other than race led to disparities in outcomes among financial analysts.55

Perhaps the most common form of case-specific opinions based on social science methods in employment litigation (other than statistical evidence) comes in the form of opinions based on a survey of employees or putative class members. Such survey evidence may be reliable depending on how the survey is formulated and implemented, but conducting a reliable survey in the midst of litigation requires care.

The most obvious concern with surveys is that the on-going litigation will contaminate responses or alter the behavior of those being surveyed. This problem doomed the EEOCs survey evidence in EEOCv. Dial Corp., where a researcher retained by the EEOC administered a questionnaire to assess whether a hostile work environment existed within the defendant corporation.56 Potential respondents included a number of plaintiff class members, and respondents were notified of the study's purpose but were told that their responses would be confidential.57 The defendant moved to exclude expert testimony based on the questionnaire, and the court ruled that, among other problems with the study, apparent bias in responses made the questionnaire results unreliable.58

One option for avoiding this contamination problem is to conduct the study in such a way as to conceal the fact of the study or at least the study's purpose, and if possible, to use persons who are blind to the study's purpose to administer the study.59 For instance, email experiments can be conducted where the apparent race, ethnicity, or gender of the correspondent is systematically varied and responses to requests within the emails are measured. Or, an experiment can be embedded in an observational study, where the race or sex of an interacting partner is systematically varied and the interactions are recorded unobtrusively to test for disparate treatment. Another option is to conduct the study with similarly situated persons who are not involved in the lawsuit. This approach was employed in Whitewayv. FedEx Kinko's Office and Print Services, Inc., a wage-and-hour class action covering center managers employed in California.60 Because agents of the defendant were not supposed to have contact with class members, an expert for the defendant conducted a study of the exempt and nonexempt duties performed by a sample of branch managers in other western states.61 This approach may be possible in any large organization where similarly situated teams, units, or branches can be observed or assigned to different conditions of a study.62

The use of survey evidence appears to be becoming more frequent in employment litigation, especially wage and hour litigation, making it likely that employment counsel will eventually encounter these issues. The use of survey evidence in wage and hour cases may receive a big boost, or big setback, depending on how the California Supreme Court rules in the Brinker Restaurant Corp. v. Superior Court case.63 In that case, the plaintiff proposed to survey class members to prove violations of meal, break, and off-the-clock requirements, without survey respondents/class members being subject to individualized hearings or cross-examination. The party and amicus briefs filed in the Brinker Restaurant case provide useful discussions of the issues presented by survey evidence and apposite case law.64

CONCLUSION

Social science evidence may exert considerable influence on the judge at class certification or on the jury at trial. In almost every case, the first goal should be to prevent the admission of unreliable expert evidence, with winning a battle of experts as the back-up strategy. By taking advantage of the arguments and strategies discussed, counsel can more effectively manage social science evidence and prevent the admission of unreliable and misleading expert opinions.

Footnote

NOTES

1. Excluded from this discussion are experts who present statistical evidence, even though these experts are often social scientists applying social scientific techniques, because statistical evidence raises somewhat different issues. Also, much of the discussion is framed in terms of making challenges to another parties' experts, but counsel offering their own experts should, of course, be prepared to address these challenges made on their own experts.

2. See Mitchell, G, Walker, L, Monahan, J, "Beyond Context: Social Facts as Case-Specific Evidence," Emory LJ., 60, 1109, 1116-1117 (2011).

3. See, e.g., Expert Report of Louise Marie Roth, Jenkins v. BellSouth Corp., 2007 WL 4286233 (W.D. Wash. Jan. 23, 2007).

4. See, e.g., Tyus v. Urban Search Management, 102 F. 3d 256, 263 (7th Cir. 1997) ("Social scientists in particular may be able to show that commonly accepted explanations for behavior are, when studied more closely, inaccurate. These results sometimes fly in the face of conventional wisdom. ... In this case, Dr. Tarini was prepared to testify about the way an advertising campaign sends a message to its target market and how an all- White campaign affects African- Americans. This kind of social research, which would demonstrate the way one of the most important industries in our country actually operates, would have given the jury a view of the evidence well beyond their everyday experience.").

5. Experts operating at this level of analysis often use variants on the "consistent with" language. For instance, Dr. William Bielby, who was involved in the landmark Dukes v. Wal-Mart case, has often testified that a company's policies and practices were "vulnerable" to racial or gender bias. E.g., Expert Report of William T. Bielby, Cremin v. Merrill Lynch, Case No. 96-cv-3773 (N.D. 111.); Expert Report of William T. Bielby, Satchell v. FedEx Express, Case No. 03-cv-2659 (N.D. CaL).

6. Expert Report of Peter Glick, Tuli v. Brigham & Women's Hospital, Inc., Case No. 1:07cv- 12338 (D. Mass. Sept. 22, 2008). For an example from the wage and hour domain, see Declaration of William T. Bielby, Jimenez v. Allstate Insurance Co., Case No. 10-cv-08486 (CD. CaL Aug. 8, 2011).

7. This method of deriving case-specific opinions from a subjective application of social science research to the facts of a case was popularized by Dr. William Bielby, one of the leading plaintiff's experts, and is sometimes called "social framework analysis." See Mitchell et al., supra ?. 2, at 1112. Courts and commentators sometimes confuse "social framework" evidence with "social framework analysis." As originally conceived by Professors Monahan and Walker, experts offering social framework evidence do not attempt to link the research to the case; indeed, they originally proposed that reliable social science propositions be provided to jurors in the form of jury instructions rather than expert testimony.

8. Sepulveda v. Wal-Mart Stores, Inc., 237 F.R.D. 229, 236 (CD. CaL 2006), affdinpart, rev'dinpart, 275 F. App'x 672 (9th Cir. 2008).

9. Preliminary Expert Report of Caren B. Goldberg, Bennett v. CSX Transportation, Inc., Case No. 4:10-cv-0l4l7 (D.S.C. May 25, 2011). For an example from the wage and hour domain, see the expert report of Dr. Richard Martell, who used the same method to conclude that the defendant company maintained common policies and a common culture that enabled off-the-clock work to occur. See Wren v. RGIS Inventory Specialists, 256 F.R.D. 180, 187 (N.D. CaL 2009) ("The expert testimony of Plaintiffs' Industrial Organizational Psychologist, Dr. Richard Martell, indicates that RGIS has created and maintained common corporate policies regarding compensation of RGIS auditors throughout the United States, as well as a common culture, both of which are deficient with respect to ensuring that auditors are paid for all time worked."). Dr. Martell' s report is available through PACER (Expert Report of Richard F. Martell in the matter of Wren v. RGIS Inventory Specialists, LLC, et al., Case No. 06-cv-05778 (N.D. CaL July 10, 208)).

10. See Fed. R. Evid. 101, 1101.

11. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2553-2554 (2011).

12. Less than a month after the Supreme Court issued its opinion in the Dukes case, the Eighth Circuit held that Daubert does not apply at the class certification stage because decisions on trial admissibility are not appropriate at the class certification stage, and it approved what it described as the district court's "focused Daubert analysis which scrutinized the reliability of the expert testimony in light of the criteria for class certification and the current state of the evidence." In re Zurn Pex Plumbing Products Liability Litigation, 2011 WL 2623342, at *6 (8th Cir. July 6, 2011). It is unclear how this standard departs from Daubert, but the Eighth Circuit stated that it involved a '"rigorous analysis' of the parties' claims to determine 'whether the defendant's liability to all plaintiffs may be established with common evidence.'" Id. Whether endorsing full Daubert scrutiny or not, presumably courts will examine whether the expert's opinions on class certification questions were based on a reliable methodology. See Joseph M. McLaughlin, McLaughlin on Class Actions § 3.07, at 3-76 (2d ed. 2006).

13. Of course, if the underlying data are provided, then an expert consultant should be retained to evaluate the data and determine whether they are being reported fully and faithfully by the opposing expert.

14. As discussed at www.projectimplicit.org.

15. See Oswald, F, Mitchell, G, Blanton, H, Jaccard, J, Tetlock, PE, "Reassessing the Predictive Power of the Race IAT: A New Meta-analysis of Criterion Studies," unpublished manuscript (2011); see also Mitchell, G, Tetlock, PE, "Antidiscrimination Law and the Perils of Mindreading," Ohio St. LJ., 67, 1023 (2006); Mitchell, G, Tetlock, PE, "Facts Do Matter: A Reply to Bagenstos," Hofstra L. Rev., 37, 737 (2009).

16. See Blanton, H, Jaccard, J, Klick, J, Meilers, B, Mitchell, G, Tetlock, PE, "Strong Claims and Weak Evidence: Reassessing the Predictive Validity of the IAT," /. Applied Psychol., 94, 567 (2009); Blanton, H, Mitchell, G, "Reassessing the Predictive Validity of the IAT II: Reanalysis of Heider & Skowronski (2007J, "North Am. J. Psychol, 13, 99 (2011).

17. See, e.g., Arlene Fink, Conducting Research Literature Reviews (3d ed. 2010); Christopher Winship, Introduction to Special Issue on Sociological Expert Evidence, Sociological Methods & Research (forthcoming 2011).

18. See Expert Report of Philip E. Tetlock, Holloway v. Best Buy, Inc., Case No. 05-cv5056 (N.D. CaL).

19. Ray v. Miller Meester Advertising, Inc., 664 N.W.2d 355, 365-366 (Minn. Ct. App. 2003). This passage was recently cited by the court in EEOC v. Bloomberg, 2010 WL 3466370, at *17 (S. D. N. Y. Aug. 31, 2010), as one of a number of reasons for excluding the EEOCs social science expert.

20. Id. at *18 ("Dr. Borgida's opinion focuses on factors that give rise to gender stereotyping and would serve merely to distract the jury's attention from considering the evidence as it applies to EEOCs pattern or practice claim and other causes of action alleged in the Second Amended Complaint.").

21. When social framework evidence purports to summarize how people usually act with respect to certain conditions or stimuli, in order to suggest how parties in the case likely behaved (e.g. , testimony that males tend to hold gender stereotypes to suggest that male managers in a case were likely to hold gender stereotypes), an additional challenge to the testimony may be made on grounds that it is propensity evidence proscribed by Federal Rule of Evidence 404(a). See King, AG, Amin, SS, "Social Framework Evidence as Inadmissible 'Character' Evidence, " Law & Psychol. Rev., 32, 1 (2008).

22. See, e.g., Borsboom, D, Measuring the Mind: Conceptual Issues in Contemporary Psychometrics, Cambridge, UK: Cambridge University Press, 73 (2005) ("It is known that if a model fits in a given population, that does not entail the fit of the same model for any given element from a population, or even for the majority of elements from that population." (citations omitted)).

23. Declaration of William T. Bielby, Ph.D. in Support of Plaintiffs' Motion for Class Certification, Dukes v. Wal-Mart Stores, Inc., Case No. 01-2252 (N.D. CaL 2003), available at http: //www. wa Ima rtclass. com/sta ticdata/reports/rß * html.

24. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2553 n.8 (2011).

25. Id. at 2555. Dr. Bielby described his method for linking general social science research to the case as "social framework analysis."

26. Id. at 2554 ("[E] ven if properly considered, Bielby 's testimony does nothing to advance respondents' case. '[Wlhether 0.5 percent or 95 percent of the employment decisions at Wal-Mart might be determined by stereotyped thinking' is the essential question on which respondents' theory of commonality depends. If Bielby admittedly has no answer to that question, we can safely disregard what he has to say. It is worlds away from "significant proof" that Wal-Mart "operated under a general policy of discrimination.").

27. For much more detailed discussions of the problems with this method, see Mitchell, G (2010); Monahan, J, Walker, L, Mitchell, G, "Contextual Evidence of Gender Discrimination: The Ascendance of "Social Frameworks," Va. L. Rev., 94, 1705, 1742-1748 (2008); Monahan, J, Walker, L, Mitchell, G, "The Limits of Social Framework Evidence," 8 Law, Probability & Risk, 307, 311-314 (2009).

28. Another common response is to invoke the Supreme Court's opinion in Price Waterhouse, where the testimony of Dr. Susan Fiske, a psychologist who has conducted research on gender stereotypes, was considered. However, invoking Dr. Fiske 's testimony in the Price Waterhouse case should be unavailing for multiple reasons. First, as noted in the text, the Supreme Court in Dukes strongly suggested this method was inappropriate. Second, the Supreme Court issued no majority opinion in Price Waterhouse, see Price Waterhouse v. Hopkins, 490 U.S. 228, 228 (1989), and Dr. Fiske's testimony did not provide a unifying rationale among the different opinions for purposes oí Marks analysis, see Marks v. United States, 430 U.S. 188, 193 (1977). Indeed, none of the individual opinions in Price Waterhouse endorsed Dr. Fiske's method. Third, Price Waterhouse predates the Daubert line of cases. Fourth, the plurality opinion authored by Justice Brennan notes that the defendant failed to object to the plaintiff expert's testimony at trial and that its appellate argument opposing her testimony "comes too late." Price Waterhouse, 490 U.S. at 255. Justice Brennan's opinion never expressly endorsed Dr. Fiske's method or conclusions but rather stated that "we are tempted to say that Dr. Fiske's expert testimony was merely icing on Hopkins' cake," going on to say that it took no expertise in psychology to see that the statements at issue in this case reflected stereotypes. See id. at 256. Fifth, Justices White and O'Connor concurred in the judgment but did not join the plurality's opinion, with both writing separate opinions in which neither endorsed the expert's opinions or method, see id. at 258-279, and with Justice O'Connor writing that the expert's testimony would not alone be sufficient to shift the burden of persuasion to a defendant, see id. at 277. Finally, the dissent's opinion, authored by Justice Kennedy, noted that the Court was constrained by the defendant's failure to object to the plaintiff's expert testimony and stated that "[tloday's opinions cannot be read as requiring fact-finders to credit testimony based on [the expert's] type of analysis." Id. at 294 n.5.

29. See, e.g., Paris v. Ford Motor Co., 2007 WL 4967217, at *7 (D.N.M. May 25, 2007) ('"Regardless of the specific factors at issue, the purpose of the Daubert inquiry is always the same: '[t]o make certain that an expert, whether basing testimony upon professional studies or personal experience employed in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.'" (citations omitted)).

30. Capra, DJ, "The Daubert Puzzle," Ga. L. Rev., 32, 699, 781 (1998).

31. See, e.g., In re Accutane Products Liability, 511 F. Supp. 2d 1288, 1290-1291 (M. D. FIa. June 15, 2007) ("An expert's methodology must be consistent with the 'methods and procedures of science' rather than being founded on 'subjective belief or unsupported speculation.' When an expert relies on the studies of others, he must not exceed the limitations the authors themselves place on the study. That is, he must not draw oveneaching conclusions." (citations omitted)); Estate of Groff v. Aquila, Inc., 2007 WL 4644707, at *10 (S.D. Iowa Sept. 28, 2007) Qf Daubert s admissibility standard requires Dr. Hall to do more than support his position with an educated guess based on general principles; he needs objective evidence to support his theories.").

32. See, e.g., Declaration of William T. Bielby, Ph.D. in Support of Plaintiffs' Motion for Class Certification at 9, 20-21, 29, 38, Dukes v. Wal-Mart Stores, Inc., Case No. 01-cv-2252 (N.D. CaI. 2003); Report of Professor Susan T. Fiske on Behalf of Plaintiffs at 5, 7, 8, 9, 21, Butler v. Home Depot, Inc., Civil Action No. 3:94-cv-4335 (N.D. Cai. Mar. 21, 1997).

33. See, e.g., EEOC v. Bloomberg, 2010 WL 3466370, at *16 (S.D.N.Y. Aug. 31, 2010) ("In addition to finding Dr. Borgida's method to be unreliable, the Court also finds that Dr. Borgida did not apply his social framework analysis reliably to the facts here. The Court is troubled by Dr. Borgida's decision to ignore completely what he referred to as 'disconforming' examples in his report. Specifically, Dr. Borgida engaged in credibility determinations, crediting testimony that supported his position while rejecting testimony that contradicted his opinion. ... An expert "should not be permitted to 'supplant the role of counsel in making argument at trial, and the role of the jury in interpreting the evidence.'" (citations omitted)).

34. For a recent example, see EEOC v. Bloomberg, 2010 WL 3466370, at *16-17 (S.D.N.Y. Aug. 31, 2010) ("Applying his methodology to the facts, Dr. Borgida reached the conclusion that 'stereotypic perceptions more likely than not influenced employment decisions about employees who are mothers and/or pregnant.' Dr. Borgida effectively intuited this conclusion: '[I]n knowing the research as I do, and in seeing and reading what I read and saw, and in looking at the relationship between what I expected and what I saw, I attached a more likely than not expression to ... that.' . . . Dr. Borgida was unable to determine how many decisions at Bloomberg were, as he believed, affected by gender stereotypic thinking.").

35. Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541, 2554 (2011).

36. Expert Report of Anthony G. Greenwald, Pippen v. State of Iowa, Case No. CL 107038 (Dist. Ct., Polk County, Iowa). Dr. Greenwald recently testified at the trial of this case. See Jeff Eckhoff, "Psychologist: Racial Bias Seen in State Hiring," The Record Herald, Sept. 24, 2011, available at http://indianolarecordherald.desmoinesregister.com/ article/201 10925/NEWS01/309250046/Psychologist-Racial-bias-seen-state-hiring (last visited Oct. 6, 2011).

37. In addition to objections under Federal Rule of Evidence 702 or a state law equivalent, Dr. Greenwald' s testimony also presents the question raised in note 20 above, namely, whether this testimony violates the ban on character evidence by asserting that all managers at Iowa agencies have the propensity to exhibit racial bias and engage in discrimination.

38. For example, the decision to admit Dr. Bielby's opinions by the district court in Dukes appears to have been positively influenced by prior acceptance of testimony by Dr. Bielby. See Dukes, 222 F. R. D. at 192 ("Dr. Bielby's testimony on sex stereotyping also has been admitted in prior cases in this district." (citation omitted)).

39. Palgut v. City of Colorado Springs, 2008 U.S. Dist. LEXIS 123115 (D. Colo. July 3, 2008).

40. Fed. R. Civ. P 35(a)(1).

41. Id.

42. Koch v. Cox, 489 F.3d 384, 391 (D.C. Cir. 2007).

43. Schlagenhauf v. Holder, 379 U.S. 104, 119 (1964).

44. Fed. R. Civ. P 35(a)(1) ("The court ... may order a party ..... " (emphasis added)).

45- See Schlagenhauf, 379 U.S. at 115 n.12 ("Although petitioner was an agent of [the defendant], he was himself a party to the action. He is to be distinguished from one who is not a party but is, for example, merely the agent of a party."); Kropp v. Gen. Dynamics Corp., 202 F. Supp. 207, 208 (E.D. Mich. 1962) (holding that the court lacked jurisdiction to compel a truck driver, a nonparty and agent of corporate defendant, to submit to a physical examination under Rule 35(a)).

46. 114 F.2d 479, 481 (D.C. Cir. 1940).

47. 144 F. Supp. 880, 882 (W. D. Pa. 1950.

48. Palgut, 2008 U.S. Dist. LEXIS 123115, at *12.

49. Case No. 04-05764 (N.D. 111.).

50. Case No. 94-4335 (N.D. CaL).

51. Case No. 07-08383 (S.D.N.Y.).

52. Case No. 04-09194 (S.D.N.Y.).

53. Report of Gregory Mitchell, Ph.D., Bridge water v. Northrop Grumman Ship Systems, Inc., 2007 WL 4267340 (S.D. Miss. 2007).

54. Report of Dr. James Outtz, Gutienez v. Johnson & Johnson, Case No. 01-cv-5302 (D.N.J). For the court's opinion denying class certification, in part on grounds that the companies were occupationally diverse and not following common practices, see Gutierrez v. Johnson & Johnson, 269 F.R.D. 430 (D.N.J. 2010).

55. Report of Dr. James Outtz, McReynolds v. Merrill Lynch, Case No. 05-cv-6583 (N.D. 111). For the court's opinion denying class certification, see McReynolds v. Merrill Lynch, 2010 WL 3184179 (N.D. 111.).

56. No. CIV.A. 99-C-3356, 2002 WL 31061088, at *l-3 (N.D. 111. Sept. 17, 2002).

57. Id. at *4-5.

58. Id. at *9 ("[T]he inclusion of a large number of class members in the survey appears to have strongly influenced the overall results, which further supports the defendant's position that the survey data do not reliably reflect the views or experiences of the overall population of relevant employees.").

59. See, e.g., Vita-Mix Corp. v. Basic Holding, Inc., 581 F.3d 1317, 1325 (Fed. Cir. 2009) (double-blind study of blend users with respect to their manner of use of stir stick in patent infringement case); Mario v. UPS, Inc., No. CV 03-04336DDP(RZX), 2005 WL 6197774, at *10 (CD. CaI. Mar. 1, 2005) (double-blind survey of employees regarding their duties in wage-and-hour case).

60. No. C 05-2320 SBA, 2007 WL 2408872, at *l-2 (N.D. CaI. Aug. 21, 2007), rev'd, 319 F. App'x 688 (9th Cir. 2009).

61. Id. at *8. The plaintiff challenged the study on grounds that it did not examine the activities of the actual class members (which is an external validity challenge on the basis of participants' characteristics, a topic we address below), but the court rejected this challenge: "FedEx argues, and White way does not effectively rebut, that there is no operational/functional difference between the centers in California and the centers in other western states surveyed." Id. (citation omitted).

62. See, e.g., id. at *9 ("[T]here remains no evidence^ that . . . the job duties/responsibilities of any Center Manager . . . are any different than another.").

63. Brinker Restaurant Corp. v. Superior Court, 165 CaI. App. 4th 25 (2008), review granted, S16Ó350 (Oct. 22, 2008).

64. The briefs filed in the Brinker case are collected at http://www.uclpractitioner.com/ Brinker. html (last visited Oct. 6, 2011). Survey issues receive considerable attention in the parties' main briefs and in amicus briefs submitted by the National Retail Federation, the California Employment Lawyers Association, and Rogelio Hernandez.