California Bar Examination Studies

Reports

Background

In keeping with a strategic goal to ensure that its admission system is timely and fair, the State Bar of California conducted a comprehensive series of studies of the California Bar Exam. The work was prompted by several factors, including a multiyear decline in pass rates. This trend led the State Bar to evaluate the appropriateness of California’s cut score, one of the highest in the nation.

The California Supreme Court, which has ultimate authority over the bar exam and cut score, directed the agency to ensure that these studies:

Identify and explore all issues affecting California Bar Exam pass rates;
Analyze and determine whether protection of potential clients and the public is served by maintaining the current pass line; and
Include broad participation of subject matter experts, stakeholders, law schools, and technical experts.

2017 Studies

The first three studies were completed in 2017. Each study was led by an outside consultant with nationally recognized expertise in the subject. In addition, the State Bar hired additional subject matter experts to serve as external reviewers of the studies’ methods and findings. A working group comprised of a single representative from the California Supreme Court, and two representatives each from State Bar Board of Trustees and the Committee of Bar Examiners, oversaw the work.

The 2017 studies included:

A historical analysis of California Bar Exam pass rate trends from 2008, 2012, and 2016, which suggested that downward shifts in law school applicant performance are a contributing factor in declining bar exam pass rates. Conducted by Dr. Roger Bolus, a nationally known psychometrician with extensive experience evaluating bar exams, the study analyzed data from 2008, 2012, and 2016. It revealed noticeable downward shifts in applicant performance, as measured by law school median LSAT scores. It suggested that approximately 20 percent of the decline in bar exam pass rates could be attributable to changing applicant abilities. However, the lack of individual student performance data limited the ability to identify a causal connection between changes in applicant abilities and bar exam passage rates. Report
A standard-setting study that defined minimally competent performance expected of entry-level attorneys and applied it to bar exam performance for the purpose of recommending a pass line. The study was led by Chad Buckendahl, PhD, a nationally recognized expert in educational assessment and testing. It involved a two-and-a-half-day workshop with 20 practicing attorneys, representing diverse demographics, geography, employment and firm type, and legal practice area. The study utilized a modified version of the Analytic Judgment Method, a structured, iterative process of group discussions and individual rating of exam responses. The method was developed in the 1990s and field tested in different settings through a multiyear study funded by the National Science Foundation. It is considered particularly suitable for evaluating constructed response (essay) questions. Participants reviewed exam content specifications, developed Performance Level Descriptors, and sorted exam responses in three categories: not competent, competent, and far exceeding minimum competence. Combining the panelists’ exam sort with actual scores, the study identified a range of scores for borderline cases, evaluated the impact of specific cut scores within the borderline range, and presented cut score recommendations for policy consideration. Report
A content validation study to assess the alignment of bar exam content with the abilities, skills, and job-related knowledge needed by an entry-level attorney, according to a national attorney job analysis. The study was also led by Chad Buckendahl, PhD, and involved a two-and-a-half-day workshop with practicing attorneys. They evaluated the fit of items, tasks, and scoring criteria relative to job-related content; reviewed knowledge and task statements from a job analysis conducted by the National Committee of Bar Examiners in 2012; and rated test items and tasks by cognitive complexity level and job relevance. The final report evaluated bar exam alignment including:
- Representation of the exam items in terms of content, cognitive complexity level relative to the knowledge and task statements of the NCBE job analysis;
- Aspects of the NCBE job analysis that are missing from the exam; and
- Aspects of the exam that are not aligned with the NCBE job analysis.
Results suggested that the exam’s content and cognitive complexity were consistent with job-related expectations of entry-level attorneys, based on the generalized national job analysis. The study also suggested that the exam’s relevance for skills needed by California entry-level attorneys could be better assessed following a California-specific attorney practice analysis. Report

2017 Report; Supreme Court Response

After completing the standard-setting study, the State Bar invited public comment, surveyed practicing attorney and applicants, and conducted public hearings. In September 2017, the State Bar issued a report to the California Supreme Court, which has ultimate authority over the California Bar Exam cut score.

The State Bar Board of Trustees offered three options for the Supreme Court’s consideration:

An interim cut score of 1390
An interim cut score of 1414
No change in the current score of 1440

The report also outlined key issues important for the Court’s policy decision–public protection, access to justice, and diversity of the legal profession.

In October 2017, the Supreme Court issued its response, maintaining the current cut score for the time being, as the State Bar continued its research.

Law School Performance Study

The fourth study, completed in 2018, examined changes in the characteristics of students taking the California Bar Exam to provide a better understanding of the declining trend of the bar passage rates. The study found that changes over time in the characteristics of exam takers accounted for between 20 and 50 percent of the decline in bar exam performance during the study period. The study was unable to account for a substantial amount of the decline in pass rates, concluding that other unexamined factors have contributed to the decade-long decrease in bar exam performance.

Dr. Roger Bolus led the study. An advisory group of law school deans participated throughout the project. The study examined detailed data on over 7,000 students (from 11 ABA law schools participating in the study) who took the California Bar Exam in July 2013, 2016, and 2017.

Individual data included information on students’ undergraduate grade point average (GPA), undergraduate major, LSAT score, final law school GPA, course work in law school, and basic demographic information. The study examined these characteristics in relation to bar exam performance—both pass/fail outcomes and scores. The study found that:

Changes over time in the characteristics of exam takers accounted for only some of the decline in bar exam performance during the study period. The study found that these changes accounted for between 20 and 50 percent of the decline in bar exam performance during the study period, depending on the performance metric chosen and year compared;
The two student characteristics that most strongly predicted performance on the California Bar Exam were a student’s GPA in the final year of law school and a student’s LSAT scores;
Changes over time in entering credentials (undergraduate GPA and LSAT) and law school credentials (final law school cumulative GPA) contributed roughly equally to that portion of the decline in bar exam performance attributable to changes in student characteristics during the study period;
The proportion of test takers who were ethnic minorities or female grew slightly over the study period, but the study found no correlation between these demographic characteristics and pass/fail outcomes among students with similar abilities; and
Ultimately, the study was unable to account for a substantial amount of the decline in pass rates, concluding that other unexamined factors have contributed to the decade-long decrease in bar exam performance. Report

California Strategies and Stories Program

Beginning in 2018, the State Bar has been working with a team of law and psychology researchers from Indiana University, University of Southern California, and Stanford University on an online program for bar exam applicants, intended to help them adopt a productive mindset while preparing for the exam. The Mindsets in Legal Education researchers designed, administered, and evaluated the intervention, known as the California Bar Exam Strategies and Stories Program. The online program includes an introductory film, stories from prior test takers, and a writing activity in which participants share insights and strategies that may be useful to them and to future test takers.

The program was first offered to all applicants for the July 2018 bar exam and again to July 2019 bar exam applicants. Results from the first two years have been promising: The program increased the likelihood of participants passing the bar exam ranging between 6.8 and 9.6 percentage points, controlling for other factors. The impact was even higher for applicants in disadvantaged groups, including those in underrepresented racial/ethnic populations and those who are first-generation college students. The program was again offered to all bar exam applicants for the October 2020 bar exam.

California Attorney Practice Analysis (CAPA)

In December 2018, the State Bar began the first California-specific study of the knowledge, skills, and abilities required by entry-level attorneys. The study collected detailed empirical data about how attorneys use their knowledge and skills to perform routine tasks in the practice of law. A working group, with members selected by the California Supreme Court from state and national stakeholder groups, oversaw the study.

The working group’s final report contains three broad recommendations designed to bring the California Bar Exam into closer alignment with the current practice of law for entry-level attorneys in California:

Focus the bar exam’s scope on the practice experience of entry-level attorneys, defined as attorneys in their first three years of practice.
Focus exam content on the legal topics that are most important and most frequently used by entry-level attorneys, reducing the number of legal topics from 13 to 8.
Assess bar exam content against the key competencies identified by the data as the most relevant to the work of entry-level attorneys.

2019 Studies

In addition to CAPA, the State Bar completed two other bar exam studies in 2019:

Differential Item Functioning Analysis, which studied the potential differential impact of exam questions by race, gender, and other factors. Examining 20 bar exams from July 2009 to February 2019, the study found no major areas of concern, but recommended further action to continue improving the exam by eliminating minor sources of differential impact.

Evaluation of Grading Processes: The State Bar evaluated its multiphase exam grading process to identify potential efficiencies and best practices in grading while continuing to ensure necessary consistency in the grading process.

These studies, along with a review of test administration procedures conducted by the California Department of Consumer Affairs, were summarized in a report to the Board of Trustees presented in May 2020.

2020 Cut Score Simulation

In March 2020, the State Bar published a simulation of the impact of different bar exam cut scores on bar passage by gender, race/ethnicity, and law school type. The simulation was based on results from 21 bar exams administered over 11 years, from February 2009 to February 2019. The study simulated what the pass rates for the bar exam would have been for various populations by race/ethnicity and gender if the cut score had been 1300, 1330, 1350, or 1390 at the time of those exams. The smallest reduction in the cut score increased the overall pass rate by 8 percentage points, while the greatest reduction in cut score increased it by 31 percentage points, with wide-ranging impacts depending on the subgroup.

For questions, contact: Ron Pi, Principal Program Analyst, Office of Research & Institutional Accountability, 415-538-2013, ron.pi@calbar.ca.gov

Additional Searches