California Bar Examination Studies Fact Sheet

Reports

• Report 1: Recent Performance Changes to the Bar Exam
• Report 2: Standard Setting Study
• Report 3: Content Validation Study
• Final Report on 2017 Bar Exam Studies
• Report 4: Performance Changes on the California Bar Examination

Background

In late 2016, the State Bar of California began a comprehensive evaluation of the California Bar Exam. The evaluation was prompted by several factors, including a multiyear decline in pass rates. This trend raised the question of whether the current pass line, or cut score, of 1440 is appropriate.

The current pass line was established over three decades ago. Critics point out that California’s pass line is higher than every other state in the country except for Delaware. Others contend that factors other than the cut score influence pass rates.

The California Supreme Court, which has ultimate authority over the bar exam and cut score, directed the agency to ensure that these studies:

1. Identify and explore all issues affecting California Bar Exam pass rates;
2. Analyze and determine whether protection of potential clients and the public is served by maintaining the current pass line; and
3. Include broad participation of subject matter experts, stakeholders, law schools, and technical experts.

2017 Studies

The first three studies were completed in 2017. Each study was led by an outside consultant with nationally recognized expertise in the subject. In addition, the State Bar hired additional subject matter experts to serve as external reviewers of the studies’ methods and findings. A working group comprised of a single representative from the California Supreme Court, and two representatives each from the State Bar Board of Trustees and the Committee of Bar Examiners, oversaw the work.

The 2017 studies included:

• A historical analysis of California Bar Exam pass rate trends from 2008, 2012, and 2016, which suggested that downward shifts in law school applicant performance are a contributing factor in declining bar exam pass rates.
  
  Conducted by Dr. Roger Bolus, a nationally known psychometrician with extensive experience evaluating bar exams, the study analyzed data from 2008, 2012, and 2016. It revealed noticeable downward shifts in applicant performance, as measured by law school median LSAT scores. It suggested that approximately 20 percent of the decline in bar exam pass rates could be attributable to changing applicant abilities. However, the lack of individual student performance data limited the ability to identify a causal connection between changes in applicant abilities and bar exam passage rates. Report
• A standard-setting study that defined minimally competent performance expected of entry-level attorneys and applied it to bar exam performance for the purpose of recommending a pass line. 
  
  The study was led by Chad Buckendahl, PhD, a nationally recognized expert in educational assessment and testing. It involved a two-and-a-half-day workshop with 20 practicing attorneys, representing diverse demographics, geography, employment and firm type, and legal practice area. The study utilized a modified version of the Analytic Judgment Method, a structured, iterative process of group discussions and individual rating of exam responses. The method was developed in the 1990s and field tested in different settings through a multiyear study funded by the National Science Foundation. It is considered particularly suitable for evaluating constructed response (essay) questions.
  
  Participants reviewed exam content specifications, developed Performance Level Descriptors, and sorted exam responses in three categories: not competent, competent, and far exceeding minimum competence. Combining the panelists’ exam sort with actual scores, the study identified a range of scores for borderline cases, evaluated the impact of specific cut scores within the borderline range, and presented cut score recommendations for policy consideration. Report

• A content validation study to assess the alignment of bar exam content with the abilities, skills, and job-related knowledge needed by an entry-level attorney, according to a national attorney job analysis. 
  
  The study was also led by Chad Buckendahl, PhD, and involved a two-and-a-half-day workshop with practicing attorneys. They evaluated the fit of items, tasks, and scoring criteria relative to job-related content; reviewed knowledge and task statements from a job analysis conducted by the National Committee of Bar Examiners in 2012; and rated test items and tasks by cognitive complexity level and job relevance.
  
  The final report evaluated bar exam alignment including: 

  • Representation of the exam items in terms of content, cognitive complexity level relative to the knowledge and task statements of the NCBE job analysis;
  • Aspects of the NCBE job analysis that are missing from the exam; and
  • Aspects of the exam that are not aligned with the NCBE job analysis.

  Results suggested that the exam’s content and cognitive complexity were consistent with job-related expectations of entry-level attorneys, based on the generalized national job analysis. The study also suggested that the exam’s relevance for skills needed by California entry-level attorneys could be better assessed following a California-specific attorney practice analysis. Report 

2017 Report; Supreme Court Response

After completing the standard-setting study, the State Bar invited public comment, surveyed practicing attorneys and applicants, and conducted public hearings. In September 2017, the State Bar issued a report to the California Supreme Court, which has ultimate authority over the California Bar Exam cut score.

The State Bar Board of Trustees offered three options for the Supreme Court’s consideration:

• An interim cut score of 1390
• An interim cut score of 1414
• No change in the current score of 1440

The report also outlined key issues related to the Court’s policy decision: public protection, access to justice, and diversity.

In October 2017, the Supreme Court issued its response, maintaining the current cut score for the time being, as the State Bar continued its research.

Law School Performance Study

The fourth study, completed in 2018, examined changes in the characteristics of students taking the California Bar Exam to provide a better understanding of the declining trend of the bar passage rates. The study found that changes over time in the characteristics of exam takers accounted for between 20 and 50 percent of the decline in bar exam performance during the study period. The study was unable to account for a substantial amount of the decline in pass rates, concluding that other unexamined factors have contributed to the decade-long decrease in bar exam performance.

California Attorney Practice Analysis

In December 2018, the State Bar began the first California-specific study of the knowledge and skills needed by entry-level attorneys. The study will collect detailed, empirical data about how attorneys use their knowledge and skills to perform routine tasks in their legal practices. Key components of the study will include focus groups and surveys of California attorneys. Castle Worldwide is conducting the study. A working group, with members selected by the California Supreme Court from state and national stakeholder groups, oversees the study.

The final report, scheduled for completion by December 2019, will set the foundation for revisiting the California Bar Exam pass line and content, as well as exam format and other aspects of the test.

Additional information

Fact Sheet: California Attorney Practice Analysis