Oct. 18, 2017 UpdateThe Supreme Court has issued its response after reviewing a Standard Setting Study of the passing score and options from the State Bar of California
In 2017 the State Bar decided to undertake four studies to evaluate the California Bar Exam. The first study, an analysis of the historical trend of bar exam passage rates based on existing data, was completed in March. The other three studies, in various stages of planning, include:
Combined, the four studies will assist the State Bar in developing recommendations on the bar exam for the California Supreme Court to consider.
In late 2016, the State Bar of California began developing a plan to conduct a comprehensive evaluation of the bar exam. The decision to evaluate the exam was prompted by a number of factors including a multi- year trend of declining passage rates. This trend raised the question of whether the current pass line, or cut score, of 1440 is appropriate.
Critics of the exam point out that California’s pass line is higher than every other state in the country except for Delaware. At the same time, since the current pass line was established over three decades ago, the percentage of students who pass the exam has increased and declined, indicating that factors other than the cut score influence passage rates.
As the State Bar began to move forward with its plans to evaluate different aspect of the bar exam, the California Supreme Court directed the agency to ensure that these studies:
The State Bar will submit a report to the Court no later than Dec. 1, 2017, providing detailed findings from the studies, recommendations for changes, if any, to the bar exam, and a timeline for implementation.
The management of the different studies is being overseen by the State Bar’s Office of Research and Institutional Accountability and under the direction of the Bar Exam Study Working Group. The Working Group is comprised of a single representative from the California Supreme Court, and two representatives each from State Bar Board of Trustees and the Committee of Bar Examiners.
The Bar Exam Study Working Group meets every other week and will continue to do so throughout 2017 to monitor the progress of the studies, receive input, and ensure that the studies are concluded in time for submission to the Court in December. While the project timelines for the individual studies vary, they are all scheduled to be completed before September, in time for the State Bar to evaluate the findings and develop recommendations for the Court to consider.
Each study is being led by an outside consultant with nationally recognized expertise in the subject. In addition, the State Bar has entered into contracts with additional subject matter experts to serve as external reviewers of the studies’ methods and findings.
As noted above, the first study examining historical trend of California Bar Exam passage rates was completed in March. Dr. Roger Bolus, a nationally known psychometrician with extensive experience evaluating bar exams. The study analyzed existing historical data on bar examinations from 2008, 2012, and 2016. The study revealed noticeable downward shifts in applicant performance, as measured by law school median LSAT scores, across different subgroups during the 9-year period.
Study results suggested that approximately 20 percent of the decline in bar exam pass rates is likely attributable to a change in applicant composition. However, the lack of individual student performance data, as opposed to measures of the median LSAT scores of law schools, limited the ability of the study to identify a causal connection between changes in applicant abilities and bar exam passage rates. In-depth analysis of possible causal linkages will depend on the collection of detailed student performance data from law schools; this data will be collected as part of the Law School Performance Study.
The State Bar has contracted with Chad Buckendahl, Ph.D., a nationally recognized expert in the field of educational assessment and testing, to lead the standard setting study. Two independent experts in psychometrics, Mary Pitoniak, Ph.D. and Tracey Montez, Ph.D., have been retained to serve as external reviewers on this study.
Standard setting studies are generally based on examinations that have already been given. This study will utilize a modified version of the Analytic Judgment Method, a structured, iterative process of group discussions and individual rating of actual exam responses. The method was developed in the 1990s and field tested in different settings through a multi-year study funded by the National Science Foundation. It is considered particularly suitable for standard-setting exercises evaluating the performance of constructed response (essay) questions.
A panel of 20 practicing attorneys, assembled from nominations or appointments provided by the Governor’s Office, the Senate and Assembly Judiciary Committees, law schools, and the State Bar Board of Trustees, will participate in the two-and-a- half day exercise to be held on May 15-17, 2017.
Qualification of the panelist candidates require that they be knowledgeable of the expectations for new attorneys, usually early career practitioners and experienced attorneys who supervise new attorneys. Drawing from a pool of 38 candidates, the Court made a final selection of panelists in early May to represent a balance of participants in terms of demographics, geography, employment and firm type, and legal practice area.
On the first day of the workshop, participants will be oriented to the standard-setting process and methods. Participants will also review the content specifications for the exam and review a general policy definition of the “target candidate” which will serve as a starting point for more detailed discussions to develop Performance Level Descriptors to guide subsequent judgments about exams.
Participants will be provided a sample of exam papers representing the full range of performance at different levels. Although the responses have already been graded, the panelists are not shown the scores and in the initial round are asked only to determine if the response fits into one of three categories: not competent, competent, and far exceeding minimum competence.
Individual evaluations are then compiled to begin identifying marginal exams. Participants will begin grouping the exams into those that represent the best of the “not competent” responses and the worst of the “competent” responses to move closer to establishing a threshold or pass line for the exam. Combining the panelists’ classification of the exams papers with the actual scores, Dr. Buckendahl will analyze the data to identify a range of the scores for borderline cases.
Additional analysis of the data will evaluate the impact of specific cut scores within the borderline range on the bar exam passage rate. A set of cut score recommendations will be presented in the final report for policy consideration.
The content validation study will examine the substantive overlap between what is measured on the exam and what is important for entry-level attorneys. As with the standard setting study, this work will also depend on the participation of subject matter experts to serve as panelists. Currently in planning stage, the content validation workshop will be held in early June 2017.
The two-and-a-half day workshop, also facilitated by Dr. Buckendahl, will involve the following steps:
Following the two day meeting, Dr. Buckendahl will produce a report that evaluates several areas of bar exam alignment including:
This study will be reviewed by the two independent psychometric experts as well referenced above for the Standard Setting Study.
The Law School Performance study aims to gather more detailed information on law school performance by linking individual measures of students – such as LSAT scores and GPAs – to bar exam performance. The goal is to recruit the participation of all California law schools and collect anonymized student performance data for specific periods from 2008 to 2016 in order to represent the breadth and historical trend of possible changes in student and examinee performance. By linking detailed data on student performance with their bar exam performance, in-depth analyses of any causal connections will shed more light on the recent trend of the declining passage rates that the limited analysis based on existing historical data noted above was unable to answer. Dr. Bolus will be the Principal Investigator of the study.
Still in its early planning stage, this study will be guided by an Advisory Group of law school deans. The State Bar is in the process of forming this body, delineating the scope of the project, defining relevant issues, and devising data collection strategies.
Since the inception of the project, the State Bar has maintained regular engagement with interested stakeholders, including periodic conference calls with law school deans to seek their input on various policy and methodological issues related to the studies. The State Bar is also in the planning stage s of designing a survey to seek the views of practical attorneys and law school students on the issue of bar examination with respect to the cut score as well as content of the exam.
When the final report of the standard setting study is completed in July 2017, the current project design includes both a formal request for comment period and two public hearings in mid-August.
For questions contact: Ron Pi, Principal Analyst, Office of Research and Institutional Accountability firstname.lastname@example.org