25 votes

State Bar of California admits it used AI to develop exam questions, triggering new furor

4 comments

  1. [2]
    SloMoMonday
    Link
    As much as I'd love to rip into idiotic uses of Language Models, this is just a symptom of much bigger issues at the state bar. Can't even suspect any foul play on the part of the Bar. They...

    As much as I'd love to rip into idiotic uses of Language Models, this is just a symptom of much bigger issues at the state bar.

    Last year, as the State Bar faced a $22-million deficit in its general fund, it decided to cut costs by ditching the National Conference of Bar Examiners’ Multistate Bar Examination, a system used by most states, and move to a new hybrid model of in-person and remote testing. It cut an $8.25-million deal with test prep company Kaplan Exam Services to create test questions and hired Meazure Learning to administer the exam.

    There were multiple problems with the State Bar’s rollout of the new exams. Some test takers reported they were kicked off the online testing platforms or experienced screens that lagged and displayed error messages. Others complained the multiple-choice test questions had typos, consisted of nonsense questions and left out important facts.

    Can't even suspect any foul play on the part of the Bar. They justified it as necessary cost cutting since the Admissions Fund seems wholly unsustainable.

    From an old press release

    The agreement will help the State Bar transition to remote and test center-based exam administration, both of which test takers prefer. These test administration changes will also help the State Bar close a significant gap in its Admissions Fund, which is projected to reach insolvency in 2026, absent further efforts to reduce costs. The State Bar projects that the new arrangement will result in annual cost savings of up to $3.8 million in exam-related expenses—enough to significantly reduce if not fill the gap.

    You can even check their budget and revenue forecasts it's clear that the Bar expects to be loosing money very soon.

    So I'm just curious if the very obvious corner cutting is simply a case of budget/manpower limitations or just the organizational decy kicking in. They signed the work off to Kaplan who decided to start their major government contract with the min-effort strategy. Wouldn't be surprised if they just let go of all the people responsible for maintaining test standards.

    But even beyond this mess, I'm very curious what happens if a professional standards group can't sustain itself. These bodies are important and fumbling an institutional responsibility like the Exams is not a very good look. You also don't want to hear about how the organization that maintains the standards for lawyers are essentially broke. That would erode my trust a lot more than the AI questions.

    Hell, I really want to know how half their imcome is "Grants" and they expect to pay more in "Grant Expenses".

    10 votes
    1. MimicSquid
      Link Parent
      This is one that I can answer, as this is my day job. (not for them, to be clear, but financial planning and analysis in general) Take a look at the trend of Grants and Grant Expenses over the...

      Hell, I really want to know how half their imcome is "Grants" and they expect to pay more in "Grant Expenses".

      This is one that I can answer, as this is my day job. (not for them, to be clear, but financial planning and analysis in general)

      Take a look at the trend of Grants and Grant Expenses over the last few years. Both of them are rising, with grants higher than grant expenses. In the 2025 forecast, grant expenses are expected to rise 10%, but grant revenue is dropping by a third. By 2026 the budget for grant expenses has been corrected to be in line with expected revenue, but many, many organizations have been blindsided by how bad the economy is getting how fast. These sorts of downward corrections are being baked into budgets all over the country.

      Digging deeper: While the difference isn't distinguished in the major budget, you can see the budget on an individual project basis by clicking on the "Budget by Fund" link below. There are drops in the expected grant revenue dedicated to the Legal Services Trust Fund, the Equal Access Fund, and the general Grants Fund, while there are much higher expenses. Two out of three of those are funds dedicated to focusing on issues that are in the federal government's crosshairs.

      Does that help? Those programs are going to be upside down next budget cycle because the federal government has gone crazy, but they're working to adjust to the startlingly fast change in financial conditions.

      11 votes
  2. boxer_dogs_dance
    (edited )
    Link
    California has a reputation as one of the most difficult/ rigorous state bar examinations. This damages that brand, badly. I hope that the bar can solve the problem.

    California has a reputation as one of the most difficult/ rigorous state bar examinations. This damages that brand, badly.

    I hope that the bar can solve the problem.

    5 votes