Legal Retrieval for Public Defenders

Dominik Stammbach [email protected]
Princeton University

Kylie Zhang [email protected]
Princeton University

Patty Liu [email protected]
Princeton University

Nimra Nadeem [email protected]
Princeton University

Inyoung Cheong [email protected]
Princeton University

Lucia Zheng [email protected]
Stanford University

Peter Henderson [email protected]
Princeton University

Abstract

AI tools are suggested as solutions to assist public agencies with heavy workloads. In public defense---where a constitutional right to counsel meets the complexities of law, overwhelming caseloads, and constrained resources---practitioners face especially taxing conditions. Yet, there is little evidence of how AI could meaningfully support defenders' day-to-day work. In partnership with the New Jersey Office of the Public Defender, we develop the OPD Resource Library, a retrieval tool which surfaces relevant appellate briefs to streamline legal research and writing. We show that existing retrieval benchmarks fail to transfer to real public defense research, however adding domain knowledge improves retrieval quality. This includes query expansion with legal reasoning, domain-specific data and curated synthetic examples. To facilitate further research, we release a taxonomy of realistic defender search queries and a manually annotated evaluation dataset for public defense retrieval. This benchmark is highly correlated with a proprietary retrieval dataset annotated by experienced public defenders. Our work improves on the status quo of realistic legal retrieval benchmarking and illustrates one approach to applying AI in a real-world public interest setting.

Dataset

Replication Package

Executive Summary: Public defenders in the United States face heavy caseloads and must conduct legal research under tight resource constraints. They often rely on past appellate briefs from colleagues to handle unfamiliar cases, yet current AI tools have not been shown to support this work reliably. Generative systems frequently produce hallucinations or unverifiable claims, while existing legal retrieval benchmarks use artificial queries that do not reflect actual defender needs.

The authors partnered with the New Jersey Office of the Public Defender to build and test the OPD Resource Library, a retrieval system that surfaces relevant passages from the office’s collection of appellate briefs, directives, and guidelines. They collected 194 real queries from experienced defenders, created a public evaluation dataset of 170 queries with 543 manually verified relevant paragraphs, and compared eight retrieval models and several rerankers under zero-shot and fine-tuned conditions. They also tested whether domain knowledge—query expansion using legal reasoning and carefully filtered synthetic training data—could raise performance.

Larger embedding models performed best in zero-shot settings, yet the strongest model reached only 37 percent Recall@5 on the public dataset. Training on existing legal benchmarks reduced accuracy, while fine-tuning on curated synthetic data and IRAC-based query expansion raised Recall@5 by several points. A reranker fine-tuned on defender annotations further improved result quality. Performance on the released dataset closely tracked results on the office’s confidential data.

These outcomes show that retrieval can deliver trustworthy assistance that mirrors current office practices without introducing the risks of generative AI. The tool gives defenders faster access to reusable arguments and precedent, potentially shortening research time and promoting consistency across cases. It also supplies the first publicly available benchmark built from authentic public-defense queries.

Offices should pilot similar closed, retrieval-only systems that return source passages rather than generated text, accompanied by clear disclaimers and safeguards against over-reliance on summaries. Researchers should develop agentic retrieval methods and larger domain-adapted models, while collecting queries from additional jurisdictions to test broader applicability. The current dataset covers one state and recent documents only; results should be validated locally before wider deployment.

1. Introduction

Section Summary: Public defenders in the US often struggle with heavy caseloads and limited resources, which can compromise the quality of legal help they provide to clients who cannot afford private lawyers. To address this, researchers worked with the New Jersey Office of the Public Defender to build an AI tool called the OPD Resource Library that uses foundation models to find and surface relevant past appellate briefs, helping defenders reuse strong legal arguments and precedents in their own cases. The project also includes real-world testing with actual defender queries, the release of a new dataset and models, and notes that existing legal AI benchmarks fall short of capturing practical office needs.

In the United States, individuals facing criminal charges have a right to counsel. This right is guaranteed by the Sixth Amendment and reaffirmed in [1]. For those unable to afford private counsel, representation is provided by public defenders. In practice however, defenders often face severe resource constraints and overwhelming caseloads ([2]), while having to navigate the complexities of today's legal system. Combined, these can undermine promises of fair and equal legal representation for clients relying on public defense.

Advances in natural language processing, particularly in foundation models, have raised hopes that AI tools could assist public defenders by streamlining time-consuming tasks, including for example legal research or drafting of briefs ([3, 4, 5]). However, despite rapid advances in model capabilities and legal benchmark performance ([6, 7]), there remain very few examples of concrete real-world implementation and evaluation of AI within public defender offices.

This gap limits our understanding of what types of AI use cases are feasible, safe, and can genuinely empower defenders in day-to-day legal practice. Public defense is a high-stakes setting where errors can directly affect the outcome of cases and clients. Hence, carefully balancing trade-offs between accuracy, reliability and risks of failure become essential design constraints in such applications. Consequently, any AI assistance for public defenders must prioritize verifiable and trustworthy outputs.

In this work, we partner with the New Jersey Office of the Public Defender (NJOPD) to identify, develop, and evaluate such an AI use case. Defenders often specialize in specific legal areas, like felonies or misdemeanors. If handed a case outside their area of specialization, they often consult colleagues to obtain past briefs within the office handling similar cases. Such past briefs, especially those from the appellate section written by experienced defenders, offer overviews, reusable legal arguments, applicable precedent and overall guidance for how to navigate similar legal circumstances.

Inspired by such current office practices, we developed the OPD Resource Library. The Library leverages foundation model-based embeddings for retrieval, along other components, to search over all appellate briefs within the office and surfaces relevant ones. By providing access to relevant briefs, defenders can reuse legal arguments, precedent and other applicable information. Thus, the Library is designed to streamline brief drafting, but also maintains more consistency within the office: defenders may be more likely to leverage already-identified winning strategies and follow best practices.

During multiple evaluation rounds, experienced defenders submitted realistic search queries and assessed the quality of the tool. The submitted queries are highly diverse: They include broad topical searches, specific legal arguments, pinpoint citations, statutory definitions and doctrinal status checks. These queries reflect the heterogeneous and practice-driven information needs of public defenders.

In contrast, existing legal retrieval datasets are often constructed somewhat artificially, e.g., by reusing bar-exam questions as queries ([9]), finding similar cases given a query case ([12]), or by heuristically linking preceding context to quoted legal passages ([13, 8]). In Table 3 in Appendix, we show that no existing U.S. legal retrieval datasets contain both realistic queries and manually verified target paragraphs. While existing datasets are valuable for controlled benchmarking and measuring progress on legal retrieval, they poorly approximate more realistic contexts. Figure 1 visualizes query embeddings from BarExam-QA ([9]), LePaRD ([8]), and public defender queries, which are separable in embedding space. In Section 5, we will show that training on such datasets decreases performance on public defense retrieval.

On the other hand, we improve recall by (1) using more recent and larger embedding models ([14, 15]) and (2) leveraging domain knowledge. To introduce domain knowledge, we generate curated synthetic training data using a fine-tuned defender query generation model, and filter that data with a fine-tuned legal reranker (similar to [16]). We further expand queries in this dataset using the IRAC framework[^1] developed for legal analysis. See Section 5 for full results.

[^1]: Issue, Rule, Application, and Conclusion.

Our work is a first step toward practical AI for public defense, inspired by current office practices. [17] argue that future benchmarks should emphasize real-world utility. In our work, we offer a step towards more realistic legal NLP benchmarking, enabled by collaboratively developing the OPD Resource Library in partnership with the NJOPD. To stimulate further research on legal retrieval for public defenders, we release a manually annotated dataset, fine-tuned models and replication code.[^2]

[^2]: replication package: https://github.com/princeton-polaris-lab/PublicDefenderRetrieval

Combined, these artifacts could help develop similar libraries for other defender offices or pro-bono clinics, but also measure progress in more realistic legal retrieval settings. More broadly, this collaboration illustrates how partnerships with public institutions allow situating NLP research in real-world applications. To summarize, we make the following contributions:

• We introduce the public defense retrieval task: retrieving relevant passages from existing appellate defense briefs. We construct an accompanying evaluation dataset comprised of 170 queries and 543 human-annotated relevant paragraphs from publicly available documents (Section 3).
• Evaluation of eight pre-trained retrieval models on public defense search. While larger models perform better, the best model only achieves 37.08% Recall@5, demonstrating substantial room for improvement on this task (Table 2).
• A taxonomy of defender search queries, discussing shortcomings of current models and informing future work on public defense search (Section 4).
• Evidence of a mismatch between existing legal retrieval benchmarks and real-world use cases. We find that fine-tuning retrieval models on existing legal retrieval benchmarks degrades performance, while legal domain adaptation, fine-tuning on carefully curated synthetic data and query expansion strategies improve performance (Section 5).

2. Public Defense Retrieval

Section Summary: Public defense offices can benefit from AI assistance in legal research by retrieving relevant paragraphs from a trusted corpus of past briefs, internal documents, and public directives, which serve as reusable surveys of legal issues and starting points for unfamiliar cases. Generative AI tools and broad case-law searches were explored but rejected due to high rates of hallucinations, failures on nuanced legal contexts, confidentiality risks, and the danger of surfacing outdated or overturned precedents. The resulting retrieval task therefore focuses on matching a defender's query—such as a citation, doctrinal question, or case description—to the most relevant existing materials within the office's own collection.

[5] group public defense work into five pillars, of which two seem especially suitable to AI assistance: evidence investigation and legal research and writing. For legal research and writing, they report that AI would be most useful to generate surveys of information, provide starting points, draft documents and narrow down case searches.

This perspective is reinforced in our collaboration with the NJOPD. A well-written brief functions as a structured survey of a legal issue. The office encourages defenders to use such briefs as starting points if they have to handle a case outside their specialization. Thus, access to such briefs streamlines legal drafting through reuse of existing materials. Lastly, these briefs contain relevant legal precedent, and thus indirectly narrow down case search.

Apart from searching for briefs, we also explored other AI use cases to empower defenders. These include (1) directly answering legal queries using generative AI with a closed-source RAG tool (NotebookLM), and (2) searching through federal and state case law. However, both failed to deliver sufficient levels of accuracy, verifiability, and transparency, hence we define public defense retrieval as searching over past briefs, materials defenders already use and trust. Building accurate search over briefs also serves as a lens into the collective knowledge acquired within an office over time, and makes that knowledge accessible.

2.1 Retrieval vs. Generation

Past work pointed out potential usefulness of generative AI for legal work ([3, 4]), which has been supported by promising legal evaluation of LLMs, such as GPT-4 passing the bar exam ([18]). [19] find that LLMs help law students draft legal documents faster. Given such evidence, the office explored the potential of LLMs to assist public defenders in their day-to-day work. Experienced defenders submitted 100 queries to NotebookLM, a Retrieval-Augmented Generation (RAG) application. Although NotebookLM had access to a small set of relevant briefs, 66% of the NotebookLM generations contained issues. The three main failure modes consisted of:

1. Hallucinations (of e.g., citations) or incorrect references to source materials.
2. Failure of the model to address nuanced legal contexts.
3. Incomplete or verbose outputs, or generating unrelated information altogether.

Strikingly, [5] report that 85% of public defenders they interviewed currently doubt AI can reliably verify research output, describing the same reasons (hallucinations, failure to handle nuanced legal context and incomplete output) NJOPD identified in their explorative evaluation. Given the high stakes of public defense, where hallucinated citations and misstated precedents can affect the outcome of cases, such failure rates are unacceptable in practice. Beyond accuracy, using commercial generative models raises confidentiality risks: case details submitted to proprietary APIs may fall outside attorney-client privilege and be subject to mandatory disclosure ([20, 21]). Generative models also provide limited transparency about sources, making it difficult for attorneys to verify output accuracy.

2.2 Brief vs. Case Law Retrieval

We also considered retrieval over all state and federal case law, which would directly return relevant precedent. However, this risks surfacing opinions that are no longer good law. As prior work shows, approximately 7.8% of lower-court decisions are later reversed on appeal ([22]). Because defenders cannot rely on non-binding or outdated precedent, case law search alone without information about whether a case is still good law seems too unreliable for practical use cases. While commercial systems include such information, automatically detecting overturned case law remains an active and unresolved research problem ([23]). In the Resource Library, we address this by including internal documents and public directives, both containing, among others, overturned case alerts and up-to-date best practices for frequently occurring legal issues.

2.3 Task

Definition

Formally, we define the public defense retrieval task as follows: Given a user query $q$, which may be related to a citation, rule reference, doctrinal question, or natural-language description of a legal concept, the goal is to retrieve the most relevant paragraphs $p_i$ from a corpus of prior briefs, other internal documents and public directives. Each query can have multiple relevant paragraphs.

3. The OPD Resource Library and the PD Retrieval Dataset

Section Summary: The OPD Resource Library is a search tool that lets public defenders quickly find relevant passages from a large internal collection of appellate briefs, directives, and guidelines by retrieving matching paragraphs along with helpful summaries and case details. To make this kind of resource available for broader research despite confidentiality limits, the authors built the Public Defense Dataset, a public benchmark that uses the same real defender queries but pairs them with paragraphs drawn from openly available legal documents. The dataset was assembled through automated retrieval, LLM filtering, and careful manual review by legally trained annotators, resulting in hundreds of verified relevant examples.

In this section, we describe the Resource Library in more detail, and the associated PD dataset construction process.

::: {caption="Table 1: Legal Search Queries and Associated Paragraphs"}

:::

3.1 OPD Resource Library Overview

The Resource Library is a retrieval system that enables defenders to search across the office’s internal corpus of appellate briefs, directives, and guidelines to locate relevant passages for new cases. The primary goal of the system is to make the collective institutional knowledge of the office, including arguments, citations, and legal reasoning, accessible within seconds. This corpus consists of 2896 briefs spanning the last 25 years, 168 internal documents and 351 public directives. We automatically split these into 140K unique paragraphs using LLM-based semantic segmentation ([24]).

When a user submits a query, the Library retrieves relevant paragraphs and presents them alongside contextual metadata (such as title and filing date). Each retrieved passage is accompanied by an LLM-generated summary of the legal issue and case facts to help users decide which returned briefs warrant closer inspection. Importantly, generation is only used to summarize existing content and help defenders quickly decide whether a brief might be relevant, not to create new arguments or citations, preserving factual reliability.

Over three evaluation rounds, experienced defenders submitted 194 queries to the tool. For each query, they provided detailed feedback: binary search result annotation (relevant or irrelevant) for up to five retrieved paragraphs, and additional freeform textual feedback. They annotated 85.6% of all returned paragraphs, from which 66% were annotated as being relevant for public defense work. Moreover, they provided textual feedback for 55.2% of the queries, which further contextualize the annotations and search results.

3.2 The Public Defense Dataset

Due to confidentiality constraints, the proprietary NJOPD dataset cannot be released. To support reproducible research, we construct the Public Defense Dataset (PD dataset), a resource that mirrors the structure and characteristics of the proprietary dataset with the same search queries, but target paragraphs from publicly available documents. We show dataset examples and annotated paragraphs in Table 1.

The PD dataset contains:

• 170 authentic public defender queries, collected from multiple evaluation rounds with senior public defenders in New Jersey (discarding all queries containing personally identifiable information).
• 96, 032 unique paragraphs segmented using LLM-based semantic chunking, the same process applied to OPD's internal corpus. They are drawn from a corpus of 856 publicly available documents, including appellate briefs and New Jersey Court/Attorney General directives.
• 543 human-annotated relevant paragraphs manually annotated and verified by the author team. All annotators have ample background in legal NLP, and half of the annotators are currently in or have completed law school.

Dataset construction.

To build this public benchmark, we scraped state and AG guidelines, and briefs from cases argued at the appellate level between 2023 and 2025. In total, this results in 856 documents: 351 directives and 505 briefs. We convert all pdfs to text using olmOCR ([25]). Afterwards, we apply LLM-based semantic segmentation ([24]) to split the documents into 96, 032 unique paragraphs.

We then obtained candidate search results for all queries obtained during the NJOPD evaluation rounds from this corpus (discarding all queries containing personally identifiable information). For each query, we retrieve the following candidates: We collect the 100 most similar paragraphs to the query using the NV-Embed model ([14]) and the 10 most similar paragraphs based on a keyword search. If we have annotated gold paragraphs from an evaluation round, we also retrieve the 10 most similar paragraphs for each gold paragraph. In total, each query yields 110–160 paragraph candidates: 100 from the NV-Embed, 10 from BM25, and 0–50 from the most similar paragraphs retrieved from annotated results. We narrow down the set of potential candidates using LLMs. First, we use GPT-4o as a judge to filter paragraphs which are potentially irrelevant. We then rerank all remaining paragraphs using a Qwen3-8B reranker ([15]) fine-tuned on the proprietary NJOPD dataset.

Next, we manually review up to seven highest scoring paragraphs (if all paragraphs were discarded by the reranker or GPT-4o, we discard the query). For this annotation, we take into account the query, our own legal expertise, the already annotated gold paragraphs from the Resource Library evaluation rounds, and additional feedback collected from those evaluation rounds. The annotation was performed by the author team, all with backgrounds in legal NLP. Additionally, half of the annotators are in or have completed law school. To compute inter-annotator agreement, an independent annotator with law school experience also annotated 100 query/paragraph pairs, using the same annotation guidelines. Annotations between the author team and that annotator result in a Cohen's Kappa of 0.36, indicating fair agreement. Most disagreements (66% of all disagreements) can be explained by the independent annotator being more lenient, and annotating slightly relevant paragraphs too. The author team was more conservative and often rejected such slightly relevant paragraphs instead.

Lastly, we anonymize party-related personally identifiable information in the dataset. We use the 31B Gemma 4 model ([26]) to anonymize all paragraphs (prompt shown in Figure 7 in Appendix). This results in 147, 044 anonymized entities (1.5 entities per paragraph). To evaluate the anonymization procedure, we manually reviewed 50 paragraphs, 25 where no information was anonymized and 25 paragraphs were at least one entity was anonymized: we did not find any personally identifiable information (PII) in paragraphs where the model did not anonymize. For the anonymized entities, we find that 67% of anonymizations are correct, while 28% contain over-anonymization: judge names or organization names insufficient to identify parties. In 5% of cases the model anonymizes non-relevant legal terms (e.g., case index numbers).

Performance on the two datasets (the proprietary NJOPD and the publicly released PD dataset) is highly correlated: Across eight zero-shot experiments, Recall@5 results in a Spearman R of 0.79 (p = 0.02) and across 20 fine-tuning experiments, the Spearman R is 0.82 (p=9.1e-6). We further show that anonymization barely affects retrieval performance: the Spearman R between the anonymized and non-anonymized version is 1.0 in zero-shot settings (perfect correlation), and 0.99 (p=2.0e-16) for fine-tuning experiments. In Table 4 in Appendix, we present dataset statistics from both the PD and the proprietary NJOPD dataset, and show that they are comparable. Combined, these findings make us confident that the PD dataset captures relevant signals about public defender search, and may also be used as a more realistic evaluation benchmark for legal NLP and AI methods.

4. Taxonomy of Defender Queries

Section Summary: Researchers analyzed real-world queries from public defenders by manually labeling them according to both the underlying search goal and the retrieval approach required. The most common goals involved locating specific legal standards, rules, or doctrines, followed by broader requests for topical arguments or briefs, while rarer ones covered term definitions, factual details, or whether a case remained good law. Keyword-style queries proved especially difficult for embedding-based systems and produced the highest failure rates, whereas complex multi-step questions pointed to the potential value of agentic retrieval methods.

To better understand public defense search, we manually annotated all queries for search intent (what a defender was searching for) and algorithmic search strategies (keyword-based, embedding-based, and agentic). We then construct a taxonomy, which includes information about how often queries have been successfully answered by the Resource Library. These can point to common failure cases in realistic legal retrieval. While commercial companies have such data at scale, little to no publicly available information on real-world legal queries can be found. We describe the annotation process in more detail in Table 5 in Appendix.

4.1 Search Intent

We find substantial variation in public defense search queries, ranging from queries asking for specific cases or rules by title, e.g., "803(c)(27)", to more natural language queries, e.g., "Find me briefs about inevitable discovery?" to questions about legality, e.g., "are tinted windows legal in New Jersey" to complex queries requiring multihop retrieval and reasoning, such as "has Counterman v. Colorado been addressed in a published new jersey opinion?". Overall, we identify eight broad categories of search objectives and plot the distributions across them in Figure 2. The majority of queries fall into either (1) legal standards, rules or doctrines, (2) search for legal arguments or briefs about certain topics, and (3) less frequently used categories.

Standards, Rules, and Doctrines.

Queries that ask for the legal standard or rules, often for a specific situation, for example "standard for ordering passenger out of a car" or "803(c)(27)". These are the most frequent queries, and closely reflect the day-to-day needs of defenders when conducting legal research. They often include keyword-based queries which embedding-based retrieval approaches struggle with.

Topical or Argument-Oriented Searches.

Queries that broadly look for briefs or passages about a certain topic, such as "find briefs about community caretaking", or specifically for legal arguments, for example "What are arguments against consent searches during illegal car stops?". Having relevant briefs about certain topics can give an overview of the legal landscape and applicable legal arguments. Failure cases include a lack of ability in current models to distinguish nuanced legal contexts: Consider the query "reverse 404b". Reverse 404(b) is when a defense lawyer introduces evidence of another person's past acts to exonerate their client, whereas the standard 404(b) rule is typically used by prosecutors to introduce a defendant's past acts to prove guilt. Embedding models for this query only return results about the standard 404(b) rule.

Other, less frequent categories include term clarifications (e.g., “difference between reasonable suspicion and probable cause”), definitions and exceptions (e.g., “booking exception to Miranda”), factual or procedural questions (e.g., “when was NERA amended?”) and questions about good law (e.g., "Is statev. pena-flores still good law?"). Failure cases here can be summarized as the model not sufficiently understanding the query, or the relevant information not being present in the indexed corpus.

4.2 Information Sources and Search Strategies.

We also annotate the required search strategy for each query. We group search strategies into embedding-based, keyword-based and agentic search (Figure 3):

• Embedding-based retrieval computes dense vector representations for queries and paragraphs, and the paragraphs closest to a query are returned as results.
• Keyword-based retrieval, such as BM25 ([27]) represent queries and documents as sparse vectors, capturing some statistic about word frequency such as TF-IDF weights. Search results are again paragraphs with the highest similarity to a query.
• Agentic search strategies ([28, 29]) refer to methods that broadly involve an agent conducting retrieval, with actions such as query expansion ([9]), multihop retrieval, and reasoning over multiple retrieval steps.

Embedding-based retrieval captures most queries, but keyword searches remain common. This is likely due to defenders’ familiarity with boolean-style, commercial legal research systems. Notably, keyword-style queries have the highest rate of unhelpful results (38%), suggesting that models optimized for natural language retrieval still underperform when users employ traditional legal search syntax.

One example for an agentic query is "has Counterman v. Colorado been addressed in a published new jersey opinion". To answer, one would first need to retrieve all published opinions referring to Counterman v. Colorado, requiring a search index over all state case law, and associated metadata about publishing status. Then, the agent would need to read all these opinions and decide whether they sufficiently address the main arguments in Counterman v. Colorado, and finally return the answer. Optimally, the agent would provide excerpts from the state opinions addressing the case, along with links to the full opinions. We believe this points to exciting avenues for future work in AI-powered public defense research.

5. Empirical Evaluation of Retrieval Models and Rerankers

Section Summary: The section evaluates retrieval models, which use embeddings to surface candidate paragraphs via semantic similarity, and rerankers that refine those candidates with more powerful cross-encoder models, testing both on a proprietary legal dataset and a comparable public one. Zero-shot tests of eight pre-trained models reveal that larger models generally deliver stronger recall, with performance on the two datasets closely correlated. Attempts to improve results through fine-tuning on existing legal benchmarks or simple synthetic data tended to hurt accuracy on the defense datasets, while more carefully constructed synthetic data showed better potential.

We evaluate two components of the Resource Library pipeline: (1) retrieval models, which encode both queries and paragraphs into embeddings and return a set of candidate results based on semantic similarity, and (2) rerankers, which re-score the top- $k$ retrieved passages using more expressive cross-encoder LLMs.

\begin{tabular}{l|cc|cc}
\toprule
\textbf{Model} & \multicolumn{2}{c|}{\textbf{NJOPD Dataset}} & \multicolumn{2}{c} {\textbf{PD Dataset}} \\
   {} & Recall@1 & Recall@5 & Recall@1 & Recall@5 \\
\midrule

all-mpnet-base-v2 & 6.79 & 19.72 & 7.44 & 19.30 \\
E5-base-v2 & 11.11 & 27.44 & 6.27 & 25.23 \\
E5-large-v2 & 11.34 & 29.61 & 8.67 & 27.40 \\
Qwen3-Embedding-0.6B & 8.69 & 30.93 & 9.28 & 29.35 \\
\hdashline
Qwen3-Embedding-4B & 10.33 & 36.84 & 11.13 & 34.19 \\
E5-mistral-7b-instruct & 14.48 & 41.97 & 11.11 & 32.61 \\
NV-Embed-v2 & \textbf{15.12} & \textbf{51.85} & 11.48 & 31.27 \\
Qwen3-Embedding-8B & 13.16 & 40.19 & \textbf{13.37} & \textbf{37.08} \\
\hdashline
fine-tuned E5-large-v2 & 10.72 & 33.71 & 10.40 & 36.26 \\

\bottomrule
\end{tabular}

Experimental Setup

All experiments are conducted on both the proprietary NJOPD dataset, and the PD dataset described in Section 3. Each paragraph in the datasets is treated as an independent retrieval unit. Given the size of the datasets, we consider the whole dataset as a test set only and report results on the whole set. For all fine-tuning experiments, we report the mean result of five runs with different seeds (confidence intervals can be found in the Appendix).

5.1 Retrieval Experiments

We evaluate retrieval performance of eight pre-trained retrieval models: all-mpnet-base-v2 ([30]), E5-base-v2 and E5-large-v2 ([31]), Qwen3-Embedding-0.6B, -4B and-8B ([15]), E5-mistral-7b-instruct ([32]) and NV-Embed-v2 ([14]). We use Recall@k (with $k=1, 5$) as the metric. This metric is informative for practitioner-facing search systems where defenders typically inspect only the top few results.

Zero-shot retrieval.

We first evaluate zero-shot performance of eight pre-trained models (Table 2). We observe that larger models (above 4B parameters) perform better than smaller models (below 1B parameters) on the NJOPD dataset and the PD dataset. The results also confirm that PD is a good approximation of the proprietary NJOPD dataset: performance of the two datasets is correlated, with a Spearman R of 0.79 (p=0.02). If we discard results from the NV-Embed model, we obtain a Spearman R of 0.89 (p=0.007). The NV-Embed model is potentially confounding, as the first version of the Resource Library simply returned the top five paragraphs found by the NV-Embed model. By construction, that model has a recall of 100% on all these examples, which in turn inflates performance numbers for this model on the proprietary dataset.

Fine-tuning on existing legal retrieval benchmarks.

Next, we fine-tuned four smaller models (all-mpnet-base-v2, E5-base-v2, E5-large-v2, Qwen3-Embedding-0.6B) on two existing legal retrieval benchmarks: BarExam-QA and LePaRD. All reported results are the mean of five independent fine-tuning runs with different seeds. Fine-tuning on these datasets leads to a decrease in Recall@5. In Figure 4, we show averaged performance gains (or losses). We show exact results for all models in Table 7 in Appendix. To illustrate, the blue bars indicate the effect of further fine-tuning models on the BarExam-QA dataset. If evaluated on the same BarExam-QA dev set, performance increases by 1.7 points in Recall@5, compared to the zero-shot performance of the same models. Training on BarExam-QA also slightly increases performance on LePaRD. However, performance on both the proprietary NJOPD and the released PD dataset decreases.

Fine-tuning on naive synthetic dataset.

Since fine-tuning on existing benchmarks does not lead to performance gain, we experiment with synthetic datasets. We first construct and evaluate a "naive" synthetic dataset generated by a Llama3-70B model ([33]). To construct the naive synthetic dataset, we generate a corresponding search query for each paragraph in our corpus. To prevent data leakage in this synthetic dataset, we remove all paragraphs which are annotated retrieval targets. We include four (query, paragraph) pairs from the proprietary dataset as few-shot examples to guide generation. The model prompt is shown in Figure 6 in Appendix. Fine-tuning on this dataset increases performance on BarExam-QA and LePARD, indicating that there is some signal about legal similarity in that synthetic dataset. However, training on this dataset substantially decreases performance on the two public defense test sets.

Fine-tuning on optimized synthetic dataset.

Next, we construct an optimized synthetic dataset. We fine-tune a Llama3-70B model using annotated (query, paragraph) pairs obtained in the Resource Library evaluation rounds. Input to the model is an annotated paragraph, output is the query for which the paragraph was retrieved. We use the same system prompt (shown in Figure 6 in Appendix). During this fine-tuning process, the model learns to generate more realistic queries. Next, we filter all (query, paragraph) pairs with a Qwen3-8B reranker fine-tuned on the proprietary NJOPD dataset, and discard all generated examples below a certain threshold. After further inspection of the dataset, we find that paragraphs containing facts, tables of contents, or other procedural content rarely appear in the annotated search results. Thus, we also filter out such paragraphs using a zero-shot Llama3-70B model. Eventually, fine-tuning on the resulting synthetic dataset improves performance on public defense datasets.

Query Expansion.

Following ([9]), we also experimented with query expansion strategies. Using Llama3-70B, we expand each query by first applying the IRAC framework (issue, rule, application, conclusion), a well-known method for legal analysis: spot the issue, identify the relevant legal rule, apply the rule to the issue, draw the conclusion. After IRAC, we then derive an expanded search query. Input to the model is the concatenation of the original query, the IRAC analysis and the expanded query. We show the system prompt and one augmented query example in Figure 9 in Appendix. We expand all public defense queries in the test set, and all curated synthetic queries. Next, we fine-tune models on that set. The E5-large-v2 model approaches the performance of the larger Qwen3-Embedding-8B model in this regime of fine-tuning with expanded queries. In the zero-shot setting, we find mixed results where query expansion increases recall for smaller models, but recall decreases for larger model.

Overall comparison.

Leveraging domain knowledge, we can improve retrieval quality. We show that query expansion eliciting IRAC traces and an optimized synthetic dataset both lead to performance increases. However, training on existing legal benchmarks and naively generated synthetic data decreases quality. We speculate this is caused by a domain shift: these datasets are simply too different from public defense retrieval. We alluded to this phenomenon in Figure 1 and provide further evidence by showing queries from all datasets in Table 13 in Appendix. We show exact results (instead of average performance) for all models in Table 7 in Appendix.

Robustness of the PD Dataset

Results between the proprietary NJOPD dataset and the released PD dataset are highly correlated. In zero-shot settings, the resulting Spearman R is 0.79 (p=0.02) for Recall@5, for fine-tuning experiments, the Spearman R consists of 0.82 (p=9.1e-6) across 20 fine-tuning experiments (four models and five different training datasets). We also report retrieval results on a non-anonymized version of the PD dataset, and report results in Table 11 in Appendix. The resulting Spearman R of the anonymized and non-anonymized version is 1.0 in the zero-shot setting (perfect correlation), the correlation after 20 fine-tuning experiments results in a Spearman R of 0.99 (p=2.0e-16). We believe these results together confirm the validity of the benchmark, although it has been annotated by the author team and undergone anonymization.

Since NJOPD documents span 25 years, and our PD dataset only contains briefs from 2023–2025, we also verify that retrieval performance is robust across different time periods of the NJOPD dataset. We report Spearman R between the PD dataset and NJOPD subsets stratified by year in Table 6 in Appendix. Correlations between the datasets are not driven by old NJOPD documents, but also generalize to more recent retrieval targets.

In all fine-tuning experiments, we train models five times with different seeds. In the main text, we consistently report the mean results of these five runs. In Table 8 in Appendix, we show the 95% confidence intervals. In almost all experiments, differences to zero-shot results of the same base model are significant. The confidence intervals of the fine-tuning experiments are moderate, with an average of 0.70 percentage points for Recall@5 for the NJOPD dataset, and an average of 0.83 percentage points for the PD dataset. These are comparable to the confidence intervals of the BarExam QA dataset (0.68 percentage points).

Given the practitioner-facing nature of the PD dataset, we report Recall@5 as the official metric (the Resource Library by default returns five search results). Recall@5 is highly correlated with other standard information retrieval metrics, including Normalized Discounted Cumulative Gain (NDCG@5), Mean Reciprocal Rank (MRR@10), and Mean Average Precision (MAP@100). The lowest of these correlations with the reported Recall@5 results is between MRR@10 and Recall@5, with a Spearman R of 0.97 (p=2.6e-12). We show detailed results for all metrics in Table 9 in Appendix.

Domain Adaptation.

We additionally experiment with legal domain adaptation, another method to add domain knowledge to models. We take a ModernBERT-large checkpoint further pre-trained on 30B tokens of US case law opinions using the masked language modeling objective from ([34]). Fine-tuning this domain-adapted checkpoint on the optimized synthetic dataset achieves a Recall@5 of 23.4, outperforming a vanilla ModernBERT-large checkpoint fine-tuned on the same dataset (Recall@5=18.7, -4.7 percentage points). We show exact results in Table 10 in Appendix.

5.2 Reranker Experiments

We evaluate six open-source rerankers on the NJOPD dataset. We additionally fine-tune the most performant reranker on the proprietary dataset, showing that this further improves performance. We use an 80-20 split to separate queries into a training and test set (stratified by queries, to make sure that no training queries appear in the test set). We then calculate the F1 score for all models. We show the main results in Figure 5.

We compare all results to a simple majority baseline. The baseline assumes a simple heuristic which treats each search result as a good result. We plot the F1 score of this majority baseline as a horizontal dashed line (at 80.3% F1). We observe that most existing rerankers underperform the simple majority baseline, with the only exception being the recently released Qwen3-8B reranker ([15]), which slightly surpasses this baseline. In contrast, fine-tuning on domain-specific data leads to significant improvements. Detailed metrics (precision, recall, accuracy, and F1) for all models are provided in Table 12 in Appendix.

6. Discussion

Section Summary: The discussion highlights a mismatch between existing legal retrieval benchmarks and the real needs of public defenders, noting that most prior datasets lack realistic queries or verified examples from actual practice. To address this, the authors release a new dataset of defender-written queries paired with relevant paragraphs, along with a taxonomy of search goals, developed through direct collaboration with a public defender's office. They also outline promising future directions such as query expansion, multi-step agentic search, and improved models, while stressing the broader value of academic-agency partnerships for creating practically useful AI tools.

6.1 Difference between Existing Benchmarks and Public Defense Retrieval

We find a distribution mismatch between existing legal retrieval datasets and our public defender search dataset, where training on legal benchmarks ([9, 8]) results in lower performance. Similarly, [35] note that outdated medical domain benchmarks are inadequate to assess current AI, and [17] emphasize that most existing benchmarks lack real-world utility. Especially for public defense work, there seem to be no suitable benchmarks to advance AI and NLP methods. Moreover, most legal retrieval datasets do not contain manually verified retrieval targets, and none contain real-world queries (Table 3 in Appendix). While the field of NLP and AI for public good and access to justice is growing ([36, 4, 8]), such efforts are limited by the availability of realistic data.

To make progress on this front, we provide several starting points to stimulate further research on public defense retrieval. We release the PD dataset containing realistic queries drafted by experienced public defenders, and manually verified corresponding paragraphs relevant to those queries. Performance on the PD dataset is correlated with a proprietary dataset created by public defenders. Next, we construct a taxonomy about search objectives of defenders and what search strategies can be employed to answer these queries. Similar to datasets like WildChat ([37]), which make real-world ChatGPT conversations accessible for research, we hope our dataset and taxonomy can inform future work in more realistic legal retrieval settings.

6.2 Collaborations between Academia and Legal Institutions

This work illustrates a collaboration between academic researchers and a public agency. We believe such partnerships can be mutually beneficial: Public agencies get to be involved in reflecting on their existing workflows and AI needs, and gain exposure to the opportunities, trade-offs, risks and barriers of AI applications.

For researchers, such collaborations allow work on AI tasks and use cases that are more closely aligned with real institutional needs and constraints. These are often absent in existing datasets and AI benchmarks, especially in the legal domain (See Table 3 in Appendix). Through frequent meetings and discussions, knowledge transfer benefits both sides: agencies clarify their AI needs, and researchers gain insight into tacit constraints that can inform better AI methods.

We believe such collaborations represent a promising research direction to work on more realistic AI applications with practical utility ([17]). As NLP and AI techniques mature and increasingly promise real-world impact, progress is often limited by the lack of realistic datasets and evaluation settings. We view our collaboration with NJOPD, and the accompanying data artifacts presented in this paper as one step toward addressing this gap.

6.3 Future Work To Improve Public Defense Retrieval

[9] find substantial gains leveraging query expansion and legal reasoning to retrieve relevant statutes for bar exam questions. We confirm this for public defense retrieval, and believe there's value in exploring such efforts in more detail. Moreover, we believe agentic search ([28]), which combines multiple retrieval steps into a single agentic workflow. During this process, the agent can explore e.g., query expansion, legal reasoning, multihop retrieval, and reranking, until a suitable search result is retrieved.

Second, the current state-of-the-art embedding models, i.e., E5-mistral ([32]), Qwen3-Embedding ([15]) and NV-Embed-v2 ([14]) already perform substantially better than smaller, older models built on top of BERT or RoBERTa models, as shown in Table 2. We believe further such advances, and especially advances in dedicated legal retrieval models, may further increase performance on public defense retrieval.

Synthetic data likely can be leveraged to further improve model performance in legal retrieval. However, we note two challenges: The first is that synthetic data must be carefully curated, as we have shown in Section 5. In Section 2, we described how NJOPD has experimented with a RAG tool, but ultimately rejected the idea: models often were imprecise in addressing specific legal questions. It appears that current models lack the ability to handle nuanced legal contexts (see also [5]), which will affect the generated data. This remains an open research problem.

6.4 Related Work

Our work contributes to a growing academic field on how to use legal NLP and AI in collaboration with public agencies. Related work includes AI assistance for automatically detecting and redacting racial covenants ([38]), automatically clearing records at scale ([39]) or AI assistance for eviction defense ([40]). Similar to our work, these projects also identified a suitable use case for AI assistance, and in collaboration with public agencies developed specialized methods to accomplish the goal.

Moreover, we release a dataset for legal retrieval, and contribute to the academic literature on legal retrieval in the United States ([12, 8, 13, 9]). In contrast to these works, we focus on retrieval for public defenders, where the goal is to retrieve relevant paragraphs from legal briefs. Following the recommendations made in [17], we have put an emphasis on real-world utility while creating this benchmark.

We do acknowledge the potential of NLP and AI methods for other types of public defense work. [5] outline a research agenda for how AI can assist public defenders, and foremost identify use cases around making sense of large volumes of data in evidence investigation. Similar to our work, public defenders in ([5]) report similar challenges of using AI for legal research and writing, however this can change rapidly as technology advances.

7. Conclusion

Section Summary: This paper explores computer tools to help public defenders find useful legal documents, developed through collaboration with the New Jersey Office of the Public Defender. The authors built a search system for internal briefs, collected real queries from defense work, and released a new dataset to support further research. Their tests showed that methods like search expansion and legal-specific model tweaks improve results, but general legal datasets often reduce performance due to mismatches with actual public defense needs, suggesting that data differences and limited datasets are bigger obstacles than model size.

In this paper, we discuss legal retrieval for public defenders. In collaboration with the New Jersey Office of the Public Defender, we identify retrieval over internal briefs as a suitable use case to assist public defense work, and developed the OPD Resource Library. The development of this tool allowed us to gather realistic public defense queries, from which we manually construct and release the PD retrieval dataset.

Query expansion, carefully curated synthetic datasets and legal domain adaptation increase retrieval performance, while in-domain fine-tuning increases reranking accuracy. However, training on existing academic legal retrieval datasets lowers performance, indicating a distribution shift between these benchmarks and the more realistic PD retrieval task. Our results suggest that progress in legal retrieval for public defense may be constrained less by model scale than by domain mismatch and lack of available datasets.

8. Acknowledgements

Section Summary: The authors express gratitude to their key partners at the New Jersey Office of the Public Defenders, especially Jennifer Perez and Alison Perrone, whose involvement made the project possible. They also credit Walker Gosrich from the NJ Innovation Authority and John Seith from the NJ Office of Information Technology for providing essential technical support. Finally, they thank Joe Krakora for his valuable insights and steady guidance over the course of the work.

We thank all our partners at the New Jersey Office of the Public Defenders, most importantly Jennifer Perez and Alison Perrone without whom this work would not have been possible. Moreover, we would like to thank Walker Gosrich from NJ Innovation Authority and John Seith from the NJ Office of Information Technology for their technical assistance, and Joe Krakora for his insights and guidance throughout the project.

9. Broader Impact Statement

Section Summary: This work creates a new NLP task and benchmark for retrieving relevant legal documents to support public defenders, along with an AI tool that assists with research by surfacing appellate briefs while deliberately avoiding generative AI for core legal decisions. The tool incorporates multiple safeguards, such as disclaimers and restricted use of AI summaries, to minimize risks like hallucinations, and the released dataset aims to enable realistic future research despite limitations from its origins in a single office. The authors stress the need for secure, closed systems using open-source models to protect client confidentiality in any similar justice-related AI deployments.

This work introduces public defense retrieval as an NLP task and releases an associated benchmark. We have built the OPD Resource Library, an AI-powered retrieval tool to surface relevant appellate briefs to support defenders'day-to-day legal research, while taking into account high stakes and low error tolerance of public defense work. The tool is deliberately designed to not be a AI decision-making system, but AI assistance mimicking existing office practices. The tool serves as an interface to the office's collective institutional knowledge and mirrors existing word-of-mouth practices.

The Library avoids using generative AI for substantive legal tasks. Exploratory evaluation revealed unacceptable failure rates in RAG-based approaches (hallucinations, failure to handle nuanced legal contexts), consistent with qualitative findings defenders report in [5]. LLM-generated summaries are included in the tool only to help defenders triage results. We implement multiple additional safeguards to encourage responsible usage: (1) We specify in the Library instructions that defenders shall not rely on AI-generated summaries for details, but for triaging whether returned documents are relevant. (2) The summaries are not visible by default, but have to be expanded. (3) If expanded, a user first sees a disclaimer (in bold and large font-size) that the summary is AI-generated and might be incomplete, factually incorrect or contain hallucinations. (4) We added CSS safeguards to prevent that these summaries can be copy-pasted.

The released PD dataset enables future research in more realistic legal retrieval settings. Performance on the public dataset is strongly correlated with a proprietary defender-annotated dataset, suggesting that it provides a valid signal for benchmarking. We hope it also serves as a starting point for similar deployments at other defender offices. However, we acknowledge that the benchmark stems from a single office in one U.S. state. Generalization to other jurisdictions or office contexts should thus always be verified. Moreover, the benchmark might be biased (1) towards retrieval targets specific to New Jersey, (2) specific annotation idiosyncrasies of the annotating defenders and the author team, and (3) we relied on automatic filtering using an ensemble of GPT-4o and a fine-tuned Qwen3-8B reranker to further reduce the number of paragraphs we manually reviewed. These limitations need to be considered in future usage of this benchmark.

Finally, any deployment of AI tools in public defense must carefully consider confidentiality. Using proprietary APIs risks falling outside attorney-client privilege and may be subject to mandatory disclosure ([20, 21]). The Resource Library was deployed as a closed system within existing secure NJ State infrastructure. Thus, the tool is only accessible to NJOPD attorneys, and relies on open-source models. We recommend other projects developing access to justice AI tools to take into account constraints around sensitive data, privilege and confidentiality, and to adopt similar safeguards.

Appendix

Section Summary: The appendix presents a series of tables that compare the authors' PD Dataset against prior legal retrieval collections, highlighting its unique use of real queries and manually verified targets drawn from appellate briefs. It also reports basic statistics on query and paragraph lengths for both proprietary and released versions, shows example queries used to build a search taxonomy, and includes correlation results linking the new data to older court opinions. Additional tables detail how fine-tuning embedding models on various legal sources affects recall performance across test sets, with confidence intervals provided for the main experiments.

\begin{tabular}{lccc}
\toprule
\textbf{Dataset} & \textbf{Domain} & \textbf{Real queries} & \textbf{Manually verified targets} \\
\midrule
LePaRD ([8]) & US case law & \ding{55} & \ding{55} \\
CLERC ([13]) & US case law & \ding{55} & \ding{55} \\
Law Search as Prediction ([41]) & US case law & \ding{55} & \ding{55} \\
AirBench ([42]) & Pile of Law & \ding{55} & \ding{55} \\ 
BVA citation prediction ([43]) & Board of Veterans' Appeals & \ding{55} & \ding{55} \\
Contract Summarization ([44]) & Contracts & \ding{55} & \ding{55} \\
COLIEE ([12]) & US case law / statutes & \ding{55} & \ding{51} \\
BarExam-QA ([9]) & US case law / bar exams & \ding{55} & \ding{51} \\ \hdashline
PD Dataset (ours) & Appellate briefs & \ding{51} & \ding{51} \\

\bottomrule
\end{tabular}

: Table 4: Dataset Statistics

Statistic	Proprietary Dataset	Released Dataset
Number of queries	194	170
Average gold paragraphs per query	2.9	3.2
Average query length (words)	8.9	9.3
Average paragraph length (words)	133.9	155.0
Type-Token Ratio (TTR)	0.15	0.13

: Table 5: Examples of Queries and Annotations for Taxonomy Construction

Query	Objective	Search Strategy
Standard for ordering passenger out of a car	standard	embeddings
803(c)(27)	rule	keyword
inevitable discovery	doctrine	keywords / embeddings
find briefs about community caretaking	topical search	keywords / embeddings
What are arguments against consent searches during illegal car stops?	legal argument	embeddings
Difference between reasonable suspicion and probable cause	term clarification	embeddings
when was NERA amended to enumerate the offenses subject to its provisions?	Factual Answer	agentic
has Counterman v. Colorado been addressed in a published new jersey opinion?	Factual Answer	agentic
booking exception to miranda	exception	embeddings
what is the definition of probable cause?	definition	embeddings
Is state v. pena-flores still good law?	good law	embeddings / agentic

: Table 6: Spearman R Correlations between NJOPD Subsets (Stratified by Release Date) and the PD Dataset

NJOPD Period	Queries	Retrieval Targets	Spearman R (p-value)
All years (overall)	194	563	0.79 (0.02)
1998--2019	114	357	1.00 (<0.001)
2020--2025	70	137	0.88 (0.004)
2023--2025	46	66	0.61 (0.108)

\begin{tabular}{llcccc}
\toprule
\textbf{Train Dataset} & \textbf{Model} & \textbf{BarExam QA} & \textbf{LePaRD} & \textbf{Internal NJOPD} & \textbf{PD Dataset} \\
\midrule

\multirow{4}{*}{BarExam QA}
   {} & all-mpnet-base-v2 & 4.52 (+2.90) & 15.75 (+1.44) & 21.80 (+2.07) & 19.67 (+0.37) \\
   {} & E5-base-v2 & 3.23 (-1.61) & 16.82 (+0.47) & 28.06 (+0.62) & 25.46 (+0.23) \\
   {} & E5-large-v2 & 3.23 (+1.61) & 18.64 (+1.11) & 32.03 (+2.43) & 28.11 (+0.71) \\
   {} & Qwen3-0.6B & \textbf{11.94} (+3.87) & 20.51 (+1.41) & 22.66 (-8.26) & 25.37 (-3.98) \\
\hdashline
\multirow{4}{*}{LePaRD}
   {} & all-mpnet-base-v2 & 3.39 (+1.77) & 32.03 (+17.71) & 18.17 (-1.56) & 17.35 (-1.96) \\
   {} & E5-base-v2 & 4.84 (+0.00) & 28.82 (+12.47) & 9.59 (-17.85) & 7.46 (-17.77) \\
   {} & E5-large-v2 & 5.97 (+4.35) & 36.52 (+18.99) & 20.01 (-9.59) & 19.41 (-7.99) \\
   {} & Qwen3-0.6B & 7.10 (-0.97) & \textbf{41.70} (+22.60) & 18.08 (-12.84) & 21.04 (-8.32) \\
\hdashline
\multirow{4}{*}{Naive Synthetic}
   {} & all-mpnet-base-v2 & 5.97 (+4.35) & 19.81 (+5.49) & 23.39 (+3.67) & 22.42 (+3.12) \\
   {} & E5-base-v2 & 3.39 (-1.45) & 18.62 (+2.28) & 20.37 (-7.07) & 17.57 (-7.66) \\
   {} & E5-large-v2 & 8.71 (+7.10) & 21.14 (+3.60) & 25.35 (-4.25) & 23.76 (-3.64) \\
   {} & Qwen3-0.6B & 6.61 (-1.45) & 22.90 (+3.80) & 22.46 (-8.47) & 21.45 (-7.91) \\
\hdashline
\multirow{4}{*}{Optimized Synthetic}
   {} & all-mpnet-base-v2 & 7.90 (+6.29) & 19.68 (+5.37) & 27.28 (+7.56) &
  24.53 (+5.23) \\
   {} & E5-base-v2 & 3.23 (-1.61) & 18.23 (+1.88) & 27.99 (+0.54) & 24.42 (-0.80) \\
   {} & E5-large-v2 & 7.26 (+5.65) & 21.28 (+3.74) & 29.84 (+0.24) & 31.09 (+3.68) \\
   {} & Qwen3-0.6B & 6.77 (-1.29) & 22.31 (+3.21) & 26.54 (-4.38) & 31.99 (+2.64) \\
\hdashline
\multirow{4}{*}{Query Expansion}
   {} & all-mpnet-base-v2 & n/a & n/a & 31.89 (+12.17) & 26.48 (+7.18) \\
   {} & E5-base-v2 & n/a & n/a & 31.65 (+4.21) & 27.82 (+2.6) \\
   {} & E5-large-v2 & n/a & n/a & \textbf{33.71} (+4.10) & \textbf{36.26} (+8.86) \\
   {} & Qwen3-0.6B & n/a & n/a & 27.03 (-3.90) & 35.32 (+5.97) \\

\bottomrule
\end{tabular}

\begin{tabular}{llcccc}
\toprule
\textbf{Train Dataset} & \textbf{Model} & \textbf{BarExam QA} & \textbf{LePaRD} & \textbf{Internal NJOPD} & \textbf{PD Dataset} \\
\midrule

\multirow{4}{*}{BarExam QA}
   {} & all-mpnet-base-v2 & 4.52 ($\pm{0.9}$)\textbf{ & 15.75 ($\pm{0.04}$)} & 21.80 ($\pm{0.25}$)\textbf{ & 19.67 ($\pm{0.32}$)} \\
   {} & E5-base-v2 & 3.23 ($\pm{0.0}$)\textbf{ & 16.82 ($\pm{0.03}$)} & 28.06 ($\pm{0.32}$)\textbf{ & 25.46 ($\pm{0.54}$)} \\
   {} & E5-large-v2 & 3.23 ($\pm{0.0}$)\textbf{ & 18.64 ($\pm{0.03}$)} & 32.03 ($\pm{0.64}$)\textbf{ & 28.11 ($\pm{0.21}$)} \\
   {} & Qwen3-0.6B & \textbf{11.94} ($\pm{0.84}$)\textbf{ & 20.51 ($\pm{0.07}$)} & 22.66 ($\pm{0.60}$)\textbf{ & 25.37 ($\pm{0.73}$)} \\
\hdashline
\multirow{4}{*}{LePaRD}
 {} & all-mpnet-base-v2 & 3.39 ($\pm{0.45}$)\textbf{ & 32.03 ($\pm{0.02}$)} & 18.17 ($\pm{0.33}$)\textbf{ & 17.35 ($\pm{0.46}$)} \\
   {} & E5-base-v2 & 4.84 ($\pm{0.0}$)\textbf{ & 28.82 ($\pm{0.02}$)} & 9.59 ($\pm{0.61}$)\textbf{ & 7.46 ($\pm{0.47}$)} \\
   {} & E5-large-v2 & 5.97 ($\pm{1.82}$)\textbf{ & 36.52 ($\pm{0.08}$)} & 20.01 ($\pm{0.67}$)\textbf{ & 19.41 ($\pm{1.29}$)} \\
   {} & Qwen3-0.6B & 7.10 ($\pm{1.49}$) & \textbf{41.70} ($\pm{0.17}$)\textbf{ & 18.08 ($\pm{1.29}$)} & 21.04 ($\pm{1.27}$)** \\

\hdashline
\multirow{4}{*}{Naive Synthetic}
 {} & all-mpnet-base-v2 & 5.97 ($\pm{0.55}$)\textbf{ & 19.81 ($\pm{0.03}$)} & 23.39 ($\pm{0.15}$)\textbf{ & 22.42 ($\pm{0.33}$)} \\
   {} & E5-base-v2 & 3.39 ($\pm{0.45}$)\textbf{ & 18.62 ($\pm{0.05}$)} & 20.37 ($\pm{0.48}$)\textbf{ & 17.57 ($\pm{0.45}$)} \\
   {} & E5-large-v2 & 8.71 ($\pm{1.10}$)\textbf{ & 21.14 ($\pm{0.12}$)} & 25.35 ($\pm{0.98}$)\textbf{ & 23.76 ($\pm{2.25}$)} \\
   {} & Qwen3-0.6B & 6.61 ($\pm{1.31}$)\textbf{ & 22.90 ($\pm{0.11}$)} & 22.46 ($\pm{1.63}$)\textbf{ & 21.45 ($\pm{1.67}$)} \\

\hdashline
\multirow{4}{*}{Optimized Synthetic}
  {} & all-mpnet-base-v2 & 7.90 ($\pm{0.45}$)\textbf{ & 19.68 ($\pm{0.04}$)} & 27.28 ($\pm{0.41}$)\textbf{ & 24.53 ($\pm{0.63}$)} \\
   {} & E5-base-v2 & 3.23 ($\pm{0.0}$)\textbf{ & 18.23 ($\pm{0.08}$)} & 27.99 ($\pm{0.36}$)** & 24.42 ($\pm{0.99}$) \\
   {} & E5-large-v2 & 7.26 ($\pm{0.71}$)\textbf{ & 21.28 ($\pm{0.08}$)} & 29.84 ($\pm{1.37}$) & 31.09 ($\pm{0.55}$)** \\
   {} & Qwen3-0.6B & 6.77 ($\pm{0.90}$)\textbf{ & 22.31 ($\pm{0.05}$)} & 26.54 ($\pm{0.62}$)\textbf{ & 31.99 ($\pm{0.49}$)} \\

\hdashline
\multirow{4}{*}{Query Expansion}
  {} & all-mpnet-base-v2 & n/a & n/a & 31.89 ($\pm{0.46}$)\textbf{ & 26.48 ($\pm{0.74}$)} \\
   {} & E5-base-v2 & n/a & n/a & 31.65 ($\pm{0.71}$)\textbf{ & 27.82 ($\pm{0.95}$)} \\
   {} & E5-large-v2 & n/a & n/a & \textbf{33.71} ($\pm{1.09}$)\textbf{ & \textbf{36.26} ($\pm{0.53}$)} \\
   {} & Qwen3-0.6B & n/a & n/a & 27.03 ($\pm{1.06}$) \textbf{ & 35.32 ($\pm{1.85}$)} \\

\bottomrule
\end{tabular}

::: {caption="Table 9: Different IR Evaluation Metrics"}

:::

\begin{tabular}{llcccc}
\toprule
\textbf{Train Dataset} & \textbf{Model} & \textbf{BarExam QA} & \textbf{LePaRD} & \textbf{NJOPD} & \textbf{PD Dataset} \\
\midrule
\multirow{2}{*}{BarExam QA}
   {} & ModernBERT-large & 1.61 ($\pm$0.00) & 7.30 ($\pm$0.06) & 1.02 ($\pm$0.18) & 0.26 ($\pm$0.00) \\
   {} & Legal-ModernBERT & 2.42 ($\pm$0.00) & 8.77 ($\pm$0.04) & 1.18 ($\pm$0.00) & 1.78 ($\pm$0.23) \\
\hdashline
\multirow{2}{*}{LePaRD}
   {} & ModernBERT-large & 3.55 ($\pm$0.90) & 32.16 ($\pm$0.12) & 6.85 ($\pm$0.40) & 5.25 ($\pm$0.60) \\
   {} & Legal-ModernBERT & 3.23 ($\pm$0.00) & 32.41 ($\pm$0.12) & 9.46 ($\pm$0.90) & 9.89 ($\pm$0.17) \\
\hdashline
\multirow{2}{*}{Naive Synthetic}
   {} & ModernBERT-large & 3.71 ($\pm$0.55) & 17.30 ($\pm$0.11) & 10.88 ($\pm$0.27) & 11.75 ($\pm$1.11) \\
   {} & Legal-ModernBERT & 4.03 ($\pm$0.00) & 17.80 ($\pm$0.05) & 12.48 ($\pm$0.51) & 16.0 ($\pm$0.83) \\
\hdashline
\multirow{2}{*}{Optimized Synthetic}
   {} & ModernBERT-large & 2.42 ($\pm$0.00) & 17.01 ($\pm$0.08) & 20.34 ($\pm$0.74) & 18.74 ($\pm$0.53) \\
   {} & Legal-ModernBERT & 4.19 ($\pm$0.45) & 17.42 ($\pm$0.05) & 20.82 ($\pm$1.11) & 23.39 ($\pm$0.83) \\
\bottomrule
\end{tabular}

\begin{tabular}{l c c c }
  \toprule
  \textbf{Experiment} & \textbf{Anonymized} & \textbf{Non-anonymized} & $\Delta$ \textbf{(Difference)} \\ \midrule
  all-mpnet-base-v2 & 19.30 & 19.55 & $-$ 0.25 \\
  E5-base-v2 & 25.23 & 25.06 & $+$ 0.17 \\
  E5-large-v2 & 27.40 & 28.25 & $-$ 0.85 \\
  Qwen3-Embedding-0.6B & 29.35 & 29.83 & $-$0.48 \\
  Qwen3-Embedding-4B & 34.19 & 34.10 & $+$0.09 \\
  E5-mistral-7b-instruct & 32.61 & 33.63 & $-$1.02 \\
  NV-Embed-v2 & 31.27 & 31.93 & $-$0.66 \\
  Qwen3-Embedding-8B & 37.08 & 37.37 & $-$0.29 \\ \bottomrule
  \end{tabular}

\begin{tabular}{l|cccc|cccc}
\toprule
\textbf{Model} & \multicolumn{4}{c|}{\textbf{All (\%)}} & \multicolumn{4}{c}{\textbf{Heldout Test (\%)}} \\
   {} & Pr & Rc & F1 & Acc & Pr & Rc & F1 & Acc \\
\midrule
majority baseline & 66.8 & 1.00 & 80.1 & 66.8 & 67.1 & 1.00 & 80.3 & 67.1 \\
bge-reranker-base & 77.4 & 59.8 & 67.5 & 62.1 & 76.5 & 58.4 & 66.2 & 62.4 \\
bge-reranker-large & 77.5 & 56.3 & 65.2 & 60.5 & 78.1 & 56.2 & 65.4 & 62.4 \\
bge-reranker-v2-m3 & 75.7 & 73.6 & 74.6 & 67.1 & 76.2 & 71.9 & 74.0 & 68.1 \\
jina-reranker-v2-base-multilingual & 76.4 & 65.1 & 70.3 & 63.8 & 81.1 & 67.4 & 73.6 & 69.5 \\
Zeroshot Llama 3.1 & 77.9 & 81.7 & 79.7 & 72.7 & 80.5 & 78.7 & 79.5 & 74.5 \\
Zeroshot Qwen3-Reranker-8B & 73.7 & 98.7 & 84.4 & 76.0 & 71.9 & 97.8 & 82.9 & 74.5 \\
\hdashline
Finetuned-Qwen3-Reranker-8B & n/a & n/a & n/a & n/a & 87.0 & 89.9 & 88.4 & 85.1 \\

\bottomrule
\end{tabular}

::: {caption="Table 13: Examples Queries from Different Datasets"}

:::

References

Section Summary: This section compiles a bibliography of sources cited in the paper, including a 1963 Supreme Court ruling on the right to counsel, recent reports on public defense workloads, and numerous academic papers on AI applications in law. The references cover studies of large language models for legal reasoning and document retrieval, benchmarks and datasets for legal NLP tasks, and foundational work in machine learning techniques such as embeddings and visualization. They also include practical guides on AI tools for lawyers and analyses of challenges like precedent handling and model limitations in legal contexts.

[1] Gideon v. Wainwright (1963). 372 U.S. 335. https://supreme.justia.com/cases/federal/us/372/335/.

[2] Nicholas M. Pace et al. (2023). National Public Defense Workload Study.

[3] Rishi Bommasani et al. (2022). On the Opportunities and Risks of Foundation Models. https://arxiv.org/abs/2108.07258. arXiv:2108.07258.

[4] Mahari et al. (2023). The Law and NLP: Bridging Disciplinary Disconnects. In Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 3445–3454. doi:10.18653/v1/2023.findings-emnlp.224. https://aclanthology.org/2023.findings-emnlp.224/.

[5] Inyoung Cheong et al. (2025). How Can AI Augment Access to Justice? Public Defenders' Perspectives on AI Adoption. https://arxiv.org/abs/2510.22933. arXiv:2510.22933.

[6] Guha et al. (2023). LEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems.

[7] Ricardo Dominguez-Olmedo et al. (2025). Lawma: The Power of Specialization for Legal Annotation. https://arxiv.org/abs/2407.16615. arXiv:2407.16615.

[8] Mahari et al. (2024). LePaRD: A Large-Scale Dataset of Judicial Citations to Precedent. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9863–9877. doi:10.18653/v1/2024.acl-long.532. https://aclanthology.org/2024.acl-long.532/.

[9] Zheng et al. (2025). A Reasoning-Focused Legal Retrieval Benchmark. In Proceedings of the 2025 Symposium on Computer Science and Law. pp. 169–193. doi:10.1145/3709025.3712219. https://doi.org/10.1145/3709025.3712219.

[10] Laurens van der Maaten and Geoffrey Hinton (2008). Visualizing Data using t-SNE. Journal of Machine Learning Research. 9(86). pp. 2579–2605. http://jmlr.org/papers/v9/vandermaaten08a.html.

[11] Pedregosa et al. (2011). Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res.. 12(null). pp. 2825–2830.

[12] Kim et al. (2023). COLIEE 2022 Summary: Methods For Legal Document Retrieval And Entailment. In New Frontiers in Artificial Intelligence: JSAI-IsAI 2022 Workshop, JURISIN 2022, and JSAI 2022 International Session, Kyoto, Japan, June 12–17, 2022, Revised Selected Papers. pp. 51–67. doi:10.1007/978-3-031-29168-$5_4. $https://doi.org/10.1007/978-3-031-29168-$5_4$.

[13] Hou et al. (2025). CLERC: A Dataset for U. S. Legal Case Retrieval and Retrieval-Augmented Analysis Generation. In Findings of the Association for Computational Linguistics: NAACL 2025. pp. 7898–7913. doi:10.18653/v1/2025.findings-naacl.441. https://aclanthology.org/2025.findings-naacl.441/.

[14] Lee et al. (2024). NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv preprint arXiv:2405.17428.

[15] Yanzhao Zhang et al. (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. https://arxiv.org/abs/2506.05176. arXiv:2506.05176.

[16] Zhuyun Dai et al. (2023). Promptagator: Few-shot Dense Retrieval From 8 Examples. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. https://openreview.net/forum?id=gmL46YMpu2J.

[17] Ott et al. (2022). Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications. 13(1). pp. 6793.

[18] Katz et al. (2024). GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 382(2270). pp. 20230254. doi:10.1098/rsta.2023.0254. https://doi.org/10.1098/rsta.2023.0254.

[19] Schwarcz, Daniel and Choi, Jonathan H. (2023). AI Tools for Lawyers: A Practical Guide. Minnesota Law Review Headnotes. 108. pp. 1. doi:10.2139/ssrn.4404017. https://ssrn.com/abstract=4404017.

[20] Cheong et al. (2024). (A) I am not a lawyer, but...: engaging legal experts towards responsible LLM policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. pp. 2454–2469.

[21] Adam Paine and Robert M. Travisano (2025). Discovery Pitfalls in the Age of AI. Epstein Becker Green.

[22] Barry C. Edwards (2019). Why Appeals Courts Rarely Reverse Lower Courts: An Experimental Study to Explore Affirmation Bias. Emory Law Journal Online. 68. pp. 1035–1073. https://scholarlycommons.law.emory.edu/elj-online/7.

[23] Li Zhang et al. (2025). Do LLMs Truly Understand When a Precedent Is Overruled?. https://arxiv.org/abs/2510.20941. arXiv:2510.20941.

[24] Smith, Brandon and Troynikov, Anton (2024). Evaluating Chunking Strategies for Retrieval. https://research.trychroma.com/evaluating-chunking.

[25] Jake Poznanski et al. (2025). olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models. https://arxiv.org/abs/2502.18443. arXiv:2502.18443.

[26] Google DeepMind (2026). Gemma 4. https://deepmind.google/models/gemma/gemma-4/. Accessed: 2026-04-30.

[27] Xing Han Lù (2024). BM25S: Orders of magnitude faster lexical search via eager sparse scoring. https://arxiv.org/abs/2407.03618. arXiv:2407.03618.

[28] Xiang Deng et al. (2023). Mind2Web: Towards a Generalist Agent for the Web. https://arxiv.org/abs/2306.06070. arXiv:2306.06070.

[29] OpenAI (2025). Operator System Card. https://cdn.openai.com/operator_system_card.pdf.

[30] Reimers, Nils and Gurevych, Iryna (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3982–3992. doi:10.18653/v1/D19-1410. https://aclanthology.org/D19-1410/.

[31] Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv preprint arXiv:2212.03533.

[32] Wang et al. (2023). Improving Text Embeddings with Large Language Models. arXiv preprint arXiv:2401.00368.

[33] Aaron Grattafiori et al. (2024). The Llama 3 Herd of Models. https://arxiv.org/abs/2407.21783. arXiv:2407.21783.

[34] Dominik Stammbach and Peter Henderson (2026). Legal Domain Adaptation of Modern BERT Models. https://arxiv.org/abs/2606.28538. arXiv:2606.28538.

[35] Yu Gu et al. (2025). The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks. https://arxiv.org/abs/2509.18234. arXiv:2509.18234.

[36] Antonia Karamolegkou et al. (2025). NLP for Social Good: A Survey of Challenges, Opportunities, and Responsible Deployment. https://arxiv.org/abs/2505.22327. arXiv:2505.22327.

[37] Wenting Zhao et al. (2024). WildChat: 1M ChatGPT Interaction Logs in the Wild. https://arxiv.org/abs/2405.01470. arXiv:2405.01470.

[38] Faiz Surani et al. (2025). AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County. https://arxiv.org/abs/2503.03888. arXiv:2503.03888.

[39] Code for America (2020). Record Clearance at Scale: How Clear My Record Helped Reduce or Dismiss 144,000 Convictions in California. Accessed: 2025-08-03.

[40] Stanford Law School Legal Design Lab (2025). Empowering Legal Aid: Developing AI Co‑Pilots for Eviction Defense and Reentry Debt Mitigation. Press release / in‑brief article by Stanford Law School. Stanford Law School, “Empowering Legal Aid” initiative description; accessed August 3, 2025. https://law.stanford.edu/press/empowering-legal-aid/.

[41] Dadgostari et al. (2021). Modeling law search as prediction. Artificial Intelligence and Law. 29(1). pp. 3–34.

[42] Chen et al. (2025). AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 19991–20022. https://aclanthology.org/2025.acl-long.982/.

[43] Huang et al. (2021). Context-aware legal citation recommendation using deep learning. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law. pp. 79–88. doi:10.1145/3462757.3466066. https://doi.org/10.1145/3462757.3466066.

[44] Manor, Laura and Li, Junyi Jessy (2019). Plain English Summarization of Contracts. In Proceedings of the Natural Legal Language Processing Workshop 2019. pp. 1–11. https://www.aclweb.org/anthology/W19-2201.