Replacing a $42K/mo OpenAI bill with on-prem 70B (anonymized)

A regional law firm — anonymized — was running a document classification and clause extraction pipeline through OpenAI's GPT-4o. Volume: about 80M tokens/month. Bill: $42,000/month, with strict per-matter data handling requirements that made the OpenAI usage uncomfortable for partners and inside counsel.

We migrated to on-prem Llama 3.3 70B with custom calibration. Hardware capex: $52,000 for a workstation with 2× A100 80GB. Amortized over three years plus power and ops: $4,200/month equivalent. Payback period: 9 months. Quality measurably better than the OpenAI baseline on the firm's domain-specific eval.

This is what the engagement looked like.

The original setup

Two distinct workloads:

Document classification. Incoming engagement letters, contracts, court filings — all needed automatic classification into ~30 case types so they could be routed to the right practice group. Roughly 60% of total traffic.

Clause extraction. Once a contract was classified, specific clauses (governing law, indemnification, termination, change-of-control) needed extraction with citation. Roughly 40% of traffic. Higher-stakes — extracted clauses fed downstream review workflows where lawyers acted on them.

Both ran through GPT-4o with carefully tuned prompts. Volume was steady around 80M tokens/month. The system worked; nobody was complaining about quality.

Why they wanted to migrate

Three converging pressures:

1. Cost trajectory. Bill was $42K/mo and growing 4-5% monthly as the firm grew. Trajectory: $65K/mo within 18 months, $100K/mo within 36 months. Unsustainable as an operating expense for a regional firm.

2. Data handling discomfort. Even with OpenAI's enterprise data handling commitments, partners weren't comfortable sending un-redacted client documents to a third-party API. Some matters involved sealed litigation or active arbitration where confidentiality was a hard requirement.

3. Compliance momentum. Bar association guidance was moving toward "if cloud, it must be your cloud" for legal AI. Not a regulation yet but obviously where it was heading.

The engagement scope: replace the API with an on-prem deployment, match or exceed quality, document everything for the firm's compliance program.

What we built

Phase 1 — measurement (week 1-2): Built an eval harness with 800 documents the firm had manually classified and clause-extracted. This became the ground truth. Ran the existing OpenAI system against it; recorded baseline accuracy.

Baseline:

• Classification accuracy: 91.2%
• Clause extraction F1: 84.6%

Phase 2 — hardware procurement (week 3-4): Specced and ordered a workstation: dual Xeon, 256GB RAM, 2× A100 80GB, redundant power, on-prem in the firm's existing server room. Total: $52,000 including a 3-year warranty.

Phase 3 — base model deployment (week 5-6): Stood up Llama 3.3 70B Instruct on the new hardware. Off-the-shelf Q4_K_M from bartowski. Wired to vLLM with continuous batching. Reproduced the existing prompts against the new model.

Baseline (stock Llama 3.3 70B):

• Classification accuracy: 86.4%
• Clause extraction F1: 78.1%

Substantially below GPT-4o. Expected — Llama 3.3 70B is genuinely competitive but not always a drop-in replacement.

Phase 4 — custom calibration (week 7-9): Built a calibration corpus from 3,000 historical documents the firm had archived. Stripped PII. Ran llama-imatrix against the unquantized base. Generated a custom Q4_K_M.

After custom calibration:

• Classification accuracy: 89.7%
• Clause extraction F1: 81.4%

Improved meaningfully. Still below GPT-4o, but closer.

Phase 5 — LoRA fine-tuning (week 10-13): Trained two task-specific LoRA adapters using the firm's labeled data. 4000 examples for classification, 2500 for clause extraction.

After LoRA fine-tuning:

• Classification accuracy: 93.8%
• Clause extraction F1: 87.1%

Both metrics now above the GPT-4o baseline. The fine-tuning closed the capability gap because the firm's specific domain (regional contract law, specific practice areas) was narrower than the general training distribution.

Phase 6 — production cutover (week 14-16): Shadow mode for 2 weeks: production traffic ran through both systems, results compared offline. No customer-facing change.

A/B test for 1 week: 20% of new documents routed to the on-prem system, others to GPT-4o. Quality verified manually by senior attorneys.

Full cutover at week 16.

Results

Production metrics six months post-cutover:

The custom model is actually more accurate than the API baseline because LoRA fine-tuning on domain-specific data beats a general-purpose model. Latency dropped by half because there's no network round-trip to OpenAI.

What the firm owns

End of engagement, the firm has:

• The on-prem hardware, fully operational
• The custom-calibrated Q4_K_M GGUF
• Two LoRA adapters (classification + clause extraction)
• The 3000-document calibration corpus (with PII strip pipeline documented)
• The 6500-example training set for the LoRAs
• The 800-document eval harness
• Runbooks for re-training, re-calibrating, and re-evaluating on a 6-monthly cycle
• A compliance memo describing the data handling for the bar association

The asset that compounds is the eval harness and the labeled data. The model itself will need updating in 18-24 months as Llama 3.3 is superseded; the data assets survive that transition.

What we measured to convince leadership

Three things had to be true for the partners to sign off on the cutover:

1. Accuracy at or above the baseline. Measured against the 800-document eval set. Custom + LoRA setup beat it. Verified.

2. Manual review by 3 senior attorneys. 60 random documents per attorney, comparing GPT-4o output to the on-prem output. Attorneys couldn't reliably pick which was which. Verified.

3. Real-world latency acceptable. Shadow mode confirmed 1-2 second response times under production load. Verified.

The cost case was secondary to these — they wouldn't have moved if quality dropped, regardless of savings.

What goes wrong on this kind of engagement

A few honest failure modes from related engagements:

Insufficient labeled data. A firm without a labeled training set for the LoRA path is stuck at the calibration-only quality level, which sometimes doesn't beat the API baseline. We were lucky here — the firm had been hand-labeling for years.

Operational immaturity. On-prem hardware requires someone watching it. Firms without internal IT capacity end up calling us at 2 AM when the service is down. We now require clients to demonstrate operational capacity before the cutover.

Model deprecation cycle. The next generation of open models will change the cost math again. Engagements that don't include a re-training plan get stranded on an old model. Our standard hand-over now includes a "re-train every 9 months" schedule and the tooling to do it without us.

What this pattern transfers to

Any engagement with:

• High-volume, narrow-domain LLM usage (80M+ tokens/mo)
• Sensitive data
• Labeled training data available
• Compliance pressure pushing toward control
• Operational maturity sufficient to run on-prem

We've shipped variations of this on 5 client deployments now. Healthcare imaging report classification, financial KYC document review, insurance claims triage. Numbers vary; the playbook is identical.