Private LLM That Eliminated 10,000+ Hours
How a leading FinTech firm used Mosaic AI and Databricks to build a private, GDPR-compliant LLM that processes complex regulatory filings in seconds — saving over 10,000 manual hours annually.
The Challenge
Drowning in Regulatory Documents
The client processes hundreds of complex regulatory filings every week — Basel IV capital adequacy reports, FCA compliance submissions, and internal credit risk assessments. Each document runs to hundreds of pages of dense financial and legal language. A team of 14 highly-paid compliance analysts spent the majority of their time reading, summarising, and cross-referencing these documents. The work was critical, error-prone, and deeply unsatisfying for the people doing it. Leadership wanted to redirect analyst time to higher-value judgement work — but couldn't afford to use a public LLM due to the highly sensitive nature of the documents.
14 Analysts Stuck in Document Triage
Senior compliance professionals were spending 60–70% of their working week on document ingestion and summarisation — work that required reading comprehension but not the expert judgement these analysts were hired for.
Public LLMs Were Off the Table
The documents contained commercially sensitive financial models, client data, and proprietary risk methodologies. Sending them to OpenAI or any public API was a non-starter from a data security and regulatory perspective.
Inconsistent Output Quality
Different analysts produced different summaries of the same document. When these summaries fed into risk models, the inconsistency introduced noise. Leadership had no way to ensure standardised output quality at scale.
The Approach
A Fully Private, Domain-Specific LLM on Databricks
ComputeLogic designed and built a private LLM infrastructure on Databricks using Mosaic AI — fine-tuned on the client's regulatory document corpus, hosted entirely within the client's own Azure tenant, with zero data leaving the security perimeter. The solution combined RAG (Retrieval-Augmented Generation) for document Q&A with fine-tuning for structured output extraction.
Use Case Definition & Data Audit
Weeks 1–2Worked with the compliance and technology teams to identify the highest-value automation targets. Audited the existing document corpus — 4,200 historical filings — and defined the structured output schema the model needed to produce.
Deliverables
- Prioritised use case matrix (12 document types identified)
- Structured output schema for 5 highest-value document types
- Data audit report (4,200 historical documents catalogued)
- Security and data residency requirements sign-off
RAG Infrastructure & Vector Store
Weeks 3–6Built the RAG pipeline on Databricks — document ingestion, chunking strategy, embedding generation using a private embedding model, and a Vector Search index. All compute and storage remained within the client's Azure tenant.
Deliverables
- Document ingestion pipeline (PDF, Word, HTML support)
- Private embedding model deployed on Databricks
- Databricks Vector Search index (4,200 historical documents)
- RAG Q&A interface for compliance analyst testing
Fine-Tuning & Evaluation
Weeks 7–12Fine-tuned a base open-source LLM (Llama 3) on the client's regulatory document corpus using Mosaic AI's training infrastructure. Built an automated evaluation harness using MLflow to track model quality across 200 held-out test documents.
Deliverables
- Fine-tuned Llama 3 model (private, hosted on Databricks)
- MLflow evaluation harness (200-document test suite)
- Model card and capability documentation
- Structured extraction accuracy: 94.3% on test set
Production Deployment & Integration
Weeks 13–16Deployed the model behind a Databricks Model Serving endpoint, integrated with the client's existing document management system, and built a simple analyst-facing web interface. Ran a 4-week parallel operation period before full handover.
Deliverables
- Production model serving endpoint (99.9% uptime SLA)
- Integration with existing document management system
- Analyst-facing web interface (React, internal deploy)
- LLMOps monitoring dashboard in Databricks
The Results
10,000 Hours Returned to the Business
Within 60 days of production deployment, the model was processing 94% of routine regulatory filings without analyst intervention. The 14-person compliance team shifted from document triage to model oversight and exception handling — doing the work they were hired for.
Routine regulatory document processing automated, freeing 14 compliance analysts from triage work and redirecting them to higher-value judgement tasks.
The fine-tuned model achieves 94.3% structured extraction accuracy on the held-out test set — exceeding the benchmark set by senior analyst output consistency.
Average processing time for a 200-page regulatory filing dropped from 45 minutes of analyst time to 8 seconds of model inference.
Entire LLM infrastructure deployed within the client's Azure tenant. No document, query, or output ever touches a public API.
“We had tried three different AI vendors before ComputeLogic. None of them could solve the data security problem. ComputeLogic built us something we actually own — running entirely in our own infrastructure. The accuracy is better than our analysts on routine documents, and our analysts now spend their time on work that actually requires their expertise.”
Tech Stack
Focus Areas
Ready for similar results?
Book a free 30-minute audit and we'll map your data estate to outcomes like these.
Request Free AuditWork With Us
Let's Architect Your Next Breakthrough
Every engagement starts with a free 30-minute audit. No pitch decks — just an honest assessment of where your data estate is today and what it could become.