From 24-Hour Lag to 15-Minute Intelligence
How a $2B retailer eliminated its fragmented legacy stack and achieved real-time inventory and pricing intelligence on the Databricks Lakehouse.
The Challenge
The Data Problem at the Heart of a Retail Giant
The client was operating a patchwork of legacy data systems — a decade-old Hadoop cluster, a Snowflake instance added as a quick fix, and a separate Oracle warehouse for financial reporting. Every morning, a team of five analysts spent hours reconciling overnight batch jobs. Business leaders were making pricing and inventory decisions based on data that was 24 hours stale. In a competitive retail landscape where margins are razor-thin and consumer behaviour shifts hourly, this was an existential bottleneck.
24-Hour Reporting Lag
Nightly batch processes meant the merchandising team couldn't react to intra-day demand signals. Flash promotions, stockouts, and competitor pricing changes went undetected until the next morning.
3 Siloed Data Systems
Hadoop, Snowflake, and Oracle each held partial truths. Joining them required hand-crafted ETL pipelines that broke frequently and required specialist knowledge to repair.
Zero Data Lineage
When numbers disagreed between dashboards — which happened weekly — no one could trace the root cause. Trust in data had eroded across the organisation, leading to 'gut feel' decisions at the executive level.
The Approach
A Phased Migration with Zero Business Disruption
ComputeLogic designed a 16-week migration programme using our proven Medallion Architecture blueprint. The critical constraint: the business could not tolerate downtime or data gaps during the peak trading season. We ran legacy and Lakehouse systems in parallel throughout, with automated reconciliation checks at every stage.
Diagnostic & Discovery
Weeks 1–2We performed a full audit of all three source systems — cataloguing 847 distinct tables, mapping 214 downstream report dependencies, and interviewing 23 data consumers across merchandising, finance, and supply chain.
Deliverables
- Data estate inventory (847 tables across 3 systems)
- Dependency map of 214 downstream reports
- Risk matrix for migration sequencing
- Executive alignment workshop
Medallion Architecture Blueprint
Weeks 3–4Designed the target-state Lakehouse architecture on Databricks — a three-layer Medallion model (Bronze raw ingestion, Silver cleansed/conformed, Gold business-ready) with Unity Catalog governance overlaid from day one.
Deliverables
- Target architecture design document
- Unity Catalog namespace and permission schema
- Delta Live Tables pipeline specifications
- Cluster sizing and cost model
Agile Build & Migration
Weeks 5–12Executed migration in five two-week sprints, prioritising the highest-value data domains first (inventory, pricing, transactions). Each sprint delivered production-ready Delta Live Tables pipelines with automated data quality checks.
Deliverables
- 5 production Delta Live Tables pipeline bundles
- Automated DQ checks covering 98.7% of critical metrics
- Legacy system parallel-run reconciliation reports
- Real-time Databricks SQL dashboards for merchandising
Cutover & Knowledge Transfer
Weeks 13–16Managed the hard cutover during a low-traffic window, decommissioned Hadoop and Snowflake connectors, and ran a 3-week internal enablement programme for the client's data engineering team.
Deliverables
- Zero-downtime production cutover
- Decommission plan for legacy systems (saving $340K/year)
- Internal team enablement programme (12 engineers certified)
- Runbook and operational documentation
The Results
The Numbers That Changed the Business
Within 90 days of cutover, the results were transformational. The merchandising team could react to demand signals in minutes rather than the next morning. The data team shrank from five manual reconciliation analysts to one platform engineer. And for the first time, the CFO had a single trusted number.
Reporting latency dropped from 24 hours to 15 minutes — enabling intra-day pricing and inventory decisions for the first time.
Decommissioning Hadoop and Snowflake eliminated redundant licensing and infrastructure costs in year one.
Automated DQ checks across all critical pipelines, replacing manual reconciliation work that consumed 5 analysts' time each morning.
The client's internal team completed ComputeLogic's Databricks enablement programme, ensuring long-term platform ownership.
“ComputeLogic didn't just migrate our data — they changed how we think about it. For the first time in a decade, our merchandising team trusts the numbers they're looking at. That trust is worth more than any infrastructure saving.”
Tech Stack
Focus Areas
Ready for similar results?
Book a free 30-minute audit and we'll map your data estate to outcomes like these.
Request Free AuditWork With Us
Let's Architect Your Next Breakthrough
Every engagement starts with a free 30-minute audit. No pitch decks — just an honest assessment of where your data estate is today and what it could become.