Databricks for construction: building a project-data lakehouse

The construction data problem

A single heavy-industrial project generates data in a dozen disconnected systems. The critical-path schedule sits in Oracle Primavera P6. Budget and actuals live in an ERP like Sage 300 CRE, Viewpoint Spectrum, or CMiC. RFIs, submittals, and drawings are in Procore, Autodesk Construction Cloud, or Aconex. Daily reports and quantities come off field apps and, more often than anyone admits, off spreadsheets emailed between the trailer and the home office.

None of those systems talk to each other. So the questions that actually matter — is this project going to finish late, is the cost-to-complete drifting, which cost codes are bleeding — require someone to export three reports and reconcile them by hand in Excel. That does not scale past a few projects, and it makes real forecasting impossible because there is no single history to learn from.

Why a lakehouse, and why Databricks

A lakehouse combines the cheap, scalable storage of a data lake with the query performance, ACID transactions, and governance of a data warehouse. That combination matters for construction because your data is both structured (schedule activities, cost codes, invoices) and unstructured (drawings, photos, PDFs) — and a lakehouse handles both on one platform instead of forcing you to run a warehouse and a lake side by side.

Databricks is built on this model, with Unity Catalog providing governance, lineage, and row- and column-level access control across every source. For a contractor that means a project executive sees the whole portfolio, a division president sees their region, and a subcontractor portal sees only its own scope — all enforced centrally, not rebuilt per report.

The medallion architecture, applied to construction

The standard pattern is three layers, and it maps cleanly onto how construction data flows:

• BronzeRaw ingests, landed as-is: P6 XER/XML exports, ERP extracts, Procore and Aconex API pulls, field-app data, and spreadsheet drops. Nothing is cleaned yet — Bronze is the durable record of exactly what each source sent.
• SilverCleaned and conformed: one activity table across every P6 update, one cost-code taxonomy reconciled across projects and ERPs, mapped WBS, deduplicated vendors. This is where the schedule and cost worlds finally line up on shared keys.
• GoldBusiness-ready marts the whole company reads from: portfolio schedule health, cost-to-complete and margin-at-risk, safety leading indicators, and the feature tables that machine-learning models train on. Dashboards and AI agents point here.

What you can build once it is in place

With every source in one governed layer, the analytics that were impossible become routine:

• Schedule analytics: DCMA-style health checks and update-over-update slippage across the whole P6 portfolio, refreshed automatically instead of rebuilt each month.
• Cost-overrun forecasting: machine-learning models that combine schedule slippage, productivity vs. plan, and change-order velocity to forecast where a project lands at completion.
• Safety leading indicators: pattern detection across observations, near-misses, and toolbox talks before an incident, not after.
• Portfolio visibility: one governed source feeding Power BI so executives, PMs, and field supers each see the slice they own.

Build it yourself, or have it built

You can staff a data-engineering team to build and run this, or have it delivered and either handed off or fully managed. Most contractors start managed — the lakehouse, pipelines, and governance are stood up in weeks rather than quarters — and bring it in-house once the team is trained. The important thing is to stage it: land one or two core sources (usually schedule plus cost) in 4–8 weeks, prove value, then add sources without ever taking the business offline.