Data Architecture & Engineering

On three measurable criteria: the share of semi-structured or streaming data (logs, events, documents) in your use cases, the skills of the team that will run the platform (SQL only, or Spark and Python too) and the total cost computed on your real queries rather than a price list. A SQL warehouse is enough for many organisations; a lakehouse makes sense when one platform must serve reporting, machine learning and generative AI without duplicating data. Either way, an open table format (Delta, Iceberg) keeps the decision reversible.

Domain by domain, with a double-run period: the old and new pipelines run in parallel and we compare results automatically (counts, aggregates, row-level differences on a sample) until every discrepancy is explained. Cutover happens one dashboard at a time, and the rollback is prepared beforehand, not during. What we do not do: a big bang over a weekend.

When a decision or action depends on data less than a few minutes old: fraud detection, replenishment, operational alerting. For daily or hourly reporting, a dbt batch orchestrated by Airflow is simpler, cheaper and easier to test. We often start with micro-batch and move to event streaming only where latency has a measured value.

By treating pipelines as software: versioned in Git, tested on every change (schema, uniqueness, freshness, volumes), deployed automatically, observed in production. Quality rules are written with business teams as data contracts, and an alert reaches a named owner before a user finds the error. This is the DataOps approach we describe in our articles.

Usually six to ten weeks for a first domain: sources connected, model documented, tests in place and one business use genuinely served. We would rather ship a narrow scope that runs than a complete architecture on paper. The rest is added domain by domain, reusing what has been proven.