DataOps: finally bringing DevOps practices to data engineering

DataOps: finally bringing DevOps practices to data engineering

CI/CD, quality as code, observability: three pillars to make your data pipelines as reliable as your applications.

Data engineering has long operated outside modern DevOps practices. While application teams adopted continuous integration, automated testing, infrastructure as code and production monitoring, data pipelines too often remained hand-deployed SQL scripts, scheduled jobs without tests, and dashboards whose errors were discovered… in the board meeting.

Those days are over. Data now feeds critical decisions, customer-facing products, regulatory obligations and, increasingly, AI models and agents. Wrong data is no longer a minor inconvenience: it is a production incident. And a production incident is prevented with the same disciplines as in software.

That is precisely the point of DataOps: applying to the data value chain the principles that have proven themselves on the DevOps side (automation, testing, fast feedback, shared responsibility), adapted to one major specificity: in a data pipeline, it is not only the code that can break, it is the data too.

Why data fell behind

Data engineering's lag behind DevOps is not a lack of skill; it has structural causes.

• Historically graphical tools. First-generation ETL tools were configured by drag and drop, in proprietary tools that were hard to version and impossible to unit test.
• A double dependency on code and data. A pipeline can be perfectly correct and still produce a wrong result because the source changed its format, volume or meaning. Classic tests are not enough.
• Environments that are expensive to duplicate. Reproducing a multi-terabyte warehouse to test a change seemed out of reach.
• A culture of “it runs”. As long as the job does not crash, everything is assumed to be fine. The question “is the delivered data correct?” never gets a systematic answer.

The cloud and the modern data stack have removed most of these obstacles: transformations written in SQL or Python, hence versionable; cloud warehouses able to clone tables at low cost; testing and observability tools designed for data. There is no technical excuse left. What remains is to change the practices.

Pillar 1: CI/CD for data pipelines

Every ETL/ELT pipeline must be versioned, tested and deployed automatically. This is the foundation; without it, the other two pillars have nothing to hold on to.

Version everything that defines a pipeline

The transformation code, of course, but also the orchestration configuration, schema definitions, quality rules, access rights and the infrastructure itself (Terraform, Pulumi or equivalent). If a change does not go through Git and a code review, it does not officially exist.

Test before deploying

A continuous integration chain for data checks, on every pull request:

• that the code compiles and follows conventions (SQL linting with SQLFluff, for example);
• that the transformations produce the expected result on test datasets (unit tests);
• that the produced data meets the defined quality rules (data tests, see pillar 2);
• that the change does not break the downstream models that depend on it.

Tools such as dbt make this approach considerably easier thanks to their native tests and their dependency graph. The state comparison feature in particular makes it possible to rebuild and test only the modified models and their descendants (the well-known slim CI), which keeps CI fast and inexpensive:

# .github/workflows/dbt-ci.yml (excerpt)
- name: Build & test modified models
  run: |
    dbt deps
    dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci

Deploy automatically and safely

Deployment to production must be the mechanical consequence of a validated merge, not a manual operation on a Friday evening. Several patterns help limit the risk:

• Isolated environments (dev, CI, staging, production), ideally created on the fly thanks to zero-copy cloning in cloud warehouses.
• The Write-Audit-Publish pattern: data is written to a staging area, audited by the quality tests, then published only if it complies.
• Blue/green deployment of tables or views, to switch over instantly and roll back without reprocessing.

Our recommendation: do not aim for perfection from day one. A first, highly profitable step is to require Git and code review on all transformations, then add a CI that runs the dbt tests on the modified models. This foundation alone eliminates a large share of the regressions we encounter in audits.

Pillar 2: Data Quality as Code

Testing the code is not enough: the data must be tested too. Great Expectations, Soda or dbt tests make it possible to define explicit data contracts, written as code, versioned and executed automatically.

The principle is simple and non-negotiable: a pipeline that delivers poor-quality data should not go to production. Quality becomes a quality gate, just as a failing unit test blocks an application release.

What a data contract looks like

With dbt, quality rules are declared as close as possible to the model, and the schema contract can be enforced at compile time:

models:
  - name: fct_commandes
    config:
      contract:
        enforced: true
    columns:
      - name: commande_id
        data_type: varchar
        data_tests: [unique, not_null]
      - name: client_id
        data_type: varchar
        data_tests:
          - relationships:
              to: ref('dim_clients')
              field: client_id
      - name: statut
        data_type: varchar
        data_tests:
          - accepted_values:
              values: ['créée', 'payée', 'expédiée', 'annulée']

With Soda, the same requirements are expressed in a declarative language that non-technical profiles can read, which makes it an excellent basis for dialogue with business teams:

checks for fct_commandes:
  - row_count > 0
  - missing_count(client_id) = 0
  - duplicate_count(commande_id) = 0
  - freshness(date_commande) < 1d

Great Expectations, for its part, offers a very rich set of expectations and automatically generated documentation, particularly useful in Python and Spark environments.

Where to place the checks

The most effective checks are spread across three points of the pipeline:

1. At the entrance, on the sources: schema, volume, freshness. This is where unannounced changes in upstream systems are caught.
2. During the transformations: uniqueness, referential integrity, business rules. This is where you catch your own mistakes.
3. At publication, on the data exposed to consumers: is the promised contract honoured? This is the last barrier before the user.

Our recommendation: start with the data that matters. Identify the ten most consumed or most critical tables (regulatory reporting, billing, executive dashboards) and cover them first with the basic tests: uniqueness, non-null, freshness, volume. Use severity levels (warning or blocking) to distinguish what must stop production from what should only alert. And above all, have the rules validated by the business: a data contract nobody has signed is just one more test.

Pillar 3: data observability

Tests check what we know how to anticipate. But in production, the most expensive incidents are the ones nobody foresaw: a source that suddenly delivers half as many rows, a column whose distribution drifts, a job that finishes three hours late. Running pipelines in production without observability is flying blind.

What to observe

Data observability generally relies on five signals:

• Freshness: is the data up to date compared with what is expected?
• Volume: is the number of rows received or produced consistent with history?
• Schema: have columns been added, removed, or changed type?
• Distribution: do values stay within plausible ranges (null rates, averages, cardinalities)?
• Lineage: which downstream tables and dashboards are affected by an incident?

Market platforms or a custom solution

Platforms such as Monte Carlo or Bigeye provide machine-learning anomaly detection on these signals, without having to write every rule by hand, along with end-to-end lineage and built-in incident management. They make it possible to detect incidents proactively, before consumers report them.

For organisations that already have an application observability stack, a custom solution based on OpenTelemetry is a credible alternative: pipelines emit traces and metrics (task durations, volumes processed, quality check results) to the same tools as the rest of the IT landscape. Complemented by OpenLineage for lineage, this approach gives a unified view of application and data health, with no dependency on an additional vendor.

The right choice depends on the size of the data estate, the maturity of the team and the cost of an incident. What does not depend on anything is the need to have one.

Organise the incident response

Detecting is useless if nobody reacts. Observability must come with:

• identified owners for each pipeline and each data product, who receive the alerts;
• explicit service levels (for example: “the sales table is available every day at 7 am with a completeness rate above 99.5%”);
• an incident process inspired by application operations: triage, communication to the affected consumers, resolution, post-mortem.

Our recommendation: treat data incidents as production incidents, with the same seriousness and the same rituals. Measure time to detect and time to resolve. And aim for a simple goal: that your data team is the one warning the business about an incident, never the other way round.

Pitfalls to avoid

Pitfall 1: buying a tool to buy a culture

No tool does DataOps for you. An observability platform whose alerts nobody reads, or a test suite that gets disabled as soon as it blocks an urgent delivery, only produces a false sense of security. DataOps is first and foremost a team discipline: systematic code review, refusing to ship without tests, clear ownership of every pipeline.

Pitfall 2: alert fatigue

Conversely, multiplying tests and monitors without any hierarchy drowns teams in notifications. After a few weeks nobody reads them anymore, and the real incident goes unnoticed. Twenty relevant checks, tied to business stakes, beat two thousand generic rules.

Pitfall 3: leaving DataOps to engineers only

The most useful quality rules are business rules: an amount cannot be negative, an active customer has necessarily placed an order in the last twelve months. If the business does not take part in defining them, the checks will stay technical and miss the errors that really matter.

DataOps and AI: a prerequisite for industrialisation

In 2026, the question of data reliability has taken on a new dimension. Generative AI and agentic AI projects consume data continuously: document bases for RAG, transactional data queried by agents, training and evaluation datasets for models.

Yet an AI fed with wrong, stale or incomplete data does not flag the error: it produces a wrong answer with confidence. The three pillars of DataOps therefore apply directly to the pipelines feeding AI. Versioning and testing document ingestion chains, checking the freshness and completeness of vector indexes, watching for drift in the data going into models: these are the same practices, applied to new consumers.

Organisations that already master DataOps move from AI prototype to production much faster. The others discover, when scaling up, that their problem is not the model but the data.

Where do you stand?

A few questions to quickly assess your DataOps maturity:

• Are all your data transformations versioned in Git and subject to code review?
• Can a change be deployed to production without manual intervention, and rolled back just as easily?
• Are your critical tables covered by quality tests that block publication on failure?
• Do you have data contracts validated with producers and consumers?
• Do you know how long it takes to detect a data incident, and who hears about it first?
• Can you tell, within minutes, which dashboards are affected by a failing source?

If several answers are no, the good news is that the first gains come quickly: Git, a CI running dbt tests on the critical models, and freshness and volume alerts are often enough to transform the perceived reliability of the data platform within a few months.

Conclusion

DataOps is neither a fad nor one more tool to add to the stack. It is the recognition of something obvious: data is a production product, and it deserves the same engineering rigour as software.

CI/CD to ship fast without regressions, Data Quality as Code to never publish wrong data, observability to detect incidents before users do: these three pillars reinforce one another. Together, they turn a data team that puts out fires into a team that delivers value with confidence.

The question is no longer “should we industrialise our data pipelines?”, but “how much is it costing us, every month, not to have done it yet?”

Looking to industrialise your data pipelines or assess the DataOps maturity of your platform? Logbrain's experts, a pure player in data, AI and cloud, support you from the audit to the implementation of your CI/CD chains, your data contracts and your observability. Let's talk about your situation