DevOps & DataOps

Not necessarily. Below about ten services, or without a team to run it, a managed service (ECS Fargate, Cloud Run, Container Apps) gives the same continuous deployment for a fraction of the operational load. Kubernetes is justified by the number of services, a portability need or specific workloads (GPU, large batch). We ask “who runs it at three in the morning?” before asking which tool.

Like an application’s, with two additions. Every change to a transformation (dbt, Spark, SQL) goes through review, code tests and data tests in an isolated environment cloned from production; only modified models and their dependants are rebuilt. Deployment follows write, audit, publish: data is exposed only after passing the checks. Our DataOps article describes this approach in detail.

By instrumenting inputs and outputs first (requests, jobs, pipelines) with OpenTelemetry, centralising logs, metrics and traces in a tool the team already reads (Grafana, Datadog, CloudWatch), then defining three or four service indicators per application and the alerts that protect them. For data: freshness, volume, schema, quality. Twenty useful alerts beat a thousand generic ones.

The same principles (version, test, deploy automatically, observe) applied to three objects instead of one: code, data and model. That adds tracking of training and evaluation sets, a model registry, performance tests before promotion and drift monitoring in production. For generative AI the model is often external, but prompts, evaluation sets and indexes are versioned the same way.

Frequency is not a goal; it is the result of small changes, automated tests and a simple rollback. The teams we support often go from a dreaded monthly release to several a week within a few months, with no extra incidents, because each release has become small and observed.