# Skyhook - Full Documentation > Skyhook is a Kubernetes Internal Developer Platform for small-to-mid engineering teams. Build, deploy, and scale applications on Kubernetes without the DevOps complexity. ## About Skyhook Skyhook provides a one-click Kubernetes platform that handles the full application lifecycle - from build through deploy, run, observe, and grow. It eliminates the need for dedicated DevOps teams by providing golden paths, automated deployments, and built-in observability. ### Key Capabilities - **Build**: Automated container builds from source code with zero Dockerfile configuration - **Deploy**: Kubernetes deployments with canary, blue-green, and rolling strategies - **Run**: Runtime management with auto-scaling and self-healing infrastructure - **Observe**: Built-in monitoring, logging, and alerting for all services - **Grow**: Scale infrastructure and teams with service catalogs and platform abstractions - **AI-Ready**: AI agent integration for Kubernetes operations and ChatOps ### Solutions - **Preview Environments**: Ephemeral environments for every pull request, enabling faster code review and QA - **Rollout Strategies**: Production-safe deployments with canary, blue-green, and rolling strategies - **Service Catalog**: Self-service catalog for deploying pre-configured services and infrastructure - **Auto-Scaling**: Automatic horizontal and vertical scaling based on real-time metrics ## Blog Posts ### Migrating from ECS to EKS: Architecture Challenges and Solutions - **URL**: https://www.skyhook.io/blog/migrating-from-ecs-to-eks - **Date**: May 2026 - **Author**: Eyal Dulberg, CTO - **Category**: Kubernetes - **Tags**: aws, ecs, eks, migration, kubernetes, for-devops ECS works. For a small surface area, it works really well. The reason teams keep migrating off it isn't ECS itself - it's that the ecosystem you actually want is on the other side of the kubectl boundary. By 2026, almost every interesting platform tool ships as a Kubernetes-first project: Karpenter, ArgoCD, External Secrets Operator, Crossplane, Backstage, Knative, every modern service mesh, every modern CRD-driven controller. ECS sits outside that ecosystem. You can build the same things on top of it - many teams have - but you build them yourself, and the rest of the industry isn't going to help you maintain it. This post is for the team that's run a production workload on ECS for years and is now seriously planning the move to EKS. We'll skip the marketing comparison and go straight to the parts that catch teams out: networking, IAM, service discovery, autoscaling, deployments, secrets, and logging. For each one we'll show what changes, what to do about it, and the gotchas we've watched real migrations hit. ## TL;DR - The hardest parts of an ECS-to-EKS migration aren't the workloads. They're networking (IP exhaustion with VPC CNI), IAM (Task Roles became IRSA, then Pod Identity), and rebuilding your deployment pipeline around a controller-driven model. - Use a strangler pattern with a shared ALB and weighted target groups. Don't cut over in a single window. - Don't lift-and-shift the deployment process. The ECS one-shot deploy doesn't translate. Adopt GitOps from day one. - The migration is worth doing for the ecosystem, not for the AWS console UX. EKS is more powerful and more work. Plan for both. ## Why teams are migrating in 2026 The ECS-to-EKS conversation used to be about cost and portability. In 2026 it's about three things: 1. **Ecosystem gravity.** Every tool you want to add to your platform - Karpenter for compute, External Secrets for secrets, ArgoCD for deploys, OpenTelemetry collectors, service meshes, autoscalers smarter than CPU - assumes Kubernetes. ECS-equivalent paths exist for some of them but always lag, and often die. 2. **Compute economics.** Fargate is a beautiful abstraction with a markup. At scale, EKS plus Karpenter on Spot lands somewhere between 30% and 60% cheaper for the same workload, depending on what fraction of your traffic is interruptible. 3. **Hiring.** The pool of engineers who know Kubernetes is now meaningfully bigger than the pool who know ECS. New hires already speak `kubectl`. Onboarding documentation writes itself. None of that means ECS is wrong. If your platform is small, your team is two people, and you don't need Kubernetes' extensibility, ECS is still the right call. The teams who should migrate are the ones who keep filing tickets that look like "we want to do X, and we have to write it ourselves on ECS, and someone else already wrote it for Kubernetes." This isn't a fringe move, either. The CNCF's 2025 survey put Kubernetes in production at 82% of container-using organizations, and a string of well-documented ECS-to-EKS migrations have made the path repeatable: - **Figma** moved off ECS to EKS in under 12 months. Their blockers were the ones above made concrete - no StatefulSets, no Helm, no clean way to run OSS like Temporal - and they landed on three EKS clusters with Karpenter for cost and resilience ([Figma engineering blog](https://www.figma.com/blog/migrating-onto-kubernetes/)). - **SailPoint** migrated 100+ microservices, standardizing on Karpenter and KEDA for autoscaling and GitOps with Kustomize and ArgoCD - the same controller-driven deploy model this post argues for ([SailPoint engineering](https://medium.com/sailpointengineering/ecs-to-eks-the-great-container-migration-b0bcf41991e9)). - **ADN**, a streaming platform with spiky release-day traffic, reported roughly a 25% cut in operational cost and faster issue identification after moving to EKS with Karpenter ([TrackIt case study](https://trackit.io/adn-case-study-ecs-to-eks-migration/)). The common thread is the strangler-style, Karpenter-first approach below - not a big-bang rewrite. ## Architecture mapping The mental model translation isn't 1:1, but it's close. Here's what maps to what: | ECS concept | EKS / Kubernetes equivalent | Notes | | --- | --- | --- | | Task Definition | Pod template (in Deployment, Job, etc.) | Container is the same. The wrapper has more knobs. | | Service | Deployment + Service | Two objects in K8s, one in ECS. | | Cluster | Cluster | Same word, different scope. EKS cluster manages a control plane; nodes are separate. | | Capacity Provider | Node Group / Karpenter NodePool | Karpenter replaces ASG-based scaling for most teams. | | Fargate (ECS) | Fargate (EKS) or EC2 nodes | EKS Fargate exists but most teams pick EC2 + Karpenter. | | Task IAM Role | Pod Identity (or IRSA) | Per-pod IAM. Pod Identity is the 2024+ default. | | ECS Service Discovery (Cloud Map) | Kubernetes Service + CoreDNS | DNS-based, in-cluster. | | Service Auto Scaling | HPA (workload) + Karpenter (nodes) | Two layers, both required. | | ECS Deployment / CodeDeploy | ArgoCD / Flux + Argo Rollouts | GitOps becomes the deploy plane. | | Parameter Store / Secrets Manager (direct) | External Secrets Operator | Same upstream, different consumption pattern. | | FireLens (awsfirelens) | Fluent Bit DaemonSet / OTel collector | Per-task became per-node. | | ALB target groups (IP mode) | AWS Load Balancer Controller (TargetGroupBinding or Ingress) | Same target groups, K8s-native config. | | awsvpc network mode | VPC CNI (default) | Conceptually identical. Operationally different at scale. | Every row in that table is a sentence. Most rows are a half-day of work. Two of them - networking and IAM - are the rows that come back and bite you. ## Challenge 1: Networking and the ENI question ECS in `awsvpc` mode gives every task its own ENI and its own VPC IP. EKS does the same thing, by default, via the AWS VPC CNI. So far so good. The catch: the VPC CNI assigns IPs to pods from the same subnet as the node, and each instance type caps how many it can attach - `(ENIs × (IPs per ENI - 1)) + 2`. A `c6i.large` tops out around 29 pods, a `c6i.xlarge` around 58, a `c6i.4xlarge` around 234. The subnet itself is the harder limit: if your pod subnets are `/24` (256 IPs) and you have a node group of 30 nodes running 200 pods, you've already used up the subnet, and you'll watch new pods hang in `ContainerCreating` with `failed to assign an IP address to container`. Three real options for fixing this: **Option A: Bigger subnets.** The cheapest fix and the one most teams pick. Drop your pod subnets to `/20` or larger, ideally one per AZ, and budget IPs based on your peak pod count plus headroom. This is a VPC redesign, which is exactly the kind of thing you don't want to discover halfway through a migration. **Option B: Prefix delegation.** Configure the VPC CNI to assign `/28` prefixes to ENIs instead of individual IPs. Each ENI now hosts 16 IPs instead of one. A `c6i.4xlarge` jumps from ~234 pods of IP capacity into the thousands - well past the per-node ceiling kubelet enforces anyway (AWS recommends a `--max-pods` of 110 for clusters up to 100 nodes, 250 above that), so IP exhaustion stops being the constraint. ```bash kubectl set env daemonset aws-node -n kube-system \ ENABLE_PREFIX_DELEGATION=true \ WARM_PREFIX_TARGET=1 ``` The cost is that prefixes are allocated whole. If you only need three IPs, you still take a `/28` (16 IPs) out of your subnet. Density up, fragmentation up. **Option C: Custom networking with a secondary CIDR.** Add a secondary `100.64.0.0/16` CIDR to the VPC and route pod traffic through it. The nodes stay in the original CIDR. This gives you huge pod IP space without renumbering your VPC, but the routing setup is non-trivial and worth doing only if you're going to be at scale. The mistake we see: teams plan their EKS migration with the same subnet footprint they used for ECS, hit IP exhaustion two weeks after going to production, and discover that fixing it requires changing the cluster's networking configuration in ways that need careful coordination. Solve this one before you ship workloads. ## Challenge 2: IAM (Task Roles became Pod Identity) In ECS, a Task Definition declares a `taskRoleArn`. AWS handles the credential delivery via a metadata service the task inherits. The container calls AWS SDKs and the SDKs find the right role automatically. EKS used to require IRSA (IAM Roles for Service Accounts), which is a federated OIDC trust between the cluster and IAM. It works, and it's still common. EKS Pod Identity (GA in late 2023) is simpler and is now the recommended path. Pod Identity is closer to the ECS Task Role model: AWS handles the credential delivery via an agent on each node, no OIDC trust, no service account annotations. ECS: ```json { "taskRoleArn": "arn:aws:iam::1234:role/payments-task", "containerDefinitions": [{ "name": "api", "image": "..." }] } ``` EKS with Pod Identity: ```yaml # 1. Create a ServiceAccount apiVersion: v1 kind: ServiceAccount metadata: name: payments namespace: payments # 2. Associate it with an IAM role (one-time, via aws CLI or IaC) # aws eks create-pod-identity-association \ # --cluster-name prod-us \ # --namespace payments \ # --service-account payments \ # --role-arn arn:aws:iam::1234:role/payments-task # 3. Reference the ServiceAccount from your Pod apiVersion: apps/v1 kind: Deployment metadata: name: api namespace: payments spec: template: spec: serviceAccountName: payments containers: - name: api image: 1234.dkr.ecr.us-east-1.amazonaws.com/api:v1.9.0 ``` The IAM role's trust policy is also simpler than IRSA's. With Pod Identity it trusts `pods.eks.amazonaws.com`, not a per-cluster OIDC provider: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] }] } ``` If you've still got IRSA-based clusters, they keep working. For greenfield, pick Pod Identity. The mental shift for ECS migrants is small: instead of attaching the role to the task definition, you attach it to the ServiceAccount the pod uses. Same idea, slightly different plumbing. The trap to avoid: don't run all your pods under the `default` ServiceAccount with one giant catch-all role. ECS forced you to think per-task, and EKS lets you keep that hygiene. One ServiceAccount per workload. One role per ServiceAccount. Least privilege survives the migration. ## Challenge 3: Service discovery and load balancing ECS gives you two service-to-service patterns: AWS Cloud Map (DNS-based service discovery) and ALB target groups for ingress. Neither survives the move untouched, but both have direct equivalents. **Internal service-to-service.** In ECS, you might have `payments-api.svc.local` resolving via Cloud Map to the running tasks. In EKS, you write a Service: ```yaml apiVersion: v1 kind: Service metadata: name: payments-api namespace: payments spec: selector: app: payments-api ports: - port: 80 targetPort: 8080 ``` Other workloads in the cluster reach it at `payments-api.payments.svc.cluster.local`, or just `payments-api` from inside the same namespace. CoreDNS handles the resolution. This is one of the parts that gets simpler in EKS, not harder. **External / ingress.** In ECS you registered tasks with an ALB target group via the service definition. In EKS, the AWS Load Balancer Controller does the same thing with two patterns: The Ingress pattern (controller provisions an ALB): ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: payments-api namespace: payments annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: payments-api port: { number: 80 } ``` The TargetGroupBinding pattern (you bring an existing ALB and target group, the controller registers pods into it): ```yaml apiVersion: elbv2.k8s.aws/v1beta1 kind: TargetGroupBinding metadata: name: payments-api namespace: payments spec: serviceRef: name: payments-api port: 80 targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:1234:targetgroup/payments-eks/abc ``` TargetGroupBinding is the killer feature for migrations. It lets you keep your existing ALB and route traffic to ECS and EKS simultaneously while you cut over. We'll come back to this in the migration section. ## Challenge 4: Autoscaling at two layers ECS has one autoscaling concept (Service Auto Scaling) that scales tasks based on metrics. EKS has two: HPA scales pods, and a node autoscaler scales nodes. Both are required. You don't want to be the team that forgot the node layer and watched pods queue at `Pending` because the cluster ran out of nodes. For pods, HPA is straightforward and direct: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api namespace: payments spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 4 maxReplicas: 40 metrics: - type: Resource resource: name: cpu target: { type: Utilization, averageUtilization: 65 } ``` For nodes, the answer in 2026 is Karpenter. The Cluster Autoscaler still works, but Karpenter is faster, smarter about instance types, and natively handles Spot. A typical NodePool: ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default requirements: - key: karpenter.k8s.aws/instance-category operator: In values: [c, m, r] - key: karpenter.k8s.aws/instance-generation operator: Gt values: ["5"] - key: karpenter.sh/capacity-type operator: In values: [spot, on-demand] limits: cpu: "1000" disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 30s ``` Karpenter looks at pending pods, picks the cheapest instance type that satisfies them, launches it directly (no ASG), and consolidates underutilized nodes when they free up. Teams typically see 30-50% node cost reduction over Cluster Autoscaler with mixed Spot/on-demand, which is most of why ECS-to-EKS pencils out economically at scale. The gotcha: HPA scales reactively on metrics that are seconds-to-minutes behind. A traffic spike that overwhelms the existing pods can take a minute or two for HPA to react and another minute for Karpenter to spin up nodes. If your workload is bursty enough that you can't tolerate that, you need pod over-provisioning (always run extra capacity) or KEDA (event-based scaling) to scale ahead of the metric. Plan for which pattern you need before you cut over. ## Challenge 5: Deployments are no longer one-shot ECS deployments are, fundamentally, an API call. CodePipeline runs `aws ecs update-service`, ECS swaps tasks based on the deployment configuration, you're done. The deployment is imperative and stateless: nothing watches your Git repo. EKS deployments can work that way (`kubectl apply -f` is the same shape) but you'd be giving up the main reason to be on Kubernetes. The standard EKS pattern is GitOps: a controller in the cluster (ArgoCD or Flux) watches a Git repo, reconciles the cluster to match it, reports drift, and lets you roll back by reverting a commit. The team that lifts-and-shifts the ECS deploy pipeline (CI runs `kubectl apply` directly) ends up with a worst-of-both-worlds setup: imperative deploys that can drift, no audit trail beyond the CI logs, and no native rollback. We've seen this pattern enough that we wrote a [longer comparison of ArgoCD versus Terraform for managing add-ons](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons), but the short version applies to your own apps too: ArgoCD owns the cluster's desired state, your CI builds images and updates manifests, the controller does the rolling update. For progressive delivery (canary, blue-green), the move from ECS CodeDeploy is natural. Argo Rollouts is the EKS equivalent, with explicit traffic splitting through your ALB: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: api namespace: payments spec: replicas: 10 strategy: canary: steps: - setWeight: 10 - pause: { duration: 5m } - setWeight: 25 - pause: { duration: 5m } - setWeight: 50 - pause: { duration: 10m } - setWeight: 100 trafficRouting: alb: ingress: payments-api servicePort: 80 ``` If you ran ECS with CodeDeploy's blue-green setup, the mental model is the same. The mechanics now live in Argo Rollouts, not in CodeDeploy. We covered the deeper tradeoffs in [Ship Without Fire Drills](/blog/ship-without-fire-drills-canary-blue-green-and-rolling-deploys). ## Challenge 6: Secrets and config ECS lets you reference SSM Parameter Store and Secrets Manager directly in a Task Definition. The agent fetches them at task start and injects them as env vars. Clean, simple, no extra components. EKS has no native equivalent. The good news: External Secrets Operator (ESO) is the universal answer and works well. The bad news: it's a controller, not a runtime injection - so secrets land in Kubernetes `Secret` objects, which means etcd, which means encryption-at-rest considerations. A SecretStore pointing at AWS Secrets Manager: ```yaml apiVersion: external-secrets.io/v1 kind: SecretStore metadata: name: aws-secrets namespace: payments spec: provider: aws: service: SecretsManager region: us-east-1 auth: jwt: serviceAccountRef: name: payments ``` A workload's secret pulled from there: ```yaml apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: payments-db namespace: payments spec: refreshInterval: 1h secretStoreRef: name: aws-secrets kind: SecretStore target: name: payments-db dataFrom: - extract: key: prod/payments/db ``` The pod consumes `payments-db` like any other Secret: env vars, volumes, whatever. ESO refreshes on the configured interval and updates the K8s Secret in place. Pods don't auto-restart on secret rotation by default - if you want that, add `reloader` (a small controller that watches Secrets and bounces dependent Deployments) or use a sidecar that re-reads the secret. For people who hate having plaintext-but-encrypted secrets in etcd at all, the alternative is Secrets Store CSI Driver - it mounts secrets as volumes directly from Secrets Manager without going through a K8s Secret object. More moving parts. Use it if your compliance posture requires it, otherwise ESO is simpler. ## Challenge 7: Logging changes shape ECS with FireLens lets you configure per-task log routing. EKS doesn't, by default. The standard pattern is a node-level Fluent Bit DaemonSet that collects everyone's logs and ships them to your destination of choice (CloudWatch, OpenSearch, Datadog, S3, anywhere). The migration cost: per-workload log routing rules from ECS need to be re-expressed as Fluent Bit filters keyed by Kubernetes labels. A minimal Fluent Bit values snippet that ships everything to CloudWatch but routes the `payments` namespace to a different log group: ```yaml config: filters: | [FILTER] Name kubernetes Match kube.* Merge_Log On Keep_Log Off outputs: | [OUTPUT] Name cloudwatch_logs Match kube.var.log.containers.*payments* region us-east-1 log_group_name /aws/eks/prod/payments log_stream_prefix payments- auto_create_group true [OUTPUT] Name cloudwatch_logs Match kube.var.log.containers.* region us-east-1 log_group_name /aws/eks/prod/default log_stream_prefix default- auto_create_group true ``` If you're moving to OTel anyway, do it now. The OpenTelemetry Collector handles logs, metrics, and traces in one DaemonSet, and the configuration model survives the next vendor switch. Teams who set up Fluent Bit in 2026 and migrate to OTel in 2027 do roughly the same work twice. ## The migration playbook: don't cut over in one window Lift-and-shift cutovers fail. The playbook that works is the strangler pattern with a shared ALB. Step by step: 1. **Stand up the EKS cluster, networking, and platform layer first.** No workloads yet. Solve the IP exhaustion question (Challenge 1) before any container runs. Install Karpenter, AWS Load Balancer Controller, External Secrets Operator, ArgoCD, Fluent Bit. Verify each one with a hello-world. 2. **Pick one non-critical workload and rebuild it on EKS, properly.** Not a port. A rebuild from the ground up using EKS-native patterns: Deployment, Service, Ingress, ESO, Pod Identity, GitOps-managed. This forces every team in your org to learn the new shape on a workload where mistakes are cheap. 3. **Bring up your second copy of a real workload behind the same ALB.** This is where TargetGroupBinding earns its place. Your existing ALB has one target group pointing at ECS; you create a second target group pointing at the EKS pods. Both target groups listen on the same ALB. You shift traffic via ALB weighted target groups: 95% ECS, 5% EKS. Then 80/20. Then 50/50. ```yaml # Listener rule on the ALB (Terraform shape) resource "aws_lb_listener_rule" "api" { listener_arn = aws_lb_listener.https.arn priority = 100 action { type = "forward" forward { target_group { arn = aws_lb_target_group.ecs_api.arn weight = 80 } target_group { arn = aws_lb_target_group.eks_api.arn weight = 20 } stickiness { enabled = false duration = 1 } } } condition { host_header { values = ["api.example.com"] } } } ``` Roll forward with confidence; roll back by changing the weights. No cutover window, no DNS TTL games. 4. **Cut workloads over one at a time.** Each one repeats step 3. The cluster slowly fills up. The ECS side slowly empties. When the last workload is cut, the ECS side is empty, and you tear it down. 5. **Decommission deliberately.** Before deleting the ECS cluster, audit for residual things: scheduled tasks, one-off run-tasks, CloudWatch alarms scoped to ECS metrics, IAM roles only used by ECS, security groups, log groups. None of those move themselves. The teams that try to do this in one weekend always regret it. The teams that take three months and run both stacks in parallel are the ones who land the migration without an outage on the front page. ## What Skyhook does for the EKS side A lot of the EKS day-one and day-two complexity above is undifferentiated. Every team migrating off ECS stands up the same set of platform components: Karpenter, ArgoCD, ESO, AWS Load Balancer Controller, Fluent Bit. Every team writes the same opinionated Helm values. Every team builds the same internal docs explaining how to deploy a service. [Skyhook](https://www.skyhook.io) is one option for skipping the platform-from-scratch part. We provision EKS clusters with the platform layer pre-installed and pre-wired, commit the manifests to your Git repos so you own them, and operate the cluster from a single UI: deployments, scaling, secrets, environments. For a team that wants Kubernetes' ecosystem without spending a quarter building the platform, it's a shortcut. For a team that already has a platform team and a strong opinion about which CRDs they want, it's a trial subscription you can rip out in an afternoon - the manifests are all in your repo. We're not pretending Skyhook is the only path. What we are saying is that "stand up the platform yourself, the way every other team has done it, with the same Helm values everyone else uses" stops being a useful exercise after the hundredth time the industry has done it. ## Wrap-up The ECS-to-EKS migration is real work, but the hard parts aren't where most teams expect. The workloads almost always come over cleanly. The pipeline rebuild is straightforward if you commit to GitOps. The bills that come due are networking (plan IPs early), IAM hygiene (Pod Identity per workload, no catch-all role), and the discipline to use the strangler pattern instead of forcing a cutover. If you do it right, the payoff isn't just "the same workloads on a different runtime." It's that every new platform capability the industry ships in the next five years will land in your cluster as a Helm chart, not a backlog item. That's the actual reason to do it. If you're sizing this up and want to talk through the migration without a sales pitch, we're around. And if you want to skip the platform-from-scratch step, [Skyhook](https://www.skyhook.io) is one way. --- ### The Kubernetes Production Readiness Checklist Every Service Needs - **URL**: https://www.skyhook.io/blog/kubernetes-production-readiness-checklist - **Date**: May 2026 - **Author**: Roy Libman, CPO - **Category**: DevOps - **Tags**: kubernetes, production-readiness, devops, kyverno, best-practices Services don't usually fail in production the way you'd expect. It's not the clever feature you spent two weeks designing. It's not the algorithm you stress-tested. It's the readiness probe that pings the database, so when Postgres has a 10-second blip the entire service goes NotReady and Kubernetes stops routing traffic. It's the deployment that only has one replica because the dev environment never needed two. It's the image tagged `:latest`, which pulled a different SHA today than it did yesterday, and now nobody can reproduce the bug. The boring stuff is what breaks. So the boring stuff is what you check. It's not a fringe problem either: in Red Hat's [2024 State of Kubernetes Security report](https://www.redhat.com/en/blog/state-kubernetes-security-2024), misconfigurations were the single biggest cause of security incidents, at 59% - ahead of vulnerabilities and failed audits. Not zero-days, not sophisticated attacks. Just things a checklist would have caught. This post is the production-readiness checklist we run against every Kubernetes service before we'll call it production-ready - the same checklist that ships built into Skyhook's [Production Readiness Framework](https://skyhook.io/solutions/production-readiness), evaluated automatically on every deployment, grouped into five categories. For each check, I'll cover what it catches, why it's on the list, and how to fix it when it fails. **TL;DR.** Production readiness is around 20 boring checks across five categories: reliability, security, resilience, performance, and resource hygiene. Most teams know they should be doing all of them. Almost none do all of them consistently, because tribal knowledge degrades fast. The fix is to automate the checklist, surface results next to every service, and let teams either fix or acknowledge with a documented reason. The enforcement layer is optional but valuable for org-wide standards. ## Why automate the checklist You can absolutely run this checklist by hand. Teams do, and the good ones internalize most of it. The cost is that "good production-readiness practices" lives in people's heads. When somebody leaves, some of it leaves with them. When a new service launches at 4pm on a Friday because a customer demoed it, half the checklist gets skipped. And the drift only compounds as the team grows and the original context fades. The other reason to automate: this list has 20-ish items, and you'll be evaluating it for every service, every deployment, forever. That's not a code-review checklist; it's a CI check. Either it runs automatically and tells you when something regressed, or it doesn't really exist. Most teams climb the same ladder, and most stall on the bottom rung: - **Documentation** - a wiki page engineers are supposed to check by hand. Compliance declines the moment a deadline hits. - **CI checks** - `kube-linter`, `kubeconform`, or `checkov` fail the PR on a bad manifest. Catches YAML mistakes, but can't see runtime state or drift. - **Admission control** - Kyverno or OPA Gatekeeper reject non-compliant resources at the door. Nothing lands without passing. - **Continuous scanning** - the same policies run on a schedule, catching drift and resources that were deployed before the policy existed. The teams that ship reliably operate on the top two rungs. The rest of this post is the checklist itself, then how to get there. Skyhook's Production Readiness Framework scans every service deployment against the checklist, marks each check pass / warn / fail, suggests an actionable fix for each failure, and lets you either fix or acknowledge with a documented reason. Optionally, it pairs with [Kyverno](https://kyverno.io/) to block non-compliant deployments at admission control. The rest of this post is what the framework actually checks, and why. ## Category 1: Reliability Reliability checks catch the failures that make your service unavailable to its users even when the cluster is otherwise fine. **Readiness probe configured.** Without a readiness probe, Kubernetes routes traffic to your pod the moment the container starts - including the first few seconds when your process is still loading config, opening database connections, or warming caches. New deployments serve errors during rollout. Fix: add a `readinessProbe` that checks the dependencies your service can't live without. **Liveness probe configured.** Without a liveness probe, a deadlocked or hung process stays in the pool until something else notices. With one, Kubernetes restarts a stuck pod automatically. Fix: add a `livenessProbe` - but keep it bare. This is the most common place teams shoot themselves in the foot. If your liveness probe checks Postgres, every Postgres incident becomes a full pod restart, which is itself load on Postgres. Liveness should answer "is this process alive?" and nothing else. **Valid probe timeouts.** The default `timeoutSeconds: 1` is short enough that any garbage-collection pause looks like a probe failure. The check fails when timeouts are too aggressive for the workload (and on the other side, it warns when they're so generous they make the probe useless). Fix: tune `timeoutSeconds`, `periodSeconds`, and `failureThreshold` to the actual latency profile of your `/health` endpoint. All three probes, with the right division of labor - and three distinct endpoints, not one shared `/health`: ```yaml containers: - name: api image: ghcr.io/org/api:v2.41.7 startupProbe: # Give slow starters up to 60s before liveness kicks in httpGet: path: /healthz port: 8080 periodSeconds: 2 failureThreshold: 30 readinessProbe: # Gates traffic - check the deps you can't serve without httpGet: path: /readyz port: 8080 periodSeconds: 5 failureThreshold: 3 livenessProbe: # Keep bare - "is the process alive?" and nothing else httpGet: path: /livez port: 8080 periodSeconds: 10 failureThreshold: 3 ``` **Rolling update strategy.** The `Recreate` strategy terminates all old pods before starting new ones - guaranteed downtime on every deploy. `RollingUpdate` is the default, but it's easy to override and forget. The check fails when a Deployment uses `Recreate` instead of `RollingUpdate`, or when `maxUnavailable` is set so high it can take the service below capacity mid-rollout. Fix: set `strategy.type: RollingUpdate` with `maxUnavailable: 0` (or a small fraction) and `maxSurge: 25%`. **Multiple replicas (≥2).** A single-replica Deployment goes to zero capacity every time a node drains, every time a pod restarts, every time Kubernetes evicts for any reason. This is the single most common cause of "our prod just went down for 30 seconds" tickets. Fix: set `replicas: 2` minimum for any service that handles user traffic. If you're using HPA, set `minReplicas: 2`. (For deployment patterns that pair with this - canary, blue-green, rolling - see [Ship Without Fire Drills](/blog/ship-without-fire-drills-canary-blue-green-and-rolling-deploys).) The replica and rollout settings together: ```yaml spec: replicas: 3 # >=2 for any user-facing service, 3 tolerates one failure strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% # Allow 25% extra pods during rollout maxUnavailable: 0 # Never drop below desired count minReadySeconds: 10 # Settle before marking a new pod available ``` ## Category 2: Security Security checks catch the misconfigurations that turn a routine vulnerability into a serious incident. (For the image and supply-chain side of this, see our [compromise audit checklist](/blog/trivy-compromise-audit-checklist).) **Non-root user.** Containers that run as UID 0 inherit root inside the container, which makes container escapes much more dangerous. The check fails when `securityContext.runAsNonRoot` is missing or false. Fix: set `runAsNonRoot: true` and `runAsUser: ` in the pod or container `securityContext`. If your image was built to run as root, rebuild it with a `USER` directive in the Dockerfile. **No privileged containers.** A privileged container has effectively the same access as a process on the host. The check fails when `securityContext.privileged: true` is set anywhere in the pod spec. There are legitimate reasons to need this (some CNI plugins, some node-level agents) but they're rare and should be exceptions you acknowledge, not defaults. Fix: remove the `privileged: true` flag and use specific Linux capabilities via `securityContext.capabilities.add` instead. **Read-only root filesystem.** A writable root filesystem means an attacker who pops the process can drop arbitrary binaries and modify the container image at runtime. The check fails when `securityContext.readOnlyRootFilesystem` is unset or false. Fix: set `readOnlyRootFilesystem: true` and mount `emptyDir` volumes for the specific paths your service needs to write to (temp directories, cache locations). **Privilege escalation disabled.** Without `allowPrivilegeEscalation: false`, a process can gain privileges via setuid binaries or capabilities. Fix: set `allowPrivilegeEscalation: false` in the container `securityContext`. The four container hardening checks above are one `securityContext` block. This matters more than it looks: running workloads as root is still common, and a single container escape on a root container lands the attacker on the node as root. ```yaml containers: - name: api securityContext: runAsNonRoot: true runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false capabilities: drop: ["ALL"] # Add back only what you need (e.g. NET_BIND_SERVICE) ``` **Default-deny Network Policy.** Without a NetworkPolicy, any pod can talk to any other pod in the cluster - so a single compromised container has full lateral movement. The check warns when a namespace has no default-deny policy. Fix: start every namespace with a default-deny, then allowlist the specific flows each service needs. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} # Applies to every pod in the namespace policyTypes: [Ingress, Egress] ``` **TLS on Ingress.** An Ingress serving plain HTTP in production leaks credentials, session tokens, and PII into any network it traverses. The check fails when an Ingress has no `tls` block or its hosts don't all have certificates. Fix: configure TLS via [cert-manager](https://cert-manager.io/) or your platform's equivalent. There is no good reason a production Ingress should still be plain HTTP in 2026. **Image tag is not `:latest`.** Tags like `:latest` (or `:stable`, or an unpinned major version) mean two pods of the same Deployment can be running different code, and rollbacks can become impossible. The check warns when an image tag is mutable or missing. Fix: pin to a specific tag (`v2.41.7`), or better, to the image digest (`@sha256:...`). Whichever you choose, treat tag mutability as an antipattern. ## Category 3: Resilience Resilience checks catch the failures that turn routine cluster operations into customer-facing incidents. **Pod Disruption Budget defined.** Without a PDB, a node drain (for an OS upgrade, an autoscaler scale-down, a node replacement) can evict every pod of your Deployment simultaneously. Fix: define a `PodDisruptionBudget` with `minAvailable: 1` (or `maxUnavailable: 50%` for larger Deployments). This is the single check that most directly catches "we did a routine cluster maintenance and accidentally took prod down." ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api-pdb spec: minAvailable: 2 # Keep at least 2 pods up during voluntary disruptions selector: matchLabels: app: api ``` **Multi-zone distribution.** A Deployment whose pods all land on nodes in the same availability zone goes dark when that zone has an issue. The check fails when nothing spreads the pods across `topology.kubernetes.io/zone`. Fix: add a `topologySpreadConstraints` block with `topologyKey: topology.kubernetes.io/zone` and `whenUnsatisfiable: ScheduleAnyway` (or `DoNotSchedule` if you want hard distribution). ```yaml spec: template: spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway # DoNotSchedule to spread hard labelSelector: matchLabels: app: api ``` **Graceful shutdown.** When Kubernetes sends SIGTERM, it starts removing the pod from Service endpoints concurrently - and that removal can lag behind the signal, so in-flight requests still hit a pod that has begun shutting down. The check warns when a pod has no `preStop` hook or a too-short `terminationGracePeriodSeconds`. Fix: add a short `preStop` sleep so endpoint removal propagates before your app stops accepting connections, and make sure the app handles SIGTERM (drain, finish in-flight work, exit) inside the grace period. ```yaml spec: terminationGracePeriodSeconds: 45 # Hard deadline before SIGKILL containers: - name: api lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 5"] # Let endpoint removal propagate ``` **Replica count appropriate for tier.** A "tier-1" production service running with 2 replicas might satisfy the previous reliability check but still be under-provisioned for its actual traffic. A flat "minimum 2" floor treats `payments-api` and `internal-tooling-dashboard` the same, which they aren't. The fix is a tiered floor: tag services by criticality and require a higher replica minimum for higher tiers. (This is one place a platform helps - Skyhook lets you set per-tier floors so the same check enforces a stricter standard where it matters.) ## Category 4: Performance Performance checks catch the misconfigurations that don't fail tests but quietly degrade your latency or your cluster's economics. **Horizontal Pod Autoscaler configured.** Without an HPA, traffic spikes either get absorbed by over-provisioned capacity (expensive) or cause cascading slowdowns (bad). The check warns when a Deployment has no HPA targeting it, and fails when the HPA's `maxReplicas` is set to a value that can't actually serve peak load. Fix: configure an HPA on CPU or a custom metric, with `minReplicas` matching your "always-on" floor and `maxReplicas` sized for peak. Set a scale-down stabilization window too, or a brief traffic dip churns your pods. ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 3 # Cover baseline traffic, not 1 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 behavior: scaleDown: stabilizationWindowSeconds: 300 # Wait 5 min before scaling down to avoid flapping ``` **Resource requests and limits set.** Without requests, the scheduler can't make smart placement decisions and your pod gets BestEffort QoS - first to be evicted under memory pressure. Without limits, a runaway process can starve everything else on the node. The check fails when a container is missing CPU requests, memory requests, or memory limits. (CPU limits are a more nuanced call - we'll cover the tradeoff in the next section.) Fix: set explicit `resources.requests` and `resources.limits` for every container. For sizing, measure first, don't guess. ```yaml containers: - name: api resources: requests: # Used for scheduling decisions cpu: "200m" memory: "256Mi" limits: memory: "512Mi" # ~2x request for spike headroom; CPU left unlimited on purpose ``` ## Category 5: Resource hygiene Resource hygiene checks are the finer-grained version of Performance: the difference between "set resources" and "set them correctly." **CPU requests defined.** Missing CPU requests means the scheduler treats your pod as needing nothing, which leads to noisy-neighbor problems and unfair scheduling. Fix: set a `resources.requests.cpu` that reflects your service's actual baseline. `100m` is a starting point for a small service; measure to refine. **Memory requests defined.** Same logic for memory: without requests, Kubernetes can't reserve capacity for your pod and the node can over-commit. Memory pressure then triggers OOM kills in arbitrary order, often killing your pod instead of the actual offender. Fix: set `resources.requests.memory` based on your service's working set, not its idle footprint. **QoS class is Guaranteed or Burstable.** The QoS class is derived from how you set requests and limits. `Guaranteed` (requests == limits, set for both CPU and memory) is the safest under pressure. `Burstable` (requests set, limits set higher) is the most common compromise. `BestEffort` (neither set) is what you want to fail every time. Fix: at minimum, set requests and limits such that Kubernetes gives your pod `Burstable` QoS. For tier-1 services, consider `Guaranteed` so they're last to be evicted. Per-container hygiene is easy to forget on the next manifest someone writes. Two namespace-level objects make the defaults stick: `LimitRange` fills in requests and limits a container omitted, and `ResourceQuota` caps what the whole namespace can consume so one team's runaway pods can't starve everyone else. ```yaml apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: production spec: limits: - type: Container defaultRequest: # Applied when a container omits requests cpu: "100m" memory: "128Mi" default: # Applied when a container omits limits memory: "256Mi" # Memory only - deliberately no CPU limit (see QoS note above) --- apiVersion: v1 kind: ResourceQuota metadata: name: production-quota namespace: production spec: hard: # Ceiling for the entire namespace requests.cpu: "20" requests.memory: "40Gi" limits.memory: "80Gi" pods: "100" ``` ![The five production-readiness categories - Reliability, Security, Resilience, Performance, and Resource hygiene - each feeding into automated enforcement](/images/blog/production-readiness-checklist-diagram.png) ## What a manifest checklist can't see The checks above are everything you can assert against a Deployment's YAML. Production readiness has a second half that doesn't live in a manifest, and it's the half that gets skipped under deadline pressure: **Observability** - Structured JSON logs to stdout/stderr, not files inside the container - A Prometheus metrics endpoint exposed (`/metrics`) - Alerts on error rate, latency, and saturation - Distributed tracing (OpenTelemetry) **Operational readiness** - SLOs defined for availability and latency - Runbooks for the common failure modes - Load testing done before go-live - A dependency map - what you call, and what calls you - Backup and restore procedures for anything stateful **Secrets management** - Secrets stored externally (External Secrets Operator, Sealed Secrets, Vault), not baked into manifests - see [how to manage secrets in Kubernetes](/blog/how-to-manage-secrets-in-kubernetes) - ServiceAccount token automount disabled unless the workload actually calls the API None of these reduce to a single field, so a YAML scanner won't catch them - but they're where the long tail of incidents lives. Teams routinely lose weeks a year to Kubernetes troubleshooting, and a lot of that is time good observability would have turned into a five-minute fix. ## Configurable to your standards, not ours The list above is what we believe production readiness means for a typical Kubernetes service in 2026. It's not the only valid opinion. Different organizations have different risk profiles, different compliance requirements, and different patterns they've already invested in. The framework lets you make those calls explicitly: - **Toggle individual checks** on or off. If your org genuinely needs privileged containers for a class of services, disable that check globally - or for a specific namespace. - **Override importance levels** per check. The default is critical / high / medium; you can change the severity to match your own tier definitions. - **Add custom checks** for org-specific requirements. Internal SLOs, naming conventions, label requirements, a DB-migration timeout that's specific to your stack - any predicate you can express against a Kubernetes resource can become a check. - **Exclude environments** from enforcement. Dev and preview environments usually don't need the same standards as prod; the framework lets you scope checks by environment. - **Acknowledge a failing check** with a documented reason. The check still shows as failed in the catalog, but it doesn't block a deploy. The reason is recorded against the check - so technical debt stays visible rather than invisible. The point of configurability isn't to let teams turn the whole thing off. It's to let teams own their version of the standard, so the standard is something they agreed to rather than something a vendor imposed. ## Two modes: surface, or block Production-readiness checks are useful at two distinct moments: when you're looking at what's already running ("is this service production-ready?") and when you're deploying something new ("should this deploy be allowed?"). **Surface mode** runs the checks continuously against every deployment and shows the results in the service catalog. Engineers see what's failing, fix or acknowledge, and the standard improves over time without anyone being blocked. This is where most teams should start. It's low-friction, high-visibility, and it surfaces existing technical debt without creating a deployment hostage situation. **Enforce mode** pairs the checks with Kyverno admission policies. Non-compliant deployments are rejected at admission control - they never reach the cluster. This is the right gear for organizations with hard compliance requirements, or for specific check classes (TLS on ingress, non-root user) where there's genuinely no acceptable exception in production. You can mix the two. A common pattern: enforce a small set of critical security checks (no privileged containers, TLS on prod ingresses), surface everything else. That way the deploy-blocking happens only for things that would be unambiguously bad, and the everyday "you should really set resource requests" stays a conversation rather than a wall. The framework supports running Kyverno in `audit` mode first - surfacing violations without blocking - so you can verify a policy won't cause a deploy storm before flipping to `enforce`. Use it. Going straight to enforce on a freshly added policy is how you discover at 9am on Monday that 30 of your existing services were already non-compliant. ## What the workflow looks like end to end 1. **Enable the framework for a service**, or for an entire environment. Skyhook starts scanning immediately - no agents to install, no sidecar to deploy. 2. **Checks run automatically on every deployment.** Results appear in the service catalog with pass / warn / fail status per check, color-coded by importance. 3. **For every failure, you get an actionable fix suggestion** - the specific YAML change to make, or a link to the standard playbook. You either apply the fix or acknowledge with a reason that goes into the audit log. 4. **When you're ready, enable Kyverno enforcement** for the specific checks where "fail" should mean "no deploy." Start in audit mode, verify, then switch to enforce. That's it. The checklist runs forever, the standard is visible, technical debt is documented rather than invisible, and the things that should be unconditional - no privileged containers in prod, no plain-HTTP ingresses, no `:latest` tags - become unconditional. ## Why this beats the wiki page Most teams already have a production-readiness checklist somewhere. It usually lives in a Notion page, gets reviewed during onboarding, and slowly diverges from reality. A new service launches and three checks get skipped because the engineer who knew about them was on vacation. Two years later, that service is the one that takes prod down. Automating the checklist isn't a process improvement; it's the difference between a standard that exists and one that doesn't. The first month feels great either way. The third year is where the gap shows. If you're building this yourself, the 20-odd checks above are the ones we'd hand you. If you'd rather inherit the framework, that's what [Skyhook's Production Readiness Framework](https://skyhook.io/solutions/production-readiness) is. Either way - the boring stuff is what saves you. Make sure it's getting checked. --- ### Backstage Fatigue: When NOT to Build an Internal Developer Platform - **URL**: https://www.skyhook.io/blog/when-not-to-build-an-internal-developer-platform - **Date**: April 2026 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: platform-engineering, idp, backstage, developer-experience, for-engineering-leaders TL;DR: Building an internal developer platform from scratch used to be the only option, and it only paid back for the largest orgs that could afford it. Self-hosted Backstage runs $450K-$800K in year one, takes 6-12 months to be useful, and stalls at around 10% adoption outside Spotify. The Gartner "80% by 2026" stat is about *large* orgs, but even there, teams that operate as federations of smaller squads land in the same trap when they don't actually have bespoke needs. This post is the honest version: when building still makes sense, when it doesn't, and the pre-built path that actually delivers what teams want. The short answer for most teams: [buy, don't build](/blog/why-pre-built-internal-developer-platforms). ## The Stat Everyone Quotes Wrong The [Gartner forecast](https://www.gartner.com/en/infrastructure-and-it-operations-leaders/topics/platform-engineering) that started this whole conversation says, verbatim: "By 2026, 80% of **large** software engineering organizations will establish platform engineering teams as internal providers of reusable services, components and tools, up from 45% in 2022." That word "large" does most of the work, but the size axis is also misleading on its own. Plenty of 500-engineer orgs run as a federation of four mostly-independent squads with their own stacks, and the build-vs-buy math inside each squad looks like a 100-engineer team's, not a 500-engineer team's. Conversely, in a 30-person engineering org with one product squad and one cluster, the math is often inverted - building looks attractive on a slide and breaks under operating it. And in a 5-person team, building an IDP is the same conversation as building your own database: technically possible, almost certainly the wrong call. The real question is not size; it is whether your needs are bespoke enough to justify the build cost when a pre-built option exists. But the stat keeps showing up in board decks at companies it was never about, and the conclusion gets reverse-engineered from there: we are an organization, 80% of organizations have one, therefore we should have one. The [DORA 2024 report](https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report), which actually surveyed practitioners rather than predicting CIO behavior, found something less flattering: orgs that increased their platform engineering focus saw throughput drop by 8% and change stability drop by 14%. The same orgs reported productivity gains of 8% individual / 10% team, so the picture is "developers feel more productive, but the team ships slower and less reliably." DORA's hypothesis was that IDPs introduce handoffs that previously did not exist. That is not a verdict on platform engineering. It is a verdict on doing it before you need it. ## What Backstage Actually Costs Backstage is the default suspect when this conversation comes up because it is the only widely-adopted open source IDP. The Spotify-adjacent numbers are sobering. | Cost line | Realistic range | | --- | --- | | Time to first useful state | 6-12 months | | Dedicated engineers to maintain | 3+ | | Year-one cost (loaded salaries) | $450K - $800K | | External adoption rate | ~10% (vs Spotify's internal 99%) | Those numbers come from vendors selling alternatives (Roadie, Cortex, Port), so treat them as directional rather than gospel. But the time-to-value range is consistent across enough independent sources that I would trust the lower bound. And the adoption number is the one that should haunt anyone considering this: [Helen Greul](https://thenewstack.io/spotifys-backstage-roadmap-aims-to-speed-up-adoption/), head of engineering for Backstage at Spotify, has said publicly that average external Backstage adoption stalls at around 10% of engineers because most adopters never get past the proof-of-concept phase. Spotify's own response was to launch Spotify Portal for Backstage, a hosted edition. When the canonical reference customer concludes that self-hosting is broken for everyone else, that should land somewhere. This is not a Backstage-is-bad argument. Backstage works exactly as advertised when you have the engineering surface area to amortize the cost. Below somewhere around 150 engineers, you do not. ## Platforms Before Portals Octopus has a piece called "Platforms Before Portals" that frames this better than I can, and GitLab made the same case in "Beyond the portal hype." The argument: a portal is navigation. If there is nothing to navigate to, the portal is theater. You are pointing developers at a beautiful index of services they could deploy to, except that the deploy experience itself is still "open a ticket with DevOps." The right order is: 1. **Paved path for deployment.** A self-serve way to ship a service from commit to production without filing a ticket. 2. **Paved path for environment.** A self-serve way to spin up a preview environment, get a database, get secrets. 3. **Paved path for observability.** Logs, metrics, and traces that are wired up by default, not as a follow-up ticket. 4. **Then a portal.** Once the underlying services are real, the portal becomes useful as a navigation layer. Most teams I talk to skip 1-3 and start at 4 because Backstage has a marketing site and the others do not. They end up with a beautiful portal pointing at the same broken deploy experience. Six months in, the portal is stale because nobody updates the catalog, and the team that built it has moved on. Charity Majors put this more bluntly in her [QCon NYC 2023 talk on platform engineering pitfalls](https://www.infoq.com/presentations/platform-engineering-teams/): "If your platform team spends a lot of time writing software, something's probably wrong." That was Pitfall #2 of eight. The point is not that platform teams should not write code. The point is that if they are writing it from scratch instead of integrating things that already exist, you are paying twice for the same outcome. ## When an IDP Genuinely Makes Sense There is a real threshold. From what I have seen across customer conversations and what shows up in the public data, dedicated internal platforms start to pay back somewhere around **~150 engineers**, with some accelerants: - **More than 3 product groups with conflicting infrastructure preferences.** When Group A wants Postgres on RDS and Group B wants Postgres in-cluster and Group C wants CockroachDB, a platform layer that says "here is Postgres, you do not get to choose" is genuinely valuable. - **Multi-cloud or multi-region by requirement, not preference.** If you have to deploy to AWS and GCP because of a customer contract, abstracting that away has obvious leverage. - **Compliance scope that affects every service.** SOC 2, HIPAA, PCI - any of these benefit from "the platform enforces it" rather than "every team has to remember." - **A platform team that exists to maintain it.** Not three SREs you renamed. A real team with a product manager and a roadmap. If you tick at least two of these and you are above 150 engineers, build the platform. Backstage may even be the right base. If you tick zero of these and you are anywhere under that line - whether you are five engineers or eighty - do something else. ## The Wrong-Reason Signals The conversations that end badly tend to start with one of these: - **"We are getting enterprise-ready."** Enterprise readiness is SOC 2, audit logs, and an SSO story. It is not a developer portal. - **"Every modern company has one."** Not at your size, they do not. - **"Our VP of Eng read a Gartner report."** See above. - **"We need to attract platform engineers."** Hiring a team to build a platform so you can hire more people for the team is recursive in a way that should set off alarms. - **"It is on our 2026 OKRs."** OKRs are not requirements. Push back. The honest reason to build an IDP is that developers cannot deploy without a ticket and you have measured the cost of that and decided a platform is the cheapest path to fix it. If that is not the reason, do not start. ## What Most Teams Actually Need Most of the value people expect from a Backstage build is achievable from a pre-built platform plus a couple of small tools. The list below applies whether you are 5 engineers, 100, or 500 split across federated squads - the difference is which row hurts first: | What developers actually want | How to deliver it without building one | | --- | --- | | Deploy without filing a ticket | Opinionated deploy platform with a UI and CLI | | Preview environment per PR | Argo Rollouts or similar, automated from CI | | Find services and their owners | A spreadsheet, then a `services.yaml` in a known repo, then maybe a portal once it hurts | | Self-serve Postgres / Redis / S3 | Crossplane or a Terraform module library with a `plan` step in CI | | Documentation that is not stale | Mintlify, GitBook, or just a `docs/` folder with a search bar | | Service catalog | The README of your monorepo, until that hurts | The pattern is the same one Kelsey Hightower has been making for a decade: do not build the abstraction until the thing you are abstracting is real and painful enough to justify it. Pick opinionated tools that already do most of what you want. Tolerate the spreadsheet. Resist the urge to build. If you do decide you need one coherent platform across all of this, [buy a pre-built one rather than start a Backstage project](/blog/why-pre-built-internal-developer-platforms). ## The Inverse Argument Here is the contrarian frame: for most teams under ~150 engineers, the deployment platform IS the developer experience. There is no separate portal to build because the thing developers interact with every day is the deploy experience itself. If that experience is good, you have an IDP. It just does not have a homepage. This is the argument behind opinionated PaaS-style tools like Skyhook, Render, Northflank, Porter, Fly, and Railway. None of us market against Backstage directly, but the implicit pitch is the same: the platform is the product. You do not need to build it. Most of these vendors say it through their product surface rather than head-on. What is missing from that conversation is a numerical threshold for *when the math flips* and a checklist for what to actually build when it does. That is what the rest of this post is for. For Skyhook specifically, this is why we ship things like preview environments per PR, golden paths, deploy strategies, and a service catalog as defaults rather than as plugins you have to assemble. A three-engineer team gets the IDP outcome on day one without ever owning a platform. A growing 80-engineer team gets the same baseline but with the room to extend it as their needs sharpen. And when an organization eventually crosses into "we genuinely need a custom platform" territory, you still have not spent engineer-quarters building the wrong thing - you have a working production baseline to either extend or replace. ## The Question to Actually Ask Stop asking "should we build an internal developer platform?" Ask: > What is the specific developer pain that an IDP would solve, and is building one cheaper than buying or simplifying the thing causing the pain? If the answer is "deploys are slow," fix deploys. If the answer is "nobody knows who owns what," start with a markdown file. If the answer is "we want to look modern," that is not an answer. The companies winning at developer experience right now are not the ones with the prettiest Backstage instance. They are the ones who shipped fewer abstractions and let developers actually deploy software. ## Further Reading - [Golden Paths and the 80/20 Rule](/blog/golden-paths-and-the-80-20-rule) - The opinionated-defaults argument from the inside: how to pick the 20% of paths that cover 80% of work without building a portal. - [Self-Serve Platforms and Service Catalogs](/blog/self-serve-platforms-and-service-catalogs) - The narrower problem an IDP is usually trying to solve, and what minimum viable looks like. - [The Rise of Platform Engineering](/blog/the-rise-of-platform-engineering-redefining-devops-for-the-modern-era) - Companion piece on when a platform team starts to make sense. This post is the contrarian counterweight to that one. - [Octopus Deploy: Platforms Before Portals](https://octopus.com/blog/platforms-before-portals) and [GitLab: Beyond the Portal Hype](https://about.gitlab.com/the-source/platform/beyond-the-portal-hype-why-you-need-a-platform-first/) - Primary sources for the platforms-first argument. --- ### After the Trivy Compromise: What Every K8s Pipeline Should Audit Right Now - **URL**: https://www.skyhook.io/blog/trivy-compromise-audit-checklist - **Date**: April 2026 - **Author**: Eyal Dulberg, CTO - **Category**: DevOps - **Tags**: kubernetes, supply-chain-security, cosign, kyverno, devops TL;DR: On March 19, 2026, attackers took over the official Trivy GitHub Action and the official Trivy Docker image. Anyone whose CI pipeline pulled the latest Trivy that day got a backdoored binary that quietly stole cloud credentials and shipped them to the attackers. Same crew (TeamPCP) then used those stolen credentials to attack npm, PyPI, and Docker Hub the following weeks. This post is the audit checklist - the same controls work whether you have three engineers or a hundred, you just have less inventory to walk through. Working YAML included, jargon explained inline. At the end, we cover what Skyhook does for the clusters we manage. ## What Made This Attack Different Most CVEs are about a flaw in a piece of software you are running. The Trivy attack was different because **the attacker became the trusted thing**. Three details made it nasty: 1. **The scanner became the malware.** Trivy is the tool teams use to decide whether *other* images are safe. When the scanner itself is the threat, your image scanner cannot save you. The malicious payload ran silently before the real scan, so CI logs looked normal. 2. **They moved version tags, not commits.** 76 of 77 version tags of `aquasecurity/trivy-action` (`v0`, `v1`, semver tags) were rewritten to point at malicious code. The underlying commit IDs (the long `abc123...` SHAs) were untouched. So anyone who wrote `uses: aquasecurity/trivy-action@v0` got the malware. Anyone who wrote `uses: aquasecurity/trivy-action@` was fine. 3. **They came back three days later.** After the cleanup, the same stolen credentials were used to push two new malicious Trivy Docker images on March 22. If your CI was pulling `aquasec/trivy:latest`, you might have grabbed a clean one Monday and a malicious one Wednesday and never noticed. The downstream damage is still being counted. [CERT-EU has attributed](https://cert.europa.eu/blog/european-commission-cloud-breach-trivy-supply-chain) a 340 GB Europa.eu data leak to credentials stolen this way, and [Datadog traced](https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-teampcp-supply-chain-campaign/) PyPI compromises in the following weeks (LiteLLM, Telnyx) back to tokens harvested from CI runs that pulled the bad binary. ## The Pattern Underneath Every supply chain attack of the last five years rhymes: SolarWinds, Codecov, `tj-actions/changed-files`, Trivy. A trusted thing gets compromised, the references that point to it (`@latest`, `@v0`, `:latest` tag) are mutable, and downstream consumers pull the new version without noticing. A "mutable reference" is anything that can change underneath you. Tags can be moved. Branches can be rewritten. A friendly name like `:latest` always points at whoever pushed last. The opposite is an "immutable reference" - a digest like `sha256:6d3c...` or a Git commit ID - which is content-addressed: change one byte and the ID changes too. The fix for this whole class of attack is structural: stop using mutable references in the places where it matters, and put one place in your stack that decides what is allowed to run. ## The Audit Order That Actually Ships You cannot do everything in one week. You can do these three things, in this order, and the next attack like Trivy is a non-event for you. ### 1. Pin GitHub Actions to commit SHAs, not tags If only one thing on this list gets done, do this. The Trivy attack was harmless to anyone pinning by SHA, because the SHAs never moved. ```yaml # BAD: tag is mutable - it can be re-pointed at malicious code - uses: aquasecurity/trivy-action@v0 # BAD: even semver tags can be force-pushed - uses: aquasecurity/trivy-action@0.28.0 # GOOD: full commit SHA - cannot be changed - uses: aquasecurity/trivy-action@915b19bbe73b92a6cf82a1bc12b087c9a19a5fe2 # v0.28.0 ``` In plain English: GitHub Actions are referenced by either a tag (a friendly name someone in the project picks and can move) or a commit SHA (the long string that uniquely identifies a snapshot of the code). Always use the SHA. Add the friendly version as a comment so humans can read it. You do not have to keep these SHAs current by hand. Dependabot does it for you, opens PRs when actions release new versions, and updates the comment automatically: ```yaml # .github/dependabot.yml version: 2 updates: - package-ecosystem: "github-actions" directory: "/" schedule: interval: "weekly" ``` This costs you one PR and zero buy-in from anyone else. Open it. ### 2. Block `:latest` and pin every container image to a digest The same idea, one layer up: a Docker image tag (`myapp:v1.2`, `aquasec/trivy:latest`) can be re-pointed at a different image at any time. A digest (`@sha256:...`) cannot. To enforce this across a Kubernetes cluster, you use a tool called **Kyverno**. The policies in this section assume **Kyverno 1.10+** (for `mutateDigest` and the `subjectRegExp` field used later in the Cosign policy). Quick definitions before the YAML: - **Kyverno** is an open-source policy engine for Kubernetes. You write policies as YAML, install them in your cluster, and they get checked every time anyone tries to create a pod. - **Admission control** is the moment Kubernetes asks "is this allowed?" before creating a resource. Kyverno plugs into that moment. - **Image digest** is the immutable fingerprint of an image: `myapp:v1@sha256:abc123...`. The tag is decorative; the digest is what gets pulled. Here is the policy. It does two things: rejects any pod that uses `:latest`, and automatically rewrites `image: foo:v1` into `image: foo:v1@sha256:...` at the moment the pod is created. ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-mutable-tags spec: validationFailureAction: Enforce # actually block, not just warn rules: - name: block-latest match: any: - resources: kinds: [Pod] validate: message: "Image tag :latest is forbidden. Pin to a specific version or digest." pattern: spec: =(initContainers): - image: "!*:latest" containers: - image: "!*:latest" - name: rewrite-tag-to-digest match: any: - resources: kinds: [Pod] verifyImages: - imageReferences: - "*" mutateDigest: true # the magic line: rewrite tag to digest at admission required: false # set to true once you also add signing ``` In plain English: the first rule rejects pods that use `:latest`. The second rule tells Kyverno "for any image, look up its current digest and rewrite the manifest so the cluster pulls by digest." Even sloppy `image: foo:v1` references get pinned automatically. If somebody force-pushes a tag tomorrow, you keep running the version you actually deployed. Pair it with a registry allowlist so workloads can only pull from registries you trust. `anyPattern` is the OR construct in Kyverno - one of the patterns must match: ```yaml - name: registry-allowlist match: any: - resources: kinds: [Pod] validate: message: "Pulls are only allowed from ghcr.io/myorg or harbor.internal." anyPattern: - spec: containers: - image: "ghcr.io/myorg/*" - spec: containers: - image: "harbor.internal/*" ``` The hard part of all this for most teams is not writing the policy - it is owning Kyverno itself: installing it, keeping it in sync with the Kubernetes version, upgrading it when the cluster upgrades, and making sure someone is still on call for it after the engineer who shipped it changes teams. This is the chassis Skyhook handles for the clusters we manage. Kyverno is a one-click addon. Policies are managed from the Skyhook UI: you pick from a baseline, paste in custom YAML for the ones you want, and toggle each policy between `Audit` (warn only) and `Enforce` (actually block) per cluster. The selection ends up in your GitOps repo so it is reviewable like any other change. If you cannot run Kyverno at all, the next-best thing is to pin digests directly in your deployment manifests using Kustomize: ```yaml # kustomization.yaml images: - name: aquasec/trivy newName: aquasec/trivy digest: sha256:6d3c1e2b7a8f4c5d9e0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e ``` You can find the digest of any image with `crane digest aquasec/trivy:0.69.3` (install `crane` from the `go-containerregistry` project) or with `docker buildx imagetools inspect aquasec/trivy:0.69.3`. ### 3. Run a "pull-through" registry as a choke point A pull-through registry sits between your cluster and the public registries (Docker Hub, GHCR, etc.). When something asks for `docker.io/library/redis:7`, the request goes to your registry first. If your registry has it cached, it serves the cached copy. If not, it fetches it from Docker Hub once, caches it, and serves it. Every subsequent pull uses your copy. **Harbor** is the most common open-source choice. This is the unsexy part of supply chain security, but it is what lets you actually do something when the next attack happens: - **Audit log**: every image that ever entered your cluster is in one place. - **Quarantine in one place**: if `aquasec/trivy:0.69.4` turns out to be malicious, you delete it from your registry once and it stops pulling everywhere. - **Layer retention**: when a maintainer rewrites history (or a registry deletes an image), you still have the version you were running. For most teams, the cloud bill for a self-hosted Harbor is in the low tens of dollars a month. The day it earns its keep, it pays for the next decade. (If you are very small - say, fewer than ten engineers and one production cluster - you can defer this and rely on the digest-pinning policy until the registry choke point becomes worth the operational overhead.) The same logic applies one layer up at the deployment level. If every team deploys differently, "pin all images by digest" is a 50-place change. If every deploy goes through one GitOps plane, it is a one-place change. This is structurally what Skyhook gives you: one Kustomize layer, one ArgoCD plane, one set of overrides. When you decide no image without `@sha256:` is allowed in prod, the policy goes in one place and applies everywhere. ## Cosign Verification: Smaller Than People Think The next step up from "block `:latest` and pin digests" is **image signing**: when you build an image, you cryptographically sign it. When the cluster tries to pull it, you verify the signature. If the signature is missing or wrong, the pull is rejected. The dominant open-source standard for this is **Cosign**, part of the **Sigstore** project. Cosign has a reputation for being a months-long project. It is not. Signing your own builds is two extra lines in your CI workflow: ```bash cosign sign --yes ghcr.io/myorg/app@${DIGEST} ``` Verifying it at the cluster is one Kyverno policy. Here is the version that proves the image was built by your repo, on your branch, by your CI: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: verify-org-signatures spec: validationFailureAction: Enforce rules: - name: cosign-keyless-org-images match: any: - resources: kinds: [Pod] verifyImages: - imageReferences: - "ghcr.io/myorg/*" attestors: - entries: - keyless: subjectRegExp: "^https://github\\.com/myorg/[^/]+/\\.github/workflows/release\\.yml@refs/heads/main$" issuerRegExp: "^https://token\\.actions\\.githubusercontent\\.com$" rekor: url: "https://rekor.sigstore.dev" mutateDigest: true required: true ``` A focused engineer can have signing + this policy in `Audit` mode across their own org's images in about a week. The reason it sometimes stretches longer is *coverage*: third-party images (operators, sidecars, base images) are not signed by your CI, so this policy will not cover them. Most teams accept narrower scope and rely on the digest-pinning policy from above for everything they did not build themselves. One small prerequisite people skip: Kyverno's webhook needs TLS, and the easiest way to handle that is to install **cert-manager** (a separate Kubernetes addon that issues certificates automatically). On Skyhook clusters cert-manager is already installed as a one-click addon, which removes one of the more common yak shaves on the path here. ## What People Miss The audit fails silently in the same places every time. Three to actually check: **Helm charts you adopted years ago.** Some `values.yaml` from a 2022 tutorial still has `image: redis:latest`. ArgoCD reconciles it on every cycle and nobody looks. Run `helm template` over your repo and grep for any image reference without `@sha256:`. **Sidecars and init containers.** Service mesh proxies, log shippers (`fluent-bit:latest` is everywhere), wait-for-db init scripts, secrets injectors. People audit the main app container and stop. The Trivy attack landed in a CI sidecar pattern; the Codecov attack in 2021 landed in a CI script. Same shape. **Operator-managed images.** When you install an operator (ArgoCD, cert-manager, Vault), it pulls *its own* images based on its source code. Those are often pinned to mutable tags that you do not control. List what is actually running with: ```bash kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.initContainers[*].image}{"\n"}{.spec.containers[*].image}{"\n"}{end}' | sort -u ``` This is also where having a single addon catalog matters. If every operator was installed by a different engineer over three years from a different Helm command, you have no inventory. If they were all installed through one addon system, you have one list to grep. Skyhook's addon catalog is structured this way - one place to see what is running, one place to override an image source, one place to pin a version. ## The Triage Pass: 30 Minutes If you have been ingesting Trivy in CI, do this first: ```bash # Look for the attacker's exfil endpoints in your repos grep -r "scan.aquasecurtiy.org" . # note the typo - that's the malicious one grep -r "tpcp-docs" . # Find every GitHub Action pinned by tag instead of SHA grep -rE "uses:\s+\S+@v[0-9]" .github/workflows/ # Find every Kubernetes manifest using :latest grep -rE "image:\s+\S+:latest" . ``` If any of those return hits, those are your weekend. ## A One-Week Plan The work scales with your inventory, not with your team size. Three engineers and one cluster will finish faster than one hundred engineers and twenty clusters, but the order is the same: 1. **Day 1:** PR pinning every GitHub Action to a SHA. Add Dependabot to keep them current. 2. **Day 2-3:** Install Kyverno (or enable it from your platform's addon catalog). Apply the `block-latest` and `mutateDigest` policies in `Audit` mode for a week to see what would fail. 3. **Day 4-5:** Stand up Harbor as a pull-through cache. Update the registry allowlist policy. 4. **Day 6:** Flip Kyverno from `Audit` to `Enforce`. Triage the inevitable surprises. 5. **Week 2-3:** Add Cosign signing to your own builds. Add the verifyImages policy in `Audit`, then `Enforce` for your own images. The first three items neutralize most of this attack class for a fraction of the work. Cosign is a focused week after that, not a quarter. ## Where Skyhook Fits If you are running on Skyhook, most of the chassis for this work is already in place. Kyverno is a first-class addon, installed in one click and kept in sync with the cluster version through the same upgrade flow as everything else. cert-manager ships the same way. Policies are managed from the Skyhook UI per cluster. A baseline of opinionated security policies is included with the addon (block host paths, block privileged containers, require non-root, require resource limits, require probes, require labels). You can add the `block-latest` and `mutateDigest` policies from this post by pasting them into the custom-policy editor, and you can flip each policy between `Audit` and `Enforce` from the same screen. The selection lives in your GitOps repo, so it is reviewable, reversible, and applied identically to every cluster. That is the structural piece that matters here. Every Skyhook deploy goes through one Kustomize layer applied through one ArgoCD plane. When you decide tomorrow that no image without `@sha256:` is allowed in prod, the change goes in one place and applies everywhere. This works the same way whether you have one cluster or twenty, and whether your team is three engineers or a hundred - the leverage actually scales up the more inventory you have. The controls in this post are the ones we are continuously adding to the default bundle so customers do not have to think about them at all. The next supply chain attack is already being planted. The good news: the controls that stopped this one will stop the next one. Pin the SHAs. Block the tags. Enforce the digests. Then go to bed. ## Further Reading - [How to Manage Secrets in Kubernetes](/blog/how-to-manage-secrets-in-kubernetes) - The other half of the supply chain story: keeping the credentials your CI does need from leaking the same way Trivy's victims did. - [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter) - How a single GitOps plane turns "pin every image by digest" from a 50-place change into a one-place change. - [Managing Kubernetes Add-ons: Argo CD or Terraform?](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons) - Where Kyverno, cert-manager, and the rest of this audit's chassis actually live. - [CERT-EU advisory on the European Commission cloud breach](https://cert.europa.eu/blog/european-commission-cloud-breach-trivy-supply-chain) - Primary source for the Europa.eu impact. - [Aqua Security's incident write-up](https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/) and [Wiz's TeamPCP analysis](https://www.wiz.io/blog/trivy-compromised-teampcp-supply-chain-attack) - Original technical breakdowns of the attack chain. --- ### Why Best-of-Breed Tooling Wins: The Case for Flexibility in Your Infrastructure Stack - **URL**: https://www.skyhook.io/blog/best-of-breed-tools - **Date**: April 2026 - **Author**: Roy Libman, CPO - **Category**: Architecture - **Tags**: architecture, platform-engineering, kubernetes, vendor-lock-in, devops-tools **TL;DR.** Best-of-breed infrastructure tools preserve capability and reduce vendor lock-in, but every additional tool adds integration and maintenance work. A thin platform layer can absorb that integration tax without replacing the underlying tools or moving their configuration out of your repositories. This post explains the tradeoff and what to require from that layer. The CNCF landscape has over 1,000 logos. If you've ever zoomed into that chart and felt overwhelmed, you're not alone. But that density isn't a problem to solve - it's a signal to read. It means the cloud-native ecosystem has produced specialized, purpose-built tools for every layer of the stack: GitOps, observability, secrets management, service mesh, certificate handling, cost optimization. And in nearly every category, the specialized tool outperforms the "good enough" module bundled inside an all-in-one platform. This is what "best-of-breed" means in practice. You pick ArgoCD for GitOps because it's the best at GitOps. You pick Prometheus for metrics because it's the best at metrics. You don't pick a platform that happens to do both, mediocrely. Most teams already operate this way. The question is whether they're doing it intentionally - or just accumulating tools and paying the price. ## The All-in-One Promise All-in-one platforms sell a compelling story: one vendor, one UI, one contract, one throat to choke when things break. For a team of five spinning up their first cluster, that simplicity is genuinely valuable. The problem shows up later. **Feature depth plateaus.** An all-in-one platform spreads engineering effort across dozens of capabilities. The GitOps module is "fine." The monitoring is "adequate." But none of it matches what a dedicated team building a single-purpose tool can deliver. You end up working around limitations instead of leveraging strengths. **Pricing becomes a trap.** Some platforms use "success penalty" pricing - the more you grow, the more you pay, often in ways that aren't obvious upfront. self-managed Red Hat OpenShift, for example, ties licensing to the underlying hardware (subscriptions are sold per core-pair or per bare-metal node). Upgrade to higher-core nodes and your bill jumps, even if your workload hasn't changed. **Lock-in compounds over time.** Moving away from an opinionated all-in-one platform often means re-architecting networking, security policies, and deployment workflows from scratch. The longer you stay, the harder it gets to leave. This isn't a theoretical risk - vendor lock-in is one of the most cited concerns in cloud strategy surveys year after year. **The AWS CI/CD example.** Many teams adopt CodePipeline, CodeBuild, and CodeDeploy because they're already on AWS - one ecosystem, one bill, tight integration. In practice, each tool is mediocre at its job compared to the specialized alternative. CodeBuild is slower and less flexible than GitHub Actions. CodeDeploy lacks the drift detection and reconciliation of ArgoCD. CodePipeline's UI makes debugging a chore. Teams end up migrating to GitHub Actions + ArgoCD anyway - but now they've burned months building on AWS's all-in-one suite first. ## Why Best-of-Breed Won This isn't a theoretical debate. The market already decided. **The numbers:** The CNCF hosts 200+ projects with hundreds of thousands of contributors worldwide. And the pull is accelerating: the [CNCF and SlashData State of Cloud Native Development report](https://www.cncf.io/announcements/2026/03/24/cncf-and-slashdata-report-finds-cloud-native-community-reaches-nearly-20-million-developers/) put the cloud-native developer community at 19.9 million in Q1 2026 - up from 15.6 million two quarters earlier, a 28% jump in six months, and now 39% of all developers worldwide. **The ecosystem is specialized by design.** Kubernetes itself was built as an extensible platform, not a monolith. Its power comes from CRDs and the operator pattern - standardized APIs that let specialized tools plug in. cert-manager handles TLS. External Secrets Operator handles secrets. Argo Rollouts handles progressive delivery. Each one does its job better than any platform's built-in equivalent. **Multi-cloud is the default.** Flexera's [State of the Cloud report](https://www.flexera.com/blog/finops/the-latest-cloud-computing-trends-flexera-2025-state-of-the-cloud-report/) has multi-cloud adoption near 89%, with organizations using an average of 2.4 public cloud providers. When your infrastructure spans AWS, GCP, and Azure, an all-in-one tool that only works well on one provider becomes a liability, not a convenience. **Best-of-breed gives you leverage.** When you're not locked into a single vendor, you can negotiate. You can switch. You can adopt a better tool when one emerges without re-platforming your entire stack. That optionality has real economic value - especially for teams that plan to be around for a while. ## The Integration Tax Is Real Here's where best-of-breed advocates usually lose credibility: they pretend integration is free. It isn't. Engineers routinely juggle a handful of separate tools to build and ship a service, and the context-switching between them is not free - it's hours a week per engineer that never reach a product backlog. Across a team, that adds up to real money. This is often called the "DevOps tax" - the cost of integrating, maintaining, and troubleshooting a multi-tool pipeline, measured in engineering hours that could have gone toward shipping features. GitLab's [DevSecOps survey](https://about.gitlab.com/blog/2022/08/24/too-many-toolchains-a-devops-platform-migration-is-the-answer/) has tracked it climbing: by 2022, nearly 40% of developers said they spent a quarter to half of their time just maintaining and integrating their toolchain, and 69% wanted to consolidate. The failure mode looks like this: your CI/CD pipeline works, but the handoff to your GitOps tool requires a custom webhook. Your monitoring catches issues, but alerting lives in a different system with its own configuration language. Your secrets management is solid, but rotating credentials requires touching three different tools. Each integration point is a potential break point. This is the honest tradeoff. Best-of-breed gives you the best tool for each job. It also gives you the job of making them work together. ## The Third Option: Best-of-Breed with a Platform Layer The choice isn't "all-in-one platform" versus "duct tape and scripts." There's a third option that's become the dominant pattern among teams that have figured this out. Team Topologies authors Matthew Skelton and Manuel Pais call this the **"thinnest viable platform"** - the smallest set of APIs, documentation, and tooling needed to let teams move fast without rebuilding the same integration work over and over. In practice, this means a platform layer that **orchestrates** best-of-breed tools rather than **replacing** them. Spotify built Backstage for exactly this reason. Their engineers were using dozens of specialized tools but had no unified way to discover, configure, and operate them. Backstage doesn't replace those tools - it provides a portal layer on top of them. It's now a CNCF incubating project used by thousands of companies, with a large community plugin ecosystem. The architecture that's emerging across the industry looks like this: | Layer | Role | Examples | | --- | --- | --- | | **Developer Portal** | Discovery, self-service, catalog | Backstage, Port | | **Platform Orchestrator** | Glue, automation, golden paths | Humanitec, Kratix | | **Best-of-Breed Tools** | Specialized capabilities | ArgoCD, Prometheus, Terraform, cert-manager | | **Infrastructure** | Compute, networking, storage | AWS, GCP, Azure, on-prem | ![All-in-One vs Best-of-Breed with a Platform Layer](/images/blog/best-of-breed-tools.png) The platform layer doesn't own your tools. It connects them. Your ArgoCD config stays in your repo. Your Prometheus rules stay where they are. The platform layer handles the integration work so your engineers don't have to. ## What to Look for in a Platform Layer Not all platforms are created equal. Some claim to "orchestrate" best-of-breed tools while quietly introducing their own lock-in. Here's what matters: **Your configuration lives in your repos.** If the platform stores your deployment configs, Helm values, or Kustomize overlays in its own proprietary format or database, you've traded one form of lock-in for another. Everything should be standard YAML/HCL in Git repos you own. **You can swap any tool.** If you decide to move from Prometheus to Grafana Mimir, or from nginx-ingress to Gateway API, the platform should make that easier - not harder. The test: can you replace any single tool in your stack without re-platforming? **It works across clouds.** AWS today, GCP tomorrow, hybrid next year. A platform layer that only works on one cloud is just lock-in wearing a different hat. **It provides opinionated defaults, not mandates.** The [80-20 rule](/blog/golden-paths-and-the-80-20-rule) applies here. Golden paths should cover 80% of use cases. For the other 20%, you need the freedom to deviate. Paths, not cages. **You can leave.** The ultimate test. If you rip out the platform layer, you should be left with standard Kubernetes manifests, standard Helm charts, and standard CI/CD pipelines that still work. The platform made your life easier - but it didn't make itself a dependency. This is the philosophy we built [Skyhook](https://www.skyhook.io) on - and the reason it isn't a row in that table. Assembling those layers yourself - a portal, an orchestrator, the best-of-breed tools, and the seams between them - is a platform team's full-time job. Skyhook collapses the stack: we orchestrate best-of-breed tools (ArgoCD, Prometheus, cert-manager, External Secrets, and dozens more) without replacing them, wire them together, and put a self-service surface on top - so a small team gets this whole architecture without staffing a platform team to build it. And it holds itself to every test above: your config lives in your Git repos as standard YAML, you can swap or modify any tool directly, and if you leave, you take everything with you. ## Making the Call If you're evaluating your infrastructure tooling strategy, here's a practical framework: **Go best-of-breed with a platform layer if:** - You want to use the best tool for each job without the integration headache - You run multi-cloud or plan to - You want to adopt new tools as the ecosystem evolves without re-platforming - You care about avoiding vendor lock-in from day one The good news: you don't need a large platform team to make this work. A platform layer handles the orchestration so even small teams get the flexibility of best-of-breed without the overhead of stitching it all together themselves. **Audit what you already have.** Most teams are already running best-of-breed tools - they just haven't invested in the platform layer to make it manageable. The integration tax is a solvable problem. You don't need to consolidate into a monolith. You need better glue. The CNCF landscape isn't going to shrink. The ecosystem will keep producing specialized tools that outperform bundled alternatives. The teams that thrive will be the ones who figured out how to use the best tool for each job - without drowning in the work of making them talk to each other. --- ### How We Automated Our Way to 5-Minute Onboarding (Hint: It's Not Just AI) - **URL**: https://www.skyhook.io/blog/how-we-automated-our-way-to-5-minute-onboarding - **Date**: March 2026 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: developer-experience, onboarding, platform-engineering, kubernetes, automation Our first onboarding flow asked users 15 questions. Language, framework, port, Dockerfile path, build command, health endpoints, environment variables. It was tedious, and half the answers required digging through the repo to find. Developers would guess, get it wrong, and not realize until the first build failed. But even when users answered everything correctly, things still broke. The Dockerfile copies a `dist/` folder that doesn't exist in CI. The build needs BuildKit but the pipeline doesn't enable it. There are seven Dockerfiles and you picked the wrong one. The port in the Dockerfile doesn't match the port in your app config. These aren't things you'd think to check - they surface as mysterious failures 30 minutes into your first deploy. The typical onboarding experience was: fill out a form, trigger a build, watch it fail, debug, fix, retry, hit the next issue. Sometimes it took hours. **We wanted to get that down to 5 minutes.** That meant solving two problems: figuring out what a codebase *is*, and figuring out what needs to *change* for it to deploy. ## Step 1: Read the Codebase The first problem is detection - language, framework, port, Dockerfile, monorepo structure. All of this information exists somewhere in your repo, but it's scattered, unstructured, and wildly inconsistent across real-world projects. A port might be in the Dockerfile, overridden in a start script, and defaulted differently by the framework. Your language is obvious from `go.mod` - unless there's also a `package.json` for build tooling, in which case it's ambiguous. We built a [detection system](/blog/reverse-ml-using-ai-to-write-rules-not-run-them) that resolves these conflicts with confidence scoring, runs in under 50ms, and handles the ambiguity without calling an LLM. That was hard enough on its own. But when we started testing across real repos, we kept running into the same pattern: detection was correct, but the first build would fail anyway. ## Step 2: Find What Will Break A Next.js app. Detection nails it - Node.js, Next.js framework, port 3000. But the Dockerfile has this: ```dockerfile COPY --chown=nextjs:nodejs .next/standalone ./ COPY --chown=nextjs:nodejs .next/static ./apps/api/.next/static ``` That `.next/` directory only exists after running `npm run build`. On the developer's machine, it's there because they built it locally. In a CI pipeline building from a clean checkout, it doesn't exist. The Docker build fails with `COPY failed: file not found`. This isn't just a detection problem. It's an *adaptation* problem. The codebase needs something to change before it can deploy - and the developer might not even realize it. We found these issues everywhere. Dockerfiles that require BuildKit syntax but CI doesn't enable it. Build contexts that reference parent directories. `ARG` declarations with no defaults. Private registry dependencies with no auth configured. Each one is a build failure waiting to happen. ## Step 3: Decide What to Do About It This is where we spent most of our engineering time. When the system finds a problem, what should it do? Our first instinct was to fix everything automatically. That's wrong. Some fixes are safe to apply silently. Others would surprise the user. And some genuinely require information only the user has. We ended up with five strategies: **Just Do It** - the fix is zero-risk and strictly better. Example: the Dockerfile uses `--mount=type=cache` (BuildKit syntax), but CI doesn't have BuildKit enabled. We add `DOCKER_BUILDKIT=1` to the workflow. BuildKit is backward-compatible. There's no scenario where this breaks anything. We don't even mention it unless you look at the detailed log. **Show & Confirm** - we're making a significant change and you should see it. Example: no Dockerfile found. We generate one based on your detected language and framework. That's a whole new file in your repo - you should see what we're creating and approve it. Same with pre-built artifacts: "Your Dockerfile expects `.next/standalone` to exist. We'll add `npm run build` before `docker build` in CI. Here's the change." **Must Ask** - we can't guess. Example: Dockerfile says `EXPOSE 8080`, but your NestJS config says port 3000. Both are valid signals. We present both options and ask which one is correct. Same with monorepo service selection - we can detect that it's a Turborepo workspace with five services, but we can't know which one you want to onboard right now. **Infer + FYI** - we pick a sensible default and tell you what we picked. Example: `ARG PORT` in the Dockerfile with no default value. We set it to the port we detected from your framework config and show you: "Set PORT=3000 based on NestJS defaults. Change this if your app uses a different port." **Follow-up** - not blocking, but worth doing later. Example: no health endpoints detected. Your service will deploy fine, but Kubernetes probes won't work properly, which means slower rollouts and potential downtime during deploys. We configure a safe default (TCP probe on the service port) and flag it as a recommended follow-up. The taxonomy sounds simple. The engineering is in correctly classifying each issue. A missing Dockerfile is "Show & Confirm" - but what if we're only 60% confident about the detected framework? Then the generated Dockerfile might be wrong, and auto-generating it with a default "yes" is risky. Confidence from detection flows directly into adaptation strategy. ## Putting It Together Here's what `skyhook init` looks like on a messy real-world repo - a Next.js app inside a Turborepo monorepo, with a Dockerfile that assumes locally-built artifacts. ``` $ skyhook init Detecting project configuration... ✓ Scanning manifests and configs ✓ Analyzing Dockerfiles ✓ Detecting monorepo structure ✓ Checking build readiness Detection Results ─────────────────────────────────────── Language: Node.js (98%) ← Dockerfile FROM node:18-alpine Framework: Next.js (85%) ← next.config.ts Port: 3002 ← package.json scripts Monorepo: Turborepo ← apps/api Adaptations ─────────────────────────────────────── ✓ Fixed: BuildKit enabled in CI workflow ⚠ Confirm: Dockerfile COPYs .next/standalone - adding 'npm run build' to CI before docker build ⚠ Confirm: ARG PORT has no default - using 3002 from package.json ``` Three things happened here beyond detection. BuildKit was silently enabled (just do it - zero risk). The pre-built artifacts issue was caught and a fix proposed (show & confirm - the user should see what's changing in their CI pipeline). And a missing ARG default was inferred from the detected port (infer + FYI). Without this, the developer would have filled out the form, hit "deploy," waited for the Docker build, and gotten `COPY failed: file not found`. Then spent 20 minutes figuring out that `.next/standalone` needs a build step before `docker build`. Then maybe hit the BuildKit issue next. Then the ARG issue after that. Instead: detection + adaptation finds all three issues upfront. The user confirms two changes and moves on. The interactive form then appears with most fields pre-filled: ``` Service name: api [auto: from monorepo path] Dockerfile: Dockerfile [auto: single Dockerfile] Environment: production ← Select ``` One question: which environment. Everything else was either detected or handled by adaptations. The system also knows when it's *not* sure. When confidence drops below 70% or signals conflict, it says so explicitly rather than guessing: ``` ⚠ Verification recommended: Node.js detected alongside Python - likely tooling only (source: package.json) ``` The worst onboarding experience isn't answering a question - it's when the platform silently picks the wrong answer and your first deploy fails for a reason you can't understand. We'd rather ask one extra question than debug a mysterious build failure. ## What We Still Get Wrong This system handles about 90% of repos correctly with zero or minimal questions. The remaining 10% are genuinely hard: - **Custom build pipelines**: If your Dockerfile shells out to a Makefile that calls a Python script that runs webpack, we're not going to trace that chain. We'll detect the language and framework, but the build command needs human input. - **Multiple services in one Dockerfile**: Multi-stage builds that produce different binaries based on build args. We detect the Dockerfile, but can't always tell which target is the one you want. - **Unconventional project structures**: A Go service where the actual entry point is three directories deep with no `main.go` at the root. Detection works, but the Dockerfile context might be wrong. For these cases, the system asks rather than guesses. We're upfront about what it can't figure out, and the interactive form is always there as a fallback. The goal was never to eliminate all questions - it was to eliminate the *unnecessary* ones. ## The Payoff We turned a 15-field form and a "hope it works" first deploy into a system that reads your codebase, finds what will break, and either fixes it or asks you about it - with enough context that the question takes seconds to answer, not minutes of investigation. Detection figures out *what* your codebase is. Adaptations figure out *what needs to happen* for it to work. Confidence scoring decides *when to act and when to ask*. Between them, onboarding goes from hours of form-filling and debugging to a few minutes and a couple of confirmations. We [wrote separately](/blog/reverse-ml-using-ai-to-write-rules-not-run-them) about how we used AI to build the detection rules. But the adaptation system - the strategy taxonomy, the confidence-based UX, knowing when to silently fix vs when to ask - that's where most of the product thinking went. AI was one piece. The rest was figuring out the right thing to do with what we found. --- ### How to Set Up ArgoCD on Kubernetes: Step-by-Step Guide - **URL**: https://www.skyhook.io/blog/how-to-set-up-argocd-on-kubernetes - **Date**: March 2026 - **Author**: Eyal Dulberg, CTO - **Category**: GitOps - **Tags**: argocd, gitops, kubernetes, tutorial, for-devops ArgoCD is a declarative, GitOps-based continuous delivery tool for Kubernetes. It watches a Git repository for changes to your manifests and automatically syncs them to your cluster - keeping your deployed state in line with what's in Git. This guide gets you from zero to a working ArgoCD installation with your first app deployed via GitOps. No fluff, no theory - just the commands and configs you need. **TL;DR**: Install ArgoCD with `kubectl apply`, port-forward to the UI, install the CLI, connect a Git repo, create an Application, and push a change. Eight steps, under 30 minutes. We also cover what you'll need beyond this for a real team. This is part of our three-part ArgoCD series. Once you're up and running, continue with [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter) for production patterns, and [ArgoCD Multi-Cluster Architecture](/blog/argocd-multi-cluster-architecture) for choosing between centralized and per-cluster deployments. While you're learning, [Radar](https://radarhq.io) auto-detects ArgoCD Applications and shows sync status alongside the resources they manage - useful as a visual companion to the CLI. ## Prerequisites You'll need: - A running Kubernetes cluster (any provider - EKS, GKE, AKS, kind, minikube) - `kubectl` configured and pointing at your cluster - `git` and a GitHub/GitLab account - A terminal with `bash` or `zsh` Verify your cluster is reachable: ```bash kubectl cluster-info ``` ## Step 1: Install ArgoCD Create a namespace and install ArgoCD with the stable manifest: ```bash kubectl create namespace argocd kubectl apply -n argocd -f \ https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml ``` This installs the non-HA (single-replica) version, which is fine for getting started. For production clusters, use the HA manifest instead: replace `install.yaml` with `ha/install.yaml` in the URL above. You can also install via Helm (`helm install argocd argo/argo-cd`) if you prefer managing ArgoCD's own configuration as a Helm release. Wait for all pods to be ready: ```bash kubectl wait --for=condition=ready pod \ --all -n argocd --timeout=120s ``` You should see 5-7 pods running: `argocd-server`, `argocd-repo-server`, `argocd-application-controller`, `argocd-applicationset-controller`, `argocd-redis`, and `argocd-dex-server`. ```bash kubectl get pods -n argocd ``` ## Step 2: Access the ArgoCD UI The quickest way to access the UI is port-forwarding: ```bash kubectl port-forward svc/argocd-server -n argocd 8080:443 ``` Open [https://localhost:8080](https://localhost:8080) in your browser. You'll get a TLS warning since it's using a self-signed certificate - that's expected for local access. ### Get the admin password ArgoCD generates an initial admin password stored in a Kubernetes secret: ```bash kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath="{.data.password}" | base64 -d ``` Log in with username `admin` and the password from above. **Important**: Change this password after your first login, or better yet, connect an SSO provider. The initial secret should be deleted once you've set up proper authentication: ```bash kubectl -n argocd delete secret argocd-initial-admin-secret ``` ## Step 3: Install the ArgoCD CLI The CLI lets you manage ArgoCD from your terminal. On macOS: ```bash brew install argocd ``` On Linux: ```bash curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 chmod +x argocd sudo mv argocd /usr/local/bin/ ``` Log in to your ArgoCD instance (with port-forward still running): ```bash argocd login localhost:8080 --username admin --password --insecure ``` The `--insecure` flag is fine for local port-forward. For production, set up proper TLS with an Ingress or Gateway. ## Step 4: Create a Sample App Repository You need a Git repo with Kubernetes manifests for ArgoCD to sync. Create a simple one: ```bash mkdir argocd-demo && cd argocd-demo git init ``` Create a file called `deployment.yaml` with a basic Nginx deployment: ```yaml # deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: demo-app labels: app: demo spec: replicas: 2 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: nginx image: nginx:1.27 ports: - containerPort: 80 resources: requests: cpu: "50m" memory: "64Mi" limits: memory: "128Mi" ``` And a file called `service.yaml` to expose it: ```yaml # service.yaml apiVersion: v1 kind: Service metadata: name: demo-app spec: selector: app: demo ports: - port: 80 targetPort: 80 ``` Commit and push to your Git provider: ```bash git add . git commit -m "Initial demo app" git remote add origin git@github.com:/argocd-demo.git git push -u origin main ``` ## Step 5: Connect the Repo to ArgoCD If your repo is public, ArgoCD can access it directly. For private repos, add credentials: ```bash argocd repo add git@github.com:/argocd-demo.git \ --ssh-private-key-path ~/.ssh/id_ed25519 ``` Or via HTTPS with a token: ```bash argocd repo add https://github.com//argocd-demo.git \ --username --password ``` Verify the connection: ```bash argocd repo list ``` ## Step 6: Deploy Your First Application Now the good part. Create an ArgoCD Application that tells it what to sync and where: ```bash argocd app create demo-app \ --repo https://github.com//argocd-demo.git \ --path . \ --dest-server https://kubernetes.default.svc \ --dest-namespace default ``` Or apply it as YAML for a more GitOps-native approach: ```yaml # argocd-application.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: demo-app namespace: argocd spec: project: default source: repoURL: https://github.com//argocd-demo.git targetRevision: HEAD path: . destination: server: https://kubernetes.default.svc namespace: default syncPolicy: automated: prune: false # Don't delete resources removed from Git selfHeal: true # Re-sync if someone edits the cluster directly syncOptions: - CreateNamespace=true ``` ```bash kubectl apply -f argocd-application.yaml ``` ## Step 7: Sync and Verify If you used `automated` sync policy, ArgoCD will sync within a few minutes. To trigger it immediately: ```bash argocd app sync demo-app ``` Check the status: ```bash argocd app get demo-app ``` You should see: ``` Name: demo-app Health Status: Healthy Sync Status: Synced ``` In the ArgoCD UI, you'll see a visual map of your deployment - the Application, Deployment, ReplicaSet, Pods, and Service all connected. Verify the pods are running: ```bash kubectl get pods -l app=demo ``` ## Step 8: Make a Change via Git This is where GitOps clicks. Edit the replica count in your repo: ```yaml # deployment.yaml - change replicas from 2 to 3 spec: replicas: 3 ``` Commit and push: ```bash git add deployment.yaml git commit -m "Scale demo app to 3 replicas" git push ``` Within 3 minutes (ArgoCD's default polling interval), you'll see the change reflected in the cluster. Or trigger it immediately: ```bash argocd app sync demo-app ``` That's the GitOps workflow: change Git, cluster converges. No `kubectl apply`, no direct cluster access needed. ## What You Have Now At this point you have: - ArgoCD installed and running - A Git repo connected as a source - An application syncing from Git to your cluster - Automated self-healing (drift correction) This is enough for a single developer experimenting with GitOps. It's not enough for a team. ## What's Missing for Production The setup above works for a demo. For a real team, you'll quickly hit these gaps: **Multi-cluster management**: You need the same addons across dev, staging, and prod - but with different resource limits, replica counts, and configs. Copying YAML files per cluster doesn't scale. See [ArgoCD Multi-Cluster Architecture](/blog/argocd-multi-cluster-architecture) for choosing between centralized and per-cluster deployments. **Developer self-service**: Your developers can't (and shouldn't) be expected to write ArgoCD Application YAML and Kustomize patches. They need an interface that generates the right config for them. **Access control**: The default `admin` account can do anything. You need RBAC with AppProjects to isolate teams and limit who can sync what. **Deletion protection**: One accidental `kubectl delete` on an ApplicationSet can cascade into deleting every Application it manages. You need multiple layers of protection. **Ordered deployments**: If your app needs database migrations before the service starts, or config before the deployment, you need sync waves and hooks - which means more YAML annotations developers need to understand. **Secrets management**: Kubernetes Secrets are base64-encoded, not encrypted. You'll need Sealed Secrets, External Secrets Operator, or a vault integration before storing anything sensitive in Git. We cover the patterns that solve these problems in [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter) - including multi-cluster file organization, ApplicationSet auto-discovery, three-layer deletion protection, and sync waves for ordered deployments. Or, if you'd rather skip the manual setup entirely - [Skyhook](https://skyhook.io) gives you a production-ready ArgoCD setup out of the box, with all of these patterns configured from day one. ## Frequently Asked Questions ### Is ArgoCD free? Yes. ArgoCD is a free, open-source CNCF project licensed under Apache 2.0. There's no paid tier or enterprise edition of the core project. Commercial platforms like [Akuity](https://akuity.io) and [Skyhook](https://skyhook.io) build management layers on top of it, but ArgoCD itself is fully free. ### How often does ArgoCD sync with Git? By default, ArgoCD polls your Git repository every 3 minutes. You can configure this with the `timeout.reconciliation` setting in the `argocd-cm` ConfigMap, or set up [Git webhooks](https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/) for near-instant sync on push. ### Can I install ArgoCD with Helm instead of kubectl? Yes. The ArgoCD community maintains an official Helm chart at `argo/argo-cd`. Run `helm repo add argo https://argoproj.github.io/argo-helm && helm install argocd argo/argo-cd -n argocd --create-namespace`. Helm gives you more control over ArgoCD's own configuration values, which is useful if you plan to manage ArgoCD's config as code. ### How do I expose ArgoCD outside the cluster? For production, replace port-forwarding with an Ingress or Gateway resource pointing to the `argocd-server` service. You'll want TLS termination - either via cert-manager with Let's Encrypt or your cloud provider's certificate manager. The [ArgoCD Ingress docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/) cover configuration for nginx, Traefik, and AWS ALB. ## Further Reading - [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter) - Multi-cluster patterns, deletion protection, sync waves, and developer self-service - [ArgoCD Multi-Cluster Architecture: Centralized vs Per-Cluster](/blog/argocd-multi-cluster-architecture) - Choosing the right deployment model for your team - [Managing Kubernetes Add-ons: Argo CD or Terraform?](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons) - When to use ArgoCD vs Terraform for different lifecycle stages - [ArgoCD Official Getting Started Guide](https://argo-cd.readthedocs.io/en/stable/getting_started/) - Full documentation from the ArgoCD project --- ### ArgoCD Multi-Cluster Architecture: Centralized vs Per-Cluster - **URL**: https://www.skyhook.io/blog/argocd-multi-cluster-architecture - **Date**: March 2026 - **Author**: Eyal Dulberg, CTO - **Category**: GitOps - **Tags**: argocd, gitops, kubernetes, multi-cluster, for-devops Installing ArgoCD on your first cluster is straightforward - we cover it in [How to Set Up ArgoCD on Kubernetes](/blog/how-to-set-up-argocd-on-kubernetes). The second cluster is where architecture decisions start. Do you point your existing ArgoCD at the new cluster, or install a separate instance? This is part of our three-part ArgoCD series. For production patterns that apply regardless of architecture, see [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter). This is the ArgoCD decision that's hardest to reverse. Migrating between architectures means redefining how every application is managed, where credentials live, and how your team operates. Choose well upfront and you avoid a painful rearchitecture later. **TL;DR**: There are three ArgoCD multi-cluster deployment models - centralized (hub-spoke), per-cluster (standalone), and agent-based (emerging). Centralized gives you a single dashboard but creates security and blast radius risks; per-cluster isolates failures at the cost of operational overhead; the agent model isn't production-ready yet. An orchestration layer above ArgoCD tips the scales toward per-cluster - it solves the management gaps while preserving isolation. ## The Three Deployment Models ### Centralized (Hub-Spoke) ![Centralized ArgoCD architecture: one ArgoCD instance in a management cluster managing multiple remote clusters](/images/blog/argocd-arch-centralized.png) One ArgoCD instance runs in a management cluster and manages deployments across your fleet. ArgoCD connects to each cluster's API server using credentials stored as Kubernetes Secrets in the argocd namespace. It polls Git, renders manifests, and pushes desired state to each cluster over the network. **What's good:** - One dashboard for sync status across all clusters - RBAC, SSO, and notifications configured once - ApplicationSets expand natively across all managed clusters - Less operational overhead with few clusters **What breaks:** - **Security surface grows with every cluster.** The management cluster stores kubeconfigs or bearer tokens for your entire fleet. Compromise it and an attacker gets credentials to everything. - **Blast radius is total.** A bad ArgoCD upgrade, misconfiguration, or OOM kill on the management cluster affects every cluster simultaneously. - **Network dependency on every cluster.** ArgoCD needs persistent access to every cluster's API server. Cross-region, cross-cloud, VPN tunnels, NAT traversal - the networking complexity compounds. - **Resource pressure scales linearly.** The application controller caches resource state for every managed cluster. At 15-20+ clusters, memory and CPU grow significantly even with controller sharding enabled. A [visibility tool like Radar](https://radarhq.io/product/cluster-audit) can give you a fleet-wide dashboard without the centralized credential footprint - each cluster runs Radar locally and reports upstream over an outbound-only tunnel. ### Per-Cluster (Standalone) ![Per-cluster ArgoCD architecture: each cluster runs its own ArgoCD instance managing local workloads](/images/blog/argocd-arch-per-cluster.png) Each cluster runs its own ArgoCD instance that manages only the local cluster. Each instance watches Git and syncs to its own cluster. No cross-cluster credentials, no cross-cluster networking. **What's good:** - **Credentials stay local.** A compromised cluster doesn't hand over access to every other cluster. - **Failures are contained.** A bad ArgoCD upgrade on Cluster A doesn't touch Cluster B. - **No cross-cluster network requirements.** ArgoCD needs to reach Git and its own API server - that's it. - **Independent lifecycle.** Teams can upgrade, test, and roll back ArgoCD per cluster on their own schedule. **What's harder:** - **No unified view.** You have N dashboards instead of one. Checking sync status across clusters means N browser tabs. - **Configuration drift.** RBAC, SSO, notification configs need replication across instances. Without automation, they drift. - **Operational overhead.** N instances to monitor, patch, upgrade, and troubleshoot. - **Cross-cluster coordination.** ApplicationSets only see their local cluster. Deploying the same app across clusters requires coordinating at the Git level, not the ArgoCD level. ### Agent-Based (Emerging) ![Agent-based ArgoCD architecture: lightweight agents on each cluster connect outbound to a central control plane](/images/blog/argocd-arch-agent-based.png) A lightweight agent on each cluster communicates back to a central control plane. Unlike hub-spoke, the control plane doesn't store cluster credentials or connect to cluster API servers. The agent pulls work and applies it locally. Connections are outbound from the workload cluster, not inbound - the control plane never reaches into cluster API servers directly. **What it promises:** - Single pane of glass without the security surface of centralized - Resource-heavy work (rendering, reconciliation) runs on the workload cluster - Outbound-only connections from agents work through firewalls without special configuration **Where it stands today:** - Technology preview in Red Hat OpenShift GitOps 1.19+ - Not yet part of core ArgoCD OSS - Feature parity with standard ArgoCD still catching up - Limited production battle-testing The agent model is where multi-cluster ArgoCD is headed long-term. For teams making architecture decisions today, it's one to watch rather than depend on. ## What Actually Drives the Decision Five factors matter more than cluster count: | Factor | Favors Centralized | Favors Per-Cluster | |---|---|---| | **Security posture** | Trusted network, low compliance bar | Multi-tenant, regulated, zero-trust | | **Blast radius tolerance** | Can absorb coordinated downtime | Prod must be isolated from dev/staging | | **Cluster count** | 2-3 clusters | 5+ (especially 10+) | | **Team structure** | Single platform team | Multiple teams with cluster ownership | | **Network topology** | Same VPC/region, simple routing | Cross-region, cross-cloud, air-gapped | Security and blast radius matter more than cluster count. Three clusters in a regulated environment should probably be per-cluster. Ten clusters in a single-team dev shop might still work centralized. The common advice is "start centralized under 5 clusters, switch to per-cluster at 10+." That's fine as a starting point, but it oversimplifies. The real question is whether you can accept the security and blast radius tradeoffs of centralized - and whether you have a plan for migrating when you can't. ## Why an Orchestration Layer Tips the Scales Here's an observation that shifts the decision: per-cluster's downsides are management problems, not architectural problems. "No unified view" means there's nothing above ArgoCD aggregating status. "Configuration drift" means there's nothing generating consistent ArgoCD configs. "Operational overhead" means there's nothing managing ArgoCD's lifecycle across clusters. The core argument for centralized is really an argument for having a management layer. Running one ArgoCD instance happens to be the simplest way to get one - until you hit the security and scaling walls. An orchestration layer above ArgoCD - something that manages visibility, configuration, and lifecycle across instances - changes the tradeoff matrix (we explore what this layer looks like in practice in [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter)): | Per-Cluster Downside | What an Orchestration Layer Provides | |---|---| | No unified dashboard | Cross-cluster visibility in one place | | Config drift between instances | Configuration generated centrally, applied per-cluster | | N instances to upgrade | Lifecycle management across the fleet | | No cross-cluster ApplicationSets | Deployment coordination at a higher level than ArgoCD | With this layer in place, per-cluster gives you genuine architectural benefits - credential isolation, blast radius containment, independent failure domains - without the operational tax that pushes teams toward centralized by default. This is why we lean toward per-cluster for most teams, even those with only a few clusters. The isolation benefits are real from day one. The operational downsides are solvable with tooling. And starting centralized means signing up for a migration you'd rather avoid when you eventually outgrow it. You can build this orchestration layer yourself - plenty of teams have. It's months of work to get right and it evolves with your infrastructure. Or you can use an existing platform. Either way, the architectural decision should be based on the properties you want (isolation vs simplicity), not on which model happens to be easier to manage without additional tooling. ## Our Recommendation **Default to per-cluster.** The security isolation and blast radius containment are architectural benefits you get immediately. The operational overhead is real but solvable with tooling above ArgoCD - and much easier to solve than migrating away from centralized later. **Centralized still makes sense** for 2-3 clusters in a single network with one team and no compliance requirements. Just plan for the migration. Teams rarely stay at 2-3 clusters. **Watch the agent model.** If you need a single pane of glass and don't have an orchestration layer, the agent approach is the most promising path to getting centralized visibility without the security tradeoffs. Give it another year or two of production hardening. For the production patterns that apply regardless of which architecture you choose - Kustomize overlays for multi-cluster YAML, ApplicationSet generators, deletion protection, sync waves, and developer self-service - read [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter). [Skyhook](https://skyhook.io) deploys ArgoCD per-cluster by default and provides the orchestration layer on top - unified visibility, generated configuration, and managed lifecycle across your fleet. The patterns in this series describe what it automates. ## Frequently Asked Questions ### How many clusters can one ArgoCD instance manage? There's no hard limit, but practically the application controller becomes resource-constrained around 15-20 clusters. ArgoCD caches the resource state for every managed cluster in memory. You can extend this with [controller sharding](https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-application-controller) - splitting clusters across multiple controller replicas - but the security and blast radius concerns of centralized don't go away with sharding. ### Can I migrate from centralized to per-cluster ArgoCD? Yes, but it's painful. You need to install ArgoCD on each cluster, recreate every Application resource locally, migrate credentials and RBAC, update your Git repo structure, and cut over - all without downtime. The migration typically takes weeks for production setups. This is why we recommend starting per-cluster: it's easier to add an orchestration layer on top than to decompose a centralized instance later. ### What is ArgoCD agent mode? ArgoCD agent mode is an emerging architecture where a lightweight agent runs on each workload cluster and communicates outbound to a central control plane. Unlike centralized ArgoCD, the control plane never stores cluster credentials or connects to cluster API servers directly. It's currently a technology preview in Red Hat OpenShift GitOps 1.19+ and not yet part of core ArgoCD OSS. ### Should I use ArgoCD or Flux for multi-cluster? Both are CNCF-graduated GitOps tools. ArgoCD has a built-in UI, ApplicationSets for multi-cluster templating, and a larger ecosystem. Flux is lighter-weight, uses native Kubernetes primitives (no custom UI), and has built-in multi-tenancy via Flux's Kustomization CRD. For multi-cluster specifically, ArgoCD's ApplicationSet generators make cross-cluster templating easier, while Flux relies more on Git structure and Kustomize. We cover ArgoCD here because it's what most teams adopt, but Flux is a solid choice - especially for teams that prefer a more Kubernetes-native approach. ## Further Reading - [How to Set Up ArgoCD on Kubernetes](/blog/how-to-set-up-argocd-on-kubernetes) - Step-by-step installation guide, from zero to your first synced application - [ArgoCD in Production: Patterns That Actually Matter](/blog/argocd-in-production-patterns-that-actually-matter) - Production patterns for deletion protection, sync waves, and developer self-service - [Managing Kubernetes Add-ons: Argo CD or Terraform?](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons) - When to use each tool for different lifecycle stages - [ArgoCD Agent Architecture](https://argocd-agent.readthedocs.io/latest/concepts/architecture/) - Technical deep dive into the agent-based model - [ArgoCD Cluster Management](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters) - Official docs for managing external clusters --- ### ArgoCD in Production: Patterns That Actually Matter - **URL**: https://www.skyhook.io/blog/argocd-in-production-patterns-that-actually-matter - **Date**: March 2026 - **Author**: Eyal Dulberg, CTO - **Category**: GitOps - **Tags**: argocd, gitops, kubernetes, platform-engineering, for-devops Installing ArgoCD takes 10 minutes. Getting your first app syncing from Git feels like magic - you push a commit, ArgoCD detects the change, and your cluster converges to the desired state. Clean, elegant, done. Then reality hits. Your second cluster needs the same addons but different resource limits. A developer accidentally deletes an ApplicationSet and takes down 12 services. Your team of 40 engineers is filing Jira tickets asking DevOps to "please update the replica count." And you're drowning in YAML files that are 80% identical across environments. The gap between "ArgoCD installed" and "developers can actually ship" is enormous. This post covers the ArgoCD best practices and production patterns that bridge it. **TL;DR**: ArgoCD is free and powerful, but running it for a real team requires patterns it doesn't ship with - separating addons from app services, Kustomize base/overlays to eliminate YAML duplication, ApplicationSet generators for multi-cluster auto-discovery, three-layer deletion protection, sync waves for ordered deployments, and a generation layer so developers never touch YAML directly. We cover all of them with working examples. ## Why GitOps, Why ArgoCD The GitOps model is simple: Git is your source of truth. Every change is a commit - auditable, reviewable, reversible. Rollback is `git revert`. Drift detection is automatic. No more SSH-ing into a cluster to `kubectl apply` a hotfix that nobody documents. ArgoCD has become the default GitOps engine for good reasons. The built-in UI gives you real-time visibility into sync status across clusters. ApplicationSets let you template deployments across dozens of clusters from a single definition. The ecosystem is massive - hundreds of integrations, active development, and battle-tested at scale by companies like Intuit (where ArgoCD originated), Red Hat, and dozens of CNCF adopters. But ArgoCD is a tool, not a platform. It gives you a powerful engine - what you build on top of it determines whether your team ships faster or drowns in YAML. This is part of our three-part ArgoCD series. Start with [How to Set Up ArgoCD on Kubernetes](/blog/how-to-set-up-argocd-on-kubernetes) if you haven't installed it yet, or see [ArgoCD Multi-Cluster Architecture](/blog/argocd-multi-cluster-architecture) for choosing between centralized and per-cluster deployments. ## Centralized vs Per-Cluster ArgoCD The first architectural decision: do you run one ArgoCD instance managing all clusters, or one per cluster? | | Centralized | Per-Cluster | |---|---|---| | **Visibility** | Single pane of glass across all clusters | Separate UI per cluster | | **Auth complexity** | High - needs credentials for every remote cluster | Low - only manages local cluster | | **Blast radius** | One misconfiguration affects everything | Isolated - failure stays local | | **Operational overhead** | One instance to maintain | N instances to upgrade, monitor, patch | | **Network requirements** | Needs network path to all clusters | No cross-cluster networking needed | **Our take**: Per-cluster is the safer default for production - credential isolation and blast radius containment are hard to retrofit later. Centralized works for small setups (2-3 clusters, single team, same network) but you'll likely outgrow it. We go deep on this decision - including how an orchestration layer above ArgoCD changes the calculus - in [ArgoCD Multi-Cluster Architecture: Centralized vs Per-Cluster](/blog/argocd-multi-cluster-architecture). Either way, the patterns in this post apply regardless of which architecture you choose. They're about what ArgoCD manages, not how many instances you run. For visibility across per-cluster ArgoCD instances without the blast radius of centralized, tools like [Radar](https://radarhq.io/vs/lens) give you a GitOps-aware view of sync status across clusters. ## Addons vs Application Services: Different Beasts This is where most ArgoCD setups start to creak. Teams treat cert-manager and their user-facing API the same way - same ArgoCD project, same sync policies, same review process. But they're fundamentally different: | | Platform Addons | Application Services | |---|---|---| | **Examples** | cert-manager, ingress-nginx, external-dns, Prometheus | user-api, payment-service, frontend | | **Owned by** | Platform / DevOps team | Application teams | | **Change frequency** | Monthly or quarterly | Multiple times per day | | **Rollout strategy** | Careful, coordinated, often cluster-by-cluster | Fast, per-service, canary or rolling | | **Risk profile** | Breaking cert-manager breaks TLS for everything | Breaking one service affects one service | | **Config source** | Helm charts from external repos | Your own repo with Kustomize overlays | Mixing these creates real problems. A developer pushing a frontend change shouldn't need to understand why their sync is queued behind a Prometheus upgrade. An addon upgrade shouldn't be blocked because a dev team has an unresolved sync error on their service. **Separate them**. Use distinct ArgoCD AppProjects for addons and application workloads. Give them different sync policies - addons get `automated: false` with manual promotion, application services get `automated: true` with `selfHeal: true`. Different RBAC rules, different notification channels. A minimal AppProject for isolating a team's workloads looks like this: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: team-payments namespace: argocd spec: sourceRepos: - "https://github.com/your-org/payment-*" destinations: - server: https://kubernetes.default.svc namespace: "payments-*" clusterResourceWhitelist: [] # No cluster-scoped resources roles: - name: deployer policies: - p, proj:team-payments:deployer, applications, sync, team-payments/*, allow - p, proj:team-payments:deployer, applications, get, team-payments/*, allow ``` This restricts the team to their own repos and namespaces - they can't accidentally deploy into another team's namespace or sync from an unauthorized repo. The directory structure should reflect this: ``` argocd/ ├── addons/ # Platform team owns this │ ├── security/ │ │ ├── cert-manager/ │ │ └── sealed-secrets/ │ ├── observability/ │ │ ├── prometheus/ │ │ └── grafana/ │ └── networking/ │ └── ingress-nginx/ │ └── workloads/ # App teams own this ├── user-api/ ├── payment-service/ └── frontend/ ``` ## Eliminating YAML Duplication Across Clusters You have 3 clusters - dev, staging, prod. You have 8 addons. That's 24 near-identical sets of YAML if you're not careful. Add a fourth cluster and you're copying files again, hoping you don't miss a value. ### Base + Overlays with Kustomize Kustomize's overlay pattern is the foundation. One `base/` directory holds the shared manifests. Each cluster gets an `overlays/` directory with only the differences. ``` addons/cert-manager/ ├── base/ │ ├── kustomization.yaml │ ├── deployment.yaml # Shared: image, ports, health checks │ └── rbac.yaml # Shared: same permissions everywhere └── overlays/ ├── dev/ │ ├── kustomization.yaml # resources: [../../base] │ └── patch-resources.yaml # memory: 256Mi, replicas: 1 ├── staging/ │ ├── kustomization.yaml │ └── patch-resources.yaml # memory: 256Mi, replicas: 2 └── prod/ ├── kustomization.yaml └── patch-resources.yaml # memory: 512Mi, replicas: 3 ``` The overlay patches are small - just the delta from base: ```yaml # overlays/prod/patch-resources.yaml apiVersion: apps/v1 kind: Deployment metadata: name: cert-manager spec: replicas: 3 template: spec: containers: - name: cert-manager resources: requests: memory: "512Mi" limits: memory: "512Mi" ``` When you need to bump the cert-manager image version, you change it once in `base/deployment.yaml`. Every cluster gets the update. ### ApplicationSet Auto-Discovery The second layer of duplication elimination: don't manually create an ArgoCD Application for each cluster. Use an ApplicationSet with a Git file generator that auto-discovers clusters. ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: cert-manager namespace: argocd spec: generators: - git: repoURL: https://github.com/your-org/infrastructure revision: HEAD files: - path: "clusters/*/config.json" template: metadata: name: "cert-manager-{{cluster.name}}" spec: project: platform-addons source: repoURL: https://github.com/your-org/infrastructure path: "addons/cert-manager/overlays/{{cluster.name}}" destination: server: "{{cluster.server}}" namespace: cert-manager syncPolicy: automated: prune: false selfHeal: true ``` Each cluster has a config file at `clusters/{name}/config.json`: ```json { "cluster": { "name": "prod-us-east", "server": "https://prod-us-east.example.com", "provider": "gke", "region": "us-east1" } } ``` Add a new cluster? Create its config file and overlay directory. The ApplicationSet picks it up automatically. No copy-pasting Application manifests. ### Group by Capability, Not by Cluster Organize addons by what they do, not where they run. Group `cert-manager` and `sealed-secrets` under `security/`. Group `prometheus` and `grafana` under `observability/`. This keeps related configuration together and makes it obvious what's deployed where. ## Letting Developers Self-Serve Without Losing Control Here's the uncomfortable truth: you can set up ArgoCD perfectly - clean directory structure, ApplicationSets, Kustomize overlays - and developers still can't ship without filing a ticket. Why? Because the interface to ArgoCD is YAML files in a Git repo. Developers need to know which directory to edit, which fields to change, what values are valid, and how Kustomize patches work. That's not developer self-service. That's "we replaced the Jira ticket with a Git commit that's equally likely to break things." Real self-service means developers interact with something that generates the right YAML for them - whether that's a CLI tool, an API, or a UI. The generated files land in a branch, go through a PR review, and merge to trigger ArgoCD sync. Developers get autonomy. Operators get a review gate. Git stays the source of truth. The pattern looks like this: 1. **Developer** requests a change (new service, config update, scaling change) 2. **Generation layer** produces valid Kustomize manifests and commits them to a feature branch 3. **PR review** by the platform team (or automated policy checks) before merge 4. **ArgoCD** picks up the merged change and syncs to the cluster The generation layer is the hard part. It encodes your organization's conventions - naming standards, resource limits, required labels, network policies. Without it, you're relying on developers to read a wiki page and get the YAML right. They won't, and you can't blame them. ## Deletion Protection: Three Layers Deep One pattern that's non-negotiable in production: deletion protection. ArgoCD is powerful enough to delete everything it manages, and a single misconfiguration can cascade fast. Layer 1 - **ApplicationSet level**: Prevent the ApplicationSet from nuking all its Applications if it's accidentally deleted. ```yaml spec: preserveResourcesOnDeletion: true ``` Layer 2 - **Sync policy level**: Prevent ArgoCD from deleting resources that are removed from Git. This catches the "someone removed a file by mistake" scenario. ```yaml spec: syncPolicy: automated: prune: false # Don't auto-delete resources missing from Git ``` Layer 3 - **Resource annotation level**: Protect individual critical resources from deletion even during intentional cleanup operations. ```yaml metadata: annotations: argocd.argoproj.io/sync-options: Delete=false ``` Use all three. Layer 1 protects against ApplicationSet deletion. Layer 2 protects against accidental file removal. Layer 3 protects your most critical resources (AppProjects, namespaces, CRDs) from any deletion path. They're additive and each catches scenarios the others miss. ## Sync Waves: Coordinating Ordered Deployments and Database Migrations Teams migrating from a PaaS to Kubernetes often hit this surprise: on a PaaS, deployment ordering is built in. You define "run migrations after deploy" and it just works. In GitOps, everything applies at once by default. ArgoCD solves this with sync waves and hooks. Sync waves assign an integer to each resource - lower numbers deploy first, higher numbers wait. Hooks run jobs at specific lifecycle points (PreSync, PostSync, SyncFail). Here's a real scenario: a team migrating from a PaaS needs their deployment to follow a specific order. First, prepare config and secrets. Then deploy the service. Then run the database migration. Finally, clear the cache. ```yaml # Wave 0: Config and secrets (must exist before the app starts) apiVersion: v1 kind: ConfigMap metadata: name: api-config annotations: argocd.argoproj.io/sync-wave: "0" data: DATABASE_URL: "postgres://db.internal:5432/app" CACHE_ENDPOINT: "redis.internal:6379" --- # Wave 1: Main deployment (waits for config to be ready) apiVersion: apps/v1 kind: Deployment metadata: name: api annotations: argocd.argoproj.io/sync-wave: "1" spec: replicas: 3 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: api image: ghcr.io/org/api:v2.1.0 envFrom: - configMapRef: name: api-config ``` For the database migration and cache cleanup, you use PostSync hooks - they only run after the main deployment succeeds: ```yaml # Wave 2: Database migration (runs after deployment is healthy) apiVersion: batch/v1 kind: Job metadata: name: db-migrate annotations: argocd.argoproj.io/hook: PostSync argocd.argoproj.io/sync-wave: "2" argocd.argoproj.io/hook-delete-policy: BeforeHookCreation spec: template: spec: containers: - name: migrate image: ghcr.io/org/api:v2.1.0 command: ["./migrate", "--direction", "up"] restartPolicy: Never backoffLimit: 1 --- # Wave 3: Cache cleanup (runs after migration completes) apiVersion: batch/v1 kind: Job metadata: name: cache-flush annotations: argocd.argoproj.io/hook: PostSync argocd.argoproj.io/sync-wave: "3" argocd.argoproj.io/hook-delete-policy: BeforeHookCreation spec: template: spec: containers: - name: flush image: ghcr.io/org/api:v2.1.0 command: ["./cache-cli", "flush", "--prefix", "v2.0-"] restartPolicy: Never backoffLimit: 1 ``` The `BeforeHookCreation` deletion policy cleans up the previous Job before creating a new one on the next sync - otherwise you'll get name conflicts. This works, but notice the effort. Four YAML files, specific annotations, an understanding of wave ordering and hook lifecycle. On a PaaS, this was a three-line config. That's not a criticism of ArgoCD - it's a more powerful model. But it's a real gap in developer experience that someone has to fill. ## The Missing Layer Every pattern in this post works. Thousands of teams use them in production. But here's what they all have in common: someone has to build and maintain them. The Kustomize base/overlay structure doesn't create itself. ApplicationSets need to be designed, tested, and updated as your needs evolve. Deletion protection has to be applied consistently - miss one resource and you're exposed. Sync waves require developers to understand ArgoCD's annotation model. And the self-service generation layer? That's a full internal product. This is the real cost of ArgoCD in production. The tool is free. The platform layer you need on top of it - the conventions, guardrails, generation tooling, and developer experience - is months of work. And it's ongoing work, because the platform evolves with your team. That's what [Skyhook](https://skyhook.io) gives you out of the box. ArgoCD runs under the hood, configured with the patterns described here. Developers deploy through a UI and CLI without touching YAML. Addons and workloads are separated with proper isolation. Deletion protection is on by default. And when a team needs sync waves for an ordered deployment, it's a configuration option - not a week of YAML engineering. If you want to build this yourself, you now have the blueprint. If you'd rather skip straight to shipping - [give Skyhook a try](https://skyhook.io). ## Further Reading - [How to Set Up ArgoCD on Kubernetes](/blog/how-to-set-up-argocd-on-kubernetes) - Step-by-step installation guide, from zero to your first synced application - [ArgoCD Multi-Cluster Architecture: Centralized vs Per-Cluster](/blog/argocd-multi-cluster-architecture) - Choosing the right deployment model for your team - [Managing Kubernetes Add-ons: Argo CD or Terraform?](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons) - Deep comparison of when to use each tool - [Ship Without Fire Drills: Canary, Blue-Green, and Rolling Deploys](/blog/ship-without-fire-drills-canary-blue-green-and-rolling-deploys) - Progressive delivery strategies that pair well with ArgoCD - [ArgoCD Sync Waves Documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) - Official reference for sync phases and waves - [ApplicationSet Generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/) - Full list of generator types for multi-cluster patterns --- ### 50-100x Faster: Building an In-Memory Kubernetes Resource Cache with SharedInformers - **URL**: https://www.skyhook.io/blog/sharedinformer-caching - **Date**: February 2026 - **Author**: Eyal Dulberg, CTO - **Category**: Kubernetes - **Tags**: kubernetes, go, performance, caching, client-go, deep-dive Every call to the Kubernetes API server costs network round-trip time plus server-side processing. For a small cluster, a LIST call might take 50-100ms. For a larger cluster with hundreds of pods, it can easily take several hundred milliseconds to seconds - the [Kubernetes SLO](https://github.com/kubernetes/community/blob/master/sig-scalability/slos/api_call_latency.md) for LIST calls allows up to 30 seconds at p99. For a dashboard that displays pods, services, deployments, and nodes, even the best case adds up fast. We wanted sub-50ms responses even for larger clusters or busy agents. This post explains how we got there using SharedInformers - the agent-side caching layer. The companion post on [push-based sync](/blog/push-sync-architecture) covers how this cached data gets from the agent to the backend. ## The Pattern: Watch, Don't Poll The naive approach to Kubernetes data is polling: ```go // Don't do this for { pods, _ := client.CoreV1().Pods("").List(ctx, metav1.ListOptions{}) cache.Store(pods) time.Sleep(10 * time.Second) } ``` Problems: - 50ms-seconds latency per request depending on cluster size - Stale data between polls - Load on API server scales with poll frequency - No notification when data actually changes The Kubernetes ecosystem solved this years ago with the **Informer** pattern. Informers use the Watch API: one initial LIST to populate the cache, then a persistent connection that receives events as they happen. ```go // SharedInformer pattern informer := cache.NewSharedInformer( &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { ... }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { ... }, }, &v1.Pod{}, 0, // No resync period ) ``` The "Shared" prefix means multiple consumers can use the same informer. Without sharing, each consumer would create its own watch connection - wasteful. ## Our Implementation We cache 17 resource types across the entire cluster: | Category | Resources | |----------|-----------| | Core | Pod, Service, Node, Namespace, PVC, ConfigMap, ServiceAccount | | Apps | Deployment, DaemonSet, StatefulSet, ReplicaSet | | Networking | Ingress, IngressClass | | Batch | Job, CronJob | | Storage | StorageClass | | Events | Event | ### Initialization ```go func InitResourceCache() error { config, err := rest.InClusterConfig() if err != nil { return err } clientset, err := kubernetes.NewForConfig(config) if err != nil { return err } // SharedInformerFactory creates informers that share the same connection factory := informers.NewSharedInformerFactory(clientset, 0) rc := &ResourceCache{ factory: factory, clientset: clientset, changes: make(chan ResourceChange, 1000), } // Register all informers rc.pods = factory.Core().V1().Pods() rc.services = factory.Core().V1().Services() rc.deployments = factory.Apps().V1().Deployments() // ... 14 more // Add event handlers for change notifications rc.pods.Informer().AddEventHandler(rc.makeHandler("Pod")) rc.services.Informer().AddEventHandler(rc.makeHandler("Service")) // ... // Start all informers factory.Start(rc.stopCh) // Wait for initial cache population if !cache.WaitForCacheSync(rc.stopCh, rc.pods.Informer().HasSynced, ...) { return errors.New("failed to sync caches") } return nil } ``` Key points: - **No resync period**: We set it to 0. The watch provides continuous updates; periodic full re-lists just add load. - **SharedInformerFactory**: All informers share resources (goroutines, connections). - **WaitForCacheSync**: Block until initial LIST completes. Don't serve stale data on startup. ### Usage Once initialized, lookups are trivial: ```go // List all pods in a namespace - 1-2ms pods, err := cache.Pods().Pods("default").List(labels.Everything()) // Get specific service - under 1ms svc, err := cache.Services().Services("default").Get("my-service") // List with label selector - 1-2ms selector := labels.SelectorFromSet(map[string]string{"app": "api"}) pods, err := cache.Pods().Pods("").List(selector) ``` Compare to direct API calls: ```go // Direct API - 50ms to seconds depending on cluster size pods, err := clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{}) ``` **Orders of magnitude faster.** For dashboards displaying multiple resource types, this turns multi-second loads into sub-10ms. ## Memory Optimization: Stripping Metadata Raw Kubernetes objects carry baggage. A typical Pod object includes: - `metadata.managedFields`: Tracks which controller owns which fields. Can be 30% of object size. - `kubectl.kubernetes.io/last-applied-configuration`: The entire manifest that was applied. 10-100KB per object. For a cache used in UI display, we don't need either. We strip them: ```go func dropManagedFields(obj interface{}) (interface{}, error) { accessor, err := meta.Accessor(obj) if err != nil { return obj, nil } // Drop managedFields - ~30% memory savings accessor.SetManagedFields(nil) // Drop last-applied-configuration annotation annotations := accessor.GetAnnotations() if annotations != nil { delete(annotations, "kubectl.kubernetes.io/last-applied-configuration") accessor.SetAnnotations(annotations) } return obj, nil } // Apply to all informers via transform function factory := informers.NewSharedInformerFactoryWithOptions( clientset, 0, informers.WithTransform(dropManagedFields), ) ``` For a cluster with 500 pods, this reduces memory from ~40MB to ~25MB. Worth it. ## Change Notifications Caching is half the story. We also need to push changes to [our sync system](/blog/push-sync-architecture). SharedInformers support event handlers: ```go func (rc *ResourceCache) makeHandler(kind string) cache.ResourceEventHandler { return cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { rc.emitChange(kind, obj, "add") }, UpdateFunc: func(oldObj, newObj interface{}) { rc.emitChange(kind, newObj, "update") }, DeleteFunc: func(obj interface{}) { rc.emitChange(kind, obj, "delete") }, } } func (rc *ResourceCache) emitChange(kind string, obj interface{}, op string) { change := ResourceChange{ Kind: kind, Operation: op, Object: obj, } select { case rc.changes <- change: default: // Channel full - drop event, log warning log.Warn("Resource change channel full, dropping event", zap.String("kind", kind)) } } ``` The channel is buffered (capacity 1000) and non-blocking. If the consumer is slow, we drop events rather than block the informer. Dropped events get recovered via [delta sync](/blog/push-sync-architecture). ## Graceful Fallback The cache might fail to initialize (permissions, network issues). We don't want that to break the application. Every lookup follows this pattern: ```go func GetPodsInNamespace(ns string) ([]v1.Pod, error) { cache := GetResourceCache() if cache != nil { pods, err := cache.Pods().Pods(ns).List(labels.Everything()) if err == nil { return pods, nil } log.Warn("Cache lookup failed, falling back to API", zap.Error(err)) } // Fallback to direct API call return clientset.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{}) } ``` This ensures: - Optimal performance when cache works (most of the time) - Reliability when cache fails (rare but possible) - No downtime or errors during startup or transient issues ## What We Don't Cache Not everything belongs in a SharedInformer cache. **Secrets**: We minimize sensitive data in memory. Even if it's convenient, caching secrets creates risk. We fetch them on-demand, and only if users opt-in to secrets read permission in the first place. **Custom Resources**: Handled separately with a dynamic informer that discovers CRDs at runtime. Different lifecycle than core resources. **High-Churn Resources**: Events are the exception - we cache them but filter aggressively. Without filtering, a busy cluster generates thousands of events per minute. ## Memory Characteristics For a typical small-to-mid cluster (500 pods, 50 services, 30 deployments), the typed informer cache looks like: ``` Raw cache size: ~11-14 MB With Go overhead: ~23-28 MB Goroutines: ~32 (2 per typed informer) Watch connections: 16 (multiplexed via HTTP/2) ``` Dynamic informers for CRDs (Argo Rollouts, cert-manager, Flux, etc.) add more - roughly 2 goroutines per discovered CRD type. A cluster with 20 CRDs adds ~40 more goroutines. Not free, but manageable. Startup cost: ``` 16 typed LIST calls: ~1-2 seconds (blocks until populated) CRD discovery: 5-10 seconds (runs in background) ``` The typed informer cache blocks on startup - you don't serve data until it's populated. CRD discovery runs in the background so the UI is usable immediately for core resources. On clusters with many CRDs (~100+), we had to fix a discovery waterfall that was taking minutes - parallelizing and caching the discovery results brought it down to seconds. ## Consistency Model SharedInformers provide **eventual consistency**, not strong consistency. Under normal operation, staleness is sub-second - events arrive almost instantly via watch. But during network partitions or API server restarts, the cache might lag. For our use case (UI dashboards), this is fine. Users see "eventually correct" data. For cases requiring strong consistency (admission controllers, operators making decisions), you'd want direct API calls with appropriate caching headers. ## Thread Safety All listers are thread-safe for reads. Multiple goroutines can query simultaneously without locking: ```go // Safe to call from any goroutine pods1, _ := cache.Pods().Pods("ns1").List(labels.Everything()) pods2, _ := cache.Pods().Pods("ns2").List(labels.Everything()) ``` The underlying store uses read-write locks internally. Writes (from informer events) don't block reads. ## Results After implementing SharedInformer caching (measured on a cluster with ~500 pods): | Metric | Before | After | |--------|--------|-------| | Pod list latency | 80ms | 1.5ms | | Service lookup | 60ms | 0.8ms | | Dashboard load (4 resource types) | 400ms+ | <10ms | | API server load | High | Minimal | On larger clusters, the "before" numbers are worse - LIST calls can take seconds - making the cache even more impactful. More importantly, it enabled our [push-based sync architecture](/blog/push-sync-architecture) - we can react to changes as they happen rather than polling. ## Key Takeaways 1. **SharedInformers are the standard pattern** for Kubernetes data access in Go. If you're hitting the API directly for every request, you're doing it wrong. 2. **Strip what you don't need.** managedFields and last-applied-configuration are often 30-50% of object size. 3. **Plan for failure.** Cache initialization can fail. Always have a fallback path to direct API calls. 4. **Eventual consistency is usually fine.** For dashboards and monitoring, sub-second staleness is acceptable. For operators making decisions, think harder. 5. **Don't cache secrets.** Minimize sensitive data in memory. The convenience isn't worth the risk. --- *This is part of our series on building real-time Kubernetes control planes. See also: [Zero-Latency Kubernetes: How We Built a Push-Based Sync Architecture](/blog/push-sync-architecture).* *Want to see this in action? [Try Skyhook free](https://www.skyhook.io) and connect a cluster - the SharedInformer cache powers every resource view. The same cache also powers [Radar](https://radarhq.io), our open-source Kubernetes explorer - try it locally in 30 seconds.* --- ### Reverse ML: Using AI to Write Rules, Not Run Them - **URL**: https://www.skyhook.io/blog/reverse-ml-using-ai-to-write-rules-not-run-them - **Date**: February 2026 - **Author**: Nadav Erell, CEO - **Category**: Platform Engineering - **Tags**: ai, machine-learning, developer-tools, platform-engineering, automation Everyone's racing to add LLMs to their runtime. We went a different direction. When we needed to build a system that detects programming languages, frameworks, and configurations from codebases, the obvious approach was to call an LLM at request time. Feed it the file tree, let it reason about what it sees, return a classification. GPT-5, Claude or Gemini would probably get it right 90% of the time. For this feature, we do use AI at runtime - but not as the starting point. Instead, we used AI extensively during *development* to generate deterministic rules that execute in milliseconds. These rules handle 90%+ of cases instantly. When they're uncertain, we optionally invoke AI - but with a head start: structured evidence and a preliminary classification that focuses the AI on specific ambiguities rather than cold-start exploration. We call this pattern **Reverse ML**: instead of training a model on data, you use AI agents to explore real-world examples and synthesize explicit rules. AI at development time to build the rules; deterministic execution by default; AI at runtime only when needed, with context that makes it faster and more accurate. ## Why Not Just Use AI at Runtime? Or Just Write Rules by Hand? There are three options for a classification problem like this: **Option 1: AI at runtime.** Let an agent explore the repo, reason about what it finds, return a result. Modern agents are good at this. But a thorough exploration takes 2-5 minutes - dozens of tool calls, reading files, reasoning about conflicts. For a CLI tool where detection is the first thing users experience, that latency kills the UX. You also lose determinism (same repo, different runs, different answers), auditability (good luck explaining a decision buried in 40 tool calls), and you pay $0.10-0.50 per repo at scale. (For Kubernetes-specific agent workloads, our [Radar MCP server](https://radarhq.io/product/mcp) is the runtime surface - agents query structured cluster state instead of scraping `kubectl` output.) **Option 2: Hand-written rules.** Engineers have built rule-based systems for decades. But the work doesn't scale. Labeling a testbed - thoroughly analyzing each repo to establish ground truth - takes 15-30 minutes per repo. Investigating detection failures takes 30-60 minutes each: open the repo, read manifests, check the Dockerfile, cross-reference framework docs, iterate. For 100+ repos across dozens of frameworks, you're looking at months. It's the kind of project that never gets prioritized because the ROI doesn't justify the effort. **Option 3: AI-assisted rules.** Use AI at *development time* to do the tedious work - labeling repos (2-5 minutes instead of 30), investigating failures (minutes instead of an hour), exploring edge cases in parallel. The output is the same readable, deterministic rules a human would write. AI just makes it economically feasible to build something this comprehensive. We chose option 3. Not because any single factor is decisive - if latency doesn't matter, if you're already building production agent infrastructure, runtime AI might be simpler. But for our constraints (instant CLI response, CI/CD determinism, auditability, and a small team that can't spend months on manual investigation), the tradeoffs made sense. The novel part isn't the rules themselves. **It's the AI-assisted development process that makes comprehensive rule-building practical at scale.** ## How It Works: A Concrete Example We built a codebase detection system that identifies languages, frameworks, ports, and configurations from project files. The goal: reduce manual onboarding inputs from 15+ fields to zero. Here's the development process, step by step. ### Phase 1: Build a Testbed (AI-Labeled) First, we curated 100+ real repositories covering: | Category | Examples | Purpose | |----------|----------|---------| | Popular frameworks | Express, FastAPI, Spring Boot | High-frequency patterns | | Emerging tools | Bun, Deno, Hono | Future-proofing | | Monorepos | Nx, Turborepo, Lerna | Complex structures | | Edge cases | No manifest, multiple Dockerfiles | Robustness | | Failure modes | Conflicting configs | Error handling | Here's where AI helps first: **establishing ground truth**. For each repo, an AI agent spends 2-5 minutes doing thorough analysis - reading files, reasoning about project structure - to determine correct classification. Language, framework, port, build commands. Manually labeling 100+ repos would take days. With AI running in parallel, we have ground truth in hours. We trust SOTA agents to get this right when given time to explore - same analysis a careful human would do, just faster. Is this ground truth perfect? No. But it's good enough to catch rule regressions, and we can always manually verify the cases that matter most. The alternative - manually labeling everything - doesn't scale. This testbed is our training set. Every rule change gets validated against all 100+ repos. ### Phase 2: AI-Assisted Rule Development When detection fails on a repo, we launch a coding agent (Claude Code) to explore: ``` Human: "Detection says this is Go, but it's actually Node.js. Explore ~/testbed/problem-repo and figure out why." Agent: 1. Lists directory structure 2. Reads package.json, go.mod, Dockerfile 3. Analyzes file contents and patterns 4. Identifies root cause 5. Proposes a detection rule ``` A typical exploration session: ``` > The repo has both go.mod and package.json. Let me check each... > go.mod exists but only contains tooling (golangci-lint, mockgen) > package.json has application deps (express, typescript) > Dockerfile uses node:18 base image > Root cause: We detected Go first because go.mod exists, but this > is actually a Node.js project that uses Go for build tooling. > Proposed rule: When both manifests exist: > 1. Check Dockerfile base image (highest signal) > 2. Check if go.mod deps are tooling-only > 3. Compare source file counts (.go vs .ts/.js) > 4. Adjust confidence based on evidence strength ``` ### Phase 3: Translate to Deterministic Code The AI proposes rules in plain language. We translate to code: **AI proposal:** > "If package.json has only devDependencies and no dependencies/main/bin fields, it's likely a script runner, not a Node.js application. Reduce confidence by 30%." **Implementation:** ```go func isScriptRunnerPackageJson(pkg *PackageJSON) bool { // No application entry points if pkg.Main != "" || pkg.Bin != nil || len(pkg.Dependencies) > 0 { return false } // Only has devDependencies (build tools, linters) return len(pkg.DevDependencies) > 0 } ``` This rule now runs in microseconds, with zero AI involvement. ### Phase 4: Validate and Iterate After adding a rule, we re-run the entire testbed: ```bash $ skyhook test-detection ~/testbed --html report.html Testbed Results ─────────────────────────────────────── Total repos: 127 Correct: 121 (95.3%) Wrong: 4 (3.1%) Low confidence: 2 (1.6%) Regressions from last run: 0 New fixes: 3 ``` If the new rule causes regressions, we refine it. The HTML report shows exactly which repos changed and why. The result: `skyhook init` now auto-detects language, framework, port, Dockerfile config, environment variables, and monorepo structure - everything users used to fill in manually - in under 50ms with no AI calls. ## The ML Parallel (and Where It Breaks Down) Why call this "Reverse ML"? The name is deliberately provocative. The parallel to traditional ML isn't perfect, but the similarities are illuminating: | Concept | Traditional ML | Reverse ML | |---------|---------------|------------| | **Training set** | Labeled examples | Testbed of real repos | | **Labeling** | Humans label data to train a model | AI labels data to validate rules | | **Training** | Gradient descent on loss function | AI exploration + human rule synthesis | | **Model** | Opaque weights | Explicit, readable rules | | **Feature engineering** | Transform raw data → features | Evidence collection → structured signals | | **Overfitting** | Model memorizes training data | Rules too specific to testbed repos | | **Inference** | Forward pass through network | Rule evaluation | The economic logic is identical: accept expensive training (development-time AI exploration) because inference (runtime rule evaluation) is cheap and fast. Evidence collection is literally feature engineering - transforming raw codebases into structured signals that make classification easier. And just like in ML, overfitting is a real concern - we mitigate it the same way: diverse test data, watching for suspiciously specific rules. Where the analogy breaks down: there's no loss function guiding us toward better rules - human judgment is the optimizer. And rules are discrete - they match or they don't, with no graceful degradation on edge cases. An ML model might get a weird input 60% right; our rules either handle it or punt to the confidence system. ## Architecture: Evidence Collection + Classification The detection system separates evidence collection from classification: first gather all signals (manifests, Dockerfile, source patterns), then apply heuristics with confidence scoring. This separation enables debuggability (see all signals, not just the winner), testability (evidence is deterministic, classification can be tuned), and transparency: ``` $ skyhook detect Detection Results ───────────────────────────────────────────────── Language: Go (85%) ← from go.mod Framework: Echo (80%) ← from github.com/labstack/echo import Port: 8080 ← from Dockerfile EXPOSE Alternatives Considered Language: Node.js (45%) ← package.json exists but tooling-only Decision Reasoning 1. Found go.mod → Go candidate (90%) 2. Found package.json → Node.js candidate (85%) 3. package.json has no deps, only devDeps → reduced to 45% 4. Dockerfile FROM golang:1.21 → confirms Go 5. Selected Go with 85% confidence ``` When confidence drops below 70% or conflicts exist, the system flags it for human review rather than guessing. ## Hybrid Mode: Best of Both Worlds Pure deterministic detection handles the common cases well. But some repos are genuinely ambiguous - conflicting signals, unusual structures, or patterns we haven't seen before. For these, we use a hybrid mode. The idea: run deterministic detection first (50ms), then optionally hand off to AI when confidence is low or the user opts in. ``` ┌─────────────────────────────────────────────────────────────────┐ │ HYBRID DETECTION │ │ │ │ Codebase ──▶ Evidence Collection ──▶ Deterministic Rules │ │ (50ms) (10ms) │ │ │ │ │ ▼ │ │ ┌──────────────────────┐ │ │ │ Confidence >= 80%? │ │ │ └──────────────────────┘ │ │ │ Yes │ No │ │ ▼ ▼ │ │ Return result AI Refinement │ │ (30-60 sec) │ │ │ │ │ ▼ │ │ Return refined │ │ result │ └─────────────────────────────────────────────────────────────────┘ ``` The key insight: **AI doesn't start from scratch**. It receives: - The structured evidence we already collected - The preliminary classification and confidence score - The specific conflicts or ambiguities detected - Access to the repo for targeted exploration This fast-tracks the AI's analysis. Instead of spending 3 minutes exploring the entire codebase, it spends 30-60 seconds investigating the specific ambiguities. The evidence collection acts like a pre-computed feature vector that focuses the AI's attention. For example, if deterministic detection says "Go (55%) vs Node.js (45%), conflict: both go.mod and package.json exist, Dockerfile ambiguous," the AI knows exactly what to investigate. It doesn't need to discover that conflict - it can immediately dig into which manifest represents the actual application. This hybrid approach gives users a choice: - **Fast mode** (default): Pure deterministic, 50ms, handles 90%+ of cases correctly - **Thorough mode** (opt-in): Deterministic + AI refinement for ambiguous cases, 30-60 seconds, higher accuracy on edge cases Neither mode is strictly better. Fast mode is right for CI pipelines and quick checks. Thorough mode is right for initial onboarding when you want high confidence. The system adapts to context rather than forcing a single tradeoff. ## Tradeoffs and Limitations This approach isn't free: **Upfront investment**: Building the testbed and iteration tooling takes time. You're trading runtime cost for development cost. AI dramatically reduces this cost - what might take months of manual investigation becomes weeks of AI-assisted exploration - but it's still nonzero. **Maintenance burden**: Rules are code. They need to be updated as the world changes. When Deno 2.0 changes its manifest format, someone has to update the rules. The flip side: updates are surgical. You change one rule, not retrain an entire model. **Coverage gaps**: Rules only cover patterns you've seen. A novel framework won't be detected until you add it to the testbed and write rules for it. The hybrid mode mitigates this - AI can handle novel patterns at runtime when confidence is low - but it's still a gap for pure deterministic mode. **Diminishing returns**: Going from 80% to 95% accuracy is 10x harder than 50% to 80%. Some edge cases aren't worth the rule complexity. Again, hybrid mode helps: let rules handle the 95% and let AI handle the long tail. For our use case - onboarding codebases to a deployment platform - these tradeoffs work. Millisecond detection with full transparency handles the common cases. AI refinement handles the edge cases when users opt in. Your mileage may vary. If your patterns change daily or your input space is truly unbounded, runtime AI might be the right choice despite its costs. But if you're solving a classification problem with knowable patterns and you care about speed, cost, or explainability, this approach is worth considering. ## Conclusion We call this Reverse ML because it inverts the traditional pattern: expensive AI exploration at development time produces cheap deterministic execution at runtime - with the option to bring AI back when the stakes justify it. The approach works for us because of a combination of factors: - We can leverage best-in-class tooling (Claude Code) at development time, saving our agent investment for where we add unique value - We want instant CLI response times, not multi-minute waits - We value transparency and auditability over probabilistic outputs - Upfront development effort pays off as free runtime execution None of these factors alone is decisive. If latency doesn't matter, if you're already building production agent infrastructure, if determinism isn't important - runtime AI might be simpler. We're not claiming this approach is universally better. But for bounded classification problems where you can enumerate the patterns, where examples exist, and where some combination of speed, cost, capability constraints, or transparency matters - this is a viable third option between "just use AI at runtime" and "write rules by hand." The novel part isn't the deterministic rules. Engineers have built rule-based systems for decades. The novel part is using AI to make comprehensive rule development economically practical - and giving that same AI a head start at runtime when the rules aren't enough. The next time you reach for an LLM at runtime, ask: could I use AI to generate rules instead? And if some cases still need AI, can I at least give it a head start? --- ### Zero-Latency Kubernetes: How We Built a Push-Based Sync Architecture - **URL**: https://www.skyhook.io/blog/push-sync-architecture - **Date**: February 2026 - **Author**: Eyal Dulberg, CTO - **Category**: Architecture - **Tags**: kubernetes, distributed-systems, architecture, performance, deep-dive When you're building a platform that shows Kubernetes cluster state, freshness and latency matter. Users expect the dashboard to reflect reality, and they expect to be able to click around quickly. A pod that crashed 25 seconds ago should show as crashed now, not on the next cache refresh. Our original architecture couldn't deliver on either front. Data was up to 25 seconds stale, and even when the cache was warm, it was often enough invalid for at least one resource type - meaning the dashboard had to fetch it from the cluster on demand, adding seconds of latency to what should feel instant. This post covers the push-based sync protocol we built to solve this. It's the companion to [our SharedInformer caching post](/blog/sharedinformer-caching), which covers the agent-side in-memory cache that feeds this system. You can read them in either order, but together they explain the full pipeline from Kubernetes API server to dashboard. ## The Problem: Pull-Based Polling Our first architecture was straightforward: ``` Frontend → Backend (cache check, 25s TTL) → Task Queue → Agent polls (1s) → Agent fetches k8s → Returns ``` When a user opened a cluster view, the backend checked its cache. Cache miss? Queue a task. The agent polled every second, picked it up, fetched data from Kubernetes, and returned it. Not slow per se - but the multi-hop round-trip plus cache TTL meant data could be up to 25 seconds stale. And that's the optimistic case. On larger clusters, individual Kubernetes API calls can take a few seconds rather than milliseconds. When a user navigates through Skyhook - clicking into a cluster, viewing different resource types - each interaction can trigger multiple fetch tasks. Even with the agent's built-in parallelism, a burst of concurrent requests could push end-to-end response times into several seconds territory. Combine that with cache staleness and the experience felt sluggish. **The issues:** - **Staleness**: Cache TTL meant users saw data up to 25 seconds old even though the agent polled frequently - **Slow under load**: On large clusters, Kubernetes API calls are slower, and bursts of user activity could saturate the agent's fetch capacity - **Reactive, not proactive**: The agent only fetched data when asked - it had no awareness of changes happening in the cluster - **Redundant work**: Agents polling every second even when nothing changed, backend re-fetching resources that hadn't been modified ## The Solution: Push-Based Sync The fix seems obvious: have the agent push changes as they happen. But the implementation has subtleties worth walking through. ### High-Level Flow ``` ┌─────────────────────────────────────────────────┐ │ Kubernetes API Server │ └────────┬────────────────────────────────────────┘ │ Watch (SharedInformers) ▼ ┌─────────────────────────────────────────────────┐ │ Agent (skyhook-connector) │ │ ResourceCache → Aggregator → Sync Worker │ └────────┬────────────────────────────────────────┘ │ POST /sync (deltas every 3s) ▼ ┌─────────────────────────────────────────────────┐ │ Backend (cluster-agent-controller) │ │ SyncStore → API Endpoints │ └─────────────────────────────────────────────────┘ ``` The agent watches Kubernetes resources using [SharedInformers](/blog/sharedinformer-caching), aggregates changes, and pushes deltas to the backend every 3 seconds. The backend stores this state and serves it directly - no round-trip to the cluster needed. Result: **sub-3-second staleness** instead of 25. ## What We Got Right (and What We Had to Iterate On) We anticipated a lot of the challenges upfront. Activity-based activation, delta sync, batching/aggregation, overflow protection, staleness fallback - these were all in the initial design. We knew that syncing all clusters all the time would be wasteful, so sync was demand-driven from day one. But some things only become obvious when you hit real clusters at scale. ### Activity-Based Activation (Day One) Sync is OFF by default. It activates when someone accesses the cluster. ```go func (s *SyncStore) RecordActivity(clusterID string) { state.LastActivityTime = time.Now() if !state.SyncEnabled && state.Mode != "manual_off" { state.SyncEnabled = true // Tell the agent to start syncing go s.onSyncControlChange(clusterID, true) } } ``` Every API endpoint that touches cluster data calls `RecordActivity()`. First access triggers sync. The agent receives a `sync_control` task, enables its sync worker, and immediately pushes a full snapshot. ### Idle Detection What about clusters that were active but aren't anymore? We run an idle checker every minute: ```go const idleTimeout = 30 * time.Minute func (c *IdleChecker) checkIdleClusters() { for _, cluster := range c.store.GetAllClusters() { if time.Since(cluster.LastActivityTime) > idleTimeout { c.store.DisableSync(cluster.ID) } } } ``` 30 minutes of no activity? Sync disabled. The agent stops pushing. Zero overhead until someone looks again. ## Delta Sync vs Full Sync Pushing the entire cluster state every 3 seconds would be wasteful. Instead, we use delta sync for normal operation and full sync for recovery. ### Delta Sync (Normal Operation) The agent tracks changes since the last push: ```json { "SyncType": "delta", "SequenceNumber": 42, "Deltas": [ { "Kind": "Pod", "Namespace": "default", "Name": "api-7d4b8c9-xk2pm", "Operation": "update", "Object": { "status": { "phase": "Running" } } } ] } ``` Each delta batch has a sequence number. The backend validates that `SequenceNumber == LastSequence + 1`. If there's a gap, something was lost in transit. ### Full Sync (Recovery) Full sync sends everything: ```json { "SyncType": "full", "SequenceNumber": 1, "Snapshots": { "Pod": [{ }, { }, { }], "Service": [{ }], "Deployment": [{ }] } } ``` Full sync triggers on: - Sync enabled (OFF → ON transition) - Sequence gap detected - Explicit request from backend - Manual resync via dashboard After full sync, sequence resets to 1 and delta sync resumes. ## Aggregation and Coalescing Kubernetes clusters are chatty. A deployment rollout can generate hundreds of events per second. Pushing every event individually would overwhelm the network. The aggregator batches and deduplicates: ```go func (a *Aggregator) coalesce(existing, new *ResourceDelta) *ResourceDelta { // add + delete = nil (resource appeared and disappeared) if existing.Operation == "add" && new.Operation == "delete" { return nil } // delete + add = update (resource was replaced) if existing.Operation == "delete" && new.Operation == "add" { return &ResourceDelta{Operation: "update", Object: new.Object} } // Otherwise: latest state wins return new } ``` If a pod is created and deleted within the same 3-second window, we send nothing. If it's updated 50 times, we send one update with the final state. ### Overflow Protection What if changes come faster than we can process? We cap the pending queue: ```go const ( maxEventDeltasPerBatch = 500 maxPendingEvents = 2000 ) ``` If pending events exceed 2000, oldest events are dropped. This prevents memory exhaustion from pathological cases (CrashLoopBackOff generating infinite events). A full resync will eventually restore consistency. ## Staleness and Fallback Push-based sync is great when it's working. But what if the agent disconnects? We can't serve stale data forever. We track staleness in three states: | State | Condition | Behavior | |-------|-----------|----------| | Fresh | Last sync < 60s | Serve from SyncStore | | Stale | 60s - 5min | Fall back to task-based fetch | | Disconnected | > 5min | Fall back to task-based fetch | The API lookup order: ```go func (s *Service) GetPods(clusterID string) ([]Pod, error) { if !s.syncStore.IsStale(clusterID) { // Fresh data available - sub-millisecond response return s.syncStore.GetPods(clusterID) } // Stale or disconnected - fall back to pull-based return s.fetchViaTasks(clusterID) } ``` This maintains backward compatibility. If push sync fails, the system degrades gracefully to the old pull-based flow. ## Resource Type Filtering Not every use case needs all 17 resource types. The backend can specify which types to sync: ```json { "type": "sync_control", "args": { "enabled": "true", "resourceTypes": "Pod,Service,Deployment" } } ``` This reduces bandwidth for focused use cases and saves memory in the agent. ## Where We Had to Iterate The core sync protocol worked well from the start. The surprises came from payload size and memory. ### Payload Size: Compression and Stripping Our first sync payloads were huge. A large cluster's pod list could exceed 2MB, and we were hitting 413 Request Entity Too Large errors on the backend. The first fix was gzip compression on all sync payloads - an 85-90% size reduction. A 2MB pod payload compressed to ~240KB. But compression doesn't help memory on the agent side. Kubernetes objects carry a lot of data you don't need for a dashboard - `managedFields`, `last-applied-configuration`, CRD validation schemas. We added [transform functions to strip these before they enter the cache](/blog/sharedinformer-caching#memory-optimization-stripping-metadata), reducing object sizes by 30-50%. These weren't all in the initial release. We added them incrementally as we tested against larger clusters and profiled what was eating memory. ### Memory: The Go Marshalling Problem On small clusters, everything was fine. On large clusters (1000+ pods, dozens of CRDs), memory usage spiked well beyond what the raw data size would suggest. The culprit: Go's JSON marshalling. Every sync cycle, we serialize the delta batch to JSON for the HTTP payload. `json.Marshal` allocates intermediate buffers - it builds the output incrementally, concatenating fragments, each step allocating. For large objects, this creates significant GC pressure. The garbage collector can't reclaim memory fast enough, and RSS climbs. We had to profile this carefully. `pprof` heap profiles showed allocations dominated by JSON serialization, not by the cache itself. The fixes were incremental: - **`GOMEMLIMIT`**: We auto-set Go's memory limit to 75% of the container's allocation (768MB of the default 1GB). This makes the GC work harder to stay within bounds, significantly reducing RSS without affecting throughput. - **Per-resource-type cache breakdown**: We built observability tooling to measure JSON-serialized size per resource type, identify which types were dominating memory, and surface the largest individual objects (like oversized ConfigMaps). - **Selective caching**: Not every resource type needs to be cached. We refined which resources to sync by default and gave the backend control over which types to request. This is an area we're still optimizing. Go's standard `encoding/json` is known for allocation-heavy marshalling, and the ecosystem is moving toward streaming alternatives. For now, the combination of field stripping, memory limits, and selective caching keeps things manageable. ### Resync Storm Protection The overflow protection and resync logic worked, but needed tuning in practice. During testing we hit a failure mode: resync loops. 1. Agent sends delta with sequence 43 2. Backend expected 42 (missed one) 3. Backend requests full resync 4. Agent sends full sync 5. Something corrupts - backend requests resync again 6. Infinite loop The fix was rate limiting: ```go const ( maxResyncPerMinute = 5 resyncBackoffDuration = 30 * time.Second ) ``` Five resyncs within a minute triggers a 30-second backoff. This gives transient issues time to resolve and prevents runaway resource consumption. ## What We Learned ### 1. Anticipate the Obvious, Profile the Rest Activity-based activation, delta sync, and graceful fallback were clear requirements from the start. But memory behavior on large clusters only became visible through profiling real workloads. Design for what you know, then measure. ### 2. Sequence Numbers Are Essential Without sequence validation, we'd silently lose events and serve inconsistent data. The gap detection and resync flow ensures eventual consistency even under network partitions. ### 3. Graceful Degradation Matters Push sync is an optimization, not a requirement. When it fails, we fall back to pull. Users see slower responses but never errors. This let us ship incrementally rather than doing a risky big-bang migration. ## Results After deploying push-based sync: - **Freshness**: Up to 25s stale → sub-3 seconds (when sync active) - **Dashboard latency**: ~80% reduction. Before, even one cache-miss resource type meant a round-trip to the cluster, adding seconds. Now every resource is served from the SyncStore in milliseconds. - **API server load**: Reduced (fewer direct fetches) - **Network efficiency**: Only active clusters sync The architecture is more complex than simple polling. But for a platform where users click through clusters, namespaces, and resource types expecting instant responses, the difference is night and day. --- *This is part of our series on building real-time Kubernetes control planes. See also: [50-100x Faster: Building an In-Memory Kubernetes Resource Cache with SharedInformers](/blog/sharedinformer-caching).* *Want to see this in action? [Try Skyhook free](https://www.skyhook.io) and connect a cluster - you'll see the push sync kick in immediately. This architecture also powers [Radar Cloud's real-time multi-cluster view](https://radarhq.io/product/timeline).* --- ### Skyhook Agent: AI That Actually Understands Your Infrastructure - **URL**: https://www.skyhook.io/blog/introducing-skyhook-agent - **Date**: December 2025 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: ai, automation, product, kubernetes, developer-experience Most AI assistants for DevOps are glorified documentation search. You ask a question, they regurgitate docs. Useful, but limited. Skyhook Agent is different. It has real-time access to your actual infrastructure - every service, configuration, dependency, and log. It doesn't just answer questions. It takes action, opening Pull Requests that you approve. ## What Makes Skyhook Agent Different The Agent sees your entire environment: - **Kubernetes configurations and deployments** - what's running, how it's configured - **Service dependencies** - which services talk to which - **Environment variables and secrets references** - what's configured where - **Logs and metrics** - what's happening right now - **Your GitOps workflows** - how changes flow through your system - **Live cluster topology** - powered by the same engine as [Radar](https://radarhq.io/product/topology), our open-source Kubernetes visibility tool This context awareness means the Agent understands your specific environment. When you ask "why is checkout slow?", it doesn't give generic advice - it checks your actual services, traces dependencies, and identifies the bottleneck. ## Key Capabilities ### Intelligent Troubleshooting Kubernetes troubleshooting usually means hours of `kubectl` commands and log analysis. The Agent compresses this: - Analyzes logs across multiple services simultaneously - Correlates errors with recent deployments or config changes - Identifies root causes, not just symptoms - Suggests and implements fixes with your approval **Example:** A pod crashes with OOM. The Agent traces the memory spike to a recent code change, identifies the service, and opens a PR to adjust resource limits. Seconds, not hours. ### Configuration Management The Agent monitors configurations against best practices and your established patterns: - Detects drift before it causes issues - Identifies security vulnerabilities in deployments - Suggests optimizations based on actual resource usage - Ensures consistency across dev, staging, and production Each recommendation comes as a ready-to-merge PR. You stay in control; the tedious work disappears. ### Natural Language Queries Skip the `kubectl` gymnastics: > "Which services depend on Redis in production?" > "Show me services with CPU limits below 500m that had OOM events last week." > "What changed in the payment service since Monday?" Instant answers from your actual infrastructure. ### Security Automation Continuous security review: - Flags exposed secrets or insecure configurations - Enforces your organization's security policies - Recommends updates for outdated dependencies - Generates audit trails for compliance ## Where to Use It The Agent works wherever you do: - **Skyhook Portal**: Integrated into each service's detail view - **Skyhook CLI**: `skyhook agent` starts a conversation - **Slack**: Chat directly in your team channels Conversations are threaded. Context persists within sessions. Multi-step troubleshooting flows naturally. ## Real Scenarios ### Production Incident at 3 AM Your monitoring alerts fire. A critical service returns 500s. The Skyhook Agent: 1. Identifies errors started 12 minutes after a deployment 2. Pinpoints a misconfigured environment variable 3. Opens a PR to roll back the change 4. Posts a summary to your incident channel Resolution: 4 minutes instead of 45. No one got paged. ### Resource Right-Sizing Your cloud bill is growing. You're not sure what's over-provisioned. Ask: *"Which services allocated more than 2x their actual usage over the past 30 days?"* The Agent returns a prioritized list with specific recommendations and opens PRs to optimize the top offenders. ### Developer Onboarding A new team member needs to understand dependencies before making changes. They ask: *"Explain how the payment service interacts with other services and what happens if it goes down."* The Agent provides dependency graphs and failure impact analysis. Context that would take days to gather manually, delivered in seconds. ## The Point Skyhook Agent isn't about replacing DevOps engineers. It's about removing the tedious parts of the job. Routine tasks get automated. Troubleshooting gets faster. Developers gain autonomy without sacrificing safety. The on-call engineer sleeps through incidents that resolve themselves. [Get started with Skyhook](/demo) and see what AI-assisted infrastructure actually looks like. Prefer to explore your cluster without AI? Try [Radar's MCP server for Kubernetes](https://radarhq.io/product/mcp) - local-first, no account required. --- ### Laravel to Kubernetes, Simplified with Skyhook - **URL**: https://www.skyhook.io/blog/laravel-to-kubernetes-simplified-with-skyhook - **Date**: November 2025 - **Author**: Eyal Dulberg, CTO - **Category**: Kubernetes - **Tags**: tutorial, laravel, kubernetes, migration, docker, for-developers Laravel is a PHP heavyweight, but running it in production usually means stitching together many moving parts: runtime, scaling, security, TLS, observability, and secrets. Kubernetes solves the runtime puzzle but introduces a new learning curve: container images, manifests, deployments, scaling, service discovery, and secrets management. Below are two ways to ship Laravel to Kubernetes: - The manual route: containerize the app, write YAML, own every knob. - The Skyhook route: let Skyhook automate Day 1 bootstrap and handle Day 2 operations for you. ## What you will build - Web app: Nginx with PHP-FPM serving Laravel - Queue workers: Horizon deployment - Scheduler: CronJob running php artisan schedule:run every minute - Database migrations: pre-deploy Job - Ingress with TLS via cert-manager - Autoscaling via HPA, logs to stdout, metrics to Prometheus ## Prerequisites | Requirement | Manual path | Skyhook path | | --- | --- | --- | | Kubernetes cluster | You create or have one | Create or connect from Skyhook | | Docker | Needed to build the image | Needed, Skyhook can assist | | Container registry | Needed | Built in or external (ECR, GCR, ACR, GHCR) | | Git repository | Needed | Needed | ## Option 1: Manual deployment ### 1) Containerize Laravel (PHP-FPM behind Nginx) PHP-FPM is not an HTTP server. Expose Nginx on port 80 and keep PHP-FPM internal. **Dockerfile (multi-stage)** ```dockerfile FROM composer:2 AS vendor WORKDIR /app COPY composer.json composer.lock ./ RUN composer install --no-dev --no-interaction --prefer-dist --no-scripts FROM node:22-alpine AS assets WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci COPY resources resources RUN npm run build FROM php:8.3-fpm-alpine AS app RUN apk add --no-cache nginx curl libzip-dev icu-dev \ && docker-php-ext-install pdo_mysql zip intl opcache WORKDIR /var/www/html # App files COPY . . COPY --from=vendor /app/vendor ./vendor COPY --from=assets /app/public/build ./public/build # Cache config and routes for prod RUN php artisan config:cache && php artisan route:cache # Nginx COPY ./infra/nginx.conf /etc/nginx/nginx.conf EXPOSE 80 CMD ["sh", "-c", "php-fpm -D && nginx -g 'daemon off;'"] ``` **Minimal Nginx config** ```nginx events {} http { server { listen 80; root /var/www/html/public; index index.php index.html; location /health { return 200 "ok"; } location / { try_files $uri $uri/ /index.php?$query_string; } location ~ \.php$ { fastcgi_pass 127.0.0.1:9000; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; } } } ``` Build and push: ```bash docker build -t registry.example.com/laravel-demo:1.0 . docker push registry.example.com/laravel-demo:1.0 ``` ### 2) Write Kubernetes manifests (abbreviated) **Web Deployment and Service** ```yaml apiVersion: apps/v1 kind: Deployment metadata: { name: laravel-web, labels: { app: laravel-web } } spec: replicas: 2 selector: { matchLabels: { app: laravel-web } } template: metadata: { labels: { app: laravel-web } } spec: containers: - name: web image: registry.example.com/laravel-demo:1.0 ports: [{ containerPort: 80 }] envFrom: [{ secretRef: { name: laravel-env } }] readinessProbe: { httpGet: { path: /health, port: 80 }, periodSeconds: 5 } livenessProbe: { httpGet: { path: /health, port: 80 }, initialDelaySeconds: 30 } resources: requests: { cpu: "200m", memory: "256Mi" } limits: { cpu: "1", memory: "512Mi" } securityContext: { runAsNonRoot: true } --- apiVersion: v1 kind: Service metadata: { name: laravel-web } spec: selector: { app: laravel-web } ports: [{ port: 80, targetPort: 80 }] ``` **Ingress with TLS via cert-manager** ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: laravel annotations: cert-manager.io/cluster-issuer: letsencrypt spec: tls: - hosts: [app.example.com] secretName: laravel-tls rules: - host: app.example.com http: paths: - path: / pathType: Prefix backend: { service: { name: laravel-web, port: { number: 80 } } } ``` **Migrations as a pre-deploy Job (works well with Argo CD hooks)** ```yaml apiVersion: batch/v1 kind: Job metadata: name: laravel-migrate annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-delete-policy: BeforeHookCreation spec: template: spec: restartPolicy: Never containers: - name: migrate image: registry.example.com/laravel-demo:1.0 command: ["php", "artisan", "migrate", "--force"] envFrom: [{ secretRef: { name: laravel-env } }] ``` **Queue workers (Horizon)** ```yaml apiVersion: apps/v1 kind: Deployment metadata: { name: laravel-horizon, labels: { app: laravel-horizon } } spec: replicas: 2 selector: { matchLabels: { app: laravel-horizon } } template: metadata: { labels: { app: laravel-horizon } } spec: containers: - name: horizon image: registry.example.com/laravel-demo:1.0 command: ["php", "artisan", "horizon"] envFrom: [{ secretRef: { name: laravel-env } }] resources: requests: { cpu: "200m", memory: "256Mi" } limits: { cpu: "1", memory: "512Mi" } ``` **Scheduler** ```yaml apiVersion: batch/v1 kind: CronJob metadata: { name: laravel-schedule } spec: schedule: "* * * * *" jobTemplate: spec: template: spec: restartPolicy: Never containers: - name: schedule image: registry.example.com/laravel-demo:1.0 command: ["php", "artisan", "schedule:run"] envFrom: [{ secretRef: { name: laravel-env } }] ``` **External Secrets example** ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: { name: laravel-env } spec: refreshInterval: 1h secretStoreRef: { name: gcp-sm, kind: ClusterSecretStore } target: { name: laravel-env } data: - secretKey: APP_KEY remoteRef: { key: projects/123/secrets/laravel-app-key } - secretKey: DB_HOST remoteRef: { key: projects/123/secrets/laravel-db-host } - secretKey: REDIS_HOST remoteRef: { key: projects/123/secrets/laravel-redis-host } ``` **HPA for web** ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: { name: laravel-web } spec: scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: laravel-web } minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } } ``` **Key Laravel env tips** ```env APP_ENV=production APP_DEBUG=false APP_KEY=base64:... # generate once and store in your cloud secrets manager SESSION_DRIVER=redis CACHE_DRIVER=redis QUEUE_CONNECTION=redis LOG_CHANNEL=stderr ``` Apply your YAML: ```bash kubectl apply -f k8s/ ``` You will still need to install and wire cert-manager, External Secrets, Prometheus, Grafana, and logging, and you will own upgrades, rollouts, and drift. ## Option 2: Ship Laravel with Skyhook (recommended) Skyhook supports a hybrid workflow. You keep your repos and clusters. Skyhook automates Day 1 bootstrap and Day 2 operations. ### Day 1: Bootstrap with Terraform (one click) - Click Create Cluster in the Skyhook console and choose cloud and region. - Skyhook commits a ready made Terraform module to your infra repo and triggers CI. - Result: VPC, managed Kubernetes, node pools, OIDC, IAM roles, Argo CD, cert-manager, External Secrets, Grafana, Prometheus, and Loki. You can also bring your own Terraform and clusters. ### Day 2: Deploy Laravel from the Skyhook UI - Create a Service by pointing Skyhook to your Git repo. - Build and push using your Dockerfile or Buildpacks. Skyhook handles registry credentials. - Environment variables and secrets mapped through your cloud secrets manager via External Secrets. - Add ons with a click such as cert-manager for automatic TLS, External Secrets for rotation, Redis or RabbitMQ charts for queues. - Deploy which generates an Github Actions flow or an Argo CD Application, commits it to your apps repo, and rolls it out. - Observe and scale a full open source observability setup with dashboards and autoscaling policies. You don’t have to deal with YAML and kubectl apply, but they’ll always be available for you if you want to dig in yourself. ## Skyhook value recap | Benefit | What it means for Laravel teams | | --- | --- | | Best practices out of the box | Production networking, probes, HPA, log aggregation, and TLS configured for you | | No lock in | Manifests live in your Git repos and run on any CNCF conformant Kubernetes | | Plug and play setup | Cluster up and add ons installed in minutes, no scratch Helm or HCL | | Empower developers | Self service deploy, debug, and rollback without waiting on DevOps | | Designed to scale | Multi cluster support, RBAC, and policy controls that grow with you | ## Conclusion You can deploy Laravel on Kubernetes by hand, but teams usually rebuild the same pieces: web, workers, scheduler, migrations, TLS, secrets, and safe rollouts. Skyhook gives you those pieces out of the box, commits them to your Git repos, and lets you operate them from a single UI. Spin up a free Skyhook workspace, point it at your repo, and ship web, workers, scheduler, and migrations behind TLS in minutes. --- ### Ship Without Fire Drills: Canary, Blue-Green, and Rolling Deploys - **URL**: https://www.skyhook.io/blog/ship-without-fire-drills-canary-blue-green-and-rolling-deploys - **Date**: October 2025 - **Author**: Eyal Dulberg, CTO - **Category**: DevOps - **Tags**: guide, best-practices, canary-deployment, blue-green, argocd TL;DR: Shipping to production is easy; shipping safely is hard. Progressive delivery strategies - Rolling (Gradual), Canary, and Blue-Green - constrain blast radius, create decision points, and give you measurable guardrails. This post explains why they matter, how each works, pros/cons, failure modes, and the key knobs you can tune. We also include concrete YAML snippets you can lift into your pipelines. At the end, we show how Skyhook.io integrates Argo Rollouts so teams can do this with a click. ## Why Progressive Delivery Matters When a release goes wrong, you don't want a binary "all or nothing" deployment. You want narrow exposure, fast feedback, and emergency brakes. Case in point: the CrowdStrike Falcon content update (July 2024). This incident shipped a flawed sensor configuration that triggered Windows BSODs (Blue Screen of Death) at a global scale. A single binary push affected millions of systems in minutes. The actionable lesson for application teams is to deploy progressively with circuit breakers: small initial exposure, metric-gated promotion, and instant rollback. Had CrowdStrike used canary deployment with a 5% traffic gate, the blast radius would have been contained to thousands of systems, and rollback would have been automatic within minutes. This kind of failure is increasingly common because modern systems have high dependency density: kernel/OS, base images, libraries, feature flags, infra add-ons, and your app. Any one of those can fail in a way that is: * Correctness-breaking (logic/compatibility/API mismatches) * Performance-degrading (p99 latency regressions, garbage collection (GC) or heap thrash, I/O contention) * Safety-critical (authorization bypass, data loss, cascading retries) Progressive delivery strategies reduce mean time to detect (MTTD) from hours to minutes and limit blast radius by introducing staged exposure and automated rollback based on real traffic signals. Studies show this approach reduces incident severity by roughly 70% and cuts incident response time from hours to minutes. ## Strategy 1: Rolling (Gradual) Updates Idea: Replace pods incrementally while keeping the service endpoint stable. New pods come up; old pods drain; traffic shifts implicitly as endpoints register or deregister. **How it works (Kubernetes Deployment):** * maxSurge: how many extra pods you can add above desired during rollout * maxUnavailable: how many of the old pods can be down during the update * Readiness/liveness/startup probes gate traffic **When to use:** * Backward-compatible changes where schema, config, and protocol compatibility is guaranteed * You don't need explicit traffic splitting; implicit load balancing is fine * Frequent, low-risk releases (e.g., configuration tuning, documentation updates) **Pros:** * Simple, native to Kubernetes; no extra controllers or custom resource definitions * Good for low-risk, frequent changes * Minimal cluster overhead **Cons:** * No explicit step-wise traffic weights, so harder to pause, measure, and abort at controlled percentages * Rollback is another rollout; may not be instant if capacity is tight. If maxUnavailable=0, rollback takes just as long as the forward rollout. * No built-in metric gates; relies entirely on liveness/readiness probes **Key knobs:** * maxSurge, maxUnavailable * Probe thresholds (failureThreshold, periodSeconds) * minReadySeconds (grace period before pod is considered ready) * Pod Disruption Budgets (PDB) to guard availability **Example (native Kubernetes Rolling):** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: api spec: replicas: 12 strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 0 template: spec: containers: - name: api image: ghcr.io/org/api:v1.9.0 readinessProbe: httpGet: { path: /healthz, port: 8080 } periodSeconds: 5 failureThreshold: 3 livenessProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 10 failureThreshold: 2 ``` ## Strategy 2: Canary Releases Idea: Shift a small, explicit percentage of production traffic to the new version, analyze service-level objectives (SLOs) and metrics, then progressively increase if healthy (or auto-rollback if not). **How it works (Argo Rollouts):** * Controller replaces Deployment with Rollout and orchestrates traffic steps * Integrates with service meshes or ingress controllers (Nginx, AWS Application Load Balancer (ALB), Istio, Linkerd, etc.) to apply precise traffic weights at the ingress layer * Analysis templates execute asynchronously during pauses, querying Prometheus, Datadog, Kayenta, or webhooks * If any metric breaches its failure threshold, the rollout aborts automatically and rolls back Important: Canary requires traffic splitting at the ingress or mesh layer. It will not work with bare Kubernetes Service load balancing, which always distributes traffic evenly across all pods. **When to use:** * Medium or high-risk changes: library upgrades, infra changes, or anything touching critical paths * New feature rollouts where you want to measure real user impact before going wide * Changes with low backward compatibility confidence **Pros:** * Precise traffic control (e.g., 2% -> 10% -> 25% -> 50% -> 100%) * Built-in pauses, automated analysis, and instant rollback on metric breach * Reduces blast radius to a measurable percentage from the start * Provides early signal on performance and correctness before wide rollout **Cons:** * Requires mesh or ingress controller integration and metric observability hygiene * Slightly more control-plane complexity than Rolling * Demands well-tuned SLO thresholds and metric lookback windows **Key knobs:** * Step weights and pause durations * Max surge/unavailable for capacity safety * Analysis templates: success/failure thresholds, metric lookback windows, failure limits * Abort conditions and rollback strategies **Example (Argo Rollouts Canary with analysis):** ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: checkout spec: replicas: 10 strategy: canary: canaryService: checkout-canary stableService: checkout-stable trafficRouting: nginx: {} steps: - setWeight: 5 - pause: { duration: 120 } - analysis: templates: - templateName: slo-99-latency args: - name: service value: checkout - setWeight: 25 - pause: { duration: 180 } - analysis: templates: - templateName: slo-99-latency args: - name: service value: checkout - setWeight: 50 - pause: { duration: 300 } maxSurge: 2 maxUnavailable: 0 template: spec: containers: - name: app image: ghcr.io/org/checkout:v2.3.1 ``` **Example (AnalysisTemplate with 99th percentile latency and error rate):** ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: slo-99-latency spec: metrics: - name: p99-latency interval: 30s successCondition: result < 400 failureLimit: 1 provider: prometheus: address: http://prometheus.monitoring:9090 query: | histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~"{{args.service}}"}[2m])) by (le)) * 1000 - name: error-rate interval: 30s successCondition: result < 0.01 failureLimit: 1 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{service=~"{{args.service}}",code=~"5.."}[2m])) / sum(rate(http_requests_total{service=~"{{args.service}}"}[2m])) - name: user-checkout-success interval: 60s successCondition: result > 0.99 failureLimit: 1 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(checkout_success_total{service=~"{{args.service}}"}[3m])) / sum(rate(checkout_attempts_total{service=~"{{args.service}}"}[3m])) ``` Analysis templates run asynchronously during each pause step. If a single metric fails (failureLimit: 1), the entire rollout aborts and triggers automatic rollback rather than waiting for all metrics to pass. This ensures fast feedback and prevents cascading user impact. Note: A canary might show healthy latency metrics but still fail in production due to missing business logic checks. Always include user-journey SLOs (e.g., login success rate, checkout completion rate) alongside infrastructure metrics. Health illusions are one of the most common silent failures in rollouts. ## Strategy 3: Blue-Green (also called Red-Black) Idea: Run two full environments (Blue \= live, Green \= new). Validate Green behind the scenes; when ready, flip the router (or swap service labels) instantly. If it breaks, flip back. **How it works:** * Traffic is directed 100% to the Blue (stable) environment. * The Green (new) environment is deployed alongside it, receiving no live traffic. * Internal smoke tests, synthetic probes, and validation checks run against the Green environment's preview endpoint. * When Green is confirmed healthy, the production router is "flipped" to send 100% of traffic to Green in a single atomic operation. * Blue is kept warm for a short time to allow for an instant rollback if issues are detected in the first few minutes. **When to use:** * Protocol or compatibility changes that demand clean switchover (e.g., gRPC -> HTTP/2, database schema migrations) * Releases requiring database migrations executed separately using the expand-migrate-contract pattern * Changes with zero tolerance for partial state inconsistency * When you need smoke tests or backfills to run before any user traffic arrives **Pros:** * Instant cutover and instant rollback (flip happens in milliseconds) * Clean separation that enables exhaustive pre-flight validation * No gradual traffic shift; users experience either Blue or Green, never a mixed state * Excellent for coordinated infrastructure changes **Cons:** * Double capacity required during release (both environments run simultaneously) * Requires careful state management: shared databases, caches, and session stores must be compatible with dual-write scenarios or stateless architectures * If something is silently broken in Green, you discover it instantly at cutover (canary would catch it earlier) **Key knobs:** * Cutover policy (manual vs. auto with delay) * Validation gates before promotion (e.g., smoke test pass rate, synthetic checks) * Scale-down delay of old stack (how long to keep Blue warm before terminating it) Important: Blue-Green deployments require careful handling of stateful systems. If your app writes to a shared database, ensure both Blue and Green can handle concurrent writes during switchover, or use feature flags to gate writes to the new version until you've completed the switchover and verified Green is stable. **Example (Argo Rollouts Blue-Green):** ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: payments spec: replicas: 12 strategy: blueGreen: activeService: payments-stable previewService: payments-preview autoPromotionEnabled: false autoPromotionSeconds: 0 scaleDownDelaySeconds: 600 prePromotionAnalysis: templates: - templateName: pre-promotion-smoke-tests args: - name: service value: payments template: spec: containers: - name: app image: ghcr.io/org/payments:v5.4.0 ``` The scaleDownDelaySeconds: 600 keeps the old Blue stack running for 10 minutes after cutover. If issues appear within that window, a rollback is instant. After 10 minutes, Blue is terminated and capacity is freed. ## Failure Modes and Guardrails These guardrails apply to all three strategies: **Schema drift:** Use the expand-migrate-contract pattern for database changes. Expand the schema first, deploy code that reads and writes the new column alongside the old one, then contract by removing the old column in a follow-up release. This ensures both old and new code can coexist. **Cache or session churn:** Version your cookies or use feature flags to invalidate old sessions during switchover. Better yet, use stateless JSON Web Tokens (JWTs) so cache eviction doesn't break user sessions mid-rollout. **Feature flags vs. releases:** Feature flags gate behavior post-deploy (e.g., enable a new algorithm for 10% of users). Release strategies gate infrastructure (which version runs). Don't conflate the two. Use both: canary to gate infrastructure rollout, then feature flags to gate feature rollout within the new infrastructure. **Health illusions:** A canary shows healthy latency metrics but real user logins fail because auth SLOs were not included in analysis templates. Always include user-journey checks (login, checkout, API calls) alongside infrastructure metrics (CPU, latency, error rate). **Noisy metrics:** Use longer lookback windows (e.g., 3-5 minute aggregations instead of 30-second), robust thresholds (e.g., p99 latency \< 400ms not \< 100ms), and guard against insufficient traffic per step (minimum 100 requests per minute before analyzing). A step with 10 requests can have wildly swinging error rates. **Capacity cliffs:** Reserve 30-40% surge capacity; cap concurrency limits on critical services; and limit horizontal pod autoscaler (HPA) aggression during rollouts (set max scale-up rate to 2x per 5 min, not 10x per 1 min). ## At-a-Glance Comparison | Strategy | Traffic Control | Capacity Overhead | Rollback Speed | Best For | | --- | --- | --- | --- | --- | | Rolling (Gradual) | Implicit via LB | Low | Moderate (re-rollout) | Safe, backward-compatible changes | | Canary | Weighted, stepped (%, duration, metrics) | Low-Medium | Fast (auto on breach) | Risky updates where metrics are decisive | | Blue-Green | Binary flip | High (dual environment) | Instant (1 command) | Clean cutovers or compatibility breaks | ## Operator Checklist: Main Settings You Control **All strategies:** * Health probes, PDBs, HPA min/max, surge buffers * Observability: golden signals, SLOs, error budgets, distributed tracing * Automated rollback policies and on-call escalation hooks **Rolling deployments:** * maxSurge, maxUnavailable * Probe thresholds (failureThreshold, periodSeconds, initialDelaySeconds) * minReadySeconds **Canary deployments:** * Step weights, pause durations * Analysis templates (queries, success conditions, failure limits) * Metric lookbacks (how much historical data to include in analysis) * Abort gates and traffic router selection (Nginx, Istio, ALB) * maxSurge, maxUnavailable **Blue-Green deployments:** * activeService, previewService * autoPromotionEnabled, autoPromotionSeconds * scaleDownDelaySeconds * Pre-promotion smoke tests and validation templates ## Migration Path: From Rolling to Canary If you're already using Rolling deployments today, canary is a natural next step. You keep the same container images and deployment process. The difference: add a traffic router (Nginx ingress or service mesh), define SLO analysis templates, and configure traffic steps (e.g., 5% -> 25% -> 50% -> 100%). Most teams can migrate from Rolling to Canary in 2-4 weeks. ## Putting It Together: A Pragmatic Release Train A realistic release pipeline combines all three strategies: 1. **Build and validate:** Run tests, static and dynamic application security testing (SAST/DAST), image scans, software bill of materials (SBOM), and policy checks. 2. **Staging validation:** Deploy to staging, run smoke tests, shadow or proxy traffic from production to staging (for realistic load testing). 3. **Production canary:** Deploy to 2% of traffic with SLO analysis gates. Promote to 10% if metrics pass. Continue to 25%, then 50%, with metric checks at each step. Auto-rollback if any SLO breaches. 4. **Full promotion or blue-green flip:** For incompatible changes, skip gradual canary and go straight to blue-green for instant switchover. 5. **Post-promotion verification:** Monitor stricter SLOs for 10-15 minutes after full rollout. Watch error budgets burn rate; if it's too high, trigger an automated alert or manual rollback. As you can see, building a robust, progressive delivery pipeline involves a lot of complex configuration, metric integration, and YAML orchestration just to get started. Most teams spend weeks tuning analysis templates and thresholds. ## One-Click Rollouts with Skyhook.io Building this yourself means: * Spending 3-4 weeks learning Argo Rollouts API, CRD semantics, and failure modes * Writing and testing AnalysisTemplates for each service * Integrating Prometheus, Datadog, or Kayenta manually * Debugging YAML syntax errors and metric query mismatches * Managing Nginx or Istio ingress rules by hand * Training your team on rollout lifecycle and manual promotion With Skyhook.io, powered by Argo Rollouts, teams don't have to hand-craft YAML every time: * Choose strategy per service (Rolling, Canary, Blue-Green) from the UI in 30 seconds * Pick traffic steps (weights and pauses) or use battle-tested presets from our library * Attach AnalysisTemplates for Prometheus (or others) in one click * Auto-rollback on SLO breaches and notify via Slack * Preview and promote safely with audit trails showing who promoted what and when * Abort or rollback live rollouts from the dashboard without touching kubectl You focus on shipping fast; Skyhook handles the orchestration. If you're already on Kubernetes, adopting progressive delivery should be a one-click decision, not a three-month project. Ready to ship without fire drills? Start a free trial on Skyhook.io today. --- ### How to Manage Secrets in Kubernetes: 5 Approaches Compared - **URL**: https://www.skyhook.io/blog/how-to-manage-secrets-in-kubernetes - **Date**: September 2025 - **Author**: Eyal Dulberg, CTO - **Category**: Kubernetes - **Tags**: kubernetes, secrets-management, security, gitops, for-devops **TL;DR.** Kubernetes Secrets deliver values to pods; they do not create, rotate, audit, or safely store those values. Use Sealed Secrets or SOPS for GitOps-managed static secrets, a vault SDK when applications should fetch secrets directly, External Secrets Operator for centralized rotation across services, or the Secrets Store CSI Driver when values should be mounted without becoming Kubernetes Secret objects. Kubernetes has a Secret resource, but it doesn't manage secrets. It's a delivery mechanism - a way to get a value into a pod. RBAC controls who can read them, and managed clusters (EKS, GKE, AKS) encrypt etcd at rest, so the basics are covered. What's missing is everything around the lifecycle: how secrets get created, how they rotate, how you audit changes, and how you store them safely in Git without leaking plaintext. Five approaches handle this, each with different tradeoffs around GitOps compatibility, rotation, multi-cluster support, and how much your application code needs to change. Here's each one with working configs you can adapt. ## Sealed Secrets [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) flips the problem: encrypt secrets so they're safe to commit to Git. The controller runs in your cluster with a private key. You encrypt locally with the `kubeseal` CLI using the controller's public key, producing a `SealedSecret` that only your cluster can decrypt. ```yaml # Generated by: kubeseal --format yaml < my-secret.yaml # Safe to commit to Git - only the target cluster can decrypt this apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: database-credentials namespace: production spec: encryptedData: # Each value is encrypted with the cluster's public key DB_HOST: AgA3f8...truncated...Q4Kz== DB_PASSWORD: AgCE7...truncated...xR9w== DB_USERNAME: AgBy2...truncated...mN1c== template: metadata: name: database-credentials namespace: production type: Opaque ``` The workflow: `kubectl create secret` locally, pipe through `kubeseal`, commit the `SealedSecret` to Git, push. The in-cluster controller decrypts it into a regular Kubernetes Secret. **Tradeoffs**: encryption keys are per-cluster, so a `SealedSecret` encrypted for cluster A won't work on cluster B. No built-in rotation - if a value changes, you re-seal and commit. If you lose the controller's private key, you re-encrypt everything. Doesn't scale well across many clusters without automation on top. **Best for**: single-cluster GitOps teams with mostly static secrets (API keys, database credentials) that change infrequently. An automation layer on top (like Skyhook) can handle the re-sealing and multi-cluster coordination, but out of the box it's manual. For auditing which workloads reference which secrets across your cluster, [Radar's cluster audit view](https://radarhq.io/product/cluster-audit) surfaces it without extra tooling. ## SOPS + ArgoCD/Flux [Mozilla SOPS](https://github.com/getsops/sops) encrypts values in-place inside YAML, JSON, or ENV files. The file structure stays readable - only the values are encrypted. SOPS supports multiple key backends: AGE, AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault. ```yaml # Encrypted with SOPS - structure visible, values encrypted # Decrypt with: sops -d secret.enc.yaml apiVersion: v1 kind: Secret metadata: name: api-credentials type: Opaque stringData: API_KEY: ENC[AES256_GCM,data:8fGk...truncated,type:str] API_SECRET: ENC[AES256_GCM,data:Qs7x...truncated,type:str] sops: kms: - arn: arn:aws:kms:us-east-1:123456789:key/abc-def-123 created_at: "2025-09-01T10:00:00Z" enc: AQICAHh...truncated version: 3.9.0 mac: ENC[AES256_GCM,data:xF7d...truncated,type:str] ``` Integration with GitOps tools happens via plugins: [helm-secrets](https://github.com/jkroepke/helm-secrets) for Helm-based workflows, or [KSOPS](https://github.com/viaduct-ai/kustomize-sops) for Kustomize. For [ArgoCD specifically](/blog/argocd-in-production-patterns-that-actually-matter), you install the plugin in the repo-server container so ArgoCD can decrypt during sync. **Tradeoffs**: every developer who touches encrypted files needs access to the encryption key (or KMS permissions). Editing an encrypted file recomputes the MAC, which means noisy diffs. Setting up the ArgoCD repo-server plugin requires a custom image or init container. **Best for**: small teams who want encrypted files in Git backed by cloud KMS, and are comfortable with the plugin setup. ## Direct vault integration via SDK Skip Kubernetes entirely. Your application fetches secrets directly from a vault at runtime using the provider's SDK. ```go // Fetch a secret from AWS Secrets Manager at startup import ( "context" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/secretsmanager" ) func getSecret(ctx context.Context, name string) (string, error) { cfg, err := config.LoadDefaultConfig(ctx) if err != nil { return "", err } client := secretsmanager.NewFromConfig(cfg) result, err := client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{ SecretId: &name, }) if err != nil { return "", err } return *result.SecretString, nil } ``` This works with HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or any provider with an SDK. Vault also supports Kubernetes auth - your pod's ServiceAccount JWT is exchanged for a Vault token, so no static credentials needed. **Tradeoffs**: every service needs SDK code and auth logic. You're coupled to a specific provider - switching from AWS Secrets Manager to Vault means code changes, not config changes. No GitOps visibility into what secrets exist or which services use them. Harder to manage consistently across environments (dev/staging/prod). **Best for**: teams already deep in a single cloud ecosystem, or applications that need dynamic, short-lived credentials (like database passwords rotated per-connection). ## External Secrets Operator (ESO) [External Secrets Operator](https://external-secrets.io/) bridges external vaults and Kubernetes. It's an operator that syncs secrets from external vaults into native Kubernetes Secrets. Your application reads a regular Secret or env var - ESO handles the vault plumbing behind the scenes. Two CRDs do the work. A `SecretStore` configures the connection to your vault: ```yaml # Tells ESO how to authenticate with AWS Secrets Manager apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: aws-secrets namespace: production spec: provider: aws: service: SecretsManager region: us-east-1 auth: jwt: serviceAccountRef: name: external-secrets-sa # Uses IRSA for zero-credential auth ``` An `ExternalSecret` declares which secrets to sync and how often to refresh: ```yaml # Syncs database credentials from AWS Secrets Manager into a K8s Secret # ESO refreshes the value every hour - rotation handled automatically apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: database-credentials namespace: production spec: refreshInterval: 1h # Check for updates every hour secretStoreRef: name: aws-secrets kind: SecretStore target: name: database-credentials # Name of the K8s Secret to create creationPolicy: Owner # ESO owns and manages this Secret data: - secretKey: DB_HOST # Key in the K8s Secret remoteRef: key: prod/database # Path in AWS Secrets Manager property: host # JSON field within the secret - secretKey: DB_PASSWORD remoteRef: key: prod/database property: password ``` ESO supports 20+ providers: Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password, Doppler, and more. You define the connection once in the `SecretStore`, then create `ExternalSecret` resources for each secret you need. **Tradeoffs**: the secrets still land as Kubernetes Secrets in etcd. You need vault infrastructure running somewhere. Setup complexity scales with the number of providers and namespaces. **Best for**: most production teams. You get a centralized source of truth with rotation, audit trails, and provider-agnostic applications. This is the approach that scales. ## Secrets Store CSI Driver The [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/) takes a different path: mount secrets as files directly from a vault into the pod's filesystem. No Kubernetes Secret object is ever created (unless you explicitly opt in). ```yaml # Defines which secrets to mount from AWS Secrets Manager apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: aws-db-secrets namespace: production spec: provider: aws parameters: objects: | - objectName: "prod/database" objectType: "secretsmanager" jmesPath: - path: host objectAlias: db-host - path: password objectAlias: db-password ``` ```yaml # Pod mounts secrets as files at /mnt/secrets/ apiVersion: v1 kind: Pod metadata: name: api-server namespace: production spec: serviceAccountName: api-server-sa containers: - name: api image: ghcr.io/org/api:v2.1.0 volumeMounts: - name: secrets mountPath: /mnt/secrets # Secrets appear as files here readOnly: true volumes: - name: secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: aws-db-secrets ``` Your application reads `/mnt/secrets/db-password` as a file. The secret never exists as a Kubernetes object in etcd. **Tradeoffs**: secrets are only available after the volume mounts - race conditions are possible if your app starts before the mount completes. No offline/disconnected operation (the vault must be reachable at pod start). Runs as a DaemonSet on every node. If you need secrets as env vars, you have to enable the optional sync-to-Secret feature, which defeats the main benefit. **Best for**: high-security environments where secrets must never touch etcd, and you can design your application to read from files. ## Decision framework | | Sealed Secrets | SOPS | Direct SDK | ESO | CSI Driver | |---|---|---|---|---|---| | **GitOps-friendly** | Yes | Yes | No | Yes | Partial | | **Rotation** | No | No | Yes | Yes | Yes | | **Multi-cluster** | Hard | Moderate | N/A | Easy | Easy | | **Setup complexity** | Low | Medium | Low (per-app) | Medium | Medium-High | | **Secrets in etcd** | Yes | Yes | No | Yes | No | | **App code changes** | None | None | Yes | None | File reads | | **Audit trail** | Git only | Git only | Vault logs | Vault logs | Vault logs | **Quick decision tree**: - Single-cluster, static secrets, GitOps workflow - **Sealed Secrets**. - Small team, cloud KMS, comfortable with plugins - **SOPS**. - Already using a vault and want apps to fetch directly - **Direct SDK**. - Production team, multiple services, need rotation and audit - **ESO** (most common choice). - Maximum security, secrets must never touch etcd - **CSI Driver**. These approaches aren't mutually exclusive. Many teams use Sealed Secrets for bootstrap secrets (like the ESO credentials themselves) and ESO for everything else. ## How Skyhook handles secrets Skyhook ships with Sealed Secrets built into its [GitOps workflow](/blog/argocd-in-production-patterns-that-actually-matter). When you add a secret through Skyhook, it fetches your cluster's public key, encrypts the value with RSA + AES-256-GCM (the same hybrid encryption Sealed Secrets uses), and opens a PR in your GitOps repo. You review, merge, and ArgoCD applies it. Batch updates across clusters and namespaces work the same way - one PR, multiple sealed secrets. For teams that prefer an external vault, External Secrets Operator is available as a [one-click addon](/blog/argo-cd-vs-terraform-for-kubernetes-add-ons) - Skyhook deploys it via ArgoCD with the Helm chart and CRDs configured. Whichever approach you choose, the goal is the same: secrets with an audit trail, encrypted at rest, and managed through the same Git workflow as everything else. ## FAQ ### Are Kubernetes Secrets encrypted? No - they're base64-encoded, which is encoding, not encryption. But this is less scary than it sounds. Kubernetes relies on RBAC as the access control boundary, not encryption of the Secret object itself. If you can `kubectl get secret`, you're already authorized to see that value - encrypting it would just add a decryption step for the same person. The storage layer (etcd) is encrypted at rest on all major managed providers. The real gaps with vanilla Secrets aren't about encryption - they're about lifecycle: no rotation, no audit trail, no safe way to store them in Git, and no management workflow beyond `kubectl create secret`. ### What is the best way to manage secrets in Kubernetes? For most production teams, External Secrets Operator (ESO) is the strongest default. It syncs secrets from an external vault (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, etc.) into native Kubernetes Secrets, with automatic rotation and audit trails. Your application code stays unchanged - it reads a normal Secret or env var. ### Can I store Kubernetes Secrets in Git? Not safely in plain text. Tools like Sealed Secrets and SOPS encrypt secret values so the encrypted form is safe to commit. Sealed Secrets uses asymmetric encryption tied to your cluster's key pair. SOPS encrypts values in-place using cloud KMS or AGE keys. Both let you keep secrets in your GitOps repo without exposing plaintext. ### How do I rotate secrets in Kubernetes? Native Kubernetes Secrets have no rotation mechanism. For automatic rotation, you need either an external vault with ESO (set a `refreshInterval` and ESO polls for updates), a CSI Driver that re-mounts on rotation, or direct SDK integration where your app fetches fresh credentials at runtime. Sealed Secrets and SOPS require manual re-encryption and a new commit. --- ### Preview Environments: A Game-Changer for Testing and Collaboration - **URL**: https://www.skyhook.io/blog/preview-environments-a-game-changer-for-testing-and-collaboration - **Date**: August 2025 - **Author**: Eyal Dulberg, CTO - **Category**: DevOps - **Tags**: explainer, preview-environments, testing, for-developers Preview environments, also known as ephemeral environments, are no longer a luxury - they’re becoming a necessity for modern development teams. These temporary setups allow developers to test, validate, and showcase their work in an environment that mirrors production. Unlike traditional staging setups, preview environments are dynamically created on demand and discarded once their purpose is served. In this post, we’ll delve into the mechanics behind preview environments, the real-world problems they solve, and why they’re reshaping software development. ## What Are Preview Environments? Preview environments are isolated, disposable, and fully automated replicas of your production environment. They are purpose-built for specific tasks, such as testing a feature, reviewing a pull request, or validating a hotfix. These environments are clean by design - free from conflicts or remnants of prior tests. **How They Work** When a developer creates a pull request or submits code for review, the CI/CD pipeline triggers the creation of a preview environment. Tools like Kubernetes and Docker orchestrate the deployment of all necessary components - applications, databases, and external services - ensuring the environment matches production as closely as possible. Developers can visually inspect a preview environment's topology with tools like [Radar](https://radarhq.io/product/topology) before merging. Once testing or validation is complete, the environment is torn down automatically, optimizing resource usage. **Example Use Case** Imagine developing a feature that integrates with a third-party API. Instead of risking interruptions or conflicts in a shared staging environment, you can spin up a preview environment tailored for this feature. After validation, the environment is discarded, leaving no residual configurations or data. ## Why Should You Use Preview Environments? ### 1. Faster Feedback Loops In traditional workflows, developers often wait hours - or even days - for a shared staging environment to become available. Preview environments remove this bottleneck. Each pull request gets its own isolated space, enabling immediate validation. **Technical Impact** * Faster code reviews, as stakeholders can interact directly with the feature instead of relying on screenshots. * Automated feedback loops integrated into CI/CD pipelines eliminate manual setup time. **Customer Insights** Teams using preview environments report a **60% reduction in feature delivery times**, as they can validate changes in real time without dependencies on shared environments. ### 2. Better Testing Testing in a shared staging environment often introduces unpredictable variables: leftover data, misaligned configurations, or undetected changes. Preview environments start fresh every time, ensuring consistent and accurate testing. **Technical Impact** * Match production setups precisely by replicating configuration, dependencies, and integrations. * Enable A/B testing or experimental feature toggles without impacting other team members. **Customer Insights** Organizations leveraging preview environments see up to **50% fewer production bugs**, as critical issues are caught earlier in the lifecycle. ### 3. Improved Collaboration Across Teams Preview environments create a live, interactive sandbox where developers, QA teams, and non-technical stakeholders can review and test changes. **How This Helps** * Marketing teams can preview new features before they’re live, ensuring alignment on messaging. * QA engineers can perform exploratory testing without blocking other team members. **Real-Life Use Case** A product manager preparing a demo for an upcoming client meeting can use a preview environment to showcase a feature in its final form, months before its official release. ### 4. Cost Efficiency Maintaining a traditional testing environments often incurs significant costs, as these setups run 24/7 regardless of usage. Preview environments, on the other hand, are ephemeral - consuming resources only when active. **Technical Insight** Cloud-native tools like Kubernetes ensure environments are dynamically provisioned, leveraging pay-as-you-go infrastructure models. This minimizes overhead costs without sacrificing quality. ### 5. Seamless Integration with CI/CD Pipelines Preview environments fit naturally into modern CI/CD workflows. Every code change can trigger automated deployment, validation, and teardown processes. **How This Helps** * Automates end-to-end testing for each pull request. * Prevents human errors by codifying environment setups in infrastructure-as-code (IaC) tools like Terraform. **Technical Example** A CI/CD pipeline using tools like ArgoCD or GitHub Actions can automatically deploy a feature branch into a preview environment, run end-to-end tests, and clean up the environment - all in a single pipeline execution. ## Challenges Preview Environments Solve ### 1. Inconsistent Testing Shared staging environments often become dumping grounds, leading to unreliable results. Preview environments eliminate this by starting fresh every time. ### 2. Resource Contention Teams waste valuable time waiting for staging environments to free up. Preview environments ensure every team member has their own isolated space, accelerating development cycles. ### 3. Communication Gaps Explaining feature behavior through documentation or screenshots often falls short. A live preview environment demonstrates functionality in a way that words cannot. ## Why Teams Love Preview Environments * **Faster Time-to-Market:** Accelerate release cycles with parallel testing. * **Higher Confidence in Deployments:** Reduce production bugs by validating in production-like setups. * **Streamlined Developer Workflows:** Free up developers to focus on writing code instead of managing environments. ## Why Are Preview Environments So Difficult to Set Up? Building a robust preview environment platform is far from straightforward. It requires replicating production setups with all dependencies, automating dynamic provisioning, and ensuring seamless CI/CD integration. Challenges include managing stateful services like databases, scaling environments cost-effectively, and aligning workflows with developer needs. Most solutions are piecemeal, requiring significant time and resources to maintain, leading to high costs and inconsistent results. **Why Skyhook?** Skyhook automates these processes, providing a seamless, cost-efficient solution that eliminates the operational burden of preview environments. Skyhook simplifies preview environments with: * Automated environment setup and teardown. * Seamless integration with popular CI/CD tools. * Built-in support for managing environment variables and secrets. ## Conclusion Preview environments represent a paradigm shift in how software is developed and tested. By providing isolated, production-like setups on demand, they empower teams to deliver higher-quality software faster. From faster feedback loops to fewer production bugs, the benefits are transformative. If you're ready to embrace the future of software development, give **Skyhook** a try and experience the power of preview environments firsthand. --- ### Managing Kubernetes Add-ons: Argo CD or Terraform? - **URL**: https://www.skyhook.io/blog/argo-cd-vs-terraform-for-kubernetes-add-ons - **Date**: July 2025 - **Author**: Eyal Dulberg, CTO - **Category**: Kubernetes - **Tags**: comparison, argocd, terraform, kubernetes, for-devops ## Why this matters A big part of the magic of Kubernetes, and maybe the main reason it has exploded in popularity, is its extensibility through standardized APIs for resources and Custom Resource Definitions (CRDs). The ecosystem has exploded with tools or ”add-ons” like cert-manager, Argo Rollouts, External Secrets Operator, Loki, Prometheus, and Grafana, just to name a few, that significantly enhance the core Kubernetes capabilities. These add-ons are not quite infrastructure, and not quite applications. Choose the wrong tool to manage them and you can end up fighting drift, breaking CI, or scrambling in production. Below you will see two of the most popular approaches to managing add-ons - Argo CD and Terraform - and why one is starting to pull away as the favorite among DevOps teams.. ## TLDR Use Terraform to bootstrap cloud and cluster. Use Argo CD to operate in-cluster add-ons. Argo CD is natively designed for this, solving a lot of problems that Terraform is ill-suited to handle. ## Key differences at a glance | Area | Argo CD | Terraform | | --- | --- | --- | | Primary lifecycle role | Day-2 app/add-on operations inside K8s | Day-1 cloud/K8s infra provisioning | | Scope | In-cluster Kubernetes resources and Helm charts via Git | Cloud infra plus Kubernetes via providers | | Operational model | Pull: continuous reconciliation and drift detection + auto-heal | Push: plan and apply on demand, drift ignored until next run | | Source of truth | Desired state in Git, live state in the Kubernetes API server | Desired state in Git (.tf), last known state in remote state (.tfstate) | | Centralization | Decentralized controllers per cluster, smaller blast radius | Centralized pipelines and shared state | | Typical ownership | Platform/DevOps for setup, then application teams | Platform/Infra teams (often too risky for non-experts) | | Visibility | Built-in UI and CLI show health, history, and diffs | CLI output unless paired with other tools | | Rollbacks | Built-in rollback to previous revisions | No native rollback - revert Git and re-apply | | Multi-cluster | ApplicationSet generators and App-of-Apps pattern | Workspaces, modules, and pipelines | | Failure modes to watch | Bad commit breaks sync but is easy to revert | State lock, provider errors, partial applies | | Best fit | Add-on upgrades, rollbacks, canary or blue/green releases | Bootstrapping clusters, VPCs, IAM, initial add-on install | ## Argo CD in action ```yaml # Minimal cert-manager Application apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd spec: project: default destination: server: https://kubernetes.default.svc namespace: cert-manager source: repoURL: https://charts.jetstack.io chart: cert-manager targetRevision: v1.15.0 helm: values: | installCRDs: true syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` **What you get** - Immediate drift repair. If someone edits a resource with kubectl, Argo CD reverts it within seconds. - Multi-cluster fan-out. ApplicationSet can generate identical apps for many clusters. - Audit trail. Every change is a Git commit that can be rolled back with a click or PR revert. ## Terraform in action ```hcl terraform { required_providers { helm = { source = "hashicorp/helm", version = "~> 2.13" } kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.29" } } } provider "kubernetes" { host = var.cluster_endpoint cluster_ca_certificate = base64decode(var.cluster_ca) token = var.cluster_token } provider "helm" { kubernetes { host = var.cluster_endpoint cluster_ca_certificate = base64decode(var.cluster_ca) token = var.cluster_token } } resource "helm_release" "cert_manager" { name = "cert-manager" repository = "https://charts.jetstack.io" chart = "cert-manager" version = "v1.15.0" namespace = "cert-manager" create_namespace = true set { name = "installCRDs" value = "true" } } ``` **What you get** - One workflow for everything. The same tool provisions VPCs, EKS or GKE, and Helm charts. - Predictable plans. terraform plan shows a diff before anything changes. - Remote state backends. S3, GCS, or Terraform Cloud keep state consistent across teams. **Trade-off**: drift inside the cluster is invisible until the next plan. Fixes may require a replace if resources diverged. ### Two real-world drift moments * **Someone deletes the cert-manager webhook** * Argo CD: Application goes OutOfSync then heals automatically. * Terraform: nothing happens until the next plan or scheduled run, then a replace may be required. * **External Secrets Operator changes a CRD schema between chart versions** * Argo CD: bump the chart version in Git, watch health, roll back instantly if custom resources fail to reconcile. * Terraform: plan shows the Helm upgrade, but CRD behavior changes surface only at apply, sometimes with manual cleanup. ## Use the right tool for the job ### Day 1 with Terraform - Provision VPC, cluster, node pools, IAM or OIDC roles. - Install Argo CD. - Output Argo CD URL and the bootstrap Git repo for later steps. ### Day 2 with Argo CD - Manage every add-on as an Argo CD Application. - Use PRs for upgrades and rollbacks. - Watch diff and health views in daily stand-ups. ### Quick decision checklist - Need to manage cloud primitives or create clusters. Choose Terraform. - Need frequent add-on upgrades, rollbacks, and clear daily visibility. Choose Argo CD. - Mixed needs. Bootstrap with Terraform, operate add-ons with Argo CD. - Regulated environment with strict windows. Use Terraform plans for infra and PR reviews for Argo CD apps. ## How we implemented this in Skyhook Skyhook ships with the hybrid model built in. ### Day 1 bootstrap - One-click Terraform module spins up VPC, cluster, node pools, and IAM or OIDC roles. - Run the module via Skyhook or commit it to your Git repository so you keep ownership. ### Day 2 and beyond - Skyhook installs and configures Argo CD automatically - no manual YAML or Helm values to manage. - Select which clusters should run Argo CD from a single dashboard. - Browse a catalog of add-ons (cert-manager, Argo Rollouts, External Secrets, Loki, Prometheus, and more) and enable them per cluster. - When you select an add-on, Skyhook opens a PR with the matching manifests in your repo - you approve, it deploys. - Updates, rollbacks, drift repair, and multi-cluster fan-out all happen through Argo CD under the hood. Skyhook surfaces health, sync status, and history in one place. The result: you get the power of Terraform for infrastructure and Argo CD for in-cluster operations, without the boilerplate and drift wars that come with wiring it all together yourself. ## Further reading - Argo CD App-of-Apps pattern: [https://argo-cd.readthedocs.io/en/stable/operator-manual/app-of-apps/](https://argo-cd.readthedocs.io/en/stable/operator-manual/app-of-apps/) - CNCF GitOps Working Group best practices: [https://opengitops.dev/](https://opengitops.dev/) --- ### PaaS Kubernetes vs. DIY Kubernetes: Why PaaS Wins for Simplicity and Efficiency - **URL**: https://www.skyhook.io/blog/paas-kubernetes-vs-diy-kubernetes-why-paas-wins-for-simplicity-and-efficiency - **Date**: May 2025 - **Author**: Nadav Erell, CEO - **Category**: Kubernetes - **Tags**: comparison, kubernetes, paas, build-vs-buy, for-engineering-leaders When it comes to deploying and managing Kubernetes, the simplicity and efficiency of PaaS solutions often outweigh the complexities of a DIY approach. In this post, we'll explore why PaaS Kubernetes is the better choice for businesses aiming to scale without operational overhead. ********* Kubernetes has revolutionized container orchestration, transforming how we deploy, manage, and scale applications. However, the question remains: should you choose a DIY (Do It Yourself) approach or a PaaS (Platform as a Service) solution? This blog post explores why PaaS Kubernetes outperforms DIY Kubernetes in simplicity and efficiency for most businesses. ## Understanding the Basics * **Kubernetes (K8s):** An open-source container orchestration platform that automates the deployment, scaling, and management of application containers. * **DIY Kubernetes:** Setting up, configuring, and maintaining your own Kubernetes cluster, usually on-premises or in a cloud environment. * **PaaS Kubernetes:** A cloud service offering Kubernetes as a platform, managing most of the infrastructure, enabling users to focus on deploying and managing their applications. ![The Responsibility Stack: DIY Kubernetes vs PaaS Kubernetes](/images/blog/paas-vs-diy-responsibility-stack.webp) ## The Dominance of PaaS Kubernetes ### Simplified Setup * DIY Kubernetes requires setting up the cluster from scratch, including node configurations and much more. * PaaS solutions offer a streamlined setup process, resulting in quicker deployments and minimized configuration errors. ### Automated Upgrades * Kubernetes frequently updates, and applying these updates on a DIY setup requires careful planning and often causes downtime. * PaaS services automatically manage and implement these updates, ensuring you're always up-to-date without manual intervention. ### Cost Efficiency * DIY may seem cost-effective initially, but considering infrastructure management, specialized staff, and potential downtimes, PaaS often proves more economical. * Many PaaS options have pay-as-you-go pricing models, so you're billed only for what you use. ### Fortified Security * PaaS providers invest heavily in security, including automated patching, threat detection, and built-in firewalls. * Securing a DIY Kubernetes setup is possible but requires extensive expertise and continuous monitoring. ### Comprehensive Developer Tools * PaaS offerings come with integrated tools for CI/CD, monitoring, logging, and more. * This facilitates a smooth development workflow and rapid issue identification and resolution. ### Seamless Scalability * While Kubernetes supports scalability, mastering it in a DIY environment can be challenging. * PaaS platforms handle scalability based on load, ensuring resources are always allocated appropriately. ## Additional Advantages of PaaS * **Cost Savings:** PaaS solutions can be more cost-effective than hiring dedicated infrastructure engineers or building an in-house infrastructure team. * **Faster Deployment:** PaaS automates DevOps processes, enabling quicker and more efficient deployments, leading to increased productivity. * **Easier Scalability:** PaaS platforms offer automated scalability features to meet user traffic demands, ensuring optimal resource utilization. * **Focus on Core Business:** PaaS frees up developers from infrastructure management, allowing them to concentrate on core product development. ## Conclusion The choice between PaaS Kubernetes and DIY Kubernetes depends on your organization's needs, expertise, and goals. However, for businesses looking to leverage Kubernetes without the operational complexities, PaaS is the clear winner. By choosing a PaaS solution which isn’t locked to any cloud vendor, businesses can unlock the vast potential of Kubernetes while maintaining simplicity and efficiency. ## Skyhook: Your Partner in Tech Decisions At Skyhook, we're always eager to engage in tech discussions and assist with your tech decisions. If you're considering Kubernetes or any other technology, don't hesitate to reach out\! --- ### Buy, Don't Build: The Case for Pre-Built Internal Developer Platforms - **URL**: https://www.skyhook.io/blog/why-pre-built-internal-developer-platforms - **Date**: April 2025 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: platform-engineering, idp, build-vs-buy, for-engineering-leaders TL;DR: Once you have decided you actually need an internal developer platform, the build-vs-buy question almost always resolves to buy. A self-built Backstage runs $450K-$800K in year one and stalls at ~10% adoption outside Spotify. A pre-built platform replaces 80% of that work for a fraction of the cost. The remaining tradeoffs are real but small. This post is what to evaluate and which flavor to pick. ## First, Make Sure You Need One Most teams that say "we need an IDP" don't actually mean it. They mean "deploys hurt, nobody knows who owns what, and onboarding takes too long." Those are real problems, but the cheapest fix is rarely a developer portal. We covered the threshold question in [Backstage Fatigue: When NOT to Build an Internal Developer Platform](/blog/when-not-to-build-an-internal-developer-platform). Short version: under ~150 engineers with one or two product groups, what you usually need is an opinionated deploy platform, not a portal. If you read that and concluded you do need a coherent platform, the next question is build vs buy. This post is about that decision. ## What "Build" Actually Costs Backstage is the open source default, so most "build" plans turn into "stand up self-hosted Backstage." The numbers below come from vendors selling alternatives - treat them as directional, not gospel. | Cost line | Realistic range | | --- | --- | | Time to first useful state | 6-12 months | | Dedicated platform engineers | 3+ | | Year-one fully-loaded cost | $450K - $800K | | Median external adoption rate | ~10% (vs Spotify's internal 99%) | The adoption number is the one that should haunt anyone considering self-hosting. [Helen Greul](https://thenewstack.io/spotifys-backstage-roadmap-aims-to-speed-up-adoption/), head of engineering for Backstage at Spotify, has said publicly that average external adoption stalls around 10% because most adopters don't make it past proof-of-concept. Spotify's own response was to launch Spotify Portal for Backstage - a hosted edition. When the canonical reference customer concludes the self-hosted experience is broken for everyone else, that should land somewhere. The non-cash cost is arguably worse. Every quarter a platform engineer spends on Backstage plugins is a quarter they didn't spend on something that moves your product. Three years in, you have a portal that 10% of engineers use and 100% of leadership has to keep funding because too much was spent to walk away. ## What "Buy" Actually Delivers Pre-built platforms come in two flavors. The right one depends on what hurts. **Hosted Backstage / portal-first.** Roadie, Cortex, Port, Spotify Portal. You get the Backstage UX without owning the chassis. Plugins, software catalog, scorecards, scorecard automation - on day one. You pay per developer. **Opinionated PaaS / platform-first.** Skyhook, Render, Northflank, Porter, Fly, Railway. You get the *outcome* an IDP is supposed to produce - one-click deploys, preview environments per PR, golden paths, secrets, observability - without a portal layer. The deploy experience itself is the developer experience. The naming around this category is intentionally fuzzy. Port markets itself as "Internal Developer Portal & Platform." Cortex calls itself both a "portal" and an "Engineering Operations Platform." Roadie is the most honestly named of the three - "hosted Backstage" - but the whole category often gets discussed as if portal-first and platform-first tools are interchangeable. They are not. None of the portal-first tools ship your code. They describe code that is already shipping: service catalogs, ownership, scorecards, dashboards. That is real value when your services already deploy reliably. It is not a substitute for the deploy experience itself. Octopus called this ["Platforms Before Portals"](https://octopus.com/blog/platforms-before-portals) for a reason: if the underlying deploy experience is the problem, a portal layered on top of it will not fix it. If you are 80 engineers with seven services and your real problem is "deploys are still tickets," portal-first won't fix that and platform-first will. If you are 150 engineers across three product groups, the deploy experience is already self-service, and your real problem is "nobody can find anything and ownership is a mess," portal-first is the right shape - but only because the platform underneath is already there. In either flavor, what you skip is everything that doesn't differentiate you: SSO/SCIM, RBAC, audit logging, the Kubernetes operators, the upgrade treadmill, the on-call rotation for the platform itself. That is the entire 80% that DIY Backstage spends two years rebuilding. ## The Honest Tradeoffs Buying isn't free in non-cash terms. Three real tradeoffs: **Less customization at the seams.** A pre-built platform has opinions. If your deploy flow has a gnarly compliance step that doesn't fit those opinions, you'll either bend your flow or use a vendor escape hatch. Most teams find this less painful than they feared, but walk through your weirdest workflow with the vendor before signing. **Vendor dependency.** You are now coupled to a roadmap that isn't yours. Mitigate by picking platforms that commit changes to *your* GitOps repo, so you can leave with your manifests intact, and that are open about their APIs. **Pricing scales with engineers, not value.** Per-developer pricing means a 200-engineer org pays roughly 10x what a 20-engineer org pays for the same platform, even if the larger org gets less marginal value because it has more in-house ops capacity. At a certain scale - Stripe, Shopify, Spotify - self-hosting Backstage actually does pencil out. That scale is much higher than most readers of this post. What you do *not* trade away by buying: extensibility (good vendors expose escape hatches), reliability (you skip the "your platform team is on call for the platform" trap), or technical depth (the pre-built ones are mostly built by people who watched DIY fail at their last company). ## What to Evaluate If you decide to buy, look for these in order: 1. **Does it solve your specific painful workflow today?** Walk through your three worst deploy scenarios with the vendor live. If they hand-wave on any of them, walk away. 2. **Is the configuration in your Git repo?** If the platform is "stateful in the vendor's database," your migration cost is locked in. Manifests-in-your-repo is the structural escape hatch. 3. **Does it cover both deploys and the boring stuff?** Preview environments per PR, secrets, observability, RBAC, SSO. If any are "on the roadmap," that's a six-month gap you'll fill yourself. 4. **What's the upgrade path when your needs sharpen?** A platform that handles you at 30 engineers but breaks at 150 is one you'll outgrow at exactly the wrong moment. 5. **Real customers in your shape.** Not logos at the top of the website - actual customers with the same engineer count and stack as you. Ask for two on a call. ## Where Skyhook Fits Skyhook is in the platform-first flavor. The thesis is that for teams under ~150 engineers, the deployment platform *is* the IDP - so we ship preview environments per PR, golden paths per language, deploy strategies, secrets, and a service catalog as defaults rather than as plugins you assemble. Configuration commits to your Git repos, so the escape hatch is structural. We are not the right fit if your real problem is "we need a beautiful Backstage portal for 250 engineers across five product groups." For most teams under that threshold, we are the version of "buy" that actually replaces the IDP project. If you are earlier in the decision and not sure you even need an IDP yet, start with [Backstage Fatigue](/blog/when-not-to-build-an-internal-developer-platform). If you have decided you do, the build-vs-buy answer is almost always buy. ## Further Reading - [Backstage Fatigue: When NOT to Build an Internal Developer Platform](/blog/when-not-to-build-an-internal-developer-platform) - The companion piece. Read this first if you are not sure you need an IDP at all. - [Golden Paths and the 80/20 Rule](/blog/golden-paths-and-the-80-20-rule) - Why a small set of opinionated paths covers most of what an IDP is supposed to do. - [Self-Serve Platforms and Service Catalogs](/blog/self-serve-platforms-and-service-catalogs) - The narrower problem most "we need an IDP" conversations are actually about. - [Helen Greul on Backstage adoption (The New Stack)](https://thenewstack.io/spotifys-backstage-roadmap-aims-to-speed-up-adoption/) - Primary source for the 10% adoption figure. --- ### Streamlining Development with Self-Serve Platforms and Service Catalogs - **URL**: https://www.skyhook.io/blog/self-serve-platforms-and-service-catalogs - **Date**: February 2025 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: explainer, service-catalog, developer-experience, self-service, for-engineering-leaders In software development, finding ways to boost efficiency without sacrificing quality is a constant challenge. One approach that's gaining traction is the use of self-serve development platforms with integrated service catalogs. These tools are helping developers navigate the complexities of modern software projects more smoothly and effectively. We've all experienced it. You need to spin up a new service or environment, but you're unable to do it on your own due to permission issues or lack of experience and knowledge of the standardization rules. So, you track down the DevOps expert, who's already juggling countless tasks, and end up in a back-and-forth exchange to get things set up. This process eats up both your time and theirs. But there's a better way. ## What Are Self-Serve Development Platforms with Service Catalogs? At their core, these platforms provide developers with easy access to a curated list of tools and services. This isn't just about convenience; it's about creating a more streamlined and standardized development process. **Key Benefits:** 1. **Speed and Efficiency:** Setting up a development environment can be time-consuming. Self-serve platforms reduce this friction by offering pre-configured tools and services that developers can access instantly. This speeds up the initial stages of a project and allows teams to focus on coding rather than setup. 2. **Consistency Across Teams:** With multiple teams working on different aspects of a project, inconsistencies can easily arise. A service catalog ensures that everyone is using the same tools and versions, reducing the risk of integration issues later on. 3. **Cost Management:** Centralizing tools and services in one platform can lead to cost savings. Instead of purchasing separate licenses or subscriptions for each tool, teams can leverage shared resources, which also helps avoid redundant spending. 4. **Developer Empowerment:** Developers often rely on other departments to provide the necessary tools and services. A self-serve platform changes that dynamic, giving developers more control and reducing delays. This autonomy not only enhances efficiency but also boosts job satisfaction. 5. **Seamless Collaboration:** By providing a single platform for all necessary tools, self-serve platforms facilitate better collaboration between teams. Everyone is on the same page, and integrating different components becomes much more straightforward. 6. **Improved Reliability:** Standardizing the tools and services used across projects reduces the chance of errors. Developers are less likely to encounter issues due to outdated or incompatible tools, leading to a higher quality end product. 7. **Scalability:** As projects grow, so do their requirements. A well-maintained service catalog can easily scale to accommodate new tools and services, ensuring that the platform evolves alongside the needs of the team. ## How Skyhook Fits In At Skyhook, we've built our platform with these principles in mind. Our self-serve development platform provides a comprehensive service catalog, enabling developers to quickly access the tools they need while maintaining consistency and control. We're focused on helping teams work more efficiently, without sacrificing quality or control. ## Final Thoughts The shift towards self-serve development platforms with service catalogs is more than just a trend - it's a necessary evolution in the way software is built. By reducing setup times, ensuring consistency, and empowering developers, these platforms are proving to be invaluable in today's fast-paced development environments. At Skyhook, we're committed to supporting this evolution, helping teams to not just meet but exceed their development goals. --- ### Platform Engineering & The Value of Golden Paths in the Light of the 80-20 Rule - **URL**: https://www.skyhook.io/blog/golden-paths-and-the-80-20-rule - **Date**: December 2024 - **Author**: Roy Libman, CPO - **Category**: Platform Engineering - **Tags**: explainer, golden-paths, platform-engineering, for-platform-engineers Explore how Platform Engineering and Golden Paths streamline software development by offering guided solutions for 80% of tasks, while leaving room for innovation in the remaining 20%. Enhance productivity without restricting creativity. In the ever-evolving realm of software development, stepping into a new role can be both thrilling and intimidating. For example, a Java developer joining a new company might find themselves not just coding in Java but navigating the complex world of DevOps, tackling operational codes, building pipelines, monitoring systems, and perhaps an entirely new cloud platform. The tasks ahead may seem overwhelming. ## Platform Engineering: The Solution to DevOps Complexity Enter Platform Engineering, a discipline curated to lighten the developers' load. By crafting efficient abstractions and offering self-service infrastructure, Platform Engineering seeks to consolidate diverse tools, paving the way for a more streamlined developer experience. It's an antidote to the potential cognitive overload that can emerge from a dense DevOps model where shared responsibilities can sometimes blur the boundaries between Development and Operations teams. ## Golden Paths: Guided Journeys in Development Platform Engineering introduces the concept of 'Golden Paths'. As described by the Cloud Native Computing Foundation, Golden Paths are "templated compositions of well-integrated code and capabilities for rapid project development." In essence, it's a pre-defined guide designed for standard tasks. But where does the 80-20 rule fit into this? The 80-20 rule suggests that 80% of the tools and configurations developers require can be catered to by conventional Golden Paths. This means for the majority of tasks, a Java Spring Boot Golden Path Template or a “React Web Frontend” guide, for instance, would suffice. Such a repository could encompass: * A sequential tutorial. * Preliminary source codes. * Dependency management provisions. * CI/CD pipeline designs. * Cloud infrastructure-as-code outlines. * Kubernetes YAML configurations. * Logging and monitoring tools. However, it's crucial to note that while these paths serve the 80% need, there remains a 20% niche requirement. For these unique, outside-the-box tasks, developers must have the autonomy to venture outside the Golden Path and formulate custom solutions. Thus, while Golden Paths provide direction, they should not be restrictive. They should be paths, not cages. The flexibility to deviate ensures innovation isn't stifled, catering to both routine and specialized tasks. ## Who Stands to Gain from Golden Paths? Golden Paths' utility stretches across roles: * **Application Developers:** They benefit from direct, clear guidance, mitigating the intricacies of infrastructure code and refining the onboarding process. * **Site Reliability Engineers (SREs):** Uniformity across services allows SREs to speed up troubleshooting during system downtimes. Tools like [Radar](https://radarhq.io/for/sre) complement golden paths by giving responders a consistent troubleshooting surface across services. * **Security Administrators:** By using Golden Paths, they can systematically enforce security policies, ensuring protection isn't compromised. * **Organization at Large:** Resource optimization and overall efficiency are enhanced organization-wide. ## Core Principles & Best Practices Golden Paths, while differing based on an organization's profile, share several universal principles: * **Provide clear, opinionated methods** to achieve specific tasks, in order to **reduce cognitive load**. * **Integrate** effortlessly with existing platforms. * **Address most routine (80%) requirements.** * **Be transparent.** The abstraction provided by a Golden Path should not be such that a developer cannot dive into the underlying infrastructure if and when needed. * **Be adaptable and non-restrictive**, ensuring they don't become golden cages. * **Be self-service**, so that developers are able to use the golden paths without being delayed by a gatekeeper. * **Be optional**, so that a developer is not forced to use them if they don’t suit their current needs. ## Harnessing External Expertise Although many companies opt to devise their own Golden Paths, there's substantial merit in seeking external proficiency. Not every organization possesses the resources or specialized knowledge for in-house creation. External experts, with their extensive know-how, can furnish ready-to-use Golden Paths, customized for distinct organizational demands, saving companies the effort of starting from scratch. Just as you don’t expect every company to build their own CRM system, you shouldn’t expect every company to build their own internal development platform from scratch\! ## In Conclusion Golden Paths, developed internally or sourced externally, can be revolutionary in boosting productivity. When done well, following the 80-20 rule, they address the majority's needs without becoming limiting in solving the unique challenges of the few. As organizations plan their road ahead, let Golden Paths, not cages, lead the way, ensuring optimal operations and enriched developer experiences. --- ### The Rise of Platform Engineering: Redefining DevOps for the Modern Era - **URL**: https://www.skyhook.io/blog/the-rise-of-platform-engineering-redefining-devops-for-the-modern-era - **Date**: October 2024 - **Author**: Nadav Erell, CEO - **Category**: Platform Engineering - **Tags**: explainer, platform-engineering, devops, for-engineering-leaders Platform Engineering is emerging as a significant trend in DevOps. Far from being just another buzzword, it represents the natural evolution of DevOps practices, addressing the pressing challenges organizations face today. ****** In the ever-evolving landscape of DevOps, there's a new trend making waves: Platform Engineering. At first glance, it might seem like just another buzzword, but when you dig deeper, you realize it's a natural evolution of DevOps practices, addressing some of the most pressing challenges faced by organizations today. ## What is Platform Engineering? Platform Engineering is about creating a set of reusable tools, services, and workflows that enable developers to build and deploy applications more efficiently. It’s the practice of designing and building self-service, scalable platforms that empower developers to focus on writing code and delivering value without getting bogged down by the underlying infrastructure. This shift is driven by the increasing complexity of modern software systems. With microservices, containerization, and cloud-native architectures becoming the norm, developers are often required to manage a complex web of tools and processes. Platform Engineering aims to simplify this by providing a curated set of tools and services that abstract away the complexity, allowing developers to focus on what they do best. ## The Need for Platform Engineering: A Real-Life Scenario Let’s consider a real-life example to illustrate the importance of Platform Engineering. Imagine a mid-sized tech company with a growing team of developers. The company has adopted a microservices architecture, and each team is responsible for deploying their services to Kubernetes clusters in the cloud. Initially, things run smoothly. The teams are small, and the developers are well-versed in Kubernetes and the associated tooling. But as the company grows, so does the complexity. New developers join the team, each with varying levels of experience with Kubernetes and DevOps practices. Some are more comfortable working with infrastructure as code, while others struggle with the steep learning curve. As a result, the teams start to experience friction. Deployments take longer, bugs slip through the cracks, and the overall productivity of the development team begins to suffer. Developers are spending more time managing infrastructure than writing code, and the velocity of the entire organization slows down. This is where Platform Engineering comes into play. Recognizing the need for a more streamlined approach, the company decides to invest in building an internal developer platform (IDP). This platform abstracts away the complexity of Kubernetes and provides developers with a self-service interface for deploying their services. Instead of writing custom scripts and YAML files, developers can now use a simple command-line tool or web interface to deploy their services, monitor their applications, and manage their infrastructure. The impact is immediate. Developers are no longer bogged down by the intricacies of Kubernetes, and they can focus on writing and deploying code. The time to deploy new features is drastically reduced, and the overall productivity of the team improves. The platform also enforces best practices, ensuring that all deployments follow a consistent process, reducing the risk of errors and improving the reliability of the services. ## The Components of a Successful Platform So, what makes a successful platform? Here are some key components: 1. **Self-Service Interfaces**: The platform should provide easy-to-use interfaces, whether through a web portal, CLI, or API, that allow developers to deploy, monitor, and manage their applications without needing to understand the underlying infrastructure. 2. **Automation**: Automation is at the heart of Platform Engineering. By automating repetitive tasks such as provisioning infrastructure, configuring services, and managing deployments, the platform reduces the cognitive load on developers and ensures consistency across the organization. 3. **Scalability**: The platform must be designed to scale with the needs of the organization. As the number of developers and services grows, the platform should be able to handle the increased load without becoming a bottleneck. 4. **Observability**: Providing developers with visibility into their applications is critical. The platform should include monitoring, logging, and alerting capabilities that allow developers to quickly identify and resolve issues. 5. **Security and Compliance**: A well-designed platform enforces security best practices and compliance requirements, ensuring that all deployments meet the necessary standards. ## Challenges and Considerations While Platform Engineering offers many benefits, it's not without its challenges. Building such a platform requires expertise in both software development and infrastructure. It also takes a very long time if you do it yourself from scratch. There's also the risk of creating a platform that is too rigid, limiting developers' ability to innovate or forcing them to work within constraints that don't align with their needs. That's where internal platform automation tools such as Skyhook come in. ## The Future of DevOps and Platform Engineering In many ways, Platform Engineering represents the next step in the evolution of DevOps. By providing developers with the tools and services they need to build and deploy applications efficiently, Platform Engineering is helping organizations overcome the challenges of modern software development and paving the way for the future of DevOps. ## Conclusion Platform Engineering is more than just a trend; it’s a strategic approach to overcoming the challenges of modern software development. By creating reusable tools and services that abstract away the complexity of infrastructure, Platform Engineering empowers developers to focus on delivering value to their customers. As we’ve seen from real-life examples, the benefits of Platform Engineering are significant. Companies that invest in building internal developer platforms are seeing improvements in productivity, consistency, and reliability, all of which are critical to their success in a rapidly changing landscape. Whether you’re a small startup or a large enterprise, Platform Engineering has the potential to transform your DevOps practices and help you stay competitive in today’s fast-paced world. So, if you’re not already thinking about Platform Engineering, now is the time to start. The future of DevOps is here, and it’s built on platforms. --- ### Why Kubernetes Wins: The Technical Case for Container Orchestration - **URL**: https://www.skyhook.io/blog/why-kubernetes-is-a-game-changer - **Date**: August 2024 - **Author**: Nadav Erell, CEO - **Category**: Kubernetes - **Tags**: explainer, kubernetes, beginner, for-developers **TL;DR.** Kubernetes continuously reconciles declared state with live state. That control loop enables self-healing, horizontal scaling, and controlled rolling updates, but it also introduces substantial operational complexity. It is usually worthwhile when you run several independently scaling services and need consistent deployment behavior; it is often overkill for a small team with one or two stable services. Your app runs fine on a single server until it doesn't. Then you need three servers. Then ten. Then you need them spread across availability zones. Suddenly you're writing bash scripts to track which container runs where, building health check loops, and waking up at 3am because a node died and nobody noticed. Kubernetes solves this. It's not magic - it's a declarative system that turns your infrastructure into code: you describe what you want, and Kubernetes makes it happen. ## What Kubernetes Actually Does At its core, Kubernetes is a control loop. You declare a desired state ("I want 3 replicas of my API server, each with 512MB of memory"), and Kubernetes continuously reconciles reality to match that state. If a container crashes, Kubernetes restarts it. If a node dies, Kubernetes reschedules those containers elsewhere. This isn't theoretical. Here's what a basic deployment looks like: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: api spec: replicas: 3 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: api image: ghcr.io/myorg/api:v2.1.0 resources: requests: cpu: "200m" memory: "256Mi" limits: cpu: "500m" memory: "512Mi" readinessProbe: httpGet: path: /healthz port: 8080 periodSeconds: 5 livenessProbe: httpGet: path: /livez port: 8080 periodSeconds: 10 ``` This YAML file replaces hundreds of lines of deployment scripts. Change the `replicas` field from 3 to 10, apply it, and Kubernetes spins up 7 new pods automatically. Change the image tag, and Kubernetes performs a rolling update - replacing pods one at a time so your service never goes down. ## The Real Benefits (With Specifics) ### Self-Healing Infrastructure When a container dies, Kubernetes restarts it. When a node fails, Kubernetes moves workloads elsewhere. But the key insight is *how fast* this happens. With `restartPolicy: Always` and properly configured probes, a crashed container typically restarts within 10-30 seconds. Compare this to a traditional VM setup where you might not notice a failure for 5-10 minutes (until monitoring alerts), then spend another 10-15 minutes manually restarting the service. ```yaml livenessProbe: httpGet: path: /livez port: 8080 initialDelaySeconds: 10 periodSeconds: 5 failureThreshold: 3 # After 3 failures (15 seconds), restart the container ``` ### Autoscaling That Works Kubernetes Horizontal Pod Autoscaler (HPA) watches metrics and adjusts replica counts automatically: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: api spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: api minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ``` When CPU hits 70% utilization, Kubernetes adds pods. When traffic drops, it scales down. This isn't a cron job checking metrics every 5 minutes -HPA evaluates every 15 seconds by default and can scale up by 100% of current replicas every 15 seconds during traffic spikes. ### Actual Portability "Run anywhere" is often marketing speak. With Kubernetes, it's closer to reality - but with caveats. Your Deployment, Service, and ConfigMap YAML files work identically on: - Amazon EKS - Google GKE - Azure AKS - Self-managed clusters on bare metal What *doesn't* transfer cleanly: LoadBalancer services (each cloud has different annotations), storage classes, IAM integrations, and cloud-specific features like AWS ALB Ingress Controller. The honest answer: you can move between clouds, but budget 2-4 weeks of engineering work to adapt cloud-specific integrations. That's still better than rewriting your entire deployment system. ### Resource Efficiency Kubernetes bin-packs containers onto nodes based on requested resources. A node with 8GB RAM can run multiple containers that each request 512MB, rather than dedicating entire VMs to each service. Real-world example: a team running 15 microservices moved from 15 t3.medium EC2 instances (one per service) to a 3-node Kubernetes cluster using t3.xlarge instances. Monthly compute costs dropped from ~$750 to ~$300 - a 60% reduction - while gaining self-healing, autoscaling, and declarative deployments. ## What Makes Kubernetes Different From Alternatives ### vs. Docker Swarm Docker Swarm is simpler to set up. You can have a cluster running in 15 minutes. But: - No built-in support for custom resource definitions (CRDs) - you can't extend the API - Limited ecosystem - no Helm charts, no operators, no Argo CD - Smaller community means fewer battle-tested patterns Swarm works for small, static deployments. Kubernetes wins when you need to grow. ### vs. Amazon ECS ECS is deeply integrated with AWS. If you're all-in on AWS and never plan to leave, ECS is simpler for basic use cases. But: - No portability -ECS task definitions don't run anywhere else - Weaker ecosystem - no equivalent to Helm, operators, or the CNCF landscape - Less community knowledge - harder to hire, fewer Stack Overflow answers ### vs. HashiCorp Nomad Nomad is lightweight and supports non-container workloads (VMs, Java JARs, binaries). It's a solid choice for mixed workloads. But: - Smaller ecosystem than Kubernetes - Fewer managed offerings (you'll likely run it yourself) - Less momentum in the industry ## The Honest Tradeoffs Kubernetes isn't free. Here's what you're signing up for: ### Complexity A minimal production Kubernetes setup requires: - The cluster itself (managed or self-hosted) - An ingress controller for routing traffic - cert-manager for TLS certificates - A monitoring stack (Prometheus + Grafana or similar) - Log aggregation - Secret management That's 5-6 systems to understand, configure, and maintain before you deploy your first app. ### Learning Curve Kubernetes has its own vocabulary: Pods, Deployments, Services, Ingress, ConfigMaps, Secrets, PersistentVolumeClaims, StatefulSets, DaemonSets, Jobs, CronJobs, ServiceAccounts, RBAC... Expect 2-4 weeks for a developer to become productive, and 2-3 months for someone to become truly proficient. ### Operational Overhead Even with managed Kubernetes (EKS, GKE, AKS), you're responsible for: - Keeping node images updated - Managing cluster upgrades (Kubernetes releases every 4 months) - Monitoring cluster health and resource usage - Debugging networking issues (and there will be networking issues) Small teams without dedicated DevOps often underestimate this. A team of 5-10 engineers might spend 20-30% of one engineer's time on Kubernetes operations. ## When Kubernetes Makes Sense Kubernetes is worth it when: - You're running 5+ services that need to scale independently - You need self-healing and automated rollouts - You want consistent deployments across environments - You're planning for growth (headcount or traffic) Kubernetes is overkill when: - You have 1-2 services with stable traffic - Your team is small (< 5 engineers) with no DevOps capacity - You're still validating product-market fit ## Reducing the Complexity The Kubernetes learning curve is real, but it doesn't have to block your team. Platforms like Skyhook abstract the operational complexity - handling ingress, TLS, monitoring, and deployments - while keeping your workloads on standard Kubernetes. Your team writes the same Deployment YAML, but you skip the 2-3 months of infrastructure setup. The goal isn't to hide Kubernetes. It's to get the benefits (self-healing, autoscaling, declarative deployments) without drowning in YAML files for cert-manager, ingress-nginx, and Prometheus. --- ### The Birth of Skyhook: Our Journey to Simplify DevOps - **URL**: https://www.skyhook.io/blog/the-birth-of-skyhook-our-journey-to-simplify-devops - **Date**: June 2024 - **Author**: Nadav Erell, CEO - **Category**: DevOps - **Tags**: company, product, origin-story, devops, for-engineering-leaders In a world increasingly driven by software, the complexity behind the scenes often goes unnoticed. But for those of us who live and breathe development, we know that building, deploying, and maintaining software isn't always as smooth as it should be. It was this realization, borne out of our own experiences, that led to the creation of Skyhook. ## A Tale of Three Techies Back when Nadav (Skyhook CEO) was at Google Cloud, he didn't think much about the tools he had at his disposal. Every day, he'd get to work, grab a coffee, and jump straight into writing code. What he didn't realize at the time was just how much of a luxury it was to have all of the infrastructure set up and working like clockwork. It wasn't until he started talking to friends at other companies that he saw just how different things could be. Eyal (Skyhook CTO), who was leading architecture at OneZero Digital Bank and eToro, had a very different experience. He spent years trying to build something like what Nadav had at Google - a smooth, efficient devops process that didn't feel like a constant uphill battle. But even after years of effort, the job was far from done. It was a tough, sometimes frustrating process, piecing together tools and systems to create something that worked. Meanwhile, Roy was at Armis, a cybersecurity company, dealing with the same challenges. As a developer turned platform PM, he knew firsthand how tough it was to build a reliable internal platform from the ground up. It took years, countless hours, and no small amount of frustration. ## The Realization After countless conversations with companies of all shapes and sizes, we started to notice a pattern. Building an efficient developer platform wasn't just a challenge - it was a massive drain on time, resources, and morale. Usually as high as 10-20% of the engineering workforce, taking months and years to reach a stable state. We kept hearing the same thing over and over: "There has to be a better way." That's when the idea for Skyhook really started to take shape. We weren't just looking to build another tool; we wanted to create a platform that would take the pain out of DevOps, something that would let developers focus on what they do best - writing code - without getting bogged down in the complexities of deployment and infrastructure. ## Enter Skyhook Skyhook is the result of our collective experiences. We designed it to be the go-to solution for software teams struggling with deployment and development headaches. Our guiding principles for building Skyhook are what differentiates us from every other solution out there: * **Don't reinvent the wheel:** Instead of re-creating solutions for the 30+ DevOps categories of tools, we simply orchestrate the best tools for you. As if we were a set of additional DevOps/Platform engineers on your team. * **Flexibility and Control:** Skyhook is installed on our customer's cloud, and all configurations are stored within our customer's code repository. Allowing full flexibility and control for our customers to make direct changes. * **No Lock-in:** Retain full control with no dependencies on specific cloud providers or third-party tools, including us\! * **Simplicity:** Allowing anyone to get started within minutes, not days, with our fully configured environment. Without having to be a DevOps expert. * **Empower Developers:** Leverage a self-service platform that allows any developer to use it from day one. * **Designed to Scale:** Our platform grows with your needs, from startup to enterprise. At the end of the day, Skyhook is about making life easier for developers and DevOps engineers. We're here to help you get your job done without all the extra hassle. Give Skyhook a try, we think you'll like it.