Production GitOps: App of Apps & Best Practices
From one app to dozens across environments β cleanly and safely.
One Application is easy. Real organizations run dozens across dev, staging, and prod. This lesson covers the App of Apps pattern, how to structure your Git repos, and the practices that keep GitOps sane at scale.
π― Learning Objectives
- Understand the App of Apps pattern
- Structure repositories for multiple environments
- Separate config from code with overlays
- Apply production GitOps best practices
The App of Apps Pattern
Instead of creating each Application by hand, you create one parent Application that itself points at a folder of child Application manifests. Argo CD then manages the children automatically.
root-app (Application)
ββ points at git path "apps/"
ββ app-frontend.yaml (Application)
ββ app-backend.yaml (Application)
ββ app-monitoring.yaml (Application)
ββ app-argocd.yaml (Application) β even Argo CD config itselfNow onboarding a new service is a single commit β add its Application YAML to apps/ and the root app syncs it into existence. This is how teams manage fleets of apps from one Git repo.
Structuring Your Repos
A common, clean layout separates what the app is from how each environment configures it, using Kustomize overlays:
my-app/
βββ base/ # shared manifests
β βββ deployment.yaml
β βββ service.yaml
β βββ kustomization.yaml
βββ overlays/
βββ dev/
β βββ kustomization.yaml # e.g. 1 replica, debug on
βββ staging/
β βββ kustomization.yaml
βββ prod/
βββ kustomization.yaml # e.g. 5 replicas, resource limitsEach environment gets its own Argo CD Application pointing at the matching overlay path. The base stays DRY; overlays hold only the differences.
π‘ Config repo vs. app repo
A widely used practice is to keep application source code in one repo and the deployment manifests in a separate βconfigβ repo. CI builds the image and bumps the image tag in the config repo; Argo CD sees the commit and deploys. This cleanly separates βbuildβ from βdeployβ.
Production Best Practices
- Everything in Git β no manual
kubectl applyin production. If itβs not in Git, it doesnβt exist. - Enable
selfHealandprunein production so drift canβt accumulate. - Protect branches β require pull-request review for the config repo; that review is your deployment approval.
- Promote by PR β moving from staging to prod is a pull request bumping a version, giving you a full audit trail.
- Use Projects (AppProjects) to restrict which repos, clusters, and namespaces a set of apps may touch β essential for multi-team clusters.
- Sync notifications β wire Argo CD notifications to Slack so the team sees deploys and failures.
β Secrets don't belong in plain Git
Never commit raw secrets. Use Sealed Secrets, External Secrets Operator, or a vault integration so the Git repo holds only encrypted or referenced secrets, not plaintext credentials.
Rollback the GitOps Way
Because Git is the source of truth, rollback is just Git:
# revert the bad change β Argo CD re-syncs to the previous state
git revert <bad-commit>
git push
# or roll an app back to a previous revision directly
argocd app rollback my-app <history-id>No special tooling, no snowflake procedure β the same workflow you already use for code.
π§ͺ Hands-on Lab
Plan a Multi-Environment Setup
You have one service that must run in dev, staging, and prod with different replica counts. Describe: (1) how youβd structure the manifests, (2) how many Argo CD Applications youβd create, and (3) how a change would promote from staging to prod.
π§ Knowledge Check
What is the App of Apps pattern?
In a GitOps workflow, how do you typically promote a change from staging to production?
πΌ Interview Preparation
How would you manage 30 microservices across three environments with Argo CD?
Summary
Youβve completed the Argo CD track: GitOps principles, installation, sync and self-healing, and scaling with App of Apps and solid production practices. You can now deploy and manage whole fleets of apps declaratively from Git. Next, we shift from deployment to debugging distributed systems with tracing in Zipkin.