Inner Sourced Infrastructure at FireStart: Terraform, GitOps, and Governance in Practice – Recap of “Inner Sourced Infrastructure” by Karim Darwish (FireStart GmbH)

Introduction: From manual setups to inner-sourced, automated cloud infrastructure

In “Inner Sourced Infrastructure,” Karim Darwish (FireStart GmbH) walks through a candid, technical journey of moving a SaaS platform’s cloud footprint from manual configuration to an automated, versioned, and traceable operating model. The pillars are clear: infrastructure as code with Terraform, an inner-sourced Git repository for collaboration, and automated pipelines as the single path to apply changes. The payoff is not just speed; it’s reproducibility, governance, and security.

The context: FireStart runs on Azure and needs a managed Kubernetes cluster, Azure Key Vault for secrets, Azure Storage Accounts for files and data, and additional managed database resources. The central question since September 2020: How do you provision and evolve these building blocks reliably, reproducibly, and transparently—without configuration drift and without manual risk?

Karim sets the scene plainly: resources were created by hand in the Azure portal, with configurations punched into wizards, while Kubernetes resources were applied manually from the command line. It all worked—until it didn’t. The underlying insight: when your infrastructure is business-critical, it deserves the discipline you apply to software code.

The starting point (2020): Manual can be fast—until it breaks

Initially, everything was created and configured manually. The team spun up the Kubernetes cluster in the Azure portal, tuned settings directly in the wizard, and applied Kubernetes configs from the CLI. It worked—until a deployment overrode a manual setting and services stumbled. Parts of the cluster had to be recreated.

That incident exposed the pain points:

• Heavy manual configuration: Each change required hands-on steps in Azure and in the cluster.
• No reproducibility: Spinning up a fresh cluster or new stage “on demand” wasn’t possible; every piece would need to be recreated manually.
• Poor visibility into actual state: Without a canonical, versioned source of truth, it was unclear which settings were really live. Drift between desired and actual configuration could happen unnoticed.

Conclusion: something had to change. The team needed infrastructure to behave like code—versioned, reviewable, and reproducible.

The inflection point: Infrastructure as Code with Terraform

The team adopted Infrastructure as Code using Terraform. Instead of clicking through the portal, infrastructure is declared in code. Terraform then uses Azure’s APIs to materialize the desired state.

Karim’s rationale fits many engineering teams:

• Resources defined in code are version-controlled and auditable. Changes are explicit and traceable.
• Software development practices apply: Module boundaries for related resources, refactorings to decouple tight coupling, and cleaner abstractions over time.
• Developers without deep DevOps/SRE expertise can contribute more easily because infrastructure is edited and reviewed like software.

The improvement was immediate: creating a new cluster became a matter of “one command,” letting Terraform orchestrate the API calls and provision the environment. As Karim put it, “that was our dream.”

The reality check: “We broke it”

The first Terraform phase came with a critical flaw: applies were run locally on developer machines. That’s where a familiar problem surfaced. While Terraform was updating one cluster, someone switched the local Kubernetes context to a different cluster. Terraform proceeded to apply changes to the wrong target, damaging the second cluster. Services needed to be restored. As Karim notes, this was “not a great time.”

The lesson is straightforward: even excellent IaC definitions won’t save you if the execution path is uncontrolled. Local states and contexts are a risk surface.

The fix: Automated pipelines and a GitOps-style flow

The response was decisive: no more local applies. Every Terraform step now runs in automated pipelines. Infrastructure changes no longer happen ad hoc on individual laptops; they flow exclusively through pull requests in a central repository—following a GitOps-style process.

Concretely:

1. All changes happen in Git: Engineers modify Terraform configurations in the inner-source repository, then open pull requests (PRs).
2. Automatic validation: A pipeline validates the proposed change. Terraform provides a plan diff that shows exactly what resources will change.
3. Review and approval: Authors and reviewers can clearly see the blast radius and intended changes. Only after approval and merge does anything get applied.
4. Automatic apply: A pipeline applies changes to the real infrastructure. No manual execution is needed—or allowed—as a matter of process.

This flow delivers clarity, reviewability, and transparency while eliminating the brittle, error-prone nature of local runs.

Inner Source: Open by default (internally), governed by process

“Inner source” here means the infrastructure repository is open inside the company. Everyone can see which resources and configurations the SaaS product relies on. Anyone from any team can contribute: fix bugs in infrastructure and add the resources required to deliver product features—all through the same PR pipeline.

Importantly, openness doesn’t mean chaos. Control is enforced by the process—PRs, reviews, Terraform plan diffs, and the pipeline as the only write path. Inner source becomes a way to combine transparency and participation with governance.

It’s also a cultural shift: infrastructure isn’t a siloed responsibility held by a few ops specialists. It’s team code, handled like software.

Governance as a feature: Least privilege and full traceability

A pipeline-first posture reshapes access control. If nobody needs to write manually to Azure or the cluster anymore, you can remove write access broadly and let the pipeline be the only entity with write permissions. That is least privilege in action: grant minimal access, to as few identities as possible.

In parallel, the PR workflow mandates transparency:

• Every change is in Git: who changed what and when is recorded by default.
• Rollbacks are natural: merges, reverts, and clear diffs are part of the history.
• Risk goes down: without spontaneous production edits, drift is less likely, and if it happens, the code baseline remains authoritative.

This governance posture didn’t exist before; it emerges naturally once you treat infrastructure like software.

“One command” remains—properly orchestrated

The “one command” vision didn’t vanish; it matured. Pipelines orchestrate Terraform with consistent runtime environments and deterministic execution. Operationally, new clusters materialize faster, and the team spends more time building product and less time manually configuring environments.

DevSecOps in view: Security scanning as a quality gate

The next iteration is to integrate security scanning into the PR flow. A security tool checks proposed changes for vulnerabilities and acts as a quality gate before anything reaches production. Security becomes part of the inner-source/GitOps flow—built in, not bolted on.

The session avoids naming specific tools. The governing idea is the same: test for security early in the lifecycle, in the same pipeline that validates and applies infrastructure.

Engineering takeaways you can apply

From “Inner Sourced Infrastructure” by Karim Darwish (FireStart GmbH), several practical principles emerge that generalize to most cloud setups:

• Treat infrastructure as code: Without declarative, version-controlled definitions, reproducibility is hard and drift is likely.
• Make Terraform plan diffs visible: Clear diffs enable meaningful reviews and reveal unintended side effects.
• Ban local applies: Keep the execution path stable and controlled. Pipelines eliminate local context hazards.
• Establish a GitOps workflow: Changes via PRs only; merge triggers apply. Governance, auditability, and collaboration become inherent to the flow.
• Use inner source internally: Open the repository so teams can fix issues and add necessary resources without bottlenecks.
• Enforce least privilege: Remove manual write access; let the pipeline be the single writer with minimal permissions.
• Integrate security into PRs: DevSecOps as a quality gate—find vulnerabilities before they ship.
• Think modularly: Use Terraform modules for related resources and recurring patterns to improve maintainability and reuse.

What measurably improves with IaC + inner source + pipelines

Even without code snippets or vendor specifics, the session makes the benefits clear:

• Speed: New stages and clusters spin up faster. No queues for “someone to click in the portal.”
• Reliability: Code is the truth. Manual drift is less frequent, and when it occurs, it’s discoverable.
• Shared ownership: Infrastructure becomes a team effort. Knowledge silos shrink.
• Quality and security: Reviews, diffs, and quality gates shift assurance left—where it’s most effective.

Or, in Karim’s own framing: the dream of a one-command cluster is real—once you orchestrate it properly and stop running applies on laptops. “We broke it” isn’t a failure; it’s the pivotal lesson that justifies a robust process backbone.

Practical steps mirrored from the FireStart journey

Based on the talk, here’s a pragmatic sequence other teams can follow:

1. Inventory and target state: List all required resources—cluster, secrets, storage, databases—and make them explicit.
2. Introduce Terraform: Capture core building blocks in code and version-control them from day one.
3. Modularize: Group related resources into modules; decouple tightly coupled parts as they emerge.
4. Pipeline over laptop: Run Terraform plan and apply only in CI/CD pipelines. Keep environments consistent and reproducible.
5. GitOps: Enforce PRs, automatic validation, plan diffs, defined reviewers. Merge triggers the apply.
6. Access model: Remove manual write access; grant minimal rights to pipeline identities.
7. Transparency and onboarding: Document the inner-source repo, set contribution guidelines, and communicate standards.
8. Security checks: Add PR security scans as a gate before changes go live.

This sequence mirrors FireStart’s evolution—from manual setup to Terraform, then to a fully governed, automated process.

Quotes and moments that stick

A few lines capture the spirit of the shift:

• “That was our dream” — reproducible stages with minimal friction.
• “We broke it” — the honest admission that local execution can be dangerous.
• “Not a great time” — the cost of unintended side effects.
• “Inner source” — internal openness that combines responsibility with speed.
• “Least privilege” — stripping write access once the process automates change.

And with some humor: “If you also want to break some clusters with us…”—a nod to learning through real-world incidents.

Conclusion: A scalable infra process is code, PRs, and a single write path

“Inner Sourced Infrastructure” by Karim Darwish (FireStart GmbH) amounts to a clear, experience-backed blueprint:

• Define infrastructure as code,
• funnel changes through PRs,
• execute in pipelines,
• enforce governance by process,
• and integrate security as a built-in quality step.

That combination yields a resilient platform that can spin up new stages quickly while lowering the risks of drift and manual errors. For teams feeling similar pain, the core message is simple and powerful: treat your infrastructure like software—and build a path where only the pipeline writes.

Follow that path, and the “dream” doesn’t just come true—it stays operable under real-world pressure.

• 

  

  Modernize your legacy web applications using Web Components

  Andreas Hässler von FireStart berichtet in seinem devjobs.at TechTalk über den Weg, wie das Team ihre legacy Applikation mithilfe von Web Components umgebaut hat – und sie sich dabei gleich mehrere Monate an Arbeit erspart haben.

  Watch now