> This means the Service must be created first, then when the corresponding NEG is created the name can be queried and added to the Terraform project.
These kind of reverse dependencies, where app level changes have to be reflected in infrastructure are source of endless bugs and headaches.
To do it right, you'd want to encode app and infra changes in the same ubiquitous tool, but "infrastructure as a code" tools such as terraform suck big times at app level deploys.
Pulumi takes steps in right direction, where it is actually not that painfull to manage everything, that is what I ended up using for very similar problem configuring GCPs load balancers for my k8s apps.
> To do it right, you'd want to encode app and infra changes in the same ubiquitous tool
Well, that's exactly what we do with Dhall, which lets us put static types on infrastructure tooling that prints out unpredictable output but in a predictable format and transform that output into the format that other dependent tools expect. Everything is then glued together with simple shell scripts and run in a CI environment.
Once you understand the pattern - use an uber-config plus typed tool outputs to generate downstream config and apply all config in an idempotent way - you wonder why you ever did things any other way.
I looked at Dhall before. It is a pure language for static configs, right? It can't help me to express "deploy k8s resource, wait for annotation to appear, take annotation value and use it to deploy infra resource". Wrapping multiple tools in a bash is exact thing I wish I didn't have to do :(
If Dhall enables safer bash glue, I'd be happy to read how
We use terraform for our kubernetes app configuration and it works well. Being able to pass values from one resource to another is extremely powerful. For example, the database instance credentials down into cloudsql proxy.
The only and most frustrating missing piece of the terraform kubernetes provider is CRDs. I hacked around it by using some yaml provider bit I would like a cleaner solution.
TF kubernetes provider works for simplest cases only. How would you wait for PV to be created, when your TF resources specify PVC? That is to help users to diagnose problems with deployments.
No tools IaC tools I know of including Pulumi, allow to describe evolutions for infra. Simple use case - deploy canary instance, check it's health, scale down canary instance, update main pool of instances.
In other words, there is no way to express sequence of changes of the same object in the single run of IaC tool. We are forced to wrap IaC tools in layers of bash to simulate it.
These kind of reverse dependencies, where app level changes have to be reflected in infrastructure are source of endless bugs and headaches.
To do it right, you'd want to encode app and infra changes in the same ubiquitous tool, but "infrastructure as a code" tools such as terraform suck big times at app level deploys.
Pulumi takes steps in right direction, where it is actually not that painfull to manage everything, that is what I ended up using for very similar problem configuring GCPs load balancers for my k8s apps.