Kustomize and Helm together

Introduction

In my opinion, although Helm charts tend to become increasingly complex and difficult to maintain over time, its deployment experience is quite straightforward, much like installing a Linux package. So I prefer to leverage well packaged Helm charts in Kustomize project. In this article, I will share my experience on this.

Project Layout

The project layout is crucial, considering both the patching pattern of Kustomize and the varied deployment environments in GitOps. I recommend the layout in GitOps Folder Structure.

|-- bootstrap
|   `-- argocd
|       |-- argoApp
|       `-- tls
|-- clusters
|-- components
|   |-- apps
|   |   |-- microservices
|   |   |    `-- serviceA
|   |   `-- middlewares
|   |        |-- rabbitmq
|   |        `-- redis
|   |
|-- helmrepo
|   |-- rabbitmq
|   |    `-- charts
|   |-- redis
|        `-- charts
|
|-- environments
|   |-- dev
|   |   |-- configs
|   |   |   |-- rabbitmq.yaml
|   |   |   `-- redis.yaml
|   |   |
|   |   `-- overlays
|   |       |-- rabbitmq    
|   |       `-- redis
|   `-- prod
|
|--kustomization.yaml

Here is the briefing:

• bootstrap - GitOps related configurations, including:
  • Declarative configuration files for GitOps tool (Argo CD in my project)
  • Declarative configuration files for application
• Cluster - Kubernetes related configurations.
• Components - Application base, including both microservices and middwares.
  • In order to test the default configuration in base level, kustomization.yaml files are put in all the microservices and middwares.
  • helmrepo folder stores file based Helm charts. It is not necessary when using online helm repos.
• Environments - Application overlays for different environments.
  • I just put some patch/transformer files here and put the kustomization.yaml file to the root folder. In this way, I don't need to configure --load-restrictor switch.
  • I put the environment specific helm value files in configs folder.

Helm Chart overlay

Kustomize provides helmCharts field which allows to integrate Helm chart directly. HelmChartInflationGenerator is the generator for helmCharts field.

Kustomize 5.0 and later versions provide additionalValuesFiles settings which is perfect for patching with Helm value files. Argo CD supports using multiple Kustomize versions simultaneously and specifies required version per application. For example:

# argocd-cm configmap
kustomize.path.v5.3.0: /custom-tools/kustomize_5_3_0

Below is the code to copy kustomize to custom tools volume. For more information, please refer to Argo CD document

# patching argo-cd-argocd-repo-server deployment
spec:
  template:
    spec:
      volumes:
        - name: custom-tools
          emptyDir: {}
      initContainers:
        - name: get-tools
          image: alpine:3.19.0
          command: [sh, -c]
          args:
            - wget ftp://<url-to-kustomize>/kustomize -O /custom-tools/kustomize_5_3_0 && chmod +x /custom-tools/kustomize_5_3_0 # need to update the url for wget kustomize
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      containers:
        - name: repo-server
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools

Important Note:

The above covers the main aspects of managing Helm with Kustomize, but there is one crucial point to note: Settings in Kustomize do not sync with Helm values. This may cause problem in Helm deployment.

When you add a prefix in Argo CD application Declarative configuration file or transformed namespace with Kustomize, Helm has no idea about these changes and will continue to generate settings based on it's templates and value files.

For example, when Argo CD deploys Helm chart according to the Application CRD in "argocd" namespace, the value .Release.Namespace in Helm chart is "argocd". If the namespace is transformed to "dev" in Kustomize, readiness probe defined in Helm chart might failed because it cannot resolve dns name with argocd.svc.cluster.local suffix. To fix the problem, the namespace of Helm chart should be configured according to the namespace defined in transformer.

So when pod cannot start after deploying helm with Kustomize, the following troubleshooting steps can be tried first:

1. Deploy the helm chart with Helm command to isolate the problem with Helm chart and environment.
2. Enable image.debug setting and then check the log in the pod. If there is any nxdomain (non-existing domain) error, the problem is most likely regarding namespace or pre-fix settings.

That's it.

And here is the sample code.