Automating Crossplane with GitLab

This is a continuation from my previous post on Crossplane

Explain the popsicle images already!

Crossplane uses a popsicle icon. Why, you may ask? As Crossplane explains in their FAQ:

We believe in a multi-flavor cloud.

Which, they readily do - even my repo reaches out to both AWS and Azure for resources!

Automating with GitLab

I've done a post on my use of GitLab (on becoming a Rockstar Programmer) and my personal website links to the GitLab repo that builds it. This is not to say that other source code management tools are inferior - I use GitHub nearly daily and I have a good colleague that swears by Gitea. GitLab's CI/CD platform is my preference for such a tool, but there are many other good choices out there (though my disdain of Groovy makes Jenkins personally challenging to me).

What GitLab offers that I like is twofold - it offers a SCM tightly integrated to its CI/CD platform and it uses yaml. In fact a core part of the CI/CD system requires yaml in the .gitlab-ci.yml file. For my Crossplane demo, my automation file looks something like this:

include:
  - template: Jobs/SAST-IaC.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml

stages:
  - validate
  - test

yamllint-components:
  stage: validate
  image: registry.gitlab.com/pipeline-components/yamllint:latest
  script:
    - yamllint ./crossplane

yamllint-config:
  stage: validate
  image: registry.gitlab.com/pipeline-components/yamllint:latest
  script:
    - yamllint ./crossplane_configure

yamllint-install:
  stage: validate
  image: registry.gitlab.com/pipeline-components/yamllint:latest
  script:
    - yamllint ./crossplane_install

This creates two stages that will run, a validate stage and a test stage.

During each of these stages, it will run any scripts listed. You will see that there are three jobs that all start with yamllint- and have a stage: validate in them. They, however, do not have a dependence on each other, so they all run at the same time. By the image they are each pulling, yamllint one might assume that this is linting my yaml - and that is correct. All three of these jobs execute during the CI/CD pipeline from this yaml

However, you may ask, there's not a line that specifically says test, so is it just extra? It is not! This is leveraging the magic of templates.

A note on secrets in git

A secret put into a git repo is a terrible thing, and the more public the repo is, the worse the outcome.

Imagine, if you will, that your AWS Access Key and Secret are committed and pushed to your repo. Someone with malicious intent could steal those, spin up some EC2 instances to mine for crypto, and you're stuck paying a $50,000 bill before you figure it out. This is an actual story from a former client of mine that did not appreciate how that panned out and the fall out from it was intense - both in the monitoring failure and the secret exposure. However, they were able to burn the keys, issue new ones, and stop the EC2 instances after about 3 days.

Imagine something worse gets committed to your repository, like an internal system password that is prevalent across your entire product. Even if you then do a brand new push without the secrets, your git history will still have it. Undoing your git history with that secret is a very messy operation, especially if it's an open project where someone might have your un-fixed git history handy. Now you not only have to do surgery on your entire repository, you also have to change this password across several systems, and if you don't have a rotation policy (or don't follow it), you're easily having teams burn a hundred or more development hours on chasing down one small mistake.

As a practice, never, ever commit anything secret to your repository, even personal ones. This is a great reason to put certain things into your .gitignore such as config.py files or a terraform *.tfstate file. In fact, whatever repository you have open locally, go check on the .gitignore file right now and check that you have something there to catch secrets. As someone who has lived through this, thank you for doing this.

Templates in Git

Now that I have that out of my system, let's talk about the two templates in my .gitlab-ci.yml

include:
  - template: Jobs/SAST-IaC.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml

The first one, Jobs/SAST-IaC.gitlab-ci.yml, grabs a preconfigured test provided by GitLab to do Static Application Security Testing (SAST) on Infrastructure-as-Code (IaC) applications. SAST checks your code against known vulnerabilities using the KICS tool. Because it's preconfigured, it will run during the test phase automatically. It can be extended, but out-of-the-box it will scan many common IaC tools, including Kubernetes and Terraform. This helps prevent from making innocent mistakes that can compromise your code, and is an easy line to add into your GitLab CI/CD file.

The second one, the Security/Secret-Detection.gitlab-ci.yml template is a secondary check (after my local check) for secrets. This is provided from GitLab as a preconfigured test using gitleaks that runs during the test phase when included.

Since I have validate before test in my gitlab-ci.yml these will run automatically when my pipeline triggers, which is on each push.

You can even dig in to see if there were secrets or issues linting.

I definitely used the linter to better follow practices. From a previous failure -

I realized I needed to clean up my yaml. Goes to show why I don't write code first thing in the morning, really.

So, you can see I automate how all my code checks happen before the integration of it - that's Continuous Integration (the CI of CD) but how does it actually deploy?

GitLab Kubernetes Agents

GitLab is awesome in that it can use Agents with your Kubernetes clusters. In the Infrastructure menu on the left-hand side of GitLab, you can see your Kubernetes options

By following their rather thorough directions, you can install an agent into your cluster that does functions for you. Please be sure to go step-by-step - omitting steps can really mess up everything and cause you start over. Eventually, you will get to writing a .gitlab/agents file, and this is where I get to admit something.

I lied to you.

CI/CD means that, for deployment, you deploy from your repo and push your code to your environment. The continuous deployment causes your code to be bundled, whatever processes applied, and deployed to your system.

GitOps however is a pull-based system. As soon as something is present, a monitor picks it up and applies it. It requires fewer steps but it means that your git repository is the source of truth and your agent will reconcile it on whatever parameter you put it. My agent file (called gitlab/agent/do-crossplane) reads as such

gitops:
  manifest_projects:
  - id: mjh/crossplane-do-demo
    default_namespace: crossplane-system
    paths:
    - glob: '/crossplane/*.yaml'
    reconcile_timeout: 3600s

What this does is create a GitOps scenario where it picks up from my repository anything in the /crossplane that is a yaml file and deploys it, checking every 60 minutes. If it is out of sync, at all, it reapplies it. By default, it will only do this on the main branch, but there are many configurations you can apply to do this on different branches.

How this works is that if I manually nuke my crossplane/norway_aks.yaml locally (with something like kubectl delete -f norway_aks.yaml), but don't remove it from my repository, every hour the GitLab agent will detect the change (commonly called drift) and fix it. This is what is meant by your git repository being your source of truth.

Between the agent and the gitlab-ci.yml I have a way to test and deploy resources entirely automatically. By writing a new yaml file, committing it to a branch, waiting for tests, and then merge it into main it does all the testing and validation and deploys the code within the hour. I can have an entire new AKS cluster simply from using by IDE - I even wrote this on my iPad using gitpod which is a surreal way to deploy an entire Kubernetes cluster using a browser window.

Back to the top

So, you can see I keep my infrastructure pretty clean and and simple, but there are many ways you can go with this. You can get more info on the magic of how Crossplane conquered CRD issues directly from the maintainers , there's a myriad of examples, or if you find something interesting or novel, feel free to reach out!