A Guide to Git with Trunk Based Development

Learn the mindset and process behind Trunk Based Development

Through my consulting business, I equip technology organizations with the latest and greatest in DevOps tooling to increase the velocity of their development teams.

Part of this means abandoning outdated workflows like “GitFlow” and replacing them with more streamlined processes — Trunk Based Development and Continuous Deployment.

We’ll get more into “Trunk Based Development” soon — but it’s important for this conversation to define what “trunk” is, as I’ve seen this misused as well.

The “trunk” is the main or master branch of your repository. Trunk is a constant. In trunk based development, you either commit to the trunk branch, or make branches and pull requests against the trunk branch. There are no long lived alternative branches to merge against such as development.

I’ve seen teams create a new branch and call it the new “trunk” every few weeks. Although I applaud the effort, it was missing the mark by quite a lot.

We’ll get more into what this process looks like end-to-end later on… First, let me spend a minute backing up my claims of GitFlow being outdated, and explain the goals of moving to a different approach.

Some of the best programming advice you’ll ever receive is from Eric Evan’s required reading, Domain Driven Design. Domain Driven Design, for the uninitiated, is a methodology for modeling business domains in code. The advice is the following:

“Make implicit concepts explicit” — Eric Evans, Domain Driven Design

What exactly does this mean? Also, how does this apply to Git or GitFlow?

First, let’s define what an “implicit concept” means. An implicit concept is something that happens as a side effect of something else. For example, a side effect of committing to the development using GitFlow is often to tie that to deploying a version of that service to an environment such as staging.

The explicit action taken is committing to a branch. The implicit action caused by that action is running a deployment process. We didn’t explicitly change our staging environment — we changed our application’s codebase on a specific branch and it updated the environment as a side effect.

We then often see the same thing committing against the “trunk” (main/master) branch. The explicit action is committing to master - the implicit action is tying some sort of deployment process to this explicit action. Expand that to several repositories and now you’ve got yourself a real headache.

So, ask yourself — how can we make the currently implicit concepts of releasing new versions of our software, explicit?

The answer: by making those concepts a part of our model. The reason for the awkwardness we are feeling is because the model doesn’t actually model our environments. Our environments in this model are entirely implicit — and as you can imagine, things like production environments are quite important pieces — one that should be modeled explicitly, for sure.

To do so, we can create repositories that explicitly model each of our environments. Meaning instead of tying the implicit concept of releasing and deploying code to our commits, we can instead explicitly define the requirements of such a release and deployment in a codebase.

For example, our staging environment’s repository might look something like this:

dependencies:
- name: content
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.75
- name: mongodb-replicaset
  repository: https://kubernetes-charts.storage.googleapis.com/
  version: 3.15.1
- name: person-model
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.20
- name: www
  repository: http://bucketrepo/bucketrepo/charts/
  version: 0.0.55

What version of each application is in staging?

The answer is no longer “whatever is in the development branch for each project”. It’s 0.0.75, 3.15.1, 0.0.20, and 0.0.55. It’s explicitly defined in an environment repository.

Meanwhile your production environment might be on www version 0.0.50 and content on 0.0.74. Also explicitly defined in code.

Much better than “whatever is in trunk of each project? 🤷‍♂️”

To accomplish this we’ll need to approach things a bit differently than you’re probably used to.

We’ll still want to tie some actions to commits, but not so many different concepts — we only want to do things that are related to the codebase we are working on. Instead we can limit that action to releasing a new versioned artifact of the project we are working on.

More explicitly, this means the first commit to the trunk branch will release version 0.0.1 of that project, bundled and ready to go – finalized and immutable. The second commit to trunk will result in release 0.0.2, and the third 0.0.3, etc. We can also run some automated tests as part of that release process, so if commit four failed those tests for example, 0.0.4 would not be released until the 5th commit which makes the tests pass again.

What exactly those versioned artifacts are will vary by the type of project you are working on, but are usually Docker images, chart repositories, or various language specific packages like NPM or Maven releases.

Next, we can hook into the release process and with each new release, caused by each verified commit to Trunk, to explicitly update the staging environment

Then, once the explicit combination of versioned artifacts are verified in the running staging an environment, they can all be moved into production in a single explicit commit

So that’s pretty much modern Trunk Based Development in a nutshell.

Not so bad, right?

You can use a tool like Jenkins X to get this set up for you out of the box. Using JX can be a bit overwhelming as it is based on Kubernetes and several tools in the Kubernetes ecosystem. In my course, DevOps Bliss, I walk through setting up a system like this end to end, as well as get into other topics like SSO, load testing, and more! Check it out if you’re interested!

What exactly is in git history?

With our Trunk Based Development workflow defined, it’s worth taking a moment to explain how to think about what your project’s git history really is.

The history of your repository is a series of commits in an ordered list. To move to different spots in that history, pointers are used. These pointers are called branches and tags.

Check out this image pulled from the Git — Branches in a Nutshell documentation.

It’s helpful to remember this as you are working with Git.

This whole structure, the commits, the pointers, etc, are all copied to your machine when you clone a repo. They are not kept up to date automatically though.

You are probably used to using git pull at this point to get the changes from the origin.

Instead, I want to show you a few alternative commands to help you with your Trunk Based Development workflow.

General Workflow

In this next section I want to capture the “day in the life” decision making and git workflows you’ll need to employ when working on a project that follows Trunk Based Development.

First of all, whether you are working directly against the Trunk branch (yes, this is ok), or on a branch that will be merged to the trunk branch, you’ll need to know what’s happening back on Github — information like, have other developers changed the code since I’ve last checked?

I’m assuming you are familiar with the basic git workflow of add, commit, push, and pull, and here is where you might think pull. I want to give you a more granular way of thinking about it instead.

To do that, we’ll start with a git fetch.

git fetch

As you are working away on your next fix or feature, either directly on the Trunk branch, or on a short-lived branch that will eventually be merged into the trunk branch, so are other developers.

They are pushing and pulling from the same remote on Github named origin as you are. Adding new commits to the log, and new pointers in the form of branches and tags.

The command git fetch simply fetches the information about these changes. It does not update your local copy with the remote changes, it simply makes git aware that those changes exist.

So, as you’re busy working, every once in awhile, run:

git fetch --all -p

The --all flag tells git to fetch from any remote you’ve added, origin or otherwise. The -p flag stands for prune which tells git it can cleanup deleted branch pointers and etc.

Then you’ll know if there are any updates you need to bring into your local copy.

Here is an example of what that looks like:

> git fetch --all -p
Fetching origin
From github.com:servicebus/kafkabus
 - [deleted]         (none)     -> origin/renovate/mocha-7.x
remote: Enumerating objects: 150, done.
remote: Counting objects: 100% (150/150), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 198 (delta 125), reused 97 (delta 92), pack-reused 48
Receiving objects: 100% (198/198), 362.90 KiB | 1.31 MiB/s, done.
Resolving deltas: 100% (144/144), completed with 6 local objects.
   98169c3..6b4ca35  master                  -> origin/master
 * [new branch]      renovate/commitizen-4.x -> origin/renovate/commitizen-4.x
 * [new branch]      renovate/kafkajs-1.x    -> origin/renovate/kafkajs-1.x
 * [new tag]         v2.0.1                  -> v2.0.1
 * [new tag]         v2.0.2                  -> v2.0.2
 * [new tag]         v2.0.3                  -> v2.0.3
 * [new tag]         v2.0.4                  -> v2.0.4

In response to the git fetch call, I can see that master has been updated on origin compared to my local version of git history: 98169c3..6b4ca35. There are also a few new branches and tags.

Running git log will verify I am still on the 98169c3 commit locally.

> git log
commit 98169c3fabf3052dd89fe0c6900bc3c11a0252a4 (HEAD -> master, tag: v2.0.3)
Author: Patrick Lee Scott <pat@patscott.io>
Date:   Sat May 23 13:48:00 2020 -0400
    fix: high throughput test already uses no transactions
# ... more commits ...

In the parenthesis we can see a few pointers for this commit: HEAD, master, and tag: v2.0.3.

With that information in mind, we’ll want to rebase our changes on to the version of history pointed to by origin/master, which was the latter, 6b4ca35, from the info about the fetch (98169c3..6b4ca35). That is to say, the changes in Github made it to Github first, before our changes, so our changes should be moved on top of it.

Depending on our situation, we’ll either want to use a regular, or an interactive rebase.

Working on a fix directly against the Trunk branch

If you are working on a fix that can be done easily, there is no issue, in my opinion, with committing directly against the trunk branch. I’ve worked on teams with 20 engineers doing this, and I promise you, it’s fine, so long as you have a solid CI/CD process. If this process fails, the worse that happens is nothing. The build is marked as failing, nothing is released or promoted.

The only rule is if you break the build it’s now your job to fix it, and that’s the top priority!

If you’re afraid of this, it’s ok to use a branch, and I’ll cover that workflow next.

So, when you are working directly on the trunk, you’ve fetched info from Github and there are new commits to incorporate, it’s now time to rebase.

git rebase

To rebase our changes on top of origin/master we run the command:

git rebase origin/master

This only works AFTER a fetch, otherwise, git doesn’t know the information about new commits and branches from the remote named origin. Make sense? Use fetch to get info, and rebase to then use that info.

All of the commits you’ve made after the initial commit, which is 98169c3 in this case, will be picked up, placed to the side, and then your history will be updated to match origin/master. This means your log of history will match the origin version of history - ending in 6b4ca35 in our example. Then your commits that were placed aside, will be placed on the end of the log.

> git rebase origin/master
First, rewinding head to replay your work on top of it...
Fast-forwarded master to origin/master.

Now, when I run git log you’ll see the latest commit has been updated to the latter commit identified by fetch earlier (98169c3..6b4ca35).

> git log
# ... your commits ...
commit 6b4ca353091a7d6a9eeba8ee5b1978112a81cabf (HEAD -> master, origin/master, origin/HEAD)
Author: Renovate Bot <bot@renovateapp.com>
Date:   Wed Nov 4 06:34:00 2020 +0000
    chore(deps): update dependency jest to v26.6.3

If there are no conflicts, this will be a pretty simple command.

If there are conflicts, they must be resolved first. To resolve conflicts, in your IDE, fix the conflicts, and when you are ready, save the changes, add them to git’s staging area, and then run git rebase --continue.

To learn more about rebase, make sure you read the Git’s docs - Git - Rebasing.

To learn more about resolving conflicts, I recommend learning the VSCode Merge tool: Version Control in Visual Studio Code

Working on a branch that will be merged into Trunk via a Pull Request

If you are doing something more than a quick fix, such as working on a new feature, you’ll likely want a branch to work on.

git rebase -i origin/master

The above git rebase works well when we are on master working on a single commit for a fix. When we are on a branch, preparing for a change to be merged into master via a Pull Request, it’s often expected that we make things nice and clean. Maintainers of open source projects will ask you to “squash and rebase” your commits into a single commit to be merged.

This keeps the history of the master branch clean and rollback’s easy to perform.

People will debate on the merits of doing this vs. not doing so, but let’s put all those reasons aside for now, and just focus on how to do it.

If you specify the -i flag or —interactive when you run the rebase command, instead of a regular rebase, you will be performing an “interactive rebase”.

This allows you to reword commits in your little segment of history, or squash multiple commits into a single commit.

Doing so drops you into an editor in your Terminal called vim, and for this reason, knowing the basics of VIM are required in order to perform the interactive rebase.

Here’s an example from the git documentation (Git — Rewriting History):

pick f7f3f6d Change my name a bit
pick 310154e Update README formatting and add blame
pick a5f4a0d Add cat-file
# Rebase 710f0f8..a5f4a0d onto 710f0f8
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

In order to squash all of the commits, use VIM to edit the messages that say pick to say s or squash for the 2nd and 3rd commits:

pick f7f3f6d Change my name a bit
squash 310154e Update README formatting and add blame
s a5f4a0d Add cat-file

This will result in a single commit, Change my name a bit, that consists of the changes from all three commits.

Say you wanted to change the commit message as well, instead you might edit the commits like this:

r f7f3f6d Change my name a bit
s 310154e Update README formatting and add blame
s a5f4a0d Add cat-file

Saving this change would then prompt you with another VIM window that allows you to type a new commit message for the r or reword flag.

git push origin/<branch> —force

After a rebase, you’ve essentially modified the history of commits in the log. If you try to push these changes to origin you will get an error! This is expected.

You need to tell git that you have intended to change the history by using the —force flag. This will replace origin’s log of commits with your new rebased log of commits.

WARNING: You should not be force pushing to master, just branches!

Rebasing when you have unsaved changes

Need to rebase but have uncommitted changes?

If you try to do this, you will receive an error saying you must commit or stash your changes in order to rebase!

> git rebase origin/master
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.

If you are ready to commit them go ahead and do so. If not, you can stash them, perform the rebase, and then retrieve your changes from the stash again.

git stash

To put all of your changes aside in a temporary space, run:

git stash

Then, you can complete the rebase:

git fetch --all -p
git rebase origin/master

After the rebase is complete, you can bring back all of the changes you were working on from your stash:

git stash pop

--autostash

The workflow above is common enough that rebase actually supports a flag, --autostash that combines all the steps into one!

git rebase origin/master --autostash

Conclusion

Although it can be more steps than git pull, sometimes pulling results in unexpected merge commits - using fetch, stash and rebase gives you more explicit control when working with git, which is very helpful when working with Trunk Based Development!

Hopefully you understand the benefits of making our previously implicit operations explicit and how you can use a few new git commands to ease that process!

If you’re looking to implement Continuous Deployment in your organization and are looking for help, reach out to me through my website patscott.io. If you’re a DIY-type of engineer like me, then check out my course DevOps Bliss instead!