Terraform Design Patterns: the Terrafile
Special thanks to my colleague Efstathios Xagoraris, who laid the original foundations for this concept, as well as the rest of the team at ITV for their valued input.
Taken straight from the documentation, Terraform
provides a common configuration to launch infrastructure — from physical and virtual servers to email and DNS providers. Once launched, Terraform safely and efficiently changes infrastructure as the configuration is evolved.
Straightforward so far.
However, what immediately struck me (coming from a development background) was the way that modules were referenced - i.e. specifying the module
the body of the module implementation.
Combining the module source with its use feels like a mix of concerns to me.
Additionally, each time you reference a module you must specify its source - even if you used that module elsewhere in your project.
I believe that abstracting the
source location to another file (separate from the implementation) would make much more sense.
Before we cover how we might achieve that, let’s quickly cover Terraform module versioning.
It is also possible (and good practice) to tack on a version at the end of the
Specifying the version (e.g. a git tag) is great as it means multiple teams can contribute to the same Terraform modules without breaking functionality for others.
However, upgrading even a single module in a project can be quite a laborious and manual process. Consider a setup with dozens or even
hundreds of ASGs (autoscale groups), spread across numerous
.tf files and various environments (QA, SIT, Stage, Prod etc.) with each
using a Terraform module to implement said ASGs.
Any non-trivial project will require many other modules e.g. Security Groups, VPCs, subnets, Route53, EBS etc. - suddenly you have a lot of things to change!
The combination of a mix of concerns with the module source and implementation with a potentially laborious and error prone module upgrade process resulted
in the creation of a
Terrafile to address these issues.
Terrafile is simple YAML config that gives you a single, convenient location that lists all your external module dependencies.
The idea is modelled on similar patterns in other languages - e.g. Ruby with its
Gemfile (technically provided by the
Here’s what a
Terrafile might look like.
Below is a simplistic example in Ruby/Rake of how you might implement the
Terrafile pattern. No gems are required (except Rake of course).
Simply place the code in a
Rakefile and execute using
Implementation is quick and easy. We’ve covered most of it already but let’s recap.
- Create your
- Implement a way to read your
Terrafileand fetch the required modules (working Ruby example above).
- Modify all
sourcevariables in your Terraform project to point to your new cached modules directory (provided by the
modules_pathmethod above) rather than GitHub e.g.
You can read more about Terraform module sources in the official documentation.
Other Considerations and Limitations
If you need to support multiple different versions of the same module (an incremental upgrade for instance), the Ruby/Rake implementation
above takes the
Terrafile key name into account. For example, the following will be deployed to
Additionally, the deletion and subsequent fetching of the Terrafile modules is very simplistic. Each time
rake get_modules is executed, all cached
modules are removed and re-fetched.
It feels repetitive and prone to error to keep specifying modules and their version information, especially for larger teams who share modules. Terraform is rapidly evolving - to keep up you must frequently update your modules.
Probably my favourite aspect of using a
Terrafile is being to see at a glance exactly which modules and which versions are being used
by a project, just like a
Gemfile or a
Puppetfile. Outdated or buggy dependencies are often the root cause of runtime issues and
this shortens the debugging and investigation considerably when things go wrong.
I’m a huge fan of Terraform and other Hashicorp products (especially Consul and Vagrant). I hope this design pattern helps others to use Terraform even more productively.