Readme — roll 0.0.7

Roll - ClojureScript flavoured DevOps.

Very simply: Roll will build you a Terraform template that you can then use to provision you an AWS application environment.

Roll is about the deployment of application services, providing an opinionated description of what services will commonly need in a given environment. Roll will help provision you a secure PaaS (platform as a service) like AWS rolling blue/green infrastructure using Terraform, ClojureScript and Lumo.

Roll operates on a per environment basis, so you can have a dedicated Roll configuration file per environment. Roll also just works against Clojure maps, so you have all the flexibility of how you derive the configuration before passing it to Roll and ultimately on to Terraform.

Roll assumes that deployable application versions are accessible in S3, and asks you to provide the start-up instructions. Therefore it’s orthogonal to whether you want to use Uberjars, Docker containers, or whatever else.

Roll is in active development, everything about it - including this document - is being modified on a regular basis. Currently it is not recommended for general usage.

Requirements

Above terraform 0.10.8.

Features

Blue/Green deploy using AutoScaling groups

For each deployed version of an application, Roll uses a different AutoScaling group. These ASGs sit behind a load-balancer (Application Load Balancer) that we use to determine which ASG is 'live' at any given point.

The approach has significant advantages in that multiple application versions can be kept running at any given point, providing you flexibility on when you want to make a specific application version live, and to provide the ability to roll back quickly.

Route 53

Roll builds up a Terraform configuration so that you can easily point a Route 53 alias towards the load-balancer, that sits in front of the ASGs.

You need to manually configure the hosted zone (i.e. xxx.cloud), but with Roll you then can automatically provision the sub-aliases (i.e. yyy.xxx.cloud) and route them accordingly.

Roll works well when each service has it’s own alias sitting inside the hosted zone.

Security and Key Management

Roll is opinionated in that each environment should have it’s own KMS Encryption Key. Each provisioned service is then automatically configured to have decryption access to the key.

Using KMS Encryption keys is a great way of handling sensitive application configuration secrets (encrypted secrets can be kept in plain text, and only decrypted by the running application service).

It’s just data

Roll takes in a Clojure map as the main argument and will give you a clojure map back, see the sample (for an example of this).

Roll has three stages, allowing you to hook in and manipulate the input/output data-structures for your own needs.

1) Pre-process the Roll config.

Roll will preprocess the Roll configuration you give it. An example of pre-processing Roll deriving the most recent application version based on the contents of the S3 releases bucket, so you don’t have to hardcode the application version manually. After the pre-processing step you can hook and it manipulate the data-structure as you wish.

2 Create the Terraform data structure.

Before handing you Terraform JSON, you can operate on the Terraform Clojure data structure directly, to add additional Terraform configuration if you wish or to simple manipulate what Roll has come up with.

3) Create final TF JSON output

This the final stage, producing Terraform JSON that is ready for Terraform to work with, to provision the infrastructure.

Compatible with Aero

We recommend that Aero be used to house the Roll configuration, to give you benefits such as be able to have references across the configuration, to pull out the pieces of commonality (to have for example, a reusable load-balancer configuration).

Usage

With Lumo

You can use Roll with Lumo to generate a Terraform JSON file, which will be picked up when you subsequently execute Terraform commands in the same directory.

npm install juxt/roll
lumo -c node_modules/@juxt/roll/src -m roll.tf <roll-edn-file>

Will generate you a file called deployment.tf.json. If you name your Roll file roll.edn then the last argument can be ommitted.

With Mach

Mach is your Make.

Here is a sample Machfile configured with Roll, Terraform and some useful AWS targets:

{tfjson {product "deployment.tf.json"
         novelty (mach.core/modified-since product ["Machfile.edn" "roll.edn"])
         produce (let [config (aero.core/read-config "roll.edn" {})]
                   (-> config
                       (roll.core/preprocess)
                       (roll.core/deployment->tf)
                       (roll.core/->tf-json)))}

 mach/m2 [[aero "1.1.2"]
          [roll "0.0.7"]]
}

From the command-line

mach tfjson
mach plan
mach apply

Enjoy!

See our Edge project for an example usage of Mach and Roll.

Configuration

See the sample Roll configuration file sample-config.edn.

See also the sample output Terraform file to get a feel for what is generated.

Justification

Why not just use a PaaS, and why would you bring all these technologies together?

Let’s justify the various pieces of Roll individually:

To PaaS or not?

Using a PaaS (Platform as a Service) such as AWS Beanstalk is a great way to get started and we’ve blogged about deploying Clojure apps to Beanstalk using Docker.

Sometimes though a PaaS will not do enough for you, and you will want finer grained control. Unfortunately as soon as you step outside the safe and comfortable confines of the PaaS, you will end up battling low level cloud constructs such as AutoScaling Groups, Launch Configurations, Security Groups etc.

Almost straight away you will want to avoid the anti-pattern of configuring everything via point and click in the cloud provider UI, and you will seek to move towards Infrastructure as Code. Infrastructure as code means the infrastructure can be version controlled, and you won’t get into the messy situation of forgetting who provisioned what, when and why.

Terraform

So let’s consider the 'Infrastructure as Code' options. In AWS we have a couple of choices, and the ones we’ve investigated heavily are CloudFormation and Terraform. There is lots of resources out there detailing why Terraform is superior to CloudFormation, but for us Terraform offers a friendlier DSL with comments (comments!), and where you can review pre-execution plans prior to Terraform making the declared written infrastructure plans a reality.

TL;DR; Terraform rocks.

ClojureScipt and Mach and Lumo

So why doesn’t the story stop there? Why friends, must we continue into the lands of ClojureScript tooling running on the server?

Alas, Terraform by itself is not enough. For example we want blue/green deployments - which in our use case involves firing up new AutoScaling groups when we want a new version deployed and to leave the old one running. When we’re happy with our candidate deployment we want to effect a load-balancer change that will redirect where the Route53 domain entry is pointing to, thus making the new AutoScaling group the one that is live. We may want to leave the previously 'live' AutoScaling group hanging around - in case we need to revert back to it.

Roll achieves this by rebuilding the Terraform configuration to match a higher level declarative input which you can readily change (i.e. to deploy newer versions of an application). Roll is a DSL that sits above Terraform, providing an opinionated description of infrastructure that Terraform is used to make happen.

Roll aims to give you the full Beanstalk like PaaS experience except there’s no PaaS and you’re completely in control. It also has extras such as setting up KMS for each 'service'.

Road map

Roll is in active development, come back soon for the full readme experience.

Developing Roll

Add these lines to your Machfile before Roll is invoked

#cp "{path-to-roll}/src"
#cp "{path-to-roll}/tf"