Idempotency & Deployments

Misc Technical Topics
1

Hi there,

First, this is a loaded post, and I thank you in advance for time spent trying to address it.

As it regards contents models, I’d like to take a declarative, code-based approach to configuration management, with a generic, automated deployment strategy which emphasizes data safety and rollback capability.

The dream is to be able to define the entire content model in code and automatically deploy from sandbox, to dev, to staging, and finally to production without having to worry about:

  1. Data loss,
  2. Ability to rollback, or
  3. Version-to-version specific migration scripts, which I don’t ever want to write.

As far as migration tools go, in my experience, I’ve found the following solutions and their approaches effective:

  1. CloudCMS decouples the underlying data from the model definition, and validates that the existing data is usable in the new type before you can deploy the change. If it’s not, you will need to update the data across all content instances before it can be deployed to production and they provide CLI & UI tools so you can make bulk changes. This means that content managers can deal with massive content changes without developers.

  2. Demandware takes a snapshot of all of the data and model as the first step in standard deployment procedure, allowing you to rollback with a single click. If a field’s type changed, then existing “active” data for that field is deleted upon deployment, so when changes to models occur, we always deploy updated data and model together. It also provide import/export of content so that content managers can deal with massive content changes without developers.

  3. Terraform allows you to declare your desired infrastructure state in code, and with one command, it inspects current state, figures out what changes need to be done current resources in order to achieve the desired state, and creates an execution plan which you can then apply upon confirmation.

  4. Gorm, a Golang ORM, provides an AutoMigration API which " will ONLY create tables, missing columns and missing indexes, and WON’T change existing column’s type or delete unused columns to protect your data." and provides a plugin interface which I’ve leveraged to automatically backup, apply and migrate column type changes, or raise error when such change is not possible.

All of these tools free us from having to think too about what mutations need to happen to current state (much less code them in anyway in order to achieve new state, freeing us to focus on the new state, while providing tooling, safeguards, rollback capabilities so we can move quickly with confidence knowing that we won’t lose data and can quickly recover if something goes wrong.

How can I achieve something similar with contentful?

2

Hi Vinnie,

We’ve been developing tools to do this at Watermark Church for the last several months. Our primary stack is Ruby on Rails, deployed to Heroku.

Our approach has been to keep data & schema concepts separate, defining Content Types as being part of the schema which should be changed & migrated in a controlled manner. To that end we’ve forked Contentful’s Contentful Migration tool. The purpose of our fork is to make Contentful migrations more similar to Rails migrations. Our tool keeps track of which migration files have already been run in a space, and only executes those migrations which have not already been run. This allows us to store all our migrations in the db/migrate directory inside the repo, and progressively deploy changes from dev to staging to production.

We hook this up to our Heroku deployments using a shell script we’ve developed, Contentful Shell. In heroku, if your Procfile declares a release command, that command is executed whenever a new version of the code is promoted to your Heroku environment. We use this release command to run bin/contentful migrate, which executes the contentful-migration CLI in batch mode over the db/migrate directory. You could just as easily accomplish this with a rake task without using Contentful Shell, but it makes things easier.

Regarding #3, we’ve developed a nodejs based tool which takes a diff of two spaces (i.e. your dev space vs your production space) and writes out a compatible migration file. This works for 95% of our migrations, but occasionally requires hand-inspection.

All of these tools are open-source at the moment, and we will continue supporting them as long as we are on the Contentful platform for our primary CMS.

Unfortunately we haven’t solved rollbacks yet, but as I understand it the Contentful team is currently working on that.

Hope this helps! Feel free to reach out to me on slack as well.