Writing proper migration scripts with Migration CLI

Hi, what is the proper way to write migration scripts using the Migration CLI tool? Is there a way write idempotent scripts? For example, the following example will be successful the first time:

module.exports = function(migration) {
  var animal = migration.createContentType('animal').name('Animal');
  animal.createField('animalName').name('Animal Name').type('Symbol');
};

On second run it will error because the content type with ID “animal” already exists. Ideally I could check if the content type exists before attempting to run. Another option would be to check for errors in the script and handle as needed.

Any tips appreciated.

Hey @riley!

Besides checking if content type exists, I think can also just handle these errors. This example: https://github.com/contentful/migration-cli/blob/master/examples/05-plan-errors.js shows exactly creating same content type twice and I suspect it’s assumed you can handle the error depending on the response.
Also, several more examples can be found here: https://github.com/contentful/migration-cli/tree/master/examples.

Let me know if that helps a bit. If it’s not, I’ll investigate other approaches.

Thanks for the quick reply, Andrey. I see errors on the command line but how does one use them? A try/catch in the JS code doesn’t work, nor do they appear to use exit codes.

Here’s my use case: I’m working with multiple spaces to feed content for each of my project environments - Dev, and Prod. My plan is to make content model changes in code and deploy to Dev space. When ready I’ll deploy it to my Prod space.

This wouldn’t work if I were to write it in a single migration file…

Version 1 - make an “animal” content type and deploy

// migration_file.js
module.exports = function(migration) {
  var animal = migration.createContentType('animal').name('Animal');
  animal.createField('animalName').name('Animal Name').type('Symbol');
};

Version 2 - add a “plant” content type and deploy

// migration_file.js
module.exports = function(migration) {
  var animal = migration.createContentType('animal').name('Animal');
  animal.createField('animalName').name('Animal Name').type('Symbol');

  var plant = migration.createContentType('plant').name('Plant');
  plant.createField('plantName').name('Plant Name').type('Symbol');
};

“plant” won’t be created because it will error out on “animal” as it already exists.

This could work if…

Option 1 - try/catch

// migration_file.js
module.exports = function(migration) {
  try {
    var animal = migration.createContentType('animal').name('Animal');
    animal.createField('animalName').name('Animal Name').type('Symbol');
  } catch (err) {
    // handle error
  }

  try {
    var plant = migration.createContentType('plant').name('Plant');
    plant.createField('plantName').name('Plant Name').type('Symbol');
  } catch (err) {
    // handle error
  }
};

Option 2 - if not exists methods… or something…

// migration_file.js
module.exports = function(migration) {
  var animal = migration.createContentTypeIfNotExists('animal').name('Animal');
  animal.createFieldIfNotExists('animalName').name('Animal Name').type('Symbol');

  var plant = migration.createContentTypeIfNotExists('plant').name('Plant');
  plant.createFieldIfNotExists('plantName').name('Plant Name').type('Symbol');
};

Let me know your thoughts.

1 Like

Hi Riley,

the reason we went with the design decision to not ship with something like createContentTypeIfNotExists was that we wanted to preserve intent as much as possible
and to make all content model migrations as explicit as possible.

We validate - even before running a single step - for every step of the migration whether it can be done and will not cause any inconsistencies.

This becomes quite difficult once createIfNotExists is implemented.
For example given a migration like:

const foo = createContentTypeIfNotExists('foo')
foo.description('It is a foo')
foo.name('Foo')

foo.createField('bar').type('Symbol').name('a bar')

What should happen if foo exists, but its description is not It is a foo?
We would have to add a createOrUpdateContentType method too, and the same for the fields.

At that point we might even have to start diffing all the contents to see what we should apply and what not.
It gets very difficult to actually see what the expressed intent of that migration is, and so also as a developer,
you cannot be sure about the outcome.

For that reason, we designed this tool with a workflow in mind where a set of spaces is evolved only via these migrations
and where every migration would be applied in sequence and only once.

If you were to accidentally apply an old migration, it would quickly catch it and inform you about your error.
That is a safety net that you would not have with createOrUpdate operations, which could basically always be applied.

What we are indeed lacking at the moment is the tooling to automatically keep track of the migrations that have been run already
so that applying the right migrations in sequence to sync up two spaces can be done in a fully automated way,
but we are looking into supporting these sort of workflows.

Lastly, if you still would like to make use of createIfNotExists and others, there is a way to implement it yourself.
We do not have try/catch available because we first process the migration script into a set of intents,
validate them and only then proceed with building payloads and sending them out to the API, so at the time you write createContentType,
no actual creation is happening yet.

However, we do support creating the migration script asynchronously and our migration files are actually pure node modules so you could do something like:

const createContentTypeIfNotExists = async function (migration, contentTypeId) {
  // Assuming you have the SDK required and set up
  const contentType = await contentfulSDK.getContentType(contentTypeId)

  if (contentType) {
    return migration.editContentType(contentTypeId)
  } else {
    return migration.createContentType(contentTypeId)
  }
}

module.exports = async function (migration) {
  const contentType = await createContentTypeIfNotExists(migration, 'foo')
  contentType.description('It is a foo')
}

That way you can dynamically build the migration script you need.

1 Like

@tim Thank you for that thorough and helpful reply. It does help to know the intent you had in mind when creating the API.

Keeping our spaces in sync is a primary concern at the moment as we’re setting up a pipeline to allow us to build in dev and promote to prod. I’ve gone with the Contentful JS SDK and built it in much the way you described. I’ll keep on eye on the migration-cli tool and see how it develops for our needs. Thanks again!

1 Like

An option I was considering for this is to use a configuration space to track the last applied version of a migration script and only apply scripts that haven’t been applied yet.

Like @riley, I need to be able to make sure my content model is in a correct state on different environments.

I’d considered the same, either a config space or just an entry in each individual space that logged the latest migration applied. That said, it doesn’t have to be stored in Contentful - the record could be in a a database, in a flat file, you just need it somewhere that your build system can check. And you’ll need a record for each space. Dev might be at v50 while Stage is at v34 and Prod is at v35.

1 Like

track the last applied version of a migration script and only apply scripts that haven’t been applied yet.

@tim and me had different thoughts around this already. Your input is very welcome here, and we will definitely take this into consideration.
In the current state there will be no “official” solution around this topic, but we’ll try to give some hints on how this could be solved.

The idea of storing it in a dedicated Content Type within the space is something we liked, as it mirrors the conceptual idea of database migrations, where you also have a table. Sadly this would mean that you have a Content Type and at least one Entry (potentially one Entry per migration) that’s quite “meta” and not really useful apart from the migration.

@dustin.aleksiuk @riley do you want to keep multiple “production spaces” in sync with this, or one per project?

That would be very interesting for me to know - with the latter, one could imagine having a <migration-name>.js.lock file with some information on when it has been applied, which could be stored in VCS together with your migration. This is also a very appealing idea to me, as it stores the information next to the actual migration and does not has the overhead of a special Content Type.
But it might come with some implications around development situations, where you might have developer-specific spaces, so this information couldn’t be stored centralized.

Let me know about your use-cases!

Best,
Stephan

2 Likes

We have a single production space called PROD and 3 other environments called DEV, TEST, and PRE. We also often have developers work on a shared LOCAL space because the DEV space is used by a deployed version of the app. When developers commit code to master, a build agent creates a package and automatically deploys it to DEV. We currently do our content type changes in the Contentful UI in LOCAL, and then run a script that pulls down the content type so it can be committed to version control.

Our ideal scenario is to be able to apply the new content model to the next environment in the chain. Another very important step is to be able to roll back changes if something goes wrong, which I realize isn’t easy.

To answer your question specifically, we only have one production space but we always have 4 deployed versions of the app pointing to different spaces.

My other dream feature would be the ability to replicate a space exactly into another existing space using the Contentful UI.

I hope that made some kind of sense. Our basic need is to have the contentful changes be promoted as part of an automated deployment.

3 Likes

Thank you @stephan.schneider. Our scenario sounds a lot like @dustin.aleksiuk. We have 4 versions of our app pointing to 4 spaces - DEV, QA, STAGE and PROD. Our hope is to develop our models in the DEV space. When complete we intend to promote that change to up the environment chain.

We have a mental model like that of database migrations, as @stephan.schneider mentioned. We’re building our models in code instead of the the Contentful UI so that it can be applied to our other environments. We’re using the UI as more of a “viewer”.

Hi @riley,

Could you comment on how your migration-style approach is going? In one project here we simply apply all the content model changes from version control using a script. We use the UI to do our work and then pull down the content model JSON for the types we changed and put it in version control.

I’m about to get going on a second project and I need to look at how we handle promoting our work across environments. We use the UI to do our work and then pull down the content model JSON for the types we changed and put it in version control.

How do you find doing your content model changes in code instead of using the UI? Are you happy you went that way?

Hi @stephan.schneider, @tim,
Currently we have the same use case as mentioned by @dustin.aleksiuk above with Prod & 3 more environments and migrating the changes using UI.Can help to provide more details on how we can pull the changes from different content models and push it to higher environments. I want to configure this as part of my CI/CD Pipeline to migrate into higher environments through a webhook. Does Contentful provides any way to migrate through Contentful UI and not using scripts(Migration-CLI
).

@contentful @stephan.schneider Has an official solution around this topic been developed? What is the best practice for maintaining Content Model integrity across space environments when migration scripts are kept in the project?

@stephan.schneider is it still not going to be an official solution?

I think state management is a crucial element of a migration tool the only thing a developer/pipeline should say to a migration tool is apply and point it to an environment. Looking at how conventional database migration tools like liquibase, flyway, rails migrations and knex do it I think would a great idea. Is there any resources how to work around this gap in functionality? To be honest one reason for us to select contentful is because of this kind of maturity/mindset https://www.contentful.com/help/cms-as-code/ can we upvote this feature somewhere?

For perpetuity - Use this tool to make writing migrations as easy as making the change in the UI in one environment, then generating a migration based on the diff!