Importing 100 entries of structured Content to a Contentful Model Programatically

APIs
1

Hello I am trying to do something very simple that would take many hours of manual work.
I am trying to upload all my json data to contentful!

The json data contains all the entries I want for a specific content model I already created. Image arrays, description, address etc.

I am trying to follow documentation
https://www.contentful.com/developers/docs/tutorials/cli/import-and-export/

but lots of it is very unclear to me.
I have a very simple repo

I would like the JSON to me added as entries into a specific content model.
What kind of configuration am I missing in my config.js file?

THanks

2

Hi @arhoy,

Assuming I understood correctly… the import/export functionality in Contentful CLI is not able to import any JSON file. It can only import very specific JSON structure - namely JSON files, that were created by exporting a JSON file with Contentful CLI.

Having said that, it is possible to use any JSON file as a source and import it via Content Management API, but that requires some custom code and the Contentful Management API client for the language of your choosing.

3

I was actually working on this yesterday and struggled to find a lot of relevant info for a CSV import, but I was able to piece some stuff together. Here is an example of a simple JavaScript import I created to import from a CSV file. Hopefully this might help as a starting point to modify for importing from JSON.

const fs = require('fs');
const Parser = require('csv-parse');
const contentful = require('contentful-management');

const client = contentful.createClient({
  accessToken: '<token>',
});

async function main () {
  const space = await client.getSpace('<spaceId>')
  const environment = await space.getEnvironment('<envId>')
  const parser = new Parser({delimiter: ',', columns: true});
  const readStream = fs.createReadStream('./movies.csv').pipe(parser);

  for await (const chunk of readStream) {
    const entry = await rowToEntry(environment, chunk)
    console.log(entry);
    entry.publish();
  }
}

async function rowToEntry(env, row) {
  const date = new Date(row.Finished.replace('-', '/'))
  return await env.createEntry('movie', {
    fields: {
      title: {
        'en-US': row.Title
      },
      dateCompleted: {
        'en-US': date.toISOString()
      },
      expectedRating: {
        'en-US': parseInt(row.Expected)
      },
      rating: {
        'en-US': parseInt(row['Rating (of 5)'])
      }
    }
  })
}

main().catch(console.error)
3 Likes
4

Interesting, thanks Blake, I did not know contentful supported CSV uploads.
I am just looking for a solution to upload entries into a content model. It has been a real struggle.

1 Like
5

Sorry, to clarify, this was a script I wrote to read a local CSV file and create entries using the Content Management API. Contentful doesn’t support CSV or other uploads, as far as I’m aware, for importing content.

6

Sorry @arhoy I believe I misunderstood what you were initially asking about. If you are wanting to import https://github.com/arhoy/contentful-server/blob/master/contentful-export-rhk3uw4rws0r-master-2020-04-22T12-46-99.json

You would need to fill in your Contentful Space ID and create/add a management API key in your example-config.json. If the content model already exists, you can use the skip-content-model option:

contentful space import --config example-config.json --skip-content-model
7

Hey Guys, Apologies for jumping in here, Im trying to do something similar. Im trying to copy an Entry and all its nested children from one contentful space to another.

I have tried the Import/Export tool but it seems like it does not resolve nested entries.

@arhoy @blake.thompson Any idea how I can achieve this?

Thank you! Again apologies for jumping into your conversation.

8

I have no idea. I’ve gave up on ocntentful though atm and found wordpress way better and easier to use for cms for my purpose. It will always be free too

1 Like
9

@syed.kazmi26 I don’t know for sure, but you might need to import the referenced entries first with the same IDs and then import the entry you wanted copy with the nested children.

10

@blake.thompson thank you for the reply,

oh you mean import the nested children entries first with the same ID’s and then import the parent entry in the end?

11

@syed.kazmi26 yes exactly, I would give that a shot!

12

Thanks Blake! Very helpful snippet.

14

I’ve started using this approach however running into a lot of issues with API call limits. Even when I add setIntervals in the loops.

For example I have another API feed (in JSON) I am looping through via a Node Express project but after 10 items I start getting the limit alerts. Is there a way around this? We will have 1500 items to import.

The next issue is checking if they exist, if they do update any new data. But that’s for stage 2.

app.get("/", (req, res) => {
  axios
    .get("https://mydata?pageSize=5")
    .then((resp) => {
      const institutions = resp.data.institutions;

      institutions.forEach((inst) => {
        console.log(inst.shortName);

        setTimeout(function () {
          //might have to run in here due to API limits.
          clientCMAPI
            .getSpace("MYspaceID")
            .then((space) => space.getEnvironment("master"))
            .then((environment) =>
              environment.createEntry("institution", {
                fields: {
                  shortName: {
                    "en-US": inst.shortName,
                  },
                  institution_id: {
                    "en-US": inst.id,
                  },
                },
              })
            )
            .then((newEntry) => {
              setTimeout(function () {
                console.log(newEntry.sys.id, newEntry.fields.shortName);
                clientCMAPI
                  .getSpace("MYspaceID")
                  .then((space) => space.getEnvironment("master"))
                  .then((environment) => environment.getEntry(newEntry.sys.id))
                  .then((entry) => entry.publish())
                  .then((entry) =>
                    console.log(`Entry ${entry.sys.id} published.`)
                  )
                  .catch(console.error);
              }, 1500);
            })
            .catch(console.error);
        }, 1500);
      });
      res.send(resp.data.institutions);
    });
});
15

A few things:

  1. The Environment API is reusable, so you don’t need to re-initialize it on each iteration
  2. createEntry returns a reference to the entry which you can use to publish
  3. I don’t see the purpose of using Express here, but OK… (why not just a node script?)

unverified pseudocode:

async function importData (institutions) {
  // ## [1] ENV API is reusable, so initialize outside of loop
  const envApiClient = await clientCMAPI
    .getSpace('MYspaceID')
    .then((space) => space.getEnvironment('master'));

  let resultData = await Promise.all(
    institutions.map(async (inst) => {

      // ## [2.a] createEntry a wrapped reference to the entry
      const currentEntry = await envApiClient.createEntry('my-type', {
        fields: (() => {
          // ## Format fields
        })(),
        metadata: {
          tags: (() => {
            // ## Format tags
          })()
        }
      });

      // ## [2.b] use this to publish
      await currentEntry.publish();

      return {
        ...inst, contentfulEntry: { ...currentEntry }
      };

    })
  );
}

If you want to do some CreateOrUpdate, you will need to have some sort of unique constraint.

  1. Define a unique constraint
  2. Update the validation of your ContentModel (if not using sys.id)
  3. Check for existing entry
    • Add try/catch around createEntry,
    • Query for existing entry at the beginning
  4. Create/Update

Personally, I prefer a pattern of “pre-generating” & storing Entry IDs & then using createEntryWithId()

/**
 * Generates a Base62 UUID via: https://github.com/shanehughes3/uuid62
 *
 * Reference:
 *  - Contentful UUID formatting: https://www.contentfulcommunity.com/t/support-for-standard-uuids/1635/6
 *
 */
export function generateLocalEntryId(): string {
  const uuid62 = require('uuid62');
  return uuid62.v1(); // NOTE: This returns a TimeStamp-Based UUID
}

// ...
await envApiClient.createEntryWithId('my-type', '<pre-generated-id>',{fields:{}})
// ...

I would be interested to better understand if there’s a risk of collision across content-types