Published content items randomly dissapear from fetched YAML

APIs Content Delivery API
1

I’ve been having the following problem with fetching new content from Contenful to Jekyll via the jekyll-contentful-data-import and contentful_bootstrap gems:

Whenever I publish a new item in a content type and fetch the data with bundle exec jekyll contentful, the latest addition is present in the YAML normally but mostly (not always) several older items that remained published but were not updated randomly disappear.

Once I make updates to them in Contentful and fetch the data again, the updated items show up in the YAML but I need to make an update to every single one that is missing - feasible when there are 5 items but not really with more.

What is causing this issue?

EDIT: I just found that other content types randomly disapear as well! This is absolutely unstable! The API keeps removing published content and I have spent past 2 hours trying to update each disapeared item only to find out that different COMPLETELY RANDOM ITEM disapears on each fetch. I have also tried to paste the YAMl manually which jekyll won’t parse because of some illegal character.

What is the problem? This is absolutely unusable!

2

Hi @jmth,

I’m pinging a colleague of mine and, in the meanwhile, could you tell us how exactly you’re fetching data using this gems? (omitting all sensitive information and credentials, of course)

3

Hey @jmth

I’m the creator and maintainer of the Jekyll integration, as @gabriel said, could you post the relevant piece of the _config.yaml file? This way I could check what may be the case, as this is something I’ve never seen happening.

Cheers

4

Hi @gabriel and @david.litvak, thanks for getting back to me.

In _config.yaml Contentful is configured on the following two places:

contentful:
  spaces:
    - boc:
        space: yyyyyyyy
        access_token: xxxxxxxxx

page_gen:
  - data: 'contentful.spaces.boc.article'
    template: 'article'
    name: 'slug'
    dir: 'articles'
    comments: true
  - data: 'contentful.spaces.boc.press'
    template: 'press'
    name: 'slug'
    dir: 'press'
    comments: true

I’m not sure whether how Gemfile is written can make a difference:

source "https://rubygems.org"
gem 'jekyll', '~> 3.3', '>= 3.3.1'
gem 'rake'
gem 'contentful_bootstrap'

group :jekyll_plugins do
  gem 'jekyll-feed', '~> 0.9.3'
  gem 'jekyll-contentful-data-import', '~> 1.6'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]
gem 'jekyll-paginate'
gem 'jekyll-sitemap'
gem 'jekyll-seo-tag'

# Some more
gem 'jekyll-last-modified-at'
gem 'hash-joiner'
gem 'json'

I only notice this problem sometimes and I often fetch the data from public wifis that can be both spotty and flagged as malicious, can it be that the request is throttled or that parts get lost?

5

Hey @jmth,

Looks like you may be having more than 100 entries in your space. You can easily solve this by adding the all_entries: true option like follows:

contentful:
  spaces:
    - example:
        space: cfexampleapi
        access_token: b4c0n73n7fu1
        all_entries: true

This will ensure that all entries are properly fetched and does the proper pagination to fetch everything even if you have more than 1000 entries (the maximum amount of entries per request).

If you need some more granular control on how to import your data, you can tweak the cda_query and the all_entries_page_size options. You can read more about this here: https://github.com/contentful/jekyll-contentful-data-import#configuration

Cheers

6

That was the problem. Thank you!