Migrating multi-value non-associative data

Migrations in Drupal 8 are very stable and highly extensible. Still there are many cases though which would require custom code. I came accross one such scenario recently and will share some knowledge here which I gathered while working on it.

One - most of the examples shared are around process plugin or the plugins available in migrate (part of Drupal core) / migrate_plus modules. Ones that I browsed:

 

My case

Usually all data that is expected in Drupal needs to have a unique id. For instance if we want to migrate multiple images into a multi-value field, we need fid or we can consider even file path as unique key and migrate it. So first migration would migrate all the files into File entity and store mapping for external unique key (fid or file path) and internal File entity id. Now for the actual content migration we use migration_lookup plugin. Here, it will fail for scenario where field doesn't have unique value accross multiple content.

So in my case I had a document, it has ID and a multi value field for - type and system for instance. These both type and system are inturn taxonomy terms and are coming from another migration. Now this multi-value field is stored as a paragraph into system since field collections are getting deprecated.

 

Here, as you can see there are two main issues.

  1. migration_lookup plugin seems not working with multiple values like this, it needs unique ids for each value
  2. keys are part of data (0 => type:field, value:value) and not like (field => value)

At first, I started creating a custom processor plugin, extending the migration_lookup. But later I started looking at all possible places where I could extended the migration process (main learning). This is what I found out

  • We can definitely define as many process plugins as we want, lot of examples shared already
  • I looked at the code in migrate_plus and we can extend data_parser_plugin plugins too, for current problem I ended it creating custom parser plugin which was extending json parser, so I modified the source data to add unique ids for each scope to have scope_id like parent_id + '-' + delta, so it will have 1-0, 1-1, 2-0, 2-1, 2-2 and so on.
  • Another plugin that looks pretty good (not for current case though) and can be used to fetch data which requires authentication is data_fetcher_plugin

This was the learning from my first actual task in migration and I felt pretty good. Thanks to all the hard work done by great contributors I could manage to get it done in a day and with only like 50 lines of custom code.