Writing from one ES index to another

asked 2018-11-26 15:08:44 -0600

Spiced Cider gravatar image

updated 2018-11-26 16:59:17 -0600

I would like to know what is the recommended way to write data from one ES index to another (same mapping in both indices - destination has few additional fields that will calculate later in the pipeline). I've tried the following and am facing the issues as listed:

  • Writing from SDC ES source to ES destination - gives the error: Field [_index] is a metadata field and cannot be added inside a document. Use the index API request parameters.
  • Tried the approach above and removed the metadata fields: shows the error with the _source field: Field [_source] is defined both as an object and a field in [event]
  • Flattening fields, removing metadata fields and renaming fields (removing prefixes) - this works, but any nested array fields also get flattented and the doc no longer matches the mapping.
  • I can use a JS evaluator (which is the solution I'm going with for now) - but this involves manually configuring field names etc. (I'm hoping there is a better way)

I'm sure there is an easy (and right) way to do this .. vs. having to add multiple processors just to get the record to match the document mapping. Can anyone please shed some light on this? Thank you !

edit retag flag offensive close merge delete