Ask Your Question

How to create multiple records from lines of text in a field

asked 2018-02-28 15:38:40 -0600

metadaddy gravatar image

I have a field in Avro that has multiple lines of Apache log. I'm trying to split the value into multiple records. I use the CR separator and kept getting java.lang.IllegalArgumentException: The delimiter cannot be a line break in the data parser . What am I doing wrong? Is this even supported?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-02-28 15:42:07 -0600

metadaddy gravatar image

This is a limitation of the underlying parser we use for delimited data - the data parser is not the right tool for this.

You can do this in the Groovy Evaluator with the following script:

for (record in records) {
  try {
    // Assuming the field in question is '/f'
    def parts = record.value['f'].split('\n')
    0.upto(parts.length - 1) {
      // Create a new record for each line in the field
      newRecord = sdcFunctions.createRecord(record.sourceId + ':${it}')
      newRecord.value = ['f' : parts[it]]
  } catch (e) {
    // Write a record to the error pipeline
    log.error(e.toString(), e)
    error.write(record, e.toString())
edit flag offensive delete link more


Could you also provide a way to do this in a JavaScript evaluator? I am attempting to do so, but keep getting "null" returned from the sdcFunctions.createRecord() function.

tlochner95 gravatar imagetlochner95 ( 2019-04-17 15:27:38 -0600 )edit

Thank you. That was my post I had posted, and your solution worked fantastically! Thanks again!

tlochner95 gravatar imagetlochner95 ( 2019-05-16 17:17:46 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-02-28 15:38:40 -0600

Seen: 642 times

Last updated: Feb 28 '18