Ask Your Question
0

Create a csv file using Jython Evaluator

asked 2020-05-15 07:22:13 -0600

PawSok gravatar image

Hi,

I would like to use the Jython Evaluator to create a file with the CSV extension that will contain selected data from records from the pipeline. I'm using Jython Evaluator, and please find the code below:

import os
import csv
from datetime import datetime

for record in records:
  try:
    srcdir = record.value['srcDir']
    filename = record.value['metadataFileName']

    headers = ['repositoryName','docID','fileName']

    with open(os.path.join(srcdir, filename + ".csv"), 'w') as csvfile:
      writer = csv.writer(csvfile, dialect = 'excel', delimiter=',')
      writer.writerow(headers)
      for record in records:
        writer.writerow([record.value['repositoryName'], record.value['docID'], record.value['fileName']])

#writing to the pipeline output
output.write(record)

  except Exception as e:
    error.write(record, str(e))

The csv file is being saved in the correct format:

image description

There is only one issue. For example, I have three records in one batch, one of them has a creation date: May 1, 2020 and the other two are from May 5, 2020. The record from 1st comes to Documents_2020-05-01 directory, from 5th to Documents_2020-05-05. Each file has its own metadata.csv file. At the moment each metadata file has all three records/rows. I want to group them.

There are three records/current view: image description

Required final view:

metadata1.csv: (in Documents_2020-05-01 folder)

image description

metadata2.csv: (in Documents_2020-05-05 folder)

image description

edit retag flag offensive close merge delete

Comments

Any particular reason why you aren't using, for example, Local FS destination with Delimited data format?

iamontheinet gravatar imageiamontheinet ( 2020-05-15 10:26:22 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-05-19 00:08:31 -0600

PawSok gravatar image

Hi iamontheinet,

I really don't know why I didn't think about this solution :) It works now. Thank you.

edit flag offensive delete link more

Comments

You're welcome!

iamontheinet gravatar imageiamontheinet ( 2020-05-19 07:47:38 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-05-15 07:22:13 -0600

Seen: 73 times

Last updated: May 15