Ask Your Question
1

JDBC Producer ignores new JSON input keys until pipeline restart

asked 2019-09-26 10:43:12 -0500

mnc gravatar image

Hi,

I might be missing something but here's the situation:

  • HTTP (Gets json array with records)
  • Flattener
  • PostgreSQL Metadata (does alter the db according to input keys, even new ones while the pipeline is running)
  • JDBC Lookup (to decide if its an update or insert)
  • JDBC Producer (one for update, one for insert, targeting a postgreSQL db)

If the pipeline is running, and i add another key to the source, the "PostgreSQL Metadata" processor alters the table and adds the new column, everything fine.

But, the producers wont fill the column until I restart the pipeline. Seems the producers are caching the schema and wont even try to use the new keys to insert them into the new column.

Is this expected behaviour? Am I missing something? Workaround for me atm is to hit the REST API with a stop/force stop/start call on that particular pipeline, but yeah, would be cool if everything just plays nicely.

cheers and TIA

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-09-26 11:08:11 -0500

metadaddy gravatar image

It looks like we cache the JDBC metadata; I think this may have been added since the PostgreSQL Metadata Processor was released, since it certainly worked as expected in the past. Please file an issue with all of the above data at https://issues.streamsets.com/ and we can take a look. Please also leave a comment here with the issue number. Thanks!

edit flag offensive delete link more

Comments

Thanks, filed an issue with: SDC-12556

mnc gravatar imagemnc ( 2019-09-26 11:16:37 -0500 )edit

Thanks - I added my take and boosted the priority.

metadaddy gravatar imagemetadaddy ( 2019-09-26 12:26:33 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-09-26 10:41:13 -0500

Seen: 27 times

Last updated: Sep 26