Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

JDBC Producer ignores new JSON input keys until pipeline restart

Hi,

I might be missing something but here's the situation:

  • HTTP (Gets json array with records)
  • Flattener
  • PostgreSQL Metadata (does alter the db according to input keys, even new ones while the pipeline is running)
  • JDBC Lookup (to decide if its an update or insert)
  • JDBC Producer (one for update, one for insert, targeting a postgreSQL db)

If the pipeline is running, and i add another key to the source, the "PostgreSQL Metadata" processor alters the table and adds the new column, everything fine.

But, the producers wont fill the column until I restart the pipeline. Seems the producers are caching the schema and wont even try to use the new keys to insert them into the new column.

Is this expected behaviour? Am I missing something? Workaround for me atm is to hit the REST API with a stop/force stop/start call on that particular pipeline, but yeah, would be cool if everything just plays nicely.

cheers and TIA