Ask Your Question
0

How to avoid pipeline to shutdown when an exception is raised

asked 2020-11-11 14:47:19 -0600

ReneMendes gravatar image

updated 2020-11-12 12:18:40 -0600

Hi,

I have a microservice pipeline that uses a JDBC Lookup stage that submits a SQL dynamically built. However, when a column is not found in the table, an exception is raised and the pipeline shuts down.

StreamSets Data Collector 3.12.0

For instance:

com.google.common.util.concurrent.UncheckedExecutionException: com.streamsets.pipeline.api.base.OnRecordErrorException: JDBC_02 - Exception executing query: 'SELECT * 
FROM ALUNOS
WHERE (1=0) OR (CD_UNIDADES='001')' - 'Column Referenced not found in table list. (CD_UNIDADES) 


 FROM ALUNOS

 WHERE ( 1 = 0 ) OR ( << Syntax Error >>  CD_UNIDADES = '001' ) 
'
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)

Is there a way to avoid this behavior (pipeline shutdown) when a exception occurs in a stage ?

Thank you.

Renê

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2021-01-13 12:09:06 -0600

strem_dev gravatar image

Yes, you can avoid this by changing the On Record Error config for any particular processer.

For example, let's say my origin is JDBC and I don't wanna stop my job for error then I will config On Record Error as Send to Error or Discard. image description

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-11-11 14:47:19 -0600

Seen: 542 times

Last updated: Jan 13