Error while sending error records to google cloud storage

asked 2020-11-10 14:58:28 -0600

Hello guys, first post here, been using streamsets for a while and I love it, but this problem has been bothering me for about two weeks now.

I did a test pipeline that should throw some error records to GCS: - The pipeline throws half of the records (non-error) to a GCS (without a problem: meaning that the GCS bucket is ok, auth is ok, configuration is ok, etc.) - Now, when it starts throwing error records to GCS, it gives me a java null pointer error, and I have no idea why this is happening. It's weird because it can send the good records to GCS, but no the errored ones? And even weirder is the fact that even if the error comes inn the pipeline, it GETS to SUCCESSFULLY record the errors there in GCS, before crashing the job in streamsets...

My main question would be, why is the streamsets job crashing, if when the good data is sent to GCS I have no problem, and since even the error records are correctly recorded in GCS, before crashing the streamsets job...

This is the pipeline test: https://ibb.co/mRS9xJx

And here is the error log:

2020-11-10 20:43:08,359 USERS_STATES GCS DUP TEST 2/USERSSTATESGCSDUPTEST280e71dc1-3ee1-400d-9919-60f73394059f  WARN    Error while running: java.lang.NullPointerException ProductionPipeline  *mateus     ProductionPipelineRunnable-USERSSTATESGCSDUPTEST280e71dc1-3ee1-400d-9919-60f73394059f-USERS_STATES GCS DUP TEST 2
java.lang.NullPointerException
    at com.streamsets.datacollector.runner.StageContext.toEvent(StageContext.java:517)
    at com.streamsets.pipeline.lib.event.EventCreator$EventBuilder.createAndSend(EventCreator.java:240)
    at com.streamsets.pipeline.stage.cloudstorage.destination.GoogleCloudStorageTarget.lambda$write$5(GoogleCloudStorageTarget.java:182)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at com.streamsets.pipeline.stage.cloudstorage.destination.GoogleCloudStorageTarget.write(GoogleCloudStorageTarget.java:143)
    at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:290)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:235)
    at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:298)
    at com.streamsets.datacollector.runner.production.BadRecordsHandler.handle(BadRecordsHandler.java:75)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:858)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:831)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:569)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:383)
    at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:527)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:109)
    at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
    at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:703)
    at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:151)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope ...
(more)
edit retag flag offensive close merge delete