Kudu error records not being written out

asked 2018-09-06 10:52:33 -0500

supahcraig gravatar image

updated 2018-09-06 14:04:50 -0500

I have a pipeline where I'm consuming from Kafka and upserting into a kudu table, and I have it set to write error records to /var/log/sdc. The pipeline was running for about 2 weeks without issue, then yesterday it started throwing a weird kudu error suggesting I had a primary key issue, and the pipeline failed. I'm thoroughly confused by the error, because looking at the preview shows no issues on the PK field, but neither does the pipeline UI show any error records.

Here's a snippet of the pipeline log,

Error while running: com.streamsets.pipeline.api.StageException: KUDU_03 - Errors while interacting with Kudu: Row error for primary key=[-128, 0, 0, 0, 111, -92, -60, -24], tablet=null, server=null, status=Timed out: can not complete before timeout: Batch{operations=193, tablet="84c0c8c073e14c26b0a3da415a84dc53" [0x00000001, 0x00000002), ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, tablet=84c0c8c073e14c26b0a3da415a84dc53, attempt=25, DeadlineTracker(timeout=10000, elapsed=9691), Traces: [0ms] sending RPC to server 053a1bbcc6b243b0a9c90f37b336fac1, [12ms] received from server 053a1bbcc6b243b0a9c90f37b336fac1 response Service unavailable: Service unavailable: Soft memory limit exceeded (at 99.05% of capacity). See https://kudu.apache.org/releases/1.6.0-cdh5.14.4/docs/troubleshooting.html
edit retag flag offensive close merge delete