Ask Your Question

Issue with posting JSON message to HTTP client

asked 2019-01-23 10:12:14 -0600

Anuradha gravatar image

updated 2019-01-25 11:44:48 -0600

metadaddy gravatar image

I have below JSON message and while posting the message to the HTTP client processors/destination fails with the error "HTTP_40 - Error sending resource. Status: 400 Reason: Bad Request"

{"payload":"{\"Channel\":\"\",\"LOB\":\"\",\"Product Type\":\"\",\"Site\":\"\",\"Percentage_Split\":\"\",\"Err\":\"CASSANDRA_06 - Could not prepare record id'Sheet1::162' due to: com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [decimal <-> java.lang.String]\"}","appName":"eis-xyz-errorMessagePublisher","appUser":"xyz@dev","businessIdentifier":"eis-xyz-errorMessagePublisher","domain":"All","environment":"Dev","event":"ETL","globalTransactionId":"7b088ee3-7391-3d74-8de3-4a35c2028a81","hostname":"","serviceName":"XYZ_NEW_SPLIT:34043fef-0591-4954-8760-420cd9ca0732:dev","status":"Error","step":"Cassandra_01"}

Version : StreamSets Control Hub 3.6.0

Pls suggest how to resolve this issue?

edit retag flag offensive close merge delete


Are you able to post a similar message with a tool such as curl? Are you able to get any information from the web service as to why this request is bad?

metadaddy gravatar imagemetadaddy ( 2019-01-23 12:30:47 -0600 )edit

I tried from POSTMAN and it is successfully posted. I even tried to change the nested JSON to string & tried. Still it fails. Updated the request JSON in the original post.

Anuradha gravatar imageAnuradha ( 2019-01-24 00:21:29 -0600 )edit

I get the data in sdc record format and use JSON generator to convert the rec to JSON format before sending the req data to HTTP client dest. observed the output from JSON generator is enclosed within " and that is not a valid JSON. Any way to remove " before sending to HTTP client?

Anuradha gravatar imageAnuradha ( 2019-01-24 03:37:20 -0600 )edit

What have you selected for Request Transfer Encoding on HTTP Client? If it's CHUNKED, can you change it to BUFFERED and see what happens? If it doesn't work, can you run preview and upload screenshot of what the record looks like coming out of the stage right before HTTP Client?

iamontheinet gravatar imageiamontheinet ( 2019-01-24 11:35:15 -0600 )edit

Request transfer encoding is already set to BUFFERED. Even if i change to CHUNKED, i get same error. I am unable to upload the screenshot. It fails with error >10 points required to upload!!! The main reason is the request JSON to HTTP client is sent with the enclosed ".

Anuradha gravatar imageAnuradha ( 2019-01-25 03:15:37 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2019-01-25 11:48:06 -0600

metadaddy gravatar image

You are 'double encoding' the record. 'JSON Generator 2' is converting the record to JSON, then the destination is encoding that JSON again, which is where the extra quotes are coming from.

Remove 'JSON Generator 2' and the destination will do the final JSON encoding for you.

edit flag offensive delete link more


the rest API is rejecting if the request has nested JSON. I added JSON generator 2 to convert the inner MAP data (original SDC record) to string. Should I use any other processors to convert it to string? Should I retain "JSON generator 1"?

Anuradha gravatar imageAnuradha ( 2019-01-25 11:59:59 -0600 )edit

You should remove JSON Generator 1 if the API is not expecting nested JSON. Setting the destination's data format to JSON will encode the entire record.

metadaddy gravatar imagemetadaddy ( 2019-01-25 12:10:03 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-01-23 10:12:14 -0600

Seen: 199 times

Last updated: Jan 25 '19