Ask Your Question
1

HTTP Destination Header

asked 2019-10-02 16:32:54 -0500

jonas_souza gravatar image

Hi experts,

I'm using a HTTP Client Destination using OATH 2 to authenticate, but I believe that the header is impacting my process, please could you confirm if this is the problem?

This is the end of the pipeline:

image description

Here is what is arriving in the last step, with the error:

image description image description

Thank you :)

edit retag flag offensive close merge delete

Comments

Sometimes small change in the data type,data format and the field names can throw such errors. Check the API and make sure your data is exactly as your API is expecting. @jonas_souza

Maithri gravatar imageMaithri ( 2019-10-03 00:07:55 -0500 )edit

Hi :) Thanks for the comment, I've tested the request in the postman , which return success.. =/ just copied and pasted my generated json.

jonas_souza gravatar imagejonas_souza ( 2019-10-03 06:50:38 -0500 )edit

How did you configure the authentication for your HTTP client target stage?

jeff gravatar imagejeff ( 2019-10-07 13:21:42 -0500 )edit

Hi Jeff, I've set the required field (tmp) Authentication Type = none Use OAuth 2 = false (because I need to renew my token) Also set the header with the token, which is working fine (Authorization : Bearer ${record:attribute('access_token')}) Data format tried with types JSON / Text

jonas_souza gravatar imagejonas_souza ( 2019-10-08 07:00:40 -0500 )edit

I found a way , I'm going to post in the answer, but performance is rubbish.... =/ I believe because I'm not using HTTP Destination

jonas_souza gravatar imagejonas_souza ( 2019-10-08 07:01:35 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-10-08 07:14:11 -0500

jonas_souza gravatar image

Hi, the process is working renewing the token, but it takes long hours.

Here is how I designed, please note that I'm learning the tool, so I believe that should have better ways to do it... feedbacks are welcome :)

image description

A brief of the pipeline:

  • JDBC -> extract 200k records from SQL Server
  • Http Client Auth -> Retrieves a token from Sales Force (Mkt Cloud)
  • Export Token -> Export the token to the header
  • Jython -> Builds the JSON record
  • Http Client Load -> Perform the load of the json

This is working, but it took 16hours to load 150k records....

What I'm thinking to do is include in the Jython stage, a code to concatenate json records, maybe it could be an improvement, or try tod do the the destination HTTP works, maybe it can has a better performance too.

Thank you :)

edit flag offensive delete link more

Comments

This is very slow, since the HTTP Client processor makes an HTTP call for each record individually. You should use the HTTP Client destination. https://requestbin.com is a good tool for debugging - send the request there and you'll see exactly what is being sent.

metadaddy gravatar imagemetadaddy ( 2019-10-09 10:42:36 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-10-02 16:32:54 -0500

Seen: 205 times

Last updated: Oct 08