Ask Your Question
1

I am unable to get streaming data from twitter

asked 2019-12-04 09:10:47 -0600

bebemazx3 gravatar image

When I run the pipeline I only get one record. I used the oauth authentication. When I used the basic with oauth I get an error saying that the resource url is invalid. Anyone has a configuration that works ? Send me the full steps .I just need to get any data to prove I can use the tool to get twitter data.

edit retag flag offensive close merge delete

Comments

Could you edit your question to include details of how you configured the HTTP Client origin? The Twitter URL you used, mode, etc.

metadaddy gravatar imagemetadaddy ( 2019-12-04 10:04:55 -0600 )edit
iamontheinet gravatar imageiamontheinet ( 2019-12-04 10:46:22 -0600 )edit

This url does not exist. I used the basic with the oauth option, also the oauth by itself. They are both giving me errors. I need a detailed configuration of how streamsets connects to twitter.

bebemazx3 gravatar imagebebemazx3 ( 2019-12-04 10:54:47 -0600 )edit

Click on the URL again.

iamontheinet gravatar imageiamontheinet ( 2019-12-04 10:56:56 -0600 )edit

1 Answer

Sort by » oldest newest most voted
0

answered 2019-12-04 11:35:25 -0600

metadaddy gravatar image

Here is the configuration that works for me:

HTTP Client Origin

  • Resource URL: https://stream.twitter.com/1.1/statuses/sample.json - this is the random sample feed - free, and useful for testing
  • Mode: Streaming - this is important if you want to stream tweets, rather than just fetch them one time
  • Authentication Type: OAuth - Twitter doesn't support OAuth 2.0 on streaming endpoints
  • Credentials Tab: credentials from your Twitter app (keys and tokens tab in Twitter)
  • Data Format Tab: JSON, multiple JSON objects, increase Max Object Length to something like 65536
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-12-04 09:10:47 -0600

Seen: 47 times

Last updated: 2 days ago