Ask Your Question

Change in File Tail origin doesn't effect pipeline while streaming

asked 2018-07-04 03:09:34 -0500

Edward gravatar image

updated 2018-07-04 03:29:13 -0500

Referring to the pipeline suggested by @iamontheinet here link text, is there any way to continuously monitor the i/p file offset so that the pipeline will get effected with changes in File Tail origin ? Could you please let me know how to make the pipeline to be adoptable to changes with i/p File Tail without stopping and resetting the origin every time I change i/p hash tags so that the streaming process doesn't get disturbed(like a new o/p file is to be created each time I change i/p keywords) !

Many Thanks!

edit retag flag offensive close merge delete


Are you testing in Preview or Start (execution) mode? It appears to be working on my end. If I add more hashtags/search terms in the text file, the record count immediately goes up. What are your other File Tail origin and HTTP Client settings?

iamontheinet gravatar imageiamontheinet ( 2018-07-05 12:04:44 -0500 )edit

I'm testing in execution mode i.e., while the pipeline is running.I've only given the path for file tail origin and the http client settings are same as in that pipeline.I used to reset origin each time to get updated tweets of changed hash tags.

Edward gravatar imageEdward ( 2018-07-05 13:12:37 -0500 )edit

Can you share what exactly do you have in your text file and how are you adding more terms to it?

iamontheinet gravatar imageiamontheinet ( 2018-07-05 13:23:58 -0500 )edit

I'm giving %23facebook%20OR%20%23FIFA in text file as it will fetch tweets with hash tags Facebook or FIFA.To add another hash tag I'm adding to it as %20OR%20Trump.(as per my observation comma separated input like trump,fifa is giving tweets with key words along with those hash tags).

Edward gravatar imageEdward ( 2018-07-05 13:32:17 -0500 )edit

But the only concern is to get tweets with only hash tags of comma separated i/p which I couldn't get yet.Please let me know how to get the tweets with hash tags(irrespective of key words,mentions)if I give FIFA,Trump in i/p file (where #FIFA,#Trump is giving HTTP_01-Status:400 Reason:Bad Request)

Edward gravatar imageEdward ( 2018-07-06 01:27:35 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-07-06 01:23:30 -0500

Edward gravatar image

The issue is resolved after moving my error records directory to another location so that the directory given for i/p file path should have only file which is input text file.(by debugging error: java.lang.IllegalStateException:LiveFileReader for 'LiveFile' is not open.) Thanks!

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-07-04 03:09:34 -0500

Seen: 59 times

Last updated: Jul 04