Support both Custom Processor/ Custom Origin in single Block

asked 2018-09-27 06:52:56 -0600

amishmshah gravatar image

Use Case: We have been evaluating StreamSets Data Collector as our data acquisition platform and have a unique use case. We have a data source which is a server accepts only 1 socket connection. The pipeline will be a client and we need to pull data from this source periodically. The way this source works is that the first commands loads data into scratch pad memory and subsequent read have to be fired to retrieve the data. This means the write and the reads are synchronous.

Now while the synchronous operation is in progress; the user may also fire a on demand request. This request has to go through this single socket; this means that same instance of the pipeline; moreover, has to be taken up only after the synchronous query is complete.

Is this possible to achieve using StreamSets Data Collectors?

We did explore writing a custom origin or custom processor but that does not meet our requirements; we would need a combination of a origin and processor to achieve this. Is this possible to create using StreamSets Data Collectors?

edit retag flag offensive close merge delete