Ask Your Question

Execute MongoDB query in StreamSets

asked 2019-02-12 10:21:13 -0600

srinath_222 gravatar image


I have a requirement to execute a query in MongoDB (retrieving data from join of multiple collections) and fetch the result as part of processors in my pipeline.

We have MongoDB Lookup processor but it does not have an option to execute query on MongoDB database like JDBC Lookup. I have an Origin as Relational and I want to fetch the data from Mongo as a lookup.

Please suggest an approach how to fetch the data from MongoDB.

edit retag flag offensive close merge delete


@iamontheinet any suggestions on this ?

srinath_222 gravatar imagesrinath_222 ( 2019-02-19 09:50:49 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2019-02-19 14:33:40 -0600

iamontheinet gravatar image


You can use one of the scripting processors to connect to MongoDB and perform (lookup) queries. Here's how in Jython Evaluator:

1) Install pymongo via pip install pymongo

2) In Jython Evaluator Init Script section add:

import sys
import pymongo
from pymongo import MongoClient
state['client'] = MongoClient('localhost', 27017)
state['db'] = state['client'].dashDB
state['collection'] = state['db'].inventory

Note: Replace sys.path with the path where you installed pymongo in step 1 as well as client connection, db and collection details as per your environment. FYI, if db and collection are not static, move them down to Script section.

3) In Jython Evaluator Script section add:

client = state['client']
db = state['db']
collection = state['collection']
for record in records:
    lookup_item = collection.find_one({"item": record.value['item']})"======== lookup_item: {}",lookup_item)
    # do something with looked up item
    # Write record to processor output
  except Exception as e:
    # Send record to error
    error.write(record, str(e))

Note: Update code as it pertains to your logic.

3) In Jython Evaluator Destroy Script section add:

client = state['client']

Hope this helps!

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools



Asked: 2019-02-12 10:21:13 -0600

Seen: 1,267 times

Last updated: Feb 19 '19