Ask Your Question
0

How to get starting with( string functions available) in streamsets transformer

asked 2020-03-27 06:21:19 -0500

Kalaivani gravatar image

Dear Team,

I'm new to streamsets transformer. I wanted to know how to write filter conditions in the filter component. Also, different string functions available in transformer.

My requirement is I wanted to filter all the records which are not starting with "#". I couldnt get much from documentation. Do I need to know about Spark and Scala.

Regards, Kalaivani

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-03-28 05:09:19 -0500

In the Filter component you can use any Spark SQL syntax that can be used in the WHERE clause of a SQL SELECT query. See documentation here:

https://streamsets.com/documentation/...

If you check for the available Spark SQL functions (https://spark.apache.org/docs/latest/...) you can find the LIKE function. With that you can easily do a NOT LIKE:

<column-name> NOT LIKE '#%'

where <column-name> is just a field in your message.

Hope that helps

BR Guido

edit flag offensive delete link more

Comments

Thanks. It worked. :) When I tried str:startswith function, it was throwing error. Can you please let me know how can I use string functions like str:concat, str:endswith, str:startswith.

Kalaivani gravatar imageKalaivani ( 2020-03-30 04:14:51 -0500 )edit

Sorry, just saw your comment today. You are inside Spark SQL, so you can't use the Streamsets expression language, you have to use Spark SQL Functions for that. Find a list of available functions here: https://spark.apache.org/docs/2.4.5/api/sql/index.html.

gschmutz gravatar imagegschmutz ( 2020-04-25 06:30:50 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-03-27 06:21:19 -0500

Seen: 29 times

Last updated: Mar 28