Ask Your Question

Revision history [back]

In the Filter component you can use any Spark SQL syntax that can be used in the WHERE clause of a SQL SELECT query. See documentation here:

https://streamsets.com/documentation/transformer/latest/help//transformer/Processors/Filter.html#task_xwl_wyv_bhb

If you check for the available Spark SQL functions (https://spark.apache.org/docs/latest/api/sql/index.html) you can find the LIKE function. With that you can easily do a NOT LIKE:

<column-name> NOT LIKE '#%'

where <column-name> is just a field in your message.

Hope that helps

BR Guido