Ask Your Question

Parsing JSON in groovy evaluator

asked 2018-01-03 11:19:54 -0600

nadelman gravatar image

I was attempting to take events produced by the aggregator processor and parse them into JSON, with the ultimate goal of re-arranging the aggregation event data, splitting out separate events for each group-by element, and then inserting the aggregations into either a relational database or elasticsearch destination. The aggregation processor's events contain a string field containing what appears to be valid JSON representing the results of each aggregation executed by the processor. I first tried using the JSON parser processor on the record to convert it to JSON, but that resulted in an IllegalArgumentException (String field could not be converted to a map). So, I turned to using a groovy processor, in which I would use a JSONSlurper to process the aggregation event string, and then apply the logic to re-form the data and produce the derived events that I wanted. On a first pass, I have just the following in groovy script:

import groovy.json.*
import groovy.json.internal.*

for (record in records) {
    try {
        groovy.json.JsonSlurper slurper = new groovy.json.JsonSlurper()
        def result = slurper.parseText(record.value) groovy.json.JsonBuilder(result).toPrettyString())
    } catch (e) {
        log.error(e.toString(), e)
        error.write(record, e.toString())

At this point, the intention is just to prove out that I can slurp the aggregation event JSON data, and log it. But, I get:

java.lang.NoClassDefFoundError: Could not initialize class groovy.json.internal.JsonParserCharArray

So, my question is how I can go about parsing a JSON string to a JSON object in a groovy processor. Do I need to do something specific to introduce JsonParserCharArray into the class path? I was under the impression that class was part of the core groovy library.

Or, is there a completely different approach I should be taking to achieve my goal?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2018-01-05 07:47:12 -0600

tmcgrath gravatar image

updated 2018-01-16 19:06:12 -0600

metadaddy gravatar image

If you update the etc/sdc-security.policy, in the groovy section, as shown in the following, your pipeline should work.

grant codebase "file:/groovy/script" {
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.util.PropertyPermission "*", "read,write";
  permission java.util.PropertyPermission "groovy.json.internKeys", "read";
  permission java.util.PropertyPermission "groovy.json.faststringutils.*", "read";

The necessity to update the JVM permissions is expected in some cases when using the Groovy Evaluator Processor.

There is an example in StreamSets docs

I haven’t reviewed the source code of the Groovy JsonSlurper and JsonBuilder classes, but I did find an old reference to JsonSlurper using System properties. I’m betting it’s one of the reasons why the permissions need to be updated in your pipeline.

We may be able to further reduce the required permissions if the log statement is removed groovy.json.JsonBuilder(result).toPrettyString())
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-01-03 11:19:54 -0600

Seen: 46 times

Last updated: Jan 16