Ask Your Question
1

Error evaluating expression : runtime.properties

asked 2017-12-19 07:04:19 -0500

Vivian Y gravatar image

updated 2017-12-19 11:15:15 -0500

metadaddy gravatar image

I followed StreamSets documentation to define a runtime property in order to set my values. I have set in sdc.properties as follows:

#Indicates the location where runtime configuration properties can be found.
runtime.conf.location=embedded
runtime.conf_development=/esb/conf

I created a file named conf in directory /esb, and restarted the pipeline. Once i call the runtime property in parameters ,using this syntax

${runtime:conf('development')}

This is the error output:

CREATION_005 - Could not resolve implicit EL expression '${runtime:conf('development')}': 
com.streamsets.pipeline.api.el.ELEvalException: CMN_0104 - Error evaluating expression 
${runtime:conf('development')}: java.lang.IllegalArgumentException: Could not resolve property 'development'
CREATION_005 - Could not resolve implicit EL expression '${runtime:conf('development')}': 
com.streamsets.pipeline.api.el.ELEvalException: CMN_0104 - Error evaluating expression 
${runtime:conf('development')}: java.lang.IllegalArgumentException: Could not resolve property 'development'
CREATION_005 - Could not resolve implicit EL expression '${runtime:conf('development')}': 
com.streamsets.pipeline.api.el.ELEvalException: CMN_0104 - Error evaluating expression 
${runtime:conf('development')}: java.lang.IllegalArgumentException: Could not resolve property 'development'
CREATION_005 - Could not resolve implicit EL expression '${runtime:conf('development')}': 
com.streamsets.pipeline.api.el.ELEvalException: CMN_0104 - Error evaluating expression 
${runtime:conf('development')}: java.lang.IllegalArgumentException: Could not resolve property 'development'

How can i further debug this?

edit retag flag offensive close merge delete

Comments

1 Answer

Sort by ยป oldest newest most voted
2

answered 2017-12-19 11:14:11 -0500

metadaddy gravatar image

updated 2017-12-20 17:44:34 -0500

Any change to sdc.properties requires a restart of SDC. This is mentioned as step 3 in the documentation:

3. Restart Data Collector to enable the changes.

Note that Runtime Properties are only read at startup regardless of whether they are embedded or located in a separate file.

Runtime Resources

If you need a more dynamic solution, you should use Runtime Resources, which are loaded every time they are referenced.

Create a file in $SDC_RESOURCES containing the configuration value - in your case, you might create a file named development with content:

/esb/conf

Use runtime:loadResource in your pipeline. Note that the entire content of the file is used as the configuration, so it's a good idea to use str:trim to remove any whitespace such as a terminating CR/LF:

${str:trim(runtime:loadResource(<file name>, <restricted: true | false>))}

Set the restricted argument to true if this is confidential data, and SDC will verify that the file is owned by the sdc user and read and writable only by that user. Assuming that your parameter is not secret, then, you would use:

${str:trim(runtime:loadResource('development', false))}

Runtime Parameters

Runtime Parameters are set per-pipeline, and referenced in that pipeline with the syntax:

${PARAM_NAME}

You can set Runtime Parameters in the pipeline configuration's Parameters tab, in 'simple edit' mode, one at a time, or in bulk, with the syntax:

[
    {
        "key": "PARAM1",
        "value": "value1"
    },
    {
        "key": "PARAM2",
        "value": "value2"
    }
]

You can stop the pipeline, set a new value for one or more Runtime Parameters, then restart the pipeline for the new values to take effect.

edit flag offensive delete link more

Comments

@metadaddy, I have one question to ask. You mentioned for runtime:loadResource ["Note that the entire content of the file is used as the configuration"], is this mean that one files can only contain one parameter? 100 parameter will need to have 100 files to store a single value?

Vivian Y gravatar imageVivian Y ( 2017-12-19 22:01:54 -0500 )edit

Can i store 100 parameter in one files? If yes, how you will suggest to be implemented?

Vivian Y gravatar imageVivian Y ( 2017-12-19 22:02:17 -0500 )edit

Only as runtime properties, which would require a pipeline restart. There are also runtime parameters that are specified per pipeline - will add more detail to my answer.

metadaddy gravatar imagemetadaddy ( 2017-12-20 17:37:04 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-12-19 07:04:19 -0500

Seen: 207 times

Last updated: Dec 20 '17