Ask Your Question

sdc.log json format

asked 2020-05-15 14:32:53 -0500

Peter Delaney gravatar image

We are sending our SDC log files (sdc.log) to ELK/Kibana for our end users to view. We have filebeat agent installed on the machine to send it to Elastic and viewed in Kibana. I am trying to get the sdc.log file totally formatted in JSON because filebeat/ELK processes JSON quite well.

Here is my :

log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern={"AppType": "aop", "Timestamp": "%d{yyyy-MM-dd}T%d{HH:mm:ss,SSS}", "User": "%X{s-user}", "Pipeline": "%X{s-entity}", "Severity":"%-5p", "Stage":"%c{1}", "Detai\ls":%m, "Throwable":"%throwable"}%n

layout as shown above is wrapped in brackets {...} so each line is Valid JSON.

{"AppType": "aop", "Timestamp": "%d{yyyy-MM-dd}T%d{HH:mm:ss,SSS}", "User": "%X{s-user}", "Pipeline": "%X{s-entity}", "Severity":"%-5p", "Stage":"%c{1}", "Details":%m, "Throwable":"%throwable"}%n

However it is the Details part of the message I NEED JSON as well.

When I explicitly log JSON in our Custom code or in our Pipelines we get JSON in Details. However when SDC logs messages it is only string values. I want to configure file so it will always place JSON in the Details

Any examples out there would be appreciated.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-05-15 14:42:10 -0500

jeff gravatar image

Data Collector currently uses Log4J 1.2 as the logging backend. It doesn't directly support what you're trying to do. However, it is possible to use a third party library. You would just need to add this jar to the api-lib directory for your Data Collector, and then use a configuration similar to the one shown there.

In the meantime, watch the SDC-14498 Jira. If and when Data Collector switches to Log4J 2, then JSON layout would be built-in and more easily available.

edit flag offensive delete link more


Thanks for that information

Peter Delaney gravatar imagePeter Delaney ( 2020-05-15 15:12:39 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-05-15 14:32:53 -0500

Seen: 30 times

Last updated: May 15