Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

sdc.log json format

We are sending our SDC log files (sdc.log) to ELK/Kibana for our end users to view. We have filebeat agent installed on the machine to send it to Elastic and viewed in Kibana. I am trying to get the sdc.log file totally formatted in JSON because filebeat/ELK processes JSON quite well.

Here is my sdc-log4j.properties :

log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern={"AppType": "aop", "Timestamp": "%d{yyyy-MM-dd}T%d{HH:mm:ss,SSS}", "User": "%X{s-user}", "Pipeline": "%X{s-entity}", "Severity":"%-5p", "Stage":"%c{1}", "Detai\ls":%m, "Throwable":"%throwable"}%n

layout as shown above is wrapped in brackets {...} so each line is Valid JSON.

{"AppType": "aop", "Timestamp": "%d{yyyy-MM-dd}T%d{HH:mm:ss,SSS}", "User": "%X{s-user}", "Pipeline": "%X{s-entity}", "Severity":"%-5p", "Stage":"%c{1}", "Details":%m, "Throwable":"%throwable"}%n

However it is the Details part of the message I NEED JSON as well.

When I explicitly log JSON in our Custom code or in our Pipelines we get JSON in Details. However when SDC logs messages it is only string values. I want to configure sdc-log4j.properties file so it will always place JSON in the Details

Any examples out there would be appreciated.