Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

HTTP Client request logging

Hi,

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

regards Peter

HTTP Client request logging

Hi,

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

regards Peter

click to hide/show revision 3
None

HTTP Client request logging

Hi,

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

regards Peter

HTTP Client request logging

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with sdc 3.4.3

HTTP Client request logging

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with sdc 3.4.3

HTTP Client request logging

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with with: sdc 3.4.33.5.0

Log-Config:

/dev/null appender

log4j.appender.null=org.apache.log4j.FileAppender log4j.appender.null.File=/dev/null log4j.appender.null.layout=org.apache.log4j.PatternLayout log4j.appender.null.layout.ConversionPattern=null

<stdout> appender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

sdc.log appender

log4j.appender.streamsets=org.apache.log4j.RollingFileAppender log4j.appender.streamsets.MaxFileSize=256MB log4j.appender.streamsets.MaxBackupIndex=10 log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

log4j.rootLogger=INFO, streamsets,stdout log4j.logger.com.streamsets=INFO log4j.logger.org.eclipse.jetty=WARN log4j.logger.com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator=WARN log4j.logger.org.apache.kafka=WARN

log4j.logger.com.streamsets.http.RequestLogger=DEBUG

HTTP Client:

Set enable request logging

Log level: fine

Verbosity: payload_text

Max entity size: 32767

When using this setup I don't see any request details in the log-file. Only when I set the log4j.rootLogger=DEBUG, I see the expected output. But even if I disable the enable request logging on the HTTP client

HTTP Client request logging

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with: sdc 3.5.0

Log-Config:

/dev/null appender

log4j.appender.null=org.apache.log4j.FileAppender log4j.appender.null.File=/dev/null log4j.appender.null.layout=org.apache.log4j.PatternLayout log4j.appender.null.layout.ConversionPattern=null

<stdout> appender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

sdc.log appender

log4j.appender.streamsets=org.apache.log4j.RollingFileAppender log4j.appender.streamsets.MaxFileSize=256MB log4j.appender.streamsets.MaxBackupIndex=10 log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

log4j.rootLogger=INFO, streamsets,stdout log4j.logger.com.streamsets=INFO log4j.logger.org.eclipse.jetty=WARN log4j.logger.com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator=WARN log4j.logger.org.apache.kafka=WARN

log4j.logger.com.streamsets.http.RequestLogger=DEBUG

HTTP Client:

Set enable request logging

Log level: fine

Verbosity: payload_text

Max entity size: 32767

When using this setup I don't see any request details in the log-file. Only when I set the log4j.rootLogger=DEBUG, I see the expected output. But even if I disable the enable request logging on the HTTP client

Output when using debug on root logger

2018-12-20 08:39:13,747 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5 DEBUG REQUEST: HttpRequestPacket ( AsyncHttpClientFilter admin 0 jersey-client-async-executor-0 method=POST url=/customer/customers:upsert query=null protocol=HTTP/1.1 content-length=-1 headers=[ User-Agent=Jersey/2.25.1 (Async HTTP Grizzly Connector 2.25.1) Host=<removed> Connection=keep-alive Accept=/* Proxy-Connection=keep-alive] ) ... 2018-12-20 08:39:13,775 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5 DEBUG RESPONSE: HttpResponsePacket ( AhcEventFilter *admin 0 pool-92-thread-1 status=400 reason=Bad Request protocol=HTTP/1.1 content-length=-1 committed=false headers=[ server=envoy date=Thu, 20 Dec 2018 08:41:46 GMT content-type=application/json x-envoy-upstream-service-time=2 transfer-encoding=chunked] )

HTTP Client request logging

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with: sdc 3.5.0

Log-Config:

/dev/null appender

log4j.appender.null=org.apache.log4j.FileAppender log4j.appender.null.File=/dev/null log4j.appender.null.layout=org.apache.log4j.PatternLayout log4j.appender.null.layout.ConversionPattern=null

<stdout> appender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

sdc.log appender

log4j.appender.streamsets=org.apache.log4j.RollingFileAppender log4j.appender.streamsets.MaxFileSize=256MB log4j.appender.streamsets.MaxBackupIndex=10 log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

log4j.rootLogger=INFO, streamsets,stdout log4j.logger.com.streamsets=INFO log4j.logger.org.eclipse.jetty=WARN log4j.logger.com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator=WARN log4j.logger.org.apache.kafka=WARN

log4j.logger.com.streamsets.http.RequestLogger=DEBUG

HTTP Client:

Set enable request logging

Log level: fine

Verbosity: payload_text

Max entity size: 32767

When using this setup I don't see any request details in the log-file. Only when I set the log4j.rootLogger=DEBUG, I see the expected output. But even if I disable the enable request logging on the HTTP client

Output when using debug on root logger

2018-12-20 08:39:13,747 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5   DEBUG   REQUEST: HttpRequestPacket (    AsyncHttpClientFilter   admin *admin  0   jersey-client-async-executor-0
   method=POST
   url=/customer/customers:upsert
   query=null
   protocol=HTTP/1.1
   content-length=-1
   headers=[
      User-Agent=Jersey/2.25.1 (Async HTTP Grizzly Connector 2.25.1)
      Host=<removed>
      Connection=keep-alive
      Accept=/*
Accept=*/*
      Proxy-Connection=keep-alive]
)
...
2018-12-20 08:39:13,775 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5   DEBUG   RESPONSE: HttpResponsePacket (  AhcEventFilter  *admin  0   pool-92-thread-1
  status=400
  reason=Bad Request
  protocol=HTTP/1.1
  content-length=-1
  committed=false
  headers=[
      server=envoy
      date=Thu, 20 Dec 2018 08:41:46 GMT
      content-type=application/json
      x-envoy-upstream-service-time=2
      transfer-encoding=chunked]
)

)