Ask Your Question
0

HTTP Client request logging

asked 2018-10-02 06:55:37 -0500

peter_f gravatar image

updated 2018-12-20 02:47:47 -0500

According to the documentation request logging can be enabled by defining the required granularity and payload type. You also have to adjust the log level of the datacollector to actually make the request logging work.

Is it possible to limit the datacollector part to just the HTTP Client? Generally raising the datacollector log level to debug to get request logging finer/finest just bloats everything up and makes it difficult to track down just request issues.

Tested with: sdc 3.5.0

Log-Config:

/dev/null appender

log4j.appender.null=org.apache.log4j.FileAppender log4j.appender.null.File=/dev/null log4j.appender.null.layout=org.apache.log4j.PatternLayout log4j.appender.null.layout.ConversionPattern=null

<stdout> appender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

sdc.log appender

log4j.appender.streamsets=org.apache.log4j.RollingFileAppender log4j.appender.streamsets.MaxFileSize=256MB log4j.appender.streamsets.MaxBackupIndex=10 log4j.appender.streamsets.File=${sdc.log.dir}/sdc.log log4j.appender.streamsets.Append=true log4j.appender.streamsets.layout=org.apache.log4j.PatternLayout log4j.appender.streamsets.layout.ConversionPattern=%d{ISO8601} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] %-5p %c{1} - %m%n

log4j.rootLogger=INFO, streamsets,stdout log4j.logger.com.streamsets=INFO log4j.logger.org.eclipse.jetty=WARN log4j.logger.com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator=WARN log4j.logger.org.apache.kafka=WARN

log4j.logger.com.streamsets.http.RequestLogger=DEBUG

HTTP Client:

Set enable request logging

Log level: fine

Verbosity: payload_text

Max entity size: 32767

When using this setup I don't see any request details in the log-file. Only when I set the log4j.rootLogger=DEBUG, I see the expected output. But even if I disable the enable request logging on the HTTP client

Output when using debug on root logger

2018-12-20 08:39:13,747 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5   DEBUG   REQUEST: HttpRequestPacket (    AsyncHttpClientFilter   *admin  0   jersey-client-async-executor-0
   method=POST
   url=/customer/customers:upsert
   query=null
   protocol=HTTP/1.1
   content-length=-1
   headers=[
      User-Agent=Jersey/2.25.1 (Async HTTP Grizzly Connector 2.25.1)
      Host=<removed>
      Connection=keep-alive
      Accept=*/*
      Proxy-Connection=keep-alive]
)
...
2018-12-20 08:39:13,775 staging.customer.upsert.kafka2transaction copy/stagingcustomerupsertkafka2transactioncopy11e3e763-efa5-44d5-ad13-d81f2dcb65c5   DEBUG   RESPONSE: HttpResponsePacket (  AhcEventFilter  *admin  0   pool-92-thread-1
  status=400
  reason=Bad Request
  protocol=HTTP/1.1
  content-length=-1
  committed=false
  headers=[
      server=envoy
      date=Thu, 20 Dec 2018 08:41:46 GMT
      content-type=application/json
      x-envoy-upstream-service-time=2
      transfer-encoding=chunked]
)
edit retag flag offensive close merge delete

Comments

Can you paste a sample of the request log line you see, when you adjust the root level to DEBUG?

jeff gravatar imagejeff ( 2018-12-19 11:28:16 -0500 )edit

@jeff, added the log output, when using debug on root logger, request logging not set (makes no difference in output if set or not)

peter_f gravatar imagepeter_f ( 2018-12-20 02:50:27 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-10-02 10:01:51 -0500

jeff gravatar image

Yes, you certainly can. As indicated in the documentation, the name of that particular logger is com.streamsets.http.RequestLogger. You can set up a different threshold for just that logger using the steps outlined here.

edit flag offensive delete link more

Comments

Hi Jeff, I already did this and have consulted the docs and have read your post before I posted the question. I tried log4j.logger.com.streamsets.http.RequestLogger=DEBUG, but it was not working. I have to set log4j.logger.com.streamsets=DEBUG to get it working. best Peter

peter_f gravatar imagepeter_f ( 2018-10-02 10:20:34 -0500 )edit

If you want to paste your full log4j config, as well as the level you selected in the pipeline, I can take another look.

jeff gravatar imagejeff ( 2018-12-07 12:06:48 -0500 )edit

Hi Jeff, Thanks for getting back on that, I updated my initial request. best Peter

peter_f gravatar imagepeter_f ( 2018-12-19 08:05:26 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-10-02 06:55:37 -0500

Seen: 472 times

Last updated: Dec 20 '18