Ask Your Question
0

What is the maximum record size Data Collector can process? [closed]

asked 2017-06-27 17:10:10 -0600

LC gravatar image

I've been increasing the max line length property in the origin, but the pipeline still doesn't process my longer lines.

edit retag flag offensive reopen merge delete

Closed for the following reason duplicate question by metadaddy
close date 2017-06-27 17:52:13.883823

1 Answer

Sort by » oldest newest most voted
0

answered 2017-06-27 17:12:03 -0600

LC gravatar image

In versions before 2.6.0.0, the Data Collector maximum record size is 1048576 bytes.

With version 2.6.0.0 and later, you can configure the parser.limit Data Collector configuration property (in the sdc.properties file) to the limit that you prefer.

As with any Data Collector configuration property, you must restart Data Collector for the change to take effect.

Note that with some data formats, you can set a limit for the largest record to process in the origin. But that limit is bounded by the Data Collector limit. For a fuller discussion, see Maximum Record Size.

edit flag offensive delete link more

Comments

Regarding the comment about parser.limit, above - i have used the HTTP Client and tested with 20MB specified as the parser.limit. This is far in excess of the size of the record I was reading - which was a single-line complex JSON document which was just over 6.5MB.

bob gravatar imagebob ( 2017-06-27 17:41:29 -0600 )edit

Question Tools

1 follower

Stats

Asked: 2017-06-27 17:10:10 -0600

Seen: 369 times

Last updated: Jun 27 '17