Dev Raw Data Source ignores batch size

asked 2019-10-24 04:00:53 -0600

Gill Bates gravatar image

updated 2019-10-24 04:01:58 -0600

I've found that if you set up "Dev Raw Data Source" as an origin and set batch size to 1 in "preview mode", it would still output the whole data as a single batch.

I've made an example setup: https://gist.github.com/AndreiPashkin...

It has a raw data source origin with three records and three Jython nodes that output "TEST 1", "TEST 2" and "TEST 3" to the logs respectively. If you launch it in preview and set "Preview Batch Size" to 1 in the preview dialog and then look into logs it would look like this:

TEST 1   
TEST 1
TEST 1
TEST 2
TEST 2
TEST 2
TEST 3
TEST 3
TEST 3

Which means that batch size is still >= 3 while it was set to 1. If it was 1 the logs would look like:

TEST 1
TEST 2
TEST 3
TEST 1
TEST 2
TEST 3
TEST 1
TEST 2
TEST 3

Did anyone encounter such a problem before?

edit retag flag offensive close merge delete