Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

By the nature of origins, we expect them to return multiple records. In your case, the origin is returning one record per line, because you have configured the data format as TEXT and left the rest of the configuration values to their defaults. The origin is seeing each new line, parsing it into a text record, and moving on to the next line/record.

The reason the processor is only "seeing" the first line is because you have configured the data format identically (TEXT with default settings, which means record separator is newline). This means that there are actually multiple "records" associated with a single input. Some of the Data Collector processors support handling of multiple records from a processor result, but unfortunately HTTP is not one of them. See the in-progress SDC-10968 Jira for updates on that being supported.

In the meantime, I recommend you configure the TEXT format for your processor differently. Use the _Use Custom Delimiter_ config option to specify a custom delimiter, other than newline. Provide a sequence of characters that will never appear in your source data (to make a random suggestion, something like <|^|>). Then, you can add a subsequent stage(s) to parse that data out of the result field into a different format as needed.

By the nature of origins, we expect them to return generate multiple records. In your case, the origin is returning one record per line, because you have configured the data format as TEXT and left the rest of the configuration values to their defaults. The origin is seeing each new line, parsing it into a text record, and moving on to the next line/record.

The reason the processor is only "seeing" the first line is because you have configured the data format identically (TEXT with default settings, which means record separator is newline). This means that there are actually multiple "records" associated with a single input. Some of the Data Collector processors support handling of multiple records from a processor result, but unfortunately HTTP is not one of them. See the in-progress SDC-10968 Jira for updates on that being supported.

In the meantime, I recommend you configure the TEXT format for your processor differently. Use the _Use Custom Delimiter_ config option to specify a custom delimiter, other than newline. Provide a sequence of characters that will never appear in your source data (to make a random suggestion, something like <|^|>). Then, you can add a subsequent stage(s) to parse that data out of the result field into a different format as needed.