Ask Your Question

How to convert Avro files into Date based folders?

asked 2019-01-28 15:14:26 -0500

nerd101 gravatar image

updated 2019-01-29 11:07:39 -0500

metadaddy gravatar image

I am trying this one pipeline whose flow is like

directory->field merger->field pivoter->field flattener->local fs
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-01-29 11:07:03 -0500

metadaddy gravatar image

You can set Time Basis in the Local FS destination Output Files tab to an expression that evaluates to a DATETIME, then set the Directory Template using Datetime Variables such as ${YYYY()}}, ${MM()} and ${DD()}.

In this example, the /pickup_datetime field contains a STRING value such as "2013-01-13 04:46:02", so we use ${time:extractDateFromString()} to convert it to a DATETIME:

image description

edit flag offensive delete link more


it shows error on '/pickup_datetime,

nerd101 gravatar imagenerd101 ( 2019-01-30 15:14:39 -0500 )edit

The expression I showed was just an example - you'll have to adapt it to your use case.

metadaddy gravatar imagemetadaddy ( 2019-01-30 15:58:31 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-01-28 15:14:26 -0500

Seen: 20 times

Last updated: Jan 29