Read from table and write to file

asked 2019-12-19 09:19:45 -0600

rajneesh75 gravatar image


I am trying to read from an impala table and write to remote server as a .csv file usingSFTP . I have an expression evaluator stage I have defined these 3 output fields expressions since these are not coming from source

/fileRef { "type": "FILE_REF", "value": { "bufferSize": 8192, "supportedStreamClasses": [""]}} /fileInfo { "type": "MAP", "value": { "filename": {"type": "STRING", "value": "test.csv"} } } /fileInfo/size { "type": "MAP", "value":{ "size": { "type": "LONG", "value": "1000000" }}}

I am not sure if I have defined all the output fields and if I have defined above fields properly. Its a simple pipeline and I don't want to go the java/groovy/jython scripting route for generating above headers. The pipeline gives runtime error on expression evaluation stage saying

EXPR_04 - Record 'select * from tablename where 1=1 order by `date::rowCount:430' cannot set value in field '/fileInfo/size'. Reason : java.lang.IllegalArgumentException: Cannot convert String field '{ "type": "MAP", "value": { "filename": {"type": "STRING", "value": "test.csv"} } }' to Map

Any ideas? Thanks

edit retag flag offensive close merge delete