Avro to Parquet conversion in Azure Data Lake Gen2

asked 2019-12-12 16:15:29 -0600

CindyRu gravatar image

updated 2019-12-12 16:28:46 -0600

metadaddy gravatar image

We were able to do the Avro to Parquet conversion

  1. in Hadoop with the Hadoop FS destination and MapReduce Executor and
  2. in Azure Data Lake Gen2 storage with the Azure Data Lake Gen2 origin and Whole File Transformer processor

Your documentation for the "Hadoop FS" destination says that "You can also use the destination to write to Azure Blob storage."

We attempted to do the Avro to Parquet conversion in Azure Data Lake Gen2 storage with the following two routes but to no avail yet. Can they be done? Or we have to use the Whole File Transformer processor??

  1. with the Hadoop FS destination (to write to Azure Data Lake Gen2 storage) and MapReduce Executor and
  2. with the Azure Data Lake Gen2 destination and MapReduce Executor
edit retag flag offensive close merge delete

Comments

We are having the same issue as mentioned above. We are using SDC 3.11 on Azure VM.

atrivedi11 gravatar imageatrivedi11 ( 2019-12-19 13:17:00 -0600 )edit