Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Hadoop FS: different schema than is used for the current file

I have a data pipeline from kafka to hive. The kafka message is json type, in pipeline I flatten entire fields and generate a new 'yyyy-MM-dd' field for partition, when run the pipeline it reported errors like the following. What's the error meaning? Why it happen and how to fix it?

image description

AVRO_GENERATOR_04 - Record sit_CREDIT_INVOCATION_HISTORY::0::3902 has a different schema than is used for the current file. Current schema is '{"type":"record","name":"credit_invocation_history_table","namespace":"report_sit","fields":[{"name":"creditscore","type":["null","double"],"default":null},{"name":"occurtime","type":["null",{"type":"int","logicalType":"date"}],"default":null},{"name":"terminal","type":["null","string"],"default":null},{"name":"userid","type":["null","long"],"default":null},{"name":"riskprocessid","type":["null","long"],"default":null},{"name":"eventcode","type":["null","string"],"default":null},{"name":"executionid","type":["null","long"],"default":null},{"name":"input_name","type":["null","string"],"default":null},{"name":"input_certno","type":["null","string"],"default":null},{"name":"productcode","type":["null","string"],"default":null},{"name":"creditstrategyid","type":["null","long"],"default":null},{"name":"creditdetail_ratevalue","type":["null","double"],"default":null},{"name":"creditdetail_creditenddate","type":["null","long"],"default":null},{"name":"creditdetail_ratetype","type":["null","string"],"default":null},{"name":"creditdetail_compoundperiod","type":["null","int"],"default":null},{"name":"creditdetail_amount","type":["null","double"],"default":null},{"name":"creditdetail_creditdecision","type":["null","string"],"default":null},{"name":"creditdetail_creditstartdate","type":["null","long"],"default":null}]}' whereas the record schema is '{"type":"record","name":"credit_invocation_history_table","namespace":"report_sit","fields":[{"name":"creditscore","type":["null","double"],"default":null},{"name":"occurtime","type":["null",{"type":"int","logicalType":"date"}],"default":null},{"name":"terminal","type":["null","string"],"default":null},{"name":"userid","type":["null","long"],"default":null},{"name":"riskprocessid","type":["null","long"],"default":null},{"name":"eventcode","type":["null","string"],"default":null},{"name":"executionid","type":["null","long"],"default":null},{"name":"input_name","type":["null","string"],"default":null},{"name":"input_certno","type":["null","string"],"default":null},{"name":"productcode","type":["null","string"],"default":null},{"name":"creditstrategyid","type":["null","long"],"default":null},{"name":"creditdetail_ratevalue","type":["null","double"],"default":null},{"name":"creditdetail_creditenddate","type":["null","long"],"default":null},{"name":"creditdetail_ratetype","type":["null","string"],"default":null},{"name":"creditdetail_compoundperiod","type":["null","int"],"default":null},{"name":"creditdetail_amount","type":["null","double"],"default":null},{"name":"creditdetail_creditdecision","type":["null","string"],"default":null},{"name":"creditdetail_creditstartdate","type":["null","long"],"default":null},{"name":"input_phone","type":["null","string"],"default":null}]}'