Ask Your Question
1

Avro schema validation failed

asked 2018-07-03 06:29:14 -0500

kutny gravatar image

I'm not able to set up Avro validation correctly. Any idea what is wrong?

streamsets

avro schema

Is it somehow possible to select Avro validation schema based on expression (e.g. table name)? Right now I'm only able to set one Schema ID per pipeline which is not very practical.

Thanks

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-07-03 14:50:51 -0500

metadaddy gravatar image

Your schema is saying that the output record should have two fields in its root, id and name; I can see that the record contains /BinLogFilename, /Type, /Table etc but not /id and /name - I suspect they're somewhere in /Data. If you manipulate the record to get the right data into those fields (Expression Evaluator or Field Renamer are useful for this), the destination should be able to write them out.

You can't configure multiple schema ID's per pipeline (that would be an interesting enhancement, to allow expressions in the Schema ID field, if you'd like to suggest it at https://issues.streamsets.com), but you can specify the Avro schema in the record header. You could configure a Static Lookup processor to provide the correct schema based on the value of some field.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2018-07-03 06:29:14 -0500

Seen: 36 times

Last updated: Jul 03