Versions

Confluent schemaRegistry - 3.1.2 Hortonworks schemaRegistry - 0.3.0

Case

By mistake new required field was added to avro schema and confluent schema registry didn’t complained about that. Later, during migration schemas to Hotronworks schemaRegistry, this problem arose.

Simplified schema examples:

{
  "type": "record",
  "name": "Event",
  "namespace": "test.namespace",
  "fields": [
    {
      "name": "topOptionalField",
      "type": [
        "null",
        {
          "type": "record",
          "name": "optionalField",
          "fields": [
            {
              "name": "required1",
              "type": "int"
            }
          ]
        }
      ],
      "default": null
    }
  ]
}

next version of the schema with one more required field required2

{
  "type": "record",
  "name": "Event",
  "namespace": "test.namespace",
  "fields": [
    {
      "name": "topOptionalField",
      "type": [
        "null",
        {
          "type": "record",
          "name": "optionalField",
          "fields": [
            {
              "name": "required1",
              "type": "int"
            },
            {
              "name": "required2",
              "type": "int"
            }
          ]
        }
      ],
      "default": null
    }
  ]
}

check compatibility

public void testRealSchemas() throws IOException {
      final String fromSchemaString = IOUtils.toString(TestAvro.class.getResourceAsStream("/avro/simple_from.json"), "UTF-8");
      final String toSchemaString = IOUtils.toString(TestAvro.class.getResourceAsStream("/avro/simple_to.json"), "UTF-8");
      final Schema fromSchema = new Schema.Parser().parse(fromSchemaString);
      final Schema toSchema = new Schema.Parser().parse(toSchemaString);

      final AvroCompatibilityChecker confluentChecker = AvroCompatibilityChecker.BACKWARD_CHECKER;

      System.out.println(confluentChecker.isCompatible(fromSchema, toSchema)); //true
      System.out.println(confluentChecker.isCompatible(toSchema, fromSchema)); //true

      System.out.println(new AvroSchemaProvider().checkCompatibility(fromSchemaString,
              toSchema.toString(),
              SchemaCompatibility.BACKWARD)); //true
      System.out.println(new AvroSchemaProvider().checkCompatibility(toSchemaString,
              fromSchema.toString(),
              SchemaCompatibility.BACKWARD)); //false
}

Facts

There is a ticket in confluent schema registry. According to it, they use avro library 1.8.1. And there is a bug in avro jira. The bg in avro was fixed in 1.8.2. And this version of libarary is used for newest version of confluent schema registry. Hortonworks implements its own validator for avro schemas AvroSchemaValidator.