Understanding the FULL compatibility of APICURIO

[AhmedRaafat14]

hey folks! Feeling a bit stupid here and need your help! 🙏

I have been playing with APICURIO lately, especially if I enable FULL compatibility as a global rule.

I'm using Json type req.Header.Set("X-Registry-ArtifactType", "JSON")

Then add a new schema like this:

{
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  },
  "required":["foo"],
  "additionalProperties": { "type": "string" }
}

Then if I try to update it I call the create endpoint with ?ifExists=RETURN_OR_UPDATE&canonical=true with a new version of the schema like this:

{
  "type": "object",
  "properties": {
    "foo": { "type": "string" },
    "bar": { "type": "string" }
  },
  "required":["foo"],
  "additionalProperties": { "type": "string" }
}

Isn't this considered a full compatible version?
If not, then what it is?
If yes, then why i get OBJECT_TYPE_PROPERTY_SCHEMAS_NARROWED error?

even if i try with:

{
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  },
  "required":["foo"]
}

then

{
  "type": "object",
  "properties": {
    "foo": { "type": "string" },
    "bar":{ "type": "string" }
  },
  "required":["foo"]
}

I'm using registry.hub.docker.com/apicurio/apicurio-registry-mysql:2.6.x-release

APICURIO Logs

     2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Selecting artifact (latest version) meta-data: testing test-record2 (behavior = SKIP_DISABLED_LATEST)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 2 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Inserting an artifact row for: testing test-record2
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.metrics.health.liveness.LivenessUtil] (executor-thread-187) Ignored intercepted exception: io.apicurio.registry.storage.ArtifactAlreadyExistsException :: An artifact with ID 'test-record2' in group 'testing' already exists.
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.metrics.health.liveness.LivenessUtil] (executor-thread-187) Ignored intercepted exception: io.apicurio.registry.storage.ArtifactAlreadyExistsException :: An artifact with ID 'test-record2' in group 'testing' already exists.
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.metrics.health.liveness.LivenessUtil] (executor-thread-187) Ignored intercepted exception: io.apicurio.registry.storage.ArtifactAlreadyExistsException :: An artifact with ID 'test-record2' in group 'testing' already exists.
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Selecting artifact (latest version) meta-data: testing test-record2 (behavior = SKIP_DISABLED_LATEST)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Selecting artifact (latest version) meta-data: testing test-record2 (behavior = DEFAULT)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.metrics.health.liveness.LivenessUtil] (executor-thread-187) Ignored intercepted exception: io.apicurio.registry.storage.ArtifactNotFoundException :: No artifact with ID 'test-record2' in group 'testing' was found.
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Selecting artifact (latest version) meta-data: testing test-record2 (behavior = SKIP_DISABLED_LATEST)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (executor-thread-187) Getting a list of all artifact rules for: testing test-record2
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 WARN <_> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (executor-thread-187) Datasource '': JDBC resources leaked: 1 ResultSet(s) and 0 Statement(s)
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): ]Visiting ObjectSchemaWrapper(wrapped={"type":"object","required":["foo"],"properties":{"bar":{"type":"string"},"foo":{"type":"string"}}}) at #
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): /properties/foo]Visiting StringSchemaWrapper(wrapped={"type":"string"}) at #/properties/foo
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): /properties/foo]Visiting formatValidator unnamed-format
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): ]Visiting ObjectSchemaWrapper(wrapped={"type":"object","required":["foo"],"properties":{"foo":{"type":"string"}}}) at #
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): /properties/foo]Visiting StringSchemaWrapper(wrapped={"type":"string"}) at #/properties/foo
2024-08-02 07:49:21 DEBUG <_> [io.apicurio.registry.rules.compatibility.jsonschema.diff.DiffContext] (executor-thread-187) [Context path (updated): /properties/foo]Visiting formatValidator unnamed-format
2024-08-02 07:49:21 INFO <_> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-187) apicurio.audit action="createArtifact" result="failure" src_ip="172.18.0.5" if_exists="RETURN_OR_UPDATE" artifact_type="JSON" error_msg="Incompatible artifact: test-record2 [JSON], num of incompatible diffs: {1}, list of diff types: [OBJECT_TYPE_PROPERTY_SCHEMAS_NARROWED at /propertySchemasAdded] Causes: OBJECT_TYPE_PROPERTY_SCHEMAS_NARROWED at /propertySchemasAdded" group_id="testing" name="test-record2" canonical="true" artifact_id="test-record2" version="1.1.0"
   

Thanks

[apicurio-bot]

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

[AhmedRaafat14]

@jsenko any input on this?

[jsenko]

Hi, you're right, the first two examples should be full compatible, that is a bug. The compatibility checker does not take into account the fact that sub-schemas for bar and additionalProperties are full compatible.

The example #3 and #4 are not full compatible, because

{
  "foo": "this is a string",
  "bar": 42
}

is valid for #3 but not #4.

[AhmedRaafat14]

Thanks, @jsenko for the response.

You are correct in #3 and #4 I have tried with additionalProperties: false in the schema and still get the same error.

Is there a work-in-progress to fix this bug, and can I monitor my issue and an MR will mention it when it is open?
Also is there an ETA for that to be fixed?

Thanks a lot!

[jsenko]

Hi, I don't have an ETA, but I'll mention this issue in the PR, so I'll be linked. As a workaround, you can relax the compatibility rule temporarily and then set it back to FULL.

[AhmedRaafat14]

Hi @jsenko !

I need your help a bit if you have time, I tried fixing that issue with compatibility with a patch of checking the schema diff:

private boolean isOptionalPropertyDiff(CompatibilityDifference diff) {
        return diff.asRuleViolation().getDescription()
                .equalsIgnoreCase(DiffType.OBJECT_TYPE_PROPERTY_SCHEMAS_EXTENDED.getDescription())
                || diff.asRuleViolation().getDescription()
                        .equalsIgnoreCase(DiffType.OBJECT_TYPE_PROPERTY_SCHEMAS_NARROWED.getDescription());
    }

Then filter out those, if I do this in the app/src/main/java/io/apicurio/registry/rules/compatibility/CompatibilityRuleExecutor.java it works fine am not sure though if this is the correct design. Check this commit and this works fine.

So, I moved this down to the Checker itself like in this commit, but this doesn't work it seems to me that it is not picking up the new function handleDifferencesBasedOnLevel for some reason that I can't locate.

I'm new to this so could you help me understand the flow?
My understanding is that:

• The rule executed is getting called and based on the switch case it determines that it will call CompatibilityRuleExecutor.
• Then in CompatibilityRuleExecutor the code gets the correct checker provider.getCompatibilityChecker() and I can see that it picks the correct one in my case which is JsonSchemaCompatibilityChecker.

Thanks 🙏