Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate classes from json schema, demo parsing #46

Closed
wants to merge 1 commit into from
Closed

Generate classes from json schema, demo parsing #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion json_schema/java/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@ plugins {
application

id("com.diffplug.spotless") version "6.9.0"
id("org.jsonschema2pojo") version "1.1.3"
}

spotless {
java {
targetExclude("**/io/opentelemetry/fileconf/schema/*.*")
googleJavaFormat()
}
kotlinGradle {
Expand All @@ -20,7 +22,8 @@ application {

dependencies {
implementation("org.yaml:snakeyaml:1.31")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.1")
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.14.2")
implementation("com.fasterxml.jackson.core:jackson-databind:2.14.2")
implementation("com.networknt:json-schema-validator:1.0.76")

testImplementation(platform("org.junit:junit-bom:5.9.1"))
Expand All @@ -30,6 +33,13 @@ dependencies {
testImplementation("org.assertj:assertj-core:3.23.1")
}

jsonSchema2Pojo {
sourceFiles = setOf(file(project.projectDir.parent.toString() + "/schema/schema.json"))

targetPackage = "io.opentelemetry.fileconf.schema"
includeSetters = false
}

tasks {
withType<Test>().configureEach {
useJUnitPlatform()
Expand Down
2 changes: 2 additions & 0 deletions json_schema/java/settings.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@ dependencyResolutionManagement {
mavenLocal()
}
}

rootProject.name = "json-schema-java"
16 changes: 12 additions & 4 deletions json_schema/java/src/main/java/io/opentelemetry/fileconfig/Application.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package io.opentelemetry.fileconfig;

import com.fasterxml.jackson.core.type.TypeReference;
import io.opentelemetry.fileconf.schema.OpenTelemetryConfiguration;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
Expand All @@ -9,7 +11,7 @@ public class Application {

public static void main(String[] args) throws FileNotFoundException {
if (args.length != 1) {
throw new IllegalArgumentException("Missing file to validate.");
throw new IllegalArgumentException("Missing file to parse.");
}

File file = new File(args[0]);
Expand All @@ -28,11 +30,17 @@ public static void main(String[] args) throws FileNotFoundException {

YamlJsonSchemaValidator validator = new YamlJsonSchemaValidator(schemaFile);
Set<String> results = validator.validate(new FileInputStream(file));
if (results.isEmpty()) {
System.out.println("Schema successfully validated.");
} else {
if (!results.isEmpty()) {
System.out.println("Error(s) detected validating schema: ");
results.stream().map(r -> "\t" + r).forEach(System.out::println);
return;
}

System.out.println("Schema successfully validated.");

OpenTelemetryConfiguration configuration =
validator.parse(new FileInputStream(file), new TypeReference<>() {});
System.out.println("Successfully parsed schema:");
System.out.println(configuration);
}
}
12 changes: 12 additions & 0 deletions json_schema/java/src/main/java/io/opentelemetry/fileconfig/YamlJsonSchemaValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import static java.util.stream.Collectors.toSet;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.networknt.schema.JsonMetaSchema;
Expand Down Expand Up @@ -53,6 +54,17 @@ Set<String> validate(InputStream yaml) {
return jsonSchema.validate(MAPPER.readTree(yamlStr)).stream()
.map(ValidationMessage::toString)
.collect(toSet());
} catch (IOException e) {
throw new IllegalStateException("Unable to validate yaml", e);
}
}

<T> T parse(InputStream yaml, TypeReference<T> typeReference) {
// Load yaml and write it as string to resolve anchors
Object yamlObj = YAML.load(yaml);
try {
String yamlStr = MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(yamlObj);
return MAPPER.readValue(yamlStr, typeReference);
} catch (IOException e) {
throw new IllegalStateException("Unable to parse yaml", e);
}
Expand Down
157 changes: 153 additions & 4 deletions json_schema/java/src/test/java/io/opentelemetry/fileconfig/SdkSchemaTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,32 @@

import static org.assertj.core.api.Assertions.assertThat;

import com.fasterxml.jackson.core.type.TypeReference;
import io.opentelemetry.fileconf.schema.Attributes;
import io.opentelemetry.fileconf.schema.Headers;
import io.opentelemetry.fileconf.schema.Jaeger;
import io.opentelemetry.fileconf.schema.JaegerRemote;
import io.opentelemetry.fileconf.schema.Limits;
import io.opentelemetry.fileconf.schema.OpenTelemetryConfiguration;
import io.opentelemetry.fileconf.schema.Otlp;
import io.opentelemetry.fileconf.schema.ParentBased;
import io.opentelemetry.fileconf.schema.Processor;
import io.opentelemetry.fileconf.schema.ProcessorArgs;
import io.opentelemetry.fileconf.schema.Propagator;
import io.opentelemetry.fileconf.schema.Resource;
import io.opentelemetry.fileconf.schema.SamplerConfig;
import io.opentelemetry.fileconf.schema.Sdk;
import io.opentelemetry.fileconf.schema.SpanLimits;
import io.opentelemetry.fileconf.schema.TraceIDRatioBased;
import io.opentelemetry.fileconf.schema.TracerProvider;
import io.opentelemetry.fileconf.schema.TracerProviderExporters;
import io.opentelemetry.fileconf.schema.Zipkin;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.net.URI;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;

class SdkSchemaTest {
Expand All @@ -14,10 +37,136 @@ void kitchenSink() throws FileNotFoundException {
YamlJsonSchemaValidator validator =
new YamlJsonSchemaValidator(new File(System.getenv("SCHEMA_FILE")));

FileInputStream fis =
new FileInputStream(System.getenv("REPO_DIR") + "/json_schema/kitchen-sink.yaml");

// Validate example kitchen-sink file in base of repository
assertThat(validator.validate(fis)).isEmpty();
assertThat(
validator.validate(
new FileInputStream(System.getenv("REPO_DIR") + "/json_schema/kitchen-sink.yaml")))
.isEmpty();

OpenTelemetryConfiguration configuration =
validator.parse(
new FileInputStream(System.getenv("REPO_DIR") + "/json_schema/kitchen-sink.yaml"),
new TypeReference<>() {});

assertThat(configuration.getSchemeVersion()).isEqualTo(0.1);

Sdk sdk = configuration.getSdk();
assertThat(sdk).isNotNull();

Resource resource = sdk.getResource();
assertThat(resource).isNotNull();
Attributes resourceAttributes = resource.getAttributes();
assertThat(resourceAttributes).isNotNull();
assertThat(resourceAttributes.getServiceName()).isEqualTo("unknown_service");

List<Propagator> propagators = sdk.getPropagators();
assertThat(propagators).hasSize(7);
assertThat(propagators.get(0).getName()).isEqualTo("tracecontext");
assertThat(propagators.get(1).getName()).isEqualTo("baggage");
assertThat(propagators.get(2).getName()).isEqualTo("b3");
assertThat(propagators.get(3).getName()).isEqualTo("b3multi");
assertThat(propagators.get(4).getName()).isEqualTo("b3multijaeger");
assertThat(propagators.get(5).getName()).isEqualTo("xray");
assertThat(propagators.get(6).getName()).isEqualTo("ottrace");

Limits attributeLimits = sdk.getAttributeLimits();
assertThat(attributeLimits).isNotNull();
assertThat(attributeLimits.getAttributeValueLengthLimit()).isEqualTo(4096);
assertThat(attributeLimits.getAttributeCountLimit()).isEqualTo(128);

TracerProvider tracerProvider = sdk.getTracerProvider();
assertThat(tracerProvider).isNotNull();

TracerProviderExporters exporters = tracerProvider.getExporters();
assertThat(exporters).isNotNull();

Otlp otlp = exporters.getOtlp();
assertThat(otlp.getProtocol()).isEqualTo("http/protobuf");
assertThat(otlp.getEndpoint()).isEqualTo(URI.create("http://localhost:4318/v1/metrics"));
assertThat(otlp.getCertificate()).isEqualTo("/app/cert.pem");
assertThat(otlp.getClientKey()).isEqualTo("/app/cert.pem");
assertThat(otlp.getClientCertificate()).isEqualTo("/app/cert.pem");
assertThat(otlp.getCompression()).isEqualTo("gzip");
assertThat(otlp.getTimeout()).isEqualTo(10000);
Headers headers = otlp.getHeaders();
assertThat(headers).isNotNull();
assertThat(headers.getAdditionalProperties()).isEqualTo(Map.of("api-key", 1234));

Zipkin zipkin = exporters.getZipkin();
assertThat(zipkin).isNotNull();
assertThat(zipkin.getEndpoint()).isEqualTo(URI.create("http://localhost:9411/api/v2/spans"));
assertThat(zipkin.getTimeout()).isEqualTo(10000);

Jaeger jaeger = exporters.getJaeger();
assertThat(jaeger).isNotNull();
assertThat(jaeger.getProtocol()).isEqualTo("http/thrift.binary");
assertThat(jaeger.getEndpoint()).isEqualTo(URI.create("http://localhost:14268/api/traces"));
assertThat(jaeger.getTimeout()).isEqualTo(10000);
assertThat(jaeger.getUser()).isEqualTo("user");
assertThat(jaeger.getPassword()).isEqualTo("password");
assertThat(jaeger.getAgentHost()).isEqualTo("localhost");
assertThat(jaeger.getAgentPort()).isEqualTo(6832);

List<Processor> spanProcessors = tracerProvider.getSpanProcessors();
assertThat(spanProcessors).hasSize(3);

Processor batchProcessor = spanProcessors.get(0);
assertThat(batchProcessor).isNotNull();
assertThat(batchProcessor.getName()).isEqualTo("batch");
ProcessorArgs args = batchProcessor.getArgs();
assertThat(args).isNotNull();
assertThat(args.getScheduleDelay()).isEqualTo(5000);
assertThat(args.getExportTimeout()).isEqualTo(30000);
assertThat(args.getMaxQueueSize()).isEqualTo(2048);
assertThat(args.getMaxExportBatchSize()).isEqualTo(512);
assertThat(args.getExporter()).isEqualTo("otlp");

batchProcessor = spanProcessors.get(1);
assertThat(batchProcessor).isNotNull();
assertThat(batchProcessor.getName()).isEqualTo("batch");
args = batchProcessor.getArgs();
assertThat(args.getExporter()).isEqualTo("zipkin");

batchProcessor = spanProcessors.get(2);
assertThat(batchProcessor).isNotNull();
assertThat(batchProcessor.getName()).isEqualTo("batch");
args = batchProcessor.getArgs();
assertThat(args.getExporter()).isEqualTo("jaeger");

SpanLimits spanLimits = tracerProvider.getSpanLimits();
assertThat(spanLimits).isNotNull();
assertThat(spanLimits.getAttributeValueLengthLimit()).isEqualTo(4096);
assertThat(spanLimits.getAttributeCountLimit()).isEqualTo(128);
assertThat(spanLimits.getEventCountLimit()).isEqualTo(128);
assertThat(spanLimits.getLinkCountLimit()).isEqualTo(128);
assertThat(spanLimits.getEventAttributeCountLimit()).isEqualTo(128);
assertThat(spanLimits.getLinkAttributeCountLimit()).isEqualTo(128);

SamplerConfig samplerConfig = tracerProvider.getSamplerConfig();
assertThat(samplerConfig).isNotNull();
// always_on and always_off are null because they have no properties.
// assertThat(samplerConfig.getAlwaysOn()).isNotNull();
// assertThat(samplerConfig.getAlwaysOff()).isNotNull();
TraceIDRatioBased traceIdRatioBased = samplerConfig.getTraceIdRatioBased();
assertThat(traceIdRatioBased).isNotNull();
assertThat(traceIdRatioBased.getRatio()).isEqualTo(.0001);
ParentBased parentBased = samplerConfig.getParentBased();
assertThat(parentBased).isNotNull();
assertThat(parentBased.getRoot()).isEqualTo("trace_id_ratio_based");
assertThat(parentBased.getRemoteParentSampled()).isEqualTo("always_on");
assertThat(parentBased.getRemoteParentNotSampled()).isEqualTo("always_off");
assertThat(parentBased.getLocalParentSampled()).isEqualTo("always_on");
assertThat(parentBased.getLocalParentNotSampled()).isEqualTo("always_off");
JaegerRemote jaegerRemote = samplerConfig.getJaegerRemote();
assertThat(jaegerRemote).isNotNull();
assertThat(jaegerRemote.getEndpoint()).isEqualTo(URI.create("http://localhost:14250"));
assertThat(jaegerRemote.getPollingInterval()).isEqualTo(5000);
assertThat(jaegerRemote.getInitialSamplingRate()).isEqualTo(.25);

assertThat(tracerProvider.getSampler()).isEqualTo("parent_based");

// TODO: add assertions for MeterProvider, LoggerProvider

System.out.print(configuration);
}
}
17 changes: 14 additions & 3 deletions json_schema/schema/schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@
"type": "string"
},
"args": {
"$ref": "#/definitions/LogRecordProcessorArgs"
"$ref": "#/definitions/ProcessorArgs"
}
},
"required": [
Expand All @@ -175,7 +175,7 @@
],
"title": "Processor"
},
"LogRecordProcessorArgs": {
"ProcessorArgs": {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a general ProcessorArgs type, or separate the types for LogRecordProcessor and SpanProcessor?

Also, the properties here are specific to batch processor. Will need to rethink how to define this type to accommodate simple, batch, and extension processors.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ProcessorArgs is ok until we have a reason to be more specific.

I suspect we'll want per type of processor arguments, much like we'll want the same for exporters. Though I guess that will only support the known processor/exporter. We should plan to provide documentation on how one would define and publish their own schemas if they wanted to support additional components

"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -198,7 +198,7 @@
"required": [
"exporter"
],
"title": "LogRecordProcessorArgs"
"title": "ProcessorArgs"
},
"MeterProvider": {
"type": "object",
Expand Down Expand Up @@ -460,6 +460,17 @@
"TracerProviderExporters": {
"type": "object",
"additionalProperties": true,
"properties": {
"otlp": {
"$ref": "#/definitions/Otlp"
},
"zipkin": {
"$ref": "#/definitions/Zipkin"
},
"jaeger": {
"$ref": "#/definitions/Jaeger"
}
},
"patternProperties": {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated classes aren't able to parse properties based on patterns. The code interpreting the parsed results will have to manually detect these patterns and parse the results as the corresponding type. We should minimize use of patternProperties to reduce this extra work.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there's a way to define an unmarshaling code in one place through some interface that can then be re-used. The collector does this by defining a component.ID that handles the parsing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did manage to customize the code generation such that a hook can be added for custom deserialization. This solves the problem, but I came across other issues in the process:

  • The code generation tool I used doesn't generate classes for definitions that aren't referenced, and it appears that pattern property references to the otlp, jaeger, and zipkin don't count. To solve this I had to break out those definitions into standalone files.
  • Once some of the definitions were split out from a single file, I had to tell the validation tool how to resolve URIs to definitions that live outside the file.

I imagine implementations in other languages will encounter similar issues.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So embedded reference definitions don't work but reference definitions in separate files does?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup. There's an issue tracking it, but its been stagnant for a couple of years now. This comment explains it:

When jsonschema2pojo was created, 'definitions' was not part of the standard. Many people started using 'definitions' to hold extra schemas but we didn't add support here because it was just a common pattern but not part of the standard.

"^otlp.*": {
"$ref": "#/definitions/Otlp"
Expand Down