Data Transformation Utilities Using JSON Schemas #16
Comments
|
@benjagm plzz assign me this issue |
|
Hi @affaannn, |
Aight thank you |
|
Hi, Hope you're doing well. I have been looking at the project ideas for GSoC and this one really intrigues me. I am confident that my skills and experience make me an ideal candidate for this project. I have a strong understanding of JSON Schema and its applications. I am comfortable working with many programming languages such as JavaScript, C++, Python and Java to name a few and can implement the required utilities using the latest version of JSON Schema. I have experience in writing tests in JSON, and I understand the importance of ensuring compatibility with different languages. The expected outcomes of the project are exciting, and I am particularly interested in providing a utility that takes a JSON Schema and a query string from a form post and converts it to JSON using the JSON Schema to know the types for the values. I believe that my experience in developing web applications and working with JSON data will be beneficial in achieving success in this area. I am fully committed to this project and can dedicate at least 175 hours to it. I am excited about the opportunity to contribute to this project and learn from the mentors. Looking forward to more details about how one can contribute to this project. |
|
@jansche where can we find the initial qualifying tasks? Have they been added or yet to be added. |
|
Hello everyone, I am a 3rd year Computer Science student at NIT Bhopal, India. Over the past few years, I have worked on several Node.js projects and gained experience working with JSON schema. I am familiar with the utilities need that has been described in this project and these are some of utilities that even I wanted for my personal use in my projects. Having developed the backend of several applications using Node.js, I am excited to take on the challenge of implementing all the utilities mentioned above as a GSoC mentee this summer. I am looking forward to completing the initial qualifying tasks and working with the community. Thank you. |
|
Hello Everyone I'm reaching out to express my interest in contributing to Postman's GSOC 2023. I have already studied the project documentation and guidelines and I have set up the development environment on my local machine. I would greatly appreciate any guidance or assistance that you can provide in getting started with contributing to the project. I actually hve some questions regarding this idea and also abput application process of GSOC2023
|
|
Hello Everyone, |
|
Hi folks, |
|
Hi everyone 👋. Thanks for your interest in this project. I'd like to invite everyone to join the JSON Schema slack to get involved with the community. It's a great place to ask questions or discuss JSON Schema. If you're new to JSON Schema, have a look at Understanding JSON Schema. |
|
A little more about this project ... For this project, you won't be contributing code to an existing application or library, you'll be the maintainer of a new library in the JSON Schema ecosystem. You can write it in any language you like, but you'll probably want to use existing components wherever possible. For example, the JavaScript library @hyperjump/json-schema (that I maintain) has utilities for working with JSON Schemas so you can focus on solving the problem at hand rather than having to build a whole new JSON Schema implementation. |
|
Hi, I'm Francielle Dellamora, a 21yo Brazilian web developer passionate about TypeScript. This project interested me because it uses technologies I am already familiar with and because it is similar to Zod, so I think it will be a fun programming challenge and an excellent opportunity to further my knowledge of schema validation you can learn more about me on my website or @francidellamora (: |
|
Hi everyone! Thanks a lot for your interest in this project Idea and JSON Schema! We'd like to run an Office Hours Session tomorrow at 09:00 PST, 18:00 CET, 22:30 IST, 14:00 BRT In the meantime we invite you to join JSON Schema Slack server in the channel #gsoc-2023 using this link: https://join.slack.com/t/json-schema/shared_invite/zt-1qnbilvie-_Tvbr6YQL29l2HZBNvCiDQ @dellamora @Atish03 @affaannn @ankit-pn @VishrutAggarwal @ishaan812 |
|
Hi Everyone, My name is Segun Adodo, I have a background in Information System. This particular GSoC project idea catches my fancy and I would be happy to contribute. Given that I am proficient and comfortable programming in Java and JavaScript, I believe I am suitably positioned to successfully deliver the project if given the opportunity. I am happy to bring my software development experience to bear on this project. One thing that interests me in this project is the opportunity to learn about JSON Schema (and perhaps other concepts/techniques/tools, not as obvious to me at the moment) as I haven't had any reason to use or study about it before. Another key attraction, is the excitement, sense of purpose and feeling of accomplishment that comes with building open-source utilities that can be used by anyone and anywhere in the world for free. My mentors and project coordinators can derive the best from me by providing guidance especially when I request for one, but most importantly, leaving me to mostly figure out things on my own works well for me since I learn better this way. Whilst working with your organization, I will be studying Python programming language but without impacting my timing and deliverables commitment to this project. This undertaking is already incorporated in my personal development plan for the period long before now. I prefer to simplify my experience when working on projects to encourage productivity. For instance, I'd rather use intuitive project folder structure, git for version control, task manager to track tasks and calendar reminders to avoid losing track of delivery/meetings timelines. I am more comfortable organizing my tasks into small bits that are measurable and highly suggestive of my progress. It offers me some psychological good this way. I look forward to the qualifying task as well as sharing a proposal on project delivery weekly schedule complete with associated milestones and deliverables. Many thanks!!! |
|
Hey guys, Just a few weeks ago, I was trying to setup Snowplow Analytics for our platform and trying to figuring out its complex setup locally. Which is when I came across the self described schemas for enrichment that they used. I was quite interested about how the enrichment process worked. I did not quite look into it back then. Then I was just looking for GSOC projects I would like to contribute to and found this. JSON Schema was what they were using for their process. Also working with a giant codebase with multiple languages on multiple surfaces connected by a lot of REST API's. I have understood how much the consistency of type safety which JSON schema helps with across languages is so important. How much easier it makes the debugging process and improves the communication between languages. Data Transformations, just as an example adding default values would be super helpful to further enhance the scope of this project and lower the boilerplate backend code required and I would love to play a part in that. I have for the most part, always worked remotely with teams. The necessity to over communicate in such an environment and adhering to the process becomes even more important to enhance efficiency. Also the need of well made documentation cannot be understated. I like to clearly communicate about my tasks and have a clear plan before I can get going on it on my own and start coding. I like to take ownership of the tasks provided to me. By the time GSOC begins, I wont be committed to any other major work since it will be in the middle of my summer break. So I will be able to provide the time necessary for the execution of this project. I take a lot of pride in my setup and organization, you will never catch me working on VSCode with Prettify and all the beautification you can get, along with nice folder structures. Among other non-development tools, I'm a big TO-DO list guy who likes to get notifications about every small event in the day since I tend to forget. For that I use Todoist. |
|
Greetings, everyone! My name is Semih, and I'm thrilled to be here to share my passion for software development. As a freshman software student, I have been working on various backend projects using technologies like JavaScript, TypeScript, Node.js, MongoDB, Redis and PostgreSQL. I also have familiarity in front-end technologies such as HTML, CSS, and React.js. In addition to my technical skills, I have been actively contributing to the community as a Postman Student Expert for a year. My role involves helping students with APIs and Postman, and this experience has helped me develop my teamwork, communication, and problem-solving skills. In summary, I am a passionate and motivated MIS student with a keen interest in backend engineering. I am excited about the opportunity to work on the GSoC 2023 project on JSON Schema data transformation where I am planning to implement this project using JavaScript and Ajv. I am committed to delivering high-quality and efficient implementations of the requested utilities, and I look forward to the opportunity to work with the mentors and the community to achieve this goal. |
|
Hello everyone! My name is Tarun, a pre final year student from BITS Pilani, India. I find this project very interesting as JSON Schema is a widely-used format for defining the structure and content of JSON data. By building data transformation utilities that use JSON schemas, this project provides a powerful way to manipulate and transform JSON data. The project's emphasis on data transformation tools shows that it is intended to address real-world issues that developers and data analysts confront. This implies that a variety of real-world situations are likely to benefit from the code and tools created via this initiative.By itself, JSON schemas are an effective tool for data validation and documentation. This project has the ability to enhance the quality of the data and decrease mistakes in downstream processes by utilising the benefits of JSON schemas in the data transformation process. As mentors and project coordinators you can get the best out of me by providing clear project goals and expectations,provide detailed information and context,give feedback and correction when required.As i will be working on this project during my summer break,i can allot as much time as needed. I keep myself organized my having a schedule to do my dialy tasks using tools such as notion,google calender etc. Looking forward to contribute to this project and learning a ton working with the mentors. |
|
Hi everyone. Thanks again for your interest in this project. In the contributor guidance document, it mentions picking up a "first issue" task for the project and "mandatory qualification task". However, that only makes sense for contributing to an existing project and this project will be built from scratch. Creating a repo and setting it up for development, testing, publishing, and contributions will all be a part of your responsibilities. So, I came up with a qualification task that we can do without a repo or any code. Qualification TaskProvide a test case that can be used as an acceptance test for any of the transformation operations described in the project description. This test should be expressed in JSON and include the inputs and expected output for the transformation operation. You can post your tests in this thread. Look over the tests others have posted before you and make sure someone hasn't already posted something testing the same thing. You can show that you understand the complexities you'll face with this project by coming up with a non-trivial test case. Example for applying defaults {
"description": "Given a schema with a default set for the 'foo' property",
"schema": {
"type": "object",
"properties": {
"foo": {
"type": "number",
"default": 42
}
},
"tests": [
{
"description": "When the schema is applied to an empty object, the 'foo' property should be set to the default value",
"instance": {},
"expected": {
"foo": 42
}
},
{
"description": "When the schema is applied to an object where 'foo' exists, the 'foo' property should not change",
"instance": { "foo": 3 },
"expected": {
"foo": 3
}
}
]
} |
|
HI everyone, My name is Atish Shah, I am pursuing Bachelor of Technology in Engineering Physics at Indian Institute of Technology Dhanbad, India and am currently a Sophomore. The thing that interests me the most about this project is its usefulness and Importance. JSON Schema are essential for data validation and structuring the data as intended by various applications. I like the idea of providing the API with some data and getting out an uniform output, like providing it with URL encoded data and getting out structured and useful data out of it. As a mentor, you just have to provide me with a clear result and I will make sure to get that as efficiently and effectively as possible. Since I will be working on this project in my summer break, there wont be any interference with the project and I would be able to commit all my time to this project. I like working in daytime and finishing my projects at least reasonable time before the deadline. As for the tools I use to keep myself organised, I use Tasks, Google Calendar and for projects I use Docker and GitHub. I am very excited to contribute to this project and am looking forward for some inputs from mentors. |
|
@jdesrosiers, we just have to provide with a json object like as a test case here, Right ? |
|
Hi everyone!! Thanks a lot for your interest in JSON Schema ideas. We invite you to join JSON Schema Slack server in the channel #gsoc-2023 Join the server using this link In #gson-2023 channel you will find there: 1)resources, 2)recording of the last office hours session, 3)clear instructions to prepare your submission to this Project Idea and 4) the best place to get immediate support. JSON Schema is all about Community and the Community is in Slack! See you there! |
|
@Atish03 No, your submission should look more like my example. You should provide a schema that will produce interesting behavior (hopefully more interesting than the simple one I used) and include at least one example of how that schema would transform one or more instances. Also, don't forget to tell me which transformation your tests are covering: filling in defaults, removing additional keywords, etc. EVERYONE: If you have questions, let's do that on slack. Then you can come here to post your submissions. That way people don't need to wade through a bunch of discussion to see what others have submitted. |
|
This is my test case for the qualification task where it includes a "language" property and an "information" property. |
|
Test case for applying defaults with some of the properties as required. |
|
Test case for removing properties from JSON document not described in the schema. |
|
Here's my solution to the qualification task. It applies defaults. |
|
Here are the test cases I cooked up: Here's my thought and work process:
|
|
Hello Community & @jdesrosiers,
This should cover most of the edge cases and scenarios that may arise when transforming a JSON document based on a JSON Schema with default values.
In this test case, the input JSON document has a property called "email" which is not described in the JSON Schema. The expected output JSON document should have the "email" property removed. The utility should only keep the properties that are described in the JSON Schema.
|
Test that verifies an invalid property is removed from the JSON document. {
"description": "This schema defines the JSON structure for a Product and a Package.",
"schema": {
"anyOf": [
{ "$ref": "#/definitions/Product" },
{ "$ref": "#/definitions/Package" }
],
"definitions": {
"Product": {
"type": "object",
"properties": {
"name": { "type": "string" },
"value": { "type": "number" }
},
"required": ["name", "value"]
},
"Package": {
"type": "object",
"properties": {
"name": { "type": "string" },
"value": { "type": "number" },
"products": {
"type": "array",
"items": { "$ref": "#/definitions/Product" }
}
},
"required": ["name", "products"]
}
}
},
"tests": [
{
"description": "Test that valid JSON document remains unchanged",
"instance": {
"name": "Product 1",
"value": 10
},
"expected": {
"name": "Product 1",
"value": 10
}
},
{
"description": "Test that invalid property is removed from JSON document",
"instance": {
"name": "Product 2",
"value": 20,
"color": "blue"
},
"expected": {
"name": "Product 2",
"value": 20
}
},
{
"description": "Test that invalid nested property is removed from JSON document",
"instance": {
"name": "Package 1",
"value": 100,
"products": [
{
"name": "Product 3",
"value": 30,
"color": "red"
},
{
"name": "Product 4",
"value": 40
}
],
"color": "green"
},
"expected": {
"name": "Package 1",
"value": 100,
"products": [
{
"name": "Product 3",
"value": 30
},
{
"name": "Product 4",
"value": 40
}
]
}
} |
Qualifying test for transforming a url into urlParametersFor my qualification task, I've used a custom function and default for fallback values.
I've also added a validity flag on each of the tests since I felt it was relevant to show the results properly. |
|
Hello everyone. My name is Kiran, and I am a pre-final year student at IIT Kharagpur. I am a full-stack developer. I am very interested in working on this project. I have worked mostly on backend development and used JSON schema very much, and this project is the right fit for me. JSON is essential for backend development, so any improvement is helpful to many people. I have mainly used Nodejs for most of my projects but I also interned as a Python developer. The project has so much potential like if we are trying to add new features to our project, we can add the latest properties to the schema and any default values, if any, and send all our data here to modify and similar for other cases. I am very excited to learn new things that come along the way working with everyone. |
Tests for Utility 1: A Utility that takes a JSON Schema and a JSON document and adds to the document any values that are missing but have a default specified for that value in the JSON Schema.{
"description":"Circular Schema Representing Tree Data Structure",
"schema":{
"type": "object",
"properties": {
"node_name": { "$ref":"#/$defs/name"},
"node_value": { "$ref": "#/$defs/value"},
"children": {
"type": "array",
"items": { "$ref": "#" }
}
},
"$defs":{
"name": { "type": "string", "default": "Untitled Node"},
"value": {"type" : "number", "default": 0}
}
},
"tests": [
{
"description":"When schema is applied to empty object, value set to default value",
"instance": {},
"expected": { "node_name": "Untitled Node", "node_value": 0 }
},
{
"description": "When schema is applied to object with missing value, value set to defalut value",
"instance": {
"node_name": "root",
"node_value": 10,
"children": [
{
"node_name": "child 1",
"children": [
{
"node_name": "grandchild 1"
},
{
"node_value": 15
}
]
},
{
"node_value": 5
}
]
},
"expected": {
"node_name": "root",
"node_value": 10,
"children": [
{
"node_name": "child 1",
"children": [
{
"node_name": "grandchild 1",
"node_value": 0
},
{
"node_value": 15,
"node_name": "Untitled Node"
}
],
"node_value": 0
},
{
"node_value": 5,
"node_name": "Untitled Node"
}
]
}
}
]
}
Tests for Utility 2: A Utility that takes a JSON Schema and a JSON document and removes properties in the JSON document that are not described in the JSON Schema. {
"description":"Circular Schema Representing Tree Data Structure",
"schema":{
"type": "object",
"properties": {
"node_name": { "$ref":"#/$defs/name"},
"node_value": { "$ref": "#/$defs/value"},
"children": {
"type": "array",
"items": { "$ref": "#" }
}
},
"$defs":{
"name": { "type": "string"},
"value": {"type" : "number"}
}
},
"tests":[
{
"description":"Test case for which object is empty",
"instance":{},
"expected":{}
},
{
"description":"Test case of object for which properties that is defined is not present in schema",
"instance": {
"node_name": "root",
"node_value": 10,
"children": [
{
"node_name": "child 1",
"children": [
{
"node_name": "grandchild 1",
"node_number": 10,
"node_value": 0
},
{
"node_value": 15,
"node_index": 6,
"node_name": "grandchild 2"
}
],
"node_value": 0
},
{
"node_value": 5,
"node_name": "Untitled Node"
}
]
},
"expected":{
"node_name": "root",
"node_value": 10,
"children": [
{
"node_name": "child 1",
"children": [
{
"node_name": "grandchild 1",
"node_value": 0
},
{
"node_value": 15,
"node_name": "grandchild 2"
}
],
"node_value": 0
},
{
"node_value": 5,
"node_name": "Untitled Node"
}
]
}
}
]
}Tests for Utility 3: A Utility that takes a JSON Schema and a query string from a form post (application/x-www-form-urlencoded) and convert the string to JSON using the JSON Schema to know the types for the values.{
"tests":[
{
"description":"When Schema is applied to empty string",
"schema":"",
"instance":"",
"expected":""
},
{
"description":"When Schema is applied to string that matches schema",
"schema":{
"description": "Schema Representing Foo Object",
"type": "object",
"properties": {
"foo": {"type": "string"},
"num": {"type": "number"},
"user": {
"type": "object",
"required": ["name", "age"],
"properties": {
"name": {"type": "string"},
"age": {"type": "number"}
}
},
"address": {
"type": "object",
"required": ["city", "state", "zip"],
"properties": {
"city": {"type": "string"},
"state": {"type": "string"},
"zip": {"type": "string", "pattern": "^[0-9]{5}$"}
}
},
"colors": {"type": "string"},
"options": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": ["value", "label"],
"properties": {
"value": {"type": "string"},
"label": {"type": "string"}
}
}
}
}
},
"instance":"?foo=bar&num[]=1&num[]=2&user[name]=John+Doe&user[age]=30&address[city]=New+York&address[state]=NY&address[zip]=10001&colors[]=red&colors[]=blue&colors[]=green&options[0][value]=option1&options[0][label]=Option+1&options[1][value]=option2&options[1][label]=Option+2&options[2][value]=option3&options[2][label]=Option+3",
"expected": {
"foo": "bar",
"num": 2,
"user": {
"name": "John Doe",
"age": 30
},
"address": {
"city": "New York",
"state": "NY",
"zip": "10001"
},
"colors": "green",
"options": [
{
"value": "option1",
"label": "Option 1"
},
{
"value": "option2",
"label": "Option 2"
},
{
"value": "option3",
"label": "Option 3"
}
]
}
}
]
} |
|
Here is my submission for the qualifying task |
|
Resubmission for Qualification Task |
|
Submission for Qualifination Task |
|
hello, what about this
|
|
@abhay-ahirkar, have you joined our Slack ( |
And ### submission for qualification task Hi! I can definitely help you with the qualification task. Here's my test case for the "rename" transformation operation: { I hope this test case demonstrates my understanding of the complexities involved in creating non-trivial test cases. Let me know if you have any questions or need further clarification. |
Utility to run defaults on nested Arrays |
|
Re-submission for qualifying task: |
|
Thanks a lot for your hard work with your messages and qualifying tasks! We are looking forward to receive your applications. Please ask us for feedback with your applications anytime, we'd love to help. Please, join the JSON Schema Slack server using this link and join channel #gsoc-2023 to stay up to date with all the communications and announcements. |
|
|
Closed as completed as part of 2023 edition. |
Brief Description
JSON Schema is designed for validation, but the information in a schema can be used to for a number of other purposes. One of those purposes is data transformation. This project aims to implement some of the transformation uses cases JSON Schema Community has been asking about the last months.
Expected Outcomes
Provide an implementation of the following utilities in any language. They should work with at least the latest version of JSON Schema. Tests should be described in JSON so others who want to write a similar tool in a different language can easily use your tests to ensure compatibility.
defaultspecified for that value in the JSON Schema.application/x-www-form-urlencoded) and convert the string to JSON using the JSON Schema to know the types for the values.?foo=1with schema{ "properties": { "foo": { "type": "number" } } }should return{ "foo": 1 }rather than{ "foo": "1" }.Skills Required
Mentors
@jdesrosiers
Project Repo: tbd
Expected Difficulty
Easy
Expected Time Commitment
175
The text was updated successfully, but these errors were encountered: