-
Notifications
You must be signed in to change notification settings - Fork 16
AsyncAPI Applications Relations Finder #1
Comments
I am trying to get myself familiar with |
@derberg Also can you please point out to some already implemented |
@arjungarg07 we do not have anything out of the box. These either would have to be created from scratch or you can try out https://solace.com/try-it-now/ that gives an event portal where you can design a system in a kind of design tool. You first create objects, connect them to explain how they are related and then you can export AsyncAPI files from there. Decide on your own what is better, play with some tool that already supports AsyncAPI so you learn how tooling can support the spec, or you learn the spec by just creating few files for some fake system, and try to "connect" them (I can of course help). you can also wait few days and there is Alvaro that works on a blog post on a real system based on AsyncAPI and he has AsyncAPI documents there, he just needs to do some fixes to that. I can keep you posted, especially that you will anyway be busy with the onboarding tasks from the parser |
Yeah, thanks for sharing the design tool. I would try to build AsyncAPI files as per the use cases described in the idea and play with it to draw some relations manually first. |
@arjungarg07 Alvaro's blog post is out https://www.asyncapi.com/blog/building-async-flight-notification-service and the project where he has AsyncAPI files is here https://github.com/amadeus4dev/amadeus-async-flight-status |
@derberg I build a prototype in which I tried to identify relations between the AsyncAPI docs on the Rideshare microservice (provided by solace) and model the output as shown in the image. This is just a basic structure and can be changed according to the requirements. Please give your feedback. Thanks😄 |
@derberg I did some research about what more functionalities we can provide by the library:
Please share your opinion on this and also throw some light on what more functionality we can provide by this library. Thanks😃 |
Loving it so far mate, I thought about this idea for very long but never had time to code it 😞 awesome to see what you're doing here and ideas that pop up in your head, really 🙏🏼 Do not worry about the structure much because as you noted, this can be modified once the actuall work on the library starts. And the structure will change for sure, will be more complex, with more data. I'm more interested at the moment what you think about making library extendable in a way that someone can provide a plugin for the library, so the output is not our custom Map but class diagram syntax from mermaid. Lemme just list all comments 1by1
I thing this would be a great thing. As I understand this would be besically a set of helpers that someone could use in frontend. So instead of writing custom code to traverse through found relations, frontend dev can just use a helper to get relations only of a given application, or only relations where given message is in play. That would be awesome, much easier then to do some "click to narrow view" magic 😄
you mean in the UI already? we are talking about library here. Of course it is good if some features are driven by how we see it in the UI, just need to know what you mean exactly and what would be the purpose of
I'm not sure what you mean here? oh wait, events are messages for you I guess, right? so you might mean here what I wrote above 😄 if yes, then yes, but it is not option on output but simply another function available imho
you mean not focusing on channels but on messages? find where they are used and show what schemas they have? Like it, it will for sure help to check if there are some inconsistencies maybe |
@derberg Thank you for the kind words, all thanks to you😄
I'm not sure what you mean here? Did you mean that someone can use this library as a plugin I guess, right?
Yeah, we can provide the output in a way that can match the syntax from mermaid, also we can search some more related tools which use the same syntax as mermaid from this collection: awesome-d3
Got it, I will look into it.
It would also be just another helper function, which can be used by developers to visualize the change in the architecture.
Yeah! @derberg Shall we solidify all these functionalities? also please tell me if we can provide any more features by this library as per your opinion so that I could jot them down in my official proposal with the workflow of how I will be implementing them😄 |
I think it would be cool to make it possible to provide plugins for this library. So by default, it outputs some custom format, but one can also install an additional plugin and use it with the library to get mermaid (as an example) format. Of course in first version we do not have to have pluggability, just a thought. We can also have a switch in first version where you can just pass option and say that instead of default you want mermaid.
definitely, mermaid is on my head only because you get UI out of the box, but other formats would be great too
ok, so you import 4 files, get a visualization of architecture and then can add new element to the "visualization" to see how it will look like if some new service will publish to some existing channel, good, makes sense. Just keep in mind that if we make it possible to add something to the structure, we should be able to later export added element into AsyncAPI - just making it clear so you do not extend the scope of the task to much, so you have enough time to finish it. We can make I think all is clear and you can start drafting a proposal |
Got it👍🏽
Sure, will keep in mind, it could be added if the project finishes before time.
Sure. Thanks! |
I'm also looking for some UI to discover relationships in a "mesh" of services. Great to see that you made something so quickly. I'm looking for quite some time for something to document event driven systems in a better way than linked confluence pages ;-) In my mind there is a web UI+API where you can register asyncAPI files and then you can browse all these in a format similar to the docs generated in https://playground.asyncapi.io/?load=https://github.com/raw/asyncapi/asyncapi/master/examples/2.0.0/simple.yml but with links between services that share events/messages. I'm not very fond of JS, I'm more a Java guy, so my first instinct is to create a nice Spring boot app to do the heavy lifting and a small vue.js UI to display everything. But that takes time and effort and when something else is developed somewhere, I could use the time more productively ;) |
@arjungarg07 repo ready and as you can see, the initial issue is also transfered |
@derberg Thanks, will soon be opening issues for further discussions 😃 |
Some discussed points:
|
@derberg Since the Apps relation goes through a given server/broker, in the case of multiple servers described in a single AsyncAPI file, which one do we have to consider? |
@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that asyncapi/spec#531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate. |
cool tool to consider https://reactflow.dev/ |
@magicmatatjahu Thanks for clearing it out. So we would distinguish relations between files on 3 levels. We could club 1st and 2nd levels too Please let me know your thoughts on this 🤔
@derberg yeah! i checked it out yesterday and it's super cool, would love to experiment with it but first focussing on MVP 😄 |
👏🏼
same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server |
I'd like to throw into the ring that structurewise it doesn't really matter is a service is running in 1, 2 or 200 instances, it shouldn't appear with a cardinality of 200 in the flow. The main point of the message approach is that you don't care on the sender side how many workers/consumers there are. It may be an important fact for the consumer to be for example single instanced, or it is stateful, but that has IMHO nothing to do with the message flow documentation. |
@pcornelissen Thanks for your insights :) I almost agree with you, but there is some situation when pinned channel only for particular server can change the rest of flow. We can use the relationship (as a user/client) to check which flow will be "created" when for example I as a user/client will send message to create a new order in e-shop. Sending the order to this same channel, but in two different server can change flow. This situation is rare, but still possible. Of course, we should in first implementation go with simple solution which will cover almost cases, and then go with more complex and rare. If we don't understand each other, it may be because I look at the current problem (app relationship) from the perspective of the user/dev who wants to send a message to a given channel and know what operations will be performed after sending the message/event, such as creating an order, adding loyalty points etc. on the example of an e-shop. |
@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :) |
Yeah! but I would like to play it safe and not make any assumptions😅
Sure thing. |
@derberg @magicmatatjahu @jonaslagoni I ran the script for on Alvaro's flight service with a slight change of adding one more server in one of the spec.
So the mermaid diagram of this output should be like this: Any feedbacks are much appreciated. |
@arjungarg07 I love it! Really :) As I see everything is good, 'publish' operation go to channel, from channel you link to service/app's 'subscribe' operation. At the moment is good, but honestly it's a hard to understand in first look what is connecting to what, I mean, you should look on diagram and see that you can publish something to (eg) FlightNotifierService when you will publish to |
Yup! I mean the whole thing is to visualize the relations and the first look at this image feels somewhat complex :(
will start coding the script for generating mermaid diagrams syntax asap 🚀 |
Do we have to add metadata information in the default output itself or have a separate API for it? |
@arjungarg07 Do you mean metadata like payload's schema of message etc? You operate on parsed spec, currently you save sub/pub only by name to array for appropriate channel name, to save metadata you can change the array to Map, so you have:
but you can change it to Map<string, ChannelMetadata>
Of course you can save as
About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to |
Yeah, message and schemas too.
Sure. I will then update the metrics 👍🏽 In the next weeks maybe we could imply these metrics as class models and extend the APis from there itself. |
@derberg @jonaslagoni @magicmatatjahu Consider we need to get the metadata information like message and schema for each channel on this spec. channels:
flight/update:
description: |
Provides updates from a subscribed flight
subscribe:
summary: Inform about the status of a subscribed flight
message:
$ref: '#/components/messages/flightStatus'
flight/queue:
description: |
Queues a flight in order to retrieve status
publish:
summary: Subscribe about the status of a given flight
message:
$ref: '#/components/messages/flightQueue' I can't find any API available in const messageInfo = doc.channel('flight/update').message(); Same for schemas. Although we do have APIs |
@arjungarg07 Because message is in the What about saving all channel's metadata (with description etc) as metadata - not only message - in relation? EDIT: We have also corresponding |
Ohhk, this I missed :) Thanks!
TBH, I didn't discuss it with Luksaz about what all to save in metadata coz i thought of asking it along the development phase. I would suggest to consider all channel's metadata as you said. EDIT: Also, do we need to have payload as it is for metadata or just some specific parts of it?
Yeah, I have been checking from here itself :) |
Payload is a shape of (usually) json which should be sent to channel or will be subscribe from channel, so payload is a |
@magicmatatjahu Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references, when we will extract the metadata information from let's say message payload do we have to consider some specific fields from it or just include the whole payload itself like this: message: {
payload: {
type: 'object',
'$id': 'PlayerItemPickupPayload',
additionalProperties: false,
properties: {
pickupTimestamp: {
type: 'string',
format: 'date-time',
description: 'The timestamp the item was picked up',
'x-parser-schema-id': '<anonymous-schema-4>'
}
}
},
'x-parser-original-schema-format': 'application/vnd.aai.asyncapi;version=2.0.0',
'x-parser-original-payload': {
type: 'object',
'$id': 'PlayerItemPickupPayload',
additionalProperties: false,
properties: {
pickupTimestamp: {
type: 'string',
format: 'date-time',
description: 'The timestamp the item was picked up'
}
}
},
schemaFormat: 'application/vnd.aai.asyncapi;version=2.0.0',
'x-parser-message-parsed': true,
'x-parser-message-name': '<anonymous-message-1>'
}
} Considering the whole payload would be a lot. Also, I was thinking of wtiting tests after implementing the metadata extraction funcitonailty. What do you think? 🤔 |
@arjungarg07 I see your problem but you also put example on which I can describe my thinking :) You can treat only
and then you don't focus which part of message is important, because you treat whole channel as metadata. Also:
No, we don't, we only handle simple circular refs :) I created issue for that asyncapi/parser-js#293 so please have it in mind when you will use What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future. |
Yeah, I just tried to iterate over it and it caused recursive stackoverflow😅
I agree with the explanation that the metadata's importance could change for every dev and treating the channel as metadata itself would be the best possible solution 👍🏽
Here
EDIT: I get it, you are right! channel's metadata could change for every service, sorry :) @magicmatatjahu Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that 🤔 |
Yes, you're right about that, but partially :) For |
@arjungarg07 they all feel kinda the same tbh, I mean the same principle behind as in case or mermaid and plantUML. As you wrote, I think more important are tests now, and a way to test all those outputs not only from unit tests point of view but also integration tests. And then something more beautiful........ React Flow ❤️ |
@derberg @magicmatatjahu I was playing a bit with React Flow 😅 and it turns out that they don't have any playground online where we can just paste the syntax and render the diagram. Then I thought of generating an HTML page coz I don't want to set up a whole react project just for a simple diagram, but for that(HTML file) I am not able to find and cdn/browser package for Any workarounds/suggestions for this? |
function for react flow should just generate the relations, the code responsible to show them, not entire website. This is a library, I integrate it however I want in my system, generate relations and enable in my HTML. So basically the goal is to focus on react flow code that shows relations between nodes, the HTML to present it can just be an example, part of test or a demo. Does it make sense? |
Yeah, so this is the simplest flow for ReactFlow. import React from 'react';
import ReactFlow from 'react-flow-renderer';
const elements = [
{ id: '1', type: 'input', data: { label: 'Node 1' }, position: { x: 250, y: 5 } },
// you can also pass a React Node as a label
{ id: '2', data: { label: <div>Node 2</div> }, position: { x: 100, y: 100 } },
{ id: 'e1-2', source: '1', target: '2', animated: true },
];
export default () => <ReactFlow elements={elements} />; So our library function for react flow should just generate the |
@arjungarg07 Yes, elements should be the exact output from function. |
@arjungarg07 Important thing: If you will use inside output the VDOM nodes, like in your example |
This issue has been automatically marked as stale because it has not had recent activity 😴 |
@arjungarg07 Can we close the issue? |
we can definitely close it, purpose of this issue was to discuss initial idea and the scope. |
Reason/Context
Would be awesome to build a library that as the input gets an array of AsyncAPI documents and analyzes them to get information about Applications described by those files:
As an output, we should get a format that can be used to generate a diagram of relations between applications with additional metadata information about them (message, schemas).
Use case:
Visualization of the architecture and understanding what happens with the system if one of the puzzles is removed.
Later on, we could use this library in AsyncAPI CLI and also the AsyncAPI Studio.
Required knowledge:
There is no need to have any experience in event-driven architecture. You do not even need to know AsyncAPI spec too much. It is a pure coding task where you will need to find matching patterns between different documents and return this information. This should be an independent library that can be later reused in AsyncAPI CLI or AsyncAPI Studio, or by others in their tools.
Description
The text was updated successfully, but these errors were encountered: