AsyncAPI Application simulation

[jonaslagoni]

Reason/Context

A simulation library that can trigger events in your applications based on AsyncAPI definition(s). The main purpose is to stimulate your application(s) in a development environment (staging?). Use it for stress testing, hunting down concurrency problems, stimulating visual applications, finding scalability issues in your application, and more.

The general use-case:

• You provide one or more AsyncAPI files
• You decide which channel(s) and applications should be stimulated and with which parameters.
• The library simulates messages to the application(s) based on the parameters, in the given protocol described in the AsyncAPI file.
• Inspect the application(s) and their state to figure out if errors occurred etc

Required knowledge:
You need to learn the basics of the AsyncAPI specification since it relies on the information in the provided documents.

Description
I suggest focusing on the important parts of the use-case which are:

• Simulation scenarios
• Scaled simulations (how do you simulate something like 1 million users/requests)
• Others focus points?

Rabbit holes / be careful around these topics:

• Specifying the Simulation scenarios
• Optimizing the code
• Supporting all protocols
• CLI for the library

Of course, these things are all suggestive and depend on what the user finds important. So never see any of these things as non-negotiable requirements and focus points.

[NektariosFifes]

Hello i am interested in participating . which technology stack you believe would best fit the needs of this library? suggestion : node ? will this be a npm library?
ideas:
-the app should be able to spawn multiple persistent and non persistent http//s connections
-create a sitemap from asyncapi file
-store the result of the tests in a database
-visualize the results either via cli or a frontend application.

[jonaslagoni]

Hey @NektariosFifes, Node (JavaScript/TypeScript) could be an option yes, to me it doesn't matter.

👍 on your ideas.

-store the result of the tests in a database

Keep in mind not everyone wants such a feature, so the library should be versatile enough to support different use-cases. But I would not be surprised if someone wants this feature.

-visualize the results either via cli or a frontend application.

I agree there has to be some kind of visualization, but to what extent might vary again, depending on the use-case.

[NektariosFifes]

for containerization docker?
Regardless of GSoC i am going to make a start and put smthing together . I will make a pull request when it is ready.

[NektariosFifes]

Project requirements

• User stories
• Developer stories
• Non functional requirements
• Implementation details

Common Vocabulary

word	meaning
simulation/ stress testing	A simulation is a collection of scenarios that are executed on existing services based on the AsyncAPI. It can be stress testing an application, simulating specific environments, etc.

User stories

id	Prioritization	Description
#S1	Should have	As a user i want to call the app with a cli command passing the required flags and arguments along with optional ones.
#S2	Should have	As a user i can view statistics during the test . Number of requests per second warnings and errors based on api responces.
#S3	Should have	As a user i want to have a choice to initiate tests based on one or multiple .yaml or .json async-api files.I can also pause , unpause the tests, as a standalone process a local or cloud docker container.
#S4	Should have	As a user i want to be able to install package from npm in @asyncapi scope.
#S5	Could have	As a user i can write my own modules/extensions to process response messages and create my own statistics. (?use prexisting senarios e.x ddos attack , query injection?)
#S6	Should have	As a user, I want to run the simulation completely autonomous using defined scenarios.

Developer stories

id	Prioritization	Description
#DS1	Must Have	Use of eslint and prettier for code formating.

Implementation details

id	Prioritization	Description
#S1	1	Check if container with provided async-spec file is already running .
#S3	1	Payload will be generated by the examples in the spec file or randomly generated.

Non-functional requirements

1. Test coverage of around 95%.
2. Type definitions for the exposed parts of the application (the modules , functions that will be used by the user for making extensions)

[akhill10]

Hey Guys, want to give some user inputs on requirements.

• Have an option to simulate messages (payload) with the examples specified in asyncapi.yaml file OR by generating random payload as per the defined message structure.

• Considering above case, it's a good option to have message validator in place for simulation.

• Have a flag option : Are server(s) defined in asyncapi.yaml is/are already up and running ? If not , we should be able to spin up the required containers/servers as per multiple asyncapi.yaml files, in order to perform simulation.

• For the load test, maybe have multiple scenarios and have an option to distribute the load among those scenarios. For example, there are two Scenarios: Scenario 1 and Scenario 2 . Suppose there are 100k requests to test on, able to distribute 70k of Scenario 1 and remaining 30k of Scenario 2.

Please Correct me, if anything doesn't make sense

[NektariosFifes]

@pawanakhil i will update the requirements. Any other suggestions? Will you take part in GSoC?

[akhill10]

Any other suggestions?

That's it as if now, maybe will update if I get any.

Will you take part in GSoC?

Maybe not or Maybe as a mentor 🤔 if I get chance (although it's upto member's of community). Be it Anyways, I am happy to help out and contribute to this project.

[derberg]

@pawanakhil this is open source project from initiative that is community driven. Nobody from some company will come and tell you to go away. We love working here in larger groups as it always enriches discussion and at the end make our tools better. So feel free to contribute and share your experience. I would say official GSoC mentors are there in case noone else can help, but it doesn't mean they are the only ones that can help drive the project forward.

[jonaslagoni]

Great start guys 👏

Maybe it is a good idea we clarify the word simulation/stress testing, what do you guys say we use the definition?

A simulation is a collection of scenarios that are executed on existing services based on the AsyncAPI. It can be stress testing an application, simulating specific environments, etc.

Test coverage?

@NektariosFifes I would suggest we add a test coverage of around 95% whenever we talk about implementing anything it is implied that it contains tests, docs, and the implementation. We need to ensure that any contributions do not break the library 🙂

* Have a flag option : Are server(s) defined in `asyncapi.yaml` is/are already up and running ? If not , we should be able to spin up the required containers/servers as per multiple `asyncapi.yaml` files, in order to perform simulation.

@pawanakhil I don't think this makes sense since, when you are doing the simulation you are expecting already existing services to be in place ready to process those requests right? However, I agree that we should use the defined servers in the AsyncAPI document 👍 Maybe the user selects a server before initiating the simulation.

As a user i can view statistics during the test on my terminal. Number of requests per second warnings and errors based on api responces.

Regarding #S2, maybe instead of test on my terminal, we extend it to be dynamic, implementation wise it shouldn't be that different. I.e. we create an interface for which people can create Reporters or whatever we can call it. Imagine I want the result exported to JSON, excel, etc. Would that make sense you think 🤔? Somewhat the same as Mocha have.

Just to add a few more use-cases: As a user, I want to run the simulation completely autonomous using defined scenarios.
This could be in a staging environment where I want to run the simulation continuously.

As you guys suggest the scenarios should be part of the AsyncAPI document, but what if scenarios are across multiple AsyncAPI documents, such as a chain of events, that would be a bit confusing won't you say? 🤔

Maybe there already exist frameworks how to document such scenarios we can re-use? 🤔

[NektariosFifes]

@jonaslagoni

I will update the requirements

Regarding #S2, maybe instead of test on my terminal, we extend it to be dynamic, implementation wise it shouldn't be that different. I.e. we create an interface for which people can create Reporters or whatever we can call it. Imagine I want the result exported to JSON, excel, etc. Would that make sense you think 🤔? Somewhat the same as Mocha have.

Idea: "Reporters" could take 1.data streams or 2.Promise Arrays with results internally, and spit out objects with a specific format. It will be up to the user to figure out how he want to visualize the results. However i think we should provide an example(probably as a web app because a command line visualization of statistics will not be easy if the container is running in a remote machine). This example visualization webapp utilizing the library might be in a separate repo. I am making a ui library for react and i will try to add components for visualizing statistics / stress tests .

[jonaslagoni]

This example visualization webapp utilizing the library might be in a separate repo. I am making a ui library for react and i will try to add components for visualizing statistics / stress tests .

Would be cool if it could do that yea 🎉

[akhill10]

@jonaslagoni

I don't think this makes sense since, when you are doing the simulation you are expecting already existing services to be in place ready to process those requests right? 
However, I agree that we should use the defined servers in the AsyncAPI document 👍 Maybe the user selects a server before initiating the simulation.

Cool. Agree with it. Although, It's a good option to have - Before starting the simulation, checking whether all the servers or user specified servers (for simulation) are running or not and notifying details of the servers which not running back to the user. Because if the servers aren't running and performing simulation doesn't make sense.

As you guys suggest the scenarios should be part of the AsyncAPI document, but what if scenarios are across multiple AsyncAPI documents, such as a chain of events, that would be a bit confusing won't you say? 🤔

Maybe there already exist frameworks how to document such scenarios we can re-use?

How are we planning to take the input of user defined scenarios ? through AsyncAPI document itself ? Yeah, then it would be definitely confusing. IMO.
how about a through a separate JSON file ? where a list of tasks or workflow of tasks are defined for simulation and then automate the task(s)/workflow(s) ( a good tool does the work - GulpJS)

[jonaslagoni]

How are we planning to take the input of user defined scenarios ? through AsyncAPI document itself ? Yeah, then it would be definitely confusing. IMO.
how about a through a separate JSON file ?

@pawanakhil Yea I agree it would probably be "best" to provide a separate file for this. Basically creating a specification for specifying simulations (if it doesn't already exist).

Another thing we have to keep in mind is how would we keep the scenario files up to date with ever-changing AsyncAPI documents?

@NektariosFifes I would have to insist that DS1 be a Must Have, it is important that such a file is created from the start since the refactor will be a pain 😄

[NektariosFifes]

@jonaslagoni
@pawanakhil

I will start working on the specification. For a start i will try to define its rules and try to make a parser.

[jonaslagoni]

I created a repository for the follow up issues and PRs: https://github.com/asyncapi/fluffy-robot

We should also figure out what it should be called 😄 #1

@NektariosFifes do you mind create an issue for the different parts that should be discussed?

[NektariosFifes]

@jonaslagoni ok 😃 i will

[AceTheCreator]

Idea: "Reporters" could take 1.data streams or 2.Promise Arrays with results internally, and spit out objects with a specific format. It will be up to the user to figure out how he want to visualize the results. However i think we should provide an example(probably as a web app because a command line visualization of statistics will not be easy if the container is running in a remote machine). This example visualization webapp utilizing the library might be in a separate repo. I am making a ui library for react and i will try to add components for visualizing statistics / stress tests .

@NektariosFifes I'd love to help contribute to the visualization of the reports. You said you're making a ui library for react, is this library specific for the visualization? or something you're just working on. Nevertheless I'd love to take a peek at the library. Mid sharing the repo?

[NektariosFifes]

@AceTheCreator I have not progressed that much just created some inline highlighted text components and a small fill/progress bar but i think this ui library will focus in the visualization of this project. My setup is kinda weird 1 project for developing testing and 1 for packaging and publishing. I will probably push the development repo here:https://github.com/asyncapi/fluffy-robot soon.

[AceTheCreator]

@AceTheCreator I have not progressed that much just created some inline highlighted text components and a small fill/progress bar but i think this ui library will focus in the visualization of this project. My setup is kinda weird 1 project for developing testing and 1 for packaging and publishing. I will probably push the development repo here:https://github.com/asyncapi/fluffy-robot soon.

Ok cool. Looking forward to it. Keep up with the good work mate 👍

[hkaur008]

@AceTheCreator I have not progressed that much just created some inline highlighted text components and a small fill/progress bar but i think this ui library will focus in the visualization of this project. My setup is kinda weird 1 project for developing testing and 1 for packaging and publishing. I will probably push the development repo here:https://github.com/asyncapi/fluffy-robot soon.

I would like to contribute to the visualization of the project. Please direct me where to get started.

[derberg]

Folks, maybe we could transfer it under the fluffy robot since the work starts soon and it no longer is just an idea?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴
It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation.
Thank you for your contributions ❤️

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴
It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation.
Thank you for your contributions ❤️

[derberg]

Shouldn't this issue be closed?