Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flow control implementation #79

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Flow control implementation #79

wants to merge 9 commits into from

Conversation

zubairov
Copy link
Contributor

@zubairov zubairov commented Feb 8, 2019

When working in the multi-tenant integration environment it's important to obey the API and consumption limits imposed by the platform. This is not only a condition for you integrations to run on the platform (and not begin suspended), but also a good integration practice to sustainably and efficiently use resources.

Imagine a system where one party (published) publishes to the queue and one or more consumers consume from the queue. If publishers are writing to the queue faster than consumers read data from the queue, queue will earlier or later be overfilled. Once one queue of your integration flow will grow to a particular limit, the complete integration flow will be suspended and author will be informed about it. Flow control is a build-in mechanism in the SDK that will help you to prevent the overflow to happen.

There are two types of flow control:

  • Static flow control - the hard limit of the events (e.g. messages published to the queue) that can be generated by component. This limit is dictated by your pricing plan and will limit protect the platform from the extensive load.
  • Dynamic flow control - the limit that is imposed based on the state of individual queue, more messages are in the queue, slower publisher could write to it.

TODOs

  • EventEmitter is async now, emit returns a promise
  • Data events are rate limited, with environment variable to configure it
  • Tests unit and integration for it
  • Switch from regular RabbitMQ Chanel to ConfirmationChannel - dynamic flow control
  • Add rate limits for all other events (errors, snapshots, etc.)
  • Add environment variables
  • Update documentation
  • Test on real components

@jhorbulyk
Copy link

I'll chat with the team about this. If we are going to redesign sailor APIs, there are other approaches & concerns that we may want to address.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants