-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pausable channels #2667
Merged
+678
−3
Merged
Pausable channels #2667
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
24 changes: 24 additions & 0 deletions
24
api/src/main/java/io/smallrye/reactive/messaging/PausableChannel.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
package io.smallrye.reactive.messaging; | ||
|
||
/** | ||
* A channel that can be paused and resumed. | ||
*/ | ||
public interface PausableChannel { | ||
|
||
/** | ||
* Checks whether the channel is paused. | ||
* | ||
* @return {@code true} if the channel is paused, {@code false} otherwise | ||
*/ | ||
boolean isPaused(); | ||
|
||
/** | ||
* Pauses the channel. | ||
*/ | ||
void pause(); | ||
|
||
/** | ||
* Resumes the channel. | ||
*/ | ||
void resume(); | ||
} |
28 changes: 28 additions & 0 deletions
28
api/src/main/java/io/smallrye/reactive/messaging/PausableChannelConfiguration.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
package io.smallrye.reactive.messaging; | ||
|
||
/** | ||
* A channel that can be paused and resumed. | ||
*/ | ||
public interface PausableChannelConfiguration { | ||
|
||
/** | ||
* The name of the property to configure whether the channel is pausable. | ||
*/ | ||
String PAUSABLE_PROPERTY = "pausable"; | ||
|
||
/** | ||
* The name of the property to configure whether the channel is initially paused. | ||
*/ | ||
String PAUSED_PROPERTY = "initially-paused"; | ||
|
||
/** | ||
* The name of the channel. | ||
*/ | ||
String name(); | ||
|
||
/** | ||
* Whether the channel is paused at subscribe time. | ||
*/ | ||
boolean initiallyPaused(); | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
# Pausable Channels | ||
|
||
Based on reactive streams, Smallrye Reactive Messaging ensures that channels are back-pressured. | ||
This means that the flow of messages is controlled by the downstream consumer, | ||
whether it is a processing method or outgoing channel. | ||
Sometimes you may want to pause the flow of messages, for example, when the consumer is not ready to process them. | ||
|
||
Injected [`@Channel`](emitter.md#retrieving-channels) streams are not subscribed to by default, so the flow of messages is controlled by the application. | ||
But for [`@Incoming`](model.md#incoming-and-outgoing) methods, the flow of messages is controlled by the runtime. | ||
|
||
Pausable channels are useful when you want to control the flow of messages within your application. | ||
|
||
## Creating a Pausable Channel | ||
|
||
To use pausable channels, you need to activate it with the configuration property `pausable` set to `true`. | ||
|
||
```properties | ||
mp.messaging.incoming.my-channel.pausable=true | ||
# optional, by default the channel is NOT paused initially | ||
mp.messaging.outgoing.my-channel.initially-paused=true | ||
``` | ||
|
||
## Controlling the flow of messages | ||
|
||
If a channel is configured to be pausable, | ||
you can get the `PausableChannel` by channel name from the `ChannelRegistry` programmatically, | ||
and pause or resume the channel as needed: | ||
|
||
``` java | ||
{{ insert('pausable/PausableController.java') }} | ||
``` | ||
|
||
!!!warning | ||
Pausable channels only work with back-pressure aware subscribers, with bounded downstream requests. | ||
39 changes: 39 additions & 0 deletions
39
documentation/src/main/java/pausable/PausableController.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
package pausable; | ||
|
||
import jakarta.annotation.PostConstruct; | ||
import jakarta.enterprise.context.ApplicationScoped; | ||
import jakarta.inject.Inject; | ||
|
||
import org.eclipse.microprofile.reactive.messaging.Incoming; | ||
|
||
import io.smallrye.reactive.messaging.ChannelRegistry; | ||
import io.smallrye.reactive.messaging.PausableChannel; | ||
|
||
@ApplicationScoped | ||
public class PausableController { | ||
|
||
@Inject | ||
ChannelRegistry registry; | ||
|
||
@PostConstruct | ||
public void resume() { | ||
// Wait for the application to be ready | ||
// Retrieve the pausable channel | ||
PausableChannel pausable = registry.getPausable("my-channel"); | ||
// Pause the processing of the messages | ||
pausable.resume(); | ||
} | ||
|
||
public void pause() { | ||
// Retrieve the pausable channel | ||
PausableChannel pausable = registry.getPausable("my-channel"); | ||
// Pause the processing of the messages | ||
pausable.pause(); | ||
} | ||
|
||
@Incoming("my-channel") | ||
void process(String message) { | ||
// Process the message | ||
} | ||
|
||
} |
191 changes: 191 additions & 0 deletions
191
...ssaging-kafka/src/test/java/io/smallrye/reactive/messaging/kafka/PausableChannelTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,191 @@ | ||
package io.smallrye.reactive.messaging.kafka; | ||
|
||
import static java.util.concurrent.TimeUnit.SECONDS; | ||
import static org.assertj.core.api.Assertions.assertThat; | ||
import static org.awaitility.Awaitility.await; | ||
|
||
import java.util.List; | ||
import java.util.concurrent.CompletableFuture; | ||
import java.util.concurrent.CopyOnWriteArrayList; | ||
import java.util.concurrent.atomic.LongAdder; | ||
|
||
import jakarta.enterprise.context.ApplicationScoped; | ||
import jakarta.inject.Inject; | ||
|
||
import org.apache.kafka.common.serialization.IntegerDeserializer; | ||
import org.apache.kafka.common.serialization.IntegerSerializer; | ||
import org.apache.kafka.common.serialization.StringDeserializer; | ||
import org.apache.kafka.common.serialization.StringSerializer; | ||
import org.eclipse.microprofile.reactive.messaging.Incoming; | ||
import org.eclipse.microprofile.reactive.messaging.Message; | ||
import org.eclipse.microprofile.reactive.messaging.Outgoing; | ||
import org.junit.jupiter.api.Test; | ||
|
||
import io.smallrye.mutiny.Multi; | ||
import io.smallrye.reactive.messaging.ChannelRegistry; | ||
import io.smallrye.reactive.messaging.PausableChannel; | ||
import io.smallrye.reactive.messaging.annotations.Blocking; | ||
import io.smallrye.reactive.messaging.kafka.base.KafkaCompanionTestBase; | ||
import io.smallrye.reactive.messaging.kafka.base.KafkaMapBasedConfig; | ||
|
||
public class PausableChannelTest extends KafkaCompanionTestBase { | ||
|
||
public static final int COUNT = 100; | ||
|
||
private KafkaMapBasedConfig commonConfig() { | ||
return kafkaConfig("mp.messaging.outgoing.out") | ||
.with("topic", topic) | ||
.put("key.serializer", StringSerializer.class.getName()) | ||
.put("value.serializer", IntegerSerializer.class.getName()) | ||
.withPrefix("mp.messaging.incoming.data") | ||
.put("topic", topic) | ||
.put("cloud-events", false) | ||
.put("commit-strategy", "throttled") | ||
.put("auto.offset.reset", "earliest") | ||
.put("key.deserializer", StringDeserializer.class.getName()) | ||
.put("value.deserializer", IntegerDeserializer.class.getName()); | ||
} | ||
|
||
@Test | ||
public void testPausableChannelInitiallyPaused() { | ||
addBeans(MyMessageProducer.class); | ||
ConsumerApp application = runApplication(commonConfig() | ||
.with("pausable", true) | ||
.with("initially-paused", true), ConsumerApp.class); | ||
ChannelRegistry pausableChannels = get(ChannelRegistry.class); | ||
PausableChannel pauser = pausableChannels.getPausable("data"); | ||
|
||
long firstStep = COUNT / 10; | ||
long secondStep = COUNT / 5; | ||
long finalStep = COUNT; | ||
await().pollDelay(1, SECONDS).untilAsserted(() -> assertThat(application.getCount()).isEqualTo(0L)); | ||
assertThat(pauser.isPaused()).isTrue(); | ||
pauser.resume(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isGreaterThan(firstStep)); | ||
pauser.pause(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isBetween(firstStep, finalStep)); | ||
assertThat(pauser.isPaused()).isTrue(); | ||
pauser.resume(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isGreaterThan(secondStep)); | ||
pauser.pause(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isBetween(secondStep, finalStep)); | ||
assertThat(pauser.isPaused()).isTrue(); | ||
pauser.resume(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isEqualTo(COUNT)); | ||
} | ||
|
||
@Test | ||
public void testPausableChannel() { | ||
addBeans(MyMessageProducer.class); | ||
ConsumerApp application = runApplication(commonConfig() | ||
.with("pausable", true), ConsumerApp.class); | ||
ChannelRegistry pausableChannels = get(ChannelRegistry.class); | ||
PausableChannel pauser = pausableChannels.getPausable("data"); | ||
|
||
long firstStep = COUNT / 10; | ||
long secondStep = COUNT / 5; | ||
long finalStep = COUNT; | ||
await().untilAsserted(() -> assertThat(application.getCount()).isGreaterThan(firstStep)); | ||
assertThat(pauser.isPaused()).isFalse(); | ||
pauser.pause(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isBetween(firstStep, finalStep)); | ||
pauser.resume(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isGreaterThan(secondStep)); | ||
pauser.pause(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isBetween(secondStep, finalStep)); | ||
pauser.resume(); | ||
await().untilAsserted(() -> assertThat(application.getCount()).isEqualTo(COUNT)); | ||
} | ||
|
||
@Test | ||
public void testPausableChannelWithPauser() { | ||
addBeans(MyMessageProducer.class); | ||
ConsumerAppWithPauser application = runApplication(commonConfig() | ||
.with("pausable", true), ConsumerAppWithPauser.class); | ||
ChannelRegistry pausableChannels = get(ChannelRegistry.class); | ||
PausableChannel pauser = pausableChannels.getPausable("data"); | ||
|
||
assertThat(pauser.isPaused()).isFalse(); | ||
await().untilAsserted(() -> { | ||
if (pauser.isPaused()) { | ||
pauser.resume(); | ||
} | ||
assertThat(application.getCount()).isEqualTo(COUNT); | ||
}); | ||
assertThat(application.getPaused()).isEqualTo(5); | ||
} | ||
|
||
@ApplicationScoped | ||
public static class ConsumerApp { | ||
|
||
LongAdder count = new LongAdder(); | ||
List<Integer> list = new CopyOnWriteArrayList<>(); | ||
|
||
@Incoming("data") | ||
@Blocking | ||
public void consume(Integer message) throws InterruptedException { | ||
list.add(message); | ||
count.increment(); | ||
Thread.sleep(50); | ||
} | ||
|
||
public List<Integer> get() { | ||
return list; | ||
} | ||
|
||
public long getCount() { | ||
return count.longValue(); | ||
} | ||
} | ||
|
||
@ApplicationScoped | ||
public static class ConsumerAppWithPauser { | ||
|
||
@Inject | ||
ChannelRegistry registry; | ||
|
||
LongAdder count = new LongAdder(); | ||
LongAdder paused = new LongAdder(); | ||
List<Integer> list = new CopyOnWriteArrayList<>(); | ||
|
||
@Incoming("data") | ||
@Blocking | ||
public void consume(Integer message) throws InterruptedException { | ||
list.add(message); | ||
count.increment(); | ||
if (count.longValue() % 20 == 0) { | ||
PausableChannel data = registry.getPausable("data"); | ||
data.pause(); | ||
paused.increment(); | ||
} | ||
} | ||
|
||
public List<Integer> get() { | ||
return list; | ||
} | ||
|
||
public long getCount() { | ||
return count.longValue(); | ||
} | ||
|
||
public long getPaused() { | ||
return paused.longValue(); | ||
} | ||
} | ||
|
||
@ApplicationScoped | ||
public static class MyMessageProducer { | ||
|
||
List<Integer> produced = new CopyOnWriteArrayList<>(); | ||
|
||
@Outgoing("out") | ||
public Multi<Message<Integer>> generate() { | ||
return Multi.createFrom().range(0, COUNT) | ||
.map(i -> Message.of(i, () -> { | ||
produced.add(i); | ||
return CompletableFuture.completedFuture(null); | ||
})); | ||
} | ||
} | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍