Start filtering events before unmarshalling #636

FasterSpeeding · 2021-06-01T17:37:42Z

Summary

This change looks into how we may be able to avoid deserializing event data when it isn't being listened to or used to set state to lower the inherent strain of shard dispatch. While intents can help with this by lowering the amount of events a bot receives, intents can't always be used to prevent unwanted events as some events just aren't covered by it and some of it's categories (such as GUILDS) are broad enough to have the potential to still lead to unnecessary events being received.

Another benefit this has over intents is that while intents are behaviour that's statically defined for the entire runtime of a shard, the behaviour of this filtering is dynamic and will shift based on what's being listened for and cached.

Due to the broad scope of the GUILD_CREATE and GUILD_UPDATE events, the raw consumer methods for these events were modified to internally filter based on the cache as well as also filtering based on registered event listeners (this was done in-favour of the generic filtering system used for other raw event consumers). This internal filtering allows the different entities present in these events to only be unmarshalled if they're specifically being cached when there's no listeners registered for the relevant guild events.

Checklist

I have run nox and all the pipelines have passed.
I have made unittests according to the code I have added/modified/deleted.

Related issues

hikari/impl/cache.py

hikari/impl/event_manager_base.py

hikari/api/cache.py

hikari/events/base_events.py

hikari/impl/cache.py

hikari/impl/event_manager.py

hikari/impl/event_manager_base.py

FasterSpeeding · 2021-08-28T03:09:46Z

@davfsa i might officially put this PR on ice for a bit cause there's other things we should prioritise over an optimisation

davfsa · 2021-08-28T08:05:45Z

@davfsa i might officially put this PR on ice for a bit cause there's other things we should prioritise over an optimisation

Sounds good to me

(rather than during event dispatch)

…eing used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config

…ta which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it

davfsa · 2021-11-12T16:35:13Z

Merging this to a separate branch so that I can add my work on top of it

* Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes

* Start filtering events before unmarshalling (#636) * Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes * Improve filtering by using per event bitmasks Co-authored-by: Lucina <luke@lmbyrne.dev>

* Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes

* Start filtering events before unmarshalling (#636) * Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes * Improve filtering by using per event bitmasks Co-authored-by: Lucina <luke@lmbyrne.dev>

* Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes

* Start filtering events before unmarshalling (#636) * Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes * Improve filtering by using per event bitmasks Co-authored-by: Lucina <luke@lmbyrne.dev>

* Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes

* Start filtering events before unmarshalling (#636) * Switch to iterating over mro during listener/waiter registration (rather than during event dispatch) * Add in a system to avoid unmarshalling data for events which aren't being used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config * Specialise guild create and update handlers to avoid unmarshalling data which isn't being cached when no listeners are registered for the event * For this to work gateway guild definition handling had to be refactored to switch to explicitly specifying which mappings it should include when calling it * Logic fixes around event checks * Register listeners by subclasses not parents (mro) (For this the subclasses need to be cached on the Event classes) * Add voodoo on new event cls callback to Event class * This is meant to be a mock way to handle the edge case of new subclassing Event types being added after the event manage has been initialised which might be unorthodox but probably has some wack use case * Switch over to mro based approach * Switch over to mro based approach * Cache whether a consumer can be dispatched or not * Slight logic cleanup * Prefer internal granularity on guild create and update methods * rename event_manager_base.as_listener to "filtered" and remove from on_guild_create and update * Also clear the dispatches for cache when waiters are depleted * Only deserialize guild object on guild create and update if necessary * Add check to shard payload dispatch and refactor consumer check logic * Internal refactors and naming scheme changes * Plus fix CacheImpl.update_me not copying the stored member entry before returning it * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add internal _FilteredMethod proto to event manager base * Move filtering to _handle_dispatch * Add trace logging calls to on_guild_create and on_guild_update * Small logic fix + add code/logic comments and docs * As an artifact of this addition, on_guild_integrations_update acn raise NotImplementedError now since it should always be skipped * Some test fixes * cache_components shouldn't ever be undefined if event_types isn't * Try the builder pattern for GatewayGuildDefinition * Switch GatewayGuildDefinition to using getter style methods for delaying deserialization * test fixes and additions * bug fixes + tests * Post-rebase fixes * Have EventManagerBase take components rather than the cache settings * remove _dispatches_for_cache + add in missing filtered decorator calls * Post-rebase fix * post-rebase fixes * Change i forgot to commit * formatting fixes * Mypy and flake8 fixes * Improve filtering by using per event bitmasks Co-authored-by: Lucina <luke@lmbyrne.dev>

FasterSpeeding added enhancement New feature or request optimization Optimizations to the code and removed enhancement New feature or request labels Jun 1, 2021

FasterSpeeding self-assigned this Jun 1, 2021

davfsa reviewed Jun 1, 2021

View reviewed changes

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from 8052e63 to 0481fb8 Compare June 1, 2021 19:33

davfsa reviewed Jun 1, 2021

View reviewed changes

hikari/impl/event_manager_base.py Outdated Show resolved Hide resolved

hikari/impl/event_manager_base.py Outdated Show resolved Hide resolved

hikari/impl/event_manager_base.py Outdated Show resolved Hide resolved

davfsa reviewed Jun 2, 2021

View reviewed changes

hikari/impl/event_manager_base.py Show resolved Hide resolved

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from e06d7d0 to faadd37 Compare June 6, 2021 15:39

FasterSpeeding commented Jun 12, 2021

View reviewed changes

hikari/api/cache.py Show resolved Hide resolved

davfsa reviewed Jun 14, 2021

View reviewed changes

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch 2 times, most recently from 868b17e to 4636e24 Compare June 30, 2021 17:20

FasterSpeeding marked this pull request as ready for review June 30, 2021 17:21

FasterSpeeding requested a review from davfsa June 30, 2021 17:22

FasterSpeeding changed the title ~~POC pitch for filtering events before unmarshalling~~ Start filtering events before unmarshalling Jun 30, 2021

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from 4636e24 to 1504419 Compare July 8, 2021 17:24

FasterSpeeding enabled auto-merge (squash) July 8, 2021 20:01

davfsa reviewed Jul 14, 2021

View reviewed changes

hikari/impl/event_manager_base.py Show resolved Hide resolved

hikari/impl/event_manager_base.py Show resolved Hide resolved

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from 9db17ab to 3c086ea Compare August 4, 2021 15:35

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from 2a58163 to dac60b4 Compare September 10, 2021 13:25

davfsa mentioned this pull request Oct 20, 2021

Split InteractionCreateEvent into 2 events for command and component events #859

Closed

1 task

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch 2 times, most recently from 8e8039f to 29200f7 Compare October 26, 2021 18:12

FasterSpeeding added 4 commits October 26, 2021 19:17

Switch to iterating over mro during listener/waiter registration

dc57b2b

(rather than during event dispatch)

Add in a system to avoid unmarshalling data for events which aren't b…

f392b93

…eing used * Add settings property to cache interface to allow for introspection * Also add "ME" resource to cache config

Logic fixes around event checks

824db06

FasterSpeeding added 8 commits October 26, 2021 19:18

Post-rebase fixes

be8faf9

Have EventManagerBase take components rather than the cache settings

4f8f47b

remove _dispatches_for_cache + add in missing filtered decorator calls

fc509fa

Post-rebase fix

bb28a55

post-rebase fixes

6740d0c

Change i forgot to commit

07f0ab3

formatting fixes

6f99d62

Mypy and flake8 fixes

09f0c81

FasterSpeeding force-pushed the task/experimental-dispatch-refactor-take-2 branch from a5f8cfe to 09f0c81 Compare October 26, 2021 18:18

This was referenced Nov 4, 2021

Ability to configure the guild member cache to only store the current user's members #878

Open

Add separate configuration option for enabling/disabling member chunking. #882

Closed

davfsa changed the base branch from master to feature/events-filtering November 12, 2021 16:34

davfsa disabled auto-merge November 12, 2021 16:34

davfsa merged commit 4eaba59 into hikari-py:feature/events-filtering Nov 12, 2021

davfsa mentioned this pull request Nov 12, 2021

Improve filtering by using per event bitmasks #903

Merged

2 tasks

FasterSpeeding deleted the task/experimental-dispatch-refactor-take-2 branch November 15, 2021 21:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start filtering events before unmarshalling #636

Start filtering events before unmarshalling #636

FasterSpeeding commented Jun 1, 2021 •

edited

Loading

FasterSpeeding commented Aug 28, 2021

davfsa commented Aug 28, 2021

davfsa commented Nov 12, 2021

Start filtering events before unmarshalling #636

Start filtering events before unmarshalling #636

Conversation

FasterSpeeding commented Jun 1, 2021 • edited Loading

Summary

Checklist

Related issues

FasterSpeeding commented Aug 28, 2021

davfsa commented Aug 28, 2021

davfsa commented Nov 12, 2021

FasterSpeeding commented Jun 1, 2021 •

edited

Loading