-
Notifications
You must be signed in to change notification settings - Fork 10k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Epic: IDistributedCache updates in .NET 9 #53255
Comments
I know it's difficult to change the existing interface, but depending on the other changes planned in this epic, it would be great if we could have distributed cachr methods that were more performance oriented, working with Span instead of byte[]. A common scenario for using the cache is saving serialized objects or large texts encoded as UTF-8. Requiring byte[] usually means copying this data at least once. Similar issues exist when reading from the cache which also returns a byte[] and thus does not allow for using rented buffers or similar optimizations. As the cache is often used several times for each request in any non trivial web application (e.g. session store, query cache, response cache), optimizations here would really pay off. |
Yup. Totally understand, @aKzenT , and that's part of the prototype. Will update with more details of the proposed API as it evolves, but the short version here is:
|
@aKzenT please see updated body |
As for L1+L2 caching, you might want to talk to developers of MS FASTER https://github.com/microsoft/FASTER, which has L1+L2 support, while L2 is not strictly out of process, more like out of main memory (disk based or azure based if I remember correctly). |
@Tornhoof aye, FASTER has come up more than a few times; because of the setup etc required, I doubt we can reasonably make that a default implementation; the most pragmatic solution there might be to provide an |
@mgravell thank you a lot for the update. I'm seeing a lot of things addressed that I've missed in the past, working on multi-server web apps. For L1+L2 Caching we have been quite happy with https://github.com/ZiggyCreatures/FusionCache in the past which is also built upon IDistributedCache. I remember that there were some features missing from IDistributedCache that made it hard to implement advanced scenarios in that library. So I would like to invite @jodydonetti to this discussion as well as he can probably best comment on these issues. One thing I remember that was missing was being able to modify the cache entry options (i.e. life time) of a cache entry without going through a Get/Set cycle. Being able to modify the lifetime allows for some advanced scenarios like invalidating a cache entry from the client (e.g. you received a webhook notifying you about changes in data) or reducing the time to allow things like stale results. Another thing related to cache invalidation, that is not really possible with the current API in an efficient way, is the removal/invalidation of a group of related cache entries. Let's say you have a cache for pages of a CMS system with each page being one entry. The CMS informs you about changes via a web hook, which invalidates the cache for all pages. Directly refreshing all pages might be expensive, so you would rather refresh them individually on demand. So you want to invalidate all page entries in the cache, but there is no way to get the list of entries from the cache, nor is there a good way to delete the entries. Our solution was to built this functionality ourself using a Redis Set that manages the related keys and then iterating through these keys and removing them one by one. But it felt very hacky as you cannot even use the same Redis Connection that the distributed cache uses, as far as I remember. |
@aKzenT re individual cache invalidation: there is a remove API that is meant to serve that function, but it doesn't allow modify of the options; I'd love to understand the need there further re group cache invalidations: that sounds a lot like the "tag" feature of output-cache, i.e. you associate entries with zero, one or more tags, and then you can invalidate an entire tag, which nukes everything associated; the problem is: that tracking still needs to go somewhere, and it isn't necessarily an inbuilt feature of the cache backend - it was a PITA to implement reasonably on redis without breaking the cluster distribution, for example (ask me how I know!). It also isn't something that can fit in the existing |
Re FusionCache: that isn't one I've seen before, but glancing at the homepage, I see that the fundamental design is pretty similar (some differences, but: very comparable). There is a common theme in these things - indeed, a lot of inspiration (not actual code) in the proposal draws on another implementation of the same that we had when I was at Stack Overflow (in that version, we also had some Roslyn analyzers which complained about inappropriate captured / ambient usage - very neat!). My point: lots of approaches converging around the same API shape. |
@mgravell we had the same experience implementing our invalidation logic for a redis cluster setup. It's really hard to get right. I would not expect the design to provide a complete solution to this issue, but maybe there is some way that would make it possible for other libraries to support that while building on top of IDistributedCache. Maybe @jodydonetti has an idea how that could work. As for modifying the options, in the case of FusionCache there is the possibility to allow stale entries, which are treated as expired, but still available in case the backend is not available. For these cases there is a separate timeout of how long you want to allow a result being stale. The TTL that is sent to the underlying IDistributedCache is then the initial TTL plus the additional stale timeout. So when you invalidate an entry, but still want to allow stale results, you cannot simply delete the entry. Instead you would want to update the timeout to being equal to the stale timeout. Hope that makes sense. |
Yep, very familiar with the idea - it is conceptually related to the "eager pre-fetch" mentioned above - with two different TTLs with slightly different meanings |
This is referring to this repo, which offered some of the highlights of this proposal as extension methods, but without any of the "meat" that drives the additional functionality. |
One thing I'm wondering is, if the choice to put the generic type parameter on the interface rather than the methods might be limitting in some cases and would require some classes to have to configure and inject multiple IDistributedCache instances. I'm not sure if that is really a problem, but it would be nice to learn, why you went for that choice, which differs from other implementations that I have seen. |
@aKzenT fair question, and it isn't set in stone, but - reasons:
Tell you what, I'll branch my branch and try it the other way. Although then I need a new name... dammit! |
@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update I will benchmark the perf of |
Would it be a option to use change token for the cache key invalidation instead of the event type proposed currently? |
@danielmarbach in reality, I'm not sure that is feasible in this scenario; the example uses shown for that API seem pretty low-volume and low issuance frequency; file config changes etc, but if we start talking about cache: we're going to need as many change tokens as we have cached entries, and crucially: the backend layer would need to know about them; I'm mentally comparing that to how redis change notifications can work, and to do that efficiently: we don't want to store anything extra, especially at all the backend layers. Given that all we actually want/need here is the |
@danielmarbach I believe it should be fairly easy to implement a GetChangeToken(string key) method as an extension method that subscribes to the event and listens to changes to the specific key. The other way arround is harder. That being said, I'm a fan of ChangeTokens and IIRC, MemoryCache uses change tokens to invalidate entries, so there might be some value to provide such an extension method directly in the framework in addition to the event @mgravell . |
To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider |
If you want to go that route, is there a particular reason why you want ICacheSerializer to be generic instead of just its methods being generic? I feel like for the basic scenario of System.Text.Json, this can just be a singleton. If someone needs to differentiate serialization by type it should be easy to do using a composite style pattern. Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example. |
Keyed Services might make that easy. Although I'm not sure if there is an easy way to register both a keyed cache and a keyed serializer and tell the DI system that it should use the serializer with the same key when injecting into the cache. |
I never liked the name of it to start with because I might want an in memory, hybrid or distributed cache and the abstraction doesn't change. I also don't like how expiration / expiration strategy is not present (possibly it is with extension methods, I haven't looked in a long time). I also don't feel very strongly about adding generics to the interface because a cache can store all different kinds of shapes. We probably should slim down our default interface, but this is what we've found useful over the past decade: https://github.com/FoundatioFx/Foundatio#caching |
Great to know, thanks! The list was never meant to be exhaustive - I'll add and clarify. The key takeaway is: we want to actively avoid breaking any implementations; the new functionality should happily work against the existing backend, with them using the additional / optional extensions if appropriate. |
@mgravell a couple of things that comes to mind, some related to serialization configuration being strongly binded per-type, some related to other things, but all related to how having clearly separated worlds (read: named caches) is frequently used. SerializationFirst off is wanting different serialization configuration for the same type but in different named caches. A more low-level but common example: it's common not to create a custom type for common types like Here's for an even nastier one to consider, which it also happened to me multiple times: the need to manually specify how to (de)serialize a specific type because of how it's shaped, like a Of course for this you may be tempted to simply say "not supported" or "not supported in v1" and call it a day, but is something to at least think about, even just to prepare for the future version when it may eventually be supported. Different ConfigurationsA totally different scenario in support for multiple named caches is different configurations. A classic example is wanting a different Another one is different sub-components, like different distributed caches or different tag-based invalidation mechanisms (what in FusionCache is the backplane). This can be good for different reasons, from smaller to bigger:
Tag-based invalidations and general namespacingAnother example is, since it seems to be part of HybridCache v1, tag-based invalidation. Of course an answer may be "just namespace your tags too, like you namespace you cache keys", for example via the common way of using a prefix, but then users will have to manually prefix everything and... that's not great. Lingua Franca and IntegrationsFor a FusionCache-HybridCache integration I don't see big problems related to not having multiple named caches in v1. The way I'm imagining the integration would be is something like this: services.AddFusionCache()
.WithOptions(...)
.WithSerializer(...)
.WithDistributedCache(...)
.AsHybridCache(...) // THE MAGIC HAPPENS HERE
; where it will register in DI an In this case users, even when directly using multiple named FusionCache instances in their apps, can still setup one of them for For example this will be useful if (as I hope/think it will be the case) So something like this: services.AddFusionCache("MyHybridCacheUsedForOutputCacheOrWhatever")
.WithOptions(...)
.WithSerializer(...)
.WithDistributedCache(...)
.AsHybridCache(...) // THE MAGIC HAPPENS HERE
; Does this make sense? A lot to digestNow, I know it's a lot to digest and a lot to ask for HybridCache v1 (FusionCache for example did not have multiple named caches for some time), but at least knowing this upfront may allow you and your team to design for a future addition and not have to make either breaking changes down the road or have a badly designed api in the future because you didn't see these things coming. Hope this helps. ps: if I forgot something I hope @aKzenT can chip in with the missing pieces! |
Thank you @jodydonetti . This sums up perfectly what I was thinking. Serialization, namespacing and default options were my main points as well and you articulated them better than I could have and added some additional aspects I did not think of, but agree 100%. As another edge case, if I develop a library with a class that uses caching internally using MyPrivateData class, how would I configure its serialization behavior as a library consumer without requiring my MyPrivateData to be exposed publically? I might get away with exposing it only internally and then providing DI extension methods to configure that, but that basically prohibits the use of any other DI frameworks. |
I'm not understanding this part: if your library uses internally a cache, why should it be configurable from the outside? |
Re the DI topic: it'll already want the expected DI APIs to lazily resolve the serializers as it sees the need. But within that: your component can still call AddHybridCache etc and configure serialization - everything should still work correctly, but I'll adds tests for that scenario. I need to take a moment to digest the multiple cache thing. |
The HybridCache needs to know how to serialize the types that my library uses. So this means at some point the caller has to directly or indirectly configure the HybridCache to correctly handle (serialize) them. So imagine I have a DataLoader class that uses HybridCache using a construction parameter and requires serialization of an internal DataCacheEntry. I need to register a matching serializer for that class so that HybridCache knows how to serialize it. I'm assuming that usually this would happen through an extension method, so that I don't need to be aware of the details as a caller. What I was wondering is, how that scenario would play out if using another DI or when constructing the class manually. Having said that, after writing everything down, I think the solution here would be for the library to provide a public serializer factory. I missed that before, so ignore my comment. |
If it hadn't been mentioned (didn't see it), I'd change the API such that the base interface is a "multi get" (set based operations). IE: it takes in keys as a list and then when some of them aren't found, it should run a function that gets the values for those keys and kicks back a dictionary. Then throw an extension method on top of that to do the single get/add. That way my fallback can go to the DB and grab N items efficiently. |
I'm not part of the team, or Microsoft in general, but having "been there done that" in the past I'd like to share a couple of issues. The main one is the fact that when working with one cache entry the input for the cache is the cache key, and not the db item id, whereas the input for the factory (called by the cache) is the item id, and not the cache key. For example (oversimplified): var id = 123;
cache.GetOrSet<Product>(
$"product/{id}", // INPUT FOR THE CACHE: CACHE KEY
_ => GetFromDb(id) // INPUT FOR THE FACTORY: ITEM ID
); So the core problem here is one of mapping between the 2 things, cache key and item id. By then going "multi", you should consider the input for the cache being something like a list of cache keys, then the cache should use that, see what is missing, and then pass the cache keys for the missing entries to the factory, which in turn should decompose the cache keys to extract the item ids. There are basically 2 paths here that can be followed:
and both ways are generally not that nice. (just as a note: personally, I'm working on something that would solve this, in what would be imho a correct way, but it takes a different approach and I need a little bit more time for that). Opinions? |
On the key vs id; this also crosses over into the alloc-free key discussion
that I had on the (new cache proposal thing that I've forgotten the name
of). If we had *that*, then yes there's some possible future additional
piece that is basically a TKey, with a TKey-to-string function somewhere
(perhaps even implied via attribute or naming), but I don't think that
piece is a "today" thing.
…On Sat, 25 May 2024, 10:58 Jody Donetti, ***@***.***> wrote:
IE: it takes in keys as a list and then when some of them aren't found, it
should run a function that gets the values for those keys and kicks back a
dictionary.
I'm not part of the team, or Microsoft in general, but having *"been
there done that"* in the past I'd like to share a couple of issues.
The main one is the fact that when working with one cache entry the input
for the cache is the *cache key*, and not the db *item id*, whereas the
input for the factory (called by the cache) is the *item id*, and not the *cache
key*.
For example (oversimplified):
var id = 123;
cache.GetOrSet<Product>(
$"product/{id}", // INPUT FOR THE CACHE: CACHE KEY
_ => GetFromDb(id) // INPUT FOR THE FACTORY: ITEM ID);
So the core problem here is one of mapping between the 2 things, *cache
key* and *item id*.
By then going "multi", you should consider the input for the cache being
something like a list of cache keys, then the cache should use that, see
what is missing, and then pass the cache keys for the missing entries to
the factory, which in turn should decompose the cache keys to extract the
item ids.
There are basically 2 paths here that can be followed:
- make the cache learn about how to manipulate the mapping between
item ids <-> cache keys (eg: with 2 lambdas, one for id -> key and the
other for key -> id)
- have the consumer (eg: you) do it yourself by parsing/splitting the
cache keys to get back the item ids
and both ways are generally not that nice.
(just as a note: personally, I'm working on something that would solve
this, in what would be imho a *correct* way, but it takes a different
approach and a I'd need a little bit more time for that).
Opinions?
—
Reply to this email directly, view it on GitHub
<#53255 (comment)>
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAEHMHHDYJVSD3JTY3QXSTZEBODZBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTEOBUGE3TKMBTGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRVGU3TENBTGEZTAM5ENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUDCNZWGIYDGNBXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGA3TGMRXGE4TMMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJSHA2DCNZVGAZTJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDKNJXGI2DGMJTGAZ2O5DSNFTWOZLSUZRXEZLBORSQ>
.
You are receiving this email because you commented on the thread.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>
.
|
I tend to use extensions like the following example in some scenarios like the one described by @mlstubblefield and @jodydonetti. But I only use it with in-memory caches... Does this flow make sense? public static class MemoryCacheExtensions
{
public static async Task<IReadOnlyCollection<TResult>> GetOrAddToCache<TResolverArg, TResult>(
this IMemoryCache cache,
IReadOnlyCollection<string> ids,
Func<TResult, string> itemIdSelector,
Func<string, TResolverArg, string> cacheKeyResolver,
Func<IReadOnlyCollection<string>, TResolverArg, Task<IReadOnlyCollection<TResult>>> dataResolver,
TResolverArg resolverArg,
TimeSpan expiryInMinutes)
{
var (fromCache, toRetrieve) = cache.GetDataByIdsFromCache<TResolverArg, TResult>(ids, cacheKeyResolver, resolverArg);
if (toRetrieve.Count == 0)
return fromCache;
var fromIo = await dataResolver(toRetrieve, resolverArg);
if (fromIo == null)
return fromCache;
cache.AddToCache(fromIo, itemIdSelector, cacheKeyResolver, resolverArg, expiryInMinutes);
foreach (var item in fromIo)
{
fromCache.Add(item);
}
return fromCache;
}
static void AddToCache<TResolverArg, TResult>(
this IMemoryCache cache,
IEnumerable<TResult> fromIo,
Func<TResult, string> itemIdSelector,
Func<string, TResolverArg, string> cacheKeyResolver,
TResolverArg resolverArg,
TimeSpan expiryInMinutes)
{
foreach (var item in fromIo)
{
cache.Set(
key: cacheKeyResolver(itemIdSelector(item), resolverArg),
value: item,
expiryInMinutes);
}
}
static (List<TData> data, List<string> remainder) GetDataByIdsFromCache<TResolverArg, TData>(
this IMemoryCache cache,
IEnumerable<string> ids,
Func<string, TResolverArg, string> cacheKeyResolver,
TResolverArg resolverArg)
{
var fromCache = new List<TData>();
var toRetrieve = new List<string>();
foreach (var id in ids)
{
if (cache.TryGetValue(cacheKeyResolver(id, resolverArg), out var data))
{
fromCache.Add((TData)data);
continue;
}
toRetrieve.Add(id);
}
return (fromCache, toRetrieve);
}
} Example: ...
var resolverArg = (
repository: _repository,
cachePrefix: 'product',
cancellationToken);
var entities = await _memoryCache.GetOrAddToCache(
ids: query.Ids,
itemIdSelector: static item => item.Id,
cacheKeyResolver: static (id, rArg) => $"{rArg.cachePrefix}:{rArg.product.Id}"
dataResolver: static (toRetrieve, rArg) => FetchFromDb(rArg, toRetrieve, rArg.cancellationToken),
resolverArg,
5); For a distributed version with L1 and L2, it would be ideal to be able to fetch and set all the keys in a single operation... |
Preview 4 NotesHi @mgravell and team, I played with the preview 4 bits and here are my notes.
NOTE: I will not touch on things that I know will already change, like At First GlanceI see that there's a Anyway I'll wait for preview 5 for that. Regarding the
As pointed out by some in the video by @Elfocrash , the auto-magic usage of any registered
Something else that's not clear is what Finally I see that the "by tag" logic is there, all the time: I get that it is desired (and I mean, I really really get it, it would be an awesome feature, and I'm playing with it in FusionCache for some time now already) but, since it is reasonably tied to the distributed backend used, I'm wondering about 2 things:
Honestly, I can see the difficulty in creating an API surface area that allows that but also allows to check for that, but still, I'd like to point that out in this first preview. HybridCache implemented on top of FusionCacheI've been able to reasonably implement very quickly a FusionCache-based adapter adapter for HybridCache, so kudos for how easy it has been. Right now in the example project I'm able to register the normal HybridCache with this: // ADD NORMAL HYBRIDCACHE
services.AddHybridCache(o =>
{
o.DefaultEntryOptions = new HybridCacheEntryOptions
{
LocalCacheExpiration = defaultDuration
};
}); and a FusionCache-based one with this: // ADD FUSIONCACHE, ALSO USABLE AS HYBRIDCACHE
services.AddFusionCache()
.AsHybridCache() // MAGIC
.WithDefaultEntryOptions(o =>
{
o.Duration = defaultDuration;
}); The "magic" part is basically registering a FusionCache-based adapter as an // THIS WILL WORK BOTH WITH THE DEFAULT HybridCache IMPLEMENTATION
// AND THE FusionCache BASED ONE
var cache = serviceProvider.GetRequiredService<HybridCache>();
Console.WriteLine($"TYPE: {cache.GetType().FullName}");
const string key = "test key";
// TAGS: NOT CURRENTLY SUPPORTED
//await cache.SetAsync("fsdfsdfds", "01", tags: ["", ""]);
// SET
await cache.SetAsync(key, "01");
var foo = await cache.GetOrCreateAsync<string>(
key,
async _ => "02"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 01)");
// REMOVE
await cache.RemoveKeyAsync(key);
foo = await cache.GetOrCreateAsync<string>(
key,
async _ => "03"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");
// NO-OP
foo = await cache.GetOrCreateAsync<string>(
key,
async _ => "04"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");
// LET IT EXPIRE
Console.WriteLine("WAITING FOR EXPIRATION...");
await Task.Delay(defaultDuration);
// SET
foo = await cache.GetOrCreateAsync<string>(
key,
async _ => "05"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 05)"); Running this with both the registrations in turn gives me this first:
and this second:
Of course right now I'm talking without tagging support, for which I'm currently throwing an exception (because of the reasons I'll highlight soon in the other issue). Apart from that I think it may be useful to have a method (either Currently I'm applying some logic here, simulating what you have described in your specs, but having some cache instance method where you can pass an Another thing I noticed is that currently it's possible to override the I see that the namespace/packages in preview 4 are "wrong" long term: if you are wondering about types forwardings and similar, my suggestion is to just "destroy everything" in the next preview. This is exactly what previews are for, and (again, personally) I'd like to update the packages, see that I'm not able to compile anymore, and know exactly what needs to be changed to allow it to compile again. For now I think it's all, thanks for sharing the update. Hope this helps. |
Great feedback. I will ponder! Minor note: there is no additional closure allocation: we pass the stateless callback as the TState, and invoke it from there via a static lambda. This trick is also available to anyone wrapping the library. |
Oh, that's neat, I didn't notice it! |
If my application has 2 instances and 1st instance updates record in L1 + L2, will 2nd instance L1 get the updated record as well with some pub/sub magic? |
Hi @mgravell and team, LazyCache supports setting the expiry of the cache entry based on the data being cached. For example: cache.GetOrAdd("token", async entry => {
var token = await GetAccessToken();
// can set expiry date using a DateTimeOffset (or a TimeSpan with RelativeToNow variant)
entry.AbsoluteExpiration = token.Expires;
return token;
}); Does |
@michael-wolfenden no, the API as implemented doesn't support that option; such options are passed in as arguments, so are fixed at the time of call. |
Is the team considered adding similar functionality? The api as is stands now doesn't allow the cache time to be set dynamically based on the data being returned which is a common scenario when caching data from a third party where they control the data's lifetime. |
@mgravell and @jodydonetti, I believe combining your efforts will result in a best-in-class cache abstraction |
Are hit/miss metrics available? How? |
Currently: not; working on it via .net metrics |
@mgravell I quickly looked at using In a related scenario, is there a pattern for when the factory can't produce the data (external service is offline), so all we have is some default value we fall back to, such as a static var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));
result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null)); The factory in the second call won't be invoked as Ideally, I need a way to signal that the creation failed and not store anything under the key. If there were a way to do this, that might then provide a workaround to the first problem, as I can do Another scenario I've yet to explore is where my "expensive" async calls to external systems that I use to build up some data that I'd like to cache, also provide some extra info I need later in my method but that I don't want cached. I'm guessing, but have yet to try, that using the state might be a way to do that but I'm curious if it's a pattern you expect to see used? |
Will do example of the first tomorrow. On the latter: if you need that value, what do you expect to use on the anticipated case when the value does come from cache? You say you need it, but if it isn't stored: where does it come from? I guess philosophically it should usually be preferable to have things work the same whether or not the callback was invoked this time, but yes you can always mutate ambient state in either the TState sense or via closures. Note that there's also a category of cases where the callback was invoked for the active result, but not by you - i.e. a stampede scenario where you're merely an observer. |
@mgravell, thanks. This morning, with a fresh eye, I just discovered the flags that seemed to give me the control I needed to avoid the set. var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null),
new() { Flags = HybridCacheEntryFlags.DisableLocalCacheWrite | HybridCacheEntryFlags.DisableDistributedCacheWrite });
result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null)); The main reason I was looking at this was akin to the FusionCache adaptive caching, where, based on the outcome of the factory, I may want not to cache (or choose a very short cache time) when the external service couldn't give me the value I needed. In that case, I may present an error to the end user (for the current request); for a subsequent request, it tries again to get a value. Using a short empty cached value might be helpful for DDos or general outages to save hitting a struggling backend for a while. However, I may also get that behaviour from Polly and the HTTP resiliency stuff. If a null/empty value is cached for the configured expiration time, say, 30 mins, that's not ideal. That said, if the cache returns empty, I can still trigger a fetch of the data and manually update the cache. So I think, I see a way to achieve this. |
Hello all, I'm interested in the |
I wanted to try the hybrid cache so I've tried implementing it in my current code base, but when I tried to add an EfCore item with cyclic fields(x->y->x->y...) it would crash, It seems to ignore any JsonOptions from the DI, and there was no way to set |
@HugoVG I'll investigate; fyi this has now moved to dotnet/extensions - I'll log an issue there tomorrow |
Could you mention me in that one so I could be kept posted. I don't mind help testing |
It is a little unclear if we have invalidation of L1 cache across instances (through Redis pub/sub or similar)? |
Update:
HybridCache
has relocated to dotnet/extensions:dev; it does not ship in .NET 9 RC1, as a few missing and necessary features are still in development; however, we expect to ship either alongside or very-shortly-after .NET 9! ("extensions" has a different release train, that allows additional changes beyond the limit usually reserved for in-box packages;HybridCache
has always been described as out-of-box - i.e. a NuGet package - so: there is no reason for us to limit ourselves by the runtime restrictions)Status: feedback eagerly sought
By
nomenclature: HybridCache - rename RemoveKeyAsync and RemoveTagAsync to "By" #55332Tl;Dr
HybridCache
API (and supporting pieces) to support more convenient and efficient distributed cache usageIDistributedCache
so that all existing cache backends work without change (although they could optionally add support for new features)IDistributedCache
Problem statement
The distributed cache in asp.net (i.e.
IDistributedCache
) is not particularly developed; it is inconvenient to use, lacks many desirable features, and is inefficient. We would like this API to be a "no-brainer", easy to get right feature, making it desirable to use - giving better performance, and a better experience with the framework.Typical usage is shown here; being explicit about the problems:
Inconvenient usage
The usage right now is extremely manual; you need to:
byte[]
)null
("no value")not null
:This is a lot of verbose boilerplate, and while it can be abstracted inside projects using utility methods (often extension methods), the vanilla experience is very poor.
Inefficiencies
The existing API is solely based on
byte[]
; the demand for right-sized arrays means no pooled buffers can be used. This broadly works for in-process memory-based caches, since the samebyte[]
can be returned repeatedly (although this implicitly assumes the code doesn't mutate the data in thebyte[]
), but for out-of-process caches this is extremely inefficient, requiring constant allocation.Missing features
The existing API is extremely limited; the concrete and implementation-specific
IDistributedCache
implementation is handed directly to callers, which means there is no shared code reuse to help provide these features in a central way. In particular, there is no mechanism for helping with "stampede" scenarios - i.e. multiple concurrent requests for the same non-cached value, causing concurrent backend load for the same data, whether due to a cold-start empty cache, or key invalidation. There are multiple best-practice approaches that can mitigate this scenario, which we do not currently employ.Likewise, we currently assume an in-process or out-of-process cache implementation, but caching almost always benefits from multi-tier storage, with a limited in-process (L1) cache supplemented by a separate (usually larger) out-of-process (L2) cache; this gives the "best of both" world, where the majority of fetches are served efficiently from L1, but cold-start and less-frequently-accessed data still doesn't hammer the underlying backend, thanks to L2. Multi-tier caching can sometimes additionally exploit cache-invalidation support from the L2 implementation, to provide prompt L1 invalidation as required.
This epic proposes changes to fill these gaps
Current code layout
At the moment the code is split over multiple components, in the main runtime, asp.net, and external packages (only key APIs shown):
Microsoft.Extensions.Caching.Abstractions
IDistributedCache
DistributedCacheEntryOptions
Microsoft.Extensions.Caching.Memory
AddDistributedMemoryCache
MemoryDistributedCache : IDistributedCache
Microsoft.Extensions.Caching.StackExchangeRedis
AddStackExchangeRedisCache
Microsoft.Extensions.Caching.SqlServer
AddDistributedSqlServerCache
Microsoft.Extensions.Caching.Cosmos
AddCosmosCache
Alachisoft.NCache.OpenSource.SDK
AddNCacheDistributedCache
AWS
This list is not exhaustive - other 3rd-party and private implementations of
IDistributedCache
exist, and we should avoid breaking the world.Proposal
The key proposal here is to add a new caching abstraction that is more focused,
HybridCache
, inMicrosoft.Extensions.Caching.Abstractions
; this API is designed to act more as a read-through cache, building on top[ of the existingIDistributedCache
implementation, providing all the implementation details required for a rich experience. Additionally, while simple defaults are provided for the serializer, it is an explicit aim to make such concerns fully configurable, allowing for json, protobuf, xml, etc serialization as appropriate to the consumer.Notes:
IDistributedCache
, consumers might useHybridCache
; to enable this, the consumer must additionally perform aservices.AddHybridCache(...);
step during registrationGetOrCreateAsync<T>
is for parity withMemoryCache.GetOrCreateAsync<T>
RemoveAsync
andRefreshAsync
mirror the similarIDistributedCache
methodscallback
(when invoked) will return a non-null
value; consistent withMemoryCache
et-al,null
is not a supported value, and an appropriate runtime error will be raisedUsage of this API is then via a read-through approach using lambda; the simplest (but slightly less efficient) approach would be simply:
In this simple usage, it is anticipated that "captured variables" etc are used to convey the additional state required, as is common for lambda scenarios. A second "stateful" API is provided for more advanced scenarios where the caller wishes to trade convenience for efficiency; this usage is slightly more verbose but will be immediately familiar to the users who would want this feature:
This has been prototyped and works successfully with type inference etc.
The implementation (see later) deals with all the backend fetch, testing, serialization etc aspects internally.
(in both examples, the "discard" (
_
) is conveying theCancellationToken
for the backend read, and can be used by providing a receiving lambda parameter)An
internal
implementation of this API would be registered and injected via a newAddHybridCache
API (Microsoft.Extensions.Caching.Abstractions
):The
internal
implementation behind this would receiveIDistributedCache
for the backend, as it exists currently; this means that the new implementation can use all existing distributed cache backends. By default,AddDistributedMemoryCache
is also assumed and applied automatically, but it is intended that this API be effective with arbitraryIDistributedCache
backends such as redis, SQL Server, etc. However, to address the issue ofbyte[]
inefficiency, a new entirely optional API is provided and tested for; if the new backend is detected, lower-allocation usage is possible. This follows the pattern used for output-cache in net8:(the intent of the usual members here is to convey expiration in the most appropriate way for the backend, relative vs absolute, although only one can be specified; the internals are an implementation detail, likely to use overlapped 8-bytes for the
DateTime
/TimeSpan
, with a discriminator)In the event that the backend cache implementation does not yet implement this API, the
byte[]
API is used instead, which is exactly the status-quo, so: no harm. The purpose ofCacheGetResult
is to allow the backend to convey backend expiration information, relevant for L1+L2 scenarios (design note:async
precludesout TimeSpan?
; tuple-type result would be simpler, but is hard to tweak later). The expiry is entirely optional and some backends may not be able to convey it, and we need to handle it lacking whenIBufferDistributedCache
is not supported - in either event, the inbound expiration relative to now will be assumed for L1 - not ideal, but the best we have.Serialization
For serialization, a new API is proposed, designed to be trivially implemented by most serializers - again, preferring modern buffer APIs:
Inbuilt handlers would be provided for
string
andbyte[]
(and possiblyBinaryData
if references allow); an extensible serialization configuration API supports other types - by default, an inbuilt object serializer usingSystem.Text.Json
would be assumed, but it is intended that alternative serializers can be provided globally or per-type. This is likely to be for more efficient bandwidth scenarios, such as protobuf (Google.Protobuf or protobuf-net) etc, but could also be to help match pre-existing serialization choices. While manually registering a specificIHybridCacheSerializer<Foo>
should work, it is also intended to generalize the problem of serializer selection, via an ordered set of serializer factories, specifically by registering some number of:By default, we will register a specific serializer for
string
, and a single factory that usesSystem.Text.Json
, however external library implementations are possible, for example:The
internal
implementation ofHybridCache
would lookupT
as needed, caching locally to prevent constantly using the factory API.Additional functionality
The
internal
implementation ofHybridCache
should also:Note that it is this additional state for stampede and L1/L2 scenarios (and the serializer choice, etc) that makes it impractical to provide this feature simply as extension methods on the existing
IDistributedCache
.The new invalidation API is anticipated to be something like:
(the exact shape of this API is still under discussion)
When this is detected, the
event
would be subscribed to perform L1 cache invalidation from the backend.Additional things to be explored for
HybridCacheOptions
:IDistributedCacheInvalidation
?Additional modules to be enhanced
To validate the feature set, and to provide the richest experience:
Microsoft.Extensions.Caching.StackExchangeRedis
should gain support forIBufferDistributedCache
andIDistributedCacheInvalidation
- the latter using the "server-assisted client-side caching" feature in RedisMicrosoft.Extensions.Caching.SqlServer
should gain support forIBufferDistributedCache
, if this can be gainful re allocatiuonsMicrosoft.Extensions.Caching.Cosmos
owners, and if possible:Alachisoft.NCache.OpenSource.SDK
Open issues
does the approach sound agreeable?namingSystem.Text.Json
, and possible an L1 implementation ( which could beSystem.Runtime.Caching
,Microsoft.Extensions.Caching.Memory
, this new one, or something else) and possibly compression; maybe a newMicrosoft.Extensions.Caching.Distributed
? but if so, should it be in-box with .net, or just NuGet? or somewhere else?how exactly to configure the serializeroptions for eager pre-fetch TTL and enable/disable L1+L2, viaTypedDistributedCacheOptions
should we add tagging support at this juncture?The text was updated successfully, but these errors were encountered: