Multiple Persistence actors of same type, different names throw Actor name \"akka.persistence.journal.inmem\" is not unique!

[MrTortoise]

My scenario involves an AtLeastOnce delivery actor that i am using as a client for 2 different services.

So i have 2 instances of the same actor type but with 2 different names - they differ as the props passes in the address to use as a path.

In the actor prestart i set up a scheduler to snapshot every second - this causes a cant stash messages more than once error however on actor restart i then get this:

GetCandidatePromotion
2015-12-09 12:32:11 [Error] "Can't stash the same message PromotionPlanning.Akka.GuranteedDelivererClientProxy+SnapshotMessages more than once"
Akka.Actor.IllegalActorStateException: Can't stash the same message PromotionPlanning.Akka.GuranteedDelivererClientProxy+SnapshotMessages more than once
   at Akka.Actor.Internal.AbstractStash.Stash()
   at Akka.Persistence.Eventsourced.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
2015-12-09 12:32:11 [Error] "Can't stash the same message PromotionPlanning.Akka.GuranteedDelivererClientProxy+SnapshotMessages more than once"
Akka.Actor.IllegalActorStateException: Can't stash the same message PromotionPlanning.Akka.GuranteedDelivererClientProxy+SnapshotMessages more than once
   at Akka.Actor.Internal.AbstractStash.Stash()
   at Akka.Persistence.Eventsourced.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
2015-12-09 12:32:11 [Debug] "Restarting"
2015-12-09 12:32:11 [Debug] "Restarting"
2015-12-09 12:32:11 [Debug] "now supervising akka://clientActorSystem/system/akka.persistence.journal.inmem"
2015-12-09 12:32:11 [Error] "Actor name \"akka.persistence.journal.inmem\" is not unique!"
Akka.Actor.PreRestartException: Exception of type 'Akka.Actor.PreRestartException' was thrown.
2015-12-09 12:32:11 [Error] "Object reference not set to an instance of an object."
[akka://clientActorSystem/system/akka.persistence.journal.inmem]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass110_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
2015-12-09 12:32:11 [Debug] "Restarted (PromotionPlanning.Akka.GuranteedDelivererClientProxy)"

tis spirals into a loop until the timeout on my expect is hit.

The workaround suggested in gitter which works is to call this immediatley after system creation
Persistence.Instance.Apply(_clientTestkit.Sys).JournalFor(null);

with this workaround i get a stash same message wtice error and then everythign continues as expected (well a message is not getting received on the server side but i suspect that is a bug in my code)

[Horusiath]

Stashing problem has been fixed in #1480 and will be solved in the next release. Problem with journal initialization was supposed to be solved by adding LazyThreadSafetyMode.ExecutionAndPublication to lazily initialized journals, however it seems that it hadn't solved the problem.

In general we need to solve problems of concurrent access to cached lazily initialized values in Persistence extension for:

• Journals
• SnapshotStores
• EventAdapters

[cconstantin]

@Horusiath the code below doesn't fail:

            var extension = Persistence.Instance.Apply(Sys);
            for (var i=0; i<100; i++)
                extension.SnapshotStoreFor(null);

This leads me to believe that the bug is actually in ActorSystemImpl:

        public override object RegisterExtension(IExtensionId extension)
        {
            if(extension == null) return null;
            if(!_extensions.ContainsKey(extension.ExtensionType))
            {
                _extensions.TryAdd(extension.ExtensionType, new Lazy<object>(() => extension.CreateExtension(this)));
            }

            return extension.Get(this);
        }

It should probably be

        public override object RegisterExtension(IExtensionId extension)
        {
            if(extension == null) return null;

            _extensions.GetOrAdd(extension.ExtensionType, t => new Lazy<object>(() => extension.CreateExtension(this)));

            return extension.Get(this);
        }

Will try to write a quick test for it.

[cconstantin]

Hmm, that doesn't solve it, although I think it's a good change to make. I'll keep investigating.

[cconstantin]

Duh, calling SnapshotStoreFor in 2 separate threads surfaces the issue. Working on a fix.