VS2017 freezing randomly with WakaTime

[jakubsuchybio]

History of my issue:
One and half year ago I started getting random freezes of Visual Studio. So I reported it to MSFT multiple times... Here is one where I got some responses on it: https://developercommunity.visualstudio.com/content/problem/135843/visual-studio-frozen-randomly.html
I was desperate, it was driving me nuts so I tried to contact MSFT directly, but couldn't.
Then I started experimenting by disabling some newly installed extensions. First one in line was OzCode, which after I disabled it freezes stopped. I contacted them, their CTO contacted me right back, connected to my machine and tried to get as much info as he could. Our resolution about this was, that it was most definitely doing some extension.

Recently:
Few days ago MSFT responded, that they will not fix this. It is some sort of Finalizer deadlock most probably. And also few days before that I finally narrowed down to extension WakaTime. Now it has been like 2 weeks with no freeze after disabling wakatime.

How it behaves:
Most cases when it freezes it is when I leave my PC for a lunch or toilet and when I get back I get to the frozen VS. Thinking that Wakatime counts my time spent in VS it triggers that I am away and in that moment it might have freezes.
The freeze is really odd one. Because Windows does not reports that VS is unresponsive (like when UI thread of a process is busy). However I can't click on anything (only exception that I could click on was when I am debugging and that perf window with GCs, CPU, Memory consumption shows up. I suspect that this behaves somehow different from other VS buttons, etc) I can't minimize the app, I can't click on Close/Exit on top right, my only option in this situation is open Task Manager and kill the Visual Studio, which results in lost work, when something isn't saved by me before freeze... A LOT OF LOST WORK...
Just to mention, at the end sometimes it froze even when I was active and working inside VS. Really a LOT of randomness with this.

I made several dumps taken when it was frozen and I think one timeline recording via dotTrace from it not being frozen to it getting frozen(it took me like one month to get this recording): https://1drv.ms/f/s!AopNkAPn2DUNrqwx72jqok3L7izBPw

[alanhamlett]

Thanks for all the detailed info, and I understand this is frustrating. Sounds like the MS team thinks it's related to a threading deadlock and the only usage of threads in the WakaTime extension are these three Task.run calls:

visualstudio-wakatime/WakaTime/WakaTimePackage.cs

Line 238 in 252bc83

Task.Run(() =>

Would you be up for trying a custom build of the WakaTime extension that disables threading? It would make VS laggy but if it fixes the freezing then we know for sure it's caused by the System.Threading.Tasks.Task library.

[jakubsuchybio]

I am sorry, I wanted to report this as a good oss citizen, however I don't want to spend more time testing something that already took so much from me. I hope you understand.
I don't think, that there would be issue with Task's library. It is used in the wild so much, that this kind of bug would already be caught.
However I looked at your code and there are some points that I think could cause problems:

• RunProcess library's Run method. Your reading of stdout and stderr is not so good. It is blocking, it might never end and I think it is giving you wrong output. Also in my recent project I found, that ReadToEnd on stdout and stderr in .NET can block your process, because it uses buffers which can get full and when they do, it blocks on that method indefinetely.
  Better way is by something like this:

            var stdOut = new StringBuilder();
            var stdErr = new StringBuilder();
            using (var process = Process.Start(startInfo))
            {
                process.OutputDataReceived += (s, e) => stdOut.Append(e.Data);
                process.ErrorDataReceived += (s, e) => stdErr.Append(e.Data);
                process.BeginOutputReadLine();
                process.BeginErrorReadLine();

                process.WaitForExit(1000*60*10); // 10 minutes
            }

• Plugin Initialization, these days best practice is to use InitializeAsync, which is provided by extensibility APIs. I see that you load things by Task.Run, but with that VS does still thinks that you are loading synchronously. See this for more info: https://docs.microsoft.com/en-us/visualstudio/extensibility/how-to-use-asyncpackage-to-load-vspackages-in-the-background?view=vs-2017
• If you are able not to use python binary and do sending to wakatime api from .NET it could stabilize it more
• And also I don't think that you need those Task.Run in AppendHeartbeat and ProcessHeartbeats. In AppendHeartbeat that operation you run in Task is really fast, no need to spawn new Task. And ProcessHeartbeats already runs on new thread that is provided by the Timer. The only thing to keep in mind is not to throttle the Timer. If you have 8s interval and ProcessHeartbeats would take longer that 8s, then the Timer would throttle.
• Lastly I would fix all Warnings on your solution. Because DTE has some limitations when some calls are not from the UI thread.
  

Other than that I don't see anything that would make VS freezing. The only hazardous thing I see is non-stop listening on those 4 event handlers, because if something (other extension or VS) would call some of those events periodically or in bursts it could be a real problem, because I don't think that you can handle like thousands of events per sec, it would definitelly throttle as well.

[alanhamlett]

I've added a setting to disable threading in v8.1.0 of the extension, and I'll release a new version today improving RunProcess to prevent blocking and stop using Task.Run when appending to the in-memory queue.

The only thing to keep in mind is not to throttle the Timer.

Currently the timer executes every 8s, but I'd like to chain the timer so it only executes 8s after the last execution has finished. I'll look into doing that after this first round of fixes.

[alanhamlett]

It is blocking, it might never end and I think it is giving you wrong output.

With debug mode turned off (default) this bug wouldn't be triggered since we skip reading process output unless debug mode is on.

[gandarez]

Maybe this PR #90 solves this problem

[alanhamlett]

Probably not fixed with #90, because that would only fix freezing on startup but this issue is about freezing after a break.

Maybe d3bb156 will fix this by preventing deadlocked io, such as @jakubsuchybio's comment above. Will release a new version of the extension after #90 is merged.

[gandarez]

@jakubsuchybio is it still happening?

[jakubsuchybio]

I abandoned WakaTime. This was just too much of reminding pain. Image that your main work IDE would freeze 10 times a day and every unsaved work would be lost. That I lived with for 1,5 years before I figured out that WakaTime was doing that.

[omarpiani]

Reinstalled today on VS2019 and no freezings

[gandarez]

I abandoned WakaTime. This was just too much of reminding pain. Image that your main work IDE would freeze 10 times a day and every unsaved work would be lost. That I lived with for 1,5 years before I figured out that WakaTime was doing that.

@jakubsuchybio I'd suggest you to reinstall the extension since it has been always improved by our team.

[gandarez]

Reinstalled today on VS2019 and no freezings
Thanks for the feedback.