MatcherUtil.cache takes too much memory

[eschava]

Prerequisites

• Plugin is in the latest version
• Issue was not reported yet
• Stack trace (if provided) contains mobi.hsz.idea.gitignore package name

Description

IDEA took too much memory and memory analyzer showed that MatcherUtil.cache takes 1.3 GB in my case. Size of the map - 24M of records, every record is 48 bytes.

Details of the project opened in IDEA:
Number of files in project: 140K
Number of lines in .gitignore: 700
Number of files under git: 44K
Number of ignored files: 93K

Steps to Reproduce

1. Just run IDEA and wait some time

Expected behavior: Used memory should be less than 1G

Actual behavior: Used memory is 1.5-2G

Reproduces how often: 100%

Versions

Plugin:

2.4.0

IDE:

IntelliJ IDEA 2018.1 EAP (Community Edition)
Build #IC-181.3986.9, built on February 27, 2018
JRE: 1.8.0_152-release-1136-b16 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

OS:

Windows 10 10.0

[hsz]

@eschava This is huge!

I'll try to fix it and release with next v2.5.0

[hsz]

@eschava

Can I ask you to install manually following build?
idea-gitignore-2.4.0-521.zip

This contains migration from the HashMap which has unlimited size to com.intellij.util.containers.IntObjectCache - by default (that I'll stick to) there is max size of 8192 items.
It should be enough for the displaying items in project tree view, also because it implements FIFO that respects cache hits count.

Please let me know it it works fine for you (regarding memory, performance and ignoring in general).

[eschava]

My IDEA hung up with the stacktrace below. Could it be related to the non-syncronized access to the cache?

"JobScheduler FJ pool 4/7" #87 daemon prio=4 os_prio=-1 tid=0x000000001fce1800 nid=0x3f08 runnable [0x000000003c60e000]
   java.lang.Thread.State: RUNNABLE
	at com.intellij.util.containers.IntObjectCache.searchForCacheEntry(IntObjectCache.java:283)
	at com.intellij.util.containers.IntObjectCache.isCached(IntObjectCache.java:191)
	at com.intellij.util.containers.IntObjectCache.containsKey(IntObjectCache.java:76)
	at mobi.hsz.idea.gitignore.util.MatcherUtil.match(MatcherUtil.java:70)
	at mobi.hsz.idea.gitignore.util.Glob$1.visitFile(Glob.java:130)
	at com.intellij.openapi.vfs.VirtualFileVisitor.visitFileEx(VirtualFileVisitor.java:132)
	at com.intellij.openapi.vfs.VfsUtilCore.visitChildrenRecursively(VfsUtilCore.java:299)
	at com.intellij.openapi.vfs.VfsUtilCore.visitChildrenRecursively(VfsUtilCore.java:331)
	at com.intellij.openapi.vfs.VfsUtilCore.visitChildrenRecursively(VfsUtilCore.java:331)
	at com.intellij.openapi.vfs.VfsUtilCore.visitChildrenRecursively(VfsUtilCore.java:331)
	at mobi.hsz.idea.gitignore.util.Glob.find(Glob.java:144)
	at mobi.hsz.idea.gitignore.util.Glob.findAsPaths(Glob.java:162)
	at mobi.hsz.idea.gitignore.codeInspection.IgnoreCoverEntryInspection.getPathsSet(IgnoreCoverEntryInspection.java:223)
	at mobi.hsz.idea.gitignore.codeInspection.IgnoreCoverEntryInspection.checkFile(IgnoreCoverEntryInspection.java:147)
	at com.intellij.codeInspection.LocalInspectionTool$1.visitFile(LocalInspectionTool.java:144)
	at mobi.hsz.idea.gitignore.psi.IgnoreFile.accept(IgnoreFile.java:114)
	at com.intellij.codeInspection.InspectionEngine.acceptElements(InspectionEngine.java:75)
	at com.intellij.codeInsight.daemon.impl.LocalInspectionsPass.lambda$visitRestElementsAndCleanup$4(LocalInspectionsPass.java:301)
	at com.intellij.codeInsight.daemon.impl.LocalInspectionsPass$$Lambda$2257/942143886.process(Unknown Source)
	at com.intellij.concurrency.ApplierCompleter.execAndForkSubTasks(ApplierCompleter.java:133)
	at com.intellij.concurrency.ApplierCompleter$$Lambda$2253/717656038.run(Unknown Source)
	at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1134)
	at com.intellij.concurrency.ApplierCompleter.lambda$wrapInReadActionAndIndicator$1(ApplierCompleter.java:105)
	at com.intellij.concurrency.ApplierCompleter$$Lambda$2255/751090051.run(Unknown Source)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:580)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:525)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:85)
	at com.intellij.concurrency.ApplierCompleter.wrapInReadActionAndIndicator(ApplierCompleter.java:116)
	at com.intellij.concurrency.ApplierCompleter.lambda$compute$0(ApplierCompleter.java:96)
	at com.intellij.concurrency.ApplierCompleter$$Lambda$2251/1875119226.run(Unknown Source)
	at com.intellij.openapi.application.impl.ReadMostlyRWLock.executeByImpatientReader(ReadMostlyRWLock.java:143)
	at com.intellij.openapi.application.impl.ApplicationImpl.executeByImpatientReader(ApplicationImpl.java:218)
	at com.intellij.concurrency.ApplierCompleter.compute(ApplierCompleter.java:96)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

[hsz]

@eschava Yeah, good point. It was a static object shared between threads. HashMap somehow was thread-safe, but this new storage isn't. I got rid of this static idea. Can you please verify it again?
idea-gitignore-2.4.0-521-2.zip

[eschava]

Now IDEA hung on startup :)

I've got similar dumps at several threads

"AWT-EventQueue-0 2018.1#IC-181.3986.9 IDEA, eap:true, os:Windows 10 10.0, java-version:JetBrains s.r.o 1.8.0_152-release-1136-b16" #20 prio=6 os_prio=0 tid=0x000000001af06800 nid=0x24d0 runnable [0x000000001ca5d000]
   java.lang.Thread.State: RUNNABLE
	at com.intellij.util.containers.IntObjectCache.searchForCacheEntry(IntObjectCache.java:283)
	at com.intellij.util.containers.IntObjectCache.isCached(IntObjectCache.java:191)
	at com.intellij.util.containers.IntObjectCache.containsKey(IntObjectCache.java:76)
	at mobi.hsz.idea.gitignore.util.MatcherUtil.match(MatcherUtil.java:67)
	at mobi.hsz.idea.gitignore.IgnoreManager.isFileIgnored(IgnoreManager.java:406)
	at mobi.hsz.idea.gitignore.IgnoreManager.isFileIgnored(IgnoreManager.java:423)
	at mobi.hsz.idea.gitignore.IgnoreManager.isFileIgnored(IgnoreManager.java:423)
	at mobi.hsz.idea.gitignore.IgnoreManager.isFileIgnored(IgnoreManager.java:423)
	at mobi.hsz.idea.gitignore.vcs.IgnoreFileStatusProvider.getFileStatus(IgnoreFileStatusProvider.java:70)
	at com.intellij.openapi.vcs.impl.FileStatusManagerImpl.calcStatus(FileStatusManagerImpl.java:137)
	at com.intellij.openapi.vcs.impl.FileStatusManagerImpl.getStatus(FileStatusManagerImpl.java:264)
	at com.intellij.openapi.vcs.impl.LineStatusTrackerManager.canCreatePartialTrackerFor(LineStatusTrackerManager.kt:373)
	at com.intellij.openapi.vcs.impl.LineStatusTrackerManager.onFileChanged(LineStatusTrackerManager.kt:287)
	- locked <0x000000073efe8d40> (a java.lang.Object)
	at com.intellij.openapi.vcs.impl.LineStatusTrackerManager.onEverythingChanged(LineStatusTrackerManager.kt:267)
	- locked <0x000000073efe8d40> (a java.lang.Object)
	at com.intellij.openapi.vcs.impl.LineStatusTrackerManager.access$onEverythingChanged(LineStatusTrackerManager.kt:71)
	at com.intellij.openapi.vcs.impl.LineStatusTrackerManager$MyFileStatusListener.fileStatusesChanged(LineStatusTrackerManager.kt:608)
	at com.intellij.openapi.vcs.impl.FileStatusManagerImpl.fileStatusesChanged(FileStatusManagerImpl.java:207)
	at com.intellij.openapi.vcs.impl.FileStatusManagerImpl.lambda$fileStatusesChanged$2(FileStatusManagerImpl.java:199)
	at com.intellij.openapi.vcs.impl.FileStatusManagerImpl$$Lambda$542/933704371.run(Unknown Source)
	at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.doRun(LaterInvocator.java:447)
	at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.runNextEvent(LaterInvocator.java:431)
	at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.run(LaterInvocator.java:415)
	at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:762)
	at java.awt.EventQueue.access$500(EventQueue.java:98)
	at java.awt.EventQueue$3.run(EventQueue.java:715)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:732)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:837)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:766)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:434)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

[hsz]

@eschava I'm very sorry for that. I've checked this package couple times before posting here and didn't even notice this kind of issue.
Maybe it's a specific for very huge project?
Is there any possibility to share it with me? Is it hosted on GitHub?

[eschava]

No, sorry, it's our closed project.