Psalm validation crashes laptop due to memory issues #10522
Comments
|
Hey @mkroeders, can you reproduce the issue on https://psalm.dev ? |
|
Well, a known issue is that Psalm threads will follow the php.ini directive (or use the memory-limit option if provided) for each thread. While each thread individually will probably consume less memory than analyzing everything in the same thread, it will inevitably have an overlap. Psalm can also be pretty resource consuming on big codebases. Both of these combined can eat up a laptop memory pretty fast. Have you tried running in a single thread? |
|
If I use the option However when I do |
|
I have the same problem If you delete the directory with the cache, it starts 1 time. Then it falls into error again |
|
Can you try with --no-cache and --threads=1 ?
I forgot that Psalm does that on its own, unless you tell it explicitly to follow the ini directive: psalm/src/Psalm/Internal/CliUtils.php Line 466 in afc2df4 |
|
Tested with
Which resulted in finishing the check so that works. Eye balling the activity monitor for memory usage I saw it spike to 2.47 GB. |
|
Not knowing the exact in and outs. But looking at how threads is detected. I assume that the amount of threads is based upon the cpu count - 1. Which got me thinking perhaps the max memory should be based upon the max amount of threads in combination with max memory. So the total combination of threads * max memory does not exceed the max system memory |
|
Apart from that, I think I have found a cause and workaround for the unnatural increase in memory consumption when using parallel processing and caching. I will send a report and PR later. |
|
Psalm uses A possible workaround for now is to define I'm here from the GitHub search to test our memory profiler (Reli 0.11.2). "Allowed memory size of" is my personal trending word these days. I've done some investigation about this issue, so I share it. PreconditionsAdditional Test scriptanalyzer.php<?php
register_shutdown_function(
function (): void {
$error = error_get_last();
if (is_null($error)) {
return;
}
if (strpos($error['message'], 'Allowed memory size of') !== 0) {
return;
}
$pid = getmypid();
$file_opt = '--memory-limit-error-file=' . escapeshellarg($error['file']);
$line_opt = '--memory-limit-error-line=' . escapeshellarg($error['line']);
system("docker run --entrypoint=php -it --security-opt=\"apparmor=unconfined\" --cap-add=SYS_PTRACE --pid=host reliforp/reli-prof -dmemory_limit=16G ./reli i:m -p {$pid} --no-stop-process {$file_opt} {$line_opt} >{$pid}.memory_analyzed.json");
}
);This code snippet kicks Reli on memory limit violations via docker, to analyze the memory usage of the dying process itself. Note that as the memory analyzing code of Reli itself isn't very memory efficient currently, I use 16GB memory_limit for the process of Reli on this test. The PHP VM used to testPHP versionphp --versionphp-memprofphp -m | grep memprofThis time I use php-memprof in combination with Reli. Reli is used to analyze where the most memory is used, and php-memprof is used to find out when the memory space is allocated. Run with 512M memory limitTo avoid Reli running out of memory, this time I reserve only 512M for psalm. I run psalm twice: the first run is used to analyse the effects of parallel processing only, while the second run is used to analyse the effects of the cache together. The 1st runMEMPROF_PROFILE=dump_on_limit php -dauto_prepend_file=analyzer.php ./vendor/bin/psalm --memory-limit=512M --threads=2 test.php The output of memprofkcachegrind /tmp/memprof.callgrind.1787474019042074
As you can see, the amount of memory allocated during However, is this really the case? The output from Relimemory_analyzed_reli=14258.memory_analyzed.jsonExtracting the summarycat $memory_analyzed_reli | jq .summaryReli can analyze 75% of the memory usage this time. Not too bad. Top 20 most memory-consuming types of memory locationscat $memory_analyzed_reli | jq .location_types_summary | jq -r '(["location_type", "count", "memory_usage"] | (., map(length*"="))),(to_entries|.[:20]|.[]|[.key,.value.count,.value.memory_usage])|@tsv' | column -t -o ' | 'Strings occupies the most memory space, while the combination of array elements and unused areas of arrays consumes as much memory. Top 10 largest stringscat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="StringContext"))| . as $path | $root|getpath($path) as $string | {path: $path|join("."), size: $string."#locations"[0].size, value: $string."#locations"[0].value, node_id:$string."#node_id"}' | jq -rs '(["size", "node_id" ,"path", "value"] | (., map(length*"="))),(sort_by(.size,.value) | .[-10:] | reverse | .[] | [.size, .node_id, .path, .value])|@tsv' | column -t -o ' | ' -s$'\t'The value is truncated to 256 bytes on Reli. As you can see, the base64-encoded serialised data for IPC occupies the majority of the area for strings. Call trace at the timing of the deathcat $memory_analyzed_reli | jq -r '(["frame_no", "function", "line"] | (., map(length*"="))),(path(.context.call_frames[]|objects) as $path | [$path[2], getpath($path).function_name, getpath($path).lineno])|@tsv' | column -tThis process died on reading the results from the children. Top 20 most memory-consuming classes of instancescat $memory_analyzed_reli | jq .class_objects_summary | jq -r '(["class_name", "count", "memory_usage"] | (., map(length*"="))),(to_entries|.[:20]|.[]|[.key,.value.count,.value.memory_usage])|@tsv' | column -t -o ' | 'Psalm instantiates many objects for representing language constructs in the analyzed codebase, especially for types and code locations. The references of top 100 largest arrayscat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="ArrayHeaderContext"))| . as $path | $root|getpath($path) as $header | $header.array_elements as $elements | {path: $path|join("."), size: $elements."#locations"[0].size, count: $elements."#count", node_id:$header."#node_id"}' | jq -rs '(["size", "count", "node_id" ,"path"] | (., map(length*"="))),(sort_by(.size) | .[-100:] | reverse | .[] | [.size, .count, .node_id, .path])|@tsv' | column -t -o ' | 'Unlike the case of strings, for arrays there are no extremely large arrays on their own, so there must be a large number of relatively small arrays. For example, there are many dynamic_properties later in the list. Oh wait, dynamic properties? Really? Digging into dynamic propertiesAs described in this article, PHP objects may have a hash table for dynamic properties internally. Normally this hash table is not allocated unless it is needed, e.g. for the use of dynamic properties. Let's extract all the dynamic properties tables and get the total size of them. cat $memory_analyzed_reli | jq '. as $root | path(..|objects|select(."#type"=="ArrayHeaderContext"))| select(.[-1] == "dynamic_properties") | . as $path | $root|getpath($path) as $header | $header.array_elements as $elements | $header.possible_unused_area as $unused_area | {path: $path|join("."), size: $elements."#locations"[0].size, unused_size: $unused_area."#locations"[0].size, count: $elements."#count", node_id:$header."#node_id"}' | jq -rs '. as $all | {count: $all|length , total_used_size: [$all[].size]|add, total_unused_size: [$all[].unused_size]|add}'The dynamic properties tables take up about 130MB of memory in total. Then take a look at one of the dynamic properties tables. What keys are there? cat $memory_analyzed_reli | jq '.context.objects_store."75083".object_properties.data.array_elements.classlike_storage.value.array_elements."microsoft\\graph\\generated\\teamstemplates\\item\\teamstemplateitemrequestbuilder".value.dynamic_properties.array_elements|keys'Keys beginning with The object having this table is an instance of This means that there are actually no dynamic properties in this case. For some reason, dynamic properties tables have been unnecessarily constructed. And the reason is in the implementation of This issue is already reported in php-src (php/php-src#10126). The 2nd runMEMPROF_PROFILE=dump_on_limit php -dauto_prepend_file=analyzer.php ./vendor/bin/psalm --memory-limit=512M --threads=2 test.php The output of php-memprofkcachegrind /tmp/memprof.callgrind.1787474184950721
In this second run, I omit the output of Reli here, as it was no different from the first run in that the serialized data for IPC and the dynamic_properties table occupy a large amount of space. A possible workaroundThe default behavior of the PHP VM allocates dynamic properties table unnecessarily. Funnily, defining So if we define I'll send a PR of this. About igbinaryUsing igbinary reduces the size of the serialized data for IPC, but the dynamic properties tables are still unnecessarily constructed even with igbinary. Worse, the workaround via I will also look into the implementation of igbinary when I have time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When running psalm on
microsoft/microsoft-graphit crashes due to spawnining multiple, ~10, PHP processes. Each process continuously increases it's memory footprint, beginning from 0,5GB to over 5GB each. Which effectively makes makes it run out of memory;Composer.json;
psalm.xml
test.php
First thought the issue was with PHPstorm so reported it there, bug report.
The text was updated successfully, but these errors were encountered: