Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve detection to avoid memory over-usage #400

Merged
merged 3 commits into from
Jul 3, 2024

Conversation

sobregosodd
Copy link
Contributor

@sobregosodd sobregosodd commented Jun 25, 2024

Rules using <... PATTERN ...> consume a lot of memory, thus it usage should be avoided.

  • npm-dll-hijacking.yml: Rule was using <... PATTERN ...>
  • npm-obfuscation: Rule was using <... PATTERN ...> and other optimizations and simplifications were introduced
output: {"errors": [{"code": 2, "level": "error", "message": "Error while running rules: \n
                    You are seeing this because the engine was killed.\n\n
                    The most common reason this happens is because it used too much memory.\n                    If your repo is large (~10k files or more), you have three options:\n                    1. Increase the amount of memory available to semgrep\n                    2. Reduce the number of jobs semgrep runs with via `-j <jobs>`. We\n                        recommend using 1 job if you are running out of memory.\n                    3. Scan the repo in parts (contact us for help)\n\n                    Otherwise, it is likely that semgrep is hitting the limit on only some\n                    files. In this case, you can try to set the limit on the amount of memory\n                    semgrep can use on each file with `--max-memory <memory>`. We recommend\n                    lowering this to a limit 70% of the available memory. For CI runs with\n                    interfile analysis, the default max-memory is 5000MB. Without, the default\n                    is unlimited.\n\n                    The last thing you can try if none of these work is to raise the stack\n                    limit with `ulimit -s <limit>`.\n\n                    If you have tried all these steps and still are seeing this error, please\n                    contact us.\n\n                       Error: semgrep-core exited with unexpected output\n\n                       [00.05][\u001b[34mINFO\u001b[0m]: Executed as: /Users/sebastian.obregoso/Library/Caches/pypoetry/virtualenvs/guarddog-PTZDzYc9-py3.10/lib/python3.10/site-packages/semgrep/bin/semgrep-core -json -rules /var/folders/83/v1gvs5x976xdn94tnyk95j8r0000gq/T/tmpau5oqo8e.json -j 10 -targets /var/folders/83/v1gvs5x976xdn94tnyk95j8r0000gq/T/tmpxbi2cjv7 -timeout 5 -timeout_threshold 3 -max_memory 0 -fast\n[00.05][\u001b[34mINFO\u001b[0m]: Version: semgrep-core version: 1.77.0\n\n                    ", "type": "SemgrepError"}], "paths": {"scanned": []}, "results": [], "skipped_rules": [], "version": "1.77.0"}

@sobregosodd sobregosodd marked this pull request as ready for review June 25, 2024 16:37
Copy link
Contributor

@christophetd christophetd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice one!

@christophetd christophetd merged commit fd5017c into main Jul 3, 2024
10 checks passed
@christophetd christophetd deleted the s.obregoso/fix_npm_obfuscation branch July 3, 2024 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants