-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(Feedback) Unclear value proposition for Action #1017
Comments
Scorecards add new checks which based on findings and trends. If the team is on top of the security updates and would find scorecard as redundant then they have a choice to make. Convincing everyone is hard. |
On a somewhat related note I wonder why |
Looking at the action a bit closer it appears it still can't handle PRs so issues like systemd/systemd#23693 (comment) can be caught only after PRs are merged (which isn't as useful as it can be). It's probably worth mentioning that in that particular case I have to admit I have no idea how the action in its current form ended up in the systemd repository. (I was on vacation and probably missed PRs where it was introduced). I'll take a closer look. Anyway I really hope that links to bogus bolt-on "security" stuff can be removed from https://api.securityscorecards.dev/projects/github.com/systemd/systemd and the security tab. |
It appears those links are all over the place partly because due to I assume some
I'm not sure why what should actually be debug messages pop up as info messages and why exactly top-level read permissions should be remediated by updating all the workflows. The "warn" shouldn't be there either (and as far as I understand that's what ossf/scorecard#2338 was supposed to address). The remaining links come from the "Pinned-Dependencies" check and flag CIFuzz and ClusterFuzzLite. They shouldn't be flagged either. As far as I can tell if the underlying scorecard bugs were fixed those links wouldn't even pop up anywhere. |
Hi @evverx thanks for reaching out, I really appreciate you sharing your concerns on the tool once you are now using it, it is a valuable feedback! About the link you mentioned, you mean the ones in the suggestions bellow? Just to clarify we are not suggesting (in this suggestions) or recommending the Harden Runner, the link mentioned is only about the online tool provided by the step security which you can use to automatically pin all dependencies (Pin-Dependencies Check) and also set the content permission to read only by default (Token Permission Check). That's why the flags marked should be the first (enable=permissions) and third ones (enable=pin), probably all of them have been flagged and since the Harden Runner is a Step Security tool, it was also added to the systemd/systemd#25205 (comment) PR, but it is not related to any scorecard check. About the Scorecard running on PRs, it currently runs on PRs but we have recently created a issue to really make the run on PR useful to maintainers #1019. About the token permissions, it seems that you are right, it doesn't make sense this json output "links" in the Token-Permission check, since there is nothing there to fix. I'll open an issue to look it closely and see if I am able to try to fix it myself to help scorecard team. About the Pinned-Dependencies, good to see that scorecard team is already aware that it is not really a security issue not to pin the action that has read only permissions (ossf/scorecard#2018) but this would not interfere with this links mentioned in the Token-Permissions. At least the Infos in the Pinned Dependencies details seems right
|
I agree that FWIW those semi-automated PRs can actually be considered harmful in the case of CFLite: ossf/scorecard#1907 (comment)
Good to know. Thanks!
Thanks!
I think the scorecard team has been aware of that issue (and a bunch of other issues) for a long time. The problem is that they have never been addressed for various reasons.
Looks like it. |
@evverx, please feel free to create an issue or discussion at https://github.com/step-security/harden-runner if you think there are areas of improvement. I also replied to your comment on systemd/systemd#25205 (comment). The whole point of having it be open source is that developers look at how it works under the hood and help improve it. |
From systemd/systemd#25205 (comment)
It was set to AGPL after I pointed out that it was relisenced on the fly as soon as it was integrated into the scorecard documentation masquerading as an open-source project with the Apache license. I doubt it would be AGPL now if that move hadn't been noticed. Regarding the harden runner I appreciate the details but I'm not here to audit or help to improve it. I think if anyone is interested they can take a look at the code and decide for themselves how hard it is to escape from that "sandbox" (or whether they need to escape from it at all). |
I already explained earlier that I was new to open-source licenses. I looked at a few projects and decided to use the license that dependabot uses. After getting feedback, I apologized, and promptly updated the license. As I mentioned before, we haven’t changed the project license since then and don’t intend to do so in the future.
That is fair. Request you to please not refer to it as bogus without a thorough audit or review. We spend a significant amount of effort building & maintaining these open-source solutions and there are many open-source contributors who provide constructive feedback. Reading a comment that the project is bogus without proper reasoning or discussion is disheartening. |
Is this some sort of cease and desist? Should I also remove the comments where I said that those semi-automated PRs fix imaginary "security" issues and in the case of CFLite they are actually malicious? |
Updating a case of unclear value proposition of the tool. Scorecard is not able to identify Cargo dependencies in Pinned-Dependencies check, it is disagreeable if some dependencies should be pinned or not in a library case, and Clippy tool is not detected in SAST check. This is seen as an "incorrect" analysis by maintainers and leads to a lack of trust of wheather the security recommendations are worth or not. Additionally, the |
In some discussions, maintainers questioned what are the real benefit of having scorecard running as an action with arguments such as:
PHP
Curl
How can we improve or show the value of scorecard as an action?
The text was updated successfully, but these errors were encountered: