-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Customization of results #729
Comments
/cc @raghavkaul |
This should be more of a config option. But isn't this supposed to be in Scorecard rather than action? That way it is consistent and Scorecard action is just another way to run scorecard. |
The Scorecard Action can build a policy on top of the results, in which case this feature need / should not be in Scorecard, especially if we don't want to enforce / support a universal policy style / language / framework. That's what I had in mind. This keeps the Scorecard results "universal", same as the cron results. The policy could be applied on top of cron results, for example, using whatever logic the consumer wants. Wdut? |
Another customization to consider is improving |
yes, that's the 143 issue above. I'll update the description. Thanks |
Ah sorry, didn't see it. My bad. |
Can you please unpack what you mean with an example? For example, I envisioned that end-users can apply a policy on top scorecard results to fit their needs. For example
|
pretty much what you said. Not sure that score customization is useful if we have more structured input, but maybe. They could also customize the criticality of results, or ignore certain "details" using the detailID field suggested in ossf/scorecard#1874 (comment) Let me know if this is a bit clearer or not. Thanks for asking |
Yes, it is! Thanks. Then why not have this functionality in Scorecard versus here. That is what I don't understand. |
Thanks @naveensrinivasan for pushing on this. Initially I thought we would not want to commit to a particular policy format in the main scorecard repo, and let consumers decide how they want to handle it. You're making a good point that we could support a simple policy format in the main scorecard repo, and it would benefit everyone. So I think I'm fine with that. @azeemshaikh38 any further thought about this? |
fyi, here's a possible policy file: Version: X
BasePolicy: BuiltInAllChecksEnforced | BuiltInAllChecksEnforced | __path_to_file__
IgnoredPaths:
IgnoredFiles:
EnforcedPaths:
EnforcedFiles:
Checks:
- CheckName:
- Status: enforced | ignored
- Criticality: XXX
- IgnoredPaths:
- a_path_
- a/**/path
- IgnoredFiles:
- a_file
- another_file
- EnforcedPaths:
...
- EnforcedFiles:
...
- Rules:
# RuleName is the same as `DetailID` in https://github.com/ossf/scorecard/issues/1874#issuecomment-1178259870
- RuleName:
- Status: enforced | ignored
- Criticality: XXX
- IgnoredPaths:
- IgnoredFiles:
- EnforcedPaths:
- EnforcedFiles: |
The community is leaning towards OPA https://github.com/open-policy-agent/opa. |
Yes. Kind of why I suggested earlier to have something simple only in the Action and not commit to something complex / to main in the main CLI as well. I have not looked into OPA or the complexity for both us or the user. I think a simple solution first would help users and would not require them to learn a new programming language. AllStar takes this approach and is able to express many policies this way. Later we can provide template OPA policy (I think we discussed this a few months ago but did not have bandwidth to look more closely). Wdut? |
That would be breaking a change in Policy. If I am not wrong @developer-guy was working on cosign with OPA. |
A lot of people want ways to opt out. Having a simple policy and actually delivering it seems like a good step forward to me. It's not always viable to do everything at once, and iterations are always necessary. We don't even know what other policies would want, so betting on OPA seems overkill. |
We should have some to measure This should be a discussion topic for bi-weekly |
I went thru the issues last week and found 3-4 issues of people asking for customization like this. It's also something people have been complaining about in conversations. |
OK, Go ahead if you have heard from people in conversations 👍 . I just wanted to be sure that we are building what customers want. Thanks for the explanation! |
I'd rather hear from more people, and also talk about it in the bi-weekly as you mentioned. Was just trying to say there's demand, but I agree with you on discussing it to decide on prioritization. Thanks again for all the questions you asked |
/cc @ethanent @jeffmendoza may be relevant to the work on AllStar policies for Scorecard. |
I requested a way to silence reports in #143. In that case, I want to relax the requirements for CI code. Not all code needs to uphold the same quality measures, mainly test code and CI code. The proposed policy file would work for me. It would allow suppressing the Pinned-Dependency check for the GitHub Actions code, while keeping it for the rest. It also looks flexible enough to further differentiate between different projects in the same repo or other parts. In the proposed policy file, there are both IgnoredPaths and IgnoredFiles. This seems somewhat redundant and a single option might work, similar to the syntax of gitignore files. |
Some ideas
Won't Fix
alerts are still shown github/codeql-action#986 (comment))The text was updated successfully, but these errors were encountered: