-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using a few OSSF scorecard checks to show more information to Actions consumers #1107
Comments
@ossf/scorecard-maintainers fyi. Thanks @svemu. The list of checks looks good to me. What help can we offer here to get this effort going? |
Hi @azeemshaikh38, I have a couple of questions I wanted some help with -
cc: @shruti-shukla |
Could you clarify how you plan to do this at scale (i.e for all GitHub Action providers)? Would every GitHub Action repo be required to include scorecard-action in their project or would GitHub internally run Scorecard for these repos for all triggers (commit push, branch protection update etc.) ?
If GitHub plans to re-create something similar to scorecard-action with support for running selected checks, could I suggest that you folks could directly contribute to
AFAIK, we don't test scorecard-action against windows runners so the confidence there is not high. Its completely likely that it would be supported on windows without any extra code so we would just need extra tests to productionize it. @laurentsimon can you think of any reason we may not be able to support windows?
This may not be needed if we can use scorecard-action to support your usecase directly?
Mostly testing. All our tests are heavily Ubuntu/Linux focused.
Yes it is safe and this is how scorecard-action also does it. The only problem I anticipate is that this is expected to be an internal only feature (hence the lack of documentation). Even though this will be supported in the long-term the usage may change.
Hmm, not sure we have something like that. What specifically were you looking for here? |
re: windows runner. The go binary should run on Windows so I don't expect problems. But as @azeemshaikh38 said, most our tests run on ubuntu runner. I think we could add some tests in the scorecard-action to run on windows runners, if we want more certainty. We can definitely do that if needed. |
We plan to run it internally on a schedule and for new releases of the action. Users will not have to do anything actively to run this.
Would love to contribute and get this feature in, just want to be sure if this is already in scope or planned, if not we can definitely look into contributing and raising a PR and would be great if any approvals on this can be prioritized so our timelines aren't at risk
Even with the action it would be great if we can possibly pass in a custom value for the tool name in the SARIF output, as it still helps to show user filtered results based on different runs. (Custom run added by user itself vs Run that GitHub would trigger).
If we can get the scorecard-action itself to run selected checks then we wont have to worry about it, till then we can keep a lookout on this.
Just wanted to see if there are known common errors that we can put safeguards for, since we are trying to run this internally we want to ensure that it runs without a hitch and does not show up as a failed attempt to the users.
It would be great if we can get certain tests added, it would give us more confidence in running on windows based runners. |
Not planned, but this is definitely within scope. If you could provide a brief description of the design/solution for configuring the checks in scorecard-action we can make sure we are all in agreement before you spend your time on coding it. And yes, I can review these PRs for you on priority.
Not sure I understood this - IIUC the plan is to show Scorecard info to Action consumers and not Action repo devs? How would Actions consumers have access to custom run added by user itself? |
@anuraag016 if GitHub runs the Action themselves, is there anything specific to your environment for the Action to detect it's being run by you and not the user? I ask because we could add some logic into the Action to detect it's running on your environment, and allow you to pass certain parameters, but without releasing this functionality to end-users. Just something to consider to reduce the amount of maintenance and support on the Action side. Once there is a proper design we're comfortable releasing for end-users (selection of checks, etc), we would then migrate your "special" integration to the "public" version of it (in the future). |
@azeemshaikh38 The plan is to show the scorecard info to both the Action repo owners and the consumers of the actions. For the Repo owners we want to remove the confusion of custom run vs the Run that GitHub would trigger. For the customers we would ensure that we pick only the results that were created by our initiated runs.
@laurentsimon yes there is a way we can identify if the workflow is run by us or by user. The event_name in the github context will have a reserved event name that can not be used by users.
This would be great if it means we can get these functionalities faster. We would be able to directly use the action instead of relying on the command line tool. |
Sounds reasonable to me. To summarize, the plan is to:
Does that sound right @anuraag016 ? @laurentsimon @ossf/scorecard-maintainers fyi. |
Let's also keep these changes "private", ie we will make the changes for the integration and don't publicize them? @anuraag016 if we were to implement the check selection via a hidden env variable, and in a few months decide we will provide a config file for public use, do you have a scalable way to patch your workflow across projects (env -> config)? NOTE: there's also this workaround #1098 (comment) that you could use to select checks, which does not require changes. Would this work? |
follow-up question: would you be pushing the results back to our servers? The Action has an option The setup uses OIDC to prove to the backend that the results originate from the correct repo. We do some verification on the workflow.yml that run scorecard before ingesting the results. I doubt this would work in your setup, since the workflow is not part of the repo's history. |
@azeemshaikh38 Yes, these seem right. One more thing is, if we can get the action working in Windows environment as well.
@laurentsimon This would work for us perfectly. Our use case is of running a workflow for repos of other users, best approach for us would be argument based selection of checks. We can go with a config file or any other way as well, just that would require a little extra maintenance on our side.
Yes, since we run the workflow dynamically for repos we can change the workflow file centrally and the new one would be used in all future runs.
We had thought of a similar workaround, but decided against it because it ends up running all checks which would mean using extra compute power than necessary, increasing the cost of running it for all the action repos present in GitHub. But this can be used to change the scorecard tool name without any changes to the action.
We are not planning on publishing the results from the workflow that we run. The authors can and will have the possibility of manually adding the OSSF starter workflow to run the checks and upload the results if they wish. |
Thanks for all the info. When do you ideally need a first PoC to try? |
@laurentsimon We plan to start testing the workflow internally by end of this month or start of the next month. An initial PoC by that time would be the most ideal for us. |
I will make a branch with the changes.
What's the event name? I''ll use this in the code to enable this internal feature just for your use case. Thanks |
Since the discussion is about the scorecard-action, I have transferred the issue to the Action repo #1107 |
I send ossf/scorecard#2773. I think it's actually simpler to make changes only to the Scorecard code. |
I like this idea as it will improve the security posture of the GitHub Actions ecosystem. Please consider adding |
I would be very cautious about indiscriminately showing metrics from the scorecard. There’s a bunch of them that don’t apply to every ecosystem (binary artifacts, for example), and some that imply inactionable recommendations (like, “go get another maintainer”). “Pinned dependencies” would be actively harmful if it applies to npm packages, and it’s not actually a good idea to apply to actions in every case, since you’d be depriving yourself of automatic bug and security fixes. |
@ljharb GitHub Actions are applications so it should avoid the problem. @anuraag016 I've made a test branch for you to try: - uses: ossf/scorecard-action@9ea03ba18bab6e2ea29983f3b0fabdb8fd20ecbc # gh-integration
env:
SCORECARD_INTERNAL_GITHUB_INTEGRATION: 1
SCORECARD_INTERNAL_GITHUB_SARIF_TOOL_NAME: tool-name
SCORECARD_INTERNAL_GITHUB_CHECKS: Binary-Artifacts, Security-Policy Please tell me if it works. Note: the tool name will have its first letter capitalized automatically https://github.com/ossf/scorecard/blob/main/pkg/sarif.go#L421. In the example above, it would be |
@laurentsimon i'm not sure what you mean by that - I run github actions on hundreds of packages. GHA is in no way unique to applications. |
I meant that GitHub Action are not "libraries". The GitHub tutorials ask users to use this template https://github.com/actions/javascript-action which uses a lock file. And to use dependabot to receive updates. |
ahhh ok, so you're talking about providing scores for actions not for the projects that consume those actions. That makes sense, thanks for clarifying! |
We at GitHub Actions are thinking of surfacing the security posture of a Repo that hosts Actions code on the Actions listing pages.
While there is information we can get by making API calls(E.g: Branch protection enabled or not), for the others (E.g:Binary artifacts, dangerous workflows )we are looking at using the OSSF checks (use the CLI in a workflow and get the score) and start to display that information on the Actions listing page. Since this is a new change, there will be enough time for the Action creators to fix before the information is shown on the listing page.
We looked at all the OSSF checks and using the following checks in the critical, high and medium categories as something that we need to check for and surface to the consumer of the Action.
Dangerous-Workflow
Binary-Artifacts
Branch-Protection
Code-Review
Dependency-Update-Tool
Vulnerabilities
Pinned-Dependencies
SAST
We will use the same scoring process for each of the checks and display the results
CC: @azeemshaikh38 , @laurentsimon
The text was updated successfully, but these errors were encountered: