Skip to content

Commit

Permalink
Merge branch 'main' into aeisenberg/audit-fix
Browse files Browse the repository at this point in the history
  • Loading branch information
@aeisenberg
aeisenberg authored May 10, 2021
2 parents 489dbb0 + ef38c53 commit 22d1f7f
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 51 deletions.
115 changes: 66 additions & 49 deletions .github/workflows/python-deps.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,37 +6,18 @@ on:
pull_request:

jobs:

test-setup-python-scripts:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
include:
- test_dir: python-setup/tests/pipenv/requests-2
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 2
- test_dir: python-setup/tests/pipenv/requests-3
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 3

- test_dir: python-setup/tests/poetry/requests-2
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 2
- test_dir: python-setup/tests/poetry/requests-3
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 3

- test_dir: python-setup/tests/requirements/requests-2
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 2
- test_dir: python-setup/tests/requirements/requests-3
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 3

- test_dir: python-setup/tests/setup_py/requests-2
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 2
- test_dir: python-setup/tests/setup_py/requests-3
test_script: $GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh 3

# This one shouldn't fail, but also won't install packages
- test_dir: python-setup/tests/requirements/non-standard-location
test_script: test -z $LGTM_INDEX_IMPORT_PATH
python_deps_type: [pipenv, poetry, requirements, setup_py]
python_version: [2, 3]

env:
PYTHON_DEPS_TYPE: ${{ matrix.python_deps_type }}
PYTHON_VERSION: ${{ matrix.python_version }}

steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
Expand All @@ -55,7 +36,7 @@ jobs:
set -x
$GITHUB_WORKSPACE/python-setup/install_tools.sh
cd $GITHUB_WORKSPACE/${{ matrix.test_dir }}
cd $GITHUB_WORKSPACE/python-setup/tests/${PYTHON_DEPS_TYPE}/requests-${PYTHON_VERSION}
case ${{ matrix.os }} in
ubuntu-latest*) basePath="/opt";;
Expand All @@ -71,35 +52,69 @@ jobs:
if [ ! -z $CODEQL_PYTHON ]; then
$GITHUB_WORKSPACE/python-setup/tests/from_python_exe.py $CODEQL_PYTHON;
fi
- name: Verify packages installed
run: |
${{ matrix.test_script }}
$GITHUB_WORKSPACE/python-setup/tests/check_requests_123.sh ${PYTHON_VERSION}
# This one shouldn't fail, but also won't install packages
test-setup-python-scripts-non-standard-location:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]

steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2

- name: Initialize CodeQL
uses: ./init
id: init
with:
tools: latest
languages: python
setup-python-dependencies: false

- name: Test Auto Package Installation
run: |
set -x
$GITHUB_WORKSPACE/python-setup/install_tools.sh
cd $GITHUB_WORKSPACE/python-setup/tests/requirements/non-standard-location
case ${{ matrix.os }} in
ubuntu-latest*) basePath="/opt";;
macos-latest*) basePath="/Users/runner";;
esac
echo ${basePath}
$GITHUB_WORKSPACE/python-setup/auto_install_packages.py "$(dirname ${{steps.init.outputs.codeql-path}})"
- name: Setup for extractor
run: |
echo $CODEQL_PYTHON
# only run if $CODEQL_PYTHON is set
if [ ! -z $CODEQL_PYTHON ]; then
$GITHUB_WORKSPACE/python-setup/tests/from_python_exe.py $CODEQL_PYTHON;
fi
- name: Verify packages installed
run: |
test -z $LGTM_INDEX_IMPORT_PATH
test-setup-python-scripts-windows:
runs-on: windows-latest
strategy:
fail-fast: false
matrix:
include:
- test_dir: python-setup/tests/pipenv/requests-2
python_version: 2
- test_dir: python-setup/tests/pipenv/requests-3
python_version: 3

- test_dir: python-setup/tests/poetry/requests-2
python_version: 2
- test_dir: python-setup/tests/poetry/requests-3
python_version: 3

- test_dir: python-setup/tests/requirements/requests-2
python_version: 2
- test_dir: python-setup/tests/requirements/requests-3
python_version: 3

- test_dir: python-setup/tests/setup_py/requests-2
python_version: 2
- test_dir: python-setup/tests/setup_py/requests-3
python_version: 3
python_deps_type: [pipenv, poetry, requirements, setup_py]
python_version: [2, 3]

env:
PYTHON_DEPS_TYPE: ${{ matrix.python_deps_type }}
PYTHON_VERSION: ${{ matrix.python_version }}

steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
Expand All @@ -117,17 +132,19 @@ jobs:
$cmd = $Env:GITHUB_WORKSPACE + "\\python-setup\\install_tools.ps1"
powershell -File $cmd
cd $Env:GITHUB_WORKSPACE\\${{ matrix.test_dir }}
cd $Env:GITHUB_WORKSPACE\\python-setup/tests/$Env:PYTHON_DEPS_TYPE/requests-$Env:PYTHON_VERSION
$DefaultsPath = Join-Path (Join-Path $Env:GITHUB_WORKSPACE "src") "defaults.json"
$CodeQLBundleName = (Get-Content -Raw -Path $DefaultsPath | ConvertFrom-Json).bundleVersion
$CodeQLVersion = "0.0.0-" + $CodeQLBundleName.split("-")[-1]
py -3 $Env:GITHUB_WORKSPACE\\python-setup\\auto_install_packages.py C:\\hostedtoolcache\\windows\\CodeQL\\$CodeQLVersion\\x64\\codeql
- name: Setup for extractor
run: |
echo $Env:CODEQL_PYTHON
py -3 $Env:GITHUB_WORKSPACE\\python-setup\\tests\\from_python_exe.py $Env:CODEQL_PYTHON
- name: Verify packages installed
run: |
$cmd = $Env:GITHUB_WORKSPACE + "\\python-setup\\tests\\check_requests_123.ps1"
powershell -File $cmd ${{ matrix.python_version }}
powershell -File $cmd $Env:PYTHON_VERSION
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ You may want to run `tsc --watch` from the command line or inside of vscode in o

### Checking in compiled artifacts and `node_modules`

Because CodeQL Action users consume the code directly from this repository, and there can be no build step during an GitHub Actions run, this repository contains all compiled artifacts and node modules. There is a PR check that will fail if any of the compiled artifacts are not up to date. Compiled artifacts are stored in the `lib/` folder. For all day-to-day development purposes, this folder can be ignored.
Because CodeQL Action users consume the code directly from this repository, and there can be no build step during an GitHub Actions run, this repository contains all compiled artifacts and node modules. There is a PR check that will fail if any of the compiled artifacts are not up to date. Compiled artifacts are stored in the `lib/` directory. For all day-to-day development purposes, this folder can be ignored.

Avoid running `npm install` (the node moddules should be up to date when you check out anyway) and instead use `npm ci` when you want to update dependencies `package.json`. If you make any changes to the `node_modules` folder , you must run `npm run removeNPMAbsolutePaths` to clean the folder and remove any locally identifying data. There is a PR check to ensure that this command has been run.
Only run `npm install` if you are explicitly changing the set of dependencies in `package.json`. The `node_modules` directory should be up to date when you check out, but if for some reason, there is an inconsistency use `npm ci && npm run removeNPMAbsolutePaths` to ensure the directory is in a state consistent with the `package-lock.json`. There is a PR check to ensure the consistency of the `node_modules` directory.

### Running the action

Expand Down

0 comments on commit 22d1f7f

Please sign in to comment.