- Swift analysis on Ubuntu is no longer supported. Please migrate to macOS if this affects you.
- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.3.
- Resolves CVE-2023-4759, an arbitrary file overwrite in Eclipse JGit that can be triggered when using untrusted third-party queries from a git repository. See the security advisory for more information.
- The following dependencies have been updated. These updates include
security fixes in the respective libraries that prevent
out-of-bounds accesses or denial-of-service in scenarios where
untrusted files are processed. These scenarios are not likely to be
encountered in most uses of CodeQL and code scanning, and only
apply to advanced use cases where precompiled query packs,
database ZIP files, or database TRAP files are obtained from
untrusted sources and then processed on a trusted machine.
- airlift/aircompressor is updated to version 0.27.
- Apache Ant is updated to version 1.10.11.
- Apache Commons Compress is updated to version 1.26.0.
- Apache Commons IO is updated to version 2.15.1.
- Apache Commons Lang3 is updated to version 3.14.0.
- jsoup is updated to version 1.15.3.
- Logback is updated to version 1.2.13.
- Snappy is updated to version 0.5.
- The experimental type
QlBuiltins::BigInt
of arbitrary-precision integers has been introduced. To opt in to this API, compile your queries with--allow-experimental=bigint
. Big integers can be constructed using the.toBigInt()
methods ofint
andstring
. The built-in operations are:- comparisons:
=
,!=
,<
,<=
,>
,>=
, - conversions:
.toString()
,.toInt()
, - arithmetic: binary
+
,-
,*
,/
,%
, unary-
, - bitwise operations:
.bitAnd(BigInt)
,.bitOr(BigInt)
,.bitXor(BigInt)
,.bitShiftLeft(int)
,.bitShiftRightSigned(int)
,.bitNot()
, - aggregates:
min
,max
, (strict
)sum
, (strict
)count
,avg
,rank
,unique
,any
. - other:
.pow(int)
,.abs()
,.gcd(BigInt)
,.minimum(BigInt)
,.maximum(BigInt)
.
- comparisons:
codeql test run
now supports postprocessing of test results. When .qlref files specify a path to apostprocess
query, then this is evaluated after the test query to transform the test outputs prior to concatenating them into theactual
results.
- The 30% QL query compilation slowdown noted in 2.18.0 has been fixed.
- A number of breaking changes have been made to the C and C++ CodeQL
test environment as used by
codeql test run
:-
The test environment no longer defines any GNU-specific builtin macros. If these macros are still needed by a test, please define them via
semmle-extractor-options
. -
The
--force-recompute
option is no longer directly supported bysemmle-extractor-options
. Instead,--edg --force-recompute
should be specified. -
The
--gnu_version
and--microsoft_version
options that can be specified viasemmle-extractor-options
are now synonyms, and only one should be specified as part ofsemmle-extractor-options
. Furthermore, is also no longer possible to specify these options via the following syntax.--edg --gnu_version --edg <version number>
, and--edg --microsoft_version --edg <version number>
The shorter
--gnu_version <version number>
and--microsoft_version <version number>
should be used.
-
- The
--build_error_dir
and--predefined_macros
command line options have been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI, but some customers with advanced setups may have been passing them through internal undocumented interfaces. Passing the option--build_error_dir
did not have any effect, and it is safe to remove the option. The--predefined_macros
option should have been unnecessary, as long as the extractor was invoked with the--mimic
option.
- Compilation of QL queries is about 30% slower than in previous releases. This only affects users who write custom queries, and only at compilation time, not at run time. This regression will be fixed in the upcoming 2.18.1 release.
- Introduced the
--include-logs
option to thecodeql database bundle
command. This new feature allows users to include logs in the generated database bundle, allowing for a more complete treatment of the bundle, and bringing the tool capabilities up-to-speed with the documentation. codeql database init
andcodeql database create
now support the--force-overwrite
option. When this option is specified, the command will delete the specified database directory even if it does not look like a database directory. This option is only recommended for automation. For directcommand line commands, it is recommended to use the--overwrite
option, which includes extra protection and will refuse to delete a directory that does not look like a database directory.- Extract
.xsaccess
,*.xsjs
and*.xsjslib
files for SAP HANA XS as Javascript. - We have updated many compiler error messages and warnings to improve their
readability and standardize their grammar.
Where necessary, please use the
--learn
option for thecodeql test run
command.
- Where a MacOS unsigned binary cannot be signed, CodeQL will now continue trying to trace compiler invocations created by that process and its children. In particular this means that Bazel builds on MacOS are now traceable.
- Fixed a bug where test discovery would fail if there is a syntax error in a qlpack file. Now, a warning message will be printed and discovery will continue.
- Beta support is now available for analyzing C# codebases without needing a working build. To use
this, pass the
--build-mode none
option tocodeql database create
.
- The
--model-packs
option is now publicly available. This option allows commands likecodeql database analyze
to accept a list of model packs that are used to augment the analysis of all queries involved in the analysis.
-
All the commands that output SARIF will output a minified version to reduce the size. The
codeql database analyze
,codeql database interpret-results
,codeql generate query-help
, andcodeql bqrs interpret
commands support the option--no-sarif-minify
to output a pretty printed SARIF file. -
A number of breaking changes have been made to the
semmle-extractor-options
functionality available for C and C++ CodeQL tests.- The Arm, Intel, and CodeWarrior compilers are no longer supported and the
--armcc
,--intel
,--codewarrior
flags are now ignored, as are all the flags that only applied to those compilers. - The
--threads
and-main-file-name
options, which did not have any effect on tests, are now ignored. Any specification of these options as part ofsemmle-extractor-options
should be removed. - Support for
--linker
, all flags that would only invoke the preprocessor, and the/clr
flag have been removed, as those flags would never produce any usable test output. - Support for the
--include_path_environment
flag has been removed. All include paths should directly be specified as part ofsemmle-extractor-options
. - Microsoft C/C++ compiler response files specified via
@some_file_name
are now ignored. Instead, all options should directly be specified as part ofsemmle-extractor-options
. - Support for Microsoft
#import
preprocessor directive has been removed, as support depends on the availability of the Microsoft C/C++ compiler, and availability cannot be guaranteed on all platforms while executing tests. - Support for the Microsoft
/EHa
,/EHs
,/GX
,/GZ
,/Tc
,/Tp
, and/Zl
flags, and all/RTC
flags have been removed. Any specification of these options as part ofsemmle-extractor-options
should be removed. - Support for the Apple-specific
-F
and-iframework
flags has been removed. The-F
flag can still be used by replacing-F <directory>
by--edg -F --edg <directory>
. Any occurrence of-iframework <arg>
should be replaced by--edg --sys_framework --edg <arg>
. - Support for the
/TC
,/TP
, and-x
flags has been removed. Please ensure all C, respectively C++, source files have a.c
, respectively.cpp
, extension. - The
--build_error_dir
,-db
,--edg_base_dir
,--error_limit
,--src_archive
,--trapfolder
, and--variadic_macros
flags are now ignored.
The above changes do not affect the creation of databases through the CodeQL CLI, or when calling the C/C++ extractor directly with the
--mimic
or--linker
flags. Similar functionality continues to be supported in those scenarios, except for CodeWarrior and the--edg_base_dir
,--include_path_environment
,/Tc
, and/Tp
flags, which were never supported. - The Arm, Intel, and CodeWarrior compilers are no longer supported and the
codeql generate log-summary
now reports completed pipeline runs that are part of an incomplete recursive predicate.
- The OWASP Java HTML Sanitizer library used by the CodeQL CLI for internal documentation generation commands has been updated to version 20240325.1.
- CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0.
- The language server that our IDE integration is built on now defaults to fine-grained dependency tracking for incremental error-checking after file changes. This slightly improves the latency of refreshing errors after local source code edits and will enable significant speedups in the future.
- We now properly handle globs (such as
folder/**/*.py
) inpaths
configuration to specify what files to include for Python analysis (see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#specifying-directories-to-scan). - TRAP import (a part of
codeql database create
andcodeql database finalize
) now supports allocating 2^32 IDs during the import process. The previous limit was 2^31 IDs.
- The beta support for analyzing Swift in this release and all
previous releases requires
g++-13
when running on Linux. Users analyzing Swift using theubuntu-latest
,ubuntu-22.04
, orubuntu-20.04
runner images for GitHub Actions should update their workflows to installg++-13
. For more information, see the runner images announcement.
-
When uploading a SARIF file to GitHub using
codeql github upload-results
, the CodeQL CLI now waits for the file to be processed by GitHub. If any errors occurred during processing of the analysis results, the command will log these and return a non-zero exit code. To disable this behaviour, pass the--no-wait-for-processing
flag.By default, the command will wait for the SARIF file to be processed for a maximum of 2 minutes, however this is configurable with the
--wait-for-processing-timeout
option. -
The build tracer is no longer enabled when using the
none
build mode to analyze a compiled language, thus improving performance.
- The
--mode
option and-m
alias tocodeql database create
,codeql database cleanup
, andcodeql dataset cleanup
has been deprecated. Instead, use the new--cache-cleanup
option, which has identical behavior.
- Improved the diagnostic message produced when no code is processed
when creating a database. If a build mode was specified using
--build-mode
, the message is now tailored to your build mode.
- The
scc
tool used by the CodeQL CLI to calculate source code baseline information has been updated to version 3.2.0.
- The
--[no-]analysis-summary-v2
and--[no-]new-analysis-summary
options that were used to enable (or disable) improved summary information printed at the end of acodeql database analyze
invocation are no longer supported. Improved summary information is now enabled for all invocations. - Support for overwriting default CodeQL SARIF run properties using the
--sarif-run-property
command line option has been removed. This removes the ability to overwrite thesemmle.formatSpecifier
,metricResults
, andcodeqlConfigSummary
properties in the SARIF run file.
-
TRAP import (a part of
codeql database create
andcodeql database finalize
) now performs better in low-memory situations. (Put another way, it now needs less RAM to achieve the same performance as before.) -
The worst-case performance of transitive closure computation (using the
+
or*
postfix operators or thefastTC
higher-order primitive in QL) has been greatly improved.
- The build of Eclipse Temurin OpenJDK that is used to run the CodeQL CLI has been updated to version 21.0.2.
- Fixes a bug where extractor logs would be output at a lower than expected
verbosity level when using the
codeql database create
command.
- Beta support has been added for analyzing Java codebases without needing a working build. To enable
this, pass the
--build-mode none
option tocodeql database create
.
-
A number of internal command line options (
--builtin_functions_file
,--clang_builtin_functions
,--disable-objc-default-synthesize-properties
,--list_builtin_functions
,--memory-limit-bytes
,--mimic_config
, and--objc
) has been removed from the C/C++ extractor. It has never been possible to pass these options through the CLI itself, but some customers with advanced setups may have been passing them through internal undocumented interfaces. All of the removed options were already no-ops, and will now generate errors.The
--verbosity
command line option has also been removed. The option was an alias for--codeql-verbosity
, which should be used instead.
- The frontend of the C/C++ extractor has been updated, improving the extractor's reliability and increasing its ability to extract source code.
-
When parsing user-authored YAML files such as
codeql-pack.yml
,qlpack.yml
,codeql-workspace.yml
, and any YAML file defining a data extension, unquoted string values starting with a*
character are now correctly interpreted as YAML aliases. Previously, they were interpreted as strings, but with the first character skipped.If you see a parse error similar to
while scanning an alias... unexpected
character found *(42)
,it likely means that you need to add quotes around the indicated string value. The most common cause is unquoted glob patterns that start with*
, such asinclude: **/*.yml
, which will need to be quoted asinclude: "**/*.yml"
.
- Fixes CVE-2024-25129, a limited data exfiltration vulnerability that could be triggered by untrusted databases or QL packs. See the security advisory for more information.
-
A new extractor option has been added to the Python extractor:
python_executable_name
. You can use this option to override the default process the extractor uses to find and select a Python executable. Pass one of--extractor-option python_executable_name=py
or--extractor-option python_executable_name=python
or--extractor-option python_executable_name=python3
to commands that run the extractor, for example:codeql database create
.On Windows machines, the Python extractor will expect to find
py.exe
on the systemPATH
by default. If the Python executable has a different name, you can set the new extractor option to override this value and look forpython.exe
orpython3.exe
.For more information about using the extractor option with the CodeQL CLI, see Extractor options.
- Fixed a bug where CodeQL may produce an invalid database when it exhausts all available ID numbers. Now it detects the condition and reports an error instead.
- There are no user-facing changes in this release.
- When executing the
codeql database init
command, the CodeQL runner executable path is now stored in theCODEQL_RUNNER
environment variable. Users of indirect tracing on MacOS with System Integrity Protection enabled who previously had trouble with indirect tracing should prefix their build command with this path. For example,$CODEQL_RUNNER build.sh
.
- Name clashes between weak aliases (i.e. aliases that are not final aliases of non-final entities) of the same target no longer cause ambiguity errors.
- Users specifying extra tracing configurations may now use the
GetRegisteredMatchers(languageId)
Lua function to retrieve the existing table of matchers registered to a given language.
- The
Experimental
flag has been removed from all packaging and related commands. - The RA pretty-printer omits names of internal RA nodes and pretty-prints binary unions with nested internal unions as n-ary unions. VS Code extension v1.11.0 or newer is required to compute join order badness metrics in VS Code for the new RA format.
-
The Python extractor will no longer extract dependencies by default. See https://github.blog/changelog/2023-07-12-code-scanning-with-codeql-no-longer-installs-python-dependencies-automatically-for-new-users/ for more context. In versions until 2.17.0, it will be possible to restore the old behavior by setting
CODEQL_EXTRACTOR_PYTHON_FORCE_ENABLE_LIBRARY_EXTRACTION_UNTIL_2_17_0=1
. -
The
--ram
option tocodeql database run-queries
and other commands that execute queries is now interpreted more strictly. Previously it was mostly a rough hint for how much memory to use, and the actual memory footprint of the CodeQL process could be hundreds of megabytes higher. From this release, CodeQL tries harder to keep its total memory consumption during evaluation below the given limit.The new behavior yields more predictable memory use, but since it works by allocating less RAM, it can lead to more use of disk storage for intermediate results compared to earlier releases with the same
--ram
value, and consequently a slight performance loss. In rare cases, for large databases, analysis may fail with a JavaOutOfMemoryError
.The cure for this is to increase
--ram
to be closer to the amount of memory actually available for CodeQL. As a rule of thumb, it will usually be possible to increase the value of--ram
by 700 MB or more, without actually using more resources than release 2.15.x would with the old setting. An exact amount cannot stated, however, since the actual memory footprint in earlier releases depended on factors such as the size of the databases that were not fully taken into account.If you use the CodeQL Action, you do not need to do anything unless you have manually overridden the Action's RAM setting. The Action will automatically select a
--ram
setting that matches the version of the CLI it uses.
- A new extractor option has been added to the JavaScript/TypeScript extractor.
Set the environment variable
CODEQL_EXTRACTOR_JAVASCRIPT_OPTION_SKIP_TYPES
totrue
to skip the extraction of types in TypeScript files. Use this to speed up extraction if your codebase has a high volume of TypeScript type information that causes a noticeable bottleneck for TypeScript extraction. The majority of analysis results should be preserved even when no types are extracted.
- Fixed an issue where CodeQL would sometimes incorrectly report that no files
were scanned when running on Windows.
This affected the human-readable summary produced by
codeql database analyze
andcodeql database interpret-results
, but did not impact the file coverage information produced in the SARIF output and displayed on the tool status page. - When analyzing Swift codebases, CodeQL build tracing will now ignore the
codesign
tool. This prevents errors in build commands or workflows on macOS that include both CodeQL and code signing.
- Java 21 is now fully supported, including support for new language features such as pattern switches and record patterns.
- Parallelism in the evaluator has been improved, resulting in faster analysis when running with many threads, particularly for large databases.
codeql database analyze
now defaults to include markdown query help for all custom queries with help files available. To change the default behaviour you can pass the new flag--sarif-include-query-help
, which provides the optionsalways
(which includes query help for all queries),custom_queries_only
(the default) andnever
(which does not include query help for any query). The existing flag--sarif-add-query-help
has been deprecated and will be removed in a future release.- The new (advanced) command-line option
--[no-]linkage-aware-import
disables the linkage-awareness phase ofcodeql dataset import
, as a quick fix (at the expense of database completeness) for C++ projects where this part of database creation consumes too much memory. This option is available in the commandsdatabase create
,database finalize
,database import
,dataset import
,test extract
, andtest run
. - The CodeQL language server now provides basic support for Rename, and you can now use the Rename Symbol functionality in Visual Studio Code for CodeQL. The current Rename support is less a refactoring tool and more a labor-saving device. You may have to perform some manual edits after using Rename, but it should still be faster and less work than renaming a symbol manually.
- The Find References feature in the CodeQL language server now supports all CodeQL identifiers and offers improved performance compared to CodeQL CLI 2.14 releases.
- The compiler generates shorter human-readable DIL and RA relation names. Due to use of an extended character set, full VS Code support for short relation names requires VS Code extension 1.9.4 or newer.
codeql database create
andcodeql database finalize
now log more diagnostic information during database finalization, including the size of each relation, their total size, and the rate at which they were written to disk.
- Fixed an internal error in the compiler when arguments to the
codePointCount
string primitive were not bound. - Fixed a bug where
codeql database finalize
would fail if a database under construction was moved between machines betweencodeql database init
andcodeql database finalize
. This should now work, as long as both commands are run by the same release of the CodeQL CLI and the extractors used are the ones bundled with the CLI. - Fixed a bug where
codeql database run-queries
would fail in some circumstances when the database path included an@
.
- C++ extraction has been updated to output more accurate C++ value categories. This may cause unexpected alerts on databases extracted with an up-to-date CodeQL when the queries are part of a query pack that was compiled with an earlier CodeQL. To resolve this, please recompile the query pack with the latest CodeQL.
codeql database analyze
andcodeql database interpret-results
can now output human-readable analysis summaries in a new format. This format provides file coverage information and improves the way that diagnostic messages are displayed. The new format also includes a link to the tool status page when theGITHUB_SERVER_URL
andGITHUB_REPOSITORY
environment variables are set. Note that that page only exists on GitHub.com, or in GitHub Enterprise Server version 3.9.0 or later. To enable this new format, pass the--analysis-summary-v2
flag.- CodeQL now supports
distinguishing file coverage information between related languages C and C++, Java and Kotlin,
and JavaScript and TypeScript. By default, file coverage information for each
of these pairs of languages is grouped together. To enable specific file coverage information for these languages, pass the
--sublanguage-file-coverage
flag when initializing the database (withcodeql database create
orcodeql database init
) and when analyzing the database (withcodeql database analyze
orcodeql database interpret-results
). If you are uploading results to a GitHub instance, this flag requires GitHub.com or GitHub Enterprise Server version 3.12 or later. - All CLI commands now support
--common-caches
, which controls the location of the cached data that is persisted between several runs of the CLI, such as downloaded QL packs and compiled query plans.
- Model packs that are used in an analysis will now be included in an output SARIF results file. All model packs now include the
isCodeQLModelPack: true
property in their tool component property bag. - The default formatting of DIL now more closely resembles equivalent QL code.
- Fixed a bug where
codeql github upload-results
would report a 403 error when attempting to upload to a GitHub Enterprise Server instance. - Fixed a bug in Python extraction where UTF-8 characters would cause logging to fail on systems with non-UTF-8 default system encoding (for example, Windows systems).
- The
resolve qlpacks --kind extension
command no longer resolves extensions packs from the search path. This matches the behavior ofresolve extensions-by-pack
and will ensure that extensions which are resolved byresolve qlpacks --kind extension
can also be resolved byresolve extensions-by-pack
.
- The query server's
evaluation/trimCache
command was previously equivalent to thecodeql database cleanup --mode=gentle
CLI command, but is now equivalent to using--mode=normal
. The new meaning of the command is to clear the entire evaluation cache of a database except for predicates annotated with thecached
keyword.
- The accepted values of the
--mode
option forcodeql database cleanup
have been renamed to bring them in line with what they are called in the VSCode extension and the query server:--mode=brutal
is now--mode=clear
.--mode=normal
is now--mode=trim
.--mode=light
is now--mode=fit
.- The old names are deprecated, but will be accepted for backwards-compatibility reasons until further notice.
- The list of failed tests at the end of a
codeql test run
is now sorted lexicographically. - The syntax of DIL now more closely resembles the QL source code that it is
compiled from. In particular, conjunctions and disjunctions now use the
familiar
and
andor
keywords, and clauses are enclosed in curly braces.
- Fixed a bug where the
$CODEQL_JAVA_HOME
environment variable was erroneously ignored for certain subsidiary Java processes started bycodeql
. - Fixed a bug in the CodeQL build tracer on Apple Silicon machines that prevented database creation if System Integrity Protection was disabled.
-
pragma[assume_small_delta]
is now deprecated. The pragma has no effect and should be removed. -
Missing override annotations on class fields now raise errors rather than warnings. This is to avoid confusion with the shadowing behavior in the presence of final fields.
-
The CodeQL CLI no longer supports ML-powered alerts. For more information, including details of our work in the AI-powered security technology space, see "CodeQL code scanning deprecates ML-powered alerts."
- The output of
codeql version --format json
now includes afeatures
property. Each key in the map identifies a feature of the CodeQL CLI. The value for a key is alwaystrue
. Going forward, whenever a significant new feature is added to the CodeQL CLI, a corresponding entry will be added to thefeatures
map. This is intended to make it easier for tools that invoke the CodeQL CLI to know if the particular version of the CLI they are invoking supports a given feature, without having to know exactly what CLI version introduced that feature.
-
You can now specify the CodeQL languages C/C++, Java/Kotlin, and JavaScript/TypeScript using
--language c-cpp
,--language java-kotlin
, and--language javascript-typescript
respectively. These new CodeQL language names convey more clearly what languages each CodeQL language will analyze.You can also reference these CodeQL languages via their secondary language names (C/C++ via
--language c
or--language cpp
, Java/Kotlin via--language java
or--language kotlin
, and JavaScript/TypeScript via--language javascript
or--language typescript
), however we recommend you refer to them via the new primary CodeQL language names for improved clarity. -
CodeQL now respects custom home directories set by the
$HOME
environment variable on MacOS and Linux and%USERPROFILE%
on Windows. When set, CodeQL will use the variable's value to change the default location of downloaded packages and the global compilation cache. -
This release improves the quality of file coverage information for repositories that vendor their dependencies. This is currently supported for Go and JavaScript projects.
- Fixed an issue with analyzing Python projects using Python 3.12.
- The QL language now has two new methods
codePointAt
andcodePointCount
on thestring
type. The methods both return integers and act the same as the similarly named Java methods on strings. For example,"abc".codePointAt(2)
is99
and("a" + 128512.toUnicode() + "c").codePointAt(1)
is a128512
.
- The tracking of RAM usage has been improved. This fixes some cases where CodeQL uses more RAM than requested.
- Fixed a JavaScript extractor crash that was introduced in 2.14.4.
- The CodeQL CLI no longer supports the
SEMMLE_JAVA_ARGS
environment variable. All previous versions of the CodeQL CLI perform command substitution on theSEMMLE_JAVA_ARGS
value (for example, replacing'$(echo foo)'
with'foo'
) when starting a new Java virtual machine, which, depending on the execution environment, may have security implications. Users are advised to check their environments for possibleSEMMLE_JAVA_ARGS
misuse.
- The Java extractor now supports files that use Lombok.
codeql database init
(andgithub/codeql-action/init@v2
on GitHub Actions) should no longer hang or crash for traced languages on 64-bit Windows machines when certain antivirus software is installed.- During
codeql pack create
andcodeql pack publish
, a source version of a pack coming from--additional-packs
can explicitly be used to override a requested pack version even if this source version is incompatible with the requested version in the pack file. Previously, this would fail with a confusing error message. - Fixed a bug where
codeql database interpret-results
hangs when a path query produces a result that has no paths from source to sink.
- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8.
-
The
<run>.tool.extensions
property in the SARIF generated bycodeql database analyze
now contains the following packs:- The containing query pack for each query that was evaluated.
- Each model pack that was specified via the
--model-packs
option, regardless of whether that model pack affected any of the evaluated queries.
Library packs are no longer included in the list.
Previously, this property contained every query and library pack that was available on the search path, regardless of whether that pack was used during the evaluation.
-
The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.8.
-
When
codeql test
generates.actual
files, they will in some cases list the query predicates in a different order than past versions. There is no need to update.expected
files, ascodeql test
sorts their results accordingly before diffing. However, when there are genuine changes in expected results, the generated.actual
file can show additional changes against the.expected
due to the reordering.
-
The functionality provided by the
codeql execute query-server
subcommand has been removed. The subcommand now responds to all JSON RPC requests with an error response. Correspondingly, this release is no longer compatible with versions of the CodeQL extension for Visual Studio Code prior to 1.7.6.This change also breaks third-party CodeQL IDE integrations that still rely on the
codeql execute query-server
subcommand. Maintainers of such CodeQL IDE integrations should migrate to thecodeql execute query-server2
subcommand at the earliest opportunity.
-
Switched from prefix filtering of autocomplete suggestions in the language server to client-side filtering. This improves autocomplete suggestions in contexts with an autocompletion prefix.
-
The CodeQL language server now checks query metadata for errors. This allows Visual Studio Code users to see errors in their query metadata without needing to compile the query.
- Fixed bug that made the
--warnings=hide
option do nothing incodeql database analyze
and other commands that evaluate queries.
- There are no user-facing changes in this release.
- The legacy option
--search-path
will now be used, if provided, when searching for the dependencies of packages that have no lock file. - CodeQL query packs that specify their dependencies using the legacy
libraryPathDependencies
property inqlpack.yml
/codeql-pack.yml
files are no longer permitted to contain acodeql-pack.lock.yml
lock file. This will lead to a compilation error. This change is intended to prevent confusing behavior arising from a mix of legacy (unversioned) and modern (versioned) package dependencies. To fix this error, either delete the lock file, or convertlibraryPathDependencies
todependencies
. - CodeQL CLI commands that create packages or update package lock files, such
as
codeql pack publish
andcodeql pack create
, will no longer work on query packs that specify their dependencies using the legacylibraryPathDependencies
property. To fix this error, convertlibraryPathDependencies
todependencies
.
- Missing override annotations on class member predicates now raise
errors rather than warnings. This is to avoid confusion with the
shadowing behaviour in the presence of final member predicates.
class Foo extends Base { final predicate foo() { ... } predicate bar() { ... } } class Bar extends Foo { // This method shadows Foo::foo. predicate foo() { ... } // This used to override Foo::bar with a warning, now raises error. predicate bar() { ... } }
- Unqualified imports can now be marked as deprecated to indicate that the import may be removed in the future. Usage of names only reachable through deprecated imports will generate deprecation warnings.
- Classes declared inside a parameterized modules can final extend parameters of the module as well as types that are declared outside the parameterized module.
- Fields are fully functional when extending types from within a module instantiation.
- Files with a
.yaml
extension will now be included in compiled CodeQL packs. Previously, files with this extension were excluded even though.yml
files were included. - When interpreting results (e.g., using
bqrs interpret
ordatabase interpret-results
), extra placeholders in alert messages are treated as normal text. Previously, results with more placeholders than placeholder values were skipped. - Windows users of the CodeQL extension for VS Code will see faster start times.
- In VS Code, errors in the current file are rechecked when dependencies change.
- In VS Code, autocomplete in large QL files is now faster.
- Member predicates can shadow final member predicates of the same arity even when the signatures are not fully matching.
- Fixed super calls on final base classes (or final aliases) so that they are now dispatched the same way as super calls on instanceof supertypes.
- Fixed a bug where running
codeql database finalize
with a large number of threads would fail due to running out of file descriptors. - Fixed a bug where
codeql database create --overwrite
would not work with database clusters. - Fixed a bug where the CodeQL documentation coverage statistics were incorrect.
- Fixed a bug where the generated CodeQL libarary documentation could generate invalid uris on windows.
- The Swift extractor now supports Swift 5.8.1.
- Temporary files and folders created by the CodeQL CLI will now be cleaned up when each CLI command (and its internal JVM) shuts down normally.
- Fixed an issue where indirect build tracing did not work in Azure DevOps
pipeline jobs in Windows containers. To use indirect build tracing in such
environments, ensure both the
--begin-tracing
and--trace-process-name=CExecSvc.exe
arguments are passed tocodeql database init
. - Improved the error message for the
codeql pack create
command when the pack being published has a dependency with no scope in its name.
-
This release enhances our preliminary Swift support, setting the stage for the upcoming public beta.
-
The
codeql database bundle
command now supports the--[no]-include-temp
option. When enabled, this option will include thetemp
folder of the database directory in the zip file of the bundled database. This folder includes generated packages and queries, and query suites. -
The structured log produced by
codeql generate log-summary
now includes a BooleanisCached
field for predicate events, where atrue
value indicates the predicate is a wrapper implementing thecached
annotation on another predicate. The wrapper depends on the underlying predicate that the annotation was found on, and will usually have the same name, but it has a separateraHash
.
-
Fixed a bug that could cause the compiler to infer incorrect binding sets for non-direct calls to overriding member predicates that have stronger binding sets than their root definitions.
-
Fixed a bug that could have caused the compiler to incorrectly infer that a class matched a type signature. The bug only affected classes with overriding member predicates that had stronger binding sets than their root definitions.
-
Fixed a bug where a query could not be run from VS Code when there were packs nested within sibling directories of the query.
This release was skipped.
- Fixed a bug in
codeql database upload-results
where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting anInvalidDefinitionException
. This issue occurred when the SARIF file contained certain kinds of diagnostic information.
- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.7.
- We recommend that customers using the CodeQL CLI in a third party CI
system do not upgrade to this release, due to an issue with
codeql github upload-results
. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1. For more information, see the "Known issues" section for CodeQL 2.12.6.
-
In
codeql pack add
, the dependency that is added to theqlpack.yml
file will now allow any version of the pack that is compatible with the specified version (^version
) in the following cases:- When no version is specified (
codeql pack add codeql/cpp-all
). - When the version is specified as
latest
(codeql pack add codeql/cpp-all@latest
). - When a single version is specified (
codeql pack add codeql/cpp-all@1.0.0
).
The
^version
dependency allows any version of that pack with no breaking changes sinceversion
. For example,^1.2.3
would allow versions1.2.3
,1.2.5
, and1.4.0
, but not2.0.0
, because changing the major version number to2
indicates a breaking change.Using
^version
ensures that the added pack is not needlessly constrained to an exact version by default. - When no version is specified (
-
Upper-case variable names are no longer accepted by the QL compiler.
Such variable names have produced a deprecation warning since release 2.9.2 (released 2022-05-16), so QL code that compiles without warnings with a recent release of the CLI should still work.
codeql database analyze
and related commands now export file coverage information by default. GHAS customers using CodeQL in third-party CI systems will now see file coverage information on the tool status page without needing to modify their CI workflows.
-
The possibility to omit
override
annotations on class member predicates that override a base class predicate has been deprecated. This is to avoid confusion with shadowing behaviour in the presence of final member predicates.class Foo extends Base { final predicate foo() { ... } predicate bar() { ... } predicate baz() { ... } } class Bar extends Foo { // This method shadows Foo::foo. predicate foo() { ... } // This used to override Foo::bar with a warning, is now deprecated. predicate bar() { ... } // This correctly overrides Foo::baz override predicate baz() { ... } }
- Fixed a bug in
codeql database upload-results
where the subcommand would fail with "A fatal error occurred: Invalid SARIF.", reporting anInvalidDefinitionException
. This issue occurred when the SARIF file contained certain kinds of diagnostic information.
-
We recommend that customers using the CodeQL CLI in a third party CI system do not upgrade to this release, due to an issue with
codeql github upload-results
. Instead, please use CodeQL 2.12.5, or, when available, CodeQL 2.12.7 or 2.13.1.This issue occurs when uploading certain kinds of diagnostic information and causes the subcommand to fail with "A fatal error occurred: Invalid SARIF.", reporting an
InvalidDefinitionException
.Customers who wish to use CodeQL 2.12.6 or 2.13.0 can work around the problem by passing
--no-sarif-include-diagnostics
to any invocations ofcodeql database analyze
orcodeql database interpret-results
.
- Several experimental subcommands have been added in support of the
new code scanning tool status page.
These include
codeql database add-diagnostic
,codeql database export-diagnostics
, and thecodeql diagnostic add
andcodeql diagnostic export
plumbing subcommands.
- Fixed a bug in
codeql database analyze
and related commands where the--max-paths
option was not respected correctly when multiple alerts with the same primary code location were grouped together. (This grouping is the default behavior unless the--no-group-alerts
option is passed.) This bug caused some SARIF files produced by CodeQL to exceed the limits on the number of paths (threadFlows
) accepted by code scanning, leading to errors when uploading results.
-
The
codeql pack install
command now accepts a--additional-packs
option. This option takes a list of directories to search for locally available packs when resolving which packs to install. Any pack that is found locally through--additional-packs
will override any other version of a pack found in the package registry. Locally resolved packs are not added to the lock file.Because the use of
--additional-packs
when runningcodeql pack install
makes running queries dependent on the local state of the machine initially invokingcodeql pack install
, a warning is emitted if any pack is found outside of the package registry. This warning can be suppressed by using the--no-strict-mode
option.
- Fix a bug in
codeql query run
where queries whose path contain colons cannot be run.
- The default value of the
--mode
switch tocodeql pack install
has changed. The default is now--mode minimal-update
. Previously, it wasuse-lock
.
- The per-pack compilation cache has been replaced with a global compilation cache
found within
~/.codeql
. codeql pack install
now uses a new algorithm to determine which versions of the pack's dependencies to use, based on the PubGrub algorithm. The new algorithm is able to find a solution for many cases that the previous algorithm would fail to solve. When the new algorithm is unable to find a valid solution, it generates a detailed error message explaining why there is no valid solution.- Added a new command,
codeql pack upgrade
. This command is similar tocodeql pack install
, except that it ignores any existing lock file, installs the latest compatible version of each dependency, and writes a new lock file. This is equivalent tocodeql pack install --mode update
. Note that the--mode
switch tocodeql pack install
is now deprecated. - Added a new command,
codeql pack ci
. This command is similar tocodeql pack install
, except if the existing lock file is missing, or if it conflicts with the version constraints in theqlpack.yml
file, the command generates an error. This is equivalent tocodeql pack install --mode verify
. Note that the--mode
switch tocodeql pack install
is now deprecated.
- The
--freeze
switch forcodeql pack create
,codeql pack bundle
, andcodeql pack publish
is now deprecated and ignored, as there is no longer a cache within a pack. - The
--mode update
switch tocodeql pack resolve-dependencies
is now deprecated. Instead, use the new--mode upgrade
switch, which has identical behavior. - The
--mode
switch tocodeql pack install
is now deprecated.- Instead of
--mode update
, usecodeql pack upgrade
. - Instead of
--mode verify
, usecodeql pack ci
.
- Instead of
- The CodeQL compiler now produces better error messages when it is unable to find a QL library that the query being evaluated depends on.
- Fixed a bug where the CLI would refuse to complete database creation if the OS reports less than about 1.5 GB of physical memory. Now an attempt will be made even on low-memory systems (but it might still run out of memory unless there's swap space available).
-
Fixed a QL evaluator bug introduced in release 2.12.1 which could in certain rare cases lead to wrong analysis results.
-
Fixed handling of
-Xclang <arg>
arguments passed to theclang
compiler which could cause missing extractions for C++ code bases. -
Fixed a bug where the
--overwrite
option was failing for database clusters.
- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.6.
- Added a new command-line flag
--expect-discarded-cache
, which gives a hint to the evaluator that the evaluation cache will be discarded after analysis completes. This allows it to avoid some unnecessary writes to the cache, for predicates that aren't needed by the query/suite being evaluated.
- The
--[no-]count-lines
option tocodeql database create
and related commands that was deprecated in 2.11.1 has been removed. Users of this option should instead pass--[no-]calculate-baseline
.
-
Query packs created by
codeql pack create
,codeql pack bundle
, andcodeql pack release
now contain precompiled queries in a new format that aims to be compatible with future (and, to a certain extent, past) releases of the CodeQL CLI. Previously the precompiled queries were in a format specific to each CLI release, and all other releases would need to re-compile queries.Published packs contain precompiled queries in files with a
.qlx
extension located next to each query's.ql
source file. In case of differences between the.ql
and.qlx
files, the.qlx
file takes priority when evaluating queries from the command line, so if you need to modify a published pack, be sure to delete the.qlx
files first.A new
--precompile
flag tocodeql query compile
can be used to construct*.qlx
file explicitly, but in all usual cases it should be enough to rely oncodeql pack create
doing the right thing. -
The
codeql database init
command now accepts a PAT that allows you to download queries from external, private repositories when using the--codescanning-config <config-file>
option. For example, you can specify the following queries block in the config file, which will checkout the main branch of thecodeql-test/my-private-repository
repository and evaluate any queries found in that repository:queries: - codeql-test/my-private-repository@main
If the repository is private, you can add a
--external-repository-token-stdin
option and supply a PAT with appropriate permissions via standard input. For more information on queries and external repositories in Code Scanning, see Using queries in QL packs. -
The baseline information produced by
codeql database init
andcodeql database create
now accounts forpaths
andpaths-ignore
configuration. -
In the VS Code extension, recursive calls will be marked with inlay hints. These can be disabled with the global inlay hints setting (
editor.inlayHints.enabled
). If you just want to disable them for codeql the settings can be scoped to just codeql files (language id isql
). See Language Specific Editor Settings in the VS Code documentation for more information. -
The CLI now gives a more helpful error message when asked to run queries on a database that has not been finalized.
- Fixed a bug where the
codeql pack install
command would fail if a CodeQL configuration file is used and the--additional-packs
option is specified.
-
Java and Kotlin analyses in this release of the CLI and all earlier releases are incompatible with Kotlin 1.7.30 and later. To prevent code scanning alerts being spuriously dismissed, Java and Kotlin analyses will now fail when using Kotlin 1.7.30 or later.
If you are unable to use Kotlin 1.7.29 or earlier, you can disable Kotlin support by setting
CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN
totrue
in the environment.
- Fixed a bug where it was not possible to run queries in CodeQL query
packs for C# that use the legacy
libraryPathDependencies
property in theirqlpack.yml
file. The associated error message complained about undefined extensional predicates.
- Fixed a bug that could cause log summary generation to fail in vscode.
- Kotlin support is now in beta. This means that Java analyses will also
include Kotlin code by default. Kotlin support can be disabled by
setting
CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN
totrue
in the environment.
- CodeQL 2.11.1 to 2.11.3 contained a bug in indirect build tracing on Windows
when using
codeql database init
with the--trace-process-level
flag. In these versions, when--trace-process-level
was set to a value greater than zero, (or left at the default value of 1), CodeQL attempted to inject its build tracer at a higher level in the process tree than the requested process level. This could lead to errors of the form "No source code found" or "Process tree ended before reaching required level". From 2.11.4 onwards, the CodeQL build tracer is injected at the requested process level.
- The
--[no-]fast-compilation
option tocodeql test run
is now deprecated.
- The
codeql pack ls --format json
deep plumbing command now returns only thename
andversion
properties for each found pack.
codeql pack download
,codeql pack install
, andcodeql pack add
will ignore CodeQL packs with pre-release versions, unless the--allow-prerelease
option is passed to the command. This brings these commands into alignment withcodeql pack publish
that will avoid publishing CodeQL packs with pre-release versions unless the--allow-prerelease
option is specified. Pre-release versions have the following format:X.Y.Z-qualifier
whereX
,Y
, andZ
are respectively the major, minor, and patch number.qualifier
is the pre-release version. For more information about pre-releases, see the Semantic Versioning specification.
- The
--[no-]fast-compilation
option tocodeql query compile
is now deprecated.
codeql resolve files
andcodeql database index-files
have a new--find-any
option, which finds at most one match.
- The build of Apache Commons Text that is bundled with the CodeQL CLI
has been updated to version 1.10.0. While previous releases shipped
with version 1.6 of the library, no part of the CodeQL CLI
references the
StringSubstitutor
class that the recently disclosed CVE-2022-42889 vulnerability applies to. We therefore do not believe that running previous releases of CodeQL exposes users to this vulnerability. - The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.5.
-
Bundling and publishing a CodeQL pack will no longer include nested CodeQL packs. If you want to include a nested pack in your published pack, then you must explicitly include it using the
include
property in the top-levelqlpack.yml
file.For example, if your package structure looks like this:
qlpack.yml nested-pack ∟ qlpack.yml query.ql
then the contents of
nested-pack
will not be included by default within the published package. To includenested-pack
, add an entry like this to the top levelqlpack.yml
file:include: - nested-pack/**
-
Using the
--codescanning-config=<file>
option incodeql database init
will now correctly process thepaths
andpathsIgnore
properties of the configuration file in a way that is identical to the behavior of thecodeql-action
. Previously,paths
orpathsIgnore
entries that end in/**
or start with/
were incorrectly rejected by the CLI. -
Fixed a bug where the
--compilation-cache
option tocodeql pack publish
andcodeql pack create
was being ignored when creating a query pack. Now, the indicated cache is used when pre-compiling the queries in it. -
Fixed a bug that would make the "Show DIL" command in the VSCode extension display nothing.
- Emit a detailed warning if package resolution fails, the legacy
--search-path
option is provided, and there is at least one referenced pack that does not use legacy package resolution. In this case,--additional-packs
should be used to extend the search to additional directories, instead of--search-path
.
- Pack installation using the CodeQL Packaging beta will now fail if a
compatible version cannot be found. This replaces the previous
behavior where
codeql pack download
and related commands would instead install the latest version of the pack in this situation.
- The
--[no-]count-lines
option tocodeql database create
and related commands is now deprecated and will be removed in a future release of the CodeQL CLI (earliest 2.12.0). It is replaced by--[no-]calculate-baseline
to reflect the additional baseline information that is now captured as of this release.
- Subcommands that compile QL accept a new
--no-release-compatibility
option. It does nothing for now, but in the future it will be used to control a trade-off between query performance and compatibility with older/newer releases of the QL evaluator. codeql database analyze
and related commands now support absolute paths containing the@
or:
characters when specifying which queries to run. To reference a query file, directory, or suite whose path contains a literal@
or:
, prefix the query specifier withpath:
, for example:codeql database analyze --format=sarif-latest --output=results <db> path:C:/Users/ci/workspace@2/security/query.ql
- It is no longer an error to call
codeql pack create <path>
with a<path>
option pointing to a file name. The CLI will walk up the directory tree and run the command in the first directory containing theqlpack.yml
orcodeql-pack.yml
file. - Fixed a concurrency error observed when using
codeql database import
orcodeql database finalize
with multiple threads and multiple additional databases on a C++ codebase.
- The CodeQL CLI now uses Python 3 to extract both Python 2 and Python 3 databases. Correspondingly, support for using Python 2 to extract Python databases is now deprecated. Starting with version 2.11.3, you will need to install Python 3 to extract Python databases.
- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4.
- You can now define which registries should be used for downloading and publishing CodeQL packs on a
per-workspace basis by creating a
codeql-workspace.yml
file and adding aregistries
block. For more infomation, see About CodeQL Workspaces.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
-
This release does not include any user-facing changes.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- When called with
--start-tracing
, thecodeql database init
command now accepts extractor options for the indirect tracing environment via--extractor-option
. Users should continue to specify extractor options for direct tracing environments by passing them tocodeql database trace-command
invocations.
- The build of Eclipse Temurin OpenJDK that is bundled with the CodeQL CLI has been updated to version 17.0.4.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- The option
--compiler-spec
tocodeql database create
(andcodeql database trace-command
) no longer works. It is replaced by--extra-tracing-config
, which accepts a tracer configuration file in the new, Lua-based tracer configuration format instead. Seetools/tracer/base.lua
for the precise API available. If you need help help porting your existing compiler specification files, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering.
- Versions of the CodeQL extension for Visual Studio Code released before February 2021 may not work correctly with this CLI, in particular if database upgrades are necessary. We recommend keeping your VS Code extension up-to-date.
- The experimental
codeql resolve ml-models
command has been deprecated. Advanced users calling this command should use the newcodeql resolve extensions
command instead.
- The
codeql github upload-results
command now supports a--merge
option. If this option is provided, the command will accept the paths to multiple SARIF files, and will merge those files before uploading them as a single analysis. This option is recommended only for backwards compatibility with old analyses produced by the CodeQL Runner, which combined the results for multiple languages into a single analysis.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- Improved error message from
codeql database analyze
when a query is missing@id
or@kind
query metadata.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- The
--format=stats
option ofcodeql generate log-summary
has been renamed to--format=overall
. It now produces a richer JSON object that, in addition to the previous statistics about the run (which can be found in thestats
property) also records the most expensive predicates in the evaluation run.
-
The
codeql resolve ml-model
command now requires one or more query specifications as command line arguments in order to determine the set of starting packs from which to initiate the resolution process. The command will locate all ML models in any qlpack that is a transitive dependency of any of the starting packs. Also, the output of the command has been expanded to include for each model the containing package's name, version, and path. -
The
buildMetadata
inside of compiled CodeQL packs no longer contains acreationTime
property. This was removed in order to ensure that the content of a CodeQL pack is identical when it is re-compiled. -
The
codeql pack download
command, when used with the--dir
option, now downloads requested packs in directories corresponding to their version numbers. Previously,codeql pack download --dir ./somewhere codeql/java-queries@0.1.2
would download the pack into the./somewhere/codeql/java-queries
directory. Now, it will download the pack into the./somewhere/codeql/java-queries/0.1.2
directory. This allows you to download multiple versions of the same pack using a single command.
- You can now include diagnostic messages in the summary produced by
the
--print-diagnostics-summary
option of thecodeql database interpret-results
andcodeql database analyze
commands by running these commands at high verbosity levels.
-
Fixed a bug where
codeql pack download
, when used with the--dir
option, would not download a pack that is in the global package cache. -
Fixed a bug where some versions of a CodeQL package could not be downloaded if there are more than 100 versions of this package in the package registry.
-
Fixed a bug where the
--also-match
option forcodeql resolve files
andcodeql database index-files
does not work with relative paths. -
Fixed a bug that caused
codeql query decompile
to ignore the--output
option when producing bytecode output (--kind=bytecode
), writing only tostdout
.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- Users of CodeQL Packaging Beta can now optionally authenticate to
Container registries on GitHub Enterprise Server (GHES) versions 3.6
and later using standard input instead of the
CODEQL_REGISTRIES_AUTH
environment variable. To authenticate via standard input, pass--registries-auth-stdin
. The value you provide will override the value of theCODEQL_REGISTRIES_AUTH
environment variable.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
-
Users can now use CodeQL Packaging Beta to publish and download CodeQL packs on GitHub Enterprise Server (GHES) versions 3.6 and later.
To authenticate to a package registry on GHES 3.6+, first create a
~/.codeql/qlconfig.yml
file. For example, the following file specifies that all CodeQL packages should be uploaded to the GHES instance with the hostnameGHE_HOSTNAME
:registries: - packages: '*' url: https://containers.GHE_HOSTNAME/v2/
You can now download public packages from GHES using
codeql pack download
.To publish any package or download private packages, authenticate to GHES by specifying registry/token pairs in the
CODEQL_REGISTRIES_AUTH
environment variable. You can authenticate using either a GitHub Apps token or a personal access token. For example,https://containers.GHEHOSTNAME1/v2/=TOKEN1,https://containers.GHEHOSTNAME2/v2/=TOKEN2
will authenticate the CLI to theGHEHOSTNAME1
andGHEHOSTNAME2
GHES instances.
-
Fixed a bug where precompiled CodeQL packages in the CodeQL bundle were being recompiled if they were in a read-only directory.
-
Fixed a bug where new versions of the VS Code extension wouldn't run two queries in parallel against one database.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- The table printed by
codeql database analyze
to summarize the results of metric queries that were part of the analysis now reports a single row per metric name independently of the verbosity level of the command. Previously, at higher verbosity levels, this table would contain multiple rows for metric names with multiple values.
-
The tables produced by
codeql database analyze
summarizing the results of any diagnostic and metric queries that were run now exclude the results of queries taggedtelemetry
. -
Uploading SARIF results using the
codeql github upload-results
command now has a timeout of 5 minutes. -
Downloading CodeQL packs using the
codeql pack download
,codeql pack install
and related commands now have a timeout of 5 minutes and will retry 3 times before failing. Similar behavior has been added to thecodeql pack publish
command. -
The
codeql generate log-summary
command will now print progress updates tostderr
.
- Fixed a bug that could make it unpredictable whether the QL compiler
reports problems about query metadata tags, and thereby make
codeql test run
fail spuriously in some cases.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
-
codeql database create
now supports the--[no-]-count-lines
option, which was previously only available withcodeql database init
. -
codeql resolve files
andcodeql database index-files
has a new--also-match
option, which allows users to specify glob patterns that are applied in conjunction with the existing--include
option.
- This release introduces experimental support for parameterized QL modules. This language feature is still subject to change and should not be used in production yet.
-
Fixed a bug that would prevent resolution of a query suite in a published CodeQL query pack that has a reference to the pack itself.
-
Fixed inaccurate documentation of what the
--include-extension
option tocodeql resolve files
andcodeql database index-files
does. The actual behavior is unchanged.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.30) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.30 instance, you need to create them with release 2.7.6.
-
There are no user-facing changes in this release.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.
-
Fixed an error where running out of memory during query evaluation would cause
codeql
to exit with status 34 instead of the 99 that is documented for this condition. -
Fixed a bug in our handling of Clang's header maps, which caused missing files for Xcode-based projects on macOS (e.g. WebKit).
-
This release of CodeQL (and all future ones) will not include the CodeQL runner, which is now deprecated. For more information, and instructions on how to migrate to using the CodeQL CLI, see CodeQL runner deprecation.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.
- Executable binaries for Windows are now digitally signed by a GitHub certificate.
- The evaluator logs produced by
--evaluator-log
now default to the maximum verbosity level and will therefore contain more information (and, accordingly, grow larger). The verbosity level can still be configured with--evaluator-log-level
. In particular,--evaluator-log-level=1
will restore the previous default behavior.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.
- The support for the output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) that were deprecated in 2.7.1 has been removed. If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering.
- The CodeQL CLI is now compatible with Windows 11 and Windows Server 2022, including building databases for compiled languages.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.
-
Commands that find or run queries now allow you to refer to queries within a named CodeQL pack. For example:
# Analyze a database using all queries in the experimental/Security folder within the codeql/cpp-queries # CodeQL query pack. codeql database analyze --format=sarif-latest --output=results <db> \ codeql/cpp-queries:experimental/Security # Analyse using only the RedundantNullCheckParam.ql query in the codeql/cpp-queries CodeQL query pack. codeql database analyze --format=sarif-latest --output=results <db> \ 'codeql/cpp-queries:experimental/Likely Bugs/RedundantNullCheckParam.ql' # Analyse using the cpp-security-and-quality.qls query suite in the codeql/cpp-queries CodeQL query pack. codeql database analyze --format=sarif-latest --output=results <db> \ 'codeql/cpp-queries:codeql-suites/cpp-security-and-quality.qls' # Analyse using the cpp-security-and-quality.qls query suite from a version of the codeql/cpp-queries pack # that is >= 0.0.3 and < 0.1.0 (the highest compatible version will be chosen). # All valid semver ranges are allowed. See https://docs.npmjs.com/cli/v6/using-npm/semver#ranges codeql database analyze --format=sarif-latest --output=results <db> \ 'codeql/cpp-queries@~0.0.3:codeql-suites/cpp-security-and-quality.qls'
The complete way to specify a set of queries is in the form
scope/name@range:path
, where:-
scope/name
is the qualified name of a CodeQL pack. -
range
is a semver range. -
path
is a file system pathIf a
scope/name
is specified, therange
andpath
are optional. A missingrange
implies the latest version of the specified pack. A missingpath
implies the default query suite of the specified pack.The
path
can be one of a*.ql
query file, a directory containing one or more queries, or a.qls
query suite file. If there is no pack name specified, then apath
must be provided, and will be interpreted relative to the current working directory of the current process.If a
scope/name
andpath
are specified, then thepath
cannot be absolute. It is considered relative to the root of the CodeQL pack.The relevant commands are:
codeql database analyze
codeql database run-queries
codeql execute queries
codeql resolve queries
-
- Fixed a bug that would sometimes lead to query evaluation on
M1-based Macs to crash with
Did not preallocate enough memory
error.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.29) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.29 instance, you need to create them with release 2.6.3.
- The CodeQL Action versions up to and including version 1.0.22 are not compatible with the CodeQL CLI 2.8.0 and later. The CLI will emit an error if it detects that it is being used by an incompatible version of the codeql-action.
-
A new extractor option has been added to the Java extractor. The flag
--extractor-option exclude='<glob>'
allows specifying a glob that describes which paths need to be excluded from extraction but still need to be compiled. This is useful when some files are necessary for a successful build but are uninteresting for analysis.See also: https://codeql.github.com/docs/codeql-cli/extractor-options/
-
Summary metrics can now associate messages with their results, for instance to report the name and number of uses of a particular API endpoint within a repository. To associate messages with summary metrics, define a query with
@kind metric
and@tags summary
metadata and use either thelocation, message, value
or themessage, value
results pattern.
- Fixed a bug where
codeql resolve upgrades
ignores the--target-dbscheme
option.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
- A bug where creation of a CodeQL database could sometimes fail with
a
NegativeArraySizeException
has now been fixed.
- The CLI and evaluator contain a number of new features in support of
internal machine learning experiments. This includes an experimental
resolve ml-models
subcommand and newmlModels
metadata in pack definition files. As these new features are not yet ready for general use, they should be ignored by external CodeQL users.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
- The CodeQL Action versions up to and including version 1.0.22 are now deprecated for use with CodeQL CLI 2.7.5 and later. The CLI will emit a warning if it detects that it is being used by a deprecated version of the codeql-action. This warning will become a fatal error with version 2.8.0 of the CLI.
-
The
codeql github upload-results
command will now print the API response body in JSON format if a--format=json
flag is given. Otherwise the command will print the URL of the SARIF upload. This URL can be used to get status information for the upload.See also: https://docs.github.com/en/rest/reference/code-scanning
-
The documentation for the
--trace-process-level
flag ofcodeql database init
(which is used with indirect build tracing on Windows) was erroneous.The help text previously claimed that
--trace-process-level=1
would inject CodeQL's build tracer into the calling process. This is actually what--trace-process-level=0
achieves. The help text has now been corrected to match the actual (unchanged) behavior.Also, some log messages incorrectly stated which process CodeQL was injected into. These have also been corrected.
- For commands that run queries, the
--timeout
option now controls the maximal time it may take to evaluate a "layer" of a query rather than a "stage". There are usually many "layers" in each "stage", but it is usually a single one of the layers in a stage that uses most of the time, so there is no need to reduce existing timeout values as a result of this change.
This release was skipped.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
- The experimental command-line option
--ml-model-path
that was introduced to support internal experiments has been removed.
-
Editing support (content assist, code navigation, etc.) in files under the
.github
directory will now work properly. This is because files under the.github
directory will now be indexed and processed by the CodeQL language server. Other hidden directories that start with.
will remain un-indexed. This affects the vscode-codeql extension and any other IDE extension that uses the CodeQL language server. -
Fixed authentication with GitHub package registries via the
GITHUB_TOKEN
environment variable and the--github-auth-stdin
flag when downloading and publishing packs. -
Fixed an incompatibility with glibc version 2.34 on Linux, where build tracing failed with an error message.
-
Fixed a bug where
codeql generate log-summary
could sometimes fail with aJsonMappingException
.
-
The CodeQL CLI for Mac OS now ships with a native Java virtual machine for M1 Macs, and this will be used by default where applicable to run the CodeQL engine, thus improving performance. Rosetta 2 is still required as not all components of the CodeQL CLI are natively compiled.
-
Commands that execute queries will now exit with status code 34 if certain errors that prevent the evaluation of one or more individual queries are detected. Previously some of these errors would crash the evaluator and exit with status code 100.
(This is currently used for "external predicate not found" errors).
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
-
The Java extractor now defaults to extracting all XML documents under 10MB in size, a change from the previous default of only extracting documents with particular well-known names (e.g.
pom.xml
). However, if the source tree contains more than 50MB of XML in total, it prints a warning and falls back to the old default behaviour. Set the environment variableLGTM_INDEX_XML_MODE
tobyname
to get the old default behaviour, orall
to extract all documents under 10MB regardless of total size. -
The experimental command-line option
--native-library-path
that was introduced to support internal experiments has been removed. -
The beta
codeql pack publish
command will now prevent accidental publishing of packages with pre-release version qualifiers. Prerelease versions are those that include a-
after the major, minor, and patch versions such as1.2.3-dev
. To avoid this change, use the--allow-prerelease
option.
-
Fixed an issue when using the
--evaluator-log
option where aNullPointerException
could sometimes occur non-deterministically. -
Fixed bugs observed when using indirect build tracing using a CodeQL distribution unpacked to a path containing spaces or on Arch Linux.
- CodeQL databases now contain metadata about how and when they were
created. This can be found in the
creationMetadata
field of thecodeql-database.yml
file within the CodeQL database directory. More information may be added to this field in future releases.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
-
Previously,
codeql test run
would fall back to looking for an accompanyingqueries.xml
file if it found aqlpack.yml
that did not declare an extractor to use when extracting a test database. This has been removed because the internal use case that neccessitated the fallback are now removed. If you suddenly encounter errors that complain of missing extractor declarations, check whether you had aqueries.xml
you were inadvertently relying on. -
When queries are specified by naming a directory to scan for
*.ql
files, subdirectories named.codeql
will now be ignored. The new QL packaging support uses subdirectories with this name of various scratch and caching purposes, so they may contain*.ql
files that are not intended to be directly user-visible. -
When copying dependencies for CodeQL packages into a query pack bundle,
*.ql
files in these dependencies will now be included inside of the query pack's.codeql
directory. -
The tables printed by
codeql database analyze
to summarize the results of diagnostic and metric queries that were part of the analysis have a new format and contains less (but hopefully more pertinent) information. We recommend against attempting to parse this human-readable output programmatically. Instead, use theruns[].tool.driver.invocations[].toolExecutionNotifications
property in the SARIF output. -
The experimental plumbing command
codeql pack packlist
has a new format for its JSON results. Previously, the results were a list of paths. Now, the results are an object with a single propertypaths
that contains the list of paths. -
The internal
qlpacks
directory of the CodeQL bundle available on the CodeQL Action releases page has a new structure. This directory is internal to the CLI and can change without notice in future releases.The currently-shipped
qlpacks
directory mirrors the structure of CodeQL package caches and looks like this:qlpacks - codeql - {lang}-all - {version} - qlpack contents - {lang}-examples - {version} - qlpack contents - {lang}-queries - {version} - qlpack contents - {lang}-upgrades - {version} - qlpack contents - ... and so on for all languages
-
The output formats SARIF v1.0.0 and SARIF v2.0.0 (Committee Specification Draft 1) have been deprecated. They will be removed in a later version (earliest 2.8.0). If you need this functionality, please file a public issue against https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering.
-
The
qlpack:
instruction in query suite definitions has been deprecated due to uncertainty about whether it is intended to include all the*.ql
files in the named pack, or only the pack's "default query suite". The behavior of the instruction is determined by whether the named pack declares any default query suite, but this means that a pack starting to declare such a suite may break the behavior of existing query suites that reference the pack from outside.We recommend replacing
qlpack:
by one of- queries: '.' # import all *.ql files from: some/pack-name version: 1.2.3 # optional
or
- import: path/to/actual/suite.ql # just that suite from: some/pack-name version: 1.2.3 # optional
A warning will now be printed when a
qlpack:
instruction resolves to a default suite, because that is the case where the effect may not be what the query suite author intended.
-
Fixed a bug where the
paths
andpaths-ignore
properties of a Code Scanning config file specified using--codescanning-config
were being interpreted the wrong way around. -
Fixed a bug where queries specified using the
--codescanning-config
option could not be run after an explicit call tocodeql database finalize
. -
Fixed a bug where
-J
options would erroneously be recognized even after--
on the command line. -
When running
codeql database analyze
andcodeql database interpret-results
without the--sarif-group-rules-by-pack
flag, the SARIF output did not include baseline lines-of-code counts. This is now fixed. -
Fixed a bug where expansion of query suites would sometimes fail if a query suite in a compiled query pack referenced that pack itself explicitly.
- Set literal expressions can now optionally contain a trailing comma after the last element.
-
Beta support for database creation on Apple Silicon has been added. It depends on the following requirements:
-
Rosetta 2 needs to be installed
-
Developer tools need to be installed. CodeQL requires the
lipo
,codesign
, andinstall_name_tool
tools to be present. -
Build systems invoking
csh
may experience intermittent crashes.
-
-
codeql database analyze
can now include query-specific help texts for alerts in the SARIF output (for SARIF v2.1.0 or later). The help text must be located in an.md
file next to (and with the same basename as) the.ql
file for each query. Since this can significantly increase SARIF file size, the feature is not enabled by default; give a--sarif-add-query-help
option to enable it. -
The query metadata validator now knows about queries that produce alert scores, so these queries no longer need to be run with a
--no-metadata-verification
flag. -
codeql database create
andcodeql-finalize
have a new flag--skip-empty
that will cause a language with no extracted source code to be ignored with a warning instead of treated like a fatal error. This can be useful with--db-cluster
where not all of the languages may exist in the source tree. It will not be possible to run queries against the skipped database. -
codeql resolve extractor
andcodeql resolve languages
now support an extended output format--format=betterjson
wich includes information about each extractor's language-specific options. -
This release introduces rudimentary support for parallelizing database creation by importing unfinished databases (or database clusters) into another unfinished database (or cluster) under creation. This is implemented by the new flag
--additional-dbs
forcodeql database finalize
, or the new plumbing commandcodeql database import
. -
codeql database create
,codeql database index-files
, andcodeql database trace-command
support a unified syntax for passing language-specific options to the extractor with the new--extractor-option
and--extractor-options-file
options. (The extractors do not make use of this yet, though).
-
The extractor for Ruby is now included. CodeQL analysis for Ruby is currently in beta. During the beta, analysis of Ruby will not be as comprehensive as CodeQL analysis of other languages. The source code of the extractor and the queries can be found in the
github/codeql
repository. -
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
- Fixed a bug where indirect tracing would sometimes not manage to observe build processes if certain environment variables were unset during the build.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
-
The option
--compiler-spec
accepted by some subcommands ofcodeql database
is deprecated. It will be removed in a later version (earliest 2.7.0). If you need this option, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub support and request an escalation to engineering. -
By default, databases created using the CodeQL CLI will now have their underlying datasets finalized, meaning that no further data can be subsequently imported into them. This change should not affect most users.
-
The
codeql resolve qlref
command will now throw an error when the target is ambiguous. The qlref resolution rules are now as follows:-
If the target of a qlref is in the same qlpack, then that target is always returned.
-
If multiple targets of the qlref are found in dependent packs, this is an error.
Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities.
-
-
Linux/MacOS: When tracing a build that involves an
execvp
/execvpe
(Linux-only)/posix_spawnp
syscall wherePATH
was not set in the environment, CodeQL sometimes would break the build. Now, CodeQL uses the correct, platform-specific fallback forPATH
instead. -
Linux/MacOS: When tracing a build that involves an
execvpe
(Linux-only)/posix_spawnp
syscall, thePATH
lookup of the executable wrongly took place in the environment provided viaenvp
, instead of the environment of the process callingexecvpe
/posix_spawnp
. Now, the correct environment is used for thePATH
lookup. -
A bug where query compilation would sometimes fail with a
StackOverflowError
when compiling a query that usesinstanceof
has now been fixed.
-
The
codeql query compile
command now accepts a--keep-going
or-k
option, which indicates that the compiler should continue compiling queries even if one of the queries has a compile error in it. -
CLI commands now run default queries if none are specified. If no queries are specified, the
codeql database analyze
,codeql database run-queries
, andcodeql database interpret-results
commands will now run the default suite for the language being analyzed. -
codeql pack publish
now copies the published package to the local package cache. In addition to publishing to a remote repository, thecodeql pack publish
command will also copy the published package to the local package cache.
-
CodeQL CLI 2.6.2 includes the same functionality as the CodeQL runner, which is being deprecated. For more information, see CodeQL runner deprecation.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
- A bug where
codeql generate log-summary
would sometimes crash with aJsonMappingException
has been fixed.
-
The CodeQL CLI now counts the lines of code found under
--source-root
whencodeql database init
orcodeql database create
is called. This information can be viewed later by either the newcodeql database print-baseline
command or the new--print-baseline-loc
argument tocodeql database interpret-results
. -
qlpack.yml
files now support an additional fieldinclude
in which glob patterns of additional files that should be included (or excluded) when creating a given CodeQL pack can be specified. -
QL packs created by the experimental
codeql pack create
command will now include some information about the build in a newbuildMetadata
field of theirqlpack.yml
file. -
codeql database create
now supports the same flags ascodeql database init
for automatically recognizing the languages present in checkouts of GitHub repositories:-
--github-url
accepts the URL of a custom GitHub instance (previously onlygithub.com
was supported). -
--github-auth-stdin
allows a personal access token to be provided through standard input (previously only theGITHUB_TOKEN
environment variable was supported).
-
- Documentation has been added detailing how to use the "indirect
build tracing" feature, which is enabled by using the
--begin-tracing
flag provided bycodeql database init
. The new documentation can be found here. This feature was temporarily described as "sandwiched tracing" in the 2.6.0 release notes.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
-
The
codeql resolve qlref
command will now throw an error when the target is ambiguous.The qlref resolution rules are now as follows:
-
If the target of a qlref is in the same qlpack, then that target is always returned.
-
If multiple targets of the qlref are found in dependent packs, this is an error.
Previously, the command would have arbitrarily chosen one of the targets and ignored any ambiguities.
-
-
The
qlpack
directive in query suites has its semantics changed. Previously, this directive would return all queries in the qlpack. Now, the directive returns only those queries matched by thedefaultSuite
directive in the query pack. Here is an example:Consider a
qlpack.yml
like the following:name: codeql/my-qlpack version: 0.0.1 defaultSuite: queries: standard
And the directory structure is the following:
qlpack.yml standard/ a.ql experimental/ b.ql
A query suite
suite.qls
like this:- qlpack: codeql/my-qlpack
Previously, would return all the queries in all subdirectories (i.e,
standard/a.ql
andexperimental/b.ql
). Now, it only returnsstandard/a.ql
, since that is the only query matched by its default suite.If you want to have the same behavior as before, you must update your query suites to use the
queries
directive with afrom
attribute, like this:- queries: . from: codeql/my-qlpack
-
Commands that evaluate CodeQL queries now support an additional option
--evaluator-log=path/to/log.json
that will result in the evaluator producing a structured log (in JSON format) of events that occurred during evaluation in order to aid debugging of query performance. The format of these logs will be subject to change with no notice as we make modifications to the evaluator.There is also a new CLI command
codeql generate log-summary
that will produce a summary of the predicates that were evaluated from these event logs. We will aim to keep this summary format more stable, although it is also subject to change. Unless you have a good reason to use the event logs directly, it is strongly recommended you use this command to produce summary logs and use these instead.For further information on these new logs and additional options to configure their format and verbosity, please refer to the CLI documentation.
-
QL classes can now be non-extending subtypes via the
instanceof
keyword, allowing for a form of private subtyping that is not visible externally. Methods of the supertype are accessible from within a non-extending subtype class through extended semantics of thesuper
keyword.class Foo instanceof int { Foo() { this in [1 .. 10] } string toString() { result = "foo" + super.toString() } }
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.28) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.28 instance, you need to create them with release 2.5.9.
-
The
physicalLocation.artifactLocation.uri
fields in SARIF output are now properly encoded as specified by RFC 3986. -
The
--include-extension
option to thecodeql database index-files
command no longer includes directories that are named with the provided extension. For example, if the option--include-extension=.rb
is provided, then a directory namedfoo.rb/
will be excluded from the indexing.
-
A new
codeql database unbundle
subcommand performs the reverse ofcodeql database bundle
and extracts a CodeQL database from an archive. -
The CLI now understands per-codebase configuration files in the format already supported by the CodeQL Action. The configuration file must be given in a
--codescanning-config
option tocodeql database create
orcodeql database init
. For some languages, this configuration can contain pathname filters that control which parts of the codebase is analysed; the configuration file is the only way this functionality is exposed. The configuration file can also control which queries are run, including custom queries from repositories that must first be downloaded. To actually use those queries, runcodeql database analyze
without any query-selection arguments. -
The CLI now supports the "sandwiched tracing" feature that has previously only been offered through the separate CodeQL Runner. This feature is intended for use with CI systems that cannot be configured to wrap build actions with
codeql database trace-command
. Instead the CI system must be able to set custom environment variables for each build action; the required environment variables are output bycodeql database init
when given a--begin-tracing
argument.On Windows,
codeql database init --begin-tracing
will also inject build-tracing code into the calling process or an ancestor; there are additional options to control this. -
This version contains beta support for a new packaging and publishing system for third-party QL queries and libraries. It comprises the following new commands:
-
codeql pack init
: Creates an empty CodeQL pack from a template. -
codeql pack add
: Adds a dependency to a CodeQL pack. -
codeql pack install
: Installs all pack dependencies specified in theqlpack.yml
file. -
codeql pack download
: Downloads one or more pack dependencies into the global package cache. -
codeql pack publish
: Publishes a package to the GitHub Container Registry. -
(Plumbing)
codeql pack bundle
: Builds a.zip
file for a CodeQL query or library pack from sources. Used bycodeql pack publish
. -
(Plumbing)
codeql pack create
: Creates a compiled CodeQL query or library pack from sources. Used bycodeql pack bundle
. -
(Plumbing)
codeql pack packlist
: Lists all files in a local CodeQL pack that will be included in the pack's bundle. Used bycodeql pack create
. -
(Plumbing)
codeql pack resolve-dependencies
: Resolves all transitive dependencies of a local CodeQL pack. Used bycodeql pack install
.
-
This release corresponds to release 1.28.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.6.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
- The QL compiler now verifies that
@security-severity
query metadata is numeric. You can disable this verification by passing the--no-metadata-verification
flag.
-
The
database index-files
anddatabase trace-command
CLI commands now support--threads
and--ram
options, which are passed to extractors as suggestions. -
The
database finalize
CLI command now supports the--ram
option, which controls memory usage for finalization. -
The
database create
CLI command now supports the--ram
option, which controls memory usage for database creation. - Thegenerate query-help
CLI command now support rendering query help in SARIF format.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
-
codeql database create
andcodeql database init
can now automatically recognise the languages present in checkouts of GitHub repositories by making an API call to the GitHub server. This requires a PAT token to either be set in theGITHUB_TOKEN
environment variable, or passed by stdin with the--github-auth-stdin
argument. -
Operations that make outgoing HTTP calls (that is,
codeql github upload-results
and the language-detection feature described above) now support the use of HTTP proxies. To use a proxy, specify an$https_proxy
environment variable for HTTPS requests or a$http_proxy
environment variable for HTTP requests. If the$no_proxy
variable is also set, these variables will be ignored and requests will be made without a proxy.
- The QL language now has a new method
toUnicode
on theint
type. This method converts Unicode codepoint to a one-character string. For example,65.toUnicode() = "A"
,128512.toUnicode()
results in a smiley, andany(int i | i.toUnicode() = "A") = 65
.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
-
codeql database create
(and the plumbing commands it comprises) now supports creating databases for a source tree with several languages while tracing a single build. This is enabled by a new--db-cluster
option. Once created, the multiple databases must be analyzed one by one. -
codeql database create
andcodeql database init
now accept an--overwrite
argument which will lead existing CodeQL databases to be overwritten. -
codeql database analyze
now supports "diagnostic" queries (tagged@kind diagnostic
), which are intended to report information about the analysis process itself rather than problems with the analyzed code. The results of these queries will be summarized in a table printed to the terminal whencodeql database analyze
finishes.They are also included in the analysis results in SARIF output formats as notification objects so they can be displayed by subsequent tooling such as the Code Scanning user interface.
-
For SARIF v2.1.0, a reporting descriptor object for each diagnostic query is output to output to
runs[].tool.driver.notifications
, orruns[].tool.extensions[].notifications
if running with--sarif-group-rules-by-pack
. A rule object for each diagnostic query is output toruns[].resources[].rules
for SARIF v2, or toruns[].rules
for SARIF v1. -
Results of diagnostic queries are exported to the
runs[].invocations[].toolExecutionNotifications
property in SARIF v2.1.0, theruns[].invocations[].toolNotifications
property in SARIF v2, and theruns[].toolNotifications
property in SARIF v1.
SARIF v2.1.0 output will now also contain version information for query packs in
runs[].tool.extensions[].semanticVersion
, if the Git commit the queries come from is known. -
-
codeql github upload-results
has a--checkout-path
option which will attempt to automatically configure upload target parameters. When this is given, the--commit
option will be taken from the HEAD of the checkout Git repository, and if there is precisely one remote configured in the local repository, the--repository
and--github-url
options will also be automatically configured. -
The CodeQL C++ extractor includes beta support for C++20. This is only available when building codebases with GCC on Linux. C++20 modules are not supported.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
- When scanning the disk for QL packs and extractors, directories of
the form
.../SOMETHING/SOMETHING.testproj
(where the twoSOMETHING
are identical) will now be ignored. Names of this form are used bycodeql test run
for ephemeral test databases, which can sometimes contain files that confuse QL compilations.
-
Query writers can now optionally use
@severity
in place of@problem.severity
in the metadata for alert queries. SARIF consumers should continue to consume this severity information using therule.defaultConfiguration.level
property for SARIF v2.1.0, and corresponding properties for other versions of SARIF. They should not depend on the value stored in therule.properties
property bag, since this will contain either@problem.severity
or@severity
based on exactly what was written in the query metadata. -
When exporting analysis results to SARIF v2.1.0, results and metric results now contain a reporting descriptor reference object that specifies the rule that produced them. For metric results, this new property replaces the
metric
property. -
codeql database analyze
now outputs a table that summarizes the results of metric queries that were part of the analysis. This can be suppressed by passing the--no-print-metrics-summary
flag.
- When using the
--sarif-group-rules-by-pack
flag to place the SARIF rule object for each query underneath its corresponding query pack inruns[].tool.extensions
, therule
property of result objects can now be used to look up the rule within therules
property of the appropriate query pack inruns[].tool.extensions
. Previously, rule lookup for result objects in the SARIF output was not well-defined when the--sarif-group-rules-by-pack
flag was passed.
- This release is identical to release 2.5.3, except that
codeql database analyze
no longer produces a generatedautomationDetails.id
field when the--sarif-category
is not explicitly provided. Previously, the--sarif-category
was autogenerated if not present. - Code Scanning users should upgrade to this version and avoid 2.5.3.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
-
When tracing a C/C++ build, the C compiler entries in compiler-settings must now specify
order compiler,extractor
. The default configuration already does this, so no change is necessary if using the default configuration. -
codeql database analyze
andcodeql database interpret-results
now report the results of summary metric queries in the<run>.properties.metricResults
property of the SARIF output. Summary metric queries describe metrics about the code analyzed by CodeQL. They are identified by the query metadata@kind metric
and@tag summary
. For example, see the lines of code summary metric query for C++. -
codeql database analyze
andcodeql database interpret-results
now calculate an automation ID and add it to the resulting SARIF. In SARIF v2.1.0, this field isruns[].automationDetails.id
. In SARIF v2, this field isruns[].automationLogicalId
. In SARIF v1, this field isruns[].automationId
. By default, this automation ID will be derived from the database language and the operating system of the machine that performed the run. It can be set explicitly using a new--sarif-category
option. -
In query metadata,
@kind alert
and@kind path-alert
are now recognized as (more accurate) synonyms of@kind problem
and@kind path-problem
, respectively. -
Diagnostic queries are now permitted by the metadata verifier. They are identified by
@kind diagnostic
metadata. Currently the result patterns of diagnostic queries are not verified. This will change in a future CLI release.
- Ensure the correct URL is generated during
codeql github upload-results
for GitHub Enterprise Server.
This release is identical to release 2.5.1, except that an internal incompatibility with the CodeQL action (and the codeql-runner that some customers use for CI integrations) has been fixed.
The fix does not affect any use cases where the CLI is downloaded from github/codeql-cli-binaries, so if you're seeing this release there, there's no need to upgrade from 2.5.1.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
- The QL compiler will now reject queries where the query metadata (if
present) at the top of the
.ql
file is inconsistent with the output format of the query. This check can be disabled by giving the--no-metadata-verification
flag. (The flag already existed but has not had any effect until now.)
-
Environment variables required for Java extraction are now propagated by the tracer. This may resolve issues with tracing and extraction in the context of certain build systems such as Bazel.
-
A number of
--check-CONDITION
options tocodeql database finalize
andcodeql dataset import
designed to look for consistency errors in the intermediate "TRAP" output from extractors erroneously did nothing. They will now actually print warnings if errors are found. The warnings become fatal errors if the new--fail-on-trap-errors
option is also given.
-
codeql resolve qlref
is a new command that takes in a.qlref
file for a CodeQL test case and returns the path of the.ql
file it references. -
codeql database analyze
andcodeql database interpret-results
have a new--sarif-group-rules-by-pack
option which will place the SARIF rule object for each query underneath its corresponding query pack inruns[].tool.extensions
. -
codeql database finalize
andcodeql dataset import
have a new--fail-on-trap-errors
option that will make database creation fail if extractors produce ill-formatted "TRAP" data for inclusion into a database. This is not enabled by default because some of the existing extractors have minor output bugs that cause the check to fail. -
codeql database finalize
andcodeql dataset import
have a new--check-undefined-labels
option that enables stricter consistency checks on the "TRAP" output from extractors.
super
may now be used unqualified, e.g.super.predicateName()
, when the declaring class has multiple super types, as long as the call itself is unambiguous.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.27) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.27 instance, you need to create them with release 2.4.6.
- By default,
codeql test
now performs additional compiler checks when extracting test code written in Java. Existing Java tests that previously passed may therefore fail due to this change, if they do not compile using thejavac
compiler. To allow time to migrate existing tests, the new behavior can be disabled by setting the environment variableCODEQL_EXTRACTOR_JAVA_FLOW_CHECKS=false
.
- Log files that contain output from build processes will now prefix
it with
[build-stdout]
and[build-stderr]
instead of[build]
and[build-err]
. In particular the latter sometimes caused confusion.
- The QL language now recognizes new
pragma[only_bind_into](...)
andpragma[only_bind_out](...)
annotations on expressions. Advanced users may use these annotations to provide hints to the compiler to influence binding behavior and thus indirectly performance.
This release corresponds to release 1.27.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.5.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.
-
Fixed a bug in
codeql test run
that causes tests to fail messily if the freshly-extracted test database needed to be upgraded in order to be compatible with the QL source under test. This would happen more often at the end of a release cycle, after updates to the QL repository had happened. -
codeql github upload-results
should now work correctly against GitHub Enterprise Server instances that are configured with a path prefix.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.
-
The C/C++ extractor can now parse more Microsoft language extensions when in C++14 and C++17 mode.
-
codeql database analyze
now reports the name and version of each QL pack used by the analysis. You can find this information in the SARIF output. In particular, theruns[0].tool.extensions
property contains an object for each QL pack used by the analysis. Each object contains thename
andsemanticVersion
of the corresponding QL pack, if such information is available. -
codeql github upload-results
is a new command that uploads a SARIF file generated by CodeQL to GitHub's Code Scanning.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.
- The
name
property inqlpack.yml
must now meet the following requirements:- Only lowercase ASCII letters, ASCII digits, and hyphens (
-
) are allowed. - A hyphen is not allowed as the first or last character of the name.
- The name must be at least one character long, and no longer than 128 characters.
- Only lowercase ASCII letters, ASCII digits, and hyphens (
- Alert and path queries can now give a score to each alert they
produce. You can incorporate alert scores in an alert or path query
by first adding the
@scored
property to the query metadata. You can then introduce a new numeric column at the end of theselect
statement structure to represent the score of each alert. Alert scores are exposed in the SARIF output of commands likecodeql database analyze
as thescore
property in the property bags of result objects.
-
The default value of the
--working-dir
options for theindex-files
andtrace-command
subcommands ofcodeql database
has been fixed to match the documentation; previously, it would erroneously use the process' current working directory rather than the database source root. -
codeql test run
will not crash if database extraction in a test directory fails. Instead only the tests in that directory will be marked as failing, and tests in other directories will continue executing.
Fixes several bugs introduced in 2.4.2, related to searching the disk for QL packs:
-
In many cases the search would scan through more of the file system than it should. Often the only effect of this was that the scan would take longer (sometimes significantly longer) but in some corner cases it could lead to packs being found that shouldn't be found, which could lead to compilation failure if different versions of the same pack exist on disk.
-
The search would terminate a fatal error if it met a directory without read permission.
-
A
provide
entry in.codeqlmanifest.json
that ended with*
would erroneously not match a.codeqlmanifest.json
in a subdirectory.
As a consequence of the latter fix, the semantics of
.codeqlmanifest.json
files has changed slightly: Directory names
that start with a dot used to not be matched by the pattern elements
*
and **
, whereas now even dotted directories match such a pattern
element. The previous behavior was never documented, and only very few
users have .codeqlmanifest.json
files of their own in the first
place, so this change is expected to have minimal practical effect.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.26) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.26 instance, you need to create them with release 2.3.4.
-
codeql query format
now checks all files rather than stopping after the first failure when the--check-only
option is given. -
codeql resolve database
will produce alanguages
key giving the language the database was created for. This can be useful in IDEs to help describe the database and suggest default actions or queries. For databases created by earlier versions, the result will be a best-effort guess. -
codeql database interpret-results
can now produce Graphviz.dot
files from queries with@kind graph
.
codeql test run
had some special compatibility support for running unit tests for the "code duplication" extractor features of certain discontinued Semmle products. Those tests have since been removed from the public QL repository, so the compatibility support for them has been removed. This should not affect any external users (since the extractor feature in question was never supported bycodeql database create
anyway), but if you runcodeql test run
against the unit tests belonging to an old checkout of the repository, you may now see some failures amongMetrics
tests.
This release corresponds to release 1.26.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.4.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.
For all purposes other than creating databases for LGTM Enterprise we recommend that you upgrade to CLI releases numbered 2.4.x or later.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
-
Much of the work done by
codeql database upgrade
now happens implicitly (and reversibly) as part of ordinary query evaluation. This should make the need to explicitly runcodeql database upgrade
much less common. However there are still some corner cases that will require it, particularly for very old databases. -
codeql test run
with a--threads
argument will now compile test queries in parallel even if they belong to the same single test directory. This can speed up localized testing considerably.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
-
Fixed bug in
codeql test run
where the--format
and--failing-exitcode
options would not work reliably when--ram
was also given -
The
$CODEQL_JAVA_HOME
environment variable will now be passed to extractors such that extractors implemented in Java can be affected too. Beware that this variable will override the JVM that executes the maincodeql
process. It should not normally be set explicitly.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
-
codeql database create
now accepts a--working-dir
option, which allows the working directory for extractor scripts to differ from the source root. This is useful in some specialized integration situations. -
codeql database create
will now pass a--compiler-spec
option on tocodeql database trace-command
. This allows adapting the build tracing process when unusual compiler toolchains are used. -
codeql database init
accepts an--allow-missing-source-root
option, which is useful in some specialized integration situations.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.25) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.25 instance, you need to create them with release 2.2.6.
-
The Java extractor no longer supports builds running on a Java 6 JRE. The minimum supported version is Java 7.
-
The interpretation of binding set annotations in QL has changed subtly. In rare cases, existing QL code that contains explicit binding set annotations on overriding class predicates may now be rejected with errors of the form "... is not bound to a value". You can fix this by adding explicit binding sets to the overridden predicate, or to the abstract class itself in the case of the characteristic predicate. For more information about binding sets, see Annotations in the QL language reference.
- You can now use binding sets on class bodies. This lets you explicitly annotate dynamically dispatched characteristic predicates.
-
Query authors can use the new subcommand
codeql generate query-help
to validate query help files and render the files as Markdown. For more information, see Testing query help files. -
The new subcommand
codeql bqrs hash
computes a stable hash of a BQRS file. -
codeql query decompile
now accepts a--kind
flag. This allows advanced users to choose which intermediate representation to show for a compiled QL query.--kind dil
shows the Datalog representation while--kind ra
shows the relational algebra representation used by the evaluator.
This release corresponds to release 1.25.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.3.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.
For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.
-
The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
-
Updated license terms with a rewritten description of what is and is not allowed. No substantive changes are intended, but the new text is hopefully easier to understand.
-
The CLI can now execute queries that use QL's
external predicate
feature. All subcommands that execute queries have a new--external
option to specify the value set for those predicates. -
A new
codeql bqrs diff
command can be used to compute the difference between two binary query result sets. -
codeql test run
has some new options to improve support for testing of extractors:--check-databases
which will runcodeql dataset check
on every test database produced during a run.--consistency-queries
which will run a set of additional queries over all the test databases produced during a run.--show-extractor-output
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
-
QL packs found through the
--search-path
option, or in a sibling directory to the unpacked CLI would erroneously take precedence over the content of the workspace when using the CodeQL extension for Visual Studio Code. This is now fixed such that the workspace takes priority. -
Two command-line options that control the amount of disk space that the QL evaluator will try to keep free of disk cache are now called
--min-disk-free
and--min-disk-free-pct
. Previously they were called--max-disk-free
instead, which made no sense. The old names are still recognized such as not to break existing scripts, but are now undocumented and deprecated.
CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new
version number because the v2.2.2
folder on the download site
originally contained the 2.2.0 binaries instead of the correct 2.2.2
ones.
If you have downloaded release 2.2.2, and codeql --version
correctly
identifies itself as being that version, you don't need to upgrade to
2.2.3.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
- Query evaluations that time out due to a
--timeout
option are no longer silently discarded. Insteadcodeql
will terminate with exit code 33. Commands that evaluate multiple queries will produce as much output as they can even if one of the queries times out.
There is no CodeQL CLI version 2.2.1. This version number was used internally to work around restrictions in the CodeQL for VS Code extension.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.24) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.24 instance, you need to create them with release 2.1.4.
- Starting with this release, the CodeQL CLI can be downloaded either
as a single
codeql.zip
file containing the CLI for all supported platforms, or as acodeql-PLATFORM.zip
that contains the files for just one platform. The single-platform zips are faster to download.
- QL now supports the definition of new types as type unions. This feature currently allows unions of branches from an already existing algebraic data type and unions of database types.
This release corresponds to release 1.24.x of LGTM Enterprise, and should be used when creating databases that will be uploaded to it. Future CLI releases (numbered 2.2.x) may produce databases that are not backwards compatible with this version of LGTM Enterprise.
For all purposes other than creating databases for LGTM Enterprise we recommend that you continue upgrading to newer CLI releases as they become available.
- A new
codeql query format
command exposes the QL autoformatter for use on the command line.
-J
command-line options that contain spaces now ought to work on Windows. They still do not work reliably on Linux or MacOS, though.
- Fixes a bug in
codeql execute cli-server
(a helper used by the VS Code extension) which would sometimes cause query compilation to fail until the extension was restarted. - Fixes a bug in
codeql database upgrade
which could lead to performance losses if the upgraded database was subsequently used with LGTM or the legacy Semmle Core product. - Fixes a bug in the QL evaluator that would sometimes lead to crashes
for queries that use the new
unique
aggregate added in release 2.1.0. - The value of the
--compilation-cache-size
option is now correctly interpreted as a number of megabytes rather than a number of bytes.
- Updated license terms to allow CI use with GitHub Actions for open-source software.
- In query suite definitions, filter
instructions that filter on the
query path
pseudo-tag will now always see the relative path to the query expressed with/
as a directory separator, independently of the platform. Previously they erroneously used the platform's directory separator, meaning that query suites developed on Windows would not work correctly on Unix systems (and vice versa) if they usedquery path
. Existing suite definitions developed on Windows may need to be updated to match the new behavior.
- A new
codeql test accept
subcommand helps automate updating the expected output for unit tests after a desired change in query behavior. This can also be done by the new--learn
option forcodeql test run
.
codeql database create
will now report an explicit error if given a--command
argument that specifies an empty string. Previously this would be accepted initially, leading to confusing failures later.
- The bundled extractors are updated to match the versions currently used on LGTM.com.
codeql resolve queries
accepts a--format=bylanguage
option. This is used to help automated workflows determine which languages to create databases for, from the queries that are available to run.- It is now possible to attempt to execute
.ql
files that are not in a QL pack. This is used by a few specialized internal workflows. However, standalone queries cannot import any of the dependencies that you would usually declare in aqlpack.yml
file, so will not be useful in most cases.
- The bundled extractors are updated to match the versions currently used on LGTM.com. These are newer than the last release (1.23) of LGTM Enterprise. If you plan to upload databases to an LGTM Enterprise 1.23 instance, you need to create them with release 2.0.1. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
-
If you pass a directory name as a command-line argument to
codeql test run
, it will now consider all.ql
or.qlref
files found under that directory to be test queries, even if they have no accompanying.expected
file. Tests that lack an.expected
file will fail, but will generate an.actual
file that you can rename to.expected
if you want to use the results.The goal of this change is to support existing workflows of experienced CodeQL users, and also to provide clear error indications if an
.expected
file is accidentally lost, renamed, or misspelled.However, if you invoke
codeql test run
on a directory tree that contains both tests and non-test queries, you will now encounter errors if any of the.ql
files can't be processed as test queries. If you're affected by this change, you can suppress these errors by:- Adding a
tests
property to this QL pack to define specify which directories contain only test queries and associated test code. For more information, see About QL packs. - Running
codeql test run
with a new--strict-test-discovery
option.
In the longer term, we recommend that you reorganize the queries so that test queries are stored in a directory tree that's separate from actual queries.
- Adding a
-
codeql database create
andcodeql database finalize
will no longer recognize a--no-duplicate-code
option. This option has never had any effect, and its positive variant--duplicate-code
previously led to a fatal error.
- A new XML extractor is included. It is not intended to be used as a stand-alone extractor, but rather to augment the data produced by other extractors. In particular, the C# and Java extractors invoke it during database creation to include information relevant to the analysis of those languages, much like LGTM.com does.
- Two new plumbing commands
codeql database index-files
andcodeql resolve files
have been added for support of invoking the XML extractor support. These commands are generally only of interest for extractor authors. - Two new plumbing commands have been added to
codeql dataset
. Themeasure
subcommand can be used to collect size information from a dataset, and thecheck
subcommand can scan a dataset for database inconsistencies. These commands are useful when developing a new CodeQL extractor. - The QL evaluator contains a number of features in support of an
internal experiment with using machine-learning techniques to
identify functions in unknown codebases as sources or sinks of
taint. This includes new command-line options
--ml-model-path
and--native-library-path
to several subcommands. As the new features are not yet ready for general use, these new options should be ignored by external CodeQL users.
- Fixes a bug that could result in empty databases for C/C++.
Previously, extraction would mistakenly be skipped for source files
compiled with the Clang compiler, if the
-fintegrated-cc1
option was specified. codeql database create
andcodeql database init
will now, as they have always been documented, refuse to create a database whose parent directory doesn't already exist.codeql test run
will no longer leave.actual
files from previous runs in the file system after a test passes.
-
QL now supports set literals, and the QL extractor can identify them with the
SetLiteral
class. For more information, see Set literal expressions in the QL language reference. -
QL now supports a uniqueness aggregate. This can express constraints that there is precisely one value. The syntax is taken from previous aggregates such as
min
andmax
.unique(int x | x = 4 or x = 2 * 2 | x)
- Fixes a problem preventing
codeql database create
from working with Python 3 on macOS. - Fixes a problem preventing
codeql database create
from finding locally installed Python packages.
- The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
codeql test run
has a new--slice
option that can be used to parallelize tests over more machines.
- The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
- Subcommands that execute queries (such as
codeql database analyze
) now have a--timeout
option that can be used to set a timeout to automatically cancel query evaluations that appear to diverge. - A new plumbing command
codeql query decompile
can display the DIL intermediate representations that is included in the output ofcodeql query compile --dump-qlo --include-dil-in-qlo
. This is useful mainly for certain internal workflows; the information produced is the same as whatcodeql query compile --dump-dil
already outputs.
- The
--debug
and--tuple-counting
options tocodeql test run
erroneously had no effect. Now they ought to work.
- Fixes a bug where
codeql test run
would fail with the messageCatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it
when running tests against themaster
branch of the CodeQL libraries for certain languages. - Otherwise identical to release 2.0.2.
- The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the versions currently used on LGTM.com. These are newer than the last release of LGTM Enterprise, so this release should not be used if you plan to upload databases to an LGTM Enterprise instance. For more information, see Preparing CodeQL databases to upload to LGTM in the LGTM admin help.
- The parent and sibling directories of the unpacked CLI are no longer
searched recursively for QL packs. QL packs will only be found if
there's a
qlpack.yml
or.codeqlmanifest.json
directly in a parent or sibling directory. This should eliminate the very long disk-scanning delays experienced by users who unpacked earlier versions of the CLI in their home directory. - Parent and sibling directories of the unpacked CLI will now be
searched for QL packs as a last resort, even if you give an explicit
--search-path
option. This means, for example, that you can define a search path in the per-user configuration file without it depending on where the CLI is unpacked. In particular, the setting can now be meaningfully used by users who let the CodeQL for VS Code extension manage the downloading and unpacking of the CLI.
- The
codeql database create
command and its relatives will no longer attempt to find extractors located in the parent and sibling directories of the unpacked CLI. This closes a security risk for users who unpacked the CodeQL CLI in their home directory. This could've resulted in arbitrary code execution if the user unpacked a file archive containing a malicious extractor anywhere in the home directory. Extractors will now only be found within the unpacked CLI itself, or in directories explicitly listed in the--search-path
. It is expected that users will only point--search-path
to locations they trust at least as much as the CLI download itself.
- This release supports executing query regression tests using the
codeql test
command. For further information, see Testing custom queries. - The error message if you try executing a query against a database
that needs to be upgraded (which can happen routinely if you're
using a fresh
master
checkout of the CodeQL libraries with the bundled extractors) will now explicitly suggest acodeql database update
command to run. The database is not automatically upgraded, as this may make it irreversibly incompatible with older versions of the CodeQL libraries. This allows users who want to compare behavior of different versions of the libraries against the same database to make a copy before they upgrade it.
- Corresponds to LGTM Enterprise release 1.23.
- The bundled extractors (which are responsible for converting source code to databases for each supported language) are updated to match the extractor versions used in LGTM Enterprise.
- No other changes to the core CLI.
- First public release.