Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
**Released on November 22nd, 2015.** This is a huge release and marks a major milestone for Kyua as it finally implements a long-standing feature request: the ability to execute test cases in parallel. This is a big deal because test cases are rarely CPU-bound: running them in parallel yields much faster execution times for large test suites, allowing faster iteration of changes during development. As an example: the FreeBSD test suite as of this date contains 3285 test cases. With sequential execution, a full test suite run takes around 12 minutes to complete, whereas on a 4-core machine with a high level of parallelism it takes a little over 1 minute. Implementing parallel execution required rewriting most of Kyua's core and partly explains explains why there has not been a new release for over a year. The current implementation is purely subprocess-based, which works but has some limitations and has resulted in a core that is really complex and difficult to understand. Future versions will investigate the use of threads instead for a simplified programming model and additional parallelization possibilities. * Issue #2: Implemented support to execute test cases in parallel when invoking `kyua test`. Parallel execution is *only* enabled when the new `parallelism` configuration variable is set to a value greater than `1`. The default behavior is still to run tests sequentially because some test suites contain test cases with side-effects that might fail when run in parallel. To resolve this, the new metadata property `is_exclusive` can be set to `true` on a test basis to indicate that the test must be run on its own. * Known regression: Running `kyua debug` on a TAP-based test program does not currently report the output in real time. The output will only be displayed once the test program completes. This is a shortcoming of the new parallel execution engine and will be resolved. * Removed the external C-based testers code in favor of the new built-in implementations. The new approach feels significantly faster than the previous one. * Fixed the handling of relative paths in the `fs.*` functions available in `Kyuafile`s. All paths are now resolved relative to the location of the caller `Kyuafile`. `Kyuafile.top` has been updated with these changes and you should update custom copies of this file with the new version. * Changed temporary directory creation to always grant search permissions on temporary directories. This is to prevent potential problems when running Kyua as root and executing test cases that require dropping privileges (as they may later be unable to use absolute paths that point inside their work directory). * The cleanup of work directories does not longer attempt to deal with mount points. If a test case mounts a file system and forgets to unmount it, the mount point will be left behind. It is now the responsibility of the test case to clean after itself. The reasons for this change are simplicity and clarity: there are many more things that a test case can do that have side-effects on the system and Kyua cannot protect against them all, so it is better to just have the test undo anything it might have done. * Improved `kyua report --verbose` to properly handle environment variables with continuation lines in them, and fixed the integration tests for this command to avoid false negatives. * Changed the configuration file format to accept the definition of unknown variables without declaring them local. The syntax version number remains at 2. This is to allow configuration files for newer Kyua versions to work on older Kyua versions, as there is no reason to forbid this. * Fixed stacktrace gathering with FreeBSD's ancient version of GDB. GDB 6.1.1 (circa 2004) does not have the `-ex` flag so we need to generate a temporary GDB script and feed it to GDB with `-x` instead. * Issue #136: Fixed the XML escaping in the JUnit output so that non-printable characters are properly handled when they appear in the process's stdout or stderr. * Issue #141: Improved reporting of errors triggered by sqlite3. In particular, all error messages are now tagged with their corresponding database filename and, if they are API-level errors, the name of the sqlite3 function that caused them. * Issue #144: Improved documentation on the support for custom properties in the test metadata. * Converted the `INSTALL`, `NEWS`, and `README` distribution documents to Markdown for better formatting online.
- Loading branch information