Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modern testing #2384

Draft
wants to merge 266 commits into
base: next
Choose a base branch
from
Draft

Modern testing #2384

wants to merge 266 commits into from

Conversation

Rot127
Copy link
Collaborator

@Rot127 Rot127 commented Jun 8, 2024

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Modernize testing.

The new unit testing is build around yaml files. They can be consumed by the C test engine (cstest) or any bindings easily.

Statistics

The new cstest logs technically two statistics. One by cmocka and one by cstest.
The reason is:

  1. We might want to group tests in the future (by arch or file for example) so we need a global statistics.
  2. Errors of incorrect yaml parsing are not tracked by cmocka.
  3. Also, cmocka doesn't provide the detailed statistics to us. So we cannot print a nice summary at the end.

Changes

  • New cstest C-implementation
  • New py_cstest Python-implementation
  • Moves almost all tests to yaml files.
  • Old tests which can't be moved (because they are effectively unit tests) are kept.
  • Add numerous test jobs in the workflow
  • Adds several CMake test targets
  • Add LoongArch Python bindings
  • Several bug fixes discovered during testing.

Todo

  • Camel to snake case in cstest.
  • No _set testing in Python code.

Test plan

Added and all green

Closing issues

closes #1984
closes #2405

@Rot127 Rot127 added the Testing Test related issue label Jun 8, 2024
@Rot127 Rot127 added this to the v6 milestone Jun 8, 2024
@github-actions github-actions bot added Auto-Sync-files Auto-Sync and removed HPPA Arch labels Jun 26, 2024
@Rot127 Rot127 mentioned this pull request Jul 4, 2024
2 tasks
@Rot127 Rot127 force-pushed the modern-testing branch 2 times, most recently from 1673b1f to c4e1ab2 Compare July 8, 2024 07:33
@github-actions github-actions bot removed X86 Arch ARM Arch Mips Arch AArch64 Arch Alpha Arch PowerPC Arch labels Jul 8, 2024
@github-actions github-actions bot added python bindings Github-files Github related files labels Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

test_riscv.py failing Modernize Capstone testing
1 participant