Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: system test parallelization: two-pass approach #23275

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft

WIP: system test parallelization: two-pass approach #23275

wants to merge 22 commits into from

Commits on Oct 21, 2024

  1. EXPERIMENTAL! In teardown, if we see leaks, ... 

    ...try to trace them back to the culprit tests

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0b66a92 View commit details
    Browse the repository at this point in the history

  2. (debug) for 21569: log play-kube command, show at end 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    bbe6c96 View commit details
    Browse the repository at this point in the history

  3. test/system: podman play --service-container slog to syslog 

    In theory when syslog is set the cleanup process should log its errors
to syslog (journald) so we can have a look at the errors in CI.
Without it podman container cleanup errors will never be logged anywhere.

In order to rey to debug  containers#21569

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
    @Luap99 @edsantiago
    Luap99 authored and edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0f8fc2c View commit details
    Browse the repository at this point in the history

  4. DO NOT MERGE: skip unneeded CI tasks 

    All we care about in this PR is system tests.

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    f7e930a View commit details
    Browse the repository at this point in the history

  5. FIXME: update docs, teardown, ... 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    64b1ef3 View commit details
    Browse the repository at this point in the history

  6. DO NOT MERGE: test tail logging fix from Luap99 fork 

    Luap99@df865c8

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    a8e939d View commit details
    Browse the repository at this point in the history

  7. FIXME-debug for k8s-file test 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    43994e0 View commit details
    Browse the repository at this point in the history

  8. why is USEC test failing 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    77d6c87 View commit details
    Browse the repository at this point in the history

  9. try parallelizing the USEC test again 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    7c22886 View commit details
    Browse the repository at this point in the history

  10. 220: add a FIXME comment about a race 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d7323b4 View commit details
    Browse the repository at this point in the history

  11. flake fix in 700: allow 137 status code from timeout command 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    b52aa69 View commit details
    Browse the repository at this point in the history

  12. fixmeup-255 debugging 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    b83d402 View commit details
    Browse the repository at this point in the history

  13. unparallelize usec 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    705fcaa View commit details
    Browse the repository at this point in the history

  14. FIXME debugs for ns leak 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d693dd7 View commit details
    Browse the repository at this point in the history

  15. FIXME: rmi pause. Is there a better place to do this? 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3cea3c8 View commit details
    Browse the repository at this point in the history

  16. system tests: safer install_kube_template() 

    Previous version was badly broken: it relied on 'make'
rebuilding a file under cwd, which is a no-no; and, in
the case where we don't have a source directory, just
blindly hoped that there'd be a system-installed .service
file with the correct path to podman.

Solution:
  . if running in source directory, run sed directly into
    destination service file in $UNIT_DIR. This is ugly
    duplication of a line in Makefile.

  . if NOT running in a source directory, check $PODMAN:
    . if it's /usr/bin/podman, continue
    . otherwise skip, because we don't know what we're testing

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    57b0c29 View commit details
    Browse the repository at this point in the history

  17. Add lots more parallel-high-load FIXMEs 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3d1610c View commit details
    Browse the repository at this point in the history

  18. CI: parallelize 450-interactive system tests 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    6adf2fe View commit details
    Browse the repository at this point in the history

  19. CI: parallelize 520-checkpoint tests 

    Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    5223552 View commit details
    Browse the repository at this point in the history

  20. pasta udp tests: new bytecheck helper 

    ...for debugging containers#24147, because "md5sum mismatch" is not
the best way to troubleshoot bytestream differences.

Because socat is run on the container, we need to build
a new testimage (20241009).

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    076ac2e View commit details
    Browse the repository at this point in the history

  21. UGH! try to address 700 timeout flake 

    I just made things worse.

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    74401be View commit details
    Browse the repository at this point in the history

  22. kube SIGINT system test: fix race in timeout handling 

    Up to now this test has been run using:

    PODMAN_TIMEOUT=2 run_podman kube play ...

...and this gives podman time to start the pod before getting
the signal.

When run in parallel, under heavy load, the above command seems
to time out before podman has gotten its act together. Weird
things happen, like weird exit status and (most crucially)
zombie containers.

Solution: wait for container to actually start before we kill it.

Signed-off-by: Ed Santiago <santiago@redhat.com>
    @edsantiago
    edsantiago committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e0764f8 View commit details
    Browse the repository at this point in the history