Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-91960: Add FreeBSD build and test using Cirrus-CI #91961

Merged
merged 3 commits into from
Sep 6, 2023

Conversation

emaste
Copy link
Contributor

@emaste emaste commented Apr 26, 2022

Cirrus-CI is a hosted CI service that supports FreeBSD, Linux, macOS,
and Winodws. Add a .cirrus.yml to provide CI coverage on pull requests
for FreeBSD 12.3 and 13.0.

@cpython-cla-bot
Copy link

cpython-cla-bot bot commented Apr 26, 2022

All commit authors signed the Contributor License Agreement.
CLA signed

@bedevere-bot
Copy link

Every change to Python requires a NEWS entry.

Please, add it using the blurb_it Web app or the blurb command-line tool.

@emaste
Copy link
Contributor Author

emaste commented Apr 26, 2022

It seems the CLA bot does a case-sensitive match but my git author data and GitHub profile don't match :(

Thank you for authorizing our application, but the CLA must be signed by the users who contributed to the PR. Authors emails are: ...@FreeBSD.org.

Let me see what I can do to fix that.

@bedevere-bot
Copy link

Every change to Python requires a NEWS entry.

Please, add it using the blurb_it Web app or the blurb command-line tool.

@erlend-aasland
Copy link
Contributor

I guess we need to enable the Cirrus CI app in our organisation for this to work? @vstinner, is that something you can do?

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

I'm interested by the Cirrus-CI approach. I contacted @ewdurbin and @ambv to know what they are thinking about it.

Having a separated buildbot CI for FreeBSD still remains relevant. For example, a buildbot could also test FreeBSD CURRENT, not only the latest STABLE.

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

Oh, I'm impressed, Cirrus-CI does provide FreeBSD CURRENT machine!

freebsd-14-0-snap (14.0-SNAP)

https://cirrus-ci.org/guide/FreeBSD/

I don't think that we should test too many FreeBSD versions. Honestly, only the latest stable should be way enough. We are already struggling with other CI issues on well supported platforms. For example, see recent GitHub Action incidents: https://www.githubstatus.com/history

@emaste
Copy link
Contributor Author

emaste commented Sep 5, 2023

We (members of the FreeBSD project) certainly see the value in having FreeBSD buildbots participating in Python's standard CI configuration, and we're working on adding new ones to replace the ones that were removed. Some folks will soon travel to EuroBSDCon and so setting up new buildbots might take a little longer than I'd like, but they will come back.

Cirrus-CI could make a good extra or stop-gap CI for FreeBSD.

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

I updated this PR on main to re-run CI jobs, but I forgot that Cirrus-CI cannot be run until it's allowed in our GitHub cpython project :-)

@ewdurbin
Copy link
Member

ewdurbin commented Sep 5, 2023

Cirrus CI should be installed and enabled for python/cpython.

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

@ewdurbin
Copy link
Member

ewdurbin commented Sep 5, 2023

looks good!
cirrus-cpython

@emaste
Copy link
Contributor Author

emaste commented Sep 5, 2023

These should be updated to currently-supported images, i.e.

    - image: freebsd-12-4-release-amd64
    - image: freebsd-13-2-release-amd64

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

freebsd freebsd_instance:freebsd-12-4-release-amd64 In progress
freebsd freebsd_instance:freebsd-13-2-release-amd64 In progress

As I wrote previously, I would prefer to have a single FreeBSD job. Can we please start with just the latest stable, FreeBSD 13?

.cirrus.yml Outdated Show resolved Hide resolved
@emaste
Copy link
Contributor Author

emaste commented Sep 5, 2023

12.4 results, for reference, from https://cirrus-ci.com/task/5457769084485632

== Tests result: FAILURE then FAILURE ==
425 tests OK.
5 tests failed:
    test.test_asyncio.test_events test.test_asyncio.test_sock_lowlevel
    test.test_asyncio.test_sslproto test_imaplib test_socket
23 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_ctypes test_dbm_gnu
    test_devpoll test_epoll test_idle test_ioctl test_launcher
    test_perf_profiler test_perfmaps test_readline test_sqlite3
    test_stable_abi_ctypes test_startfile test_tcl test_tkinter
    test_ttk test_ttk_textonly test_turtle test_winconsoleio
    test_winreg test_wmi
8 tests skipped (resource denied):
    test_curses test_peg_generator test_smtpnet test_socketserver
    test_urllib2net test_urllibnet test_winsound test_zipfile64
5 re-run tests:
    test.test_asyncio.test_events test.test_asyncio.test_sock_lowlevel
    test.test_asyncio.test_sslproto test_imaplib test_socket
Total duration: 23 min 53 sec
Total tests: run=39,271 failures=10 skipped=1,499
Total test files: run=458/461 failed=5 skipped=23 resource_denied=8 rerun=5
Result: FAILURE then FAILURE

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

A new FreeBSD 13.2 (amd64) buildbot was just set up by @zware: python/buildmaster-config#388 , That's so cool!

3 tests are failing: https://buildbot.python.org/all/#/builders/1223/builds/3

3 tests failed:
    test_gdb test_tarfile test_tempfile

I wrote a fix for test_tempfile: #108964 It seems like @zware uses ZFS which doesn't support os.chflags(), whereas the test pass on my UFS filesystem.

@sorcio wrote a fix for tarfile: #108950

I cannot reproduce the test_gdb failure locally (with gdb 13.1) on my 13.2-RELEASE-p2 VM.

I would prefer to first focus on FreeBSD 13 before considering to fix tests on FreeBSD 12.

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

.cirrus.yml Outdated
- image: freebsd-12-4-release-amd64
- image: freebsd-13-2-release-amd64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- image: freebsd-12-4-release-amd64
- image: freebsd-13-2-release-amd64
- image: freebsd-13-2-release-amd64

@vstinner
Copy link
Member

vstinner commented Sep 5, 2023

I modified your PR to change the command running tests and I removed the FreeBSD 12 job (for now), to re-run Cirrus-CI job.

Can you try to add a NEWS entry? See: #91961 (comment)

Or I can take over your PR and add it if you prefer.

@emaste
Copy link
Contributor Author

emaste commented Sep 5, 2023

I would prefer to first focus on FreeBSD 13 before considering to fix tests on FreeBSD 12.

Yeah -- FreeBSD 12.x is EOL at the end of this year anyway, so I don't think it warrants much effort. I pasted the results just because I had already started the run. I started another with your suggested changes, at https://cirrus-ci.com/task/5969562588413952

On 13.2 it reports:

Total duration: 44 min 50 sec
Total tests: run=39,443 failures=7 skipped=1,379
Total test files: run=461/461 failed=4 skipped=24 resource_denied=5 rerun=5
Result: FAILURE then FAILURE

Can you try to add a NEWS entry?

Tried via blurb-it.

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

Cirrus-CI is a hosted CI service that supports FreeBSD, Linux, macOS,
and Winodws.  Add a .cirrus.yml to provide CI coverage on pull requests
for FreeBSD 12.4 and 13.2.
.cirrus.yml Outdated Show resolved Hide resolved
Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let me now see if the latest job with new config (--with-pydebug) does pass: https://cirrus-ci.com/task/4982289931698176

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

Aha, test.test_concurrent_futures.test_shutdown hangs: that's the #107219 issue.

.cirrus.yml Outdated Show resolved Hide resolved
@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

The job was blocked. I added --timeout=600 to put a timeout of 10 minutes.

...
0:16:49 load avg: 0.27 running: test.test_concurrent_futures.test_shutdown (10 min 36 sec)
0:17:19 load avg: 0.17 running: test.test_concurrent_futures.test_shutdown (11 min 6 sec)

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

Aha, now test_gdb decided to fail :-)


======================================================================
FAIL: test_bt (test.test_gdb.PyBtTests.test_bt)
Verify that the "py-bt" command works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 804, in test_bt
    self.assertMultilineMatches(bt,
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 300, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x5ce229: file ../Python/bltinmodule.c, line 1258.\n\nBreakpoint 1, builtin_id (self=<optimized out>, v=42) at ../Python/bltinmodule.c:1258\n1258\t    PyObject *id = PyLong_FromVoidPtr(v);\nTraceback (most recent call first):\n  <built-in method id of module object at remote 0x800cce570>\n  (unable to read python frame information)\n' did not match '^.*\nTraceback \\(most recent call first\\):\n  <built-in method id of module object .*>\n  File ".*gdb_sample.py", line 10, in baz\n    id\\(42\\)\n  File ".*gdb_sample.py", line 7, in bar\n    baz\\(a, b, c\\)\n  File ".*gdb_sample.py", line 4, in foo\n    bar\\(a=a, b=b, c=c\\)\n  File ".*gdb_sample.py", line 12, in <module>\n    foo\\(1, 2, 3\\)\n'

======================================================================
FAIL: test_bt_full (test.test_gdb.PyBtTests.test_bt_full)
Verify that the "py-bt-full" command works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 824, in test_bt_full
    self.assertMultilineMatches(bt,
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 300, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x5ce229: file ../Python/bltinmodule.c, line 1258.\n\nBreakpoint 1, builtin_id (self=<optimized out>, v=42) at ../Python/bltinmodule.c:1258\n1258\t    PyObject *id = PyLong_FromVoidPtr(v);\n#1 <built-in method id of module object at remote 0x800cce570>\n#4 (unable to read python frame information)\n' did not match '^.*\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \\(a=1, b=2, c=3\\)\n    baz\\(a, b, c\\)\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \\(a=1, b=2, c=3\\)\n    bar\\(a=a, b=b, c=c\\)\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in <module> \\(\\)\n    foo\\(1, 2, 3\\)\n'

======================================================================
FAIL: test_pyup_command (test.test_gdb.StackNavigationTests.test_pyup_command)
Verify that the "py-up" command works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 761, in test_pyup_command
    self.assertMultilineMatches(bt,
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 300, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x5ce229: file ../Python/bltinmodule.c, line 1258.\n\nBreakpoint 1, builtin_id (self=<optimized out>, v=42) at ../Python/bltinmodule.c:1258\n1258\t    PyObject *id = PyLong_FromVoidPtr(v);\n#4 (unable to read python frame information)\n#9 <built-in method pyobject_vectorcall of module object at remote 0x80163ef90>\n' did not match '^.*\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file <string>, line 12, in baz \\(args=\\(1, 2, 3\\)\\)\n#[0-9]+ <built-in method pyobject_vectorcall of module object at remote 0x[0-9a-f]+>\n$'

======================================================================
FAIL: test_up_then_down (test.test_gdb.StackNavigationTests.test_up_then_down)
Verify "py-up" followed by "py-down"
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 790, in test_up_then_down
    self.assertMultilineMatches(bt,
  File "/tmp/cirrus-ci-build/Lib/test/test_gdb.py", line 300, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x5ce229: file ../Python/bltinmodule.c, line 1258.\n\nBreakpoint 1, builtin_id (self=<optimized out>, v=42) at ../Python/bltinmodule.c:1258\n1258\t    PyObject *id = PyLong_FromVoidPtr(v);\n#4 (unable to read python frame information)\n#9 <built-in method pyobject_vectorcall of module object at remote 0x80163ef90>\n#4 (unable to read python frame information)\n' did not match '^.*\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file <string>, line 12, in baz \\(args=\\(1, 2, 3\\)\\)\n#[0-9]+ <built-in method pyobject_vectorcall of module object at remote 0x[0-9a-f]+>\n#[0-9]+ Frame 0x-?[0-9a-f]+, for file <string>, line 12, in baz \\(args=\\(1, 2, 3\\)\\)\n$'

----------------------------------------------------------------------

@sorcio
Copy link
Contributor

sorcio commented Sep 6, 2023

test_gdb should be skipped. The skip check currently checks for Darwin but I see no real reason for that. Checking for clang should be enough:

if 'Clang' in platform.python_compiler() and sys.platform == 'darwin':
raise unittest.SkipTest("test_gdb doesn't work correctly when python is"
" built with LLVM clang")

PR: #108993

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

I'm working on a different fix for test_gdb when Python is built with clang: PR #108999.

I would prefer to land it before enabling Cirrus CI on the main branch.

@vstinner vstinner merged commit 2cd170d into python:main Sep 6, 2023
22 of 23 checks passed
@bedevere-bot
Copy link

There's a new commit after the PR has been approved.

@vstinner, @erlend-aasland: please review the changes made to this pull request.

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

Thanks @emaste! This PR wasn't merged earlier before because some tests were failing and there was no core dev available to handle issues specific to FreeBSD. Apparently, some happy FreeBSD developpers are now available and helped me to fix the last tests, so I merged your change to add the FreeBSD CI. Thanks @emaste for working on this PR!

Obviously, thanks to Cirrus-CI platform for providing free budget of GCP VMs! Let's see how it goes in terms of budgets. We can may tune resources usage later depending on the consumption, and see with Cirrus-CI if they are fine with supporting us.

By the way, I'm surprised how fast is the CI! While jobs can be queued for a few minutes, once they are launched, it's really fast! Faster than all other existing CIs (Ubuntu, macOS, Windows): FreeBSD is blazing fast! 🏎️ 🎉

For now, FreeBSD stays at Tier-3, since Tier-2 needs 2 active core developers fixing FreeBSD specific issues, whereas right now, I'm the only one who is maintaining the FreeBSD support. Maybe I can motivate another core dev to take care of FreeBSD ;-)


For now, the CI is only enabled on the main branch and only FreeBSD 13 is tested by Cirrus CI and the newly added buildbot (by @zware). Later, we can consider testing other Python branches and more FreeBSD versions (CURRENT?).

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

I scheduled a new job (by rebasing a PR on main): https://cirrus-ci.com/task/4946742500655104

Result: SUCCESS! All tests pass, great!

Total tests: run=39,512 failures=1 skipped=1,363
Total test files: run=461/461 skipped=24 resource_denied=1 rerun=1

test.pythoninfo (interesting bits):

CC.version: FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
build.Py_DEBUG: Yes (sys.gettotalrefcount() present)

os.cpu_count: 2

os.getgid: 0
os.getgroups: 0
os.getuid: 0
os.login: root


os.supports_effective_ids: ['access']
os.supports_fd: ['chdir', 'chmod', 'chown', 'execve', 'listdir', 'pathconf', 'scandir', 'stat', 'statvfs', 'truncate', 'utime']
os.supports_follow_symlinks: ['access', 'chflags', 'chmod', 'chown', 'link', 'stat', 'utime']

os.uname: posix.uname_result(sysname='FreeBSD', nodename='cirrus-task-4946742500655104', release='13.2-RELEASE', version='FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC', machine='amd64')

platform.platform: FreeBSD-13.2-RELEASE-amd64-64bit-ELF
platform.architecture: 64bit ELF

pwd.getpwuid(0): pwd.struct_passwd(pw_name='root', pw_passwd='', pw_uid=0, pw_gid=0, pw_gecos='Charlie &', pw_dir='/root', pw_shell='/bin/csh')


resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CORE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_DATA: (34359738368, 34359738368)
resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_KQUEUES: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_MEMLOCK: (134217728, 134217728)
resource.RLIMIT_NOFILE: (234963, 234963)
resource.RLIMIT_NPROC: (12186, 12186)
resource.RLIMIT_NPTS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_SBSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_STACK: (536870912, 536870912)
resource.RLIMIT_SWAP: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_VMEM: (9223372036854775807, 9223372036854775807)

resource.pagesize: 4096

ssl.OPENSSL_VERSION: OpenSSL 1.1.1t-freebsd  7 Feb 2023

subprocess._USE_POSIX_SPAWN: False

Tests:

436 tests OK.

10 slowest tests:
- test_tools: 4 min 8 sec
- test_capi: 3 min 18 sec
- test_tokenize: 1 min 57 sec
- test.test_multiprocessing_spawn.test_processes: 1 min 31 sec
- test.test_multiprocessing_fork.test_processes: 1 min 24 sec
- test_math: 1 min 16 sec
- test.test_concurrent_futures.test_wait: 1 min 15 sec
- test.test_multiprocessing_forkserver.test_processes: 1 min 5 sec
- test_regrtest: 58.0 sec
- test_subprocess: 56.6 sec

24 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_ctypes test_dbm_gnu
    test_devpoll test_epoll test_idle test_ioctl test_launcher
    test_perf_profiler test_perfmaps test_readline test_sqlite3
    test_stable_abi_ctypes test_startfile test_tcl test_tkinter
    test_ttk test_ttk_textonly test_turtle test_winconsoleio
    test_winreg test_winsound test_wmi

1 test skipped (resource denied):
    test_zipfile64

1 re-run test:
    test.test_concurrent_futures.test_shutdown

Total duration: 12 min 34 sec
Total tests: run=39,512 failures=1 skipped=1,363
Total test files: run=461/461 skipped=24 resource_denied=1 rerun=1
Result: FAILURE then SUCCESS

@emaste
Copy link
Contributor Author

emaste commented Sep 6, 2023

See also https://discourse.llvm.org/t/rfc-redefine-og-o1-and-add-a-new-level-of-og/72850/22

@sorcio
Copy link
Contributor

sorcio commented Sep 6, 2023

See also https://discourse.llvm.org/t/rfc-redefine-og-o1-and-add-a-new-level-of-og/72850/22

I can confirm from local runs that compiling with clang -O0 makes it debuggable and pass (not skip) almost-all test_gdb tests on both Linux and FreeBSD. I will test with macOS as soon as I get the chance. Probably we can identify some specific optimization pass which is enabled by clang -Og and disable that one, instead of going -O0, if we are careful about different clang versions.

Also the only tests that does skip because of "Unable to read information on python frame" is test_print_after_up and it's a bit fishy because with one more py-up it would work and I'm not sure where the extra frame comes from. (Ironically, I don't have an environment where the test does pass, right now)

If we work on some of this stuff, maybe we can avoid some unnecessary skips. New issue? :)

@vstinner
Copy link
Member

vstinner commented Sep 6, 2023

FYI In Fedora, we decided to build python3-debug with gcc -O0. See the rationale: https://src.fedoraproject.org/rpms/python3.12/blob/rawhide/f/python3.12.spec#_814

If we work on some of this stuff, maybe we can avoid some unnecessary skips. New issue? :)

This PR is closed and I also closed the related issue, so yes please. But I'm not sure of what you are proposing. Python cannot workaround C compiler / debugger limitations. The best that we can do is to skip test_gdb tests when gdb failed to retrieve some arguments because they are <optimized out>.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants