ChAruco regression on tough perspective

[feature-engineer]

System Information

openCV python version: 4.8.0.74 - 4.10.0.84
Operating System: Windows 10 (also on Linux Manjaro)
Python version: 3.12.4

Detailed description

A test code which had worked so far (up to OpenCV 4.7.0.72) fails on OpenCV 4.8 and up (up to and including 4.10).

detector = CharucoDetector(board, params)
corners, ids, _, _ = detector.detectBoard(img)

Fails for sufficiently extreme perspective warp and returns None for both corners and ids instead of lists containing them even though it returns those corners and ids with OpenCV 4.7 (correctly).

Steps to reproduce

def test_not_working():
    img_size = (6560, 4160)
    cam_mtx = np.array([
        [300, 0, img_size[0] / 2],
        [0, 300, img_size[1] / 2],
        [0, 0, 1]], dtype=float)
    dist_coeffs = np.array([[-5.7e-7, 1.7e-8, -2e-7, -3.8e-7, -1.4e-10]])
    board_shape = (17, 11)
    board = cv2.aruco.CharucoBoard(
        board_shape, 8, 4,
        cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_100)
    )
    b_old = board.generateImage(img_size)
    b_old = np.dstack([b_old]*3)  # make and R,G,B image.
    board_w, board_h = 136, 88
    old_pts = np.array([
        [-board_w, -board_h],
        [board_w, -board_h],
        [-board_w, board_h],
        [board_w, board_h]
    ])
    new_pts = np.array([
        old_pts[0] - np.array([2, 2]),
        old_pts[1] - np.array([0, 1]),
        old_pts[2] + np.array([3, -0.4]),
        old_pts[3]
    ])
    h, _ = cv2.findHomography(new_pts, old_pts)
    b_new = cv2.warpPerspective(b_old, h, b_old.shape[::-1][1:])
    params = cv2.aruco.CharucoParameters()
    params.cameraMatrix = cam_mtx
    params.distCoeffs = dist_coeffs
    params.tryRefineMarkers = True
    d = cv2.aruco.CharucoDetector(board, params)
    corners, ids, _, _ = d.detectBoard(b_new)
    assert corners is not None
    assert ids is not None
    assert len(corners) > 4
    assert len(corners) == len(ids)

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)