Test failure on macos-14 CI (M1 Mac, MPS torch backend): test_mps_batched_mask_to_box #380
Comments
|
Thanks for all the work on this @GenevieveBuckley! Just two brief comments from my side:
|
Ok, we can try that. The current version of pytorch is 2.2.0 Details (click to expand)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Test
test_mps_batched_mask_to_boxfails on ourmacos-14CI.tests/test_sam_annotator/test_vendored.py::TestVendored.test_mps_batched_mask_to_box=================================== FAILURES =================================== __________________ TestVendored.test_mps_batched_mask_to_box ___________________ self = <test.test_vendored.TestVendored testMethod=test_mps_batched_mask_to_box> @unittest.skipIf(not (torch.backends.mps.is_available() and torch.backends.mps.is_built()), "MPS Pytorch backend is not available") def test_mps_batched_mask_to_box(self): > self._test_batched_mask_to_box(device="mps") test/test_vendored.py:37: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <test.test_vendored.TestVendored testMethod=test_mps_batched_mask_to_box> device = 'mps' def _test_batched_mask_to_box(self, device): from micro_sam._vendored import batched_mask_to_box mask, expected_result = self._get_mask_to_box_data() > mask = torch.as_tensor(mask, dtype=torch.bool, device=device) E RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 7.93 GB). Tried to allocate 256 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure). test/test_vendored.py:21: RuntimeErrorDetails:
macos-14CI runner, an M1 Mac with 7GB of memory available.ubuntu-latest,windows-latest,macos-latest. At the time of writing,macos-latest provides a CPU only mac, the github actionsmacos-12` image.test_vendored.pyfile and see the same error (so it is probably not a memory leak coming from other tests)os.environ["PYTORCH_MPS_HIGH_WATERMARK_RATIO"] = "0.0"as suggested by the error message, did not help.torch.mpsoptions (see the docs here):torch.mps.set_per_process_memory_fraction(2.0)did not help. Two is the maximum possible value allowed.torch.mps.empty_cache()did not help.torch.mps.current_allocated_memory()andtorch.mps.driver_allocated_memory()reported zero bytes of allocated MPS memory before the failing line of the test.The text was updated successfully, but these errors were encountered: