GraalPy doesn't raise OverflowError when CPython does

[kgonia]

I was trying to build cffi wheel with GraalPy, but tests are failing on GraalPy because of an inconsistency with the CPython implementation. The code below raises an OverflowError with CPython but doesn't with GraalPy.

import pytest
import sys

from _cffi_backend import new_primitive_type, new_pointer_type, new_array_type


def test_graalpy_overflow_behavior_on_large_array_size():
    """
    This test asserts that creating an array type with a size larger than
    sys.maxsize raises an OverflowError.

    Expected behavior:
    - CPython: Should raise OverflowError. Test will PASS.
    - GraalPy: Should NOT raise OverflowError. Test will FAIL.
    """
    p = new_primitive_type("int")
    
    pytest.raises(OverflowError, new_array_type, new_pointer_type(p), sys.maxsize + 1)

[kgonia]

After more digging, I understand this comes from the difference in sys.maxsize between CPython and GraalPy, which stems from VM limitations.

In Python, sys.maxsize matches the C limit, so trying to allocate anything bigger correctly throws an OverflowError. However, GraalPy has a sys.maxsize limit lower than the actual C limit. This causes sys.maxsize + 1 to be a valid size for C allocations on GraalPy, so no OverflowError is raised, causing the test failure.

Is this on your radar? Is there anything that can be done, perhaps with the Foreign Function & Memory Access API?

[msimacek]

Thanks for the report. It's been on our TODO list for a while, but we didn't consider it a high priority issue, since this mostly comes up in tests for edge cases. Indeed, the implementation could use native memory and we already do have such a storage for primitive types hidden behind an experimental flag. But for lists of general objects, you cannot just get a raw pointer to a Java object that wouldn't later get moved by the GC, We'd need to allocate a native stub header for the every object like we do when passing Java objects to C and do refcounting and reference queues. It's doable, but the overhead might make it impractical for anybody who actually wants to use large data structures.

Note that you can get the C pointer size with struct.calcsize("P") if you want to compute your own C maxsize.