[Python] Support DictionaryArray -> numpy/pandas conversion when the decoded data exceeds the 32-bit offset limit

[pearu]

Describe the enhancement requested

Once #50840 is fixed (#50841), converting a dictionary array whose decoded form exceeds INT32_MAX bytes raises cleanly instead of crashing:

import numpy as np, pyarrow as pa

arr = pa.DictionaryArray.from_arrays(
    np.zeros(50_000_000, dtype=np.int16),
    pa.array(["a" * 50]),
)
np.asarray(arr)
# pyarrow.lib.ArrowInvalid: Take operation overflowed binary array capacity

That is a strict improvement over a segfault, but the conversion still fails on data that is entirely representable in the output. The result of np.asarray here is a numpy object array of Python str — a format with no 2 GiB limit. The failure comes purely from an intermediate step.

Why it fails

Array.to_numpy(zero_copy_only=False) sets decode_dictionaries=True. In arrow_to_pandas.cc, ConvertChunkedArrayToPandas() then decodes by casting to the dictionary's value type:

const auto& dense_type =
    checked_cast<const DictionaryType&>(*arr->type()).value_type();   // string, int32 offsets
RETURN_NOT_OK(DecodeDictionaries(options.pool, dense_type, &arr));

So a 50M × 50-byte decode has to fit in a 32-bit-offset string array. It cannot — hence the error. But that dense array is a pure implementation artifact; nothing downstream needs it to be a string array specifically.

Approach (a): widen the intermediate type

Use large_string/large_binary as the decode target instead of string/binary.

Pros

• Very small, local change — essentially picking a different dense_type
• Low risk; reuses the existing cast/take machinery unchanged
• Invisible to users: the final output is an object array either way
• Fixes the to_numpy and to_pandas paths together

Cons

• Still materializes the entire dense array (2.5 GB in the example) purely as scratch
• Still creates one Python object per element — 50M separate str objects for 50M indices — so peak memory stays very large even though the input holds a single distinct value
• Slightly increases memory for the common, non-overflowing case (8-byte offsets), unless a size pre-check is added to decide when to promote
• Treats the symptom rather than the redundancy

Approach (b): skip the dense intermediate entirely

For the object-output path, convert each dictionary value to a PyObject once, then walk the indices and Py_INCREF the corresponding object into the output array.

Pros

• No dense intermediate at all — the 2.5 GB scratch buffer disappears
• For the example, one Python string is allocated instead of 50M; memory and time drop by orders of magnitude
• Removes the size ceiling entirely rather than raising it
• Also speeds up the ordinary, non-overflowing dictionary → object conversion, which is the common categorical/pandas path
• Output is semantically identical: object arrays hold references, and Python strings are immutable, so sharing is unobservable

Cons

• Larger change: needs a dedicated dictionary branch in the object-writer instead of reusing DecodeDictionaries
• Null indices and index bounds must be handled explicitly rather than inherited from the cast
• Only covers the object-output path; any path that genuinely needs a dense Arrow array (e.g. strings_to_categorical) would still want (a)
• Callers relying on distinct object identity per element would see shared references — not a documented guarantee, and already untrue for interned strings, but worth noting

Recommendation

(b), because it addresses the actual redundancy rather than raising a ceiling. Dictionary-encoded data is used precisely when values repeat, so materializing one Python object per index rather than per dictionary entry is wasted work in every case — the overflow is just where it becomes fatal. It also improves the common path, not only the pathological one.

(a) remains worth keeping in reserve for any remaining code path that must produce a dense array; the two are complementary rather than exclusive.

I'm happy to implement (b), pending agreement on the direction.

Component(s)

Python

---

🤖 Drafted by Claude Code (an AI agent) and reviewed & approved by pearu.