GH-22232: [C++][Python] Introduce optional default_column_type parameter

[vladborovtsov]

Rationale for this change

Add an optional default_column_type parameter to the CSV reading API (C++ and Python) to provide a fallback type when per-column types aren’t specified, improving schema consistency and complementing the existing column_types logic.

What changes are included in this PR?

• c++: new convert option "default_column_type" to augment logic around column_types parameter
• 3 reader tests: DefaultColumnTypePartialDefault, DefaultColumnTypeAllStringsWithHeader, DefaultColumnTypeAllStringsNoHeader). The last two tests are inspired by PARTIAL FIX: Improve leading zeros preservation with dtype=str for dict-based dtypes pandas-dev/pandas#62242 and BUG: pyarrow read_csv engine stripping leading zeros with dtype=str pandas-dev/pandas#57666
• python: corresponding changes to make cpp change consumable from python
• python: extended test_convert_options test - include. default_column_type
• python: added new test "test_default_column_type" which tests how the field impacts schema; also test implicitly verifies leading zero preservation
• relevant documentation update for python component;

Are these changes tested?

Yes. Existing and new tests are passing.

C++:

> [==========] Running 3 tests from 1 test suite.
> [----------] Global test environment set-up.
> [----------] 3 tests from ReaderTests
> [ RUN      ] ReaderTests.DefaultColumnTypePartialDefault
> [       OK ] ReaderTests.DefaultColumnTypePartialDefault (3 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsWithHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsWithHeader (0 ms)
> [ RUN      ] ReaderTests.DefaultColumnTypeAllStringsNoHeader
> [       OK ] ReaderTests.DefaultColumnTypeAllStringsNoHeader (0 ms)
> [----------] 3 tests from ReaderTests (4 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 3 tests from 1 test suite ran. (4 ms total)
> [  PASSED  ] 3 tests.

All:

> [==========] 264 tests from 46 test suites ran. (452 ms total)
> [  PASSED  ] 264 tests.

pyarrow:
New tests are passing.

Are there any user-facing changes?

I believe this change is backward compatible. Parameter is optional and its default value doesn't change the existing behavior; All the existing rests are passing.

Maybe relevant: #22232

Relates to #47502

• GitHub Issue: [C++] CSV reader: add a default column type (or sentinel mapping) to avoid per-column enumeration #47502

• GitHub Issue: [C++] CSV reader: Ability to not infer column types. #22232

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[vladborovtsov]

@github-actions crossbow submit preview-docs

[github-actions]

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/18062577036

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #47502 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

Thank you @vladborovtsov for the contribution.
I will add info about the proposed solution in the original issue (#22232) so I can see opinions from C++ devs on the proposed solution.

[github-actions]

⚠️ GitHub issue #22232 has been automatically assigned in GitHub to PR creator.

[vladborovtsov]

Hi @AlenkaF
I'm happy to continue the labour and discussion to get that merged.
As for AI, it wasn't used much, although I tried :) With such huge codebase the generation quality is quite low.

[AlenkaF]

Happy to see a response!
All good, it is totally OK to use gen AI wisely ;)

I will wait for an opinion from a C++ dev and in the meantime try to look at the Python part.

[vladborovtsov]

Hi @AlenkaF
Any feedback?

[AlenkaF]

The Python part looks good to me. I only added a comment for the docstring examples.

@rok @raulcd do you have any opinions on the C++ approach to handle this use case?

[rok]

I only read the c++ part. I think the way option is defined and checked is good.

A nit: I'd move c++ tests into arrow/cpp/src/arrow/csv/column_builder_test.cc and use available machinery there. See how CheckInferred is used here:

arrow/cpp/src/arrow/csv/column_builder_test.cc

Lines 441 to 457 in ad9279e

	TEST_F(InferringColumnBuilderTest, MultipleChunkTimestamp) {
	auto options = ConvertOptions::Defaults();
	auto tg = TaskGroup::MakeSerial();
	std::shared_ptr<ChunkedArray> expected;
	ChunkedArrayFromVector<TimestampType>(timestamp(TimeUnit::SECOND),
	{{false}, {true}, {true}},
	{{0}, {0}, {1542129070}}, &expected);
	CheckInferred(tg, {{""}, {"1970-01-01"}, {"2018-11-13 17:11:10"}}, options, expected);
	options.timestamp_parsers.push_back(TimestampParser::MakeStrptime("%Y/%m/%d"));
	tg = TaskGroup::MakeSerial();
	ChunkedArrayFromVector<TimestampType>(timestamp(TimeUnit::SECOND),
	{{false}, {true}, {true}},
	{{0}, {0}, {1542067200}}, &expected);
	CheckInferred(tg, {{""}, {"1970/01/01"}, {"2018/11/13"}}, options, expected);
	}

Edit: column_builder_test.cc is checking behavior only on single columns, perhaps keep a couple multi-column tests in the current location to test for that.

[vladborovtsov]

There is not too much to add into column_builder_test.cc: the introduced default_column_type affects only reader. And reader using typed columns when default specified, for details see cpp/src/arrow/csv/reader.cc.

I’ve added a guard test to ensure that my changes don’t override the type inference logic. Not really sure if it it makes sense to have them there.

As for the reader tests, I’ve tried to keep them representative while minimizing the amount of lines with recent changes.

[AlenkaF]

Gentle ping @rok, maybe @raulcd?

[raulcd]

I am unsure about the approach as I am not an expert on the C++ side of the CSV reader. I'll investigate a little. In the meantime, could you fix the related lint failure and rebase main to fix some of the CI issues?