branch-4.0: [feature](file-cache) Disable cache writes after remote scan threshold

[bobhan1]

What problem does this PR solve?

Issue Number: None

Related PR: #65058

Problem Summary:

This backports #65058 to branch-4.0.

Remote scans can read a large amount of data from object storage. Before this change, a query that had already exceeded its useful cache-write budget could still keep writing later remote cache misses into file cache, increasing cache churn and IO overhead.

This PR adds the session variable file_cache_query_limit_bytes to control remote-scan read-through file-cache writes per query on each BE:

• < 0: disabled. This is the default.
• = 0: do not write remote-scan cache misses into local file cache from the start of the query.
• > 0: allow remote-scan cache writes until the next cache block would exceed the byte threshold. Once the threshold is exceeded, later cache misses continue reading from remote storage but skip additional local file-cache writes.

The limiter is query-wide within a BE and is propagated through scan, segment loading, preload, segment footer, column-reader metadata, and inverted-index metadata paths. The BE dynamic config file_cache_query_limit_segment_meta, default false, optionally includes segment footer and segment metadata cache writes in the same limit. Profile counters expose the threshold, whether remote-only-on-miss was triggered, and specialized cache-write bytes and timers.

The backport maps master storage-path and API changes to the corresponding branch-4.0 olap/vec code paths. Master-only modules that do not exist on branch-4.0 are not introduced.

Release note

Add file_cache_query_limit_bytes for query-level remote-scan file-cache write limiting, and add optional segment metadata accounting with file_cache_query_limit_segment_meta.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  Validation:

  • git diff --check
  • ./build.sh --be --fe --cloud -j100
  • ./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.ShortCircuitQueryContextTest (8 tests passed)
  • Related BE ASAN unit tests for file-cache profiles and limiter behavior, cached remote reads, inverted-index IO context, scanner, primary-key index, segment footer, external column metadata, and column-reader cache (51 tests across 8 suites passed)
  • env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d cloud_p0/cache/remote_scan_no_write_file_cache -g docker -runMode=cloud -dockerSuiteParallel 1 (2 suites passed)

• Behavior changed:

  • No.
  • Yes. Remote scans can stop writing local file cache after file_cache_query_limit_bytes is reached while continuing to read misses from remote storage. Optional segment metadata accounting can include segment footer and metadata cache writes in the same query-wide threshold.

• Does this need documentation?

  • No.
  • Yes. The new user-visible knobs should be documented in a follow-up docs PR.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[bobhan1]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 76.19% (16/21) 🎉
Increment coverage report
Complete coverage report

[bobhan1]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 76.19% (16/21) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 95.36% (411/431) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.51% (25637/35849)
Line Coverage	54.49% (272529/500184)
Region Coverage	52.06% (225920/433968)
Branch Coverage	53.43% (97167/181853)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 2.44% (6/246) 🎉
Increment coverage report
Complete coverage report