Help: Oops! A fatal internal error occurred (ArrayIndexOutOfBoundsException)

[mindcrunch4u]

Hello!

I am running the cpp-security-extended.qls on a 20GB database

• CodeQL-CLI: version 2.17.4
• command:
  • codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv --threads 32 --ram 102400
  • codeql database analyze codeql_db path/to/1.0.0/codeql-suites/cpp-security-extended.qls --format=csv --output =20240629_result.csv

Both commands (with or without threads and ram options) had the same result.

The output (without --threads option, and --ram set to 102400):

(omitted)
[83/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Best Practices/Likely Errors/CommaBeforeMisleadingIndentation.ql.
[84/91] Loaded /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/MissingCheckScanf.qlx.
[85/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck.ql.
[86/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Critical/SizeCheck2.ql.
[87/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/FailedExtractorInvocations.ql.
[88/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractionWarnings.ql.
[89/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Diagnostics/ExtractedFiles.ql.
[90/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfCode.ql.
[91/91] No need to rerun /home/zywang/Opensource/codeql/codeql/qlpacks/codeql/cpp-queries/1.0.0/Summary/LinesOfUserCode.ql.
Starting evaluation of codeql/cpp-queries/Critical/DoubleFree.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@43a39avp
java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
The RA to evaluate was:

    {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
    return r1

(eventual cause: ArrayIndexOutOfBoundsException "arraycopy: last destination index 1506635235 out of bounds for long[224938403]")
        at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
        at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
        at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
        at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 1506635235 out of bounds for long[224938403]
        at java.base/java.lang.System.arraycopy(Native Method)
        at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:44)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
        at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
        at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
        at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
        at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
        at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
        at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize$0(SimpleHeadStep.java:32)
        at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
        at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
        at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        ... 7 more

Seems like many queries have been completed, but this one triggers an exception.

Any suggestion is appreciated!

[smowton]

Could you check if this is fixed with 2.17.5? There's a possibility the snag has already been fixed.

[mindcrunch4u]

Hello @smowton !

Tried with 2.17.6 (since it's the latest version)

Now I get this error:

Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/PotentiallyExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-416/IteratorToExpiredContainer.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-732/UnsafeDaclSecurityDescriptor.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-497/ExposedSystemData.ql.
Starting evaluation of codeql/cpp-queries/Security/CWE/CWE-807/TaintedCondition.ql.
Oops! A fatal internal error occurred. Details:
com.semmle.util.exception.CatastrophicError: An error occurred while evaluating DataFlowImplCommon::Cached::edgeOrder/1#6d956c05/2@e09049d4
java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
The RA to evaluate was:

    {2} r1 = AGGREGATE `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_range`, `_DataFlowPrivate::DataFlowCallable.totalorder/0#dispred#48dd4ff9__DataFlowImplCommon::Cached::TMkCal__#rank_term` ON In.5, In.6, In.7 WITH RANK<1 ASC2 ASC> OUTPUT , Agg.0, Agg.1
    return r1

(eventual cause: IllegalArgumentException "Maximum array length exceeded: 2147483640")
        at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
        at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:593)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:410)
        at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:93)
        at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:677)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: Maximum array length exceeded: 2147483640
        at com.semmle.inmemory.eval.aggregate.TupleListList.checkSize(TupleListList.java:41)
        at com.semmle.inmemory.eval.aggregate.TupleListList.prepareForAdd(TupleListList.java:32)
        at com.semmle.inmemory.eval.aggregate.TupleListList.addList(TupleListList.java:59)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.commitCurrentRun(AggregateEvaluator.java:463)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator$GroupAndJoin.addTuple(AggregateEvaluator.java:512)
        at com.semmle.inmemory.eval.CancelCheckingSink.addTuple(CancelCheckingSink.java:18)
        at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:84)
        at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
        at com.semmle.inmemory.relations.AbstractRelation.deduplicateMap(AbstractRelation.java:138)
        at com.semmle.inmemory.eval.aggregate.AggregateEvaluator.evaluate(AggregateEvaluator.java:256)
        at com.semmle.inmemory.pipeline.AggregateStep.generateTuples(AggregateStep.java:43)
        at com.semmle.inmemory.pipeline.SimpleHeadStep.lambda$forwardInitialize$0(SimpleHeadStep.java:32)
        at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
        at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:88)
        at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
        at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
        ... 7 more

Any idea how should I proceed?

[smowton]

I'm afraid at least for now you have simply exceeded our maximum problem size, specifically by making us try to compute a ranking larger than the maximum Java array size. For the time being I'm afraid the only solution would be to bound the scope of analysis more tightly, typically by considering more components of your software opaque external libraries (things that are only seen as binary objects or libraries during database creation), and less of them the source code that is subject to analysis (compiled from source during database creation).

As an example, codeql database create -l cpp -c "gcc test.c test2.c -o binary" might be reduced in scope like ... make test2.c into a library test2.a ...; codeql database create -l cpp -c "gcc -c test.c -ltest2 -o binary" -- so that test.c is analysed in full depth, while the functions in test2.c are considered opaque. The downside is that data-flow queries will only find results that do not originate, flow through or terminate inside test2.c, so this consideration should inform what subset of your source you consider important to build from source and therefore analyse in depth.

[mindcrunch4u]

@smowton thank you for the advice!

[jbj]

I'm afraid you've run into scalability problems with the totalorder predicate that was introduced in 2.17.4. If you downgrade to 2.17.3 it should work. We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

[mindcrunch4u]

@jbj thank you! will give 2.17.3 a try, and I wish the relevant people a nice summer vacation!

[aschackmull]

We've implemented most of what's needed to avoid the totalorder predicate in future versions, and the rest of the work will happen as soon as the relevant people are back from summer vacation.

This work is now complete, and should be part of 2.18.1 once that releases (likely around July 26th).

[mindcrunch4u]

@aschackmull thank you! will give 2.18.1 a try once it's out!