Missing source files in CodeQL database creation

[northsuperman]

After creating multiple CodeQL databases, I've noticed that many times some of the source files are not in the the database. Therefore, I can't run my queries on the complete source.

I thought it could be a result of compiling a project that produces multiple executables, but this happens on source code the result in a single executable too.

Does anyone have an idea why this could happen?

The command I'm using to create the db is:

codeql database create *db_name* -l=cpp -c  "build.sh"

where "build.sh" is a file with several CC commands.

[intrigus-lgtm]

Is the build using some sort of caching?
Codeql only sees the compiler invocation for the individual source files.
If a file does not get compiled, because it is cached, it will not be detected by codeql.
So I'd suggest cleaning your build dir and trying again.

Could you be a little bit more specific on which files are missing?
For example, if you have source files that are only compiled on armv8 and you compile for x86-64, then the armv8 files will not be seen by codeql (because they are not compiled).

Hope this helps.

[northsuperman]

Thank you very much for replying!

Caching:
I have made sure that no caching occurs inside my container, and that none of the prior build file exists.

Architecture:
Could you explain what you mean by:

...if you have source files that are only compiled on armv8 and you compile for x86-64...

I will try to find some pattern in the missing files and if I do I will share my findings.

Your help is much appreciated.

[northsuperman]

I can't see anything unique about the missing files.

I have tried to compile through SCons both with some proprietary version of gcc and with clang. Both have files missing in the db.

[intrigus-lgtm]

Is the project you're building public?
Could you share the build.sh file you are using to build the project? Is the project using CCACHE?
What output do you get when running codeql --version?
And finally, how many source files are you compiling and which operating system/environment do you use?