Skip to content

/ codeql-cli-binaries

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

codeql does not use lgtm.yml file to create database #5

Open
singleghost opened this issue Nov 22, 2019 · 6 comments
Open

codeql does not use lgtm.yml file to create database #5

singleghost opened this issue Nov 22, 2019 · 6 comments
Labels
CLI

Comments

@singleghost
Copy link

@singleghost singleghost commented Nov 22, 2019 •
edited

I create a lgtm.yml file in my project like this.

extraction:
  javascript:
    index:
      include: ""
      exclude:
      - "benchmarks"
      - "test"
      - "testing"
      filters:
      - exclude: "**/*.js"
      - exclude: "**/*.html"
      - exclude: "**/*.ts"
  python:
    index:
      include: "aaaa"
      filters:
        exclude: "**/*.py"
  cpp:
    after_prepare: "mkdir -p ${LGTM_SRC}/build"
    configure:
      command: "(cd ${LGTM_SRC}/build; cmake .. -DCMAKE_C_FLAGS='-no-pie')"
    index:
      build_command: "(cd ${LGTM_SRC}/build; make)"

Then I use codeql to create a database, codeql database create -s ~/softwares/my-codeql-project --language=cpp codeql -j=20, but the output of the codeql shows that it does not use the lgtm.yml as configuration file and build failed. Part of the output:

[2019-11-22 13:22:53] [build] Makefile:132: recipe for target 'all' failed
[2019-11-22 13:22:53] [build-err] + '[' -f build.ninja ']'
[2019-11-22 13:22:53] [build-err] + '[' -d ../_lgtm_build_dir ']'
[2019-11-22 13:22:53] [build-err] + cd ..
[2019-11-22 13:22:53] [build] Semmle autobuild: no supported build system detected.
[2019-11-22 13:22:53] [build-err] + for f in build build.sh
[2019-11-22 13:22:53] [build-err] + '[' -x build ']'
[2019-11-22 13:22:53] [build-err] + '[' -f build ']'
[2019-11-22 13:22:53] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/home/dddong/softwares/codeql-home/codeql/tools/linux64/preload_tracer, /home/dddong/softwares/codeql-home/codeql/cpp/tools/autobuild.sh])
[2019-11-22 13:22:53] [build-err] + for f in build build.sh
[2019-11-22 13:22:53] [build-err] + '[' -x build.sh ']'
[2019-11-22 13:22:53] [build-err] + '[' -f setup.py ']'
[2019-11-22 13:22:53] [build-err] + echo 'Semmle autobuild: no supported build system detected.'
[2019-11-22 13:22:53] [build-err] + exit 1
A fatal error occurred: Exit status 1 from command [/home/dddong/softwares/codeql-home/codeql/cpp/tools/autobuild.sh]
@hmakholm
Copy link
Contributor

@hmakholm hmakholm commented Nov 22, 2019

It is true that the CodeQL CLI does not use lgtm.yml files when creating databases. There are good arguments that it should, even though some of the possible parts of an lgtm.yml (especially configuring how the LGTM builders get hold of dependencies) are firmly outside the scope of the CodeQL CLI.

We're having some internal discussions to figure out what the behavior should be.

For now, you will need to carry out the configuration manually and give the index command as a --command argument to codeql database create.
@singleghost
Copy link
Author

@singleghost singleghost commented Nov 25, 2019

So is it correct to pass multiple --command argument like this? codeql database create --language=cpp codeql -c="cmake . -DCMAKE_C_FLAGS='-no-pie'" -c="make" -j=20 It doesn't generate any errors but I'm not sure if the behaviour inside codeql is correct. When I query the codebase in vscode, the results seem to be incomplete.
@hmakholm
Copy link
Contributor

@hmakholm hmakholm commented Dec 9, 2019 •
edited

Yes, you can pass multiple --command arguments.

However, as long as it's only the last of them that actually invokes the C compiler you can also just run the cmake command manually before you start codeql database create and then only run the actual make invocation as part of database create.
@singleghost
Copy link
Author

@singleghost singleghost commented Dec 10, 2019

Thank you very much!
@hpsan
Copy link

@hpsan hpsan commented Apr 18, 2020

Hey @hmakholm - Could you recommend an alternative for the xml_mode config available in lgtm.yml? I can't quite think of a way to use the --command option to work around it.
@p0
Copy link
Collaborator

@p0 p0 commented Apr 20, 2020

Hey @hspan -- CodeQL 2.1.0 added XML indexing support, which hadn't been reimplemented in the CodeQL CLI before that. What value for xml_mode would you like to emulate? The Java and C# extractors for CodeQL should now behave in the same way as the default LGTM extractors.
@adityasharad adityasharad added the CLI label Jul 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CLI
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@adityasharad @singleghost @hmakholm @p0 @hpsan
You can’t perform that action at this time.