Thank you very much for this report. To help triage can you please:

• confirm which version of codeql you are using (codeql --version)
• Provide information about the version of AOSP you are trying to build.

I note that macOS instructions for AOSP suggest building on a separate volume via a sparse disk-image due to case-sensitivity issues. If you are doing this can I suggest that you monitor the space available on this build volume as the CodeQL database will take a substantial amount of room. The CodeQL database does not need to live on the same volume as the build - so feel free to put it elsewhere (I usually use /tmp).

And finally, what happens if you run:

TEMPORARY_DISABLE_PATH_RESTRICTIONS=true codeql database create /tmp/new-database2 --language=java --command='make services'

(This disables some new restrictions AOSP is adding - see https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools).

Hi Mat, thanks for the quick response!
codeql --version:
codeql --version CodeQL command-line toolchain release 2.3.1.
For example, I'm trying to build android 10 r11.
And I also compiled it successfully without codeql.
This the result of your command, seems like the build succeeds but the finalizing process fails.
Finalizing database at /tmp/new-database2. [build-err] Scanning for files in /Volumes/v0/android-10.0.0_r11... [build-err] /tmp/new-database2: Indexing files in in /Volumes/v0/android-10.0.0_r11... [build-err] /tmp/new-database2: Running in /Volumes/v0/android-10.0.0_r11: [/Users/user/codeql-home/codeql/xml/tools/index-files.sh, /tmp/new-database2/working/files-to-index13111614080267729321.list] [build] [2020-10-20] [build] [2020-10-19 17:56:50] [ERROR] Parsing error in file /Volumes/v0/android-10.0.0_r11/development/samples/training/basic/FragmentBasics/AndroidManifest.xml [build] [2020-10-20] [build] com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs. [build] [2020-10-20] [build] at [row,col {unknown-source}]: [16,5] [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:605) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:467) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.readPIPrimary(BasicStreamReader.java:3917) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2055) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.parse(StAXXmlPopulator.java:307) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.doit(StAXXmlPopulator.java:192) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.extractFile(XmlExtractor.java:84) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.extract(SimpleExtractor.java:156) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.run(SimpleExtractor.java:52) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.main(XmlExtractor.java:29) [build] [2020-10-20] [build] 17:56:50.425 [main] ERROR com.semmle.extractor.xml.StAXXmlPopulator - Parsing error in file /Volumes/v0/android-10.0.0_r11/development/samples/training/basic/FragmentBasics/AndroidManifest.xml [build] [2020-10-20] [build] com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs. [build] [2020-10-20] [build] at [row,col {unknown-source}]: [16,5] [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:605) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:467) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.readPIPrimary(BasicStreamReader.java:3917) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2055) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.parse(StAXXmlPopulator.java:307) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.doit(StAXXmlPopulator.java:192) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.extractFile(XmlExtractor.java:84) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.extract(SimpleExtractor.java:156) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.run(SimpleExtractor.java:52) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.main(XmlExtractor.java:29) [build] [2020-10-20] [build] [2020-10-19 17:56:53] [ERROR] Parsing error in file /Volumes/v0/android-10.0.0_r11/tools/apkzlib/src/test/resources/testData/packaging/AndroidManifest.xml [build] [2020-10-20] [build] com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 start byte 0x9c (at char #5, byte #-1) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:683) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.parse(StAXXmlPopulator.java:307) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.doit(StAXXmlPopulator.java:81) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.extractFile(XmlExtractor.java:84) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.extract(SimpleExtractor.java:156) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.run(SimpleExtractor.java:52) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.main(XmlExtractor.java:29) [build] [2020-10-20] [build] Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0x9c (at char #5, byte #-1) [build] [2020-10-20] [build] at com.ctc.wstx.io.UTF8Reader.reportInvalidInitial(UTF8Reader.java:302) [build] [2020-10-20] [build] at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:188) [build] [2020-10-20] [build] at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) [build] [2020-10-20] [build] at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:967) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.getNext(StreamScanner.java:738) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:1995) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069) [build] [2020-10-20] [build] ... 6 common frames omitted [build] [2020-10-20] [build] 17:56:53.165 [main] ERROR com.semmle.extractor.xml.StAXXmlPopulator - Parsing error in file /Volumes/v0/android-10.0.0_r11/tools/apkzlib/src/test/resources/testData/packaging/AndroidManifest.xml [build] [2020-10-20] [build] com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 start byte 0x9c (at char #5, byte #-1) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:683) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.parse(StAXXmlPopulator.java:307) [build] [2020-10-20] [build] at com.semmle.extractor.xml.StAXXmlPopulator.doit(StAXXmlPopulator.java:81) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.extractFile(XmlExtractor.java:84) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.extract(SimpleExtractor.java:156) [build] [2020-10-20] [build] at com.semmle.cli.SimpleExtractor.run(SimpleExtractor.java:52) [build] [2020-10-20] [build] at com.semmle.cli.XmlExtractor.main(XmlExtractor.java:29) [build] [2020-10-20] [build] Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0x9c (at char #5, byte #-1) [build] [2020-10-20] [build] at com.ctc.wstx.io.UTF8Reader.reportInvalidInitial(UTF8Reader.java:302) [build] [2020-10-20] [build] at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:188) [build] [2020-10-20] [build] at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) [build] [2020-10-20] [build] at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:967) [build] [2020-10-20] [build] at com.ctc.wstx.sr.StreamScanner.getNext(StreamScanner.java:738) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:1995) [build] [2020-10-20] [build] at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069) [build] [2020-10-20] [build] ... 6 common frames omitted

Thank you for the update. I have managed to reproduce the original issue with AOSP master (Disallowed PATH tool) myself and have a fix in progress.

I have not managed to reproduce the second issue yet.

Hi Matt, can I do something to help you reproduce the second issue?

@to-the-batmobile: I'm interested in the following behaviours:

1. What command does codeql database create think it is running? (Near the top of the output from codeql there will be a line:

Running command [.*] in .*.

I'd like to know what that says.

2. What is the disk usage like?

3. What happens if you add -j1 to the options you pass to the make command?

For reference here are the commands I have been using to try and reproduce (cribbed from https://source.android.com/setup):

# Install dependencies - Linux Only
sudo apt-get install git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig

# Set disk and install dependencies (macOS only)
hdiutil create -type SPARSE -fs 'Case-sensitive Journaled HFS+' -size 250g ~/android.dmg.sparseimage
brew install git gnupg2

# Install repo
LOCAL_BIN=$HOME/bin
mkdir -p ${LOCAL_BIN}
PATH=${LOCAL_BIN}:$PATH
curl https://storage.googleapis.com/git-repo-downloads/repo > ${LOCAL_BIN}/repo
chmod a+x ${LOCAL_BIN}/repo

# macOS setup for each session
hdiutil attach ~/android.dmg.sparseimage -mountpoint /Volumes/android;
ulimit -S -n 1024
cd /Volumes/android

# Download source
# Make sure you have >250 GB free....
mkdir aosp
cd aosp

repo init -u https://android.googlesource.com/platform/manifest

# The -j here really is limited by your network bandwidth...
repo sync -c -j4
# Make a cup of tea

# Do a build under CodeQL
source build/envsetup.sh
# For my checkout of AOSP `make` has been made an alias - the `-c` option here is an expansion of that alias.
codeql database create -l java -c "`pwd`/build/soong/soong_ui.bash --make-mode -j1 services" /tmp/aosp-db-java

@matt-gretton-dann
Hi, I tried your last command but I still get the same error, maybe it's a parsing issue that only exists in OSX?

The disk usage is enough, I also compiled it successfully without codeql

(Note: Matt is out of office but may be able to get back to this next week).

@hmakholm Thanks! is there a possibility that one of you guys may be able to take a quick look into this? I bet it's encoding/parsing-related issues... I'm trying to dig into the code and realize by myself but I think you might be able to also assist regarding this issue.