It would be great if we could cache some of the heavy download (Android SDK and NDK and CrystaX NDK) in Travis. That should not only speed up build but also make them more reliable.

We could either try to bind a Docker image directory with the Travis file system and cache it. Or directly cache the entire Docker build. I've played with caching a Docker container, but this is not built-in in Travis and needs to be done in a kind of hacky way which is painful.
So we may take a look at the other solution i.e. mounting a download directory and try to cache it first.