Open
Conversation
default is 2 minutes; we're exceeding 85% disk and so GC was removing prefetched images
Contributor
Author
|
/test gke-latest-qa-e2e-tests |
Contributor
|
Images are ready for the commit at 9fb8f60. To use with deploy scripts, first |
node-accessible is disk minus OS/etc (~42GB gke-latest) GC hits at 85% and evicts images not used in 2 minutes. Neither the 85% nor the 2 minutes can be increased.
Contributor
Author
|
/test gke-latest-qa-e2e-tests |
This reverts commit 5aab5eb.
Contributor
Author
|
/test gke-latest-qa-e2e-tests |
The prefetcher pulls images by tag via the CRI API, which stores them indexed by tag name. When tests reference the image as tag@sha256:<manifest-list-digest>, containerd 2.x cannot resolve it with imagePullPolicy: Never because the manifest list digest is not indexed as a named image by the CRI pull-by-tag path. This caused ErrImageNeverPull on every node regardless of disk size, as the image was present on disk but not findable by digest. Images referenced by tag only (busybox-1-33-1, nginx-1-12-1, etc.) worked fine with the same Never pull policy. Remove the @sha256: digest from TEST_IMAGE so it matches how the prefetcher stores the image. Keep TEST_IMAGE_SHA available for API queries that need the digest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/test gke-latest-qa-e2e-tests |
|
@davdhacs: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
default image garbage-collection expiration is 2 minutes (and cannot be increased). we're exceeding 85% disk and so GC was removing prefetched images