Skip to content

ROX-32457: Add more packages to VM index report fixture#18338

Merged
guzalv merged 5 commits intomasterfrom
master-base/gualvare/ROX-32457-add-more-packages-to-vm-indexreport-fixture
Dec 30, 2025
Merged

ROX-32457: Add more packages to VM index report fixture#18338
guzalv merged 5 commits intomasterfrom
master-base/gualvare/ROX-32457-add-more-packages-to-vm-indexreport-fixture

Conversation

@guzalv
Copy link
Contributor

@guzalv guzalv commented Dec 29, 2025

Description

Without this, the VM index report fixture used to generate load contains 507 packages - those installed in a default RHEL 9 VM. When simulating VMs with more packages, we generate reports with duplicates.

This PR adds more unique packages to the list so that we can simulate VMs with more packages installed without needing to repeat packages, therefore making them realistic.

How data was gathered

First I registered the RHEL 9 subscription:

sudo subscription-manager register --org=XXXXXX --activationkey=XXXXXX

Then I wanted to install all possible packages from the default enabled repos in RHEL 9 (rhel-9-for-x86_64-baseos-rpms, rhel-9-for-x86_64-appstream-rpms) plus the ansible automation repo (ansible-automation-platform-2.3-for-rhel-9-x86_64-rpms):

dnf install -y --skip-broken --nobest '*'

However that creates huge transactions that take a long time to process and eventually fail due to conflicts. After several iterations, this vibe-coded script did the trick:

#!/bin/bash

BATCH_SIZE=10
LOG_FILE="/tmp/batch-install.log"
PROGRESS_FILE="/tmp/batch-install-progress.txt"

echo "=== Batch Package Installation Started at $(date) ===" | tee -a "$LOG_FILE"
echo "Initial package count: $(rpm -qa | wc -l)" | tee -a "$LOG_FILE"

# Get list of available packages, excluding kernel* and firmware packages
echo "Building package list..." | tee -a "$LOG_FILE"
sudo dnf repoquery --available --qf "%{name}" | sort -u | \
  grep -v "^kernel" | \
  grep -v "^iwl.*-firmware$" | \
  grep -v "^linux-firmware$" > /tmp/pkg-list.txt

TOTAL_PKGS=$(wc -l < /tmp/pkg-list.txt)
echo "Found $TOTAL_PKGS packages to install" | tee -a "$LOG_FILE"

# Track progress
INSTALLED=0
FAILED=0
BATCH_NUM=0

while IFS= read -r line; do
  BATCH_PKGS+=("$line")

  # When we have a full batch, install it
  if [ ${#BATCH_PKGS[@]} -eq $BATCH_SIZE ]; then
    BATCH_NUM=$((BATCH_NUM + 1))
    echo "" | tee -a "$LOG_FILE"
    echo "[$(date +%H:%M:%S)] Batch $BATCH_NUM: Installing ${BATCH_PKGS[*]}" | tee -a "$LOG_FILE"

    if sudo dnf install -y --skip-broken --nobest "${BATCH_PKGS[@]}" >> "$LOG_FILE" 2>&1; then
      INSTALLED=$((INSTALLED + ${#BATCH_PKGS[@]}))
      echo "✓ Batch $BATCH_NUM succeeded" | tee -a "$LOG_FILE"
    else
      FAILED=$((FAILED + ${#BATCH_PKGS[@]}))
      echo "✗ Batch $BATCH_NUM failed (likely conflicts)" | tee -a "$LOG_FILE"
    fi

    # Update progress
    CURRENT_COUNT=$(rpm -qa | wc -l)
    echo "Progress: Batch $BATCH_NUM/$((TOTAL_PKGS / BATCH_SIZE)) | Current packages: $CURRENT_COUNT" | tee "$PROGRESS_FILE" | tee -a "$LOG_FILE"

    # Clear the batch
    BATCH_PKGS=()
  fi
done < /tmp/pkg-list.txt

# Install remaining packages (less than BATCH_SIZE)
if [ ${#BATCH_PKGS[@]} -gt 0 ]; then
  BATCH_NUM=$((BATCH_NUM + 1))
  echo "" | tee -a "$LOG_FILE"
  echo "[$(date +%H:%M:%S)] Final batch $BATCH_NUM: Installing ${BATCH_PKGS[*]}" | tee -a "$LOG_FILE"

  if sudo dnf install -y --skip-broken --nobest "${BATCH_PKGS[@]}" >> "$LOG_FILE" 2>&1; then
    INSTALLED=$((INSTALLED + ${#BATCH_PKGS[@]}))
    echo "✓ Final batch succeeded" | tee -a "$LOG_FILE"
  else
    FAILED=$((FAILED + ${#BATCH_PKGS[@]}))
    echo "✗ Final batch failed" | tee -a "$LOG_FILE"
  fi
fi

echo "" | tee -a "$LOG_FILE"
echo "=== Installation Complete at $(date) ===" | tee -a "$LOG_FILE"
echo "Final package count: $(rpm -qa | wc -l)" | tee -a "$LOG_FILE"
echo "Attempted: $TOTAL_PKGS packages in $BATCH_NUM batches" | tee -a "$LOG_FILE"

This took several hours and resulted in ~5400 packages being installed.

Note: a large disk is also needed, the lines with trailing comments are the most critical to set it up:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
...
spec:
  dataVolumeTemplates:
  - metadata:
      name: rhel9-large-disk
    spec:
      source:
        registry:
          pullMethod: node
          url: docker://registry.redhat.io/rhel9/rhel-guest-image:latest
      storage:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 200Gi # Enough space to install packages
        volumeMode: Filesystem
  ...
  template:
    spec:
      domain:
        devices:
          autoattachVSOCK: true
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: rootdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          ...
      volumes:
      - dataVolume:
          name: rhel9-large-disk
        name: rootdisk
      - cloudInitNoCloud:
          userData: |
            growpart:
              mode: auto
              devices: ['/']  # Auto-expand partition
            resize_rootfs: true  # Auto-resize filesystem to use full disk
        ...
        name: cloudinitdisk

After packages were installed, I ran roxagent to generate an index report:

$ sudo ./roxagent --verbose > report.json

And finally I copied it to my local machine and extracted the data from the report with a jq filter (LLM-generated), which I then pasted into the fixture file:

cat report.json | jq -r '
      .indexV4.contents as $c |
      $c.packages | to_entries | map(
        .value as $pkg | .key as $id |
        {
          name: $pkg.name,
          version: $pkg.version,
          repo: (($c.environments[$id].environments[0].repositoryIds // []) | map(select(startswith("rhel-") or startswith("ansible-"))) | .[-1] // "unknown")
        }
      ) |
      sort_by(.name) |
      map("\t{\"\(.name)\", \"\(.version)\", \"\(.repo)\"},") |
      .[]
    '

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

This PR adds test data, used by tests.

How I validated my change

I ran fake workloads and fake vsock load and verified that the generated reports contain real packages without repetition

@guzalv
Copy link
Contributor Author

guzalv commented Dec 29, 2025

This change is part of the following stack:

Change managed by git-spice.

@openshift-ci
Copy link

openshift-ci bot commented Dec 29, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Dec 29, 2025

Images are ready for the commit at f29fd9b.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.10.x-680-gf29fd9b71e.

Real data taken from a VM with 5465 packages
This repo was added for some reason to my test VM, let's take advantage
of this extra data rather than throwing it away
@guzalv guzalv force-pushed the master-base/gualvare/ROX-32457-add-more-packages-to-vm-indexreport-fixture branch from c727c27 to 81a571d Compare December 29, 2025 14:29
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @github-actions[bot], your pull request is larger than the review limit of 150000 diff characters

@codecov
Copy link

codecov bot commented Dec 29, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.87%. Comparing base (1a40afd) to head (f29fd9b).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #18338   +/-   ##
=======================================
  Coverage   48.87%   48.87%           
=======================================
  Files        2623     2623           
  Lines      198099   198099           
=======================================
+ Hits        96821    96825    +4     
+ Misses      93887    93883    -4     
  Partials     7391     7391           
Flag Coverage Δ
go-unit-tests 48.87% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@guzalv guzalv marked this pull request as ready for review December 29, 2025 19:19
@guzalv guzalv requested a review from a team as a code owner December 29, 2025 19:19
@guzalv guzalv merged commit 52218c8 into master Dec 30, 2025
96 checks passed
@guzalv guzalv deleted the master-base/gualvare/ROX-32457-add-more-packages-to-vm-indexreport-fixture branch December 30, 2025 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants