We have added the the getting started scripts a while ago and they are already conceptualized as tests. The return 0 if they worked, or any other return code if they failed. We would like to run them in Jenkins before every release, to save us time in the release process. Currently, the tests have to be run manually with is time consuming and error prone.

Testing should be possible for nightly as well as release branches

Implementation

Operators

In the operator, every test script should have a name conforming to test_*.sh. (i'd say the script cannot make any assumptions about where it is running)

Jenkins

Most of the work will need to be done on the Jenkins side. Jenkins should:

• Look for all the test_*.sh scripts in the repo
• start a cluster for each one. No operators need to be installed
• run the script and capture the output

We would add another "Tab" in jenkins (like nightly, custom, etc.) for the docs tests.

Steps

As usual, we'll

• Make an OPA test branch with the test_*.sh scripts (done: spike/docs-testing)
• try it with OPA in Jenkins
• if everything works, roll it out to all operators

Acceptance criteria

• Jenkins has a new tab for docs tests
• the tab contains all operators as well as the main documentation itself
• in each test a branch can be selected

Tests working

• Main docs CI docs testing documentation#367 ⚠️ Follow-Up: Get the LDAP test/docs running in the CI/Cloud documentation#378
• Airflow [Merged by Bors] - Fix/docs test nightly to dev airflow-operator#249 🟢
• Druid [Merged by Bors] - switched to dev and added some timeouts druid-operator#412 🟢
• Hbase [Merged by Bors] - switched to dev and added some timeouts hbase-operator#328 🟢
• HDFS [Merged by Bors] - Fix/docs test nightly to dev hdfs-operator#331 🟢
• Hive [Merged by Bors] - Fix/docs test nightly to dev hive-operator#320 ❗ Not working yet: Get the getting started guide script working in CI hive-operator#329
• Kafka [Merged by Bors] - Renamed getting-started.sh -> getting_started.sh kafka-operator#561 ⚠️ (Outside connection currently not possible, see Kafka brokers are not accessible from outside of the cluster because they advertise the InternalIP only kafka-operator#569)
• NiFi [Merged by Bors] - Fix/docs test nightly to dev nifi-operator#439 ⚠️ (connection problems in T2: Docs test doesn't run in T2 created cluster nifi-operator#442)
• OPA [Merged by Bors] - test fixes opa-operator#426 🟢
• Spark [Merged by Bors] - Fix/docs test nightly to dev spark-k8s-operator#220 🟢
• Superset [Merged by Bors] - switched to dev and added some timeouts superset-operator#338 🟢
• Trino [Merged by Bors] - Fix/docs test nightly to dev trino-operator#397 🟢
• ZooKeeper [Merged by Bors] - Fix/docs test nightly to dev zookeeper-operator#655 🟢

Not all tests are working right now, but the basic infrastructure is in place. I made tickets for the tests that do not work:

Follow-Up tickets

• Docs test doesn't run in T2 created cluster nifi-operator#442
• Kafka brokers are not accessible from outside of the cluster because they advertise the InternalIP only kafka-operator#569
• Get the LDAP test/docs running in the CI/Cloud documentation#378
• Get the getting started guide script working in CI hive-operator#329