Skip to content

Remove kolla entrypoint pattern, harden SecurityContext - #427

Open
stuggi wants to merge 1 commit into
openstack-k8s-operators:mainfrom
stuggi:remove-kolla
Open

Remove kolla entrypoint pattern, harden SecurityContext#427
stuggi wants to merge 1 commit into
openstack-k8s-operators:mainfrom
stuggi:remove-kolla

Conversation

@stuggi

@stuggi stuggi commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Replaces the kolla_start/kolla_set_configs/config.json staging pattern with direct volume mounts to final config paths and direct service commands for watcher-api, watcher-applier, watcher-decision-engine, db-sync, and db-purge.

Key changes:

  • Replace kolla_start with direct commands per workload (httpd -DFOREGROUND, watcher-applier/decision-engine --config-dir, watcher-db-manage upgrade, bash dbpurge.sh)
  • Remove KOLLA_CONFIG_STRATEGY env vars and 5 config.json templates
  • Mount configs via SubPath to /etc/watcher/watcher.conf.d/, /etc/httpd/, /etc/my.cnf
  • Use pod.RestrictiveSecurityContext/RestrictivePodSecurityContext from lib-common (serviceuser.WatcherUID) on all workloads; db-sync and db-purge were missing pod-level SecurityContext -- added
  • AutomountServiceAccountToken: false on all 5 workloads
  • SCC: clean anyuid -> nonroot-v2 swap
  • httpd.conf: User/Group apache -> watcher (WSGIDaemonProcess user=watcher evidence proves dedicated system user exists)
  • ApacheGID as supplemental group for RPM-shipped conf.d files
  • DefaultMode 0440 on all config Secrets; deduplicated triplicated config0440AccessMode vars to single package-level declaration
  • Removed dead WatcherUserID constant (migrated to serviceuser.WatcherUID)
  • Removed local GetLogVolume/GetLogVolumeMount wrapper functions, replaced all call sites with direct volume.WritableDirVolume/ WritableDirVolumeMount calls
  • Replaced local GetRunHttpdVolume/Mount, GetVarLogHttpdVolume/Mount, GetConfigSecretVolumes with lib-common volume helpers
  • Fix watcher-api probe assignment: startup probe was on the log sidecar (tail -F) instead of the httpd container, and the log sidecar had HTTP probes testing httpd's port instead of its own process. Moved startup probe to httpd, removed all probes from log sidecar.
  • Fix db-sync: removed GetVolumeMounts/GetVolumes calls that added conflicting SubPath mounts from the shared config-data volume into /etc/watcher/watcher.conf.d/ (directory doesn't exist in image); db-sync uses its own db-sync-config-data volume with whole-directory mount instead
  • Fix db-purge: same pattern -- replaced GetVolumeMounts/GetVolumes with inline whole-directory mount; added my.cnf SubPath mount
  • Also fixes: TLS cert/key and memcached mTLS staging-path bugs; missing /run/httpd and /var/log/httpd emptyDirs; db-purge script invoked via bash (non-root can't execute 0740 Secret-mounted file)
  • Update functional and kuttl tests

Jira: OSPRH-33504
Jira: OSPRH-33503

Depends-On: openstack-k8s-operators/lib-common#728

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign stuggi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5317ecd3fc7748a9816055ef3b492b78

✔️ openstack-meta-content-provider-master SUCCESS in 3h 26m 01s
✔️ watcher-operator-validation-master SUCCESS in 2h 21m 04s
✔️ openstack-meta-content-provider-epoxy SUCCESS in 2h 58m 34s
✔️ watcher-operator-validation-epoxy SUCCESS in 2h 02m 30s
✔️ watcher-operator-validation-epoxy-ocp4-18 SUCCESS in 2h 03m 01s
✔️ noop SUCCESS in 0s
watcher-operator-kuttl FAILURE in 1h 04m 28s

Comment thread internal/watcher/dbpurgecronjob.go Outdated
// that only requires read access (satisfied by FSGroup=WatcherUID),
// sidestepping the question of whether the non-root, non-owning
// container has the file's execute bit at all.
args := []string{GetScriptVolumeMount().MountPath + "/dbpurge.sh"}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this is just

set -x
export PURGE_AGE=${PURGE_AGE:?"Please specify PURGE_AGE variable."}

echo y | watcher-db-manage --config-dir /etc/watcher/watcher.conf.d/ --debug purge -d ${PURGE_AGE}

we only used a script to be abel to use it form kolla

so the alternitive would be ot just make this

echo y | watcher-db-manage --config-dir /etc/watcher/watcher.conf.d/ --debug purge -d ${PURGE_AGE}

i dont thinke the echo y is even needed since i belive the commadn does not prompt for conformation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call, I had not looked into the script and was just focusing for now to remove kolla calls and just use what was used inside, so just moved the script. I'll update it to remove it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it really asks for confirmation:

sh-5.1$ watcher-db-manage --config-dir /etc/watcher/watcher.conf.d/ --debug purge -d 5
/usr/lib/python3.9/site-packages/watcher/common/scheduling.py:43: DeprecationWarning: Using class 'GreenThreadPoolExecutor' (either directly or via inheritance) is deprecated: Eventlet support is deprecated. Please migrate your code and stop using Green executor.
  pool = futurist.GreenThreadPoolExecutor(int(max_workers))
+-------+------------+-----------------+--------+--------------+---------+-------+
| Goals | Strategies | Audit Templates | Audits | Action Plans | Actions | Total |
+-------+------------+-----------------+--------+--------------+---------+-------+
|   0   |     0      |        0        |   0    |      0       |    0    |   0   |
+-------+------------+-----------------+--------+--------------+---------+-------+
There are 0 objects set for deletion. Continue? [y/N]

MountPath: "/etc/watcher/watcher.conf.d/00-default.conf",
SubPath: DefaultsConfigFileName,
ReadOnly: true,
},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stuggi
stuggi force-pushed the remove-kolla branch 6 times, most recently from 05ff77a to bf700c2 Compare August 12, 2026 16:01
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/427e7852892b455bb119b2f673cd4df1

openstack-meta-content-provider-master FAILURE in 5m 23s
⚠️ watcher-operator-validation-master SKIPPED Skipped due to failed job openstack-meta-content-provider-master
openstack-meta-content-provider-epoxy FAILURE in 6m 05s
⚠️ watcher-operator-validation-epoxy SKIPPED Skipped due to failed job openstack-meta-content-provider-epoxy
⚠️ watcher-operator-validation-epoxy-ocp4-18 SKIPPED Skipped due to failed job openstack-meta-content-provider-epoxy
✔️ noop SUCCESS in 0s
⚠️ watcher-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider-master

@stuggi

stuggi commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c2d72a07f264461895f1c6e3e3057465

✔️ openstack-meta-content-provider-master SUCCESS in 1h 09m 46s
watcher-operator-validation-master POST_FAILURE in 27m 48s
openstack-meta-content-provider-epoxy FAILURE in 16m 16s
⚠️ watcher-operator-validation-epoxy SKIPPED Skipped due to failed job openstack-meta-content-provider-epoxy
⚠️ watcher-operator-validation-epoxy-ocp4-18 SKIPPED Skipped due to failed job openstack-meta-content-provider-epoxy
✔️ noop SUCCESS in 0s
watcher-operator-kuttl RETRY_LIMIT in 24m 41s

@stuggi

stuggi commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/206923f6927f491ba89c199f44253d66

openstack-meta-content-provider-master FAILURE in 5m 27s
⚠️ watcher-operator-validation-master SKIPPED Skipped due to failed job openstack-meta-content-provider-master
✔️ openstack-meta-content-provider-epoxy SUCCESS in 3h 15m 42s
watcher-operator-validation-epoxy FAILURE in 1h 32m 32s
✔️ watcher-operator-validation-epoxy-ocp4-18 SUCCESS in 2h 24m 09s
✔️ noop SUCCESS in 0s
⚠️ watcher-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider-master

@stuggi

stuggi commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@stuggi
stuggi requested a review from amoralej August 13, 2026 11:49
@centosinfra-prod-github-app

Copy link
Copy Markdown

This change depends on a change that failed to merge.

Change openstack-k8s-operators/lib-common#728 is needed.

@stuggi

stuggi commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

recheck

Replaces the kolla_start/kolla_set_configs/config.json staging pattern
with direct volume mounts to final config paths and direct service
commands for watcher-api, watcher-applier, watcher-decision-engine,
db-sync, and db-purge.

Key changes:
- Replace kolla_start with direct commands per workload
  (httpd -DFOREGROUND, watcher-applier/decision-engine --config-dir,
  watcher-db-manage upgrade, watcher-db-manage purge)
- Remove KOLLA_CONFIG_STRATEGY env vars and 5 config.json templates
- Mount configs via SubPath to /etc/watcher/watcher.conf.d/,
  /etc/httpd/, /etc/my.cnf
- Use pod.RestrictiveSecurityContext/RestrictivePodSecurityContext
  from lib-common (serviceuser.WatcherUID) on all workloads; db-sync
  and db-purge were missing pod-level SecurityContext -- added
- AutomountServiceAccountToken: false on all 5 workloads
- SCC: clean anyuid -> nonroot-v2 swap
- httpd.conf: User/Group apache -> watcher (WSGIDaemonProcess
  user=watcher evidence proves dedicated system user exists)
- ApacheGID as supplemental group for RPM-shipped conf.d files
- DefaultMode 0440 on all config Secrets; deduplicated triplicated
  config0440AccessMode vars to single package-level declaration
- Removed dead WatcherUserID constant (migrated to
  serviceuser.WatcherUID)
- Removed local GetLogVolume/GetLogVolumeMount wrapper functions,
  replaced all call sites with direct volume.WritableDirVolume/
  WritableDirVolumeMount calls
- Replaced local GetRunHttpdVolume/Mount, GetVarLogHttpdVolume/Mount,
  GetConfigSecretVolumes with lib-common volume helpers
- Fix watcher-api probe assignment: startup probe was on the log
  sidecar (tail -F) instead of the httpd container, and the log
  sidecar had HTTP probes testing httpd's port instead of its own
  process. Moved startup probe to httpd, removed all probes from log
  sidecar.
- Fix db-sync: removed GetVolumeMounts/GetVolumes calls that added
  conflicting SubPath mounts from the shared config-data volume into
  /etc/watcher/watcher.conf.d/ (directory doesn't exist in image);
  db-sync uses its own db-sync-config-data volume with whole-directory
  mount instead
- Fix db-purge: run watcher-db-manage purge directly instead of via
  shell script; removed dbpurge.sh template, scripts secret, and
  script volume/mount; purge age passed as CLI arg
- Also fixes: TLS cert/key and memcached mTLS staging-path bugs;
  missing /run/httpd and /var/log/httpd emptyDirs
- Update functional and kuttl tests

Jira: OSPRH-33504
Jira: OSPRH-33503

Depends-On: openstack-k8s-operators/lib-common#728

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Martin Schuppert <mschuppert@redhat.com>
@stuggi

stuggi commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

/test functional

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants