feat(self-sovereign-cutover): step 11 — pivot Crossplane Provider CRs to Sovereign Harbor xpkg mirror

Adds cutover-step-11-crossplane-provider-pivot, modelled on step 10's
two-phase pattern, that rewrites every `pkg.crossplane.io/v1.Provider`
CR's `spec.package` host literal from `xpkg.upbound.io/...` to
`harbor.<SOVEREIGN_FQDN>/proxy-xpkg/...` and pushes the same edit to
local Gitea so the bootstrap-kit Kustomization reconcile doesn't
revert the live patch.

Why Step 04 (containerd registries.yaml.v2 mirror) does NOT cover this
even though it registers `xpkg.upbound.io → harbor.<sov>/proxy-xpkg`:
Crossplane's package manager uses `go-containerregistry`'s
`remote.Image()` DIRECTLY from inside the `crossplane-system`
controller Pod (source: `internal/xpkg/fetch.go`), NOT through the
kubelet/containerd CRI client. Containerd mirror config is irrelevant
to it. The ONLY way to redirect Provider package fetches is to
rewrite each Provider's `spec.package` host literal.

The bootstrap-kit ships THREE Provider CRs all carrying the upstream
xpkg literal (clusters/_template + clusters/omantel.omani.works +
clusters/otech.omani.works). None were patched by any prior cutover
step — so every Provider package fetch (initial install, version bump,
ProviderRevision reconcile of an inactive revision, Pod-restart-with-
evicted-cache, any new operator-installed Provider) hit
xpkg.upbound.io directly post-handover. Principle #11 violation.
Caught by the TBD-V24 empirical investigation 2026-05-20.

Step 11 changes:
- NEW templates/11-crossplane-provider-pivot-job.yaml (~270 lines):
  Phase 1 kubectl patches every Provider CR (cluster-scoped, idempotent,
  skip-if-CRD-absent for early-handover window); Phase 2 git push edits
  every clusters/*/infrastructure/provider-*.yaml in local Gitea.
- 09-cutover-status-configmap.yaml: totalSteps "10" → "11" plus
  step.crossplane-provider-pivot.* status keys.
- values.yaml: append `xpkg.upbound.io` to harbor.mothershipAuthsToStrip
  (credential hygiene now covers the xpkg upstream too) and to
  egressTest.blockedDomains (TBD-V23's deny-egress hold proof must
  block xpkg.upbound.io alongside the other 3 mothership families);
  add stepTimeouts.crossplaneProviderPivotSeconds (600s) and
  crossplaneProviderPivot.{upstreamHost,registryPath} overlay knobs.
- rbac.yaml: ClusterRole gains pkg.crossplane.io.providers
  [get,list,watch,update,patch] + apiextensions.k8s.io.
  customresourcedefinitions [get,list,watch] (for CRD-presence probe).
- Chart.yaml: 0.1.36 → 0.1.37 with full changelog entry.
- blueprint.yaml: 0.1.36 → 0.1.37 lockstep.
- clusters/_template/bootstrap-kit/06a-bp-self-sovereign-cutover.yaml:
  pin 0.1.36 → 0.1.37 with comment.
- chart/tests/cutover-contract.sh: bump step_count + mode_job_count
  assertions 10 → 11 / 9 → 10; new Case 22 verifies Step 11 patches
  Provider CRs, rewrites Gitea YAML, and the RBAC + values are wired.

Validation:
- `helm template platform/self-sovereign-cutover/chart` smoke-renders
  cleanly with all 11 step ConfigMaps.
- `bash platform/self-sovereign-cutover/chart/tests/cutover-contract.sh`
  green on all 22 cases.
- `go test ./products/catalyst/bootstrap/api/internal/handler/...
  -count=1` passes (62.8s) — cutover handler reads steps dynamically
  via label selector, no hardcoded list to update.
- Did NOT use --dry-run=server. Cluster-side validation deferred to
  the operator walk on a fresh multi-region prov per anti-theater
  discipline.

Refs #2034 (TBD-V24 — closes only after operator-walk-with-screenshot
on a fresh multi-region prov verifies Provider CRs reconcile from
harbor.<sov-fqdn>/proxy-xpkg, NOT from xpkg.upbound.io).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
hatiyildiz 2026-05-20 04:48:36 +02:00
parent b30454a681
commit 64554e1276
8 changed files with 581 additions and 22 deletions

View File

@ -378,7 +378,32 @@ spec:
# latent gap step-06 Phase-1.6 was silently relying on since
# chart 0.1.31). totalSteps bumped 9 → 10; contract test asserts
# the shift via new Case 21. (Refs #2034)
version: 0.1.36
#
# 0.1.37 (TBD-V24 MISS-3, issue #2034, 2026-05-20): NEW step 11
# (crossplane-provider-pivot) — pivots every Crossplane Provider
# CR's `spec.package` from `xpkg.upbound.io/...` to
# `harbor.<SOVEREIGN_FQDN>/proxy-xpkg/...` so the Crossplane
# package manager (which uses go-containerregistry DIRECTLY,
# bypassing containerd) fetches Provider packages from the
# Sovereign-local Harbor mirror. Step 04's registries.yaml.v2
# mirror DOES register xpkg.upbound.io → proxy-xpkg, but
# Crossplane's fetcher Pod bypasses the kubelet/containerd CRI
# client entirely so the mirror is irrelevant — the ONLY way to
# redirect Provider package fetches is to rewrite each
# Provider's `spec.package` host literal. The bootstrap-kit ships
# 3 Provider CRs all carrying the upstream xpkg literal
# (clusters/_template + clusters/omantel.omani.works +
# clusters/otech.omani.works); none were patched by any prior
# cutover step. Closes the TBD-V24 audit gap for the Crossplane
# tether family (4th tether: xpkg.upbound.io). Phase-1 kubectl
# patch + Phase-2 git push to local Gitea (same shape as Step
# 10). RBAC gains pkg.crossplane.io.providers [update,patch] +
# apiextensions.k8s.io.customresourcedefinitions read for the
# CRD-presence probe. `harbor.mothershipAuthsToStrip` +
# `egressTest.blockedDomains` both gain `xpkg.upbound.io` for
# lockstep. totalSteps bumped 10 → 11; contract test asserts
# the shift via new Case 22. (Refs #2034)
version: 0.1.37
sourceRef:
kind: HelmRepository
name: bp-self-sovereign-cutover

View File

@ -5,7 +5,7 @@ metadata:
labels:
catalyst.openova.io/section: pts-2-3-per-sovereign-supporting-services
spec:
version: 0.1.36
version: 0.1.37
card:
title: self-sovereignty-cutover
summary: |

View File

@ -189,7 +189,68 @@ name: bp-self-sovereign-cutover
# Contract test (tests/cutover-contract.sh) asserts shift from 9 → 10
# step ConfigMaps and from 8 → 9 job-mode ConfigMaps. New Case 21
# verifies Step 10's wrapper + subchart patches are wired.
version: 0.1.36
#
# 0.1.37 (TBD-V24 MISS-3, issue #2034, 2026-05-20): NEW step 11
# (crossplane-provider-pivot) — pivots every `pkg.crossplane.io/v1.
# Provider` CR's `spec.package` host literal from `xpkg.upbound.io/...`
# to `harbor.<SOVEREIGN_FQDN>/proxy-xpkg/...` so the Crossplane package
# manager fetches Provider packages from the Sovereign-local Harbor
# `proxy-xpkg` project, NOT from xpkg.upbound.io.
#
# Root cause this addresses: Crossplane's package manager uses
# `github.com/google/go-containerregistry`'s `remote.Image()` DIRECTLY
# from the `crossplane-system` controller Pod (source:
# `internal/xpkg/fetch.go`). It does NOT route fetches through the
# kubelet's CRI containerd client, so `/etc/containerd/certs.d/` and
# `/etc/rancher/k3s/registries.yaml` mirror config (rewritten by Step
# 04's DaemonSet) is irrelevant to it. The bootstrap-kit ships THREE
# Provider CRs all carrying `spec.package =
# xpkg.upbound.io/crossplane-contrib/provider-hcloud:v0.4.0`
# (clusters/_template + clusters/omantel.omani.works +
# clusters/otech.omani.works) — none were patched by any prior cutover
# step. Result: every Provider package fetch (initial install,
# version bump, ProviderRevision reconcile of an inactive revision,
# Pod-restart-with-evicted-cache, any new operator-installed Provider)
# hit xpkg.upbound.io directly post-handover — a direct violation of
# Principle #11 (no tether to external upstreams after handover).
# Caught by the TBD-V24 empirical investigation 2026-05-20.
#
# Step 04 (containerd registries.yaml.v2) does NOT catch this even
# though it DOES register `xpkg.upbound.io → harbor.<sov>/proxy-xpkg`
# — that mirror is consumed only by containerd (kubelet image pulls),
# which Crossplane bypasses. The ONLY way to redirect Provider package
# fetches is to rewrite each Provider's `spec.package` host literal.
#
# Step 11 adds a Phase-1 live `kubectl patch provider.pkg.crossplane.io`
# loop over every Provider CR (cluster-scoped). Skip-if-absent guards
# for: the CRD not yet installed (very-early-handover window) and an
# already-pivoted package literal (re-run idempotency). Phase-2 git
# pushes sed edits of every `clusters/*/infrastructure/provider-*.yaml`
# to local Gitea so the bootstrap-kit Kustomization reconcile doesn't
# revert the Phase-1 live patch within ~1 min.
#
# Lockstep changes:
# - `harbor.mothershipAuthsToStrip` gains `xpkg.upbound.io` (Step 06
# Phase-0 credential hygiene now covers the xpkg upstream too).
# - `egressTest.blockedDomains` gains `xpkg.upbound.io` (TBD-V23's
# deny-egress hold proof must block xpkg.upbound.io alongside the
# other 3 mothership families).
# - `stepTimeouts.crossplaneProviderPivotSeconds` (600s default).
# - `crossplaneProviderPivot.{upstreamHost,registryPath}` overlay
# knobs (defaults: xpkg.upbound.io, proxy-xpkg).
# - RBAC ClusterRole gains `pkg.crossplane.io.providers
# {get,list,watch,update,patch}` + `apiextensions.k8s.io.
# customresourcedefinitions {get,list,watch}` (the latter for the
# CRD-presence probe).
#
# totalSteps bumped from "10" → "11" in 09-cutover-status-configmap.yaml.
# Contract test (tests/cutover-contract.sh) asserts shift from 10 → 11
# step ConfigMaps and from 9 → 10 job-mode ConfigMaps. New Case 22
# verifies Step 11 patches Provider CRs and rewrites Gitea YAML.
# (Refs #2034 — closes after operator-walk-with-screenshot on a fresh
# multi-region prov verifies Provider CRs reconcile from
# harbor.<sov-fqdn>/proxy-xpkg.)
version: 0.1.37
description: |
Catalyst Self-Sovereignty Cutover Blueprint. Installs DORMANT — this
chart ships eight step ConfigMaps (PodSpec ConfigMaps, one per step),
@ -243,6 +304,17 @@ description: |
kubectl patch; Phase-2 git push to local Gitea so the
bootstrap-kit Kustomization doesn't revert the override.
(TBD-V24 MISS-1)
11 cutover-step-11-crossplane-provider-pivot mode=job
Patch every `pkg.crossplane.io/v1.Provider` CR's spec.package
from `xpkg.upbound.io/...` → `harbor.<SOVEREIGN_FQDN>/
proxy-xpkg/...` so the Crossplane package manager fetches
Provider packages from Sovereign-local Harbor, NOT from
xpkg.upbound.io. Phase-1 kubectl patch (skip if CRD absent or
already-pivoted); Phase-2 git push to local Gitea so the
bootstrap-kit Kustomization doesn't revert the override.
Crossplane bypasses containerd (uses go-containerregistry
directly), so Step 04's registries.yaml mirror does NOT catch
Provider package fetches. (TBD-V24 MISS-3)
Plus:
self-sovereign-cutover-status ConfigMap

View File

@ -43,7 +43,7 @@ data:
cutoverFinishedAt: ""
currentStep: ""
currentStepIndex: "0"
totalSteps: "10"
totalSteps: "11"
progressPercent: "0"
failedStep: ""
lastError: ""
@ -91,3 +91,7 @@ data:
step.vcluster-registry-pivot.finishedAt: ""
step.vcluster-registry-pivot.result: ""
step.vcluster-registry-pivot.jobName: ""
step.crossplane-provider-pivot.startedAt: ""
step.crossplane-provider-pivot.finishedAt: ""
step.crossplane-provider-pivot.result: ""
step.crossplane-provider-pivot.jobName: ""

View File

@ -0,0 +1,319 @@
{{- /*
Step 11 — crossplane-provider-pivot (TBD-V24 MISS-3, issue #2034).
Pivots every `pkg.crossplane.io/v1.Provider` CR's `spec.package` host
literal from `xpkg.upbound.io/...` to `harbor.<SOVEREIGN_FQDN>/
proxy-xpkg/...` so the Crossplane package manager fetches Provider
packages from the Sovereign-local Harbor `proxy-xpkg` project, NOT
from `xpkg.upbound.io`. Without this step, every Provider package
fetch (initial install, version bump, ProviderRevision reconcile of
an inactive revision, Pod-restart-with-evicted-cache, any new
operator-installed Provider) hits `xpkg.upbound.io` directly — a
direct violation of Principle #11 (no tether to external upstreams
after handover).
Why Step 04 (containerd registries.yaml.v2 pivot) does NOT catch it
───────────────────────────────────────────────────────────────────
Crossplane's package manager uses `github.com/google/go-container
registry`'s `remote.Image()` DIRECTLY from inside the
`crossplane-system` controller Pod (see crossplane source
`internal/xpkg/fetch.go`). It does NOT route fetches through the
kubelet's CRI containerd client, so the `/etc/containerd/certs.d/`
and `/etc/rancher/k3s/registries.yaml` mirror config (rewritten by
Step 04's DaemonSet) is irrelevant to it. The package pull is a
plain HTTPS dial-out from the Crossplane controller Pod to the
literal hostname in `spec.package`.
There is no first-class registry-mirror knob on Crossplane v1
`Provider`/`ProviderRevision`/`Lock`/`Configuration` CRDs and the
chart's `global.imageRegistry` controls only the controller image,
not the package URL. The ONLY way to redirect Provider package
fetches is to rewrite each Provider's `spec.package` host literal.
Patches applied (idempotent merge-patch + Gitea YAML edit)
──────────────────────────────────────────────────────────
The bootstrap-kit ships THREE Provider CRs that all carry
`spec.package = xpkg.upbound.io/crossplane-contrib/provider-hcloud:v0.4.0`:
- clusters/_template/infrastructure/provider-hcloud.yaml
- clusters/omantel.omani.works/infrastructure/provider-hcloud.yaml
- clusters/otech.omani.works/infrastructure/provider-hcloud.yaml
Each per-Sovereign overlay copy ships only into ITS Sovereign cluster.
The live K8s patch is therefore scoped per-cluster (only the local
Sovereign's Provider CR is present at cutover time). The Gitea YAML
edit fixes ALL infrastructure provider-*.yaml files under
`clusters/<sov>/infrastructure/` defensively so a fresh prov pulls the
already-pivoted spec from local Gitea.
Phase inventory
───────────────
Phase 1 Live K8s patch — `kubectl patch provider.pkg.crossplane.io`
for every Provider CR (cluster-scoped). Computes the new
package literal by replacing the `xpkg.upbound.io` prefix
with `harbor.<SOVEREIGN_FQDN>/proxy-xpkg` while preserving
the path + tag. Triggers an immediate Crossplane revision
reconcile so the new package is fetched within seconds.
Phase 2 Push YAML edit to local Gitea — sed-in-place every
`clusters/<sov>/infrastructure/provider-<name>.yaml` carrying the
literal `xpkg.upbound.io/...`. Without this phase, the
bootstrap-kit Kustomization reconcile (every ~1 min from
local Gitea) would silently revert Phase 1 back to the
upstream URL within minutes.
Idempotency
───────────
- Phase 1: a merge-patch with the SAME spec.package literal is a
no-op (kubectl emits "no change", resourceVersion unchanged). The
step reads the current literal first and SKIPs the patch if
already pivoted.
- Phase 2: the sed pattern targets `xpkg.upbound.io/` so a re-run
against a YAML where the literal has been replaced finds no match
and exits cleanly. We also guard with a sentinel grep so the git
commit is skipped when the diff is empty.
Order rationale: 11 (after Step 10)
────────────────────────────────────
Placed at order 11 to run LAST. Functionally it depends on:
- Step 02 (harbor-projects) so the `proxy-xpkg` Harbor project
exists for the new pulls (chart values.yaml ships proxy-xpkg in
the canonical list since chart 0.1.x).
- Step 06 Phase-0 (helmrepository-patches) so the ghcr-pull Secret
carries auth for `harbor.<sov-fqdn>` (Crossplane fetch uses the
same pull-secret machinery via K8sFetcher's pullSecretFromConfig).
- Step 04 / containerd is NOT a hard dep (Crossplane bypasses it)
but running after Step 04 ensures the rest of the cluster is on
the Sovereign-local mirror plane before the Provider re-fetches.
Image phase: post (uses the same alpine/k8s image as Step 10).
*/ -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: cutover-step-11-crossplane-provider-pivot
namespace: {{ .Release.Namespace }}
labels:
{{- include "bp-self-sovereign-cutover.labels" . | nindent 4 }}
app.kubernetes.io/component: cutover-step
bp.openova.io/cutover-order: "11"
bp.openova.io/cutover-mode: "job"
data:
stepName: crossplane-provider-pivot
podSpec: |
serviceAccountName: {{ include "bp-self-sovereign-cutover.serviceAccountName" . }}
restartPolicy: Never
activeDeadlineSeconds: {{ .Values.stepTimeouts.crossplaneProviderPivotSeconds }}
containers:
- name: crossplane-provider-pivot
# alpine/k8s carries both kubectl AND git so we can patch live
# Provider CRs AND push the YAML edit to local Gitea (same
# image Steps 06/10 use for the same reason).
image: harbor.openova.io/proxy-dockerhub/alpine/k8s:1.31.4
imagePullPolicy: IfNotPresent
env:
- name: SOVEREIGN_FQDN
value: {{ .Values.sovereign.fqdn | quote }}
- name: GITEA_INTERNAL_URL
value: {{ .Values.sovereign.giteaInternalURL | quote }}
- name: GITEA_USERNAME
valueFrom:
secretKeyRef:
name: {{ .Values.gitea.adminSecretRef.name }}
key: {{ .Values.gitea.adminSecretRef.usernameKey }}
- name: GITEA_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.gitea.adminSecretRef.name }}
key: {{ .Values.gitea.adminSecretRef.passwordKey }}
- name: GITEA_ORG
value: {{ .Values.gitea.org | quote }}
- name: GITEA_REPO
value: {{ .Values.gitea.repo | quote }}
- name: UPSTREAM_HOST
value: {{ .Values.crossplaneProviderPivot.upstreamHost | quote }}
- name: REGISTRY_PATH
value: {{ .Values.crossplaneProviderPivot.registryPath | quote }}
volumeMounts:
- name: tmp
mountPath: /tmp
command: ["/bin/sh", "-c"]
args:
- |
set -eu
# Compute the Sovereign-local target prefix. The Crossplane
# Provider's `spec.package` is HOST + PATH + ":" + TAG
# (e.g. xpkg.upbound.io/crossplane-contrib/provider-hcloud:
# v0.4.0). We rewrite ONLY the HOST + PATH-prefix segment
# so the upstream path + tag are preserved untouched.
target_host="harbor.${SOVEREIGN_FQDN}"
target_prefix="${target_host}/${REGISTRY_PATH}"
upstream_prefix="${UPSTREAM_HOST}"
echo "[crossplane-provider-pivot] target host: ${target_host}"
echo "[crossplane-provider-pivot] upstream: ${upstream_prefix}"
echo "[crossplane-provider-pivot] target prefix: ${target_prefix}"
# ---- Phase 1: live K8s patch on every Provider CR ----
#
# `kubectl get providers.pkg.crossplane.io` is cluster-scoped
# (Provider is a non-namespaced CR). We iterate the list and
# patch any Provider whose .spec.package host == UPSTREAM_HOST.
# Skip-if-absent guard: if the CRD itself isn't installed
# (very early on a freshly-handed-over Sovereign before
# bp-crossplane lands), exit 0 cleanly — the bootstrap-kit
# will eventually install Crossplane + the Provider CR, and
# Phase-2 has already rewritten the YAML so the on-install
# spec.package will be the pivoted one.
if ! kubectl get crd providers.pkg.crossplane.io >/dev/null 2>&1; then
echo "[crossplane-provider-pivot] SKIP Phase 1 — providers.pkg.crossplane.io CRD not installed yet"
echo "[crossplane-provider-pivot] Phase 2 will still rewrite YAML in Gitea so the eventual install pulls from ${target_host}"
phase1_skip=1
else
phase1_skip=0
fi
ok=0
skip=0
fail=0
if [ "${phase1_skip}" -eq 0 ]; then
# List Provider CRs. JSONPath emits one line per Provider
# with `<name> <spec.package>` separated by tab so a
# while-read loop can parse without word-splitting on the
# package string (which can contain colons and slashes).
providers_tsv=$(kubectl get providers.pkg.crossplane.io \
-o 'jsonpath={range .items[*]}{.metadata.name}{"\t"}{.spec.package}{"\n"}{end}' 2>/dev/null || echo "")
if [ -z "${providers_tsv}" ]; then
echo "[crossplane-provider-pivot] no Provider CRs found in cluster (skip Phase 1)"
else
# Use a heredoc + IFS=$'\t' to safely parse the TSV.
printf '%s\n' "${providers_tsv}" | while IFS="$(printf '\t')" read -r name pkg; do
if [ -z "${name}" ]; then continue; fi
case "${pkg}" in
"${upstream_prefix}/"*)
# Compute new spec.package by replacing host prefix.
pkg_suffix="${pkg#${upstream_prefix}/}"
new_pkg="${target_prefix}/${pkg_suffix}"
echo "[crossplane-provider-pivot] PATCH ${name} ${pkg} -> ${new_pkg}"
# Merge-patch only spec.package — preserves every
# other spec field (revisionActivationPolicy,
# packagePullPolicy, controllerConfigRef, etc.).
patch_json=$(printf '{"spec":{"package":"%s"}}' "${new_pkg}")
if kubectl patch provider.pkg.crossplane.io "${name}" \
--type=merge --patch "${patch_json}" >/dev/null 2>&1; then
echo "[crossplane-provider-pivot] OK ${name}"
else
echo "[crossplane-provider-pivot] FAIL ${name}" >&2
fi
;;
"${target_prefix}/"*)
echo "[crossplane-provider-pivot] SKIP ${name} (already pivoted: ${pkg})"
;;
*)
echo "[crossplane-provider-pivot] SKIP ${name} (package host not ${upstream_prefix} or ${target_host}: ${pkg})"
;;
esac
done
fi
fi
# ---- Phase 2: push YAML edit to local Gitea ----
#
# bootstrap-kit Kustomization reconciles provider-*.yaml from
# local Gitea every ~1 min. Without this phase, Phase 1's
# live patch is reverted on the next reconcile because the
# YAML in Gitea still carries the xpkg.upbound.io literal.
# We sed-in-place every clusters/*/infrastructure/provider-
# *.yaml that carries the upstream host literal in its
# `spec.package` line.
export HOME=/tmp
git config --global user.name "self-sovereign-cutover"
git config --global user.email "cutover@${SOVEREIGN_FQDN}"
git config --global advice.detachedHead false
gitea_host="$(printf '%s' "${GITEA_INTERNAL_URL}" | sed -E 's|^https?://||' | cut -d: -f1 | cut -d/ -f1)"
for i in $(seq 1 30); do
if nslookup "${gitea_host}" >/dev/null 2>&1; then break; fi
sleep 5
done
push_url=$(printf '%s' "${GITEA_INTERNAL_URL}" | sed -E "s,^(https?://),\1${GITEA_USERNAME}:${GITEA_PASSWORD}@,")"/${GITEA_ORG}/${GITEA_REPO}.git"
redacted=$(printf '%s' "${GITEA_INTERNAL_URL}/${GITEA_ORG}/${GITEA_REPO}.git")
echo "[crossplane-provider-pivot] cloning ${redacted}"
cd /tmp
rm -rf repo
git clone --depth 1 --branch main "${push_url}" repo >/dev/null 2>&1
cd repo
# Find every clusters/*/infrastructure/provider-*.yaml that
# references the upstream host literal in its package spec.
# `grep -l` returns only filenames; `find` ensures we don't
# pick up unrelated provider-* files elsewhere in the tree.
edited=0
matches=$(find clusters -type f -path '*/infrastructure/provider-*.yaml' 2>/dev/null || true)
if [ -z "${matches}" ]; then
echo "[crossplane-provider-pivot] no clusters/*/infrastructure/provider-*.yaml present in mirror"
else
# Iterate, sed each file in-place. We escape `/` in the
# upstream/target prefixes so they don't terminate sed's
# delimiter; using `,` as delimiter is the simpler workaround
# (the prefixes never contain commas).
for file in ${matches}; do
if grep -q "package: ${upstream_prefix}/" "${file}" 2>/dev/null; then
if sed -i -E "s,(package:[[:space:]]+)${upstream_prefix}/,\1${target_prefix}/," "${file}"; then
echo "[crossplane-provider-pivot] rewrote ${file}"
edited=$((edited+1))
else
echo "[crossplane-provider-pivot] WARN sed failed on ${file}" >&2
fi
elif grep -q "package: ${target_prefix}/" "${file}" 2>/dev/null; then
echo "[crossplane-provider-pivot] SKIP ${file} (already pivoted)"
else
echo "[crossplane-provider-pivot] SKIP ${file} (no upstream package literal found)"
fi
done
fi
echo "[crossplane-provider-pivot] sed edited ${edited} files"
if [ "${edited}" -eq 0 ]; then
echo "[crossplane-provider-pivot] no edits required — already pivoted in Gitea or files absent"
exit 0
fi
git add clusters
if git diff --staged --quiet; then
echo "[crossplane-provider-pivot] git diff empty after sed — nothing to commit"
exit 0
fi
git commit -m "cutover: pivot Crossplane Provider spec.package to harbor.${SOVEREIGN_FQDN}/${REGISTRY_PATH}" >/dev/null
push_err=$(git push origin main 2>&1) || {
echo "[crossplane-provider-pivot] FATAL: git push failed" >&2
printf '%s\n' "$push_err" >&2
exit 1
}
echo "[crossplane-provider-pivot] pushed to ${redacted}"
# Trigger an immediate Flux reconciliation so the new YAML
# lands without waiting for the polling interval.
kubectl annotate --overwrite gitrepository openova \
-n flux-system \
"reconcile.fluxcd.io/requestedAt=$(date +%s)" >/dev/null || true
echo "[crossplane-provider-pivot] step complete"
resources:
requests: { cpu: 50m, memory: 128Mi }
limits: { memory: 384Mi }
securityContext:
runAsNonRoot: true
runAsUser: 1001
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
volumes:
- name: tmp
emptyDir: {}

View File

@ -90,6 +90,12 @@ rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["gateways"] # step 06 Phase -1 (#1871, chart 0.1.32): wait for cilium-gateway Programmed=True before URL rewrite
verbs: ["get", "list", "watch"]
- apiGroups: ["pkg.crossplane.io"]
resources: ["providers"] # step 11 (TBD-V24 MISS-3, chart 0.1.37): pivot Provider.spec.package off xpkg.upbound.io
verbs: ["get", "list", "watch"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"] # step 11 probes for providers.pkg.crossplane.io CRD presence before patching
verbs: ["get", "list", "watch"]
# ───────────────────────────────────────────────────────────────
# UPDATE / PATCH / DELETE — separate from create per RBAC rule.
@ -109,6 +115,9 @@ rules:
- apiGroups: ["helm.toolkit.fluxcd.io"]
resources: ["helmreleases"]
verbs: ["update", "patch"] # step 06 phase-1.6 (catalog override) + step 10 vcluster-registry-pivot (TBD-V24 MISS-1)
- apiGroups: ["pkg.crossplane.io"]
resources: ["providers"]
verbs: ["update", "patch"] # step 11 crossplane-provider-pivot (TBD-V24 MISS-3)
- apiGroups: ["cilium.io"]
resources: ["ciliumnetworkpolicies"]
verbs: ["delete", "patch", "update"] # step 08 removes the policy at end-of-test

View File

@ -14,9 +14,11 @@
# 3. Each step ConfigMap MUST carry data keys:
# stepName (always)
# podSpec (mode=job only)
# 4. EXACTLY 10 step ConfigMaps must render (steps 1..10; step 9
# 4. EXACTLY 11 step ConfigMaps must render (steps 1..11; step 9
# gitea-token-mint added in chart 0.1.30, TBD-C18; step 10
# vcluster-registry-pivot added in chart 0.1.35, TBD-V24 MISS-1).
# vcluster-registry-pivot added in chart 0.1.36, TBD-V24 MISS-1;
# step 11 crossplane-provider-pivot added in chart 0.1.37,
# TBD-V24 MISS-3).
# 5. Step 04 must be mode=daemonset-wait.
# 6. The status ConfigMap (default name self-sovereign-cutover-status)
# MUST render with helm.sh/resource-policy: keep so a chart
@ -44,14 +46,13 @@ echo "[cutover-contract] Case 1: chart renders with default values"
helm template smoke . > "$TMP/render.yaml"
echo " PASS ($(wc -l < "$TMP/render.yaml") lines)"
echo "[cutover-contract] Case 2 + 4: exactly 10 step ConfigMaps render with required labels"
echo "[cutover-contract] Case 2 + 4: exactly 11 step ConfigMaps render with required labels"
# Use yq if present (the CI runner installs it for the blueprint-release
# guards); fall back to grep counting on workstations without yq.
# Step 9 (gitea-token-mint) added in chart 0.1.30 (TBD-C18); step 10
# (vcluster-registry-pivot) added in chart 0.1.35 (TBD-V24 MISS-1,
# issue #2034) to pivot the bp-*-vcluster HelmReleases' image.repository
# from harbor.openova.io → harbor.<SOVEREIGN_FQDN> so vCluster Pods
# pull from the Sovereign-local Harbor mirror post-cutover.
# (vcluster-registry-pivot) added in chart 0.1.36 (TBD-V24 MISS-1);
# step 11 (crossplane-provider-pivot) added in chart 0.1.37 (TBD-V24
# MISS-3, issue #2034) to pivot Provider.spec.package off xpkg.upbound.io.
if command -v yq >/dev/null 2>&1; then
# yq emits `---` separators between matched docs; filter those out
# before counting names. `grep -E '^cutover-step-'` matches only the
@ -62,28 +63,28 @@ else
# — count distinct order values, which equals step count.
step_count=$(grep -c 'bp.openova.io/cutover-order:' "$TMP/render.yaml")
fi
if [ "${step_count}" -ne 10 ]; then
echo "FAIL: expected 10 step ConfigMaps, got ${step_count}" >&2
if [ "${step_count}" -ne 11 ]; then
echo "FAIL: expected 11 step ConfigMaps, got ${step_count}" >&2
exit 1
fi
echo " PASS (10 step ConfigMaps)"
echo " PASS (11 step ConfigMaps)"
echo "[cutover-contract] Case 3: required data keys present"
# stepName key must exist on every step ConfigMap (10 total).
# podSpec key must exist on every job-mode step (9 of 10 — step 04 is daemonset-wait).
# stepName key must exist on every step ConfigMap (11 total).
# podSpec key must exist on every job-mode step (10 of 11 — step 04 is daemonset-wait).
mode_job_count=$(grep -c 'bp.openova.io/cutover-mode: "job"' "$TMP/render.yaml")
if [ "${mode_job_count}" -ne 9 ]; then
echo "FAIL: expected 9 job-mode step ConfigMaps, got ${mode_job_count}" >&2
if [ "${mode_job_count}" -ne 10 ]; then
echo "FAIL: expected 10 job-mode step ConfigMaps, got ${mode_job_count}" >&2
exit 1
fi
podspec_keys=$(grep -c '^ podSpec: |' "$TMP/render.yaml")
if [ "${podspec_keys}" -lt 9 ]; then
echo "FAIL: expected at least 9 podSpec keys (one per job-mode step), got ${podspec_keys}" >&2
if [ "${podspec_keys}" -lt 10 ]; then
echo "FAIL: expected at least 10 podSpec keys (one per job-mode step), got ${podspec_keys}" >&2
exit 1
fi
stepname_keys=$(grep -c '^ stepName:' "$TMP/render.yaml")
if [ "${stepname_keys}" -lt 10 ]; then
echo "FAIL: expected at least 10 stepName keys, got ${stepname_keys}" >&2
if [ "${stepname_keys}" -lt 11 ]; then
echo "FAIL: expected at least 11 stepName keys, got ${stepname_keys}" >&2
exit 1
fi
echo " PASS (data keys present on every step)"
@ -531,4 +532,81 @@ if ! awk '/^kind: ClusterRole$/,/^---$/' "$TMP/render.yaml" \
fi
echo " PASS (Step-10 wired to pivot vCluster HRs to local Harbor)"
echo "[cutover-contract] Case 22: Step-11 crossplane-provider-pivot patches Provider CRs (TBD-V24 MISS-3)"
# Chart <0.1.37 shipped NO Crossplane Provider package pivot. Result:
# every Provider package fetch (initial install, version bump, inactive
# ProviderRevision reconcile, Pod-restart-with-evicted-cache) hit
# xpkg.upbound.io directly post-handover — a direct violation of
# Principle #11. Step 04's registries.yaml.v2 mirror is irrelevant to
# Crossplane because Crossplane's fetcher uses go-containerregistry's
# remote.Image() directly, bypassing the kubelet/containerd CRI client.
# Caught by TBD-V24 empirical investigation 2026-05-20.
#
# Chart 0.1.37 adds Step-11 that:
# - kubectl patches every pkg.crossplane.io/v1.Provider CR's
# spec.package, swapping `xpkg.upbound.io/...` →
# `harbor.${SOVEREIGN_FQDN}/proxy-xpkg/...` while preserving the
# full upstream path + tag.
# - git push edits clusters/*/infrastructure/provider-*.yaml to
# local Gitea so the bootstrap-kit Kustomization reconcile
# doesn't revert Phase-1 within ~1 min.
# - Idempotent on re-run (skip-if-already-pivoted on Phase-1,
# no-op grep guard on Phase-2).
# - Skip-if-absent for the CRD (very-early-handover window where
# bp-crossplane hasn't installed yet — Phase-2 still rewrites
# Gitea so the eventual install pulls from local Harbor).
#
# Guard against future regressions that drop the step.
if ! grep -q 'cutover-step-11-crossplane-provider-pivot' "$TMP/render.yaml"; then
echo "FAIL: Step-11 crossplane-provider-pivot ConfigMap missing (TBD-V24 MISS-3)" >&2
exit 1
fi
if ! grep -A20 'cutover-step-11-crossplane-provider-pivot' "$TMP/render.yaml" | grep -q 'bp.openova.io/cutover-order: "11"'; then
echo "FAIL: Step-11 not labelled bp.openova.io/cutover-order=11 (TBD-V24 MISS-3)" >&2
exit 1
fi
# Default upstream host must be xpkg.upbound.io (the Crossplane Provider
# package upstream the cutover pivots OFF of).
if ! grep -A60 'cutover-step-11-crossplane-provider-pivot' "$TMP/render.yaml" | grep -q 'value: "xpkg.upbound.io"'; then
echo "FAIL: Step-11 missing UPSTREAM_HOST=xpkg.upbound.io env (TBD-V24 MISS-3)" >&2
exit 1
fi
# Default registry path must be proxy-xpkg (the Harbor proxy-cache
# project that mirrors xpkg.upbound.io — created by Step 02).
if ! grep -A60 'cutover-step-11-crossplane-provider-pivot' "$TMP/render.yaml" | grep -q 'value: "proxy-xpkg"'; then
echo "FAIL: Step-11 missing REGISTRY_PATH=proxy-xpkg env (TBD-V24 MISS-3)" >&2
exit 1
fi
# The script body must call kubectl patch on provider.pkg.crossplane.io.
if ! grep -q 'kubectl patch provider.pkg.crossplane.io' "$TMP/render.yaml"; then
echo "FAIL: Step-11 missing kubectl patch provider.pkg.crossplane.io invocation (TBD-V24 MISS-3)" >&2
exit 1
fi
# Phase-2 must sed the package literal in clusters/*/infrastructure/
# provider-*.yaml — guard the find path is wired.
if ! grep -q "path '\*/infrastructure/provider-\*.yaml'" "$TMP/render.yaml"; then
echo "FAIL: Step-11 Phase-2 missing provider-*.yaml find pattern (TBD-V24 MISS-3)" >&2
exit 1
fi
# RBAC: ClusterRole must permit update/patch on
# pkg.crossplane.io.providers (Phase-1 kubectl patch).
if ! awk '/^kind: ClusterRole$/,/^---$/' "$TMP/render.yaml" \
| grep -B1 -A3 '"providers"' | grep -E 'verbs:.*"patch"|verbs:.*"update"' >/dev/null; then
echo "FAIL: ClusterRole missing pkg.crossplane.io.providers [update|patch] verb (TBD-V24 MISS-3)" >&2
exit 1
fi
# RBAC: ClusterRole must permit get/list/watch on
# apiextensions.k8s.io.customresourcedefinitions (CRD-presence probe).
if ! awk '/^kind: ClusterRole$/,/^---$/' "$TMP/render.yaml" \
| grep -B1 -A3 '"customresourcedefinitions"' | grep -q 'verbs:.*"get"'; then
echo "FAIL: ClusterRole missing apiextensions.k8s.io.customresourcedefinitions read verbs (TBD-V24 MISS-3)" >&2
exit 1
fi
# xpkg.upbound.io must be added to mothershipAuthsToStrip + blockedDomains.
if ! grep -q '"xpkg.upbound.io"' "$TMP/render.yaml"; then
echo "FAIL: xpkg.upbound.io not present in chart values (mothershipAuthsToStrip / blockedDomains) (TBD-V24 MISS-3)" >&2
exit 1
fi
echo " PASS (Step-11 wired to pivot Provider CRs to local Harbor proxy-xpkg)"
echo "[cutover-contract] All gates green."

View File

@ -133,6 +133,17 @@ harbor:
mothershipAuthsToStrip:
- "ghcr.io"
- "harbor.openova.io"
# TBD-V24 MISS-3 (chart 0.1.37, 2026-05-20): xpkg.upbound.io is the
# Crossplane Provider package upstream — Step 11 pivots every
# `Provider.spec.package` literal from xpkg.upbound.io to
# `harbor.<sov-fqdn>/proxy-xpkg`, so any standing auth for the
# upstream xpkg host on the ghcr-pull Secret is no longer needed.
# Strip it for the same credential-hygiene rationale as the other
# mothership entries above (CLAUDE.md §3 Principle #11). Defence-
# in-depth: even though ghcr-pull rarely carries xpkg auth today
# (Provider packages typically pull anonymously), an operator
# overlay that adds xpkg credentials gets the same clean-up here.
- "xpkg.upbound.io"
# Seven proxy-cache projects + their upstream registry endpoints.
# Names are stable because step 06 (helmrepository-patches) +
# registries.yaml v2 (registry-pivot) rewrite paths against these
@ -332,6 +343,15 @@ egressTest:
- "github.com"
- "ghcr.io"
- "harbor.openova.io"
# TBD-V24 MISS-3 (chart 0.1.37): xpkg.upbound.io is the Crossplane
# Provider package upstream. Step 11 pivots every Provider's
# spec.package literal off this host, so the deny-egress hold proof
# MUST also block xpkg.upbound.io — otherwise a Provider revision
# reconcile during the hold window could silently fall through to
# the upstream xpkg host and break the proof. The deny implementation
# is TBD-V23's responsibility; this entry is the contract this list
# must carry once TBD-V23's CiliumNetworkPolicy applies.
- "xpkg.upbound.io"
# Status sentinel ConfigMap — initial state machine state. catalyst-api
# updates the per-step fields as each Job completes.
@ -429,6 +449,12 @@ stepTimeouts:
# + one git clone/push to local Gitea. Bounded by the Gitea push
# round-trip; 600s is generous (typical run completes in <60s).
vclusterRegistryPivotSeconds: 600
# Step 11 (crossplane-provider-pivot, TBD-V24 MISS-3) — Provider CR
# patch loop + one git clone/push to local Gitea. Bounded by the
# Gitea push round-trip + the Provider revision reconcile triggered
# by the patch (Crossplane is typically <30s to re-fetch from local
# Harbor); 600s is generous (typical run completes in <60s).
crossplaneProviderPivotSeconds: 600
# Step 10 (vcluster-registry-pivot, TBD-V24 MISS-1, issue #2034) — pivot
# the three bp-*-vcluster HelmReleases' image.repository from
@ -454,6 +480,32 @@ vclusterRegistryPivot:
# pinned a different upstream vCluster version.
imageTag: "0.20.0"
# Step 11 (crossplane-provider-pivot, TBD-V24 MISS-3, issue #2034) —
# pivot every `pkg.crossplane.io/v1.Provider` CR's `spec.package` host
# literal from `xpkg.upbound.io/...` to
# `harbor.<SOVEREIGN_FQDN>/proxy-xpkg/...` so the Crossplane package
# manager fetches Provider packages from the Sovereign-local Harbor
# `proxy-xpkg` project. Without this step the Provider package fetch
# remains tethered to xpkg.upbound.io indefinitely post-handover —
# Step 04 (containerd registries.yaml.v2 mirror for xpkg.upbound.io)
# does NOT catch it because Crossplane's `internal/xpkg/fetch.go` uses
# go-containerregistry's `remote.Image()` DIRECTLY, NOT through the
# kubelet/containerd CRI client. See 11-crossplane-provider-pivot-job.yaml
# for the full protocol.
crossplaneProviderPivot:
# Upstream package host literal to match in `spec.package`. The
# cutover-step script swaps this prefix for `harbor.<sov-fqdn>/
# ${registryPath}` while preserving the rest of the package path
# and tag. Per-Sovereign overlay can override if a future Crossplane
# release ships from a different upstream OCI host.
upstreamHost: "xpkg.upbound.io"
# Sovereign-local Harbor proxy-cache project path that proxies
# `xpkg.upbound.io`. Step 02 (harbor-projects) creates this project
# from `.Values.harbor.proxyProjects[name=proxy-xpkg]`. Do NOT
# diverge from that project name unless the proxyProjects list is
# updated in lockstep.
registryPath: "proxy-xpkg"
# Step 09 (gitea-token-mint, TBD-C18) — bootstrap the Gitea API token
# that the SME provisioning service uses for tenant repo materialisation.
#