fix(catalyst-bootstrap-api): wire CATALYST_NEWAPI_ADMIN_TOKEN + correct CATALYST_NEWAPI_ADDR (Refs #2021)

Bundles the two halves of the broken ADR-0003 §3.2 NewAPI admin-API hook so the path goes from dormant-and-misconfigured to actually live: 1. catalyst-api Deployment (bp-catalyst-platform) now sets: - CATALYST_NEWAPI_ADDR = "http://newapi-bp-newapi.newapi.svc.cluster.local:3000" (literal — dual-mode Helm+Kustomize contract) - CATALYST_NEWAPI_ADMIN_TOKEN via secretKeyRef on `catalyst-newapi-admin-token` key ADMIN_API_TOKEN (optional:true) 2. bp-newapi ExternalSecret target now carries emberstack/reflector mirror annotations (default reflector-allowed-namespaces = "catalyst-system") so the Secret rendered in the `newapi` namespace is materialised in the catalyst-api Pod's namespace (same cross-namespace seam as sme-secrets / catalyst-gitea-token). 3. main.go default URL fallback corrected from the NXDOMAIN `http://newapi.newapi.svc` to the canonical Service URL `http://newapi-bp-newapi.newapi.svc.cluster.local:3000` (same root cause as TBD-V14 / PR #2017: bp-newapi.fullname renders `<Release.Name>-<Chart.Name>` and bootstrap-kit slot 80 sets `releaseName: newapi` against chart `bp-newapi`). 4. newapi/client.go godoc + main.go comments updated to the correct Service URL. Chart lockstep (Inviolable Principle #14): - bp-newapi 1.4.32 -> 1.4.33 - bp-catalyst-platform 1.4.224 -> 1.4.225 - bootstrap-kit pins both in lockstep. Validation: - go test ./internal/newapi/... ./internal/handler/... PASS - go build ./cmd/api/ PASS - helm template products/catalyst/chart/ renders CATALYST_NEWAPI_ADDR=http://newapi-bp-newapi.newapi.svc.cluster.local:3000 + CATALYST_NEWAPI_ADMIN_TOKEN secretKeyRef on catalyst-newapi-admin-token/ADMIN_API_TOKEN. - kubectl kustomize products/catalyst/chart/templates/ renders the same env vars (dual-mode contract preserved). - helm template platform/newapi/chart/ -s templates/external-secret.yaml --api-versions=external-secrets.io/v1beta1 renders the reflector annotations on target.template.metadata.annotations. Per CLAUDE.md §0 anti-theater discipline this PR uses Refs #2021 (NOT Closes). Issue closes only after a fresh-prov operator walks /console/sme/users -> Add User and observes `sme-users: NewAPI admin client wired` at catalyst-api startup + the row transitions to state=newapi_created (no `newapi client not wired` sentinel, no NXDOMAIN for `newapi.newapi.svc`). Refs #2021 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 05:59:48 +02:00 · 2026-05-20 05:59:48 +02:00 · 49bc87c200
commit 49bc87c200
parent 9d9feccff7
9 changed files with 199 additions and 11 deletions
--- a/clusters/_template/bootstrap-kit/13-bp-catalyst-platform.yaml
+++ b/clusters/_template/bootstrap-kit/13-bp-catalyst-platform.yaml
@ -809,7 +809,7 @@ spec:
      # SHA changes (bp-catalyst-platform embeds the built marketplace
      # assets via that image). Threading customer-chosen values into
      # the install POST is a follow-up (TBD-V18-D).
-      version: 1.4.224
+      version: 1.4.225
      sourceRef:
        kind: HelmRepository
        name: bp-catalyst-platform
--- a/clusters/_template/bootstrap-kit/80-newapi.yaml
+++ b/clusters/_template/bootstrap-kit/80-newapi.yaml
@ -9,7 +9,12 @@
 # Catalyst signup hook (delivered by unified-rbac in #802 against the
 # contract recorded in ADR-0003) reads the `catalyst-newapi-admin-token`
 # Secret rendered by this chart's ExternalSecret to issue per-user API
-# keys against NewAPI's admin API at `http://newapi.newapi.svc`.
+# keys against NewAPI's admin API at
+# `http://newapi-bp-newapi.newapi.svc.cluster.local:3000` (canonical
+# in-cluster Service URL — the bp-newapi `<Release.Name>-<Chart.Name>`
+# helper renders `newapi-bp-newapi` for `releaseName: newapi` against
+# chart `bp-newapi`; pre-TBD-V15 / #2021 this comment cited the
+# wrong bare-`newapi` Service name).
 #
 # Wrapper chart: platform/newapi/chart/
 # Catalyst-curated values: platform/newapi/chart/values.yaml
@ -167,7 +172,14 @@ spec:
      # bp-sandbox 0.3.2 which mounts SIGNING_KEY as the MCP's
      # `SANDBOX_JWT_SECRET` env (closes auth-gate-stays-in-test-mode
      # silent-breakage).
-      version: 1.4.32
+      # 1.4.33 (TBD-V15 #2021, 2026-05-20): catalyst-newapi-admin-token
+      # ExternalSecret target now carries reflector mirror annotations
+      # (default to `catalyst-system`) so the rendered Secret is
+      # available in the catalyst-api Pod's namespace via secretKeyRef.
+      # Companion to bp-catalyst-platform 1.4.225 which adds the
+      # secretKeyRef itself + the corrected CATALYST_NEWAPI_ADDR
+      # literal (`http://newapi-bp-newapi.newapi.svc.cluster.local:3000`).
+      version: 1.4.33
      sourceRef:
        kind: HelmRepository
        name: bp-newapi
--- a/platform/newapi/chart/Chart.yaml
+++ b/platform/newapi/chart/Chart.yaml
@ -1,5 +1,23 @@
 apiVersion: v2
 name: bp-newapi
+# 1.4.33 — TBD-V15 / #2021 (2026-05-20, sister of TBD-V14 / PR #2017):
+# the `catalyst-newapi-admin-token` ExternalSecret now reflector-
+# mirrors the rendered Secret from `newapi` into `catalyst-system`
+# so the bp-catalyst-platform catalyst-api Pod can resolve
+# CATALYST_NEWAPI_ADMIN_TOKEN via secretKeyRef. Pre-fix the Secret
+# only existed in the `newapi` namespace, so the same-namespace-only
+# Kubernetes secretKeyRef on catalyst-api landed empty → main.go
+# guard short-circuited → POST /api/v1/sme/users returned
+# `newapi client not wired` (`sme_users.go:413`), blocking the
+# ADR-0003 §3.2 user-create hook. The companion fix in
+# bp-catalyst-platform 1.4.225 (TBD-V15) adds the secretKeyRef +
+# the corrected CATALYST_NEWAPI_ADDR literal
+# (`http://newapi-bp-newapi.newapi.svc.cluster.local:3000`).
+#
+# `catalystIntegration.reflectorNamespaces` knob (default
+# `catalyst-system`) lets operator overlays widen the mirror to
+# additional namespaces without rebuilding the OCI artifact
+# (docs/INVIOLABLE-PRINCIPLES.md #4).
 # 1.4.27 (TBD-A39 #1834, 2026-05-19): replace the Helm-`lookup`-based
 # DSN Secret render with a post-install/post-upgrade Job that polls
 # CNPG's `<cluster>-app` Secret until populated, composes the SQL_DSN
@ -335,7 +353,7 @@ name: bp-newapi
 # `lookup valkey.valkey.svc.cluster.local: no such host`). Same hostname
 # already documented as canonical in products/catalyst/chart/values.yaml
 # bp-cnpg-pair comments.
-version: 1.4.32
+version: 1.4.33
 appVersion: "0.13.2"
 description: |
  Catalyst Blueprint scratch chart for NewAPI — multi-tenant LLM
--- a/platform/newapi/chart/templates/external-secret.yaml
+++ b/platform/newapi/chart/templates/external-secret.yaml
@ -50,6 +50,35 @@ spec:
  target:
    name: {{ $secretName | quote }}
    creationPolicy: Owner
+    # Reflector mirror into `catalyst-system` (TBD-V15 / #2021, sister
+    # of TBD-V14 / #2017): the catalyst-api Pod (in catalyst-system)
+    # consumes ADMIN_API_TOKEN via secretKeyRef on
+    # `catalyst-newapi-admin-token` to drive the ADR-0003 §3.2
+    # user-create hook. Kubernetes secretKeyRef is same-namespace-only,
+    # so emberstack/reflector is the canonical cross-namespace seam
+    # (same pattern as sme-secrets, catalyst-gitea-token, cnpg-cluster
+    # secrets — see products/catalyst/chart/templates/sme-services/
+    # sme-secrets.yaml lines 207-227 for the canonical comment block).
+    #
+    # The target Secret carries `reflection-allowed=true` +
+    # `reflection-auto-enabled=true` so the destination Secret is
+    # materialised in catalyst-system on first reconcile and kept in
+    # sync on source rotation (the ExternalSecret refreshInterval
+    # rotates the source from OpenBao; reflector rotates the mirror
+    # from the source).
+    #
+    # allowed-namespaces is scoped to `catalyst-system` only — the
+    # bytes never leak to any unrelated namespace. Operator overlays
+    # MAY widen this list via .Values.catalystIntegration.
+    # reflectorNamespaces but defaults to the catalyst-system seam
+    # used by every Sovereign install.
+    template:
+      metadata:
+        annotations:
+          reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
+          reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: {{ $ci.reflectorNamespaces | default "catalyst-system" | quote }}
+          reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
+          reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: {{ $ci.reflectorNamespaces | default "catalyst-system" | quote }}
  data:
    - secretKey: ADMIN_API_TOKEN
      remoteRef:
--- a/platform/newapi/chart/values.yaml
+++ b/platform/newapi/chart/values.yaml
@ -589,6 +589,17 @@ catalystIntegration:
  # the operator on a schedule documented in
  # docs/CATALYST-CLI-AGENT.md.
  existingSecret: "catalyst-newapi-admin-token"
+  # Comma-separated list of namespaces emberstack/reflector mirrors the
+  # rendered Secret into (TBD-V15 / #2021). Default `catalyst-system`
+  # is the canonical install namespace of bp-catalyst-platform's
+  # catalyst-api Deployment, which consumes ADMIN_API_TOKEN via
+  # secretKeyRef as CATALYST_NEWAPI_ADMIN_TOKEN to drive the ADR-0003
+  # §3.2 user-create hook. The Kubernetes secretKeyRef is same-
+  # namespace-only, so the reflector mirror is the canonical cross-
+  # namespace seam (same pattern as sme-secrets in the bp-catalyst-
+  # platform chart). Operator overlays MAY widen this list when other
+  # controllers in other namespaces need the same bearer locally.
+  reflectorNamespaces: "catalyst-system"
  # ExternalSecret render — gated, default true. Sources the admin token
  # from the operator's OpenBao via a ClusterSecretStore (default
  # `vault-region1`, the bp-external-secrets-stores default). The remote
--- a/products/catalyst/bootstrap/api/cmd/api/main.go
+++ b/products/catalyst/bootstrap/api/cmd/api/main.go
@ -572,7 +572,18 @@ func main() {
 			// NewAPI admin client — wired only when both env vars are
 			// present (the bp-newapi blueprint provisions the
 			// catalyst-newapi-admin-token ExternalSecret per #799).
-			if addr := env("CATALYST_NEWAPI_ADDR", "http://newapi.newapi.svc"); addr != "" {
+			//
+			// Default URL fixed in TBD-V15 / #2021 (2026-05-20): pre-fix
+			// default `http://newapi.newapi.svc` was NXDOMAIN — same
+			// root cause as TBD-V14 / #2017 (bp-newapi.fullname helper
+			// renders `<Release.Name>-<Chart.Name>` = `newapi-bp-newapi`
+			// when releaseName=newapi against chart=bp-newapi per
+			// bootstrap-kit slot 80). Canonical in-cluster URL is
+			// `http://newapi-bp-newapi.newapi.svc.cluster.local:3000`.
+			// The bp-catalyst-platform chart now also exports
+			// CATALYST_NEWAPI_ADDR explicitly so the literal lives in
+			// values rather than this code default (belt-and-braces).
+			if addr := env("CATALYST_NEWAPI_ADDR", "http://newapi-bp-newapi.newapi.svc.cluster.local:3000"); addr != "" {
 				if token := os.Getenv("CATALYST_NEWAPI_ADMIN_TOKEN"); token != "" {
 					deps.NewAPIClient = newapi.New(addr, token)
 					log.Info("sme-users: NewAPI admin client wired", "addr", addr)
--- a/products/catalyst/bootstrap/api/internal/newapi/client.go
+++ b/products/catalyst/bootstrap/api/internal/newapi/client.go
@ -1,10 +1,14 @@
 // Package newapi is a minimal admin-API client for the in-cluster
 // NewAPI service consumed by the ADR-0003 user-create hook (step 2).
 //
-// Per ADR-0003 §3.2 the client targets `http://newapi.newapi.svc` —
-// the in-cluster Service DNS, never an Ingress hostname — with a
-// bearer token sourced from the `catalyst-newapi-admin-token`
+// Per ADR-0003 §3.2 the client targets the in-cluster Service DNS
+// `http://newapi-bp-newapi.newapi.svc.cluster.local:3000` (NOT the
+// Ingress hostname). The Service name is `<Release.Name>-<Chart.Name>`
+// per bp-newapi.fullname helper (releaseName=newapi against chart
+// bp-newapi per bootstrap-kit slot 80 → `newapi-bp-newapi`). Auth is
+// a bearer token sourced from the `catalyst-newapi-admin-token`
 // ExternalSecret rendered by the bp-newapi blueprint (issue #799).
+// Default URL corrected in TBD-V15 / #2021 (sister of TBD-V14 / #2017).
 //
 // Idempotency:
 //
@ -35,7 +39,8 @@ type Client struct {
 }

 // New returns a Client. addr is the in-cluster Service URL
-// (e.g. http://newapi.newapi.svc); token is the admin bearer.
+// (e.g. http://newapi-bp-newapi.newapi.svc.cluster.local:3000);
+// token is the admin bearer.
 func New(addr, token string) *Client {
 	return NewWithHTTP(addr, token, &http.Client{Timeout: 30 * time.Second})
 }
--- a/products/catalyst/chart/Chart.yaml
+++ b/products/catalyst/chart/Chart.yaml
@ -1,5 +1,47 @@
 apiVersion: v2
 name: bp-catalyst-platform
+# 1.4.225 — TBD-V15 / #2021 (2026-05-20, sister of TBD-V14 / PR #2017):
+# catalyst-api Deployment now wires CATALYST_NEWAPI_ADDR + the
+# CATALYST_NEWAPI_ADMIN_TOKEN secretKeyRef so the ADR-0003 §3.2
+# user-create hook (POST /api/v1/sme/users → applyHook step 2 →
+# POST /api/v1/admin/users on bp-newapi) actually fires. Pre-fix
+# BOTH env vars were unset → main.go startup logged
+# `CATALYST_NEWAPI_ADMIN_TOKEN unset; NewAPI hook step 2 will
+# fail-closed` and the operator UsersPage `Add User` flow returned
+# the sentinel `newapi client not wired` on every install.
+#
+# Fix:
+#   - api-deployment.yaml: literal `CATALYST_NEWAPI_ADDR=
+#     http://newapi-bp-newapi.newapi.svc.cluster.local:3000`
+#     (matches the bp-newapi Service name rendered by the
+#     `<Release.Name>-<Chart.Name>` helper per bootstrap-kit slot 80
+#     `releaseName: newapi` against chart `bp-newapi` — identical
+#     root cause as TBD-V14 / PR #2017 which fixed bp-sandbox's
+#     NEWAPI_BASE_URL default).
+#   - api-deployment.yaml: `CATALYST_NEWAPI_ADMIN_TOKEN` secretKeyRef
+#     on `catalyst-newapi-admin-token` (key ADMIN_API_TOKEN,
+#     `optional: true` so Sovereigns without an OpenBao seed fail
+#     closed rather than InvalidContainerConfig).
+#   - main.go: default URL in the `env("CATALYST_NEWAPI_ADDR", ...)`
+#     fallback updated to the canonical Service URL (belt-and-
+#     braces if the chart env var is ever dropped from a per-
+#     Sovereign overlay).
+#   - newapi/client.go godoc + main.go comment updated to the
+#     correct Service URL.
+#
+# Companion bump: bp-newapi 1.4.32 → 1.4.33 adds the reflector
+# mirror annotations on the ExternalSecret target so the rendered
+# Secret lands in `catalyst-system` (the same-namespace seam used
+# by every other Catalyst cross-namespace secret — see
+# templates/sme-services/sme-secrets.yaml for the canonical
+# pattern). Bootstrap-kit pins both charts in lockstep
+# (Inviolable Principle #14).
+#
+# DoD: PR closes #2021 only after a fresh-prov operator walks
+# `/console/sme/users` -> Add User and the catalyst-api log shows
+# `sme-users: NewAPI admin client wired` at startup + the row
+# materialises with `state=newapi_created` (no `newapi client not
+# wired` sentinel + no NXDOMAIN for `newapi.newapi.svc`).
 # 1.4.222 — TBD-V18 / #2026 (2026-05-20): marketplace AppDetail now
 # renders the per-instance configSchema (replicas / disk / backup
 # for Postgres-backed bundles, replicas / persistence for Redis, etc.)
@ -2070,8 +2112,8 @@ name: bp-catalyst-platform
 # was already shipped on 1.4.197 (PR #1820 lineage); this completes
 # the data-layer side so the dropdown finally appears on multi-region
 # Sovereigns. Refs #1821, DoD D20.
-version: 1.4.224
-appVersion: 1.4.224
+version: 1.4.225
+appVersion: 1.4.225
 # 1.4.183 — fix(httproute): omit default sectionName so multi-zone
 # Sovereigns attach via Cilium Gateway hostname matcher (Closes #1884,
 # TBD-A30). Pre-1.4.183 every catalyst-system HTTPRoute pinned
--- a/products/catalyst/chart/templates/api-deployment.yaml
+++ b/products/catalyst/chart/templates/api-deployment.yaml
@ -627,6 +627,66 @@ spec:
                  name: sme-secrets
                  key: JWT_SECRET
                  optional: true
+            # ── NewAPI admin client (ADR-0003 §3.2, TBD-V15 / #2021) ─────
+            # CATALYST_NEWAPI_ADDR + CATALYST_NEWAPI_ADMIN_TOKEN feed the
+            # NewAPI admin-API client in
+            # products/catalyst/bootstrap/api/cmd/api/main.go around the
+            # `if addr != "" && token != ""` guard. Both env vars must
+            # land non-empty for the SME-user create hook (step 2:
+            # POST /api/v1/admin/users) to actually fire — when token
+            # is empty catalyst-api logs `CATALYST_NEWAPI_ADMIN_TOKEN
+            # unset; NewAPI hook step 2 will fail-closed` at startup
+            # and `POST /api/v1/sme/users` returns terminal
+            # `newapi client not wired`. Pre-fix BOTH envs were unset
+            # by default → the operator UsersPage `Add User` flow on
+            # `/console/sme/users` never actually provisioned an
+            # upstream NewAPI user, even though the surface returned
+            # 200 (with a sentinel error in LastError).
+            #
+            # CATALYST_NEWAPI_ADDR — canonical in-cluster Service URL.
+            # `newapi-bp-newapi` (NOT `newapi`) because the bp-newapi
+            # chart helper renders `<Release.Name>-<Chart.Name>` and
+            # bootstrap-kit slot 80 sets `releaseName: newapi` against
+            # chart `bp-newapi`. Same root cause as TBD-V14 / PR #2017
+            # which fixed the identical mistake in bp-sandbox's
+            # NEWAPI_BASE_URL default.
+            #
+            # LITERAL value (not Helm template directive) — dual-mode
+            # contract: this file is consumed by BOTH Helm AND Kustomize
+            # (contabo-mkt's clusters/contabo-mkt/apps/catalyst-platform).
+            # Helm directives in `value:` fields break the Kustomize
+            # build (`yaml: invalid map key`). Operator override path
+            # is the `catalystApi.env` additional-env patch on the
+            # per-Sovereign HelmRelease overlay (Helm-only codepath,
+            # takes precedence at template-render time).
+            - name: CATALYST_NEWAPI_ADDR
+              value: "http://newapi-bp-newapi.newapi.svc.cluster.local:3000"
+            # CATALYST_NEWAPI_ADMIN_TOKEN — admin bearer from the
+            # `catalyst-newapi-admin-token` Secret rendered by the
+            # bp-newapi chart's ExternalSecret (key ADMIN_API_TOKEN,
+            # source ClusterSecretStore `vault-region1`, OpenBao path
+            # `sovereign/<sovereign-fqdn>/newapi/admin-token`).
+            #
+            # Same cross-namespace pattern as `sme-secrets` above: the
+            # ExternalSecret lands the Secret in the `newapi` namespace
+            # (bp-newapi targetNamespace), and emberstack/reflector
+            # mirrors it into catalyst-system. The mirror annotations
+            # live on the bp-newapi-side ExternalSecret.target.template
+            # so the catalyst-system copy resolves the secretKeyRef
+            # below at Pod-start.
+            #
+            # optional: true — Sovereigns that haven't yet seeded the
+            # OpenBao path won't have the admin token materialised; in
+            # that case the env var lands empty, the main.go guard
+            # short-circuits, and `POST /api/v1/sme/users` surfaces
+            # `newapi client not wired` to the operator UI (intended
+            # fail-closed behaviour per ADR-0003 §3.2).
+            - name: CATALYST_NEWAPI_ADMIN_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: catalyst-newapi-admin-token
+                  key: ADMIN_API_TOKEN
+                  optional: true
            # KEYCLOAK_BOOTSTRAP_TIER_ROLES — EPIC-3 slice T2 (#1098/#1146).
            # When "true", a goroutine on catalyst-api startup calls
            # EnsureTierRealmRoles (internal/keycloak/realm_bootstrap.go)