Kubeflow on the Homelab — What Changed from Upstream

Dex → Keycloak + oauth2-proxy · SeaweedFS/MinIO → Ceph RGW · own Istio → shared mesh · + MLflow · Author: George Ezejiofor

Kubeflow on the Homelab — What Changed from Upstream Dex → Keycloak + oauth2-proxy · SeaweedFS/MinIO → Ceph RGW · own Istio → shared mesh · + MLflow · Author: George Ezejiofor People · 3 Profiles (namespaces) · Architecture component People 3 Profiles (namespaces) kubeflow-gateway · shared Istio · Let's Encrypt · Architecture component · was: own Istio kubeflow-gateway shared Istio · Let's Encrypt was: own Istio oauth2-proxy · reverse proxy · 10 upstreams · SSO — replaces Dex · reworked oauth2-proxy reverse proxy · 10 upstreams reworked Keycloak · realm kubeflow · CNPG · SSO — replaces Dex · replaces Dex Keycloak realm kubeflow · CNPG replaces Dex Google · upstream identity · Architecture component Google upstream identity Central dashboard · front door · MLflow tile · kubeflow · user header = X-Forwarded-Email · added tiles Central dashboard front door · MLflow tile added tiles Katib · TPE HPO · katib-mysql · kubeflow · user header = X-Forwarded-Email Katib TPE HPO · katib-mysql Notebooks · Jupyter · CUDA image · kubeflow · user header = X-Forwarded-Email · ext_authz /notebook Notebooks Jupyter · CUDA image ext_authz /notebook Pipelines 2.15 · Argo · 5 schedules · kubeflow · user header = X-Forwarded-Email Pipelines 2.15 Argo · 5 schedules Model Registry · own Postgres · kubeflow · user header = X-Forwarded-Email Model Registry own Postgres KServe · Knative serverless · kubeflow · user header = X-Forwarded-Email KServe Knative serverless 3× RTX A1000 · NVIDIA gpu-operator · Architecture component · added 3× RTX A1000 NVIDIA gpu-operator added MySQL (bundled) · mlpipeline · metadb · Architecture component · CNPG planned MySQL (bundled) mlpipeline · metadb CNPG planned Ceph RGW · s3.georgehomelab.com · Architecture component · replaces SeaweedFS Ceph RGW s3.georgehomelab.com replaces SeaweedFS MLflow 3.12 · ns mlops · own SSO · mlops · mlflow.georgehomelab.com · added MLflow 3.12 ns mlops · own SSO added CloudNativePG · mlflow_backend_db · mlops · mlflow.georgehomelab.com CloudNativePG mlflow_backend_db https all paths OIDC brokered X-Forwarded-Email /katib/ /jupyter/ /pipeline/ /model-registry/ /kserve-endpoints/ nvidia.com/gpu: 1 runs · MLMD artifacts model.bst external tile runs · params artifacts SSO — replaces Dex kubeflow · user header = X-Forwarded-Email mlops · mlflow.georgehomelab.com Legend Frontend Backend Database Cloud Security External

Auth swaps

  • • No Dex: Keycloak realm kubeflow, Google as the IdP
  • • oauth2-proxy in reverse-proxy mode hands every request to the Central dashboard first
  • • Apps sit behind it on their own paths; header = X-Forwarded-Email everywhere
  • • Notebook servers (/notebook/*) are also checked by a CUSTOM ext_authz policy

Platform swaps

  • • KFP artifacts: OBC kfp-artifacts on Ceph RGW (SeaweedFS removed)
  • • Shared cluster Istio + cert-manager; PERMISSIVE mesh, STRICT on kubeflow-sso
  • • Secrets from Vault via VSO; every PVC on ceph-block; one ArgoCD AppSet, ~23 apps
  • • MLflow 3.12 in mlops: runs on CNPG, artifacts on RGW; pipeline steps log to mlflow-mlflow.mlops:5000

Still upstream

  • • KFP + ML Metadata on bundled MySQL (CNPG database exists, unused)
  • • Katib on katib-mysql; Model Registry on its own Postgres
  • • Trainer v2 installed, no TrainJobs yet