Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,14 @@ body:
description: Include the Helm command and values file.
validations:
required: true

- type: textarea
id: continuation
attributes:
label: Continuation Prompt
description: Write the prompt a future contributor or agent can use if this bug fix needs follow-up work.
placeholder: |
Fix ...
Validate by ...
Update evidence in ...
validations:
required: false
19 changes: 18 additions & 1 deletion .github/ISSUE_TEMPLATE/feature.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,21 @@ body:
description: List concrete checks that make this issue done.
validations:
required: true

- type: textarea
id: evidence
attributes:
label: Evidence Update
description: What README, docs, roadmap, or evidence-map updates should accompany the implementation?
validations:
required: false
- type: textarea
id: continuation
attributes:
label: Continuation Prompt
description: Write the prompt a future contributor or agent can use to continue this delivery node.
placeholder: |
Implement ...
Validate by ...
Update evidence in ...
validations:
required: true
19 changes: 18 additions & 1 deletion .github/ISSUE_TEMPLATE/task.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,21 @@ body:
label: Acceptance Criteria
validations:
required: true

- type: textarea
id: evidence
attributes:
label: Evidence Update
description: What README, docs, roadmap, or evidence-map updates should accompany this task?
validations:
required: false
- type: textarea
id: continuation
attributes:
label: Continuation Prompt
description: Write the prompt a future contributor or agent can use to continue this delivery node.
placeholder: |
Implement ...
Validate by ...
Update evidence in ...
validations:
required: true
14 changes: 13 additions & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,19 @@

-

## Evidence Update

- [ ] README, docs, roadmap, or evidence map updated
- [ ] Not applicable because:

## Continuation Prompt

A future contributor or agent should be able to continue with:

> Implement ...
> Validate by ...
> Update evidence in ...

## Security Notes

- No customer assets, renderer binaries, license files, credentials, or cloud account secrets are included.

19 changes: 18 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,19 @@ Reusable Kubernetes building blocks for GPU-backed render worker orchestration o

This repository is a public engineering reconstruction of a render-farm orchestration layer. It does not contain employer source code, customer assets, V-Ray binaries, license files, cloud account identifiers, or production secrets.

## What This Demonstrates

This project demonstrates production-shaped cloud infrastructure work in a public-safe form: reusable Helm packaging, queue-driven worker orchestration, GPU scheduling controls, AWS/EKS integration points, CI validation, and explicit public/private safety boundaries.

For reviewers, the intended signal is not a toy renderer. It is the orchestration layer around private render workers: how workloads are packaged, scaled, scheduled, validated, documented, and separated from proprietary assets.

## Scope

The first reusable component is a Helm chart for render worker orchestration:

- GPU-backed worker `Deployment` for render workloads.
- Optional KEDA `ScaledObject` for SQS backlog driven scaling.
- Optional KEDA `TriggerAuthentication` for pod-identity based SQS scaler access.
- Kubernetes `Job` template for post-render processing.
- `ServiceAccount` annotations for IRSA or EKS Pod Identity style integrations.
- ConfigMap and Secret injection points.
Expand Down Expand Up @@ -53,6 +60,13 @@ helm template render-worker charts/render-worker -f examples/values-aws-gpu-sqs.

The default chart is safe for public review: KEDA is disabled in `values.yaml`, no secrets are populated, and no cloud resources are created by Helm.

## Reviewer Path

1. Run `make lint` and `make template`.
2. Inspect [docs/evidence-map.md](docs/evidence-map.md) for claim-to-file references.
3. Read [docs/architecture.md](docs/architecture.md) for scope boundaries and production integrations.
4. Check open issues for the next hardening nodes and merged pull requests for review history.

## Development Workflow

This repo uses GitHub features intentionally:
Expand All @@ -64,6 +78,10 @@ This repo uses GitHub features intentionally:

See [docs/development.md](docs/development.md) and [docs/roadmap.md](docs/roadmap.md).

## Delivery Nodes

Future work is tracked as GitHub issues and written so another contributor or agent can continue from a clean handoff. See [docs/delivery-prompts.md](docs/delivery-prompts.md) for continuation prompts.

## Resume Evidence

See [docs/evidence-map.md](docs/evidence-map.md) for how this repository maps to Kubernetes, EKS, GPU worker orchestration, S3 asset flows, KEDA/SQS scaling, and CI/CD resume claims.
Expand All @@ -74,4 +92,3 @@ See [docs/evidence-map.md](docs/evidence-map.md) for how this repository maps to
- Do not commit V-Ray binaries or license files.
- Do not commit AWS access keys or account identifiers.
- Keep production-specific values in private deployment repos or secret managers.

5 changes: 3 additions & 2 deletions charts/render-worker/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ metadata:
{{- include "render-worker.labels" . | nindent 4 }}
data:
{{- if .Values.configMap.data }}
{{- toYaml .Values.configMap.data | nindent 2 }}
{{- range $key, $value := .Values.configMap.data }}
{{ $key }}: {{ printf "%v" $value | quote }}
{{- end }}
{{- else }}
RENDER_PROFILE: "default"
{{- end }}
{{- end }}

2 changes: 2 additions & 0 deletions charts/render-worker/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ metadata:
labels:
{{- include "render-worker.labels" . | nindent 4 }}
spec:
{{- if not .Values.keda.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "render-worker.selectorLabels" . | nindent 6 }}
Expand Down
5 changes: 4 additions & 1 deletion charts/render-worker/templates/keda-scaledobject.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,8 @@ spec:
queueURLFromEnv: {{ .Values.keda.trigger.queueURLFromEnv | quote }}
queueLength: {{ .Values.keda.queueLength | quote }}
awsRegion: {{ .Values.keda.trigger.awsRegion | quote }}
{{- if .Values.keda.authentication.enabled }}
authenticationRef:
name: {{ default (printf "%s-sqs-auth" (include "render-worker.fullname" .)) .Values.keda.authentication.name }}
{{- end }}
{{- end }}

12 changes: 12 additions & 0 deletions charts/render-worker/templates/keda-triggerauthentication.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{{- if and .Values.keda.enabled .Values.keda.authentication.enabled }}
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: {{ default (printf "%s-sqs-auth" (include "render-worker.fullname" .)) .Values.keda.authentication.name }}
namespace: {{ .Values.namespace.name }}
labels:
{{- include "render-worker.labels" . | nindent 4 }}
spec:
podIdentity:
provider: {{ .Values.keda.authentication.podIdentity.provider | quote }}
{{- end }}
5 changes: 3 additions & 2 deletions charts/render-worker/templates/secret.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ metadata:
{{- include "render-worker.labels" . | nindent 4 }}
type: Opaque
stringData:
{{- toYaml .Values.secret.stringData | nindent 2 }}
{{- range $key, $value := .Values.secret.stringData }}
{{ $key }}: {{ printf "%v" $value | quote }}
{{- end }}
{{- end }}

20 changes: 20 additions & 0 deletions charts/render-worker/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,26 @@
"queueLength": {
"type": "string",
"minLength": 1
},
"authentication": {
"type": "object",
"properties": {
"enabled": {
"type": "boolean"
},
"name": {
"type": "string"
},
"podIdentity": {
"type": "object",
"properties": {
"provider": {
"type": "string",
"minLength": 1
}
}
}
}
}
}
},
Expand Down
6 changes: 5 additions & 1 deletion charts/render-worker/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ keda:
pollingInterval: 10
cooldownPeriod: 300
queueLength: "1"
authentication:
enabled: false
name: ""
podIdentity:
provider: aws
trigger:
type: aws-sqs-queue
queueURLFromEnv: QUEUE_URL
Expand Down Expand Up @@ -101,4 +106,3 @@ postprocessJob:
memory: 4Gi
ttlSecondsAfterFinished: 7200
backoffLimit: 1

9 changes: 8 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ It owns reusable Kubernetes primitives:
- ConfigMap and Secret injection points.
- Post-processing `Job` template.

## Design Constraints and Tradeoffs

- The chart treats renderer images as private adapters. Public examples use placeholders because renderer binaries, license files, and customer scenes must stay outside this repository.
- KEDA owns runtime worker replica count when autoscaling is enabled. The chart omits `Deployment.spec.replicas` in that mode so Helm upgrades do not reset an active worker pool.
- KEDA scaler authentication is modeled separately from worker AWS access. The worker ServiceAccount handles job execution permissions; `TriggerAuthentication` handles scaler access to queue metrics.
- S3/SQS/EKS names in examples are placeholders. Production infrastructure, IAM policies, and cloud resources are expected to live in private environment repositories.
- Post-processing is currently represented as a chart-rendered `Job`; its production upgrade semantics are tracked separately because Kubernetes Job pod templates are immutable after creation.

## Workload Flow

```text
Expand Down Expand Up @@ -47,4 +55,3 @@ Typical production integrations include:
## Public Repository Boundary

The public repository uses placeholders and examples. Real renderer binaries, license files, customer scenes, cloud account ids, and production values belong in private systems.

94 changes: 94 additions & 0 deletions docs/delivery-prompts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Delivery Prompts

Use these prompts to continue development from a fresh checkout. Each prompt is scoped to one GitHub issue and one pull request.

## Node: Chart Adapter Seams

```text
Repository: RayLi-Muye/renderfarm-k8s-toolkit
Issue: #12 Define chart adapter seams for queue, storage, and identity

Implement a deeper public values interface for queue, storage, and identity adapters without breaking the current AWS GPU/SQS example.

Expected files:
- charts/render-worker/values.yaml
- charts/render-worker/values.schema.json
- charts/render-worker/templates/*
- examples/*
- docs/architecture.md
- docs/evidence-map.md

Validation:
- helm lint charts/render-worker
- helm template render-worker charts/render-worker
- helm template render-worker charts/render-worker -f examples/values-local.yaml
- helm template render-worker charts/render-worker -f examples/values-aws-gpu-sqs.yaml

Evidence update:
- Update docs/evidence-map.md with adapter claims, verification commands, status, and boundaries.
```

## Node: Kubeconform and Install Integrity

```text
Repository: RayLi-Muye/renderfarm-k8s-toolkit
Issue: #15 Add kubeconform validation and Helm install integrity checks

Add Kubernetes schema validation and Helm install integrity checks to GitHub Actions.

Expected files:
- .github/workflows/ci.yml
- scripts/render-examples.sh if useful
- docs/development.md
- docs/evidence-map.md

Validation:
- CI passes on a PR.
- The workflow validates default, local, and AWS GPU/SQS rendered manifests.
- Helm installation is pinned and integrity-checked.

Evidence update:
- Update docs/evidence-map.md for schema validation and CI hardening.
```

## Node: Mock Render Worker

```text
Repository: RayLi-Muye/renderfarm-k8s-toolkit
Issue: #7 Add mock render worker and local smoke test path

Add a public mock worker path that exercises the chart contract without proprietary renderer binaries.

Expected files:
- workers/mock-render-worker/*
- examples/*
- charts/render-worker/values.yaml if new env contract is needed
- .github/workflows/ci.yml if the smoke test runs in CI
- docs/architecture.md
- docs/evidence-map.md

Validation:
- Local smoke test runs without AWS credentials.
- CI runs at least one lightweight mock path if practical.
- README and evidence map clearly distinguish the mock worker from private production renderer images.
```

## Node: Security Hardening

```text
Repository: RayLi-Muye/renderfarm-k8s-toolkit
Issue: #5 Add security hardening examples for AWS permissions and secrets

Add public-safe security examples for AWS permissions, secrets, and network isolation.

Expected files:
- examples/*
- charts/render-worker/templates/*
- docs/architecture.md
- docs/evidence-map.md

Validation:
- Helm lint and template rendering pass.
- Security examples use placeholders only.
- Docs explain where production IAM, Secrets Manager, External Secrets, and NetworkPolicy decisions live.
```
12 changes: 12 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ Every PR should include:
- Acceptance criteria.
- Rendered manifest evidence, when Helm templates changed.
- Security notes when touching Secrets, ServiceAccounts, or IAM-related annotations.
- Evidence-map updates when the PR changes reviewer-facing claims.
- A continuation prompt for the next delivery node.

## Issue Workflow

Expand All @@ -43,3 +45,13 @@ Issues should be small enough to implement and review independently. Prefer one
- Documentation.
- Security hardening.

## Loop Engineering

The preferred loop is:

```text
issue -> branch -> implementation -> local validation -> PR -> GitHub Actions -> merge -> evidence update -> next issue
```

Do not batch unrelated hardening, demo, and documentation work into one pull request unless the evidence update depends on all of them.

Loading