👻 Ghost-Pay

Infrastructure-Level Automated Payments — Push payment logic down to the infrastructure layer

Overview

Ghost-Pay demonstrates how to implement an automated x402 payment flow using Kubernetes-native architecture and Istio Service Mesh. By pushing payment logic down to the infrastructure layer, business agents can complete value exchange with zero awareness.

Two independent agents — deployed in separate Kubernetes namespaces — each purchase different types of paid intelligence via x402 micropayments, all settled transparently by the Istio sidecar:

Agent	Namespace	Purpose
Ghost-News-Agent	news	Crypto market intelligence — buys deep analysis, on-chain signals, competitor reports
Ghost-Security-Agent	security	Cybersecurity — buys IP threat checks, CVE bulletins, active threat feeds
Ghost-Paymaster	ghost-pay	Central payment service — signs x402 USDC payments for all namespaces

┌────────────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster (kind)                       │
│                                                                    │
│  namespace: news              namespace: ghost-pay                 │
│  ┌────────────────┐           ┌────────────────┐  ┌─────────────┐  │
│  │ Ghost-News-    │           │ Ghost-Paymaster│  │ x402        │  │
│  │ Agent          │──────────▶│ Agent          │  │ Provider    │  │
│  │ ┌────────────┐ │  /v1/pay  │ /v1/pay        │  │ (BASE       │  │
│  │ │ Lua Filter │─┼──────────▶│                │  │  Sepolia)   │  │
│  │ └────────────┘ │           └────────────────┘  └──────┬──────┘  │
│  └────────┬───────┘                                      │         │
│           └──────────────────────────────────────────────┘         │
│  namespace: security                                               │
│  ┌────────────────┐                                                │
│  │ Ghost-Security-│──────────▶  (same Paymaster + Provider)        │
│  │ Agent          │                                                │
│  │ ┌────────────┐ │                                                │
│  │ │ Lua Filter │─┼──────────▶                                     │
│  │ └────────────┘ │                                                │
│  └────────────────┘                                                │
│                  402 → pay → retry (auto, per namespace)           │
└────────────────────────────────────────────────────────────────────┘

Automated Payment Flow (x402)

Step	Initiator	Receiver	Action	Protocol
1	Ghost-News/Security-Agent	x402 Provider	GET /premium-resource	HTTP
2	x402 Provider	Agent sidecar	402 + PAYMENT-REQUIRED header	x402 v2
3	Lua Filter	Ghost-Paymaster	POST /v1/pay (payment_required_header)	HTTP
4	Ghost-Paymaster	(local signing)	Sign USDC payment with EVM key	x402 SDK
5	Ghost-Paymaster	Lua Filter	Return payment_signature	HTTP
6	Lua Filter	x402 Provider	GET /premium-resource + PAYMENT-SIGNATURE	x402 v2
7	x402 Provider	x402 Facilitator	Verify & settle on BASE testnet	x402/EVM
8	x402 Provider	Agent	200 OK + PAYMENT-RESPONSE header	HTTP

Tech Stack

• Local Kubernetes: kind
• Service Mesh: Istio
• Agent Framework: Google ADK
• Payment Protocol: x402 (v2 — PAYMENT-REQUIRED / PAYMENT-SIGNATURE headers)
• Network: BASE Sepolia testnet (eip155:84532)
• Token: USDC (testnet)
• Facilitator: x402.org/facilitator
• Sidecar Filter: Istio Lua EnvoyFilter (Envoy)

Project Structure

ghost-pay/
├── AGENTS.md                      # Agent definitions and interaction spec
├── Makefile                       # Automation commands
├── kind/
│   └── cluster-config.yaml        # Kind cluster configuration
├── agents/
│   ├── news/                      # Ghost-News-Agent — crypto market intelligence
│   │   ├── Dockerfile
│   │   ├── agent.py               # ADK Agent: market reports, analysis, competitor intel
│   │   ├── tools.py               # HTTP resource request tools
│   │   └── main.py                # FastAPI service + chat UI
│   ├── security/                  # Ghost-Security-Agent — cybersecurity threat intel
│   │   ├── Dockerfile
│   │   ├── agent.py               # ADK Agent: IP threat check, CVE bulletins
│   │   ├── tools.py               # check_ip_threat + request_resource tools
│   │   └── main.py                # FastAPI service + chat UI
│   └── paymaster/                 # Ghost-Paymaster — central payment service
│       ├── Dockerfile
│       ├── tools.py               # x402 payment signing / quota management
│       └── main.py                # FastAPI service (/v1/pay, /dashboard)
├── x402-provider/                 # Real x402 paid API (BASE testnet USDC)
│   ├── Dockerfile
│   └── main.py                    # Market intel + threat intel endpoints
├── k8s/                           # Kubernetes manifests
│   ├── namespace.yaml             # ghost-pay, news, security namespaces
│   ├── config.yaml                # ConfigMap (prices, facilitator URL)
│   ├── secrets.yaml.example       # Secret template
│   ├── news/deployment.yaml       # Ghost-News-Agent (namespace: news)
│   ├── security/deployment.yaml   # Ghost-Security-Agent (namespace: security)
│   ├── paymaster/
│   │   ├── configmap.yaml         # Per-namespace budgets
│   │   └── deployment.yaml
│   ├── x402-provider/deployment.yaml
│   └── istio/
│       ├── peer-authentication.yaml
│       ├── authorization-policy.yaml  # news + security → paymaster /v1/pay
│       ├── destination-rule.yaml
│       ├── lua-filter-news.yaml       # 402 interceptor for ghost-news-agent
│       └── lua-filter-security.yaml   # 402 interceptor for ghost-security-agent
└── scripts/
    ├── setup.sh
    ├── build.sh
    ├── deploy.sh
    ├── demo.sh
    └── teardown.sh

x402 Endpoints

Market Intelligence (Ghost-News-Agent)

Endpoint	Price	Description
GET /public/market-overview	Free	General crypto market overview
GET /premium/deep-analysis	$0.003 USDC	On-chain signals & investment recommendation
GET /premium/competitor-intelligence	$0.005 USDC	Competitor landscape & market share
GET /premium/ultra-premium-report	$20.00 USDC	Classified intelligence (triggers budget protection)

Threat Intelligence (Ghost-Security-Agent)

Endpoint	Price	Description
GET /public/threat-feed	Free	Known malicious IPs & active campaigns
GET /security/ip-threat-check?ip=<IP>	$0.002 USDC	IP reputation score, threat category & risk assessment
GET /security/vulnerability-bulletin	$0.004 USDC	Latest CVE bulletins & active exploit campaigns

Namespace Budgets

Managed by Ghost-Paymaster via k8s/paymaster/configmap.yaml:

Namespace	Budget	Consumer
news	$2.00 USDC	Ghost-News-Agent
security	$1.00 USDC	Ghost-Security-Agent
ghost-pay	$1.00 USDC	Internal / testing

Quick Start

Prerequisites

• Docker
• kind
• kubectl
• istioctl
• EVM wallet with BASE Sepolia testnet USDC — get from CDP Faucet

1. Configure Secrets

cp k8s/secrets.yaml.example k8s/secrets.yaml  # Fill in base64-encoded key/address

2. One-Command Setup

# Full flow: create cluster → build images → deploy
make all

# Or step by step:
make setup     # Create kind cluster + install Istio
make build     # Build all Docker images (news, security, paymaster, x402-provider)
make deploy    # Apply ConfigMap, Secrets, and deploy to K8s

Run the Demo

make demo      # Run end-to-end x402 USDC payment flow demo

Check Status

make status            # Cluster and Pod status (all namespaces)
make logs-news         # Ghost-News-Agent logs
make logs-security     # Ghost-Security-Agent logs
make logs-paymaster    # Ghost-Paymaster logs
make logs-provider     # x402-provider logs

Teardown

make teardown  # Delete cluster and images

Security

• mTLS: All inter-Pod communication enforces Istio mutual TLS
• Authorization: Only agents in news and security namespaces may call Paymaster's /v1/pay
• Budget isolation: Each namespace has an independent spending quota enforced by the Paymaster
• Secrets: EVM private key is stored in a K8s Secret, never in ConfigMap or images
• Audit: All payment transactions are recorded in Paymaster's transaction ledger (/v1/transactions)

TODO / Enhancements

💸 Payments & x402 Protocol

• Wallet balance from chain — Paymaster's initial wallet balance is currently hardcoded in ConfigMap; it should query the actual on-chain balance via RPC instead
• Finer-grained cost policy — Current policy is per-namespace budget only; add per-payment limit, per-endpoint price cap, rate limiting, etc.
• x402 multi-chain support — Currently hardcoded to BASE Sepolia (eip155:84532); extend Paymaster and x402-provider to support additional EVM chains (e.g. Ethereum mainnet, Polygon, Arbitrum) configurable per namespace or per endpoint
• x402 Bazaar discovery integration (docs) — Agents currently hit hardcoded endpoint URLs. Integrate the x402 Bazaar discovery extension so resources are discovered dynamically instead: (1) Server side — declare the paid x402-provider routes as discoverable by attaching the bazaar extension to their route config, so the facilitator catalogs them; (2) Buyer side — have the news/security agents (or the Paymaster on their behalf) query the facilitator's GET /discovery/resources endpoint via with_bazaar(HTTPFacilitatorClient(...)) (x402.extensions.bazaar) to list available services, then filter by type=http, price cap (e.g. maxAmountRequired), and network before issuing a request; (3) Paymaster — optionally cross-check the discovered resource metadata against namespace budgets to pre-validate affordability before signing
• Concurrent payment failure on testnet facilitator — Under concurrent requests (e.g. multiple IP threat checks fired simultaneously), the public x402.org/facilitator returns invalid_exact_evm_transaction_failed because its testnet submission wallet does not manage nonces under concurrency — concurrent transferWithAuthorization broadcasts collide in the mempool. This is a testnet-only limitation; a production-grade facilitator (e.g. Coinbase's production x402 facilitator) handles concurrent settlement correctly via proper nonce queuing and multi-wallet pools. No code change is needed for production deployment.
• Lua filter: general x402 retry (method + body + headers) — Current retry in the Lua filter is hardcoded to GET with no body or custom headers. x402 spec requires replaying the original request + PAYMENT-SIGNATURE. For providers that use POST or require API keys / custom Accept headers, the retry silently drops them. A general implementation requires buffering the original method, body, and safe request headers in envoy_on_request dynamic metadata (subject to a configurable size limit), or migrating to a WASM filter / egress gateway pattern.
• Robust JSON parsing for settlement responses — The Lua filter's json_get_str uses a minimal regex that truncates on escaped quotes within string values. Safe for paymaster responses (self-controlled schema) but fragile for provider settlement responses. Replace with a proper Lua JSON library (e.g. dkjson) injected at build time, or Envoy's built-in cjson module if available in the deployed Envoy version.

🛡️ Reliability & Resilience

• Persistent per-namespace memory — Paymaster spending state is in-memory only; it should be backed by an external DB (e.g. PostgreSQL / Redis) for durability across restarts
• Circuit breaker for Paymaster calls — Lua filter has no circuit breaker or retry; if Paymaster becomes unavailable, every agent request will timeout and cascade-fail until recovery
• Rate limiting on /v1/pay — No backpressure on the Paymaster endpoint; a misbehaving agent could flood it with rapid payment requests
• Bounded transaction ledger — In-memory _tx_ledger grows unbounded; add a max-size cap or TTL-based eviction to prevent paymaster OOM in long-running deployments
• Lua filter httpCall timeout tuning — The payment path blocks the downstream stream for up to 30 s (Paymaster) + 15 s (provider retry) + 5 s (explorer update) = 50 s worst-case. Under heavy concurrent 402 traffic each Envoy worker coroutine is held for the full round-trip. Tighten timeouts based on measured p99 latencies and add a pass-through fallback on Paymaster timeout (rather than hard 502) to bound blast radius.

🔒 Security & Access Control

• Secure key management — Wallet private key should not be stored in a Kubernetes Secret; migrate to a dedicated secrets manager such as AWS KMS, AWS Secrets Manager, or HashiCorp Vault
• Dashboard & transaction endpoint authentication — /dashboard, /v1/transactions, /v1/spending are currently unauthenticated; add bearer token or restrict via Istio AuthorizationPolicy

📊 Observability & Audit

• Block confirmations on dashboard — Paymaster's /dashboard currently shows only tx_hash and explorer_url; call the BASE Sepolia RPC (eth_getTransactionReceipt + eth_blockNumber) to compute confirmations = latestBlock - txBlockNumber and display it alongside each transaction row, refreshing every ~2 s via a lightweight polling endpoint (e.g. GET /v1/transactions/{tx_id}/confirmations)
• Metrics, alerting & Grafana dashboard — Expose Prometheus metrics for payment volume, failure rate, quota usage; wire up Grafana dashboards and alerting rules
• Distributed tracing & correlation IDs — No X-Request-ID / W3C traceparent propagation across agents and Lua filter; makes cross-namespace debugging very difficult
• OpenTelemetry & financial-grade audit — Integrate OTel SDK into all agents and x402-provider; deploy OTel Collector + Jaeger/Tempo in ghost-pay namespace; instrument /v1/pay with payment.process spans (attributes: tx_id, namespace, amount_usd, signature_hash, chain_id); add HMAC-SHA256 integrity hash per span and append-only OTLP log export for non-repudiation; expose payment_total, payment_amount_usd_total, payment_duration_seconds Prometheus metrics with Grafana dashboard and alerting rules

🚀 Developer Experience & Ops

• Helm chart — Package all Kubernetes manifests as a Helm chart for easier deployment and configuration management
• Integration test suite — No automated tests for happy path, quota exhaustion, signature failure, or provider unavailability scenarios
• OpenAPI spec for agent APIs — No formal spec for /v1/pay, /v1/transactions, or the internal payment endpoints; add OpenAPI/Swagger definitions to ease client development and contract testing
• Pin dependency versions — requirements.txt uses loose version constraints (e.g. >=); pin to exact versions (==) for reproducible builds

🌐 Networking

• External x402 provider support — derive_cluster in the Lua filter incorrectly appends .svc.cluster.local to any hostname containing a dot (e.g. api.example.com becomes api.example.com.svc.cluster.local). Real-world x402 providers are almost always off-cluster. Fix: check for an existing .svc.cluster.local suffix before appending; treat any other multi-label hostname as external. External providers additionally require an Istio ServiceEntry and must be within scope of the Sidecar resource's egress rules — a namespace-scoped Sidecar that restricts egress will silently drop calls to both the external provider and the Paymaster cluster.

Related Links

• Coinbase x402 Official Docs
• x402 Python SDK
• x402-demo (Golang reference)
• CDP Faucet (Get Testnet USDC)
• Bazaar

License

Apache 2.0