Gravity Labs — Architecture Reference

00 / Users & Actors

Users & Actors

Four distinct user types interact with the Gravity Labs platform. Each has its own access model, primary surfaces, and operational footprint — from learners in isolated lab VMs to partners co-selling sponsored cohorts.

Primary persona · Browser

Participants / Learners

Learners access labs through the browser with least-privilege credentials. Each session receives a uniquely isolated cloud environment. Interaction flows through Guacamole (RDP/SSH) or Code-Server; the AI scoring agent evaluates submissions continuously and surfaces results without blocking the lab experience.

Surface: Participant app Lab access: Guacamole · Code-Server Scoring: Real-time AI agent

How to read this section. Narrative on the left explains intent; the reference rail mirrors API-style docs — fields, enums, and a compact schema — so you can scan like product or partner documentation.

Persona schema · Participants

persona required

enum participant

primary_surface

Participant web app (labs, submissions, status)

lab_access

Apache Guacamole (RDP/SSH) · Code-Server terminals

trust_tier

Tenant-scoped JWT · per-lab network isolation

{
  "persona": "participant",
  "session": "browser",
  "scoring": "ai_agent_streaming",
  "egress": ["guacamole", "codeserver"]
}

Scoring runs asynchronously; UX never blocks on model latency.

Persona schema · Trainers

persona required

enum trainer_evaluator

catalog_scope

Lab templates, cohort wiring, scoring rubric hooks

live_ops

Proctor View · participant timeline · anomaly flags

human_override

Recorded override on AI scores — full audit trail

{
  "persona": "trainer",
  "surfaces": ["catalog", "proctor"],
  "approval_workflow": "override_with_reason",
  "audit": "append_only"
}

Trainers reconcile automation with policy; overrides never delete machine output.

Persona schema · Admins

persona required

enum platform_admin

admin_plane

Tenant CRUD · subscriptions · infra health · event launch

hitl_gates

Approvals for blueprint, procurement, and pre-launch

blast_radius

Highest — changes fan out to all tenants in scope

{
  "persona": "admin",
  "rbac": "global_operator",
  "gates": ["G1_blueprint","G2_procurement","G3_launch"],
  "telemetry": "full_stack"
}

Use breaker policies: split prod changes from staging experimentation.

Persona schema · Sponsors

persona required

enum sponsor_partner

visibility

Cohort usage · seat burn · branded leaderboard (read-heavy)

billing_ref

Linked sponsor account · invoice segmentation

data_exposure

Aggregates by default — no raw participant PII

{
  "persona": "sponsor",
  "access": "read_predominant",
  "reports": ["usage","outcomes","brand_kit"],
  "pii": "aggregated_only"
}

Share enough to prove ROI while keeping learner data minimized.

01 / Access & Edge Layer

Access & Edge Layer

All external traffic enters through Nginx acting as the ingress controller inside AKS. Throttling and rate limiting are managed at the AKS/application layer — no external API gateway or WAF in the current architecture.

Technology

Nginx Ingress Controller deployed inside AKS. Single entry point for all frontend and API traffic. TLS termination, URL routing, upstream health checks.

NginxTLS terminationURL routingHealth probes

Throttling & Rate Limiting

Rate limiting handled at two layers:

Nginx: limit_req_zone per IP at ingress
auth-service: Redis-backed per-IP counter with SETNX + EXPIRE pattern
AKS Network Policy: pod-to-pod traffic rules enforced by Kubernetes NetworkPolicy resources

Routing Table

/api/v1/auth/* → auth-service:8001 /api/v1/tenant/* → tenant-service:8002 /api/v1/events/* → event-service:8003 /api/v1/labs/* → lab-service:8004 /api/v1/agents/* → agent-gateway:8010 /api/v1/billing/* → billing-service:8006 /* (static) → frontend pod

Why no Azure Application Gateway?

Azure Application Gateway + WAF v2 is an excellent managed service but is Azure-specific and adds cost. Nginx inside AKS provides the same routing and TLS termination capabilities, runs identically on AWS/GCP, and keeps the stack cloud-agnostic. WAF rules can be added via Nginx ModSecurity module if needed in a future phase.

02 / Full Stack

Full Architecture Stack

Click any layer to expand. Each layer links to its detailed section below.

Client Layer

React 18TypeScriptTailwindZustand

⌄

Single-page application served from AKS. Communicates exclusively through Nginx. Handles auth token storage, silent refresh, and role-based UI rendering. → see detail

Access & Edge

NginxAKS IngressTLSRate Limiting

⌄

Nginx ingress controller inside AKS. TLS termination, URL-based routing to microservices, per-IP rate limiting. No external WAF layer. → see detail

Application Services

FastAPIPython 3.11asyncpgmany microservices

⌄

Each business domain is an independent FastAPI microservice in AKS. auth-service ✅ Phase 1. Many more added each sprint. → see detail

AI Agents Layer

8 agentsLangGraphClaude APIKEDA autoscale

⌄

Each agent is its own container in AKS. KEDA scales each agent independently based on RabbitMQ queue depth. → see detail

Workflow Orchestration

Temporaldurable execution3 core workflows

⌄

Temporal orchestrates all long-running lab lifecycle workflows. Crash-safe, replay-on-failure, supports HITL wait signals. → see detail

Messaging & Cache

RabbitMQRedisasync eventsJWT store

⌄

RabbitMQ for async inter-service events. Redis for ephemeral stateful data — JWT tokens, rate limit counters, SSO state, session cache. → see detail

Data Layer

PostgreSQL 16single instancemultiple databasesself-managed

⌄

One PostgreSQL 16 instance in AKS. Each microservice owns its own database. Enforced at the Postgres user/permissions level — no cross-service queries possible. → see detail

Infrastructure & CI/CD

TerraformHelmACRGitHub ActionsArgo CD

⌄

Dev+Stage on shared AKS cluster (namespaced). Production on dedicated AKS cluster. GitHub Actions builds + pushes to ACR. Argo CD syncs cluster to Git. → see detail

Monitoring & Alerting

PrometheusGrafanaLokipath-based alerts

⌄

Prometheus scrapes all pods. Grafana manages dashboards and alert routing (path-based). Loki aggregates logs. All inside AKS. → see detail

03 / Client Layer

Client Layer

Single-page application served as static files from an Nginx pod inside AKS.

Technology

React 18 + TypeScript, Tailwind CSS, Zustand (state), Axios (HTTP), React Hook Form + Zod (validation), Framer Motion. Built with Vite.

Why This Stack

React 18 concurrent rendering handles real-time lab status without UI freezing
Zustand — lightweight, no Redux boilerplate
Tailwind — styles co-located with components
Vite — sub-second hot reload

Gravity Labs Example

Live lab status

Participant clicks "Launch Lab." React polls lab-service every 3s via Axios. Zustand useLabStore updates the status chip — Analyzing → Designing → Provisioning → Ready — without re-rendering the whole page.

04 / API Gateway

API Gateway — Nginx

Nginx runs as a Kubernetes deployment. Single entry point for all API traffic from the frontend. Internal service-to-service calls bypass it entirely on the cluster network.

Why Not a Cloud API Gateway

Azure API Management / AWS API Gateway are cloud-specific and costly at scale. Nginx is free, runs in AKS, and the config is version-controlled. Migrating clouds means copying one nginx.conf — zero code changes.

Rate Limiting Configuration

limit_req_zone $binary_remote_addr zone=api:10m rate=100r/m; limit_req zone=api burst=20 nodelay; limit_req_status 429;

Combined with Redis-backed per-service rate limiters in each FastAPI service for fine-grained control.

05 / Application Services

Application Services Layer

Every business domain is its own FastAPI microservice. The list below grows each sprint. The pattern is consistent — one service, one database, one Helm chart, one pipeline.

Tech Pattern

FastAPI — async, OpenAPI docs, Pydantic validation
SQLAlchemy 2.x async + asyncpg
Alembic — migrations per service
structlog — structured JSON logs (Loki-ready)
Each service calls /internal/verify on auth-service to validate JWTs

Service Inventory

Service	Domain	Phase
auth-service	Identity, JWT, SSO	Phase 1 ✅
tenant-service	Orgs, seats, SAML	Phase 2
event-service	Hackathons, cohorts	Phase 2
lab-service	Lab lifecycle + Temporal	Phase 3
catalog-service	Lab templates	Phase 3
scoring-service	5-dim grading	Phase 3
billing-service	Tokens, subscriptions	Phase 4
notification-service	Email, in-app alerts	Phase 4
analytics-service	Reports, dashboards	Phase 4

Why microservices at scale

At 25,000 concurrent users, the scoring engine is under maximum load while billing is idle. A monolith scales everything together and wastes compute. Microservices let us scale scoring-service to 200 pods while billing stays at 2.

06 / AI Agents Layer

AI Agents Layer

Each of the 8 AI agents is its own independently deployed microservice in AKS. Agents are orchestrated by Temporal workflows and communicate via RabbitMQ.

Agent Runtime

LangGraph — agent state machine & tool calls
Claude API — primary LLM for lab design & analysis
Azure OpenAI GPT-4o — fallback for high-volume scoring
FastAPI — same framework as app services
Temporal Activity Workers — each agent registers as a Temporal worker

Why Each Agent = Own Container

During a 25K event, scoring-agent needs 300+ replicas. Lab-architect needs 3. Bundled = you scale both to 300.
LLM timeout in hint-dispenser doesn't crash provisioning-agent
Independent versioning — hotfix scoring without redeploying the fleet
KEDA autoscaling per agent based on RabbitMQ queue depth

Agent Scaling (KEDA)

Agent	Max Pods
lab-architect-agent	10
dataset-forge-agent	50
provisioning-agent	100
qa-validator-agent	50
live-observer-agent	200
hint-dispenser-agent	100
scoring-agent	300
cross-validator-agent	100

25,000-user event — scoring wave

At T-30min, 20,000 participants submit. The scoring queue fills with 20,000 messages. KEDA sees the queue depth and scales scoring-agent from 5 → 280 pods in under 90 seconds. Each pod calls Claude API for commentary and writes results to the scoring DB. Queue drains in ~8 minutes. KEDA scales back down automatically.

07 / Workflow Orchestration

Workflow Orchestration — Temporal

Temporal provides durable execution for all long-running, multi-step lab lifecycle workflows. Write the workflow as normal Python code — Temporal guarantees it completes correctly even if servers crash mid-execution.

Without vs With Temporal

Without: Background job calls Terraform → Graph API → DB update. Server crashes after Terraform. You have a live VM, no license, leaked cost, broken state. Manual recovery required.

With Temporal: Workflow replays from the last checkpoint. Terraform step completed? Skip it. Resume at Graph API. Zero data loss, zero manual recovery.

3 Core Workflows

ProvisionLabWorkflow — AI design → dataset → VM/container → license assign → dataset inject → URL return. 2–8 min.
GradeLabWorkflow — scoring scripts → AI commentary → 5-dimension score → store results.
DestroyLabWorkflow — revoke licenses → delete Entra users → Terraform destroy → log cost. Atomic teardown.

ProvisionLabWorkflow — step by step

1

lab-architect-agent

NL description → IaC template + scoring rubric (LangGraph + Claude API)

G1

HITL Gate 1 — Blueprint Review

Admin approves lab manifest. Temporal waits on signal indefinitely.

2

dataset-forge-agent

Generates N unique synthetic datasets (Faker + Pandas). One per participant.

G2

HITL Gate 2 — Procurement Review

Admin reviews subscription list + per-user cost. Temporal waits for approval.

3

provisioning-agent

Runs Terraform/Bicep. Creates VMSS environment. Assigns Microsoft licenses via Graph API.

4

qa-validator-agent

Dry-run: checks connectivity, subscriptions, dataset, network isolation. Pass/fail report.

G3

HITL Gate 3 — Pre-Launch

Admin reviews QA report + final cost. One click → lab goes live.

✓

Lab Ready

Guacamole URL + credentials returned. live-observer-agent begins monitoring.

08 / Messaging & Cache

Messaging & Cache

RabbitMQ carries async events between services. Redis stores all ephemeral stateful data. Both are self-managed inside AKS for cloud portability.

RabbitMQ — Async Messaging

Queue	Publisher → Consumer
lab.provision.requested	lab-service → provisioning-agent
event.started	event-service → lab-service
lab.ready	lab-service → notification-service
submission.received	lab-service → scoring-agent
score.ready	scoring-agent → analytics-service

Redis — Cache & Token Store

refresh:{user_id}:{jti} → "1" TTL 7d refresh_tokens:{user_id} → Set{jti} login_attempts:{ip} → count TTL 15m blocklist:{jti} → "1" TTL=token TTL sso_state:{state} → data TTL 10m lab_status:{lab_id} → JSON TTL 1h rate_limit:{svc}:{ip} → count TTL 1m

Redis — Why (replacing Valkey)

Redis is the industry standard — broad team familiarity, strong client ecosystem, and Azure Cache for Redis provides a fully managed HA option when moving to production. The same redis.asyncio client works against both self-managed Redis in AKS and managed Azure Cache for Redis — zero code changes when switching.

Multi-cloud portability

Self-managed Redis in AKS for Dev/Stage. Azure Cache for Redis for Production (managed HA, geo-replication). AWS ElastiCache for Redis if moving to AWS. Config change only — no code changes.

09 / Data Layer

Data Layer — PostgreSQL

One PostgreSQL 16 instance running in AKS. Each microservice owns its own logical database inside that instance. Self-managed for full cloud portability.

Database Layout

gravitylabs_auth ← auth-service only gravitylabs_tenant ← tenant-service only gravitylabs_events ← event-service only gravitylabs_labs ← lab-service only gravitylabs_scoring ← scoring-service only gravitylabs_catalog ← catalog-service only gravitylabs_billing ← billing-service only gravitylabs_temporal ← Temporal internal

Why Single Instance

Simpler ops — one backup, one upgrade, one monitoring target
Cheaper — one pod vs 8 separate pods
Logical isolation enforced at Postgres user level — wrong credentials = no access
Easy to split individual DBs to dedicated instances later if one service needs to scale

Why Self-Managed

Azure Database for PostgreSQL Flexible Server and AWS Aurora are excellent but cloud-specific. Running Postgres in AKS means identical setup on Azure, AWS, GCP, or on-prem. Backup runs as a Kubernetes CronJob executing pg_dump → cloud object storage (Azure Blob / S3).

Connection Pooling

PgBouncer — Why It Is Critical

PgBouncer is a lightweight proxy that sits between all application pods and PostgreSQL. It multiplexes thousands of application connections into a small fixed pool of real Postgres connections — preventing connection exhaustion at event scale.

Scenario	Without PgBouncer	With PgBouncer
25K event peak	~2,000+ direct connections → Postgres OOM crash	2,000 app connections → 20 real Postgres connections
KEDA scale-out	300 pods × 2 = 600 new connections in 90s → storm	Pool absorbs burst transparently
Postgres default	max_connections = 100 (hard limit)	max_client_conn = 5000 visible to apps

PgBouncer Configuration

[pgbouncer] pool_mode = transaction max_client_conn = 5000 default_pool_size = 20 reserve_pool_size = 5 reserve_pool_timeout = 3 server_idle_timeout = 600 auth_type = scram-sha-256

SQLAlchemy compatibility fix

Transaction pooling mode breaks prepared statements. Set statement_cache_size=0 in SQLAlchemy connect_args across all services. One-line change — nothing else required.

How It Fits in AKS

All microservice pods ↓ PgBouncer pod (ClusterIP: port 5432) ↓ PostgreSQL pod (internal only)

Services connect to pgbouncer:5432 — never directly to Postgres. Connection string host change only. All query code unchanged.

Without PgBouncer = world record event fails

At 25,000 users the platform has 2,000+ simultaneous DB-connected pods. PostgreSQL cannot handle this. PgBouncer is the difference between the event running and not running.

10 / Infrastructure & CI/CD

Infrastructure & CI/CD

Terraform provisions infrastructure. Helm packages services. GitHub Actions builds. Argo CD deploys. Dev and Stage share one AKS cluster; Production gets its own dedicated cluster.

Cluster Topology

Environment	Cluster	Namespaces
Dev	Shared AKS cluster	`gl-dev`
Stage	Shared AKS cluster	`gl-staging`
Production	Dedicated AKS cluster	`gl-prod`
World Record	2nd prod cluster (multi-region)	`gl-prod-2`

Why shared Dev+Stage, dedicated Prod

Dev and Stage don't need SLA guarantees — sharing a cluster cuts infra cost significantly. Production gets a dedicated cluster for full isolation, independent scaling, and zero blast radius from staging activity.

CI/CD Pipeline

1

Dev pushes PR

Feature branch → PR opened on GitHub

2

GitHub Actions — CI

Run tests → lint → build Docker image → push to ACR with git SHA tag

3

Update Helm values

Actions commits new image tag to charts/*/values.yaml

4

Argo CD detects change

Watches Helm chart repo. Marks app OutOfSync.

5

Argo CD syncs AKS

Auto-sync to Dev/Stage. Manual approval gate for Production.

11 / Monitoring & Alerting

Monitoring & Alerting

Prometheus scrapes metrics from every pod. Grafana manages all dashboards and alert routing. Loki aggregates logs. All deployed inside AKS alongside the application.

Prometheus — Metrics

Deployed in AKS. Scrapes all pods via ServiceMonitor CRDs. Key metrics:

Request rate, latency p50/p95/p99, error rate per service
Active lab count, provisioning queue depth
RabbitMQ queue depth per queue
Redis hit/miss rate, memory usage
Agent LLM token usage + latency
PostgreSQL connections, query time

Grafana — Dashboards & Alerts

Grafana is the single pane of glass — dashboards and alert management. Alert sources: Prometheus (primary) and CloudWatch (AWS metrics for multi-cloud deployments).

Event OperationsAgent FleetInfrastructureBusiness Metrics

Grafana Alert Routing — Path-Based

Alerts are routed by severity and domain using Grafana's built-in Alerting with path-based routing rules:

Alert fires→ Grafana evaluates route→ Match path

Path Match	Destination
severity=critical	PagerDuty (immediate page)
domain=lab-provisioning	Ops Slack channel
domain=billing	Finance Slack channel
severity=warning	Email digest
severity=info	Log only

Live event operations

Ops team watches the Event Operations dashboard on a large screen during a live event. Provisioning queue depth rises → KEDA auto-scales → queue drains. If the queue doesn't drain within 5 minutes, Grafana fires a critical alert → PagerDuty pages the on-call engineer directly.

12 / Peripheral Services

Peripheral Services

Services outside the core application layer. Object storage uses cloud-managed services per deployment target. Email and CDN are provider-agnostic OSS/SaaS choices.

Concern	Choice	Azure	AWS	Rationale
Object Storage	Cloud-managed	Azure Blob Storage	Amazon S3	S3-compatible API across all providers. Config-only swap between clouds.
Email	Sendgrid	Azure Communication Services	Amazon SES	SMTP-compatible. Swap by changing SMTP host config only.
CDN / DDoS	Cloudflare	Azure Front Door	CloudFront	Cloudflare is provider-agnostic — sits in front of any cloud origin.
Secret Management	Azure Key Vault (cloud) / HashiCorp Vault (on-prem)	Azure Key Vault	AWS Secrets Manager	AKS has native Key Vault integration via CSI driver.
Lab Streaming	Apache Guacamole (AKS)	Azure Virtual Desktop	Amazon AppStream	Open-source, browser-based, runs anywhere. No client install.
VM Labs Compute	Azure VMSS	Azure VMSS	AWS Auto Scaling Groups	Warm pool enabled. Pre-built images. Auto-scaling. Isolated networks.

13 / Scale Strategy

Scale Strategy

From 500 to 25,000 participants — the platform scales horizontally at every layer.

500

Standard event
Single region
Default node pool

5,000

Large event
Node autoscale
Quota pre-request

15,000

Multi-region
East US 2 + West US 2
Traffic Manager

25,000

World record
4 regions
Sponsored licenses

Scale Mechanisms

Mechanism	What It Does
HPA	Scales pods based on CPU / memory
KEDA	Scales agent pods based on RabbitMQ queue depth
Cluster Autoscaler	Adds / removes VM nodes in AKS node pool
Temporal worker pools	Provisioning workers scale with workflow load
PgBouncer	Connection pooling in front of Postgres for 10K+ connections
Azure quota tickets	Submitted 6–8 weeks before large events for vCPU headroom

14 / Open Decisions

Architecture Decision Records

Decisions made and pending. Updated each sprint.

DECIDED

Agent deployment: each agent = own container

Independent scaling. KEDA scales scoring-agent independently during events.

DECIDED

Single PostgreSQL, multiple databases

Self-managed in AKS. Cloud-agnostic. Logical isolation at DB user level.

DECIDED

CI/CD: GitHub Actions + Argo CD

Actions builds + pushes to ACR. Argo CD GitOps-syncs AKS. Drift detection included.

DECIDED

Cluster topology: Dev+Stage shared, Prod dedicated

Shared cluster for non-prod cuts cost. Prod cluster is fully isolated with independent scaling.

DECIDED

Cache: Redis (replacing Valkey)

Same redis.asyncio client. Azure Cache for Redis for managed production HA. Code unchanged.

DECIDED

Object storage: cloud-managed per provider

Azure Blob on Azure, S3 on AWS, GCS on GCP. S3-compatible API across all three — config-only swap.

DECIDED

Alerting: Grafana path-based routing

Grafana manages all alert rules and routing. Prometheus + CloudWatch as metric sources. PagerDuty for critical path.

PENDING

JWT algorithm: HS256 → RS256 upgrade

Phase 1 uses HS256 (symmetric). Phase 2 should migrate to RS256 — services hold only public keys.

CRITICAL

Microsoft Partner enrollment

CSP procurement blocked until enrollment completes. 2–4 week timeline. Blocks Phase 3 lab provisioning.

PENDING

SMS / Push notifications

Twilio vs self-hosted vs Azure Communication Services. Evaluate in Phase 4.

Gravity Labs Platform Architecture

Users & Actors

Participants / Learners

Trainers / Evaluators

Platform Admins

Sponsors / Partners

Access & Edge Layer

Full Architecture Stack

Client Layer

API Gateway — Nginx

Application Services Layer

AI Agents Layer

Workflow Orchestration — Temporal

Messaging & Cache

Data Layer — PostgreSQL

Infrastructure & CI/CD

Monitoring & Alerting

Peripheral Services

Scale Strategy

Architecture Decision Records

Agent deployment: each agent = own container

Single PostgreSQL, multiple databases

CI/CD: GitHub Actions + Argo CD

Cluster topology: Dev+Stage shared, Prod dedicated

Cache: Redis (replacing Valkey)

Object storage: cloud-managed per provider

Alerting: Grafana path-based routing

JWT algorithm: HS256 → RS256 upgrade

Microsoft Partner enrollment

SMS / Push notifications