Functional Specification & Technical Design · Final Edition

Gravity Labs FSD

Complete functional specification and technical design for the Gravity Labs platform. Covers architecture, cloud abstraction, subscription engine, AI agent implementation, API design, and data models.

March 2026 Engineering · Architects · DevOps · Product Confidential

01 / System Architecture Overview

System Architecture Overview

Cloud-native, horizontally scalable platform built around a cloud-abstraction layer enabling deployment on Azure, AWS, GCP, or on-premises Kubernetes. All application logic is cloud-agnostic; cloud-specific behaviour is encapsulated in adapter modules selected at startup.

Layer	Components	Technology Options
Presentation	Participant Portal, Admin Portal, Super Admin, Public Site	React 18 + TypeScript + Tailwind CSS. White-label themeable via CSS variables.
API Gateway	Rate limiting, auth, routing, API key management	Nginx Ingress (current) · Azure APIM (Azure) · Kong (on-prem)
Application Backend	REST API, business logic, HITL workflow state, RBAC	FastAPI (Python 3.11), async, 3+ replicas via Kubernetes Deployment
AI Agent Orchestration	8 agents, HITL gates, LangGraph stateful pipelines	LangGraph (Python). Claude API (primary) / Azure OpenAI GPT-4o (MLP-compliant)
Subscription Engine	License Manager, Subscription Monitor, credit metering	FastAPI microservice. Microsoft Graph API, GitHub API, Databricks API, CSP REST API
Lab Orchestration	Provisioning queue, warm pool manager, environment lifecycle	Temporal.io (on-prem portable) · Azure Service Bus / AWS SQS / RabbitMQ
Compute — Containers	Code-Server, GPU notebook labs	AKS / EKS / GKE / vanilla K8s. KEDA autoscaling.
Compute — VMs	Full Windows/Linux VM labs, warm pool	Azure VMSS / AWS EC2 ASG / GCP GCE MIGs / VMware vSphere. Packer for images.
Compute — Azure Sandbox	Ephemeral Azure tenant, capped subscription per participant	Azure Subscription Pool. ARM/Bicep deployment. Azure Budget + Policy hard deny cap.
Streaming Gateway	Browser-to-desktop streaming for VM and GUI labs	Apache Guacamole (RDP/VNC/SSH) + Neko WebRTC (GPU/GUI) + Code-Server HTTPS
Data	User data, event data, scores, subscriptions	PostgreSQL (OLTP) + Redis (session cache, warm pool) + Cloud Object Storage
Observability	Metrics, logs, traces, alerts	Prometheus + Grafana + Loki (AKS) · Azure Monitor / CloudWatch (cloud-managed)

Network Topology

All workloads run inside a private VNet/VPC. No workload has a direct public IP. VM lab networks are fully isolated from each other and from the application network. Azure Portal Sandbox tenants are completely separate Azure tenants — zero network connectivity to the GL platform tenant.

02 / Cloud Abstraction Layer

Cloud Abstraction Layer

Python abstract base classes define all cloud-dependent operations. At startup the deployment target is read from gravity-labs.config.yaml and the correct adapter implementations are loaded dynamically.

from abc import ABC, abstractmethod class IComputeProvider(ABC): async def provision_container(self, spec: ContainerSpec) -> EnvironmentHandle: ... async def provision_vm(self, spec: VMSpec) -> EnvironmentHandle: ... async def terminate_environment(self, handle: EnvironmentHandle) -> None: ... async def get_streaming_url(self, handle: EnvironmentHandle) -> str: ... class IIdentityProvider(ABC): async def create_ephemeral_user(self, event_id: str) -> EphemeralUser: ... async def revoke_user(self, user_id: str) -> None: ... class ISubscriptionProvider(ABC): async def provision_azure_sandbox(self, cap_usd: float) -> AzureSandbox: ... async def claim_license(self, license_type: str, quantity: int) -> LicenseClaim: ... async def revoke_license(self, claim_id: str) -> None: ... async def get_credit_usage(self, sandbox_id: str) -> CreditUsage: ...

Interface	Azure	AWS	On-Premises
`IComputeProvider`	AKS + VMSS; azure-mgmt-compute SDK	EKS + EC2 ASG; boto3	K8s Python client + pyVmomi (vSphere)
`IIdentityProvider`	Entra ID; MSAL Python + Graph API	AWS Cognito; boto3 cognito-idp	Keycloak 23; python-keycloak + ldap3
`IStorageProvider`	Azure Blob Storage; azure-storage-blob	S3; boto3 + presigned URLs	MinIO; minio Python client
`ISecretStore`	Azure Key Vault; azure-keyvault-secrets	AWS Secrets Manager; boto3	HashiCorp Vault 1.15; hvac Python
`IMessageQueue`	Azure Service Bus; azure-servicebus	SQS; boto3 sqs client	RabbitMQ; aio-pika · Kafka (high-throughput)
`IObservability`	Azure Monitor + App Insights; OTel exporter	CloudWatch; OTel AWS contrib	Prometheus + Grafana + Loki; OTel Prometheus

03 / Configuration File

gravity-labs.config.yaml

Single configuration file that controls all cloud-specific behaviour. Switching deployment targets requires only editing this file — no code changes required.

deployment: target: azure # azure | aws | gcp | on_prem | hybrid region: eastus compute: container_backend: aks # aks | eks | gke | kubernetes vm_backend: vmss # vmss | ec2_asg | gce_mig | vsphere identity: provider: entra_id # entra_id | cognito | cloud_iam | keycloak tenant_id: ${TENANT_ID} storage: provider: azure_blob # azure_blob | s3 | gcs | minio secrets: provider: key_vault # key_vault | secrets_manager | secret_manager | vault queue: provider: service_bus # service_bus | sqs | pubsub | rabbitmq observability: provider: azure_monitor # azure_monitor | cloudwatch | gcp_ops | prometheus subscription_management: azure_csp_enabled: true github_enabled: true databricks_enabled: true

04 / Subscription Management Engine

Subscription Engine — Internal API

Dedicated FastAPI microservice (subscription-engine, 2+ replicas). PostgreSQL for records. Redis for real-time credit usage (30-second polling, served instantly to dashboards).

Endpoint	Method	Description
`/api/v1/subscriptions/manifest`	POST	Agent submits lab blueprint. Returns full subscription manifest with quantities and total event cost for admin HITL review.
`/api/v1/subscriptions/approve`	POST	Admin approves (or modifies) manifest. Triggers bulk procurement. Records approver and timestamp in audit log.
`/api/v1/subscriptions/provision/{session_id}`	POST	Called by Provisioning Agent on lab start. Activates all approved licenses for this participant session.
`/api/v1/subscriptions/usage/{session_id}`	GET	Returns real-time credit usage. Served from Redis. Polled every 30 seconds by participant Subscription Dashboard.
`/api/v1/subscriptions/topup`	POST	Participant submits top-up request. Fires admin notification within 30 seconds.
`/api/v1/subscriptions/topup/{id}/approve`	POST	Admin approves top-up. Increases credit cap via Azure Budget API. Refreshes Redis cache immediately.
`/api/v1/subscriptions/revoke/{session_id}`	POST	Called on session termination. Revokes all licenses, deactivates Azure subscription, destroys ephemeral Entra ID user. 100% within 5 minutes.

Microsoft CSP License Provisioning — Graph API

# Assign M365 / Copilot license POST https://graph.microsoft.com/v1.0/users/{userId}/assignLicense { "addLicenses": [{"skuId": "{COPILOT_FOR_M365_SKU_ID}"}], "removeLicenses": [] } # Revoke on session end POST https://graph.microsoft.com/v1.0/users/{userId}/assignLicense { "addLicenses": [], "removeLicenses": ["{COPILOT_FOR_M365_SKU_ID}"] }

05 / Azure Portal Sandbox

Azure Portal Sandbox Provisioning

Pool of pre-created Azure subscriptions. Each tagged with pool status. Auto-scales when available count drops below low-water mark. 30-minute quarantine after session end before returning to pool.

# Azure Portal Sandbox — step by step 1. Claim subscription from pool → mark status=in_use 2. Deploy ARM/Bicep template: az deployment sub create --template-file lab_template.bicep \ --parameters participantId={id} creditCap={cap} 3. Create ephemeral Entra ID user: POST https://graph.microsoft.com/v1.0/users { userPrincipalName: user-{uuid}@{tenant}.onmicrosoft.com, passwordProfile: { password: {generated}, forceChangePasswordNextSignIn: false } } 4. Assign Contributor role on assigned resource group only 5. Apply Azure Budget + Policy hard cap (deny effect) 6. Deliver credential card to participant portal + email

Hard Spending Cap via Azure Policy

{ "mode": "All", "policyRule": { "if": { "field": "tags['gravity-session-id']", "equals": "{session_id}" }, "then": { "effect": "deny" } } } # Budget alert at 80% fires admin notification. # Policy deny fires at 100% — no additional spend possible.

Credential Card Delivered

Portal URL: portal.azure.com
Username: user-[code]@[tenant].onmicrosoft.com
Temporary password (auto-generated)
Subscription ID + Resource Group name
Credit cap, current balance, session expiry time

Applicable Certifications

AZ-104: Azure Administrator
AZ-305: Azure Solutions Architect
AZ-500: Azure Security Technologies
AZ-700: Azure Network Engineer
DP-203: Azure Data Engineer
SC-300: Identity & Access Administrator

06 / Real-Time Credit Metering

Real-Time Credit Metering Pipeline

Asyncio background task polls all active sessions every 30 seconds. Results stored in Redis with 60-second TTL. Participant dashboard polls usage endpoint for near-real-time display.

Data Sources per Provider

Azure: Cost Management REST API for subscription spend
GitHub: Usage API for Actions minutes + Copilot seats
Databricks: REST API for DBU consumption
Snowflake: REST API for credit consumption

Alert Thresholds

80% of credit cap: Amber alert in participant dashboard + admin notification via WebSocket + email
100% of credit cap: Red alert + "Request More Credits" prompt + admin notification. Azure Policy deny blocks all further spend.

07 / LangGraph State Machine

HITL Agent Workflow — LangGraph Design

The lab creation workflow is a LangGraph stateful graph. Three HITL gate nodes pause execution, persist state to PostgreSQL, notify the admin, and await explicit human approval before resuming.

class LabCreationState(TypedDict): request: LabRequest manifest: Optional[LabManifest] gate1_approved: bool gate1_edits: Optional[dict] subscription_manifest: Optional[SubscriptionManifest] gate2_approved: bool datasets: Optional[list[DatasetRef]] procurement_status: Optional[ProcurementStatus] qa_result: Optional[QAResult] gate3_approved: bool event_id: Optional[str] error: Optional[str] # Graph definition workflow = StateGraph(LabCreationState) workflow.add_node('lab_architect', lab_architect_agent) workflow.add_node('hitl_gate_1', hitl_gate_node) # PAUSES workflow.add_node('dataset_forge_preview', dataset_forge_preview_agent) workflow.add_node('hitl_gate_2', hitl_gate_node) # PAUSES workflow.add_node('bulk_procurement', license_manager_agent) workflow.add_node('dataset_forge_generate', dataset_forge_generate_agent) workflow.add_node('qa_validator', qa_validator_agent) workflow.add_node('hitl_gate_3', hitl_gate_node) # PAUSES workflow.add_node('launch_event', launch_event_node)

# HITL Gate Node — Interrupt & Resume async def hitl_gate_node(state: LabCreationState, config: RunnableConfig): gate_id = config['configurable']['gate_id'] # gate_1 | gate_2 | gate_3 thread_id = config['configurable']['thread_id'] await db.update_workflow_state(thread_id, gate_id, state, 'AWAITING_APPROVAL') await notify_admin(thread_id, gate_id, state) # WebSocket + email raise NodeInterrupt(f'Awaiting human approval at {gate_id}') # Admin approval endpoint — resumes the graph @router.post('/api/v1/workflows/{thread_id}/gates/{gate_id}/approve') async def approve_gate(thread_id, gate_id, edits, admin): await graph.update_state( {'configurable': {'thread_id': thread_id}}, {f'{gate_id}_approved': True, f'{gate_id}_edits': edits} ) await graph.ainvoke(None, {'configurable': {'thread_id': thread_id}}) await audit_log.record(thread_id, gate_id, admin.id, 'APPROVED', edits)

08 / Agent Specifications

Agent Technical Specifications

Lab Architect Agent

Purpose: NL description → validated Lab Manifest for Gate 1.

RAG Index: Azure AI Search index of MS FY26 course catalogue (70+ cert tracks), known lab patterns, subscription SKU catalogue with pricing. Updated monthly.

Failure handling: Pydantic validation failure → retry up to 3 times with validation error fed back in prompt. After 3 failures: alert admin, return partial manifest for manual completion.

Dataset Forge Agent

Generation: Faker 19 + Pandas 2 + NumPy 1.26. Unique seed per participant: seed = hash(event_id + participant_id). Parallel via multiprocessing.Pool.

Uniqueness Check: Pairwise Pearson correlation on key numeric columns. If any pair has r > 0.95: regenerate with adjusted seed. Max 3 retries.

Storage path: datasets/{event_id}/{participant_id}/dataset.parquet

Provisioning Agent

Routing logic:

code_workspace → AKS + Code-Server
full_vm_windows → VMSS + Guacamole RDP
full_vm_linux → VMSS + Guacamole SSH
azure_portal_sandbox → Subscription Pool
gpu_lab → GPU AKS + Jupyter
browser_desktop → container + Neko WebRTC

Warm Pool: Atomic POP from Redis (< 1 second). Cold provision fallback: 20–40 seconds.

QA Validator Agent

Check	Pass Criteria
Connectivity	Streaming handshake < 10s
Subscription	All licenses status=active
Dataset	DATASET_URL set, schema valid
Tool Install	All tools launchable
Network Isolation	Cannot reach other participant's IP

Output: Green / Amber / Red per check. Red = hard failure, Gate 3 blocked.

Live Observer Agent

Activity signals: Container labs: shell command log stream. VM labs: Guacamole keystroke/mouse event stream. Azure sandbox: Azure Activity Log resource events.

Idle detection: No activity > 5 minutes → trigger Hint Agent. Idle > 15 minutes → admin alert in Proctor View.

Progress estimation: LLM estimates which task participant is working on. Updates Proctor Dashboard in real-time.

Hint Dispensing Agent

Trigger: Live Observer idle threshold exceeded, or participant clicks "Get Hint".

Prompt principle: "Provide one specific, actionable hint. Do NOT give the answer. Max 2 sentences. Be contextual to what they appear to be attempting."

Score impact: Each hint deducts 2% from Independence dimension (max 10% total). Participant sees penalty warning and must confirm before hint is delivered.

09 / Scoring Dimensions

5-Dimension Scoring Model

Scoring & Assessment Agent evaluates each submission against 5 dimensions. Cross-Validation Agent then tests the solution against 2–3 unseen reserve datasets to verify genuine skill generalisation.

Dimension	Weight	Evaluation Method
Functional Correctness	40%	Automated test suite. Azure sandbox: Azure Resource Graph API to verify required resources exist with correct config. Code labs: test scripts against submitted code.
Architectural Quality	20%	LLM evaluation of architecture: least privilege, naming, resource organisation, tagging, service selection rationale. Scored against rubric from Lab Manifest.
Resource Efficiency	15%	Azure Advisor recommendations against participant's subscription. Score based on right-sizing, unused resources, and cost optimisation suggestions identified.
Robustness (Cross-Validation)	15%	Cross-Validation Agent tests submission against 2 additional unseen datasets. Pass rate = robustness score. Ensures solution generalises beyond participant's own dataset.
Speed & Independence	10%	Time to complete minus hint penalty deductions. Normalised against event median completion time.

Cross-Validation Agent

Uses 2–3 reserve datasets per lab (generated by Dataset Forge but withheld from participants). Runs submitted solution against each reserve dataset. Pipeline completes within 3 minutes of submission. Results fed into final score calculation and narrative score report with certification exam objective mapping.

10 / API Specification

API Specification

Authentication

Admin and Participant portals: JWT bearer tokens (RS256) issued by IIdentityProvider. External integrations: API key in X-API-Key header. LTI 1.3: OAuth 2.0 PKCE flow for grade passback. API returns HTTP 403 if a workflow advances to the next stage without passing all prior HITL gates — enforced at API layer, cannot be bypassed.

Lab Lifecycle Endpoints

Endpoint	Method	Request / Response
`/api/v1/labs/start`	POST	Body: `{ event_id, participant_id }` → `{ session_id, environment_type, access_url, credential_card?, expires_at }`
`/api/v1/labs/{session_id}/status`	GET	`{ status: provisioning\|ready\|active\|terminated, progress_pct, streaming_url, subscription_summary }`
`/api/v1/labs/{session_id}/submit`	POST	multipart/form-data `{ artifacts[], notes }` → `{ submission_id, estimated_score_ready_in_seconds }`
`/api/v1/labs/{session_id}/score`	GET	`{ dimensions: {correctness, architecture, efficiency, robustness, independence}, total_score, narrative_feedback, exam_objective_mapping }`
`/api/v1/labs/{session_id}/terminate`	POST	Triggers environment cleanup, license revocation, subscription quarantine. → `{ status: terminating, cleanup_eta_seconds }`
`/api/v1/events/{event_id}/leaderboard`	GET	`{ rank, participant_display_name, total_score, completion_time, hint_count }[]`

Admin / Workflow Endpoints

Endpoint	Method	Description
`/api/v1/admin/labs/generate`	POST	Start HITL workflow. Body: `LabRequest` → `{ thread_id, status: awaiting_gate_1 }`
`/api/v1/admin/workflows/{id}/gates/1/approve`	POST	Gate 1 approval with optional manifest edits. Triggers dataset preview + subscription manifest.
`/api/v1/admin/workflows/{id}/gates/2/approve`	POST	Gate 2 approval. Triggers procurement + dataset generation.
`/api/v1/admin/workflows/{id}/gates/3/approve`	POST	Gate 3 launch approval. Makes event live → `{ event_id, status: live }`
`/api/v1/admin/events/{event_id}/sessions`	GET	All sessions with status, progress, subscription credit usage, hint count, current task estimate.
`/api/v1/admin/events/{event_id}/cost`	GET	Real-time cost: total, per-participant avg, by subscription type, budget vs actual.
`/api/v1/admin/pool/resize`	POST	Resize warm VM pool or Azure subscription pool.

11 / Data Models

Core PostgreSQL Schema

CREATE TABLE sessions ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), event_id UUID NOT NULL REFERENCES events(id), participant_id UUID NOT NULL, environment_type VARCHAR(32) NOT NULL, status VARCHAR(16) DEFAULT 'provisioning', streaming_url TEXT, azure_sub_id TEXT, azure_user_upn TEXT, dataset_url TEXT, started_at TIMESTAMPTZ DEFAULT NOW(), expires_at TIMESTAMPTZ, terminated_at TIMESTAMPTZ, total_score NUMERIC(5,2), hint_count INT DEFAULT 0 ); CREATE TABLE events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tenant_id UUID NOT NULL, title TEXT NOT NULL, lab_manifest JSONB NOT NULL, status VARCHAR(16) DEFAULT 'draft', participant_count INT, starts_at TIMESTAMPTZ, ends_at TIMESTAMPTZ, total_cost_usd NUMERIC(10,2), created_by UUID NOT NULL ); CREATE TABLE subscription_claims ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), session_id UUID NOT NULL REFERENCES sessions(id), vendor VARCHAR(32) NOT NULL, sku_id TEXT, status VARCHAR(16) DEFAULT 'active', claimed_at TIMESTAMPTZ DEFAULT NOW(), revoked_at TIMESTAMPTZ, credit_used_usd NUMERIC(10,4) );

Redis Key Schema (Session & Warm Pool)

warm_pool:{cloud}:{region}:{type} → List[vm_id] # atomic LPOP for sub-second claim session:{session_id}:credits → JSON TTL 90s session:{session_id}:progress → JSON TTL 60s lab_status:{session_id} → JSON TTL 1h rate_limit:{service}:{ip} → count TTL 1m

12 / On-Premises Requirements

On-Premises Deployment Requirements

Component	Minimum Spec	Notes
Kubernetes Cluster	3 control-plane + 5+ worker nodes, 8 vCPU / 32GB RAM each	K8s 1.28+, OpenShift 4.12+, Rancher/RKE2. Helm 3.10+ required.
VMware vSphere	vSphere 7.0+; ESXi hosts with 32+ vCPU / 128GB RAM	pyVmomi SDK for VM provisioning/clone/destroy from golden OVA templates.
MinIO (Object Storage)	500GB NFS-backed PVC	S3-compatible API. Same IStorageProvider adapter code as AWS S3.
Keycloak (Identity)	2 clustered instances, 2 vCPU / 4GB RAM each	LDAP/AD federation. SAML/OIDC for portal SSO.
HashiCorp Vault (Secrets)	3-node HA Raft cluster	Auto-unseal via HSM if available. Kubernetes auth method for pod identity.
RabbitMQ / Kafka	3-node cluster	RabbitMQ for provisioning queues. Kafka for high-throughput agent telemetry.
Prometheus + Grafana + Loki	2 vCPU / 8GB RAM per node	Pre-built Gravity Labs dashboards included in Helm chart.
Container Registry	Harbor or JFrog Artifactory	Replaces ACR/ECR/GCR. All images pre-loaded for air-gapped operation.
PgBouncer	2 vCPU / 2GB RAM	Connection pooler in front of PostgreSQL. Transaction pooling mode. Essential for 1,000+ concurrent sessions. Set `statement_cache_size=0` in SQLAlchemy for PgBouncer compatibility.

13 / Security Model

Security Model

Authentication

JWT bearer tokens (RS256) — Phase 2 upgrade from HS256
Access token TTL: 15 minutes
Refresh token TTL: 7 days, JTI stored in Redis
JTI blocklist checked on every request
Dummy bcrypt on unknown email (prevents timing attacks)
Per-IP rate limiting via Redis SETNX + EXPIRE

Infrastructure Security

All workloads in private VNet/VPC — no direct public IPs
Azure Key Vault / HashiCorp Vault for all secrets
Managed Identity for pod-to-service authentication
Network Security Groups (NSG) per subnet
Private Endpoints for all PaaS services
RBAC enforced at API layer AND Kubernetes RBAC

Lab Isolation

Each participant lab in a separate private subnet
Azure Portal Sandbox: completely separate Azure tenant
Network isolation verified by QA Validator Agent before launch
Guacamole session recording for compliance labs
All agent decisions audit-logged with approver + timestamp
SOC 2 Type II + ISO 27001 compliance evidence packages

14 / Data Layer

Data Layer — PostgreSQL + Redis + PgBouncer

Single PostgreSQL 16 instance in AKS, multiple logical databases (one per service). PgBouncer in front for connection pooling. Redis for all ephemeral stateful data.

PostgreSQL — Database Layout

gravitylabs_auth ← auth-service gravitylabs_tenant ← tenant-service gravitylabs_events ← event-service gravitylabs_labs ← lab-service gravitylabs_scoring ← scoring-service gravitylabs_catalog ← catalog-service gravitylabs_billing ← billing-service gravitylabs_temporal ← Temporal internal

PgBouncer Configuration

[pgbouncer] pool_mode = transaction max_client_conn = 5000 default_pool_size = 20 reserve_pool_size = 5 reserve_pool_timeout = 3 server_idle_timeout = 600 auth_type = scram-sha-256

SQLAlchemy compatibility: set statement_cache_size=0 in connect_args.

Why PgBouncer is Critical

25K-user event: 2,000+ simultaneous app connections
PostgreSQL default max_connections: 100
PgBouncer multiplexes 2,000 app connections into 20 real Postgres connections
KEDA scale-out: 300 scoring-agent pods × 2 connections = 600 new connections in 90s — absorbed by PgBouncer without impacting Postgres
Without PgBouncer: world-record event cannot run on a single Postgres instance