Data Flow¶

This page describes how data flows through the Gremia system, from user input to task execution and back.

Flow 1: Manifest Generation¶

When a user describes their business in the Builder, the system generates a team manifest.

sequenceDiagram
    participant U as User
    participant B as Builder (Web)
    participant C as Cloud (FastAPI)
    participant A as Architect Agent
    participant RAG as Knowledge Base
    participant DB as Supabase

    U->>B: "I need agents for my logistics company"
    B->>C: POST /api/v1/chat (SSE stream)
    C->>A: Run architect graph
    A->>RAG: Query MCP registry + industry templates
    RAG-->>A: Relevant tools and patterns
    A->>A: Design agents, workflows, security
    A-->>C: Stream chunks (node progress)
    C-->>B: SSE: data: {"node":"architect","data":{...}}
    A->>A: Assemble team_manifest.json
    A-->>C: Final manifest
    C-->>B: SSE: data: {"node":"finalize","data":{"manifest_json":{...}}}
    C-->>B: SSE: data: [DONE]
    B-->>U: Display manifest preview
    U->>B: Save manifest
    B->>C: POST /api/v1/manifests
    C->>DB: INSERT manifest (JSONB, encrypted)
    C-->>B: 201 Created

Data transformations¶

User message (string) is sent as part of a ChatRequest with session context
Architect agent processes through a LangGraph state machine with nodes:
Research node: queries the RAG knowledge base for relevant MCP servers
Design node: creates agent definitions and workflow graphs
Finalize node: assembles and validates the complete manifest
Manifest is validated against the Zod schema before storage
Storage: Manifest is encrypted with AES-256-GCM and stored as JSONB in PostgreSQL

Flow 2: Shell Connection¶

When the Shell connects to the Cloud, it goes through authentication, key exchange, and manifest loading.

sequenceDiagram
    participant S as Shell (Tauri)
    participant N as Nginx
    participant C as Cloud
    participant DB as Supabase

    S->>N: WSS upgrade + JWT in Sec-WebSocket-Protocol
    N->>N: Verify client cert (optional mTLS)
    N->>C: Forward with X-Client-Cert header
    C->>C: Validate JWT
    C-->>S: 101 Switching Protocols
    S->>C: {"type":"auth","shell_id":"..."}
    C->>C: Bind shell_id to user_id
    C->>S: {"type":"key_exchange","session_key":"..."}
    S->>S: Store AES-256 session key
    Note over S,C: All further messages encrypted

    S->>C: GET /api/v1/manifests/{id}
    C->>DB: SELECT manifest (encrypted)
    C->>C: Decrypt manifest
    C-->>S: Manifest JSON
    S->>S: Parse mcp_requirements
    S->>S: Spawn MCP servers

Data at each stage¶

Stage	Data	Encryption
JWT in header	User identity claims	TLS 1.3 (transport)
Auth message	Shell ID	TLS 1.3
Key exchange	32-byte AES key	TLS 1.3
Manifest download	Full manifest JSON	TLS 1.3 + AES-256-GCM (at rest)
Tunnel messages	Tool calls/results	TLS 1.3 + AES-256-GCM (session key)

Flow 3: Task Execution¶

When a user submits a task, it flows through multiple components.

sequenceDiagram
    participant U as User
    participant S as Shell
    participant C as Cloud
    participant E as Execution Engine
    participant A as Supervisor Agent
    participant W as Worker Agent
    participant T as Tool Bridge
    participant M as MCP Server

    U->>S: "Generate Q1 sales report"
    S->>C: task message (encrypted)
    C->>E: Create execution session
    E->>A: Assign to supervisor
    A->>A: Plan: 1) fetch data, 2) analyze, 3) format

    A->>W: Delegate "fetch data"
    W->>T: tool_call: sheets.read_range
    T->>C: Send via tunnel
    C->>S: tool_call message (encrypted)
    S->>M: JSON-RPC tools/call (stdin)
    M-->>S: JSON-RPC result (stdout)
    S->>C: tool_result (encrypted)
    C->>T: Resolve pending future
    T-->>W: Tool output
    W-->>A: Data fetched

    A->>W: Delegate "analyze"
    W->>W: Process data with model
    W-->>A: Analysis complete

    A->>W: Delegate "format report"
    W->>T: tool_call: filesystem.write_file
    T->>C: Send via tunnel
    C->>S: tool_call message
    S->>S: Check requires_approval
    S->>U: Show approval dialog
    U->>S: Approve
    S->>M: JSON-RPC tools/call
    M-->>S: Result
    S->>C: tool_result
    C->>T: Resolve
    T-->>W: File written
    W-->>A: Report formatted

    A->>E: Task complete
    E->>C: task_response (is_final: true)
    C->>S: task_response (encrypted)
    S-->>U: "Report generated successfully"

Execution Data Model¶

erDiagram
    EXECUTION ||--o{ STEP : has
    EXECUTION {
        string execution_id PK
        string shell_id
        string user_id
        string manifest_id
        string status
        string task_description
        string final_output
        string error
    }
    STEP {
        string step_id PK
        string execution_id FK
        string agent_id
        string event_type
        json data
        timestamp created_at
    }

Flow 4: Certificate Rotation¶

mTLS certificates are automatically renewed before expiry.

sequenceDiagram
    participant S as Shell
    participant C as Cloud
    participant DB as Supabase

    loop Every 5 minutes
        S->>S: Check cert expiry
        alt Expires in < 1 hour
            S->>S: Generate EC P-256 key pair
            S->>S: Create CSR (CN=shell-{id})
            S->>C: POST /api/v1/certs/sign
            C->>C: Sign CSR (24h validity)
            C->>DB: Record serial + expires_at
            C-->>S: certificate_pem + ca_pem
            S->>S: Store new cert material
        else Still valid
            S->>S: Skip rotation
        end
    end

Flow 5: Billing and Usage Tracking¶

Every model invocation is tracked for billing.

sequenceDiagram
    participant A as Agent
    participant C as Cloud
    participant B as Billing Service
    participant DB as Supabase
    participant ST as Stripe

    A->>C: Model API call
    C->>C: Count input/output tokens
    C->>B: Record usage event
    B->>B: Calculate cost (model tier * tokens)
    B->>DB: INSERT usage_event
    B->>DB: UPDATE credit_balance (decrement)

    Note over B: Monthly cycle
    ST->>C: Webhook: invoice.paid
    C->>B: Reset credits for period
    B->>DB: UPDATE credit_balance

Usage Event Structure¶

Field	Example
`event_type`	`execution_step`
`model`	`balanced`
`tokens_input`	1500
`tokens_output`	800
`cost_cents`	17

Cost calculation: (input_tokens / 1000 * input_rate) + (output_tokens / 1000 * output_rate)

sequenceDiagram
    participant U as User
    participant C as Cloud
    participant DB as Supabase

    U->>C: GET /api/v1/privacy/data-export
    C->>DB: SELECT consent_records WHERE user_id = ?
    C->>DB: SELECT audit_entries WHERE actor_id = ?
    C->>DB: SELECT erasure_requests WHERE user_id = ?
    C->>C: Compile response
    C-->>U: JSON with all personal data

Data Storage Summary¶

Data	Table	Encrypted	RLS
User accounts	`auth.users` (Supabase)	Supabase managed	Yes
Manifests	`manifests`	AES-256-GCM	Yes
Chat sessions	`builder_sessions`	AES-256-GCM	Yes
Executions	`workflow_executions`	No (operational)	Yes
Consent records	`consent_records`	No (legal record)	Yes
Audit log	`audit_entries`	AES-256-GCM (details)	Service only
Credit balances	`credit_balances`	No (operational)	Yes
Usage events	`usage_events`	No (operational)	Yes
Certificates	`certificate_issuances`	No (serial only)	Service only
Rate limits & cache	Upstash Redis	No (ephemeral)	N/A

Network Diagram¶

graph TB
    subgraph Public Internet
        B[Browser - Builder]
        D[Desktop - Shell]
    end

    subgraph DMZ
        NG[Nginx Reverse Proxy]
    end

    subgraph Private Network
        O[Orchestrator Instances]
        RD[(Upstash Redis)]
    end

    subgraph Database Tier
        S[(Supabase PostgreSQL)]
    end

    subgraph External APIs
        AN[Anthropic API]
        SR[Stripe API]
    end

    B -->|HTTPS 443| NG
    D -->|WSS 443| NG
    NG -->|HTTP 8000| O
    O -->|TLS| S
    O -->|TLS| AN
    O -->|TLS| SR
    O -->|TLS| RD

All external communication uses TLS 1.3. Internal communication between Nginx and the orchestrator uses plain HTTP within the private network (Docker bridge). The database connection uses TLS. Redis (Upstash) is accessed over TLS for rate limiting, execution caching, and WebSocket relay.