Data Flow¶

This page describes how data flows through the Gremia system, from user input to task execution and back.

Flow 1: Manifest Generation¶

When a user describes their business in the Builder, the system generates a team manifest.

sequenceDiagram
    participant U as User
    participant B as Builder (Web)
    participant C as Cloud (FastAPI)
    participant A as Architect Agent
    participant RAG as Knowledge Base
    participant DB as Supabase

    U->>B: "I need agents for my logistics company"
    B->>C: POST /api/v1/chat (SSE stream)
    C->>A: Run architect graph
    A->>RAG: Query MCP registry + industry templates
    RAG-->>A: Relevant tools and patterns
    A->>A: Design agents, workflows, security
    A-->>C: Stream chunks (node progress)
    C-->>B: SSE: data: {"node":"architect","data":{...}}
    A->>A: Assemble team_manifest.json
    A-->>C: Final manifest
    C-->>B: SSE: data: {"node":"finalize","data":{"manifest_json":{...}}}
    C-->>B: SSE: data: [DONE]
    B-->>U: Display manifest preview
    U->>B: Save manifest
    B->>C: POST /api/v1/manifests
    C->>DB: INSERT manifest (JSONB, encrypted)
    C-->>B: 201 Created

Data transformations¶

User message (string) is sent as part of a ChatRequest with session context
Architect agent processes through a LangGraph state machine with nodes:
Research node: queries the RAG knowledge base for relevant MCP servers
Design node: creates agent definitions and workflow graphs
Finalize node: assembles and validates the complete manifest
Manifest is validated against the Zod schema before storage
Storage: Manifest is encrypted with AES-256-GCM and stored as JSONB in PostgreSQL

Flow 2: Shell Connection¶

When the Shell connects to the Cloud, it goes through authentication, key exchange, and manifest loading.

sequenceDiagram
    participant S as Shell (Tauri)
    participant N as Nginx
    participant C as Cloud
    participant DB as Supabase

    S->>N: WSS upgrade + JWT in Sec-WebSocket-Protocol
    N->>N: Verify client cert (optional mTLS)
    N->>C: Forward with X-Client-Cert header
    C->>C: Validate JWT
    C-->>S: 101 Switching Protocols
    S->>C: {"type":"auth","shell_id":"..."}
    C->>C: Bind shell_id to user_id
    C->>S: {"type":"key_exchange","session_key":"..."}
    S->>S: Store AES-256 session key
    Note over S,C: All further messages encrypted

    S->>C: GET /api/v1/manifests/{id}
    C->>DB: SELECT manifest (encrypted)
    C->>C: Decrypt manifest
    C-->>S: Manifest JSON
    S->>S: Parse mcp_requirements
    S->>S: Spawn MCP servers

Data at each stage¶

Stage	Data	Encryption
JWT in header	User identity claims	TLS 1.3 (transport)
Auth message	Shell ID	TLS 1.3
Key exchange	32-byte AES key	TLS 1.3
Manifest download	Full manifest JSON	TLS 1.3 + AES-256-GCM (at rest)
Tunnel messages	Tool calls/results	TLS 1.3 + AES-256-GCM (session key)

Flow 3: Task Execution¶

When a user submits a task, it flows through multiple components.

sequenceDiagram
    participant U as User
    participant S as Shell
    participant C as Cloud
    participant E as Execution Engine
    participant A as Supervisor Agent
    participant W as Worker Agent
    participant T as Tool Bridge
    participant M as MCP Server

    U->>S: "Generate Q1 sales report"
    S->>C: task message (encrypted)
    C->>E: Create execution session
    E->>A: Assign to supervisor
    A->>A: Plan: 1) fetch data, 2) analyze, 3) format

    A->>W: Delegate "fetch data"
    W->>T: tool_call: sheets.read_range
    T->>C: Send via tunnel
    C->>S: tool_call message (encrypted)
    S->>M: JSON-RPC tools/call (stdin)
    M-->>S: JSON-RPC result (stdout)
    S->>C: tool_result (encrypted)
    C->>T: Resolve pending future
    T-->>W: Tool output
    W-->>A: Data fetched

    A->>W: Delegate "analyze"
    W->>W: Process data with model
    W-->>A: Analysis complete

    A->>W: Delegate "format report"
    W->>T: tool_call: filesystem.write_file
    T->>C: Send via tunnel
    C->>S: tool_call message
    S->>S: Check requires_approval
    S->>U: Show approval dialog
    U->>S: Approve
    S->>M: JSON-RPC tools/call
    M-->>S: Result
    S->>C: tool_result
    C->>T: Resolve
    T-->>W: File written
    W-->>A: Report formatted

    A->>E: Task complete
    E->>C: task_response (is_final: true)
    C->>S: task_response (encrypted)
    S-->>U: "Report generated successfully"

Execution Data Model¶

erDiagram
    EXECUTION ||--o{ STEP : has
    EXECUTION {
        string execution_id PK
        string shell_id
        string user_id
        string manifest_id
        string status
        string task_description
        string final_output
        string error
    }
    STEP {
        string step_id PK
        string execution_id FK
        string agent_id
        string event_type
        json data
        timestamp created_at
    }

Flow 4: Certificate Rotation¶

mTLS certificates are automatically renewed before expiry.

sequenceDiagram
    participant S as Shell
    participant C as Cloud
    participant DB as Supabase

    loop Every 5 minutes
        S->>S: Check cert expiry
        alt Expires in < 1 hour
            S->>S: Generate EC P-256 key pair
            S->>S: Create CSR (CN=shell-{id})
            S->>C: POST /api/v1/certs/sign
            C->>C: Sign CSR (24h validity)
            C->>DB: Record serial + expires_at
            C-->>S: certificate_pem + ca_pem
            S->>S: Store new cert material
        else Still valid
            S->>S: Skip rotation
        end
    end

Flow 5: Billing and Usage Tracking¶

Every model invocation is tracked for billing.

sequenceDiagram
    participant A as Agent
    participant C as Cloud
    participant B as Billing Service
    participant DB as Supabase
    participant ST as Stripe

    A->>C: Model API call
    C->>C: Count input/output tokens
    C->>B: Record usage event
    B->>B: Calculate cost (model tier * tokens)
    B->>DB: INSERT usage_event
    B->>DB: UPDATE credit_balance (decrement)

    Note over B: Monthly cycle
    ST->>C: Webhook: invoice.paid
    C->>B: Reset credits for period
    B->>DB: UPDATE credit_balance

Usage Event Structure¶

Field	Example
`event_type`	`execution_step`
`model`	`balanced`
`tokens_input`	1500
`tokens_output`	800
`cost_cents`	17

Cost calculation: (input_tokens / 1000 * input_rate) + (output_tokens / 1000 * output_rate)

sequenceDiagram
    participant U as User
    participant C as Cloud
    participant DB as Supabase

    U->>C: GET /api/v1/privacy/data-export
    C->>DB: SELECT consent_records WHERE user_id = ?
    C->>DB: SELECT audit_entries WHERE actor_id = ?
    C->>DB: SELECT erasure_requests WHERE user_id = ?
    C->>C: Compile response
    C-->>U: JSON with all personal data

Data Storage Summary¶

Data	Table	Encrypted	RLS
User accounts	`auth.users` (Supabase)	Supabase managed	Yes
Manifests	`manifests`	AES-256-GCM	Yes
Chat sessions	`builder_sessions`	AES-256-GCM	Yes
Executions	`workflow_executions`	No (operational)	Yes
Consent records	`consent_records`	No (legal record)	Yes
Audit log	`audit_entries`	AES-256-GCM (details)	Service only
Credit balances	`credit_balances`	No (operational)	Yes
Usage events	`usage_events`	No (operational)	Yes
Certificates	`certificate_issuances`	No (serial only)	Service only

Network Diagram¶

graph TB
    subgraph Public Internet
        B[Browser - Builder]
        D[Desktop - Shell]
    end

    subgraph DMZ
        NG[Nginx Reverse Proxy]
    end

    subgraph Private Network
        O[Orchestrator Instances]
    end

    subgraph Database Tier
        S[(Supabase PostgreSQL)]
    end

    subgraph External APIs
        AN[Anthropic API]
        SR[Stripe API]
    end

    B -->|HTTPS 443| NG
    D -->|WSS 443| NG
    NG -->|HTTP 8000| O
    O -->|TLS| S
    O -->|TLS| AN
    O -->|TLS| SR

All external communication uses TLS 1.3. Internal communication between Nginx and the orchestrator uses plain HTTP within the private network (Docker bridge). The database connection uses TLS.