Architecture

Overview of how Friday Assistant is structured.

System Diagram

graph TB
    subgraph Client Layer
        A[LiveKit Client]
        B[User Authentication]
    end

    subgraph Agent Core
        C[Assistant Agent]
        D[Realtime Session]
        E[Context Manager]
    end

    subgraph LLM Integration
        F[Google Gemini 2.0]
        G[Voice Interface]
    end

    subgraph Tool Categories
        H[MS365 Tools]
        I[Context Tools]
        J[Generic Tools]
        K[BayerNet Tools]
    end

    subgraph External Services
        L[Microsoft Graph API]
        M[Weather API]
        N[Web Search]
        O[BayerNet API]
    end

    A -->|MSAL Token| B
    B -->|Authenticated Session| C
    C --> D
    C --> E
    D <-->|Realtime Audio| F
    F --> G
    C --> H
    C --> I
    C --> J
    C --> K
    H --> L
    J --> M
    J --> N
    K --> O

Components

Client Layer

The frontend handles user authentication via Azure AD (MSAL) and maintains the LiveKit connection for real-time audio/video streaming.

Agent Core

The Python backend manages:

• Assistant Agent: Main orchestration logic
• Realtime Session: LiveKit room connection and media handling
• Context Manager: Per-user state and preferences

LLM Integration

Google Gemini 2.0 Flash provides:

• Low-latency voice responses
• Multi-modal input support
• Function calling for tool execution

Tool Categories

Modular tool system organized by domain:

• MS365 Tools: Email, calendar, contacts, Teams
• Context Tools: User state management
• Generic Tools: Weather, web search
• BayerNet Tools: Internal company data

Data Flow

1. User authenticates via Azure AD
2. MSAL token passed to agent via LiveKit metadata
3. Agent extracts user context from JWT
4. Voice input streamed to Gemini
5. Gemini responds with voice output or tool calls
6. Tool results fed back to conversation