Skip to content

Backend Agent

The backend is a Python application built on the LiveKit Agents framework. It handles voice interactions using Google Gemini 2.0 Flash and connects to various enterprise services.

Project Structure

bayer-assistant-agent/
├── agent.py                 # Main entry point
├── prompts.py              # LLM instruction templates
├── requirements.txt        # Dependencies
├── Dockerfile             # Container config
├── logs/                  # Application logs
└── tools/                 # Tool implementations
    ├── ms365/            # Microsoft 365 integrations
    │   ├── email/
    │   ├── calendar/
    │   ├── contacts/
    │   ├── teams/
    │   └── people_search_tool.py
    ├── context/          # User context management
    ├── generic/          # Utility tools
    └── bayernet/         # Internal enterprise tools

Core Features

Voice Interaction

  • Real-time processing via Gemini 2.0 Flash
  • Context-aware conversation handling
  • Noise cancellation through LiveKit BVC

Authentication

  • MSAL token extraction from participant metadata
  • Per-user context isolation
  • Automatic token management

Context Management

User context is extracted from JWT tokens and maintained throughout the session:

{
    "user": {
        "given_name": "John",
        "email": "john.smith@bayer.com",
        "department": "Research & Development",
        "timezone": "Europe/Berlin"
    }
}

Adding Tools

  1. Create tool file in appropriate category directory
  2. Implement async function with RunContext parameter
  3. Register in Assistant.__init__() tools list

Example:

async def my_tool(context: RunContext, param: str) -> str:
    """Tool description for LLM."""
    result = do_something(param)
    return json.dumps({"success": True, "data": result})