This guide walks you through deploying Genesis AI Agents natively on Databricks using Databricks Apps, Compute Clusters, and Lakebase PostgreSQL.

Overview

Genesis runs natively on Databricks without requiring external infrastructure. The deployment uses three Databricks-native components:

Component	Purpose
Databricks App	Serves the web UI and handles real-time WebSocket connections
Compute Cluster	Runs the Genesis backend server in a custom Docker container
Lakebase PostgreSQL	Provides real-time message queuing between components

Architecture Diagram

Component Flow:

Browser connects to Databricks App via WebSocket
Databricks App sends notifications to Lakebase PostgreSQL
PostgreSQL Worker (in compute cluster) listens for database notifications
Worker forwards messages to Genesis Backend for processing

Prerequisites

Before you begin, ensure you have:

Databricks Workspace with access to Databricks Apps, Compute clusters with custom container support, and Lakebase PostgreSQL.
Databricks CLI installed and configured.
Docker for building the Genesis container image.
LLM Access via Databricks Model Serving endpoints or OpenAI.

Install and configure the Databricks CLI:

pip install databricks-cli
databricks configure --token

Step 1: Set Up Lakebase PostgreSQL

Lakebase is Databricks’ managed PostgreSQL service that provides real-time messaging between components.

1.1 Create a Lakebase PostgreSQL Instance

Navigate to Lakebase

In your Databricks workspace, go to Catalog → External Locations → Lakebase.

Create Instance

Click Create Lakebase Instance and configure:

Name: genesis-db
Size: Small (sufficient for most deployments)

Note Connection Details

Record the connection information:

Host: instance-xxxxx.database.cloud.databricks.com
Port: 5432
Default database: postgres

1.2 Create the Genesis Database and Schema

Connect to your Lakebase instance and run the provided SQL scripts:

# First, create the database (connect to the default 'postgres' database)
psql -h instance-xxxxx.database.cloud.databricks.com -U your_username -d postgres \
  -f genesis_bots/apps/genesis_server/deployments/databricks/sql/create_database.sql

# Then, create the schema and triggers (connect to 'genesis_app' database)
psql -h instance-xxxxx.database.cloud.databricks.com -U your_username -d genesis_app \
  -f genesis_bots/apps/genesis_server/deployments/databricks/sql/postgres_schema.sql

The schema script creates:

Message queue tables (app_requests, app_responses)
WebSocket connection tracking (websocket_connections, websocket_messages)
Real-time NOTIFY/LISTEN triggers for instant message delivery
Indexes for optimal performance

Step 2: Build and Push the Genesis Container

The Genesis backend runs in a custom Docker container on Databricks Compute.

2.1 Build the Docker Image

From the Genesis repository root:

# Build the Databricks-specific image
docker build -f genesis_bots/apps/genesis_server/deployments/databricks/Dockerfile.databricks \
  -t genesis-databricks:latest .

2.2 Push to Container Registry

Push the image to a registry accessible by Databricks (Docker Hub, Azure ACR, AWS ECR, or Databricks Container Registry):

# Example: Push to Docker Hub
docker tag genesis-databricks:latest your-registry/genesis-databricks:latest
docker push your-registry/genesis-databricks:latest

Step 3: Create the Databricks Compute Cluster

3.1 Create Cluster with Custom Container

Navigate to Compute

In your Databricks workspace, go to Compute and click Create Cluster.

Configure Cluster Settings

Setting	Value
Cluster name	`genesis-backend-cluster`
Cluster mode	Single Node
Databricks Runtime	Runtime with custom container
Docker image URL	`your-registry/genesis-databricks:latest`
Node type	At least 4 cores, 16GB RAM recommended

3.2 Set Environment Variables

In the cluster’s Advanced Options → Spark → Environment Variables, add:

PostgreSQL Connection
Databricks Model Serving
OpenAI

# PostgreSQL Connection (Lakebase)
POSTGRES_HOST=instance-xxxxx.database.cloud.databricks.com
POSTGRES_PORT=5432
POSTGRES_DB=genesis_app
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password
POSTGRES_SSLMODE=require

Genesis Backend

BACKEND_URL=http://localhost:8080 BACKEND_WS_URL=ws://localhost:8080

bash # LLM Configuration - Databricks Model Serving BOT_OS_DEFAULT_LLM_ENGINE=databricks DATABRICKS_BASE_URL=[https://your-workspace.cloud.databricks.com/serving-endpoints](https://your-workspace.cloud.databricks.com/serving-endpoints) DATABRICKS_API_KEY=dapi...your_token DATABRICKS_MODEL_NAME=databricks-claude-sonnet-4

bash # LLM Configuration - OpenAI BOT_OS_DEFAULT_LLM_ENGINE=openai OPENAI_API_KEY=sk-...your_key

3.3 Configure Init Script

The init script starts both the Genesis backend server and the PostgreSQL worker on cluster startup.

Upload Init Script

bash databricks workspace import \ genesis_bots/apps/genesis_server/deployments/databricks/cluster/databricks_init.sh \ /Workspace/Users/your-email/databricks_init.sh

Add to Cluster Configuration

In Advanced Options → Init Scripts, add the path: /Workspace/Users/your-email/databricks_init.sh

3.4 Start the Cluster

Click Create Cluster and wait for it to start. The init script will automatically launch Genesis.

Step 4: Deploy the Databricks App

The Databricks App serves the frontend UI and proxies requests to the backend.

4.1 Deploy the App

From the Genesis repository root, run the provided deployment script:

# Set your workspace path (update with your email)
export WORKSPACE_PATH="/Workspace/Users/your-email/genesis-app"

# Run the deploy script
./genesis_bots/apps/genesis_server/deployments/databricks/app/deploy_app.sh

4.2 Get the App URL

databricks apps get genesis-app

The app will be available at a URL like: https://genesis-app-xxxxx.cloud.databricksapps.com

Step 5: Verify the Deployment

5.1 Check Component Status

Databricks App

bash databricks apps get genesis-app databricks apps logs genesis-app

Compute Cluster

In a Databricks notebook attached to the cluster: bash %sh ps aux | grep -E "genesis|postgres_worker"

Genesis Health

bash curl [https://genesis-app-xxxxx.cloud.databricksapps.com/api/health](https://genesis-app-xxxxx.cloud.databricksapps.com/api/health)

5.2 Check Database Connectivity

Run in SQL editor connected to Lakebase:

SELECT 'websocket_connections' as table_name, COUNT(*) as count
FROM websocket_connections
UNION ALL
SELECT 'app_requests', COUNT(*) FROM app_requests;

5.3 Access the UI

Open your browser and navigate to your Genesis app URL. You should see the Genesis dashboard.

Next Steps

After successful deployment:

Configure LLM Settings

Access Genesis settings to configure your preferred LLM provider

Add Data Connections

Connect Genesis to your Databricks data sources (Unity Catalog, Delta tables)

Create Agents

Start building AI agents for your data workflows

Set Up Authentication

Configure SSO or other authentication methods if needed

Updating Genesis

To update your Genesis deployment:

Build New Container Image

Build a new container image with the latest code.

Push to Registry

Push the updated image to your container registry.

Restart Compute Cluster

Restart the compute cluster to pull the new image.

Redeploy App (if needed)

If frontend changes are included: bash databricks apps deploy genesis-app --source-code-path "$WORKSPACE_PATH"

Getting Started

Creating A Mission

Blueprint

Genesis Data Agents

Genesis Agent Security Controls

Genesis Data Agent's Toolkit

Setup

Slack and Teams

Data Connections

Deployment Options

Deploying Genesis on Databricks

Overview

Architecture Diagram

Prerequisites

Step 1: Set Up Lakebase PostgreSQL

1.1 Create a Lakebase PostgreSQL Instance

1.2 Create the Genesis Database and Schema

Step 2: Build and Push the Genesis Container

2.1 Build the Docker Image

2.2 Push to Container Registry

Step 3: Create the Databricks Compute Cluster

3.1 Create Cluster with Custom Container

3.2 Set Environment Variables

Genesis Backend

3.3 Configure Init Script

3.4 Start the Cluster

Step 4: Deploy the Databricks App

4.1 Deploy the App

4.2 Get the App URL

Step 5: Verify the Deployment

5.1 Check Component Status

5.2 Check Database Connectivity

5.3 Access the UI

Next Steps

Configure LLM Settings

Add Data Connections

Create Agents

Set Up Authentication

Updating Genesis

Getting Started

Creating A Mission

Blueprint

Genesis Data Agents

Genesis Agent Security Controls

Genesis Data Agent's Toolkit

Setup

Slack and Teams

Data Connections

Deployment Options

​Overview

​Architecture Diagram

​Prerequisites

​Step 1: Set Up Lakebase PostgreSQL

​1.1 Create a Lakebase PostgreSQL Instance

​1.2 Create the Genesis Database and Schema

​Step 2: Build and Push the Genesis Container

​2.1 Build the Docker Image

​2.2 Push to Container Registry

​Step 3: Create the Databricks Compute Cluster

​3.1 Create Cluster with Custom Container

​3.2 Set Environment Variables

​Genesis Backend

​3.3 Configure Init Script

​3.4 Start the Cluster

​Step 4: Deploy the Databricks App

​4.1 Deploy the App

​4.2 Get the App URL

​Step 5: Verify the Deployment

​5.1 Check Component Status

​5.2 Check Database Connectivity

​5.3 Access the UI

​Next Steps

Configure LLM Settings

Add Data Connections

Create Agents

Set Up Authentication

​Updating Genesis

Overview

Architecture Diagram

Prerequisites

Step 1: Set Up Lakebase PostgreSQL

1.1 Create a Lakebase PostgreSQL Instance

1.2 Create the Genesis Database and Schema

Step 2: Build and Push the Genesis Container

2.1 Build the Docker Image

2.2 Push to Container Registry

Step 3: Create the Databricks Compute Cluster

3.1 Create Cluster with Custom Container

3.2 Set Environment Variables

Genesis Backend

3.3 Configure Init Script

3.4 Start the Cluster

Step 4: Deploy the Databricks App

4.1 Deploy the App

4.2 Get the App URL

Step 5: Verify the Deployment

5.1 Check Component Status

5.2 Check Database Connectivity

5.3 Access the UI

Next Steps

Updating Genesis