Skip to main content

​
Authentication Setup

1

Go to Google Cloud Console

Access the concole here
2

Selecting/Creating Your Project

To select or create your project, note your PROJECT_ID (in all lower case)
3

Navigate To Service Accounts

Navigate to “IAM & Admin” > “Service Accounts”
4

Creating A Service Account

Click “Create Service Account”
5

Fill In The Details

Fill in the required details:
  • Name: (e.g., “genesis-bigquery-sa”)
  • Description: Optional
6

Grant The Following Roles

  • BigQuery Data Viewer
  • BigQuery Job User
  • BigQuery User
7

Creating And Downloading The JSON Key File

  • Go to the “Keys” tab
  • Click “Add Key” > “Create new key”
  • Choose JSON format
NOTE: The downloaded key file will contain your service account credentials including project_id, private_key, client_email and other necessary authentication details.

​
Environment Variable Setup

Before connecting, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to your key file path 
# Linux/MacOS
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
Reference Example: 
export  GOOGLE_APPLICATION_CREDENTIALS="/Users/justin/Documents/keys/2gdrivegenesistest-8b76dc054630.json"

# Windows PowerShell
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\your\service-account-key.json"

# Windows CMD
set GOOGLE_APPLICATION_CREDENTIALS=C:\path\to\your\service-account-key.json
IMPORTANT: Make sure you set this before running the Genesis server and set it in the same terminal window where you will be running Genesis.

​
Test Harness Available

In the /database_demos/bigquery folder that gets installed with Genesis, you can find an additional readme and a small Python program that can be used as a test harness to confirm that your BigQuery connection is working.  You can use this if you have any issues connecting to BigQuery with the procedure below.

​
Adding BigQuery Connection in Genesis

Tell Eve to add the BigQuery connection using this format.  Make sure your lowercase is in all lowercase. Copy the prompt below to follow along 
Eve, add a connection to BigQuery called my_bigquery and use the connection string: bigquery://project_id. Allow all bots to access.
Reference Example: 
Eve, add a connection to BigQuery called my_bigquery, use connection string: bigquery://analytics-project-123, allow all bots to access.
Example Used: 
Eve, add a connection to BigQuery called my_bigquery, use connection string: bigquery://gdrivegenesistest, allow all bots to access.

Eve adds a connection to BigQuery

​
Check your Database Connections

After setting up the connection, verify it on the Database Connections tab in the Genesis GUI.

my_bigquery and other databases connected appear here

​
See What Schemas Are Visible In Your Project

Copy the prompt below to follow along 
Eve, use query_database to list the catalog_name (as project_id) and schema names visible in the my_bigquery connection

Eve listing the Project ID and Schema Names visible in the connection

​
Adding BigQuery Datasets To The Harvest

You can harvest specific datasets by asking Eve to add all visible schemas, or you can list specific schemas. Copy the prompt below to follow along 
Eve, add project/database gdrivegenesistest in connection my_bigquery to the harvest, include all schemas

Eve adding my_bigquery to the harvest

After a few minutes, you can see results of the harvest on the Config / Harvester Summary page in the Genesis GUI.

Harvester Status is where you can see all of the data sources being harvested

In this case, there is only one table in one of these datasets, called random_numbers.  Let’s ask Eve to search for it. Copy the prompt below to follow along

Using the SearchMetadata tool she can now find this table and knows where it is located

​
Removing The Connection

 You also have the option to remove the connection.
NOTE: This will also remove related harvest information
remove the connection to my_bigquery

​
Troubleshooting

​
Common Issues and Solutions

  • Verify the key file path is correct
  • Ensure environment variable is properly set
  • Check if the service account key is valid
  • Environment variable not set correctly
  • Key file not accessible or corrupted
  • Check if all required IAM roles are granted
  • Verify the service account has the necessary permissions
  • Ensure project access is enabled
  • Confirm project ID is correct
  • Verify service account has access to the project
  • Validate dataset and table permissions
  • Check SQL syntax and dataset references
  • Make sure to refer to “projects” and “datasets”, which are the Bigquery equivalent to “Databases” and “Schemas.”
⌘I