Skip to main content

​
Overview

Genesis supports enterprise governance through environment separation and credential isolation. This guide explains the purpose and role of Dev, QA, and Prod environments when deploying Genesis, independent of the underlying platform.

​
Core Principle: Environment Isolation

Genesis mirrors traditional team-based access control:
  • Development teams work in sandboxes with dev credentials
  • QA runs in isolated test environments—typically automated test suites, with optional manual or UAT validation
  • Production operations use limited-scope credentials with approval gates
With Genesis, these same principles apply - Genesis agents operate within environment-specific boundaries defined by credential access, not deployment platform.

​
The Three Environments

​
Development Environment

Purpose: Experimentation, iteration, and initial pipeline generation Characteristics:
  • Genesis agents generate and test new pipeline code
  • Connected to development data sources (sample data, dev databases, dev warehouses)
  • No production credentials or access
  • Failures have no customer impact
  • Developers iterate freely without approval gates
Credential Scope:
  • Dev database connections
  • Dev warehouse compute
  • Test API keys
  • Non-production data only
Typical Activities:
  • “Generate a pipeline to process vendor CSV files”
  • “Add a new transformation to this existing pipeline”
  • “Test schema evolution handling”
  • Iterating on data quality rules

​
QA/Testing Environment

Purpose: Validation before production deployment Characteristics:
  • Pre-production environment for testing pipeline code
  • Automated test suites (regression, integration, data quality) run as part of CI/CD—typical for enterprises
  • Performance testing with production-like volumes
  • Isolated from production data and credentials
  • Optional manual or UAT validation where required by process
Credential Scope:
  • QA database connections
  • QA warehouse compute
  • Test environment APIs
  • Anonymized or synthetic data matching production schema
Typical Activities:
  • Deploy to QA and run automated pipelines against QA data
  • Automated regression, integration, and data quality tests validate outputs
  • Performance tests with production-like data volumes
  • Optional user acceptance testing or manual sign-off where policy requires

​
Production Environment

Purpose: DevOps support, monitoring, and troubleshooting Characteristics:
  • Genesis agents perform production support operations
  • Read access to production data for analysis
  • Root cause analysis and debugging
  • Audit logging enabled
Credential Scope:
  • Production database connections (read-only for analysis)
  • Production warehouse compute (for investigations)
  • Production API keys (monitoring/observability)
  • Real customer/business data (for troubleshooting context)
Typical Activities:
  • Root cause analysis: “This pipeline failed at 3am, what happened?”
  • Error triage: “Why are we seeing data quality issues in the customer table?”
  • Performance investigation: “Why is this transformation taking 2 hours instead of 20 minutes?”
  • Fix generation: Genesis analyzes the issue and generates PR with proposed fix
  • Monitoring and alerting: Genesis analyzes production logs and metrics
Critical distinction: Genesis in production does NOT execute pipeline code. It investigates issues, analyzes production behavior, and generates fixes that go through your standard approval process.

​
Security Boundary: Credential Isolation

The hard boundary: Each Genesis instance only has access to credentials for its designated environment. This prevents:
  • Accidental production writes during development
  • Test code running against production databases
  • Development experiments impacting live operations
  • Direct production modifications without approval

​
Standard Development Lifecycle

​
1. Development Phase

Environment: Genesis-Dev Genesis agents generate or modify pipeline code. Developers iterate on logic, test transformations, and validate against dev data sources. Output: Generated code (Python, SQL, dbt, PySpark) Human Gate: None - free experimentation

​
2. Version Control

Genesis-generated code exports to your Git repository. Standard pull request workflow:
  • Peer review of generated code
  • Automated linting and testing
  • Discussion of approach and edge cases
Output: Approved, merged code Human Gate: PR approval required

​
3. QA/Testing Phase

Environment: Genesis-QA Code deploys to QA environment. Automated test suites (regression, integration, data quality) execute against QA data and validate outputs. In many enterprises, QA passage is driven by automated tests in CI/CD; manual or UAT sign-off is used only where policy requires it. Output: Test results; approval or sign-off per your process Gate: Automated test passage required; human sign-off if required by your governance

​
4. Production Deployment

Via CI/CD: Approved code deploys to your production data platform (Snowflake, Databricks, etc.) Your orchestration tools (Airflow, dbt Cloud, Snowflake Tasks) execute the pipelines in production. Human Gate: Deployment approval, change management ticket (if required)

​
5. Production Support

Environment: Genesis-Prod When issues occur, Genesis agents in production:
  • Analyze errors and logs
  • Investigate data quality problems
  • Perform root cause analysis
  • Generate fixes as pull requests
These fixes then cycle back through steps 2-4 (Git → QA → Prod deployment). Human Gate: PR approval for any Genesis-generated fixes

​
Production Support Workflow

​
Example: Pipeline Failure at 3am

  1. Alert fires: Production pipeline fails
  2. Genesis-Prod investigates:
    • Reads error logs from orchestrator
    • Queries production data to understand state
    • Analyzes recent schema changes
    • Reviews upstream data sources
  3. Genesis determines cause: “Upstream vendor added a new column that breaks schema validation”
  4. Genesis generates fix:
    • Updates schema validation logic
    • Adds handling for new column
    • Creates PR with fix and explanation
  5. Standard approval process:
    • On-call engineer reviews PR
    • Automated tests run in CI/CD
    • Fix deployed to QA; automated test suites validate
    • After QA passes (and any required sign-off), deployed to production
  6. Pipeline resumes: Your orchestrator retries with fixed code
Key point: Genesis identified the problem and proposed the solution, but the fix still went through your governance process.

​
Common Change Scenarios

​
New Data Source

  1. Dev: Genesis generates ingestion pipeline, test with sample files
  2. Git: Code review, validate error handling
  3. QA: Automated suites run against historical test data and validate transformations
  4. Prod Deploy: CI/CD deploys to production, orchestrator executes
  5. Prod Support: Genesis-Prod monitors for issues

​
Production Error Investigation

  1. Prod Support: Genesis-Prod analyzes error
  2. Dev: Genesis-Dev reproduces issue and develops fix
  3. Git: PR with fix, peer review
  4. QA: Automated regression (and other) tests confirm fix works
  5. Prod Deploy: Fix goes live through CI/CD

​
Performance Degradation

  1. Prod Support: Genesis-Prod identifies slow query
  2. Dev: Genesis-Dev optimizes transformation logic
  3. QA: Automated performance tests validate improvement
  4. Prod Deploy: Optimized code deployed

​
Human-in-the-Loop Control Gates

Genesis respects your existing governance: Key insight: Genesis accelerates development AND troubleshooting, not deployment or execution. Your approval gates remain fully in place.

​
Best Practices

​
Environment Separation

Deploy separate Genesis instances per environment
  • Different credentials for each
  • Network isolation where possible
  • Clear naming conventions
Never share credentials across environments
  • Dev credentials stay in dev
  • Prod credentials only in prod (read-only for Genesis)
  • Use secrets management systems
Use version control as source of truth
  • All Genesis-generated code in Git
  • Deployment from Git, not from Genesis UI
  • Tag releases for traceability

​
Production Access

Genesis-Prod has read-only database access
  • Can query for troubleshooting
  • Cannot modify data directly
  • Cannot execute pipeline code
Audit all production investigations
  • Log what Genesis queries
  • Track what issues were investigated
  • Document fixes proposed

​
Credential Management

Least privilege principle
  • Genesis-Dev: dev credentials only
  • Genesis-QA: QA credentials only
  • Genesis-Prod: read-only prod credentials
Rotate credentials regularly
  • Genesis picks up new credentials automatically
  • No code changes required
  • Maintain audit trail
Use secrets vaults
  • Use your organization’s secrets management system
  • Never hardcode credentials
  • Centralized rotation and revocation

​
Deployment Discipline

Always test in QA first
  • No direct-to-prod deployments
  • Validate with production-like data volumes
  • Performance testing before release
Code review everything
  • Even AI-generated code needs human validation
  • Review for security, performance, correctness
  • Document approval decisions
Monitor after deployment
  • Track pipeline success rates
  • Alert on failures or anomalies
  • Genesis-Prod can investigate alerts

​
What NOT to Do

Don’t give Genesis-Prod write access
  • Read-only for analysis and troubleshooting
  • Fixes must go through standard process
Don’t skip QA for “urgent” fixes
  • Even Genesis-generated fixes need automated (and any required manual) validation
  • Use hotfix process if needed, but run automated tests first
Don’t use prod credentials in dev
  • Even “read-only” access is dangerous
  • Use anonymized data instead
Don’t bypass code review
  • Genesis-generated fixes still need human approval
  • Emergency changes get expedited review, not no review