> ## Documentation Index > Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt > Use this file to discover all available pages before exploring further. # Quality Assurance and Testing > Genesis software quality assurance and testing procedures applied to every release ## 1. Purpose and Scope This document defines the quality assurance and testing procedures that Genesis applies to every release of its software. It is intended for enterprise customers, security reviewers, and technical stakeholders who require a clear understanding of the controls Genesis uses to ensure functional correctness, performance, security, and overall reliability of the product before it is delivered. The procedures described in this document apply to all releases of Genesis software and are executed continuously throughout the development and release lifecycle. ## 2. Quality Objectives Genesis software is validated against six quality objectives. Every release must satisfy each of these objectives before it is made available to customers: 1. **Functional Correctness** — The software behaves according to its intended purpose. 2. **Performance** — The software meets specific response times and throughput targets under load. 3. **External Compatibility** — The software interoperates correctly with all supported third-party providers, including large language model (LLM) providers. 4. **Security** — The software is free of known vulnerabilities at the time of release. 5. **Component Integrity** — Individual software components behave correctly in isolation (unit-level correctness). 6. **Code Quality** — The software adheres to documented engineering standards and architectural guidelines. ## 3. Testing Lifecycle Testing is performed continuously across five distinct stages of the development lifecycle: * **Pre-commit validation**: applies to code on the developer's machine. * **Pre-merge validation**: applies to pull requests before they are merged into the main code branch. * **Continuous validation of the main code branch** * **Release Candidate validation** * **Customer Release validation** ### 3.1 Stage 1 — Pre-Commit Validation Before any source code change is accepted into the central code repository: * Engineers are required to validate functionality on local development environments. * Automated linting and code-quality checks are enforced through pre-commit hooks. This prevents committing non-compliant code. * An AI assistant generates automated tests for any new functionality, which are then verified by a human engineer. ### 3.2 Stage 2 — Pre-Merge Validation Every pull request is subject to automated validation before it can be merged into the main code branch. The merge is blocked until all required checks pass. On every proposed change, the Genesis Continuous Integration system (CI) automatically runs the following checks: * A suite of approximately **500 integration tests** that provide rapid feedback on functional correctness. * A suite of over **3,000 unit tests** that validates component-level correctness. * A compatibility check against all supported LLM providers. * Static code analysis and architectural review by automated review tools. Their findings are systematically addressed before merge. * Source-code security analysis, which detects unsafe coding practices such as hard-coded passwords or SQL injection. * For changes that touch container or dependency files: build-time vulnerability scanning of the resulting container. The security controls are described in detail in Section 4.4. ### 3.3 Stage 3 — Continuous Validation of the Main Code Branch A suite of more comprehensive and longer-running checks executes on a recurring schedule against the main branch. This stage exists to detect regressions, performance degradation, and newly discovered vulnerabilities that may be missed by the lighter pull request tests. * **Comprehensive integration tests**, executed against a full system image, run **every three hours** to detect functional regressions. * A **standardized load test** executes **every weekday** to detect performance regressions. * An **expanded compatibility benchmark**, exercising multiple end-to-end scenarios across all supported LLM providers, executes **weekly**. * A **security vulnerability scanner** is run against the container image built from the latest main branch **every two hours** to detect newly disclosed HIGH and CRITICAL vulnerabilities. This is described in detail in Section 4.4.3. Failures detected at this stage trigger immediate notification to the engineering organization for triage and remediation. ### 3.4 Stage 4 — Release Candidate Validation When a new release candidate is produced, the following validation is performed before the candidate is considered eligible for promotion to General Availability (GA): * **Automated User Acceptance Testing (UAT).** The release candidate is automatically deployed to a dedicated quality assurance environment, where an automated UAT suite is executed. The UAT suite exercises end-user workflows. It was developed by modeling the actions of a human quality assurance engineer. * **Pre-Production Validation.** Each release candidate is deployed to a pre-production environment that contains a production-scale dataset with over 7,000 conversation threads and 30 GB of data. This exceeds the volume of a typical customer deployment. This environment is used daily by Genesis personnel, including non-technical users, to perform real work, and thus provides real-world validation under realistic usage patterns. * **Performance Validation.** The standardized load test is re-executed against the release candidate. * **Compatibility Validation.** The full compatibility benchmark is re-executed against the release candidate for all supported LLM providers. * **Security Validation.** Vulnerability scanning is re-executed against the release candidate immediately upon its creation and on every modification. ### 3.5 Stage 5 — Customer Release Validation Before a release candidate is promoted to General Availability and made available to customers, the following procedure is executed: 1. The release must have been tested successfully in the pre-production environment. 2. All automated test suites, including UAT, performance, compatibility, and security, are re-executed. 3. **Upgrade testing** is performed in a dedicated release-validation environment. The new release is installed over the previous Generally Available version to ensure that customer upgrades will succeed without data loss or regression. 4. **Fresh-install testing** is performed in the same release-validation environment to ensure that new customer deployments will succeed. 5. Change log documentation is produced via an AI-assisted process. 6. A manual test protocol is executed to verify that newly added features that appear in the change log work as expected, and that no regressions occur. ## 4. Test Categories in Detail ### 4.1 Functional Correctness Functional correctness is validated through a combination of: * **Pre-commit local validation** by engineers. * **Approximately 500 fast integration tests** on every code change. * **Comprehensive integration tests** executed every three hours against the system container image. * **Actual usage** of release candidates by Genesis personnel in the pre-production environment. * **Automated UAT** that is executed for every release candidate. * **Manual test protocol** that is executed for every customer release. Automated tests are generated with the assistance of AI tooling and are subject to mandatory human review before they are accepted into the test suite. ### 4.2 Performance Performance is validated through: * **Automated load test** that is executed every weekday against the latest main branch, which verifies that critical tasks complete within their allotted time budget under load. * **Real-world performance monitoring** in the pre-production environment, which carries a dataset larger than that of a typical customer. * **Mandatory load test execution** before any release is promoted to General Availability. ### 4.3 External Compatibility Compatibility with supported third-party providers is validated through: * A lightweight **compatibility check** executed against all major LLM providers on every code change. * A more comprehensive **compatibility benchmark** that runs weekly against the latest main branch, and for every release candidate and customer release. This check exercises multiple end-to-end scenarios across all major LLM providers. * An **LLM configuration matrix QA harness** that applies provider configuration cells against live servers, runs the hidden built-in `llm_qa_lite` and `llm_qa_full` blueprint probes (started by name; they are not listed in the Blueprint Library), and verifies deterministic tier wiring and token-exact mission artifacts. The harness uses differential gating to catch config-attributable stale-serving regressions when switching between LiteLLM-engine providers (`azureopenai`, `databricks`, `litellm_*`, and **LiteLLM Proxy**). See [Blueprint Library and Missions → LLM configuration matrix probes](/docs/Blueprint/Blueprint-Library-And-Missions#llm-configuration-matrix-probes) for probe blueprint details. ### 4.4 Security The Genesis security validation program operates at multiple levels, as described below. #### 4.4.1 Source Code Security Analysis Every pull request is automatically analyzed for security anti-patterns and critical correctness defects before it can be merged. The analysis covers a comprehensive set of rules including hard-coded credentials, insecure cryptography, SQL injection, shell injection, XML processing vulnerabilities, and insecure SSL/TLS usage, among other code-correctness defects that have security implications. Any pull request that introduces a security or correctness violation is automatically blocked from merging. #### 4.4.2 Detecting Secrets in Code Every pull request is automatically scanned for leaked credentials, API keys, authentication tokens, private keys, and cloud credentials. Detection of any such secret automatically blocks the pull request from merging. #### 4.4.3 Dependency Vulnerability Scanning The Genesis dependency vulnerability scanner runs every two hours and is applied to: * The container image built from the latest main branch. * The latest release candidate. * The latest customer GA branch. Additionally, every release candidate is checked upon creation, and every customer release is checked before promotion to GA. The images are automatically scanned for HIGH and CRITICAL vulnerabilities in their Python and Node.js dependencies using Trivy by Aqua Security, an industry-standard vulnerability scanner. This scanner was selected after a structured evaluation of multiple alternatives against an external reference vulnerability report. #### 4.4.4 The Importance of Continuous Vulnerability Scanning Several hundred new vulnerability advisories are published every week. Therefore, a software version that was deemed secure at one point in time may become not secure at a later stage. To detect such situations, Genesis runs a vulnerability scan **every two hours** not only on the latest main branch, but also on the latest release candidate and on the generally available version. When a security vulnerability in any of those versions is detected, a failure notification is sent to the engineering team, and an AI-proposed solution is automatically crafted. Human engineers can then quickly apply the necessary security fixes, which allows keeping the software current and free of vulnerabilities. #### 4.4.5 Documented Accepted-Risk Process In limited cases, a security finding may be a confirmed false positive or may reference a vulnerability for which no fix is yet available. Such cases are handled through a documented accepted-risk process. * False positives in code must be annotated with the specific security rule identifier and a written justification. * Vulnerabilities in container-level dependencies with no available fix are recorded in an explicit ignore list and then tracked for re-evaluation as soon as a fix becomes available. They are listed on every security report for visibility. All accepted-risk entries are reviewed periodically and removed once a fix is available. #### 4.4.6 Blocking Vulnerable Releases A release cannot be promoted to General Availability while any HIGH or CRITICAL vulnerability is outstanding against the release candidate or its underlying artifacts. All security controls described above are re-executed against every release candidate and every customer release as part of the customer release validation procedure (Section 3.5). ### 4.5 Component Integrity (Unit Testing) * A suite of more than **3,000 unit tests** executes on every pull request. * A pull request cannot be merged into the central repository unless all unit tests pass. * Unit tests are generated with the assistance of AI tooling and are subject to mandatory human review. ### 4.6 Code Quality * Automated linting is enforced both within the development environment and at commit time through pre-commit hooks. * Documented coding standards and architectural guidelines govern all code contributions. * Every pull request is subject to automated architectural and code review. An automated workflow ensures that all review findings are addressed before a change is merged. ## 5. Governance and Enforcement The following controls ensure that the procedures described above are consistently applied: 1. **Merge Gating.** Proposed code changes that do not satisfy the required automated checks cannot be merged into the central code repository. 2. **Pre-Commit Enforcement.** Code that does not satisfy code-quality criteria cannot be committed by engineers. 3. **Continuous Monitoring.** The aggregated state of all quality checks across the latest code, the active release candidate, and the most recent customer release is monitored continuously by the engineering organization. 4. **Failure Notification.** Failures detected at any stage trigger immediate notifications to the engineering organization for triage and remediation. 5. **Release Gating.** No release is promoted to General Availability until all stages of the customer release validation procedure have been completed successfully. ## 6. Summary Genesis applies a multi-layered testing program in which every release is validated against six quality objectives across five lifecycle stages, ranging from individual code changes through customer release. The program combines automated testing, AI-assisted test generation under human review, continuous real-world usage by Genesis personnel on a production-scale dataset, mandatory upgrade and fresh-install validation, early-adopter deployment, and a layered security validation program that operates at the source-code, build-artifact, and published-artifact levels. These controls operate in concert to ensure that software delivered to customers meets Genesis's standards for correctness, performance, compatibility, security, and code quality.