Skip to main content

​
Overview

Genesis integrates with Talend Open Studio through APP_CONFIG — not through Config -> Connections. Agents use the talend_tools group to:
  • List jobs by parsing the project’s process/*.item files.
  • Describe job structure (components, schemas, connections, lineage summary, and observed migration evidence from the .item XML). See Talend Tools for the evidence fields and redaction behavior.
  • Run already-built standalone job artifacts and poll their output.
Talend Studio itself has no headless API. Building jobs still happens in Studio; Genesis executes the exported artifacts. For tool parameters and return shapes, see Talend Tools.

​
Prerequisites

​
Configuration

Set values in Config -> Advanced Settings (search for talend) or through environment variables / config profiles.

​
Local execution (default)

Use this layout when Genesis can read the project and job artifacts from its own filesystem:
  1. Set talend.project_path to the Talend project root (the directory that contains talend.project and process/). Example: /opt/talend/workspace/MY_PROJECT Env: TALEND_PROJECT_PATH
  2. Build jobs in Talend Studio (Build Job → standalone export) and unzip the artifacts under a single directory.
  3. Set talend.jobs_dir to that directory. Example: /opt/talend/jobs Env: TALEND_JOBS_DIR
  4. Optionally tune talend.run_timeout_seconds (default 900) — maximum seconds before Genesis kills a hung job.
Where Genesis runs as root and the image provides the restricted cortex_sandbox user (Snowflake, EKS and self-hosted Docker), a job runs as cortex_sandbox, without the server’s credentials: its environment holds only PATH, locale, TLS trust-store and proxy settings, JAVA_HOME, and its own home directory, and the job starts in its own build directory. That user must be able to read and search talend.jobs_dir and the job’s files, and the job must get its connection settings from its context (--context_param values or the build’s context files), not from Genesis’s environment. Where Genesis cannot switch users (Genesis Desktop, a local development server, or any server not running as root), jobs run as the server’s own user, as before.

​
Remote talend-runner

When Genesis cannot access built jobs locally (for example Genesis on GKE and jobs on a dedicated VM):
  1. Deploy a talend-runner HTTP service on a host that has JDK access and the built job artifacts on disk.
  2. Set talend.runner_url to the service base URL (no trailing slash). Example: http://10.0.0.5:8899 Env: TALEND_RUNNER_URL
  3. Keep talend.project_path pointed at a local project checkout if you want describe_talend_job to work. The runner lists and executes built jobs only; it does not expose .item internals.
  4. Leave talend.jobs_dir empty when talend.runner_url is set — list/run operations use the remote service instead.
The runner speaks a simple JSON API: GET /healthz, GET /jobs, POST /run, and GET /run/{run_id}. Genesis tools return the same shapes whether execution is local or remote.
In a Databricks App, run_talend_job is refused because a local Java process would run as the App user beside its credentials. The refusal message points to talend.runner_url, but this tool currently refuses the call before using that remote runner as well. Remote execution from the App therefore requires a code change; configuring talend.runner_url alone does not enable it.

​
Talend Cloud (reserved)

talend.cloud_base_url and talend.cloud_token are defined in the schema for a future Talend Cloud Management Console surface. They are ignored by the current built-artifact execution path.

​
Verify the setup

From a Genesis chat thread with talend_tools enabled:
  1. Call list_talend_jobs — confirm project_jobs and/or built_jobs are populated.
  2. Call describe_talend_job with a name from project_jobs.
  3. Call run_talend_job with a name from built_jobs, then poll get_talend_run.
If both lists are empty, check the resolved project_path and jobs_dir in the list_talend_jobs response and fix the configuration paths.

​
Lineage scanning

Twin’s Talend scanner parses the same .item format to publish job components and data-flow edges into the Context Graph. Configure talend.project_path (or point the scanner at the project) as part of your Digital Twin harvest setup. See Digital Twin and Context Graph.