> ## Documentation Index > Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt > Use this file to discover all available pages before exploring further. # Palantir Foundry Migration Tools > Plan and execute a Palantir Foundry to Databricks migration, then prove each landed dataset with Genesis Prover. ## Overview The `palantir_foundry_migration_tools` group moves data and pipeline logic from a Palantir Foundry tenant into Databricks. It never writes to Foundry. Assign it only to agents that also have the Databricks and repository tools needed for the target side. For read-only inventory and query operations, use [`palantir_foundry_tools`](/docs/Integrations/Palantir-Foundry/Palantir-Foundry#use-with-agents) instead. Every migration needs two saved connections: a `palantir_foundry` source and a Databricks target. Credentials resolve through the Connections service. The Before running it, confirm that the authenticated user can read both connections; see [Prover connection access](/docs/Genesis-Prover/Overview#connection-access-and-run-provenance) for the separation between agent identity and connection access. Databricks execution steps use the existing `databricks_action` path, not a second executor. ## Migration workflow Call the tools in this order for a pipeline migration: 1. `foundry_plan_migration` — plan pipeline order from Foundry lineage and discover the tenant's ontologies. Before translating, inspect each pipeline's `translatable` and `not_translatable_reason` fields. A Foundry job spec stays in dependency order but has `translatable: false`; handle it first as its reason directs, by landing its output with `foundry_extract_dataset(copy_derived=true)` or porting the job by hand. Translate only the entries marked `true`. 2. `foundry_translate_pipeline` — generate a PySpark script for one pipeline. 3. `foundry_publish_notebook` — publish that script into the Databricks workspace as a notebook. 4. `foundry_extract_dataset` — land inputs in the schemas named by the translation's `next_step`; when `source_schema` is set, not every input belongs in the same schema. 5. `foundry_materialize_ontology` — optionally land selected ontology object types. 6. `databricks_action(action="run_notebook")` — run the published workspace notebook with an explicit, operator-approved `cluster_id`. 7. `foundry_verify_migration` — characterize both sides and submit one Prover proof. For `foundry_verify_migration` identity, see [Prover guidance](/docs/Genesis-Prover/Agents-And-The-API#unattended-mission-visibility). For a signed-in person, planning also opens the person's migration project the first time this space is planned, then resumes it on later plans, including plans from another chat. Read the returned `project` block: its `goals` report progress, and `gaps` and `banner` explain what the plan could not read. When `project.instruction` is present, tell the person what is missing and do not report the migration complete, even if a goal reads **Met for the planned scope**. A plan without a signed-in person has no project block. Project scope and recovery options are described in [Planning project behavior](#planning-project-behavior). ### Planning project behavior Each person's project is bound to the Foundry connection and space, not the chat thread. If the response has `project_candidates`, ask which live project to keep and call the planner again with its `project_id`; the other live projects for the space are retired. An empty candidate list means no project is live, so plan again without `project_id`. A `project_error` explains why no project was bound; the Foundry plan itself is still returned. After evidence has been recorded, the project's goal scope is fixed. If a later plan differs, `scope_diff` reports the difference: `would_add` and `would_drop` are proposed changes, not applied changes; `new_gaps` are recorded even though they cannot be read. On the person's explicit request, `extend_scope=true` applies additions without removing anything, while `new_attempt=true` retires the existing project and starts a fresh one. `project_id`, `extend_scope`, and `new_attempt` are honored only in the person's own chat turn. The response's `retired` and `still_live` fields report whether older projects were retired successfully. These project operations read and write Genesis's database after Foundry planning: one `SELECT`, plus one transaction and three `SELECT`s the first time a space is planned; three `SELECT`s on a later plan, with one `UPDATE` and three additional `SELECT`s when its scope changes. ### Filing copies and translations In a signed-in person's chat, a whole `foundry_extract_dataset` copy is filed under the person's migration project before it starts. A `foundry_translate_pipeline` translation is associated with the project and records its destination when the script is saved. Both tools match the Foundry connection and dataset or pipeline to the plan's scope. If no live project covers the item, the call is refused; plan its space first with `foundry_plan_migration`, then retry. If several projects cover it, ask the person to choose from `project_candidates` and retry with that project's `project_id` in the person's own chat turn. A supplied project id must name one of the person's live projects that covers the item. The project fixes destination fields as work goes ahead. A copy that disagrees with the project's Databricks connection, catalog, or schema is refused and names the value to use; a project-filed copy must use a full Databricks connection id in `type.name` form. A translation must agree with the project's catalog and schemas. If the work is not part of a migration, pass `standalone=true` to either tool. It defaults to `false`; `project_id` defaults to `null`. A copy with `row_limit` is a sample and is not tracked. Mission, scheduled-flow, and scheduled-job work is also untracked. Calls from outside Genesis are untracked unless they specify `project_id`. Genesis records each filed copy on the project's `landed` goal by itself; nothing needs to be noted by hand. Once a minute it looks for copies that have finished and are not recorded yet. A copy still running is not checked: its table can match before it ends, so Genesis waits for the copy to report how it ended. Its automatic look-ups stop 24 hours after the copy was filed, but an ending reported later still gets the copy checked and recorded. A failed copy whose check cannot answer (for example, a stopped SQL warehouse) is also tried again only within those 24 hours; after that it is checked again only when someone cites it. A finished copy is tried until it is recorded. A `foundry_publish_notebook` publish is bound the same way, by the pipeline the script was translated from: in an ordinary chat with no covering project it is refused before the workspace is touched, and `standalone=true` or `project_id` work as they do for a translation. Once a bound publish lands, Genesis registers that version of the notebook on the project's `rebuilt_and_ran` goal, and each `databricks_action(action="run_notebook")` of its `workspace_path` on the same connection is filed under the project and recorded there by itself once the run ends; the run's answer says so in `project_tracking`, and says why when a run is not tracked. A run proves only the publish it was filed against, and only while that publish is still the pipeline's latest: a later publish of the pipeline (to any path or connection), or anything else published at its path, even untracked, makes it not count and makes an earlier pass stale. Within that, a run counts when it succeeded, every task ran the published notebook, and it started after that publish. A run that can never count (it ran another path, Databricks no longer has it, or its publish named no connection) is listed as "Cannot be checked"; any other run is checked automatically for 24 hours. A failed run is shown as "Last run failed" with its reason, and the goal stays unmet until a later run succeeds. Publishing again makes the goal stale for that pipeline until a run of the new notebook is recorded. Name `databricks_connection_id` when publishing: runs of a notebook published without one are listed under the goal as "Cannot be checked", with the reason, and never count. A planned project's `rebuilt_and_ran` expects every pipeline the plan can translate; a pipeline it cannot translate, such as a job spec, is a gap of the plan. A finished copy's table must still hold that copy, with the dataset's stamp and Foundry's row count. The table's identity is read before and after: a table written, or dropped and re-created, during the check is checked again. Each checked copy gets one note, shown as "Recorded by the server for" the project's person, which says when the copy was checked. That verdict is the table as it was at that time; a recorded copy is not checked again. A check that cannot answer yet (for example, a stopped SQL warehouse) is tried again after 1, 5, 15 and 30 minutes, then hourly; the project shows the copy as waiting to check, with the reason. An older copy whose table now holds a newer copy of the same dataset, filed under the same project, reads as replaced and does not count; it never turns `Landed` failed. A table that no longer exists, or that holds a copy which failed, is recorded as failed. A copy that failed before it landed leaves `Landed` as it was, and a failed copy is never counted as finished. `project_note` never records a filed copy. Citing a filed copy's receipt alone on the `landed` goal writes no note. For a copy not yet recorded, it asks that the copy be recorded with its next check's verdict, normally within a minute; for a recorded copy it answers with the recorded verdict and the time it was checked. Registering the dataset by hand, a waiver included, is refused. The job can be paused with `jobs.project_evidence_recorder.enabled`; filed copies then wait and are recorded when it resumes. See [Foundry copy evidence recording settings](/docs/Setup/Advanced-Settings#foundry-copy-evidence-recording) for its default, cost, and timing. ### Ontology coverage in the plan `foundry_plan_migration` returns an `ontology` block alongside the pipeline plan. It lists each tenant ontology and its object types as work to land with `foundry_materialize_ontology`; each object type includes its API name, display name, primary key, and property count. The block also reports `object_types_planned`, `action_type_count` per ontology, `link_types_planned: false`, and `action_types_planned: false`. Link types are not materialized, and action types are returned as stubs to re-author. Ontology coverage distinguishes an empty estate from a read gap. A tenant that does not serve the ontology API has `route_served: false` and does not make the plan incomplete. A listed ontology with no object types is also not work to land; the tenant's seeded `default` ontology may be empty, so the ontology's presence alone does not establish that it models data. If the token cannot list ontologies, `list_error` is set and `incomplete_reasons` includes `ontology_unexamined` with `ontology: null`. If a listed ontology cannot be expanded, it appears in `unreadable` and gets its own `ontology_unexamined` reason. In either case, `plan_complete` is false because ontology coverage is unmeasured, not absent. See [Palantir Foundry connections](/docs/Setup/Connections#palantir-foundry-connections) for the required token access. The ontology read costs one HTTP round trip to list the tenant's ontologies and one more per ontology listed. A two-ontology tenant therefore adds three calls to the planner's existing metadata reads. Planning reads metadata only; it does not materialize ontology rows or write to Foundry. During verification, the Prover includes Foundry-declared keys from dataset schemas and published Pipeline Builder output checks when they can be resolved against the dataset schema. See [Foundry catalog and declared keys](/docs/Genesis-Prover/Overview#foundry-catalog-and-declared-keys) for the scope and unread cases. The translated script preserves Foundry's output column names and case. It avoids duplicate column names; if the compiler cannot determine a node's columns in advance, the script checks them at runtime and stops at that node if duplicates are present. Before reading a derived input, the script checks whether its target table exists. If it is missing, the error names the Foundry pipeline that builds it and directs you to run that pipeline successfully on Databricks first, or land a copy of the dataset from Foundry. If the producer is a job spec rather than a Pipeline Builder pipeline, the dataset must be copied from Foundry. ### Keeping Foundry copies separate from rebuilt tables By default, `foundry_translate_pipeline` reads and writes in `target_schema`. To keep Foundry input copies in a different schema, pass `source_schema` with the schema name alone (not `catalog.schema`); a value containing `.` or no letters or digits is refused. The generated script reads inputs that no Pipeline Builder pipeline builds from `source_schema`; it reads this pipeline's outputs and inputs built by another Pipeline Builder pipeline from `target_schema`, where their producer writes them. Outputs are always written to `target_schema`. Do not move or copy tables between the schemas; land or rebuild each input in the schema the script reads. Use the same two schemas for every pipeline in the migration so a downstream script looks for its upstream table in the right schema. ```python theme={null} foundry_translate_pipeline( pipeline_rid="ri.foundry.main.pipeline.example", target_catalog="analytics", source_schema="foundry_raw", target_schema="rebuilt", ) ``` Follow the returned `next_step` for each input instead of landing every input in one schema. It gives the exact `foundry_extract_dataset` call: ordinary source inputs and job-spec outputs go to `source_schema`; an input built by another Pipeline Builder pipeline goes to `target_schema` if that producer is not being migrated; and an input this pipeline also outputs is copied to `target_schema` only when its table does not exist yet. If migrating an upstream Pipeline Builder pipeline, translate and run it first with the same schemas. A dataset whose origin is `unknown` is assumed to be a source input; check `inputs_with_assumed_schema` in the result and confirm that no pipeline builds it before trusting a copy in `source_schema`. ### Polling asynchronous extractions after restart Use `foundry_get_extraction` to poll an asynchronous extraction and `foundry_list_landings` to inspect target tables. Every running extraction and the 50 most recently finished remain in process memory. When a recovery watch was recorded, its status can also be recovered after a backend restart, and a terminal poll receipt remains available for seven days after its notice is acknowledged. When the watch has a thread target, the background watcher can notify that thread with the recovered status. After a restart, `interrupted` means the copy was interrupted; `may not have started` means the extraction was recorded before its start was confirmed. In either case, the landing may contain a partial copy and the target may already have been promoted, so inspect both before retrying. `no longer tracked` means the outcome is unknown. `unavailable` means a current-boot watch exists but its copy is absent from this process's memory. A `cancelled` result also carries an error and no `next_step`; inspect the landing and target before retrying. If no recovery record is available, the poll cannot establish what committed. Each poll costs an agent tool call and may read saved status. Poll at a human pace using the handle's time estimate. A translated or executed pipeline is not proof of equivalence; use `foundry_verify_migration` for the auditable result. ### Paging datasets without a unique key `foundry_extract_dataset` reads datasets larger than one `50,000`-row chunk in pages. It first seeks along a column measured unique, or partitions by a column whose values form groups that each fit in a chunk. If neither works, it hashes every column to assign rows to buckets and reads each bucket once. The result reports `pagination_mode: "row_hash"`, with `pagination_key_column` describing the split and `partition_group_count` and `partition_max_group_size` reporting the bucket count and largest bucket. In seek mode, Genesis writes each page of up to `50,000` rows to the Databricks landing table before reading the next page. For asynchronous extractions, `groups_total` is the planned number of seek pages or partition groups, and `groups_done` counts pages or groups already landed. Choosing the bucket count costs up to six Spark `GROUP BY` jobs over the whole dataset; extraction then runs one filtered scan per bucket. Each bucket must contain at most `50,000` rows, and the fallback stops at `2,000` buckets. If it still cannot page the dataset, the error distinguishes a size limit (more than `100,000,000` rows) from more than `50,000` identical rows. Pass `row_limit` to land a bounded sample; for the size limit, split the source into several Foundry datasets, and for duplicate rows, aggregate them with a count column. ### Receipts for finished dataset copies A successful whole-dataset copy returns `copy_receipt_id`, the id of a durable receipt Genesis writes after promotion. The receipt records the dataset RID, target table, Foundry and Databricks connection identifiers when available, the matching source and landed row counts, and the copy id stamped on the table as the `genesis.foundry_copy_id` table property. It also records the dataset's branch head before counting and after the copy only when that head's status is confirmed as `COMMITTED`. A missing head, a head not confirmed as committed, or an unreadable branch/status leaves that point unknown; neither recorded head pins which rows were copied. If the two heads differ, the result note says a transaction committed during the copy. The receipt is evidence that Genesis completed the copy and the counts agreed; it does not prove the table contents match Foundry. Use `foundry_verify_migration` for that comparison. Genesis writes a receipt only when the call did not include `row_limit` and both counts are whole numbers that agree. For a dataset with no schema, the counts are the number of files, with one landed row per file. A successful copy without a receipt returns `copy_receipt_id: null` and `copy_receipt_note` explaining why; a failed copy returns neither field. A non-null receipt can also have a `copy_receipt_note` when a branch-head read or the Databricks connection identifier needs attention. If `databricks_connection_id` is not a full `type.name` identifier, the receipt does not record a Databricks connection. A credential rejected on the branch-head read after promotion does not undo the copy or receipt; the result carries `fault: "credential_rejected"` and `remediation`, and the receipt's ending branch head is unknown. For a copy without `row_limit`, Genesis reads the branch and its head's status before counting. If the counts qualify for a receipt, it reads them again after the copy. A completed head check uses two Foundry requests, each issued once without retry or waiting on `Retry-After`. The five-second caller budget narrows the branch request's socket timeout, then passes whatever remains to the status request instead of leaving it at the client's 120-second default. If the branch read consumes the whole budget, the status request is skipped and the head is recorded as unknown. These socket timeouts bound connection and each wait for bytes separately, not the total wall-clock duration. Writing the receipt adds one write to Genesis's metadata database. For `async_mode` copies, the receipt is written when the background copy finishes and appears in the successful `foundry_get_extraction` result, not when the extraction is handed off. Copies from a backing dataset during ontology materialization follow the same receipt rules; ontology objects paged through the object API are not dataset copies and have no receipt. ### Complex columns and conversion losses `foundry_extract_dataset` lands Foundry `ARRAY` columns as Databricks arrays and `STRUCT` columns as structs, including nested arrays and structs when their declared shape can be carried. `MAP` columns, containers with missing or unsupported nested types, and structs without named fields land as JSON text instead. The result's `widened_to_string` lists each such landed column as ` ()`; use it to identify columns whose Databricks type is text rather than the Foundry type. The result's `values_lost_to_type` reports per-column counts of source values that could not be converted and were landed as `NULL`. A whole-copy receipt is not written when this field is non-empty: fix the conversion loss before treating the landed table as a complete copy. When the copy was filed under a migration project, Genesis records it as failed, so it cannot count toward `Landed`; the promoted table remains available for inspection. Nested member conversion failures can null the whole array or struct value. ### Pinning and protecting the extraction target Pass `expected_table` as the full `catalog.schema.table` name when the copy must land at one specific destination. Genesis refuses the extraction before reading rows if the Foundry dataset name resolves to a different table. This also catches `copy_derived=true` when its copy destination would be moved aside from the expected table. Set `require_absent=true` when the target must be created only if it does not already exist. If the target exists when promotion runs, Genesis does not replace it; the staged rows remain in the `__landing_` table reported by `foundry_list_landings`. ```python theme={null} foundry_extract_dataset( dataset_rid="ri.foundry.main.dataset.example", databricks_connection_id="warehouse-connection", target_catalog="main", target_schema="migration", expected_table="main.migration.airports", require_absent=True, ) ``` ### A rejected credential is refused, not planned around These tools read a tenant one resource at a time, and each read degrades on its own so that a single unreadable dataset cannot stop a migration: its lineage is reported as `unknown`, its table takes a name derived from its rid, its schema is left out of the map. That is the right answer for a metadata route a tenant does not serve, and the wrong one for a rejected token, which fails every read — the plan would then describe a sparse estate instead of naming the credential. So a Foundry `401` refuses the whole call. The result carries `success: false`, `fault: "credential_rejected"`, the error Foundry returned, and a `remediation` naming the instant **this connection's stored copy** of the credential was written. That instant is the one that matters: Genesis authenticates with the copy in the connection's secret row, not with whatever the tenant issued most recently, so a token renewed in Foundry after that instant has never reached the connection. A refusal that arrives after work has already been done — an ontology type landed, seek pages or partition groups staged — still reports what was written and the landing table it went to, because a refusal must not be the reason a completed write becomes unreachable. **This covers `401` only.** A token that is valid but lacks permission answers `403`, and a `403` really is a fact about the resource it names, so those stay per-dataset: the lineage reads `unknown`, the dataset is listed in `unknown_origin_datasets`, and — because `plan_complete` is computed from external ingestion, unreadable connections and dependency cycles alone — a scope whose every dataset was denied can still report `plan_complete: true` with an empty plan. Read `unknown_origin_datasets` alongside the verdict until that is addressed. Publishing and running are separate steps. `foundry_publish_notebook` writes the notebook the customer's engineers open and review; it runs nothing and reads no data. When the publish result has `execution_ready: true`, run its `workspace_path` with `databricks_action(action="run_notebook")`, pass an explicit operator-approved `cluster_id` (an existing cluster id, or the literal `serverless` only when approved), retain the returned `databricks_run_id`, and poll it with `databricks_action(action="check_run_status")`. Do not omit `cluster_id`: omission selects serverless implicitly. After a successful Job run, call `foundry_verify_migration` once with `rescan: true` for each output mapping, then poll or resume that same proof without starting another rescan. The publish result also includes SHA-256 receipts for the generated script and imported notebook; those establish byte chronology, not which bytes Databricks executed. `foundry_run_pipeline_script` remains a diagnostic Spark Connect path. It can run the generated script and report what it wrote, but it creates no Jobs run receipt and is not the intentional-demo execution record. Its `declared_output_tables` lists literal table names in `.saveAsTable(...)` calls, including calls with the table name on the line after the opening parenthesis. On success, `output_tables` reports those tables with their row counts. A run still going after about four minutes has not failed. The call returns `status: "running"` with a `pipeline_run_id`, and the run continues. Poll `foundry_get_pipeline_run` for its result instead of calling `foundry_run_pipeline_script` again, which would start a second run over the same tables. Pipeline runs are tracked in the backend's memory, and only the 50 most recently finished runs stay pollable. After a backend restart, or once 50 later runs have finished, the id is unknown; the output tables show what the run wrote. At most four pipeline runs go at once in one backend. A fifth call is refused with `status: "not_started"` and the ids of the running ones to poll; it starts nothing. Each output whose Delta version changes during the run is marked as a rebuild, even if the script later fails. The run attempts to remove any Foundry-copy or sample stamp and sets the table property `genesis.foundry_rebuilt_by` to `foundry_run_pipeline_script`, so a marked output cannot be adopted or read as a landed copy. If the script succeeds but a changed output cannot be marked, the tool returns `success: false` and names the unmarked tables. Cost: one full Spark run that reads the inputs and overwrites the outputs, one count per output, two Delta history reads per declared output (before and after), and two property statements per output the run wrote. ### What publishing will not overwrite The workspace import replaces whatever is at a path and offers no precondition, so `foundry_publish_notebook` establishes what is there first, with one export, and refuses rather than replacing. It publishes onto a path Databricks confirms does not exist, and over a notebook whose provenance header records the same Foundry pipeline — that second case is how a re-translated pipeline updates its own notebook. It refuses a notebook belonging to a different pipeline, a notebook carrying no provenance to identify it by, a script that carries none of its own, and a path the export could not read at all; each refusal names the reason and points at `name` for publishing under a distinct notebook name. Two Foundry display names can fold to one workspace name (`a/b` and `a.b` both become `a_b`), which is what makes the check necessary rather than defensive. A script is also refused when it carries the Databricks cell separator (`# COMMAND ----------`) on a line of its own: Databricks reads that line as a cell boundary wherever it appears, so the notebook would be a different program from the script that was translated and run. ## Tool reference | Tool | Required parameters | Important optional parameters | Purpose | | - | - | - | - | | `foundry_plan_migration` | `space_rid: string` | `connection_id: string`, `project_id: string \| null` (`null`), `extend_scope: boolean` (`false`), `new_attempt: boolean` (`false`) | Walk a Foundry space or folder and return pipelines, source datasets, derived datasets, dependency order, and the tenant's ontology object types as landing work. Its `source_datasets` are the dataset landing work order: land only those datasets from Foundry. Each pipeline includes `translatable` and `not_translatable_reason`; job specs remain in order but must not be passed to `foundry_translate_pipeline`. For a signed-in person, opens or resumes their migration project. Use `project_id` to choose among `project_candidates`; set `extend_scope` only when the person asks to add to a fixed scope, or `new_attempt` only when they ask to start over. These three options are honored only in the person's own chat turn. See [Planning project behavior](#planning-project-behavior) for gaps, fixed-scope differences, and database cost. | | `foundry_translate_pipeline` | `pipeline_rid: string`, `target_catalog: string`, `target_schema: string` | `connection_id: string`, `source_schema: string \| null` (`""`), `standalone: boolean` (`false`), `project_id: string \| null` (`null`) | Compile one Pipeline Builder pipeline to a PySpark script, return a `script_sha256` receipt, and report its input/output table mapping, unsupported nodes, and each output's `non_reproducible_columns` (columns drawn afresh on every run, with the derivation recorded). In a migration chat, files the translation under the person's covering project and records its destination when the script is saved. Set `source_schema` to the schema holding Foundry input copies when it differs from `target_schema`; omit it to read and write in `target_schema`. Use `standalone=true` for work outside a migration or `project_id` when the person selected a project. A job-spec key is refused before a Foundry call with `fault: "not_a_pipeline_builder_pipeline"`. See [filing copies and translations](#filing-copies-and-translations) for project selection and exceptions. | | `foundry_extract_dataset` | `dataset_rid: string`, `databricks_connection_id: string`, `target_catalog: string`, `target_schema: string` | `row_limit: integer \| null`, `async_mode: boolean` (`false`), `copy_derived: boolean` (`false`), `expected_table: string \| null` (`null`), `require_absent: boolean` (`false`), `standalone: boolean` (`false`), `project_id: string \| null` (`null`) | Copy a Foundry dataset to Delta. A whole copy in a migration chat is filed under the person's covering project before it starts; a `row_limit` sample is not tracked. Use `copy_derived=true` to land a derived dataset or protect an existing rebuilt table; copied rows are recorded as copied, not recomputed. Use `expected_table` to pin the resolved destination and `require_absent` to refuse an existing target instead of replacing it. Use `standalone=true` for a copy outside a migration or `project_id` when the person selected a project. See [filing copies and translations](#filing-copies-and-translations) for project selection and exceptions, and [receipts for finished dataset copies](#receipts-for-finished-dataset-copies) for receipt conditions and result fields. | | `foundry_get_extraction` | `extraction_id: string` | — | Read the status and result of an asynchronous extraction. A successful poll includes a top-level `next_step` from the landing result (for a SQL landing, `foundry_verify_migration`) and the landing result, including its `copy_receipt_id` when one was written; a failed poll includes `error` and no `next_step`. See [polling asynchronous extractions after restart](#polling-asynchronous-extractions-after-restart) for retained status and restart outcomes. | | `foundry_list_landings` | `databricks_connection_id: string`, `target_catalog: string`, `target_schema: string` | — | List tables previously landed by the migration tools. | | `foundry_adopt_copy` | `dataset_rid: string`, `table: string` | `databricks_connection_id: string \| null` (`null`) | Adopt an existing Databricks table as the landed copy of a Foundry dataset after checking its last writer. It stamps the table without moving rows; see [Adopting an existing landed copy](#adopting-an-existing-landed-copy). Costs up to three Databricks statements. | | `foundry_materialize_ontology` | `ontology: string`, `databricks_connection_id: string`, `target_catalog: string`, `target_schema: string` | `object_types: string[] \| null`, `space_rid: string` | Land selected Foundry ontology object types as Delta tables and disclose link types and actions that are not materialized. Large types use the pipeline-output dataset identified from `space_rid`; without it, they are refused instead of being paged through the object API. | | `foundry_publish_notebook` | `script_path: string`, `workspace_dir: string` | `databricks_connection_id: string`, `name: string \| null`, `standalone: boolean` (`false`), `project_id: string \| null` (`null`) | In a migration chat, files the publish under the person's covering project, whose `rebuilt_and_ran` goal then records each run of the notebook by itself; see [filing copies and translations](#filing-copies-and-translations). Import a translated script into the Databricks workspace as a notebook, split at the compiled script's function boundaries, with a provenance header and an output-contract table. Returns `script_sha256`, `notebook_sha256`, `execution_ready`, structural warnings, and the next step for running the workspace notebook. Publishes only onto a path the workspace confirms is free, or over the notebook recording the same Foundry pipeline, so re-translating a pipeline updates its own notebook. | | `foundry_run_pipeline_script` | `script_path: string`, `databricks_connection_id: string` | — | Diagnose a PySpark script through Spark Connect and report `declared_output_tables`, successful-run `output_tables` with row counts, logs, and the next verification step. The run confirms table DDL in the script named by `script_path`, including the translator's drop of its time-series carrier when the run's first write fails. Changed outputs are marked as rebuilds rather than copies; see the diagnostic-run guidance above. This does not create a Databricks Jobs run receipt; use `databricks_action(action="run_notebook")` for the intentional demo. | | `foundry_get_pipeline_run` | `pipeline_run_id: string` | — | Read the status and result of a pipeline run that `foundry_run_pipeline_script` returned as `running`. A finished poll includes `result`; a succeeded run includes a top-level `next_step`, and a failed run includes `error`. | | `foundry_verify_migration` | `dataset_rid: string`, `databricks_connection_id: string`, `target_table: string`, `foundry_space: string` | `mode: string` (`prove`), `spec_hash: string` (`""`), `key_columns: string[] \| null`, `float_rel_tolerance: number` (`1e-6`), `foundry_copy_table: string` (`""`), `translation_script_path: string` (`""`), `title: string` (`""`), `rescan: boolean` (`false`), `wait_seconds: integer` (`0`) | Register and characterize the Foundry source and Databricks target, then submit a Prover proof. Pass a sealed `spec_hash` to grade the same proof against written migration intent; without it, the result is comparison evidence only. In the default `prove` mode, a source dataset with no table schema is also compared file by file with a landed `document`/`file_path` table when measurable; see [Foundry datasets without a schema](/docs/Genesis-Prover/Overview#foundry-datasets-without-a-schema). `title` optionally overrides the generated comparison headline; by default it is `Foundry : dataset → Databricks . ()`. For a warehouse-side keyed diff of a landed Foundry copy, use `mode: "quick_keyed_check"` with `foundry_copy_table` and `key_columns`. Pass `translation_script_path` for a table built by a translated pipeline to identify columns drawn afresh on every run. `wait_seconds` requests inline waiting within one shared, 240-second call budget; see [Verification wait budget](#verification-wait-budget). Foundry scan-budget behavior and retained partial measurements are described in [Prover characterization](/docs/Genesis-Prover/Overview#1-estates-and-characterization). | ### Verifying a translated pipeline output When `target_table` was built by a translated pipeline, pass the `script_path` returned by `foundry_translate_pipeline` as `translation_script_path`. The result includes `non_reproducible_by_construction`, mapping each recorded column to its derivation: these values are drawn afresh on every run, on Foundry and Databricks, so they cannot match by construction. In a Prover proof, value, content, and count readings for these columns are indeterminate with cause `non_reproducible_by_construction`—neither a difference nor a match—unless a person licenses them; structural findings are unaffected. The full-table digest excludes these columns and compares the rest. The `quick_keyed_check` mode does not use this Prover classification and still counts mismatches in them. See the [Prover comparison guide](/docs/Genesis-Prover/Overview#2-comparison-the-system-diff) for how to interpret the cause. The translated script is read once from the uploads repository for this check. If the script predates the `NON_REPRODUCIBLE_COLUMNS` record or was edited to remove it, `non_reproducible_by_construction` is `null` and `non_reproducible_note` says no columns were checked. An empty mapping means the translation recorded that no output columns are non-reproducible. Omit `translation_script_path` for a landed source dataset. The Prover comparison performed by `foundry_verify_migration` also explains when a float tolerance could not be checked and can license a BigQuery TIME-to-text landing only when live values prove the canonical representation; see the [Prover comparison guide](/docs/Genesis-Prover/Overview#2-comparison-the-system-diff). ### Comparing a landed Foundry copy When `foundry_extract_dataset(copy_derived=true)` lands a dataset, it checks whether the destination table is absent or stamped as an earlier copy of that same Foundry dataset. In either case, the copy lands under the dataset's table name. Otherwise the existing table is left untouched and the copy lands as `__foundry_copy`; the result's `table` names the copy and `left_untouched` names the spared table. If that alternate name is also occupied by something other than this dataset's copy, or the warehouse cannot confirm the tables' status, extraction is refused. This applies regardless of the dataset's reported origin and lets a translated pipeline's rebuilt table stay intact for comparison. The destination checks for `copy_derived=true` cost one to four Databricks statements before rows are counted or extracted. Do not repeat a `copy_derived=true` landing after a pipeline has written that table: a rebuilt table can retain the earlier copy's stamp and then be replaced. A landing without `copy_derived` does not check the table's stamp and replaces the table with the dataset's name, so do not use it where a pipeline has rebuilt that table. Pass the landed table to `foundry_verify_migration` to compare it with the rebuild: ```python theme={null} foundry_verify_migration( dataset_rid="ri.foundry.main.dataset.example", databricks_connection_id="warehouse-connection", target_table="catalog.schema.flights", foundry_space="ri.foundry.main.space.example", mode="quick_keyed_check", key_columns=["flight_id"], foundry_copy_table="catalog.schema.flights__foundry_copy", ) ``` `foundry_copy_table` is optional (default `""`), applies only to `quick_keyed_check`, and requires `key_columns`. The copy and target must be three-part Databricks table names. The tool describes both tables, then compares them in one Databricks statement; the diff scans and joins the two tables in the warehouse and returns totals plus at most 50 sample keys. It does not read Foundry or create a Prover run, so it compares the copy as it was landed, not the current tenant data. Float, double, and real columns use the relative tolerance `float_rel_tolerance` (default `1e-6`). Duplicate keys make the verdict `INDETERMINATE`; columns present on only one table make an otherwise matching result `FAIL`. Without `foundry_copy_table`, `quick_keyed_check` pages Foundry and Databricks by key. That path has a 200-second budget; if it runs out, the tool returns an `INDETERMINATE` verdict with keys localized so far, `truncation_reason: "time_budget"`, and a `next_step` to land Foundry's rows and compare the copy. Use `mode: "prove"` instead when you need an auditable Prover comparison. If background hand-off is refused for an asynchronous `foundry_extract_dataset` or a large `foundry_materialize_ontology`, the tool returns immediately with `success: false`, `status: "failed"`, an `extraction_id`, and an `error`. The extraction was not started; the returned ID identifies the failed attempt, not queued work. ### Verification wait budget `foundry_verify_migration` always returns its answer inside the bot turn. Its inline wait has one 240-second budget for the entire call, not a separate allowance for each run. The space walk, estate reachability checks, and target description also use time from this budget. The `wait_seconds` parameter (default `0`) requests how long to wait inline; values above the time remaining in the budget are clamped. Negative values are rejected. Leave it at `0` to submit work without waiting inline. The call can submit two characterizations and a proof. If the inline wait ends before those runs finish, they continue running and the answer includes their run ids so you can poll them with `get_run`; the wait ending does not stop the work. Long-running tools return handles. For verification, `stage` identifies whether the run is still `characterizing`, `proving`, or `grading`; poll the returned run ids with the Prover's `get_run` tool or call `foundry_verify_migration` again with the same arguments. When `spec_hash` is supplied, Genesis reads and verifies the sealed specification before estate I/O and attaches it to the proof. `grading` means the comparison exists but its specification grade is still running; wait for the proof to reach a terminal status before treating it as a verdict. `spec_hash` applies only to `mode: "prove"`; supplying it to the keyed-contract quick check is refused. A characterization can enter a proof only after its own run succeeds and exposes a sealed snapshot. A terminal non-success is reported and does not fall back to an older snapshot or start a replacement characterization. Foundry reads that return exactly 1,000 rows may be refused if the preview role could have capped them; see [Foundry SQL result caps](/docs/Genesis-Prover/Overview#foundry-sql-result-caps) for the probe, access remedy, and limits of this check. `foundry_verify_migration` returns `stage: "withheld"` when its comparison compared no readings and at least one reading was withheld. Treat this as **not proven**, not as a clean result: the counts are empty because nothing was measured. Read `comparison.value_coverage.statement` for what was not compared. When `comparison.value_coverage.recharacterize` identifies stale side(s), call `foundry_verify_migration` once with the same arguments and `rescan: true` to start fresh measurements; if it returns `stage: "characterizing"`, poll with `get_run` or call again without `rescan: true`, because repeating the rescan restarts the scans. When `recharacterize` is `null`, no re-scan is prescribed because the comparison could not identify which side to measure again. Every `foundry_verify_migration` answer starts with `audience: "agent"` and `audience_note`, marking its proof result or refusal as guidance for the calling agent, not text to forward. See [Prover guidance](/docs/Genesis-Prover/Agents-And-The-API#reporting-a-completed-run) for how to report the result to a person. For the comparison card shown when verification runs in chat, including its live Prover theater and migration pass score, see [Review agent-generated comparisons in chat](/docs/User-Guide/Chat-And-Threads#review-agent-generated-comparisons-in-chat). ### Adopting an existing landed copy Dataset copies landed by `foundry_extract_dataset` without a `row_limit` are stamped with the Foundry dataset RID they copy. A row-limited derived-dataset copy is also stamped, but marked as a sample and not used as a dataset source in a spec grade; a row-limited source-dataset landing is unstamped unless it uses `copy_derived=true`. These stamps let an eligible keyed Foundry-source obligation read the already-landed rows from Databricks; the Prover guide explains when that route is selected and how its evidence is reported. Use `foundry_adopt_copy` only when the table owner confirms that an existing table is the copy of the named dataset. It does not land or rewrite rows. It adopts an unstamped table only when Delta history attributes its last write to `manual_query`; a write attributed to a job, notebook, or pipeline, an unreadable history, a stamp for another dataset, or the rebuild marker `genesis.foundry_rebuilt_by` is refused. If the table already carries the requested stamp, the tool returns `adopted: false` without writing. ```python theme={null} foundry_adopt_copy( dataset_rid="ri.foundry.main.dataset.example", table="catalog.schema.airports", databricks_connection_id="warehouse-connection", ) ``` The result includes `table`, `dataset_rid`, and `adopted`, or `success: false` and an `error`. Adoption can issue up to three statements on the Databricks warehouse: read table properties, inspect Delta write history, and write the copy stamp. For how the Prover verifies a Foundry copy that is already in its target, see [the data-work proving skill](/docs/Genesis-Prover/Overview#the-data-work-proving-skill). ### Materializing large ontology types `foundry_materialize_ontology` measures each selected object type before reading its rows. Types with at most `50,000` objects use the ontology object API. A type above that limit is not paged through the API; pass `space_rid` for the Foundry space or folder whose pipelines build the ontology so Genesis can resolve the pipeline-output dataset and extract it through the same staged, chunked path as `foundry_extract_dataset`. The backing-dataset match requires the object's exact primary-key column and at least `90%` of its properties as dataset columns. If no dataset matches, more than one matches, the count cannot be measured, or `space_rid` is omitted, the object type is listed in `failed_object_types` rather than guessed or paged. A successful backing-dataset entry reports `materialized_from: "backing_dataset"`, the `backing_dataset_rid`, and provenance in the form `materialized_from_backing_dataset `. These are the dataset's rows; an ontology that filters, joins, or derives properties may not match them row for row. Large successful materializations can be asynchronous; poll their `extraction_id` with `foundry_get_extraction`. ## Costs and safeguards `foundry_extract_dataset`, `foundry_materialize_ontology` when it uses a backing dataset, `foundry_run_pipeline_script`, and `foundry_verify_migration` run customer compute. When `foundry_verify_migration` receives `spec_hash`, grading also reads both estates. Configure Foundry request timeouts in [Advanced Settings](/docs/Setup/Advanced-Settings#connector-resolved-connection-cache). Foundry resource identifiers are stable migration keys. Keep the source dataset RID, the target table, `proof_run_id`, and `diff_hash` together in the migration report.