> ## Documentation Index > Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt > Use this file to discover all available pages before exploring further. # Outpost Troubleshooting > What each Genesis Outpost status and diagnosis means, and how to fix it. When an Outpost or a route through it does not work, Genesis says why in plain language, with a fix: on the Outpost's card and in the banner at the top of **Config -> Outposts**, under the Outpost picker in the connection form, on the red **Outpost down** badge, in **Test Connection** results, and in agent answers. A query through a down Outpost fails with the same diagnosis instead of a raw driver error, for example: ```text theme={null} Outpost acme-prod unreachable since 14:02 UTC (it was connected, then its tunnel dropped). Fix: Check that the Outpost is still running: ... ``` Genesis sees the tunnel from its side only. When the cause is on the Outpost's side, such as a refused sign-in, **the Outpost's own log names it**. Read it first: | Where it runs | Log | | - | - | | Laptop | The terminal running `outpost run` | | Linux VM | `sudo journalctl -u genesis-outpost -f` | | Docker | `docker logs -f genesis-outpost-` | | Kubernetes | `kubectl -n genesis-outpost logs deploy/genesis-outpost- -f` | On Genesis's side, the tunnel runs as a separate Genesis worker process, and the Genesis log carries its events as `outpost tunnel worker:` lines, each followed by one JSON object with an `event` field. The events below are logged at WARNING, so they show at the default log level. Others, such as `stream_refused` and `signed_in`, are logged at INFO and show only when Genesis logs at INFO: | `event` | Meaning | | - | - | | `started` | The worker started; `admission` is the admission mode (`uid`, `userns` or `off`). | | `sign_in_refused` | An Outpost's sign-in was refused; `code` and `slot` say why and which. | | `route_refused` | A route an Outpost declared was not held; `reason` says why. | | `admission_refused` | A connection to a tunnel listener was refused by admission. | | `admission_failed` | Admission cannot work here, so the worker did not start. | If no worker runs, Genesis logs why once, as `outpost tunnel worker: not running: ...`. ## Diagnoses Genesis has not heard from the Outpost since Genesis last started. Usually one of: * The Outpost is not running. Start it. * Its sign-in is refused: the key does not match the one registered on the Snowflake user, the user is disabled, or the Outpost's IP address is outside the user's network policy. See [Sign-in refused](#sign-in-refused-in-the-outpost-log). * `GENESIS_ENDPOINT` is wrong, or the host cannot make outbound HTTPS connections on port 443 (check firewalls and `HTTPS_PROXY`). Genesis cannot tell these apart from its side; the Outpost's log can. The Outpost was connected and then went away. Check that it is still running: its host may be stopped or asleep. A laptop or suspended VM reconnects on its own within seconds of waking. If it is running, read its log: a sign-in refusal there means its credential was revoked or rotated. When Genesis's tunnel refuses the Outpost's tunnel key, Genesis says so instead (key\_refused, below). Genesis's tunnel refused the tunnel key the Outpost signs in with: it is wrong, or it was replaced when a new key was issued. Issue a new key (**Install**, then **New tunnel key**) and restart the Outpost with it. The diagnosis clears as soon as the Outpost signs in. Genesis sees only which Outpost slot tried to sign in: if the Outpost already runs with its newest key, an older install of it is still running with a replaced key. Stop that one. The sign-in happens inside the tunnel on every deployment model, Databricks Apps included, so Genesis sees this refusal wherever it runs. The Outpost's `OUTPOST_SLOT` does not match the slot Genesis assigned it (shown on its card), so no route through it can work. Set `OUTPOST_SLOT` to the value the fix names and restart the Outpost. Something answered on the Outpost's control address, but not with a proof made with the tunnel key Genesis issued this Outpost, so Genesis did not use the answer. Usually the Outpost runs with an older key (a new one was issued since) or a release before 0.7.0. Restart it, on Outpost 0.7.0 or later, with its current key; if you no longer have the key, issue a new one (**Install**, then **New tunnel key**). The Outpost is connected, but a connection's `host:port` is not in its `ALLOWED_DESTINATIONS`. Add the named destinations on the Outpost and restart it. This is your own allowlist working as intended; only add systems you want Genesis to reach. The destination is a cloud metadata service or a link-local address, or resolves to one. Outposts never reach these, and no setting changes that. Use the system's own host name or address. The Outpost is allowed to reach the destination but could not, from inside your network: * **Does not resolve inside your network:** check the host name and the DNS server the Outpost's machine uses. * **Did not accept a connection:** check that the service is running on that port and that firewalls between the Outpost and the host allow the connection. The Outpost is connected, but after Genesis re-sent its routes it still does not hold the route to this destination. Read the Outpost's log for the routes it applied; a refusal there names the reason. Check that `OUTPOST_CAPABILITIES` includes `tunnel` and that `ALLOWED_DESTINATIONS` lists the destination, then restart the Outpost. Genesis cannot route this `host:port` because Genesis itself already uses that port in your deployment. In the Snowflake Native App this affects port **8082**, and its debug shell port (**21234**) is also reserved; see [Snowflake Native App endpoints](/docs/deployments/Snowflake-Native-App/Installation-And-Configuration). This is a known limitation being removed. Until then, reach the system on another port if it offers one (for example HTTPS on 443 or 8443), or contact Genesis support. Connections reach this host name through more than one Outpost. Genesis routes the host through none rather than sending some connections, and their credentials, into the wrong network. Use one Outpost for the host, or give one system a distinct host name or its IP address. A conflicting new connection or pattern change is refused; this diagnosis can appear when a conflict already exists in saved settings. Another connection routes this host through an Outpost, so all connections reach it through that Outpost even if this one is set to **Directly**. The test result names the Outpost and explains the route without changing the test outcome. To change where this connection goes, see [Connect a System Through an Outpost](/docs/Setup/Outposts-Connect-Systems). The Outpost holds the route, but Genesis cannot make the host name resolve to it, so a driver reports "could not translate host name". Genesis maps a routed name by writing a hosts file, and in this deployment `/etc/hosts` is read-only and no writable hosts file is preloaded. A Databricks App resolves routed names from version 1.19.97 (its launcher preloads nss\_wrapper); upgrade an older App, and if it still fails, read the App log at start for why the Outpost host-name resolver was not installed. On a self-hosted Genesis, let the Genesis process write `/etc/hosts`. The Outpost is connected, but Genesis's tunnel refused to hold its route; the diagnosis includes the reason: * `port_in_use` or `listen_failed`: Genesis cannot listen on that port, usually because Genesis itself already uses it. Treat it like `port_in_use` above. * `too_many_routes`: the Outpost declared more than 254 routes. Spread the systems over more Outposts. * `outside_slot`, `duplicate_address` or `malformed`: the Outpost declared a route Genesis does not accept from it (a tunnel address outside its own `127.77..x` range, two routes on one address, or a route Genesis cannot read). Check `OUTPOST_SLOT` against its card and restart the Outpost; if it persists, send Genesis support the Outpost's log. The Outpost declares a refused route again on its own, backing off up to 10 minutes, and its log shows `Genesis refused route; backing off` with the reason. Genesis's tunnel refused a connection to this route because it could not attribute the connection to Genesis's own processes. A connection test reports this diagnosis only when the test's connection closed before an answer and the worker recorded a refusal for this listener after the test began. An earlier refusal, such as an agent's probe, is not attributed to the test. If the Outpost went down after a refusal and the test then closed, Genesis reports the route outage instead. If agent-run code made the request, use a saved connection to reach the system instead; see [Who can use a tunnel listener](/docs/Setup/Outposts-Security#who-can-use-a-tunnel-listener). If Genesis itself made the request, send Genesis support the tunnel worker's `admission: refused` log line and its reason. Administrators can temporarily disable the check with [`outpost.enforce_tunnel_admission`](/docs/Setup/Advanced-Settings#outpost-routing). ## An agent tool or agent code cannot reach a routed host A routed host is meant to be reached through a connection routed through its Outpost. Agent tools and code that agents run hold no Outpost grant, so Genesis refuses them routed hosts where they make their requests (see [Routed hosts inside Genesis](/docs/Setup/Outposts-Security#routed-hosts-inside-genesis)). An agent that fetches a routed host's URL itself, opens it in the browser, or connects to it from Python it runs is refused, with a message such as: ```text theme={null} Invalid url: pg.corp.lan is reached through Outpost acme-prod, and only a connection routed through that Outpost may reach it. Use a connection to this system (its own tool or query) instead of fetching the address directly. ``` This is expected. Create or choose a connection to the system routed through the Outpost (**Where is this system?** -> **Through a Genesis Outpost**), and have the agent use that connection. The same applies to a tunnel address (`127.77.x.y`). In a Databricks App, the MCP servers, coding-agent CLIs and dashboards Genesis starts for agents do not resolve routed host names at all; the Twin lineage server does. ## Sign-in refused (in the Outpost log) An Outpost signs in twice: first at the platform's front door (a Snowflake service user, a Databricks service principal, or a self-hosted ingress), then inside the tunnel with its own tunnel key. Most refusals appear only in the Outpost's log, each as a `tunnel session ended; reconnecting` line whose `error` names the cause. Genesis shows the Outpost as **Not connected**, or as **key\_refused** when its tunnel refused the key. | Log line (in `error`) | Meaning | Fix | | - | - | - | | `snowflake token exchange failed (...)` with Snowflake error `390422` ("Incoming request with IP ... is not allowed to access Snowflake") | The Outpost's egress IP is outside the service user's network policy. | Add the Outpost's egress IP to the network policy, or run the Outpost from the pinned address. | | `snowflake token exchange failed (...)` naming a JWT or public-key error | The private key does not match the public key registered on the user, or the account or user name is wrong. | Re-register the public key with `ALTER USER ... SET RSA_PUBLIC_KEY`, and check `SNOWFLAKE_ACCOUNT` and `SNOWFLAKE_USER`. | | `the tunnel endpoint refused the connection (HTTP 401/403)` | The front door refused the credential: disabled user, missing `APP_OUTPOST_TUNNEL` grant, or missing CAN USE on the Databricks App. On a Databricks App the line goes on to name the service principal and the fix. | Check the identity steps in [Install an Outpost](/docs/Setup/Outposts-Install#how-the-outpost-signs-in). | | `websocket: bad handshake` on a self-hosted Genesis or a Databricks App (Genesis answered `/outpost-tunnel` with HTTP 503) | Genesis's tunnel worker is not running: `outpost.enabled` is off, no Outpost has a tunnel key yet, the Databricks App is older than 1.19.97, or the worker is restarting. Genesis logs the reason as `outpost tunnel worker: not running: ...`. | Turn Outposts on, issue the Outpost's tunnel key (**Install**, then **New tunnel key**), upgrade an older Databricks App, and check the Genesis log. | | `the tunnel endpoint does not speak genesis-outpost.v1: the upgrade was refused with HTTP 400; ...` | The endpoint does not run the Genesis-native tunnel: a Genesis release older than the native tunnel, or a proxy in front of Genesis that drops the WebSocket subprotocol. | Upgrade Genesis, and make sure a proxy in front of `/outpost-tunnel` passes WebSocket upgrades unchanged. | | `the tunnel endpoint does not speak genesis-outpost.v1: sign-in refused: ...` (for example `Genesis accepts protocol versions 1 to 1`) | Genesis and the Outpost share no protocol version. | Upgrade whichever is older. | | `Genesis refused this Outpost's tunnel key: ...`, then `Genesis refused the tunnel key: sign-in refused: key_refused` | The key in `OUTPOST_TUNNEL_KEY` (self-hosted: in `OUTPOST_TUNNEL_AUTH`) is not the one Genesis last issued this Outpost, or `OUTPOST_SLOT` is not its slot. | Issue a new tunnel key (**Install**, then **New tunnel key**), set it, check `OUTPOST_SLOT`, and restart the Outpost. | | `Genesis refused the tunnel key: sign-in refused: unknown_slot` | No Outpost with a tunnel key holds this `OUTPOST_SLOT` in Genesis. | Set the slot shown on the Outpost's card, and issue its key if it has none. | | `the tunnel endpoint did not prove it holds this Outpost's key: ...` (logged at ERROR) | The endpoint accepted the sign-in but could not prove it holds this Outpost's key, so the Outpost carries nothing for it. | Check that `GENESIS_ENDPOINT` is your own Genesis. | | `sign-in refused: busy` or `sign-in refused: auth_timeout` | Genesis was already signing in many sessions at once, or the sign-in did not finish within 10 seconds (a slow or interrupted network path). | None needed if it clears on the next attempt; otherwise check the network path to Genesis. | A front-door refusal, a refused key, and an endpoint that does not speak the protocol or cannot prove the key are retried every 30 seconds with a fresh credential, so the Outpost reconnects on its own once you fix the cause. An Outpost before 0.7.0 cannot connect to a Genesis that runs the native tunnel: Genesis answers its upgrade with HTTP 400 and the Outpost's log shows a failed WebSocket handshake. Upgrade the Outpost to 0.7.0 or later. ### Genesis ended the session These lines mean the Outpost was signed in and Genesis ended the session: | Log line | Meaning | Fix | | - | - | - | | `Genesis ended this tunnel session: another Outpost with the same OUTPOST_SLOT signed in (superseded)` | Two installs run with the same slot; the newer one replaced the older. | Stop the other install, or give each Outpost its own slot. | | `Genesis ended this tunnel session: this Outpost's tunnel key was rotated or revoked` | A new key was issued, or the key was revoked. | Restart the Outpost with its current key. | | `Genesis is ending this tunnel session` with reason `shutting_down` | Genesis is restarting. | None: the Outpost reconnects on its own. | | `tunnel session ended; reconnecting` naming a keepalive timeout, `keepalive_timeout` or a closed connection | Two keepalive pings, 20 seconds apart, went unanswered, or the connection dropped. | None if it reconnects; otherwise check proxies and load balancers between the Outpost and Genesis for WebSocket idle timeouts of 20 seconds or less. | Genesis keeps a route's listener for 30 seconds after a session ends, so a quick reconnect does not drop the Outpost's routes. ## A connection through the Outpost is closed at once Genesis carries each connection to a routed host as its own stream inside the Outpost's tunnel, and closes the connection at once (the driver reports the connection closed or reset) when the stream cannot open. The reason is in the Outpost's log as a `stream refused` line, or in the Genesis log as an `outpost tunnel worker` line with `"event":"stream_refused"` (logged at INFO, so it shows only when Genesis logs at INFO): | Reason | Meaning | Fix | | - | - | - | | `outpost_disconnected` (Genesis) | The Outpost's tunnel dropped and it has not reconnected yet. | Wait for the reconnect; if the Outpost stays down, see the diagnoses above. | | `rate_limited` (Genesis) | More new connections on one route than 50 a second, after a burst of 200, and the connection waited 2 seconds without a turn. | Use a connection pool, or spread the load over time. | | `too_many_streams` | The route already has 256 open connections. | Lower the connection pool size, or close idle connections. | | `not_allowed` (Outpost) | The destination is outside the Outpost's `ALLOWED_DESTINATIONS`. | See `destination_refused` above. | | `blocked` (Outpost) | The destination is a cloud metadata or link-local address. | See `destination_blocked` above. | | `dial_failed` (Outpost) | The destination did not accept the connection from inside your network. | See `destination_unreachable` above. | | `not_declared` | The route is not one the Outpost holds right now (it was just removed or not yet re-declared). | Retry; if it persists, see `route_missing` above. | ## A secret from the Outpost cannot be read A connection whose password is an `outpost_/` reference fails with the Outpost's reason, which names the store it reads (`aws-secrets-manager`, `azure-key-vault` or `gcp-secret-manager`), and a fix for that store. The Outpost's log has an audit record for every read. If the Outpost read succeeds but **Test Connection** rejects the credential after rotation, Genesis may have been using its cached reference value. See [Connections](/docs/Setup/Connections) for how to retry and where to correct the value. | The reason says | Fix | | - | - | | `secret not allowed by Outpost policy` | Add `secrets` to `OUTPOST_CAPABILITIES`, and make `ALLOWED_SECRET_PREFIX` cover the name (in Genesis spelling, with `/`). | | `not allowed; use letters, digits ...` or `contains "--"` | Azure Key Vault and Google Secret Manager cannot hold that name. Rename the secret using letters, digits and `-` (and `_` in Google), with `/` between parts. | | `no secret named genesis--prod--pg (Genesis name genesis/prod/pg)` (Azure) or `no secret projects//secrets/genesis--prod--pg/versions/latest` (Google) | Create the secret under that name, each `/` written as `--`; in Google, check `GCP_PROJECT` and that the secret has an enabled version. | | `refused the Outpost's identity ... Key Vault Secrets User` | Assign the **Key Vault Secrets User** role on the vault to the Outpost's managed identity, workload identity or service principal. Role assignments can take a few minutes to apply. | | `refused the Outpost's identity ... Secret Manager Secret Accessor` | Grant `roles/secretmanager.secretAccessor` to the Outpost's service account. | | `could not sign in to Azure` | The VM has no managed identity (or several; set `AZURE_CLIENT_ID` to the one to use), the AKS pod lacks workload identity, or on a laptop run `az login`. With `HTTPS_PROXY`, add `169.254.169.254` to `NO_PROXY`. | | `no Google Application Default Credentials` or `could not get a Google token` | Attach a service account to the VM with the `cloud-platform` scope, set up GKE workload identity, or on a laptop run `gcloud auth application-default login`. | | `set GCP_PROJECT` | The credentials do not name a project (common for a laptop login); set `GCP_PROJECT`. | | `AccessDeniedException` or `ResourceNotFoundException` (AWS) | Allow `secretsmanager:GetSecretValue` on the secret for the Outpost's role, and check `AWS_REGION`. | ## Reinstalling or restarting an Outpost You can reinstall an Outpost, move it to another machine, or restart it at any time. Click **Install** on its card to get the commands again. Keep the same `OUTPOST_SLOT`. You do not need to recreate routes. The Outpost restores its last routes from its state directory when it restarts. If it starts with none (a new machine, or a Kubernetes pod with an empty state volume), Genesis re-sends them automatically: the next time a query or connection test through it fails to open, and in its hourly check. ## Ask Eve Administrators can ask Eve to diagnose an Outpost in plain language, for example: * "Why is Outpost acme-prod down?" * "Check route: can Outpost acme-prod reach db.corp.internal on port 5432?" * "Explain routing: which Outpost will my connection to jira.corp.internal use?" Eve relays the same diagnosis and fix as the Outposts page, and can check a host from inside your network through the Outpost.