Pulse Monitoring

​

Overview

Pulse is the operational monitoring suite inside Genesis. It brings live system health, background jobs, downloads, datasets, metrics, slots, and user info into a single workspace.

​

Main Pulse views

In OAuth- or OIDC-protected deployments, the Live Logs, health, Server, and CWA Monitor / Slots views use your current signed-in UI session for protected fetches and live streams. That includes health checks, recent-error queries, log streaming, and the guarded restart action. The Server view and its runtime telemetry (GET /api/health/detailed, GET /api/health/server-panel-telemetry) require access_config_panel when auth is enabled. The CWA Monitor / Slots view loads the full agent-slots payload from GET /api/admin/slots, which also requires access_config_panel. If a Pulse Server request returns 403 without an X-Request-Id response header, Pulse identifies it as a proxy or firewall blocking the request before it reaches Genesis, rather than a Genesis permission denial, and shows the API path. A 403 with the header is reported as a Genesis permission denial and includes the returned error detail when available. The Metrics Upload card tracks scheduled bridge-mode uploads. Direct OTLP metrics export is configured separately on Metrics and does not use the bridge upload queue. After two consecutive job-level upload failures (each job attempt already retries three times), bridge uploads pause for one hour. Each subsequent failed attempt doubles the pause, up to 24 hours; uploads resume automatically after the backoff, and a successful upload clears the failure streak. For bot-attributed LLM usage and prompt-cache miss telemetry in exported metrics, see Metrics.

​

Costs

Open Pulse → Costs (/pulse/costs) to review the serving instance’s spend and what caused it. The page is available on every install to a signed-in user with Pulse panel access. It shows “Recording since” the moment this install began recording spend; a window that starts earlier shows “No data before” that moment and is clamped to it. Until the cost data’s indexes are built (the first quiet window after an upgrade) the page says “Preparing cost data” and shows no figures. Pulse → Costs is the supported cost investigation page; the built-in Cost & Usage Dash app is optional for older workflows. Metrics Dashboard continues to show Messages, Users, and Tokens. A saved link to its former Cost tab opens Pulse → Costs. Choose a 24 hours, 7 days, 30 days (the default), or 6 months reporting window. The summary includes spend-over-time and token-use charts, alongside the statement that distinguishes known priced spend from unpriced usage, reserved Context Graph tokens, incomplete attribution, and rows outside the instance total. Known priced spend counts only calls with a catalog price. A call at an estimated price (default rates because the model has no catalog price, or a price whose source was not recorded) is reported on its own Estimated price line, with its dollars and request count; it is never added to the known priced total. Unpriced usage is a count, never $0. A freshness panel identifies whether chat or Context Graph producers are current, stale, preparing, failed, disabled, or unsupported. While cost data is being prepared the page checks every 60 seconds, then refreshes the summary and spend controls every 30 seconds while the page is visible. Activity and execution drilldowns remain paused until figures are ready. An up-to-date Context Graph rollup with no recorded usage says nothing has been recorded since the store's recording cutover, or that nothing has been recorded yet when no cutover is available; it does not report an unknown or stale state. When an activity contains only tokens reserved against the Context Graph cap and no requests or measured usage, its cost appears as **reserved only**, not `$0`. If the install has no configured name, Costs omits its generated fallback identifier from empty-state and spend headlines. The statement can open the Written while cost accounting was off, Arrived after their day was sealed, and Revised after their day was sealed lines to their ledger records. Showing those records requires access_config_panel. The first two lines describe records outside the total; the revised-after-seal line shows the latest revision while the total keeps the value counted when the day was sealed. The gap ledger is not split by instance, so its records can include other instances on a shared store. Records load a page at a time. See the Costs API reference for the response fields and access requirement. The Breakdowns section splits spend by source or activity group, model, agent, provider, pricing quality, attribution quality, operation, origin, and call type. Operation, origin, and call type use the summary’s grouped figures; their tables load separately from the other breakdowns. Select a breakdown row to filter Activity detail; select another chart bucket or breakdown row to change the detail, or clear the filters to return to the full window. Chart bars show hourly or daily buckets depending on the selected range. Click a bucket, or choose one from the bucket picker and select Show activities, to scope Activity detail to that bucket. These selections are kept in the page URL and remain in effect after a reload. Activity-based breakdowns can be partial while more activity rows or model entries remain to be loaded. Genesis AI model rows are labeled Genesis AI (or by their Genesis AI role) instead of exposing the underlying model identifier. Their pricing source is shown as Genesis AI, not as the upstream rate catalog used internally. Expand an activity in Activity detail to see its executions inline, then expand an execution to inspect retained Calls. Activity rows show known priced spend, quality qualifiers, execution count, last-seen time, and whether the activity is recurring. See the Costs API reference for the true, false, and null meanings of the recurring indicator. Where available, links open the scheduled job, Digital Twin source, or scheduler run history. Unattributed spend can appear as separate chat and Context Graph activity rows; expanding either row drills into only that producer’s executions and evidence. When a detail filter is active, the expanded activity lists the cost records behind its filtered amount instead of runs; an execution count may still include all runs of the activity rather than only runs within the filtered spend. Without filters, an activity with no run started in the selected window shows its cost records for that window, since there is no run to expand. If the activity list and its records resolve the selected time window at different moments, Costs warns that their windows differ and offers Re-read to fetch them together. Calls show compact call or hourly Context Graph evidence. Call details include cache read/write token counts, and Context Graph bucket details include reserved embedding tokens. A dash means a count was not recorded, not that it was zero. Calls can be complete, partial, or expired; an expired execution keeps its summary even after raw call evidence passes the retention boundary. When you select a Context Graph source activity, the execution list also shows that source’s sweep status. This on-demand Twin read shows the effective per-source limits (or that the source is unbounded), the latest recorded sweep outcome and stop reason, exhausted budget units, and whether a halted sweep has room under the current budget. When usage was recorded, the panel also shows the sweep’s total active time, objects, queries, and containers across its cycles, with configured limits alongside the corresponding usage. It labels totals for an in-progress or halted sweep as used so far; those sweeps may continue. Before any outcome is recorded, the panel says the source has not been scanned; if Twin cannot provide the status, it marks the source unavailable. The source activity’s known priced spend appears with the sweep details. See the Costs API reference for the response fields. The Spend controls panel shows the Context Graph background token cap and per-agent budgets. Adjust cap edits the same cap as the Digital Twin spend page; a blank cap means unlimited background scanning, while 0 blocks all background scanning. The page also links to the Digital Twin editor. If the cost-accounting rollup job is enabled, its default interval is jobs.cost_accounting_rollup.interval_seconds=300 seconds, so newly written facts can take up to the next rollup pass to appear in the totals. The initial rollup wait is separate from this interval: the page checks for readiness every 60 seconds until the first rollup is ready, then refreshes the figures every 30 seconds while the page is visible.

​

CLI Workspaces

Open Pulse → CLI Workspaces (/pulse/cli-workspaces) to monitor the running workspaces created by the direct CLI terminal. The view is available when cli_terminal.enabled is true (the default is false) and the signed-in user has view_sandboxes. The list refreshes every 10 seconds and shows each workspace’s owner, status, age, CPU, and memory. When the Kubernetes metrics server is unavailable, live resource usage is shown as unavailable while the configured limits remain visible. Users with manage_sandboxes can restart or cancel a workspace after confirming the action. Restart deletes and recreates the workspace pod; cancel deletes the pod. In both cases, the user’s volume and saved sessions are kept, and a new workspace is created on the next connection. The page also displays the Genesis CLI connection command for the server being viewed.

​

Mission Value

Open Pulse → Mission Value (/pulse/mission-value) to review the current deployment instance’s completed-mission volume and manual-versus-Genesis time. The page requires view_all_missions and Pulse panel access. Choose a 30-, 90-, or 365-day reporting range, optionally filter by normalized work type, and page through the mission drilldown. The report is read when the page loads or its filters change; it is not a scheduled background refresh. The page shows exact, estimated, and incomplete observations; candidate and filter counts; work-type shares; cycle, lead, human-wait, and signed-savings totals; and the percentage of observed missions covered by a manual baseline. Signed saved work is manual baseline minus Genesis cycle for exact, baseline-covered missions, so negative values remain visible. A mission’s quality details are available in the drilldown. Configure manual baselines and their environment-override behavior in Mission Value settings. Work type is derived from blueprint_name, then project_type as a fallback; renaming either value changes grouping and baseline matching. The current baseline is applied retroactively to matching exact missions in the selected range. For the HTTP contract, see Mission Value endpoints. The Live Logs view subscribes to the live stream and then loads recent history. Genesis queues any matching lines that arrive during that bootstrap window and emits them once after the historical batch returns, which avoids missing or duplicated lines as the stream connects. Live entries now carry contextual attribution when Genesis can resolve it. In practice that means you can filter or inspect logs by bot, thread, tool, origin, phase, request, user, or module path instead of relying only on the message text. Slack lifecycle, warning, and error events are also forwarded into the same log stream, so Slack reconnect activity stays visible in Pulse.

​

Genesis Enhancements summary

The Genesis Enhancements Pulse card now includes a compact dormant-review summary alongside the ticket lifecycle counts. It shows the dormant-review scheduler status, the current mode (dry-run, drafts, or scan-only), queued review count, generated draft count, failure count, and the latest reviewed-thread total from the most recent run. Use the link in that card to jump back to Config -> Support and Enhancements when you need to change dry_run, create_drafts, or the saved scheduler settings. The card refreshes every 15 seconds, so it is a quick operator view rather than the full configuration surface.

​

Agent Scorecard

Open Pulse → Agent Scorecard (/pulse/agent-scorecard) to review how agents are performing under the agent-review grading pipeline. The page requires Pulse panel access (access_pulse_panel) and is active only when agent review is enabled for the deployment. When it is disabled, scorecard requests return 404 and the page shows a disabled-state message with a link to Config → Advanced Settings → Agent Review. Other request failures show the normal load-failure message. Use the header controls to scope the view: When data exists for the selection, Genesis shows KPI cards, trend charts, a workflow breakdown table, and a backfill panel.

​

KPI cards

​

Trend charts

Genesis plots time-bucketed series for first-shot success rate, steering turns per task, known priced spend (USD), latency (p50 and p95), tool error rate, and user feedback (Met / Not met). When the price split includes estimated spend or requests, that spend is plotted as its own Estimated price series beside known priced spend. Unpriced or unknown-price calls do not contribute to spend. If no calls have a known price, the chart explains that there are no priced calls; if a bucket has no model-call records or its cost data is unavailable, the chart says cost is unavailable. First-shot charts show a suppression message instead of a line when every bucket in the selection is corrections only.

​

Workflow-type breakdown

The breakdown table lists each workflow type with its Grading policy (Full or Corrections only), task count, first-shot rate (only for Full rows), and steering turns per task.

​

Backfill classification

Use Backfill classification to grade historical threads that were not yet classified:

1. Select a date window.
2. Choose Estimate — Genesis counts candidate threads and returns an estimated cost. The Run backfill button appears only after a fresh estimate for the current window.
3. Choose Run backfill to start processing from that estimate.

Changing the date window discards any stale estimate so a run cannot start against the wrong window. The Runs table below lists prior classification and backfill jobs with mode, status, window, threads scanned, turns classified, estimated cost, actual cost, and created time. Genesis polls the table while a run is running. In the Runs table, the actual-cost cell pairs each unpriced-call count with a known-priced amount when available (otherwise marking the amount unknown); see Chat and Threads for the related thread summary.

​

Agent Learning

Open Pulse → Agent Learning (/pulse/agent-learning) to review recurring steering patterns surfaced by the agent-review pipeline, adopt or dismiss them, and measure whether adopted learnings improved agent behavior. The page is the Pulse companion to Agent Scorecard. It requires Pulse panel access (access_pulse_panel) and is active only when agent review is enabled for the deployment. When it is disabled, learning requests return 404 and the page shows a disabled-state message with a link to Config → Advanced Settings → Agent Review. Other request failures show the normal load-failure message. Use the header Agent selector to scope all three tabs to one bot or All agents.

​

Candidates queue

The Candidates tab is the default view. Use the status pills to filter Open (default), Adopted, Dismissed, or All. Genesis preserves the filter in the URL (?status=) so you can share or bookmark a view. Candidates are ranked by the server — distinct users first, then occurrence count, then most recently seen — and the UI renders them in that order. Each card shows the pattern summary, workflow type, table anchor (or Unanchored when no anchor could be resolved), occurrence count, distinct-user count, and the first/last seen dates. Candidate decision permissions and the disabled-control behavior are described in the RBAC practical guide. Expand Evidence on a card to open the chat threads that triggered the pattern. For Open candidates you can: Genesis asks for confirmation before accept or dismiss. An accept is tied to the candidate text rendered in the card: Genesis finds or creates that candidate’s Action Review item, and the authenticated person decides and applies it. A candidate whose text changed after the card was loaded is not published; the queue reports that it changed so you can read the updated candidate before trying again. After the Action Review attempt settles, Genesis only adopts an anchored learning whose matching text landed during that attempt; an older matching learning is not reused. While an accept is running or its outcome is unsettled, the row shows Held, disables both decisions, and may show the hold end time. If an accept may already have published the learning, the queue links to the holding Action Review item and tells you whether to wait for reconciliation or retry after the hold; do not submit a second accept from the stale row. If another operator already resolved the candidate, Genesis shows a conflict message and refreshes the list.

​

Improvement report

The Improvement report tab compares task metrics before and after each adoption. Use the Horizon pills to choose the server default window or 7 days, 14 days, or 30 days around the adoption date (?horizon= in the URL). Each adoption card shows the headline, source, agent, workflow type, anchor (when present), adoption date, and the selected horizon. When enough graded tasks exist in both windows, Genesis plots Steering turns / task (lower is better) and First-shot success (higher is better) as before → after dumbbell charts. When a metric is unavailable — for example because the workflow uses a corrections only grading policy or the windows are too thin — Genesis shows an explicit unavailable note instead of a number. When overall task volume is below the minimum per window, the card shows Not enough data with the before/after task counts. Adoption sources appear as Auto-supersession, Steering pattern, or Manual.

​

Adoption history

The Adoption history tab lists every recorded adoption, newest first. The Adoption ledger table includes adopted time, source, agent, workflow type, anchor, learning link, and who created the adoption. When a weakness has derived coaching status, its coaching status text links to the matching adoption in this ledger. The link opens Agent Learning with the agent selected and the history filtered to that adoption; use Clear filter to return to the full ledger. If the server does not provide a matching adoption id, the coaching status remains text without a link. When a wiki learning was written and the adoption is anchored, View learning opens the wiki page at that anchor. Summary-only adoptions (no wiki statement) show a Summary-only badge. Adoptions with a statement but no resolvable anchor page show Learning (no anchor page).

​

CWA Monitor / Slots summary

The CWA Monitor / Slots page loads the full slots payload from GET /api/admin/slots, including per-thread user identity fields shown in the acquired and waiting tables. In auth-enabled deployments you need access_config_panel; without it the page cannot load slot details. GET /api/health/slots returns the same operational counters and timing fields but omits user_id, user_display_name, thread_purpose, and thread_name. In auth-enabled deployments that route requires a signed-in session (anonymous callers receive 401). Tokenless load-balancer probes should use GET /api/health or GET /api/health/ready instead. The CWA Monitor / Slots summary separates actively running work from temporary cooldown holds so capacity is easier to interpret at a glance. When a slot limit is configured, the Occupied summary highlights overall usage: green below 70%, yellow at 70% or higher, and red when all slots are occupied. If a refresh of GET /api/admin/slots fails — for example during a network interruption or while your session cannot reach the admin endpoint — the page shows a red banner at the top. When Genesis can surface an underlying client error detail (such as Network Error), the banner reads Failed to refresh slot data (Network Error) — retrying automatically with that detail in place of Network Error. When no detail is available, the banner reads Failed to refresh slot data — retrying automatically. The page keeps its normal auto-refresh cycle and clears the banner after a successful fetch.

​

Live Logs filtering and drill-down

Use the Live Logs toolbar to move quickly between broad monitoring and request-level troubleshooting. The More actions menu exposes disk-backed downloads that are not limited to the in-memory Pulse buffer: Genesis writes the dedicated error log to the path configured by GENESIS_ERROR_LOG_FILE (default logs/errors.log). Set GENESIS_ERROR_LOG_FILE to an empty string to disable error logging; download actions then return a not-found response. Retention of rotated daily files is controlled by GENESIS_ERROR_LOG_BACKUP_COUNT (default 1). In Raw view, Genesis opens at the newest log lines and keeps auto-following while you stay near the bottom. Press Enter while the Raw log viewer or page background has focus to insert a blank spacer row between log groups during live troubleshooting. Spacer rows stay visible in Raw view, remain selectable so you can drag through them when copying a log range, and do not affect level counts, displayed totals, or filter options. Long raw lines stay fully visible with horizontal scrolling instead of truncating the end of the entry. If you are tracing a failing API or UI action, capture the X-Request-Id response header and use the Request filter in Live Logs to isolate the related entries.

​

Jobs controls

The Jobs view groups operator-facing background work so you can pause or resume the right layer without hunting through internal job ids. Toggle state is persisted in runtime config, so jobs that you disable there stay disabled across restarts. In auth-enabled deployments, job switches and gateway status rows require access_config_panel. Without that permission, job switches are disabled and Pulse does not load the saved job settings or gateway status. The optional jobs below are off by default. Enable them from Pulse -> Jobs; the listed config paths are the persisted settings behind those controls. Pulse omits transient internal handles such as one-off download or refresh ids from this view so the list stays operator-oriented. Azure Container Instances sessions are included in sandbox timeout enforcement by default. Set jobs.sandbox_monitor.enforce_azure to false to disable timeout enforcement for Azure Container Instances; its type is bool and default is true.

​

Server thread observability

The Server view now breaks Python runtime threads into provenance-aware families so long-lived pools, background workers, and unattributed threads are easier to distinguish at a glance. To inspect the full Python thread list and each thread’s origin, open Pulse → Server and select the information icon beside Thread Diagnostics. In SPCS, open the displayed endpoint in the browser session you are signed in with. In other deployments, run the displayed curl command with a Personal Access Token; see Personal Access Tokens for how to create and use one. The endpoint is GET /api/health/threads. When provenance metadata is not available for a family yet, Pulse falls back to the older prefix-based summary so the Server view still surfaces the busiest thread groups. The Server view also shows the async prompt dispatcher used for prompt ingress. These counters help distinguish active work, queued newer prompts, completed dispatches, and dispatch failures. The same Server view also exposes a dedicated thread-service block so worker startup, RPC health, route fallbacks, and thread search behavior are visible without running a live probe manually. When Genesis derives the Server tile health from thread-service RPC metrics, Genesis classifies individual RPC calls at or above 10 seconds as slow and at or above 30 seconds as very slow. Large-thread run_compression work can legitimately take many minutes, so Genesis isolates compression’s contribution using per-action counters in rpc_metrics.actions (for example run_compression.slow_calls and run_compression.very_slow_calls) and evaluates slow/very-slow totals only for other RPCs such as reads, writes, and search. Slow or very-slow compression alone never pins the tile yellow or red; compression failures still surface when they increment RPC error or timeout totals. Because non-compression slow_calls is a cumulative since-boot counter, one to four slow calls after cold-cache reads or post-startup IPC backpressure do not by themselves elevate the tile to warning — Genesis waits for five or more slow non-compression calls before treating slow-call volume as sustained noise. Any non-compression very-slow call marks the tile critical. Other signals — errors, timeouts, route fallbacks, or an inactive worker — still surface as warning or critical on their own. When the Thread Service Process badge is warning or critical, the Server view shows a Why flagged list with every condition contributing to the badge, ordered from most severe to least severe. The corresponding metric rows are marked so you can connect each reason to its counter. The list can identify an inactive worker, main-process reads or writes, very-slow RPCs, RPC errors or timeouts, route fallbacks, sustained slow RPCs, and search errors or timeouts. RPC failure reasons can include the action, outcome, age, and error message; search-error reasons can include the latest search error. These counters accumulate since the server process started, and the displayed window uses the process uptime when the metric snapshot does not provide its own uptime.

​

Server restart access

The Server view includes a guarded Restart action for admins and for roles that include the manage_scheduler permission. In OAuth- or OIDC-protected deployments, Genesis sends the restart request through your current signed-in UI session so the required auth token is included automatically. To prevent accidental restarts, you must type RESTART before confirming. Restarting terminates the current backend process, drops in-flight work, and disconnects active sessions. In orchestrated environments such as Kubernetes or SPCS, the container usually starts again automatically; unmanaged local processes may simply stop.

​

When to use Pulse

Use Pulse when you need to answer questions like:

• Is Genesis healthy right now?
• Are background jobs running correctly?
• Are datasets or metrics uploads lagging?
• Are there recent operational errors or download issues?

​

Related pages

• Scheduler
• Metrics
• Harvester Status