> ## Documentation Index
> Fetch the complete documentation index at: https://haico.gr/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Tracing & data collection
> How HAI-Co² records every co-construction: agent execution traces through Phoenix and LangSmith, and a complete, replayable first-party dataset in Postgres.
HAI-Co² is, among other things, a **data-collection platform for studying human-AI co-construction**. Every session produces two complementary records:
1. **Execution traces** of how the agent thought: the LLM calls, prompts, completions, token usage, tool calls, and timings, captured through OpenTelemetry and viewable in Phoenix (and optionally LangSmith).
2. **A first-party structured dataset** in the application's own Postgres database: the full message history, a per-turn snapshot of the entire workspace, the conversation trajectory and its branches, the agent's reasoning trace, and every artifact produced.
Together these let you reconstruct, replay, and analyse exactly what happened on any turn of any conversation. This page covers both streams and the consent and privacy model around them.
## Two recording streams
```ascii theme={null}
a user turn
│
┌────────────────────┴─────────────────────┐
▼ ▼
EXECUTION TRACES FIRST-PARTY DATASET
(how the agent thought) (what the system stored)
OpenTelemetry / OpenInference Postgres (the app's own DB)
auto-instruments LangChain • workspace_snapshots (per turn)
+ LangGraph • LangGraph checkpointer (messages)
│ • conversation_users / _branches
▼ • artifacts (typed outputs)
Phoenix (self-hosted) • documents / objectives /
LangSmith (optional) todos / preferences
• prompts + completions • user_activity (audit + IP)
• token usage, latency • users (identity + consent)
• tool calls / args / returns
• grouped by session.id = thread_id
```
```mermaid theme={null}
flowchart TD
U["A user turn"] --> TR["Execution traces
(how the agent thought)"]
U --> DS["First-party dataset
(what the system stored)"]
TR --> OTEL["OpenTelemetry / OpenInference
auto-instruments LangChain + LangGraph"]
OTEL --> PX["Phoenix (self-hosted)
+ LangSmith (optional)
prompts · completions · tokens · tool calls
grouped by session.id = thread_id"]
DS --> PG[("Postgres
snapshots · messages · trajectory ·
branches · artifacts · activity · identity")]
```
## Agent tracing (Phoenix & LangSmith)
A single module, [`backend/app/services/observability.py`](https://github.com/petrosrapto/HAICO/blob/main/backend/app/services/observability.py), is the entry point. `init_observability()` runs once at application startup (from the FastAPI lifespan in [`main.py`](https://github.com/petrosrapto/HAICO/blob/main/backend/app/main.py), before any agent is built) and activates backends based on the `observability.provider` flag, which takes one of `none`, `langsmith`, `phoenix`, or `both`.
Both backends **auto-instrument** LangChain and LangGraph globally, so there are no per-call callbacks to maintain:
* **Phoenix** is registered through `phoenix.otel.register(..., auto_instrument=True)`, which patches LangChain via OpenInference and exports spans over OTLP.
* **LangSmith** is activated by setting LangChain's `LANGCHAIN_*` environment variables; LangChain then traces itself.
If a backend's packages are missing or it is disabled, that backend is skipped and the app runs unaffected.
### What gets captured
Because instrumentation sits at the LangChain/LangGraph layer, a trace records the agent's actual execution:
* every **LLM call**: the rendered prompt, the completion, the model, token usage, and latency;
* every **tool call**: arguments and return values, including the `(content, artifact)` results;
* **LangGraph** node transitions and the model-node retry wrapper;
* **errors and retries** along the way.
Spans are tagged for analysis. `get_session_context(thread_id)` wraps a turn so every span carries `session.id = thread_id`, which groups all of a conversation's spans into one Session in Phoenix. `get_trace_context(...)` layers on optional metadata keys and string tags. Separately, the [`@workspace_tool`](https://github.com/petrosrapto/HAICO/blob/main/backend/app/tools/_decorator.py) decorator logs each tool's `action_and_reasoning` sentence, the same plain-language line the user sees in the Internal Reasoning trace.
### Admin trace deep-links
From the admin dashboard, a **Trace** button next to any conversation opens that thread's Phoenix session. The backend resolves the link lazily: `GET /api/admin/threads/{thread_id}/trace` looks up the session through the Phoenix REST API (authenticated with the System `PHOENIX_API_KEY`), builds the public URL from `phoenix.web_base_url`, and caches it on `conversation_users.phoenix_url` so later requests are a single DB read. The link is `null` until Phoenix has a session for the thread, and the whole feature is disabled when `phoenix.web_base_url` is empty. The same resolved link is embedded in the GitHub issue that the "create issue from report" action files (see [User feedback](/docs/feedback)).
### Configuration
Tracing is configured per environment in `deployment//backend/config.*.yaml`, with secrets supplied through the environment. Secrets (the LangSmith API key and the Phoenix System API key) never live in YAML.
| Setting (`observability.*`) | Env var | Default | Purpose |
| ---------------------------- | ----------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `provider` | (YAML only) | `phoenix` in all deployments | Which backends to run: `none` / `langsmith` / `phoenix` / `both`. |
| `phoenix.enabled` | (YAML only) | `false` in code, `true` in the YAML | Turn Phoenix on. |
| `phoenix.collector_endpoint` | (YAML only) | `http://phoenix:4317` | Phoenix OTLP gRPC collector (the Compose network). |
| `phoenix.project_name` | (YAML only) | `haico` (`haico_local` / `haico_dev` / `haico_prod`) | Phoenix project to file spans under. |
| `phoenix.web_base_url` | (YAML only) | empty | Public Phoenix UI base the admin's browser opens (e.g. `https://haico.gr/phoenix`). Empty disables admin trace deep-links. |
| `phoenix.api_base_url` | (YAML only) | `http://phoenix:6006` | Phoenix REST API base used to resolve a session's deep-link (the Compose network). |
| `phoenix.api_key` | `PHOENIX_API_KEY` | empty | Phoenix System API key (auth is on): needed both for OTLP exports and to resolve trace deep-links; from env only. |
| `langsmith.enabled` | (YAML only) | `false` in code and in the YAML | Turn LangSmith on (still needs a key). |
| `langsmith.project` | `LANGCHAIN_PROJECT` | `haico` | LangSmith project name. |
| `langsmith.api_key` | `LANGCHAIN_API_KEY` / `LANGSMITH_API_KEY` | empty | Required to actually enable LangSmith; from env only. |
| `langsmith.endpoint` | `LANGCHAIN_ENDPOINT` | `https://api.smith.langchain.com` | LangSmith API endpoint. |
The default everywhere is `provider: phoenix` with `langsmith.enabled: false`, so tracing goes only to the self-hosted Phoenix and **no session content is sent to LangSmith** (an external SaaS). This keeps prompts and completions inside the deployment's own infrastructure, which is the appropriate default for a GDPR research deployment handling personal data. To also send traces to LangSmith, set `provider: both`, `langsmith.enabled: true`, and provide `LANGCHAIN_API_KEY`. To turn tracing off entirely, set `observability.provider: none`.
### Phoenix deployment
Phoenix runs as its own container (`arizephoenix/phoenix`) in each environment's `docker-compose.yml`, alongside Postgres, the backend, and the frontend. It serves its UI on port **6006** and its OTLP collector on **4317**; the backend exports to `http://phoenix:4317` over the Compose network. Traces persist to a SQLite database in a named volume, and the default retention policy is **0 days, meaning traces are never purged**.
In dev and production it is exposed behind the reverse proxy at `/phoenix` (`PHOENIX_HOST_ROOT_PATH=/phoenix`), so it is reachable at `https://dev.haico.gr/phoenix` and `https://haico.gr/phoenix`; locally it is bound to `127.0.0.1:6006`.
**Two Phoenix gotchas, both covered in [Deployment](/docs/deployment).** `PHOENIX_SECRET` must be **at least 32 characters** or the container crash-loops. And with Phoenix auth enabled, its OTLP collector is protected too, so the backend needs a Phoenix System API key in `PHOENIX_API_KEY` or trace exports fail with `UNAUTHENTICATED`.
## First-party data collection (the research dataset)
This is the part that makes HAI-Co² a study instrument rather than just a traced app: independently of Phoenix, the backend records a **complete, self-contained, replayable account of every co-construction** in its own database. The schema is documented in full in [Database schema](/docs/database-schema); the table below is the data-collection view of it.
| What is collected | Where (table) | When written | Notable contents |
| ------------------------------- | ------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Per-turn workspace snapshot** | `workspace_snapshots` | Before the agent acts on each turn | `user_message`, full `document_content` and `document_title`, `objective_text`, `preferences_json`, `todos_json`, tagged by `turn_index` (unique per `(thread_id, turn_index)`) |
| **Full message history** | LangGraph checkpointer tables | Every turn (by LangGraph) | Serialised Human / AI / Tool messages: prompts, completions, tool arguments and returns |
| **Conversation trajectory** | `conversation_users` | On each turn | `turn_index`, `message_count`, title, ownership, soft-delete marker |
| **Branch lineage** | `conversation_branches` | On each branch/fork | `parent_thread_id`, `branch_point_turn`, reason, status |
| **Reasoning trace** | application logs + the SSE stream | Every tool call | the `action_and_reasoning` sentence per tool call (also surfaced in the UI) |
| **Typed artifacts** | `artifacts` | When a tool emits one | `artifact_type`, full `payload` (JSONB), tagged by `turn_index` (see [Artifacts](/docs/artifacts)) |
| **Live workspace** | `documents`, `objectives`, `todos`, `preferences` | On each agent or human edit | the current state of each panel |
| **Activity / audit** | `user_activity` | login, register, query, consent | `action`, `ip_address`, `timestamp`, free-text detail |
| **Identity** | `users` | On register / OAuth | username, email, `auth_provider`, `google_id`, hashed password, `recording_consent_at` |
Two of these deserve emphasis, because together they make a turn fully reconstructable:
* **Workspace snapshots** are the backbone. Snapshot *N* is written at the **start** of turn *N*, capturing the triggering user message plus the entire workspace the agent is about to act on (document, objective, preferences, todos), each embedded as self-contained JSON. That is enough to re-render the exact context the agent saw on any past turn, which is also what powers restore and branching. The timing rule (snapshot `k+1`, or the live rows, gives the state *after* turn `k`) is detailed in [Database schema](/docs/database-schema).
* **The LangGraph checkpointer** stores the verbatim message history (every Human, AI, and Tool message, with serialised prompts, completions, and tool I/O), so the conversation can be replayed message by message.
Because every workspace table is scoped by `thread_id` and joined to a user through `conversation_users(thread_id, user_id)`, the whole dataset is attributable per user and per conversation for analysis.
## Consent & privacy
HAI-Co² records research data, so it carries an explicit **consent marker**. The `users.recording_consent_at` timestamp is `NULL` until the user consents; `POST /api/auth/consent` sets it once (idempotently) and logs a `recording_consent_granted` activity event. The field is returned on the user object so the frontend can gate the recording-consent flow.
What is stored about a user: username and email, `auth_provider` and (for Google sign-in) `google_id`, a bcrypt password hash (null for OAuth-only accounts), the consent timestamp, and an `ip_address` recorded on each activity event (login, register, query, consent). Generic error messages on registration avoid confirming whether an email already exists.
**Retention.** There is no automatic purge in the application: workspace snapshots and LangGraph checkpoints persist indefinitely, deleting a conversation is a soft delete (it sets `deleted_at` and preserves the workspace data), and Phoenix's trace retention is set to never purge. Removing data is therefore a deliberate administrative action. Plan retention according to your study's data-governance and ethics requirements.
## Where to read next
* [Database schema](/docs/database-schema): every table, snapshots, and how restore and branching reconstruct a turn.
* [Conversation branching](/docs/conversation-branching): how the trajectory tree is built and forked.
* [Artifacts](/docs/artifacts): the typed outputs recorded in the `artifacts` table.
* [Deployment](/docs/deployment): the Phoenix service, its secrets, and the reverse-proxy route.