Add logfire-datasets skill for creating hosted datasets

[duncankmckinnon]

Summary

Adds a locally owned logfire-datasets skill. The Logfire MCP server has no dataset tools, so agents currently have no guidance for creating hosted datasets in Datasets & Experiments. The skill walks an agent through:

1. Configuring credentials: an API key (project:read_datasets, project:write_datasets) for the datasets API, and a read token for script-side queries. The read token can come from the UI or be minted with the Logfire CLI through the shared auth.md flow, as logfire-evals does; the CLI output is captured straight into .env. Includes a fixed handoff: credentials live in a gitignored .env, and the agent never sees the values. Read-only checks, plus a project_id check, run before any write.
2. Agreeing on the case shape: inputs, expected output, metadata. Inputs, outputs and metadata are JSON objects, because the API rejects a bare scalar expected_output.
3. Discovering source data: span inventory, attribute keys, value sampling, and relating spans through the trace tree. The skill assumes nothing about the data's format; the agent learns it from the data and confirms the selection with the user.
4. Writing a project-specific Python script that retrieves rows, maps them to cases, validates them, dry-runs, and pushes only after user confirmation.
5. Verifying the hosted case count.

Running evaluations against a dataset is left to logfire-evals.

Files

• plugins/logfire/skills/logfire-datasets/SKILL.md: the workflow.
• references/trace-extraction.md: a discovery-driven method for extraction, covering engine rules, the discovery sequence, how to read what discovery shows, scrubbed values, and choosing rows. It makes no assumptions about span names, attribute keys, or payload shapes.
• references/python-scripts.md: guidance for writing the script with logfire.query_client and logfire.experimental.api_client. Covers credentials, stages, the case dict format, local validation, push behaviour, reading datasets back, and async clients.
• Registered in scripts/logfire-skill-sources.json as owned by pydantic/skills, mirrored to skills/logfire-datasets/, and the ownership test's skill count bumped from 9 to 10. READMEs and CONTRIBUTING updated.

Design notes

• No bundled script. No other skill in the catalog ships scripts, and a template invites agents to copy its example shape. The reference describes what the script must do and the SDK details instead.
• Uploads use add_cases() with dicts on an existing dataset, not push_dataset(). Only the dict form carries source_trace_id/source_span_id, so cases link back to their traces. push_dataset() also replaces dataset-level evaluators that may have been configured in the UI.
• Stable case names (for example derived from span_id) make re-runs upsert instead of duplicate.
• SDK behaviour was checked against the logfire 4.39 and 5.2.0 sources (the API is the same in both) and against valcore's working hosted-dataset integration.

Testing

• bash scripts/check-skill-sync.sh
• python3 -m unittest discover -s scripts/tests (7 tests pass)
• claude plugin validate .
• End-to-end run of the skill: built a bazaar-pr-reviews dataset (8 code-review cases) from delegated sub-agent runs in a real project and pushed it. Hosted content and trace links were verified against the source by md5. A re-push left the count at 8.

What the end-to-end test changed

• The query API rejects API keys (401 Invalid read token), even with project:read, so script-side queries need a read token. The credentials step now distinguishes LOGFIRE_API_KEY (datasets) from LOGFIRE_READ_TOKEN (queries). It also documents an MCP-export fallback with md5 integrity checks for small sets.
• Credential handoff pattern: keys go in a gitignored .env, because a variable the user exports after the agent session starts never reaches the agent's commands. Scripts look for .env in the working directory or project root.
• Generalised extraction guidance: Logfire scrubber placeholders ([Scrubbed due to '...'], which removed 7 of 15 candidate rows), relating spans through parent/child joins, values that share boilerplate and defeat naive keyword filters, and confirming wording-based filters with the user. Format-specific tables for particular instrumentation were removed so the skill stays general purpose.
• Upserting a named case bumps its version even when its content is unchanged.

Follow-up (upstream)

This PR narrows the logfire-datasets description to hosted datasets so it doesn't overlap logfire-evals. Routing setup requests to logfire-datasets, and pointing logfire-evals' hosted-dataset section at it, needs changes outside this repo, in two places:

• pydantic/logfire (logfire-sdk/logfire/.agents/skills/logfire-evals and logfire-setup): the public owners this repo syncs from, per scripts/logfire-skill-sources.json.
• pydantic/logfire-cli (skills/logfire-evals, skills/logfire-setup): the release-bound CLI adaptations bundled into logfire wizard / logfire skill, which are ported by hand rather than synced.

🤖 Generated with Claude Code

[duncankmckinnon]

Moving ownership of logfire-datasets to pydantic/logfire, since it documents SDK functionality (logfire.experimental.api_client): pydantic/logfire#2586. That PR also adds the setup routing row and the logfire-evals boundary.

Back to draft until it merges. Then, per CONTRIBUTING's ordering rule, this PR becomes the ownership-map change: point logfire-datasets in scripts/logfire-skill-sources.json at pydantic/logfire / logfire-sdk/logfire/.agents/skills/logfire-datasets (the sync loops over the manifest, so no separate sync_skill entry is needed), move it to the upstream rows in CONTRIBUTING, and let the sync bring in the files.