docs: center scraping guidance on auto (#1255)

* docs(concepts): update server architecture and default http strategy

* docs(guides): update strategy migration notes and direct tls deployment

* docs(adapters): document default httpx strategy, cli flags, mcp schemas, and error handling

* docs(infrastructure): sync environment variables and timeout hierarchy with falcon

* docs: center scraping guidance on auto

Deepen the public strategy interface around auto and default while keeping HTTP and server adapters behind their implementation seams. Remove migration narration and incomplete deployment side paths so end users can act on scrape outcomes instead of transport internals.

* docs: correct strategy contract details

State capture persistence and MCP discovery behavior exactly as implemented while keeping compatibility aliases out of recommended user guidance. Use the runtime UnknownStrategy name in troubleshooting.

Author: gildesmarais

src/content/docs/creating-custom-feeds.mdx

@@ -22,7 +22,7 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory]
 3. Validate: `html2rss validate your-config.yml`
 4. Live-check: `html2rss test your-config.yml`, then ship with `html2rss apply your-config.yml`
 5. Mount into `html2rss-web` or contribute to html2rss-configs
-6. Escalate to `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) only when Faraday is not enough
+6. Keep `strategy: auto`; if results are incomplete, confirm the companion scraper is configured before adding site-specific controls
 
 `html2rss feed` is a Thor alias for `apply`. `html2rss auto` aliases `scrape` (one-shot, no YAML).
 

src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx

@@ -50,7 +50,7 @@ Put `BOTASAURUS_SCRAPER_URL` in the MCP `env` block — a shell export does not
 
 ## Golden loops
 
-**Articles now:** `scrape` (or `batch_scrape`). `strategy: "auto"` already falls back to Botasaurus when configured — do not retry with explicit `faraday` after `auto`. Empty items can still be success; follow `next_step`.
+**Articles now:** `scrape` (or `batch_scrape`). Keep `strategy: "auto"` so html2rss can choose the available fetch path. Empty items can still be success; follow `next_step` instead of retrying with another strategy.
 
 **Durable YAML:** optional `inspect` → `recon` → `capture` → `test` → `apply`.
 

src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx

@@ -29,9 +29,6 @@ Print YAML to stdout:
 <Code
   code={`
   html2rss capture https://example.com/articles
-  # Pin Botasaurus for JS-rendered listings
-  BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\
-  html2rss capture https://example.com/articles --strategy botasaurus
   # Hint the item card when auto detection is weak
   html2rss capture https://example.com/articles --items_selector ".post-card"
   # Analyze a local HTML fixture
@@ -46,7 +43,7 @@ Print YAML to stdout:
 
 Common options:
 
-- `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`)
+- `--strategy` — request strategy override; defaults to `auto` (see [Strategy](/ruby-gem/reference/strategy/))
 - `--items_selector` — CSS selector hint for item cards
 - `--limit` — maximum articles kept while deriving selectors (default `25`)
 - `--max-redirects` / `--max-requests` — request budget overrides
@@ -62,10 +59,9 @@ See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full
   require 'html2rss'
   # Derive a config hash (channel + items selector with enhance: true)
   config = Html2rss.capture('https://example.com/articles')
-  # Pin strategy or provide an items hint
+  # Provide an item-card hint when automatic detection is weak
   config = Html2rss.capture(
-    'https://spa-site.com',
-    strategy: :botasaurus,
+    'https://example.com/articles',
     items_selector: '.article-card'
   )
   File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
@@ -76,11 +72,11 @@ See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full
 
 ## How It Works
 
-1. **Request** — `FeedPipeline` (AutoFallback when `:auto`)
+1. **Request** — fetch the page using the configured strategy (`auto` by default)
 2. **Discover** — AutoSource extracts admitted articles
 3. **Segment** — SST Segmenter strategies `:list` → `:cluster` → `:semantic`
 4. **Gate** — emit an items selector only when enough articles match
-5. **Assemble** — `{ items: { selector:, enhance: true } }` plus channel. When AutoFallback selects a concrete transport (or you pin one), Capture **stamps** `strategy:` into the YAML so later `html2rss apply` / `Html2rss.feed` replay the same transport.
+5. **Assemble** — `{ items: { selector:, enhance: true } }` plus channel. Capture records the concrete strategy that produced the draft so `html2rss apply` / `Html2rss.feed` can reproduce that fetch path.
 
 When the quality gate fails, selectors are omitted (`has_selectors: false`) rather than inventing attribute selectors. Hint with `--items_selector` or refine by hand.
 
@@ -99,7 +95,7 @@ MCP `capture` returns that YAML in `payload.yaml`. `validate` / `apply` accept t
 
 1. Validate: `html2rss validate my-feed.yml`
 2. Render: `html2rss apply my-feed.yml`
-3. Tighten the items selector, strategy, or `request.botasaurus` options if needed
+3. Tighten the items selector if needed; keep `auto` unless diagnosis proves a fixed fetch mode is required
 4. For Feed Directory contributions, add `directory.topics` and keep `enhance: true` unless chrome leaks (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config))
 
 ## Related

src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx

@@ -9,9 +9,9 @@ Some websites load their content dynamically using JavaScript. Static fetch path
 
 ## Solution
 
-Default `strategy: auto` automatically tries `faraday` first, then falls back to `botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. This handles many JS-rendered listing pages without needing custom configuration.
+Keep the default `strategy: auto` and configure `BOTASAURUS_SCRAPER_URL`. html2rss can then choose an available fetch path based on the scrape result, which handles many JavaScript-rendered listing pages without per-feed strategy configuration.
 
-When a site requires browser rendering or anti-bot bypass by default, you can explicitly set `strategy: botasaurus` and configure request controls under `request.botasaurus`:
+If a site still needs browser-specific navigation, waits, or scrolling, pin `strategy: botasaurus` and configure those advanced controls under `request.botasaurus`:
 
 <Code
   code={`
@@ -92,7 +92,7 @@ Configure browser actions under `request.botasaurus`:
 
 ### JSON Loaded Over XHR
 
-When Botasaurus uses the browser tier, captured JSON XHR/fetch bodies feed AutoSource `xhr_articles` automatically (enabled by default). Prefer `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) for SPA listing pages that hydrate article lists over the network rather than embedding them in HTML. See [Auto Source](/ruby-gem/reference/auto-source/) and [Strategy](/ruby-gem/reference/strategy/#botasaurus).
+Browser-rendered fetches can pass captured JSON XHR/fetch bodies to AutoSource `xhr_articles` automatically (enabled by default). With the companion scraper configured, keep `auto` for SPA listing pages that hydrate article lists over the network; pin `botasaurus` only when you need its browser-specific controls. See [Auto Source](/ruby-gem/reference/auto-source/) and [Strategy](/ruby-gem/reference/strategy/#botasaurus).
 
 ## Performance Considerations
 
@@ -102,7 +102,7 @@ Browser-based extraction uses more resources than static HTTP fetching because i
 - Executes JavaScript and handles DOM events
 - Manages browser pools and network emulation
 
-Use static HTTP fetching (`faraday`) for static content, and lean on `auto` or explicit `botasaurus` strategies when browser rendering is required. See the [Strategy Reference](/ruby-gem/reference/strategy/) for details.
+Keep `auto` for normal use. Pin a concrete strategy only when diagnosing a site or configuring browser-specific behavior. See the [Strategy Reference](/ruby-gem/reference/strategy/) for details.
 
 ## Related Topics
 

src/content/docs/ruby-gem/reference/auto-source.mdx

@@ -35,7 +35,7 @@ What each scraper does:
 - **`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`).
 - **`microformats2`:** Parses Microformats2 `h-entry` markup, extracting `p-name`, `e-content`, `u-url`, `dt-published`, `p-author`, `p-category`, and `u-photo` / `u-featured` media.
 - **`json_state`:** Walks in-page JSON (`<script type="application/json">`, `window.__NEXT_DATA__`, `window.__NUXT__`, `window.STATE`) for arrays with `title`/`url` pairs.
-- **`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during a Botasaurus **browser** scrape (no extra HTTP). Empty for Faraday and Botasaurus HTTP-request tiers.
+- **`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during browser execution (no extra request). It is empty when the selected fetch path does not capture browser responses.
 - **`wordpress_api`:** Detects `<link rel="https://api.w.org/">` and pulls posts from the REST API. See [WordPress API](/ruby-gem/reference/wordpress-api/).
 - **`sitemap`:** Locates XML sitemaps (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering by priority and recency, with Google News tags (`<news:news>`).
 - **`meta_oembed`:** OpenGraph/Twitter meta tags plus JSON oEmbed (`<link rel="alternate" type="application/json+oembed">`).

src/content/docs/ruby-gem/reference/cli-reference.mdx

@@ -37,7 +37,7 @@ Command: `html2rss scrape [URL]`
 
 Available options:
 
-- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`.
+- `--strategy`: Optional request strategy (`auto`, `default`, `botasaurus`, `local_file`). Defaults to `auto`, which chooses an available fetch path based on scrape results. Use a concrete value only for diagnosis or strategy-specific controls.
 - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`.
 - `--limit`: Maximum number of articles to extract during discovery (defaults to `25`).
 - `--items_selector`: Optional CSS selector hint for item extraction.
@@ -76,11 +76,11 @@ When no extractable items are found, `scrape` classifies likely causes instead o
 
 Known anti-bot interstitial responses (for example Cloudflare challenge pages) are surfaced explicitly as blocked-surface errors.
 
-If all fallback tiers run but still extract zero items, html2rss raises:
+If `auto` exhausts the available fetch paths without items, html2rss raises:
 
 - `No RSS feed items extracted after auto fallback ...`
 
-If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run.
+If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so `auto` can use the companion scraper.
 
 Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed.
 
@@ -123,7 +123,7 @@ Command: `html2rss apply YAML_FILE [feed_name]`
 
 Available options:
 
-- `--strategy`: Request strategy override (`auto`, `faraday`, `botasaurus`, `local_file`).
+- `--strategy`: Request strategy override (`auto`, `default`, `botasaurus`, `local_file`).
 - `--params`: Dynamic parameters passed as key-value pairs (e.g. `--params id:42 section:news`).
 - `--max-redirects`: Maximum redirects to follow per request.
 - `--max-requests`: Total request budget allowed for this feed build.
@@ -150,15 +150,15 @@ Command: `html2rss capture [URL]`
 
 Available options:
 
-- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`.
+- `--strategy`: Optional request strategy (`auto`, `default`, `botasaurus`, `local_file`). Defaults to `auto`.
 - `--items_selector`: Optional CSS selector hint for item extraction.
 - `--limit`: Maximum number of articles to keep (defaults to `25`).
 - `--max-redirects`: Maximum redirects to follow per request.
 - `--max-requests`: Maximum requests to allow for this feed build.
 - `--input`: Local HTML file path to read input from without making network requests.
 - `--explain`: Print capture quality JSON to stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`). Stdout stays YAML.
 
-When AutoFallback (or a pinned strategy) selects a concrete transport, the printed YAML includes a top-level `strategy:` so later `html2rss apply` uses the same hop.
+Capture records the concrete strategy that produced the draft as a top-level `strategy:` so later `html2rss apply` can reproduce that fetch path.
 
 ### MCP
 

src/content/docs/ruby-gem/reference/configuration.mdx

@@ -16,7 +16,6 @@ Global configuration is thread-safe and implements an RCU (Read-Copy-Update) mec
 Html2rss.configure do |config|
 config.log_level = :info
 config.min_ttl = 60
-config.default_strategy = :faraday
 config.headers = { 'User-Agent' => 'MyCustomUserAgent/1.0' }
 end
 `}
@@ -76,15 +75,15 @@ Defines HTTP headers that are globally appended/prepended to all requests. You c
 
 ### `default_strategy`
 
-Sets the default scraper strategy name used when a feed configuration doesn't specify a `strategy`. The strategy name must correspond to a registered strategy.
+Overrides the strategy used when a feed configuration does not specify one. Keep the gem default (`auto`) unless every feed in the process requires a fixed fetch mode.
 
 - **Type**: `Symbol`, `String`, or `nil`
-- **Default**: `nil` (falls back to the gem default strategy, usually `auto`)
+- **Default**: `nil` (uses `auto`)
 
 <Code
   code={`
   Html2rss.configure do |config|
-    config.default_strategy = :faraday
+    config.default_strategy = :auto
   end
   `}
   lang="ruby"

src/content/docs/ruby-gem/reference/mcp-server.mdx

@@ -28,9 +28,9 @@ Daemon logs go to **stderr** (stdout is JSON-RPC). Default `LOG_LEVEL` for the M
 
 ## Strategy
 
-`scrape` and `capture` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback in one call. Prefer native RSS/Atom when present; weak homepage URLs may be rewritten via entry resolution.
+`scrape` and `capture` use `strategy: "auto"` by default so html2rss can choose an available fetch path from the scrape result. Prefer native RSS/Atom when present; weak homepage URLs may be rewritten via entry resolution.
 
-`inspect` with `auto` stays on Faraday. Pin `strategy: "botasaurus"` when inspect needs browser rendering.
+Keep `auto` for normal use. Pin `strategy: "botasaurus"` only when a diagnostic requires browser rendering or browser-specific controls.
 
 Read `html2rss://runtime` for `botasaurus_configured` (boolean only). Set `BOTASAURUS_SCRAPER_URL` on the **MCP process** env.
 
@@ -62,7 +62,7 @@ Golden path for durable YAML: `inspect` (optional) → `recon` → `capture` →
 
 One-shot article extraction as JSON Feed items (no saved config).
 
-- **Parameters:** `url` (required); `strategy` (`auto` / `faraday` / `botasaurus`, default `auto`); `limit` (default `25`); optional `items_selector`
+- **Parameters:** `url` (required); `strategy` (`auto` / `default` / `botasaurus`, default `auto`); `limit` (default `25`); optional `items_selector`
 - **Payload:** `items`, plus totals / strategy fields when present
 - Empty items can still be `ok: true` — follow `next_step` / `guidance`
 
@@ -120,7 +120,7 @@ Ship gate: build RSS from a config. Required `url` plus exactly one of `config`
 | ----------------------- | ------------------------------------------------------------------------------------------------------------------ |
 | `html2rss://schema`     | Feed config JSON Schema                                                                                            |
 | `html2rss://extractors` | Extractor names                                                                                                    |
-| `html2rss://strategies` | `auto`, `faraday`, `botasaurus` (not `local_file`)                                                                 |
+| `html2rss://strategies` | Runtime strategy names for client discovery; prefer `auto`, `default`, or `botasaurus`                             |
 | `html2rss://runtime`    | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` — never the scraper URL |
 
 `validate` / `apply` reject `strategy: local_file` and `request.local_file_path`. Use CLI `--input` for fixtures.