fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885) by beaglemoo · Pull Request #3222 · karakeep-app/karakeep

beaglemoo · 2026-10-23T09:44:06Z

Fixes #2885. Supersedes #2944, which was closed when my fork was deleted by mistake during a cleanup.

Reddit now blocks unauthenticated .json requests, and single-post oauth.reddit.com requests made with an anonymous guest token, with a 403 "blocked by network security" page, so Reddit bookmarks no longer get their content. #2887 adds an OAuth client-credentials path, but Reddit closed self-serve API app creation under its Responsible Builder Policy, so many self-hosters can't obtain credentials.

This adds an anonymous path that reads the rendered post page instead of the blocked JSON:

If the crawled DOM already contains the post's <shreddit-post>, the data is extracted from it with no extra request.
Otherwise the URL is normalised to https://www.reddit.com/comments/<id>/ and fetched with a browser User-Agent. Reddit serves a lightweight JS challenge (solution = seed repeated twice, hidden jsc_token field). The plugin solves it, sends the submit without cookies (Reddit rejects it otherwise), follows redirects by hand so the clearance cookies set on the 301 are kept, and parses the rendered HTML.
Extracted: title, author, subreddit, date, image, post body and the top 10 top-level comments by score.
Only post URLs are handled, and only the matching shreddit-post#t3_<id> is used, so subreddit, profile and home feeds are left to the generic metascraper rules.
The optional OAuth path (REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET) is kept. It now requests the canonical post, so every permalink of a post maps to the same cache entry, and it falls back to the anonymous path on any non-OK response (the token is dropped only on 401/403).
If both fail, the plugin returns the HTML-escaped DOM post title as content, so generic readability never captures Reddit's cookie-consent banner.
fetchWithProxy gains support for redirect: "manual", which returns the 3xx response instead of following it. URL validation and proxy selection still run on every request; no other caller passes redirect, so existing behaviour is unchanged.

Tests: 21 new (challenge flow with mocked hops including the cookieless submit and both 301 and direct-200 responses, the shreddit extractor, URL normalisation, the feed-page guard, the canonical OAuth request, fallback title escaping, and the redirect: "manual" option).

Verified end to end against live Reddit on 2026-10-23 on a self-hosted instance with no credentials configured: Reddit bookmarks crawl with title, author, subreddit, post body and top comments.

Note: this deliberately solves Reddit's anti-bot challenge. Flagging that explicitly in case it raises ToS concerns for the project.

…eep-app#2885) Reddit now blocks unauthenticated .json metadata requests. PR karakeep-app#2887 adds an OAuth client-credentials path, but Reddit closed self-serve API app creation, so many self-hosters cannot obtain credentials. This keeps the OAuth path (used when REDDIT_CLIENT_ID/SECRET are set) and adds an anonymous fallback: with a browser User-Agent, Reddit serves a lightweight JS challenge (solution = seed repeated twice) rather than a hard block. The metascraper warms up on the HTML post URL, solves the challenge, submits the hidden form to earn clearance cookies, and reuses them for the .json request. Cookies are threaded manually since fetchWithProxy (node-fetch) has no cookie jar. If the challenge format changes it falls back to the existing DOM scraping.

- Resolve the challenge form action with new URL(action, base) so absolute or protocol-relative actions no longer build a malformed submit URL. - On a 401/403 from the OAuth path, drop the cached token and fall through to the anonymous challenge path instead of failing the crawl.

If the JSON body fetch fails (e.g. Reddit rate-limits us), returning no readable content let the generic readability pass scrape the rendered page, which on Reddit captures the cookie-consent banner as the article body and poisoned tagging/summaries. Fall back to the DOM post title instead so the stored content stays meaningful and a later re-crawl can recover the body.

Many Reddit posts (link/discussion posts) have an empty body, so the useful context lives in the comments. Parse the comments listing and fold the top ~10 comments (by score, with author) into readableContentHtml after the post body, so the reader view and the tagger get real content instead of just the title. HTML is sanitized downstream by DOMPurify.

The OAuth path only fell back to the anonymous challenge on 401/403, so a 429 rate limit or transient 5xx from oauth.reddit.com returned the failing response and kept reusing the cached token. Fall back on any non-OK status; only drop the cached token on 401/403 (a rate limit or server error does not mean the token is invalid).

…ascraper

greptile-apps · 2026-10-23T09:48:51Z

Confidence Score: 4/5

[Medium impact] The changes affect the Reddit bookmark-crawling workflow.

Fix redirect cookie collection before merging so the anonymous fallback works when clearance is granted through a redirect.

Findings

Clearance cookies disappear during redirects ▶
Fragments skip the HTML warmup ▶
Fallback titles lose text ▶

Fix with agent prompt

### Issue 1
apps/workers/metascraper-plugins/metascraper-reddit.ts:261-262
If the challenge response sets a clearance cookie and redirects, `fetchWithProxy` discards that cookie before `browserGet` sees it. Neither the redirect request nor the later JSON request receives it, so anonymous crawling remains blocked. Collect cookies from each response and apply them before following the next redirect.

### Issue 2
apps/workers/metascraper-plugins/metascraper-reddit.ts:333
For a post URL ending in a fragment without a query, such as `/comments/abc/title/#section`, `buildJsonUrl` produces `/comments/abc/title.json#section`. This replacement does not match that string, so the warmup requests the JSON endpoint instead of the HTML page needed for the challenge. Build the warmup URL by changing `URL.pathname`, rather than matching the whole URL string.

```suggestion
  const htmlUrlObj = new URL(jsonUrl);
  htmlUrlObj.pathname = htmlUrlObj.pathname.replace(/\.json$/, "");
  const htmlUrl = htmlUrlObj.toString();
```

### Issue 3
apps/workers/metascraper-plugins/metascraper-reddit.ts:676-677
`fallbackDomTitle` returns plain text, but this return value is later treated as HTML. When JSON fetching fails, a title such as `Working with vector<T>` loses `<T>` when the HTML is sanitized. Escape `domTitle` before returning it, preferably inside a paragraph, so the fallback preserves the title.

```suggestion
      const domTitle = fallbackDomTitle({ htmlDom });
      return domTitle ? `<p>${escapeHtml(domTitle)}</p>` : undefined;
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds OAuth credentials and an anonymous Reddit challenge flow. It also includes top-level comments in the reader view.

Reddit bookmarks can fetch post data with OAuth or an anonymous challenge.
Reddit bookmarks include top comments in their saved reader content.

Intentional behavior recorded: beaglemoo explicitly chose a title-only fallback after JSON failure to avoid saving the cookie-consent banner.

_{Reviews (1) · Last reviewed commit: "style(workers): apply oxfmt and fix no-i..." · Reviewed by Greptile}

greptile-apps · 2026-10-23T09:48:54Z

+  const response = await fetchWithProxy(url, { headers });
+  storeSetCookies(response, jar);


Clearance cookies disappear during redirects

If the challenge response sets a clearance cookie and redirects, fetchWithProxy discards that cookie before browserGet sees it. Neither the redirect request nor the later JSON request receives it, so anonymous crawling remains blocked. Collect cookies from each response and apply them before following the next redirect.

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/workers/metascraper-plugins/metascraper-reddit.ts Line: 261-262 Comment: **Clearance cookies disappear during redirects** If the challenge response sets a clearance cookie and redirects, `fetchWithProxy` discards that cookie before `browserGet` sees it. Neither the redirect request nor the later JSON request receives it, so anonymous crawling remains blocked. Collect cookies from each response and apply them before following the next redirect. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-23T09:48:54Z

+  // reuse the clearance cookies for the JSON request.
+  const jsonUrl = buildJsonUrl(url, false);
+  const jar: CookieJar = new Map();
+  const htmlUrl = jsonUrl.replace(/\.json(\?.*)?$/, "$1");


Fragments skip the HTML warmup

For a post URL ending in a fragment without a query, such as /comments/abc/title/#section, buildJsonUrl produces /comments/abc/title.json#section. This replacement does not match that string, so the warmup requests the JSON endpoint instead of the HTML page needed for the challenge. Build the warmup URL by changing URL.pathname, rather than matching the whole URL string.

Suggested change

const htmlUrl = jsonUrl.replace(/\.json(\?.*)?$/, "$1");

const htmlUrlObj = new URL(jsonUrl);

htmlUrlObj.pathname = htmlUrlObj.pathname.replace(/\.json$/, "");

const htmlUrl = htmlUrlObj.toString();

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/workers/metascraper-plugins/metascraper-reddit.ts Line: 333 Comment: **Fragments skip the HTML warmup** For a post URL ending in a fragment without a query, such as `/comments/abc/title/#section`, `buildJsonUrl` produces `/comments/abc/title.json#section`. This replacement does not match that string, so the warmup requests the JSON endpoint instead of the HTML page needed for the challenge. Build the warmup URL by changing `URL.pathname`, rather than matching the whole URL string. ```suggestion const htmlUrlObj = new URL(jsonUrl); htmlUrlObj.pathname = htmlUrlObj.pathname.replace(/\.json$/, ""); const htmlUrl = htmlUrlObj.toString(); ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-23T09:48:55Z

+      const domTitle = fallbackDomTitle({ htmlDom });
+      return domTitle ?? undefined;


Fallback titles lose text

fallbackDomTitle returns plain text, but this return value is later treated as HTML. When JSON fetching fails, a title such as Working with vector<T> loses <T> when the HTML is sanitized. Escape domTitle before returning it, preferably inside a paragraph, so the fallback preserves the title.

Suggested change

const domTitle = fallbackDomTitle({ htmlDom });

return domTitle ?? undefined;

const domTitle = fallbackDomTitle({ htmlDom });

return domTitle ? `${escapeHtml(domTitle)}` : undefined;

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/workers/metascraper-plugins/metascraper-reddit.ts Line: 676-677 Comment: **Fallback titles lose text** `fallbackDomTitle` returns plain text, but this return value is later treated as HTML. When JSON fetching fails, a title such as `Working with vector<T>` loses `<T>` when the HTML is sanitized. Escape `domTitle` before returning it, preferably inside a paragraph, so the fallback preserves the title. ```suggestion const domTitle = fallbackDomTitle({ htmlDom }); return domTitle ? `${escapeHtml(domTitle)}` : undefined; ``` --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

pullfrog

Important

The anonymous challenge solver matches the pre-2026-09 form field name, so it never submits the solution — and the PR's own cited source now says the .json endpoint it targets is blocked for signed-out clients. Both are worth reconciling before merge.

Reviewed changes

Anonymous Reddit fallback — warms up on the post HTML page, solves the embedded js_challenge form, threads clearance cookies through a manual jar, then retries the .json request when no OAuth credentials are configured.
OAuth client-credentials path — fetches and caches an app token (5-minute safety buffer, in-flight de-dup), routes reads to oauth.reddit.com, and falls through to anonymous on any non-OK response (dropping the token only on 401/403).
Top comments in content — parses kind: "t1" children and folds the ten highest-scored comments into readableContentHtml, sanitized downstream by DOMPurify.
Content guard — falls back to the DOM title instead of letting generic readability capture Reddit's cookie-consent banner when the JSON fetch fails.
Config — new REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET wired into serverConfig.reddit.

ℹ️ The anonymous `.json` premise may already be overtaken by Reddit

Independently of the field-name issue inline, the approach of reusing challenge clearance cookies against www.reddit.com/...json looks like the older technique. The gist this PR cites now states Reddit blocks the .json API for signed-out clients (~2026-10) and that the challenge's token_v2 guest token must instead be sent as a Bearer to oauth.reddit.com; glance's current, working solver does exactly that and has moved off www.reddit.com/*.json entirely. Worth re-verifying against live Reddit before relying on the anonymous path — the claimed end-to-end success may predate this change.

Technical details

# Anonymous .json path may be obsolete

## Background
- `metascraper-reddit.ts:329-343` reuses warm-up/challenge cookies for `www.reddit.com/<path>.json`.
- Reddit reportedly stopped serving `.json` to signed-out clients (~2026-10), even with the full challenge cookie set (`loid`, `token_v2`, `csrf_token`, ...).

## Evidence
- The gist this PR cites now documents in its module docstring: "Reddit blocks the .json API for unauthenticated clients (~2026-10), but the normal HTML post page still carries everything" — it now parses the HTML page instead of the challenge/.json route.
- glance (current main, `internal/glance/widget-reddit.go`) sends the challenge's `token_v2` as `Authorization: Bearer` to `oauth.reddit.com` for signed-out requests rather than reusing cookies on `www.reddit.com`.

## Required outcome
- Confirm the anonymous path still returns JSON against live Reddit; if not, pivot to the guest-token-on-oauth approach or drop the anonymous path in favour of DOM parsing (which the plugin already does as a fallback).

## Open questions for the human
- Does the author's end-to-end verification still pass on current Reddit, and on which date was it run?

^{｜ Fix it ➔ ｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

pullfrog · 2026-10-23T09:55:09Z

Anchored detail for the field-name issue in the review above (couldn't attach it inline):

apps/workers/metascraper-plugins/metascraper-reddit.ts:271-281

Reddit renamed the JS-challenge form field from token to jsc_token around 2026-09. The regex here matches the literal name="token", which is not a substring of name="jsc_token", so solveRedditChallenge hits the guard at lines 273-276 and returns early — the solution is never submitted and no clearance cookie is earned. The gist this PR cites has been updated to name="(jsc_token|token)" with an explicit rename note, and glance's working solver matches name="jsc_token" and submits the param under jsc_token.

Suggested fix:

  const token = /name="(jsc_token|token)"\s+value="([0-9a-f]+)"/.exec(html);
  const action = /<form[^>]*action="([^"]+)"/.exec(html);
  if (!seed || !token || !action) {
    // Not a solvable challenge page (or the format changed) - nothing to do.
    return;
  }
  const params = new URLSearchParams({
    solution: seed[1] + seed[1],
    js_challenge: "1",
    [token[1]]: token[2],
    jsc_orig_r: "",
  });

Using the captured field name ([token[1]]) keeps the submit correct for either format. Given this is exactly the kind of silent format drift that broke it, a small unit test feeding sample challenge HTML and asserting the submit URL/params would be worth adding.

Sources: the cited gist (# Field was "token" originally, renamed to "jsc_token" ~2026-09.) and glanceapp/glance internal/glance/widget-reddit.go (name="jsc_token", param jsc_token).

^{｜ View workflow run ｜ via Pullfrog ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

…rect: manual

…ookies

…RL normalisation

…e feed pages

… the bookmarked permalink

beaglemoo · 2026-10-23T14:03:22Z

Thanks to Pullfrog and Greptile, the review was right. Reddit renamed the challenge field to jsc_token and now blocks anonymous .json, so the old path had stopped working. I have reworked the anonymous path to parse the rendered post HTML, fixed the redirect-cookie and fragment issues and the title escaping, and re-verified it against live Reddit today. Details are in the updated description.

pullfrog

✅ No new issues found.

Reviewed changes

Re-reviewed the five commits pushed since the prior pullfrog review (01fb2766 → 2acfba68), which rework the anonymous Reddit path in response to the earlier feedback and add test coverage.

Rendered-page extraction replaces the blocked .json route — the anonymous path now GETs the post page, solves the JS challenge, and parses <shreddit-post> / <shreddit-comment> markup for title, author, subreddit, timestamp, body, image, and top comments; the OAuth client-credentials path is unchanged.
Challenge field rename handled — parseRedditChallenge accepts both token and jsc_token and echoes the token back under the name the form served, resolving the action against the page URL.
Manual-redirect support in fetchWithProxy — a new followRedirects flag (false only when the caller passes redirect: "manual") lets browserGet capture Set-Cookie from intermediate 3xx hops; each hop stays SSRF-validated and redirects off reddit.com are not followed, so the jar cannot leak.
Cookie-jar fixes — deletions (Max-Age=0 or empty value) are now honoured, and the challenge submit is sent without cookies on its first hop (avoids the edgebucket 403).
Escaped title fallback — readableContentHtml wraps the DOM-title fallback in escaped  HTML via titleToHtml, and post URLs are normalized before cache keying.
Tests — new metascraper-reddit.test.ts (17 tests) and network.redirect.test.ts (3 tests), all passing locally.

The prior review's two concerns (stale token field name, and the anonymous .json premise being obsolete) are both addressed by these commits.

^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

pullfrog

✅ No new issues found.

Reviewed changes

Re-reviewed the two commits pushed after the previous review (2acfba68 → 77f5dcdc), which scope the plugin to the bookmarked post and canonicalize the OAuth request.

Feed pages ignored — new extractRedditPostId reduces a URL to its lower-cased post id (or undefined); fetchRedditPostData returns early for non-post Reddit URLs, and extractFromShreddit now requires the <shreddit-post id="t3_<id>"> matching the bookmarked post, so subreddit/profile/home feeds no longer leak unrelated posts. The image, title, and readableContentHtml rules also early-return for non-post URLs, letting the generic metascraper rules apply.
Canonical OAuth request — buildOauthJsonUrl(postId) now always reads https://oauth.reddit.com/comments/<id>.json?sort=top&limit=25&depth=1, and the result cache is keyed by post id, so every permalink of a post (including comment permalinks) shares a single result.
Tests — added coverage for the feed-page guard, the matching-post-id requirement, the canonical OAuth URL, and cross-permalink cache reuse; the suite is now 24 tests and passes locally.

The prior review's feedback remains addressed and no new issues were found in this delta.

^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

beaglemoo added 6 commits October 23, 2026 10:33

style(workers): apply oxfmt and fix no-inferrable-types in reddit met…

01fb276

…ascraper

beaglemoo mentioned this pull request Oct 23, 2026

fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885) #2944

Closed

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

pullfrog Bot reviewed Oct 23, 2026

View reviewed changes

beaglemoo added 7 commits October 23, 2026 14:38

feat(network): let fetchWithProxy return redirect responses with redi…

ed0e5c0

…rect: manual

fix(crawler): handle renamed jsc_token challenge field and redirect c…

a0bce56

…ookies

fix(crawler): parse rendered Reddit post HTML instead of blocked .json

5dfc6c3

fix(crawler): escape fallback Reddit title in readable content

008184e

test(crawler): cover Reddit shreddit extraction, challenge flow and U…

2acfba6

…RL normalisation

fix(crawler): only extract Reddit data for the bookmarked post, ignor…

5d79c1c

…e feed pages

fix(crawler): request the canonical Reddit post from OAuth instead of…

77f5dcd

… the bookmarked permalink

pullfrog Bot reviewed Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885) - #3222

fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885)#3222
beaglemoo wants to merge 13 commits into
karakeep-app:mainfrom
beaglemoo:fix/reddit-crawl-anonymous-fallback

beaglemoo commented Oct 23, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

greptile-apps Bot Oct 23, 2026

Uh oh!

pullfrog Bot left a comment

Uh oh!

pullfrog Bot commented Oct 23, 2026

Uh oh!

beaglemoo commented Oct 23, 2026

Uh oh!

pullfrog Bot left a comment

Uh oh!

pullfrog Bot left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		const response = await fetchWithProxy(url, { headers });
		storeSetCookies(response, jar);

		const domTitle = fallbackDomTitle({ htmlDom });
		return domTitle ?? undefined;

Conversation

beaglemoo commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

pullfrog Bot left a comment

Choose a reason for hiding this comment

ℹ️ The anonymous .json premise may already be overtaken by Reddit

Uh oh!

pullfrog Bot commented Oct 23, 2026

Uh oh!

beaglemoo commented Oct 23, 2026

Uh oh!

pullfrog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pullfrog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

beaglemoo commented Oct 23, 2026 •

edited

Loading

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

ℹ️ The anonymous `.json` premise may already be overtaken by Reddit