Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/examples/stop-crawler.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
id: stop-crawler
title: Stopping the crawler early
---

import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import StopCrawlerSource from '!!raw-loader!roa-loader!./stop-crawler.ts';

Calling <ApiLink to="basic-crawler/class/BasicCrawler#stop">`crawler.stop()`</ApiLink> ends the run gracefully: no further requests are dispatched, the ones already being processed finish, and `crawler.run()` resolves. Requests still waiting in the queue are left there untouched, so a later `run()` picks up where this one left off. This is the way to stop once a result limit is reached, without draining the queue or marking the remaining requests as handled.

To abort immediately without waiting for in-flight requests, use <ApiLink to="basic-crawler/class/BasicCrawler#teardown">`crawler.teardown()`</ApiLink> instead.

<RunnableCodeBlock className="language-js" type="cheerio">
{StopCrawlerSource}
</RunnableCodeBlock>
30 changes: 30 additions & 0 deletions docs/examples/stop-crawler.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
async requestHandler({ request, $, enqueueLinks, pushData, useState }) {
const state = await useState({ found: 0, limit: 10 });

const price = $('.price').first().text();
if (price) {
await pushData({ url: request.url, price });
state.found++;
}

// Once we have what we came for, stop dispatching new requests. Requests already in
// flight finish normally and `crawler.run()` resolves; whatever is still queued stays queued.
if (state.found >= state.limit) {
crawler.stop('Collected enough prices');
return;
}

await enqueueLinks();
},
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let's maybe show how to resume the crawler as well? (what do I pass in run() the second time?)


// Calling `run()` again with no arguments continues with the requests left in the queue.
// Here we raise the limit first, otherwise the first request would stop the crawler again.
const state = await crawler.useState({ found: 0, limit: 10 });
state.limit = 20;
await crawler.run();
19 changes: 19 additions & 0 deletions docs/guides/proxy-management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,25 @@ The `newUrlFunction` takes no arguments and returns a string with the proxy URL

Our crawlers will now use the selected proxies for all connections.

### Proxies in browser crawlers

A browser is launched with a single proxy, so in `PlaywrightCrawler` and `PuppeteerCrawler` the session's proxy only applies to a page if the browser it lands in was launched with the same one. By default, the pool opens the page in any browser with free capacity, which can ignore the session's proxy. Two `launchContext` options fix that:

- `useIncognitoPages: true` - every page gets its own browser context with the session's proxy applied to it. Pages share no cookies or cache with each other; the session's cookies are still carried over.
- `browserPerProxy: true` - the pool only reuses a browser that was launched with the page's proxy, and launches a new one otherwise. Useful when you need a shared context per proxy, at the cost of more browser instances.

```js
import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
proxyConfiguration,
launchContext: {
browserPerProxy: true,
},
// ...
});
```

## IP Rotation and session management

Each call to <ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
Expand Down
13 changes: 13 additions & 0 deletions docs/guides/result-storage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ Crawlee has several result storage types that are useful for specific tasks. The

By default, Crawlee storage is managed by the <ApiLink to="fs-storage/class/FileSystemStorageBackend">`FileSystemStorageBackend`</ApiLink> class, which stores all information as local files in the respective storage type folders.

To keep everything in memory and never touch the disk - handy in tests or short-lived scripts - pass a <ApiLink to="core/class/MemoryStorageBackend">`MemoryStorageBackend`</ApiLink> instead. Both backends are exported from `crawlee`:

```js
import { CheerioCrawler, MemoryStorageBackend } from 'crawlee';

const crawler = new CheerioCrawler({
storageBackend: new MemoryStorageBackend(),
// ...
});
```

Setting `CRAWLEE_PERSIST_STORAGE=false` has the same effect globally, without touching the code.

## Key-value store

The key-value store is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.
Expand Down
Loading