Add an option to spread indexing pipelines over time - #6655
Conversation
927f22c to
2262d4a
Compare
loutPhilipps
changed the title
| let indexing_cycle_opt: Option<IndexingCycle> = | ||
| (spread_indexing_pipelines || cooperative_indexing_permits_opt.is_some()).then(|| { | ||
| IndexingCycle::new( | ||
| &pipeline_id, | ||
| indexing_settings.commit_timeout(), | ||
| cooperative_indexing_permits, | ||
| cooperative_indexing_permits_opt, | ||
| ) | ||
| }); |
There was a problem hiding this comment.
This is where we enable the IndexingCycle, either if spreading pipelines or if cooperative indexing is enabled
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 927f22cfca
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
eb67cf2 to
815d873
Compare
815d873 to
e84a525
Compare
| let num_pipelines = 100; | ||
| let num_steps = 15; | ||
| let semaphore = Arc::new(Semaphore::new(num_threads)); |
There was a problem hiding this comment.
My two cents is that having the semaphore in the reconcile test doesn't change anything to the test so I decided to test the general case instead. I can add another test for nudging to phase with cooperative indexing if needed.
1 participant
Tip
This PR is better reviewed commit-by-commit.
Problem
An indexer node's pipelines are started at the same time with the same commit timeout, leading to them committing at the same time. This is not an efficient usage of resources when there are many pipelines on a single indexer node.
This problem was addressed in #4877 with the cooperative indexing feature by introducing both a time spread of pipelines, as well as a semaphore to prevent running too many pipelines concurrently.
When having a large index though, we don't really want to have a semaphore, but do want to spread indexing pipelines.
What this PR does
Add an option
enable_spread_indexing_pipelineson the indexer's config. For backwards compatibility reasons, enabling cooperative indexing also enables this feature, with the addition of a semaphore.The implementation of
CooperativeIndexingCyclehas been mostly left intact, with theindexing_permitsbecoming optional.