Consider creating a trusted list of IPs for fine tuned SSR (to filter out aggressive bots)

Is your feature request related to a problem? Please describe.

In order to provide better protections against aggressive bots, DSpace should consider adding a "trusted IPs" configuration, to allow trusted bots easier harvesting access to DSpace (for SEO or similar) and also limit SSR to only "on campus" or other similar IP ranges.

What special behaviors would trusted bots possibly have?

Trusted bots/IPs would always have SSR enabled (to allow for harvesting). This would also allow sites to potentially disable SSR site-wide, blocking out any un-trusted bots which cannot parse Javascript. Other known IPs (like "on campus" users) would also have SSR enabled.
(Optionally) Trusted bots/IPs might be treated differently by our ui > rateLimiter. Or, perhaps there's just a separate configuration for trusted bots in the rateLimiter, to allow sites to potentially loosen rules for trusted bots and tighten rules elsewhere.
Trusted bots would be known, verifiable bots which could also be configured to serve cached SSR content (see cache settings) if necessary for performance.

How would trusted bots be identified?

Trusted bots would need to be identified by IP address/range. Identifying bots by user-agent is no longer recommended practice because it's trivial to fake.

DSpace could populate an initial list of trusted bots from trusted sources such as:

https://www.gstatic.com/ipranges/goog.json -> IPs of all Googlebots
https://www.bing.com/toolbox/bingbot.json -> IPs of all Bing bots
https://duckduckgo.com/duckduckbot.json -> IPs of all DuckDuckGo bots
Any bot IPs which are auto-populated would also need to be auto-updated from the trusted sources on a regular basis, e.g. once per week. These lists won't change frequently, but IPs may be added/removed.

Individual DSpace sites should also have a configuration (or similar) which allows them to add/remove/update additional "trusted bots". This allows sites to give access to other trusted crawlers / harvesters, or maybe remove access if a previously-trusted bot becomes problematic.

How will this alleviate issues with aggressive bots?

By providing a way to identify the good/trusted bots, DSpace sites could then turn off SSR without any impact on SEO (because sites could configure SSR to only be used for trusted bots).

Turning off SSR helps to deter aggressive bots since many of those bots currently attempt to harvest/parse HTML. Even if they do parse Javascript, they'd do all that processing on the client-side, which would not overwhelm the node server where dspace-angular runs.

Describe the solution you'd like

While I don't have an exact design in mind, we may wish to consider if a list of "trusted IPs" is best kept on the backend or fronted.
* If kept on the frontend, it'd likely need to be a large JSON file which can be parsed at runtime (and rebuilt on demand from configs or similar).
* If kept on the backend, we'd have more flexibility of keeping it in the database or configuration (or a bit of both). However, we'd still need to pass the raw data to the frontend via JSON.

Rough Implementation Proposal

For simplicity, consider storing the list of trusted IP addresses as a JSON file, likely on the backend (as it will need a script to regularly update it, and most of our regular cronjob-style scripts are on the backend).

The JSON file we store locally should use the JSON-Based format for publishing IP Ranges of Automated HTTP Clients, as that's the same format used by all search engines.

Create a new set of configurations in dspace.cfg (or a similar *.cfg file) which is structured similar to this:

# Configure trusted IP sources, which are all defined below.
# Additional sources can be added by adding a new "webui.trusted-ipranges.source.*" config for each
webui.trusted-ipranges.sources = google, bing, duckduckgo, custom

# Google source is a JSON file which will be parsed
# (NOTE: All these JSON files are the same format for every search engine)
webui.trusted-ipranges.source.google = https://www.gstatic.com/ipranges/goog.json

# BingBot source
webui.trusted-ipranges.source.bing = https://www.bing.com/toolbox/bingbot.json

# DuckDuckGo source
webui.trusted-ipranges.source.duckduckgo = https://duckduckgo.com/duckduckbot.json

# Custom source is a list of IP addresses which may correspond to on-campus users or similar
webui.trusted-ipranges.source.custom = 192.168.0.0/24, 192.168.2.2

# Additional sources can be added by just adding new "webui.trusted-ipranges.source.*" configs
# The text after "source" allows you to label the meaning behind the IPs.
# webui.trusted-ipranges.source.custom2 = 192.168.5.5, 192.168.6.6

# Schedule for how frequently the trusted IP ranges should be regenerated via config
# This example shows it running at 2:15AM every Saturday
# When commented out, this feature would be disabled
# webui.trusted-ipranges.cron = 0 15 2 ? * SAT

The DSpace backend needs a script (e.g. ./dspace update-trusted-ipranges) which can loop through the webui.trusted-ipranges.sources and gather ALL IP ranges together into a single, large JSON file (named something like [dspace]/config/trusted-ipranges.json).
- This script should have a "cron" setting (see example above), allowing it to be run on a semi-regular basis, e.g. about once per week, because the IPs of search engines don't change frequently. The script should overwrite the existing JSON file each time it runs.
- Think of this script as similar to the generate-sitemaps script which auto-updates Sitemaps on a regular basis also using a "cron" configuration.

The single combined JSON file would use the standard JSON format used by all search engines, and look similar to below. Notice how the "service" in this JSON format would list the source of the IP address/range. That can allow the UI to potentially have different configurations for different services (see examples below).

{
   "creationTime": "2024-01-03T10:00:00.121331",
   "prefixes": [
     {
       "ipv4Prefix": "157.55.39.0/24",
       "service": "bing"
     },
     {
       "ipv4Prefix": "8.8.8.0/24",
       "service": "google"
     },
     {
       "ipv4Prefix": "104.43.54.127/32",
       "service": "duckduckgo"
     },
     {
       "ipv4Prefix": "192.168.0.0/24",
       "service": "custom"
     },
     ...
}

The backend will need to make the trusted-ipranges.json available to the frontend, perhaps proxied similar to how sitemap files generated on the backend are available in the frontend.
- This might mean making this JSON file visible publicly. However, if we determine that is a security or privacy risk we likely could limit its access only to the server where the UI is running, as this file primarily needs to be visible to the server.ts file in the UI.

The User Interface will need to be updated to request this trusted-ipranges.json and parse it to determine whether SSR is enabled. An additional SSR config will need to be added similar to this:

ssr: 
  # When this is enabled, only trusted IPs listed in `trusted-ipranges.json` will trigger SSR.
  enableTrustedIPRanges: true
  # (OPTIONAL) we could even make it configurable based on service type in "trusted-ipranges.json"
  # This example would only enable SSR for "google" and "custom" services in the "trusted-ipranges.json" file.
  enableTrustedIPRangeServices:
    - service: "google"
    - service: "custom"

(OPTIONAL) We optionally could update the "cache" settings to customize which cache is used for which service.

cache: 
  serverSide:
    # This example shows us configuring that the "google" and "bing" IPs should all use the "botCache"
    botCache:
      max: 1000
      trustedIPRangeServices:
        - service: "google"
        - service: "bing"
    # But, the "custom" IPs should use the "anonymousCache" instead
    anonymousCache:
      max: 100
      trustedIPRangeServices:
        - service: "custom"

Finally, obviously in the UI's server.ts, we'd need a script/library that can quickly determine if the current user IP matches any of those in the trusted-ipranges.json. If a match is found, then SSR is triggered (and caching is triggered, if enabled). If not, then it falls back to CSR only.

Describe alternatives or workarounds you've considered

DSpace doesn't currently have a solution that lines up with this idea:

DSpace can support only enabling SSR for some pages. But, that's a global config and doesn't allow you to change the settings per IP
DSpace can support rateLimiting as well, but again it's a global config. No way to have different settings for different IP ranges
DSpace also supports basic cached of SSR with a bot-specific configuration. But, that bot-specific configuration uses the isbot library which tries to identify bots based on user-agent (which is less reliable and may not work at all for highly aggressive bots)

Additional information

This issue is related to #4565 and comes out of discussions with the Google Scholar team on Sept 8, 2026.

I don't think managing the network or the traffic volume should be a concern of this software. Adding those, it will add more complexify to DSpace configurations and to also for troubleshooting issues.

Other solutions like Anubis could be considered.

The harsh reality is that waves of extremely high (and hard to manage) bot traffic have become the new normal, and that situation does not look like it will change any time soon. It is becoming harder and more resource- and cost-prohibitive to host DSpace and other repositories. Any approaches that might help folks keep their DSpace instances up and running should be considered, as they will likely be of use to someone, somewhere, under some conditions. Clearly, folks are already finding it necessary to turn off SSR to deal with this non-hypothetical problem, so this solution seems like it would be a valuable mitigator to minimize the downsides of this defense.

I think we are trying to circumvent a problem that is caused by DSpace's frontend needing too many resource. We should investigate in a central cache for dspace-angular and in a much smaller footprint regarding CPU and memory consumption. I know that this is not easy and we already tried to go this way. But, I don't like the idea to switch off SSR for most requests.

We introduced SSR for bots and for accessibility. The idea of Open Access is that content should be crawlable and indexable (compare the declaration of the budapest open access initiative from 2002). If SSR is to intensive for servers, we should discuss if there is a way to pregenerate some of it's contents or the whole pages and store them on disk. We need to come to the point, that we can serve html without using so many resources as we currently do.

The kernel.org maintainers stated in a blog post:

In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity.

source: https://people.kernel.org/monsieuricon/creepy-crawlies

To me, it means we should look which features we have for unauthenticated requests, for example on community, collection or item views, but we should not switch of SSR if it really is still needed for accessibility and for bots.

While I hate the trouble bots make, I think it is important to do the best that we can to provide our contents (see my previous comment -> Open Access).

If SSR is to intensive for servers, we should discuss if there is a way to pregenerate some of it's contents or the whole pages and store them on disk.

Because we host DSpace instalations, In order to display specific and static content for each repository, like in the image:

we've created an Angular component used to load and display static HTML content dynamically, typically news content on the home page. It behavior is to Download the file as plain text using Angular’s HttpClient and the downloaded HTML is injected into the template element.

this.divBinder.nativeElement.innerHTML = html;

that HTML request is cached (outside DSpace).

I think there are really two separate questions here:

Should repositories remain openly accessible to humans and machines? Yes.
Should repository operators have the option to avoid spending SSR resources on every unknown automated client? I think they should.

I would support implementing this as a configurable feature.

One possible policy would be to provide SSR to verified/trusted crawlers that benefit from it, while allowing other automated traffic to fall back to CSR. Google Scholar is an obvious example where SSR matters and the crawler can be verified.

Technically, the benefit is straightforward: an SSR cache miss requires the Node process to execute Angular and render the page on behalf of the requester. With CSR, that rendering work moves to the client. REST/API, database and bandwidth costs remain, so this is not a complete bot mitigation mechanism, but it removes one expensive part of the request path.

And this is not really about trying to hide content from determined crawlers. Modern crawlers that genuinely want the content can execute JavaScript using a headless browser and render the application themselves. The difference is that DSpace no longer has to spend its own SSR resources doing that work for every unknown requester.

This would not replace better caching, rate limiting, or further SSR performance work. It would simply give repository operators another useful control when automated traffic is causing operational problems.

There are trade-offs, including legitimate but unknown crawlers receiving CSR instead of SSR, which is why I think this makes most sense as an optional, configurable feature.

@kshepherd

I still like the idea of static generated pages, that @kshepherd suggested in #3183. I see a huge potential in having pre-generated static pages for unauthorized users and dynamic pages for authorized users.

Can we take any additional ideas from the lessons learned in this paper? 👉 https://journal.code4lib.org/articles/18489

@tdonohue

@tdonohue , In my view, DSpace currently has an untapped opportunity that could noticeably improve performance and, in particular, reduce the number of requests to the database and Solr (and therefore the load on RAM and the file system) caused by bots.

Unlike the SSR mechanism (in the DSpace frontend), we do not currently cache REST API requests. API requests to item endpoints, especially for in_archive items, often return unchanged data over long periods and do not need to be recalculated by the backend each time.

Combined with limiting browsing and search options for unauthenticated users, caching REST API requests could significantly reduce the load on the DSpace backend.

In this context, we should investigate DSpace issue #9552 further, as it may negatively affect REST API request caching.

I’d like to raise two more points that illustrate limitations of this "whitelist approach":

We use Cloudflare in front of our DSpace frontend, https://openhsu.de. All requests, except those originating from the backend, pass through Cloudflare. This means the frontend only sees Cloudflare’s IP addresses, so at that point I can no longer distinguish between trusted and less trusted IP addresses. This issue is likely more general and not specific to Cloudflare.
The SEOHealthIndicator can only work correctly if the public IP address used when the backend accesses dspace.ui.url is also added as a trusted IP address to the whitelist. If this is overlooked, the error I described in SEOHealthIndicator does not robustly handle failures while fetching or evaluating UI URLs (SSR enabled/disabled, robots.txt enabled/disabled) DSpace#13139 occurs.

Regarding the list of URL path patterns to exclude from SSR, I have further consolidated the exclude patterns in PR #6279.

Since none of the remaining exclude patterns actually needs regex features anymore, I ask the community whether regex matching is still needed for the SSR exclusion mechanism, or whether a simple prefix check (path.startsWith(prefix)) would be sufficient.

@saschaszott

@saschaszott

We use Cloudflare in front of our DSpace frontend, https://openhsu.de/. All requests, except those originating from the backend, pass through Cloudflare. This means the frontend only sees Cloudflare’s IP addresses, so at that point I can no longer distinguish between trusted and less trusted IP addresses. This issue is likely more general and not specific to Cloudflare.

Do you use mod_remoteip and Cloudflares CF-Connecting-IP header to restore the origin IP's ? This way we are able to distinquish original IPs on the server side.

Instructions from here:
https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/

@phahl-sysadm

@phahl-sysadm , thanks for your note. I'm not sure whether we can change this on our side since our Cloudflare configuration is managed by central IT. We use Cloudflare's Website and Web Application Protection feature: https://www.cloudflare.com/products/ddos/

In nginx I use this to restore the origin IP, /etc/nginx/conf.d/cloudflare-realip.conf:

# Use client's real IP for logging instead of Cloudflare's
# Last update: 2026-08-10
#
# See: https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2a06:98c0::/29;
set_real_ip_from 2c0f:f248::/32;
real_ip_header CF-Connecting-IP;

There is a similar configuration for Apache httpd, but I do not use it there as we are Cloudflare → Amazon ALB → DSpace so I only need to trust Amazon ALB.

@tdonohue

As I mentioned during the developer's meeting @tdonohue
My colleague is preparing a PR with redis caching - he said it should be done soon, hopefully next week.
The way it will work is it will allow the user to define paths that should be cached and for how long - then of course if the user will request the website that exists in redis, that version will be returned. This will likely require thinking about which pages to cache - so we will try to provide some recommendations if the PR goes through.

Regarding the question of whether we should server-side render (SSR) collection and community pages. I think the information gain for search engines would be limited, since collections and communities generally have very little descriptive metadata in practice.

Also, the 10 items shown on a collection or community page are somewhat arbitrary, as they're sorted by "relevance". All other associated items are "hidden" on subsequent pages. However, each item's landing page is accessible through the sitemap, so search engines don't need to reach them via the collection or community page. Therefore, excluding collections and communities from SSR would result in little (no) information loss for search engines.

@saschaszott

@saschaszott : I still believe community/collections should be indexed by search engines because many sites use their names & descriptions as important information. See my comment on your PR here for more information: #6279 (comment) Basically, excluding them from search engines may make some content less findable in DSpace sites.

That said, I think all that the search engines need to index for Communities & Collections are the title/description. The list of items can be hidden from the SSR results, and in fact it already is excluded from SSR results (check SSR of https://demo.dspace.org/communities/c964546c-f8ec-4386-aa2a-6939843a0808 for example and you'll see that the items are not included in SSR and are only loaded once you switch to CSR).

I'd recommend moving this discussion to the related PR as it's not directly related to this ticket.

To reduce the SSR load from community and collection search I opened a PR that excludes those paths: #6344

Maybe someone in this discussion thread would like to review and test it?

I've made a couple of SSR optimizations (not sure if these help to reduce the aggressive bot problems): #6352, #6347

Unfortunately, Web Bot Auth is not yet an official web standard, but it could provide a robust solution to the problem described here (and avoid reliance on IP addresses and User-Agent headers): https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/

There is an increase in agent traffic on the Internet. Many agents choose to identify their traffic today via IP Address lists and/or unique User-Agents. This is often done to demonstrate trust and safety claims, support allowlisting/denylisting the traffic in a granular manor, and enable sites to monitor and rate limit per agent operator. However, these mechanisms have drawbacks: …

Instead than hardcode rules, matches against configurable list of IP, in dspace to disable SSR I would suggest to implement a mechanism that can driven by the network layer (cdn, reverse proxy, etc) in front of dspace. It could be a custom header that if set would disable SSR (i.e. X-SSR: false)

18. @phahl-sysadm , thanks for your note. I'm not sure whether we can change this on our side since our Cloudflare configuration is managed by central IT. We use Cloudflare's Website and Web Application Protection feature: https://www.cloudflare.com/products/ddos/

Hi! No Dspace specific settings are needed on Cloudflare side. Changes you need, can be done using apache/Nginx on your server side.

If you are using Apache, enable mod_remoteip and just add the following into your remoteip.conf file

# Declare the header field which should be parsed for useragent IP addresses
RemoteIPHeader CF-Connecting-IP

# Declare client intranet IP addresses trusted to present
# the RemoteIPHeader value
RemoteIPInternalProxy  XXXXX 
RemoteIPInternalProxy  XXXXX

and so on. Replace XXXX with Cloudflare IP's.

Updated list of Cloudflare IP's, you can get from the link: https://www.cloudflare.com/ips/

Deny / Allow lists are something that all of our services need. That suggests factoring it out of DSpace altogether, into the webserver or further out into network infrastructure.

Request rate limiting would be useful for pushing the load back until capacity is available. I've noted elsewhere that dspace-angular behaves poorly when faced with both 429s and timeouts. We should make DSpace more resilient to these problems.

Navigation Menu

Repository navigation

Consider creating a trusted list of IPs for fine tuned SSR (to filter out aggressive bots) #6212

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Rough Implementation Proposal

Describe alternatives or workarounds you've considered

Additional information

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Consider creating a trusted list of IPs for fine tuned SSR (to filter out aggressive bots) #6212

Description

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Rough Implementation Proposal

Describe alternatives or workarounds you've considered

Additional information

Activity

paulo-graca commented on Sep 10, 2026

dheles commented on Sep 10, 2026

pnbecker commented on Sep 10, 2026

pnbecker commented on Sep 10, 2026

paulo-graca commented on Sep 10, 2026

vidiecan commented on Sep 14, 2026

pnbecker commented on Sep 15, 2026

saschaszott commented on Sep 15, 2026

saschaszott commented on Sep 15, 2026

23 remaining items

saschaszott commented on Sep 29, 2026

saschaszott commented on Sep 29, 2026

phahl-sysadm commented on Oct 8, 2026

saschaszott commented on Oct 8, 2026

alanorth commented on Oct 8, 2026

Dawnkai commented on Oct 8, 2026

saschaszott commented on Oct 8, 2026

tdonohue commented on Oct 8, 2026

tinsch commented on Oct 13, 2026

saschaszott commented on Oct 13, 2026

saschaszott commented on Oct 13, 2026

abollini commented on Oct 15, 2026

phahl-sysadm commented on Oct 16, 2026

mwoodiupui commented on Oct 29, 2026

mwoodiupui commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions