Browse Source

Update maxScrapers to 2 (from 50) to slow down scraping and reduce CPU and Disk I/O

master
James Mills 1 month ago
parent
commit
19127bf374
Signed by: prologic GPG Key ID: AC4C014F1440EBD6
  1. 2
      internal/crawl_task.go

2
internal/crawl_task.go

@ -13,7 +13,7 @@ import (
log "github.com/sirupsen/logrus"
)
const maxScrapers = 50
const maxScrapers = 2
type CrawlTask struct {
*BaseTask

Loading…
Cancel
Save