How Agency Shared Monthly Crawl Pages, website allocations, unlimited scans, and page add-ons work.
Agency Shared Crawl Pool
The Agency plan gives you unlimited domains, unlimited pageviews, and unlimited manual & scheduled scans, with one Shared Monthly Crawl Pool of 20,000 pages. Expand the pool anytime by buying +10,000 Monthly Crawl Page packs through Stripe Checkout.
Pageviews vs Crawl Pages
| Term | Meaning |
|---|---|
| Pageviews | Visitor traffic recorded by the live CMP script. Agency includes unlimited pageviews. |
| Crawl Pages | Pages CookieShift’s scanner may visit during a crawl. Agency uses a shared allocation pool — pages are reserved per website, not spent every time you scan. |
Shared Crawl Pool
Every Agency workspace has one pool:
- Base: 20,000 Shared Monthly Crawl Pages( Allocate Crawl )
- Add-ons: +10,000 Monthly Crawl Pages each (stackable: 30k → 40k → 50k …)
- The pool is an allocation envelope — running another scan does not deduct another block of pages from the pool
Website Allocation
Each website has a Monthly Crawl Allocation. That number is the maximum pages CookieShift may crawl on that website during any scan.
Think of allocation as reserved crawl capacity for that site — not a one-time spend.
Example:
| Website | Allocation |
|---|---|
| Client A | 2,000 |
| Client B | 5,000 |
| Client C | 8,000 |
| Remaining pool | 5,000 |
Total allocated (15,000) must never exceed the shared pool (20,000).
Allocation can be higher than pages discovered
You may allocate more than the current discovered page count (for example, discover 700 and allocate 1,500). That reserves room for the site to grow. The only hard limit is the remaining shared pool.
Unlimited scans
You may run Quick Scan, Full Scan, Manual Scan, and Scheduled Scan as often as you want.
Every scan respects that website’s allocation:
| Website allocation | Site size | Each scan crawls |
|---|---|---|
| 2,000 | 1,750 pages | Up to 1,750 (fully covered) |
| 2,000 | 4,800 pages | Up to 2,000 — Crawled 2,000 of 4,800 discovered pages |
| 1,500 | 700 pages | Up to 700 now; later scans can use more as the site grows, up to 1,500 |
Running another scan reuses the same allocation. It does not take another 1,500 (or 2,000) pages out of the shared pool.
Partial crawls do not fail. Increase the website’s allocation anytime for a deeper crawl.
Coverage
| Status | Meaning |
|---|---|
| Fully Covered | Allocation covers the pages discovered today |
| Partial Coverage | Allocation is lower than pages discovered — scans stop at the allocation |
| Reserved for growth | Allocation is higher than pages discovered — extra capacity for future pages |
| Needs allocation | Allocation is 0 — reserve pages when ready (scans are not hard-blocked) |
Coverage % ≈ allocation ÷ pages discovered (capped at 100%). Values above 100% mean you reserved extra capacity for growth.
New website onboarding (Agency)
When you add a website on the Agency plan:
- Enter the website URL and connect the property
- CookieShift runs URL discovery immediately (homepage, menus, footer, sitemaps, robots.txt, internal links, pagination) — the same discovery engine the Scanner uses. No monthly allocation is required for this step.
- Review Pages discovered, Discovery confidence, Discovery sources, and Estimated crawlable pages
- Choose how to reserve crawl capacity:
- Allocate all discovered pages
- Choose custom allocation (editable number / slider / + / − — may be more or less than discovered, up to the remaining pool)
- Configure later (you can still scan; reserve pages anytime from Agency Management or Billing)
- Finish setup → open the Scanner
Non-Agency plans keep the simpler flow: Add Website → Connect → Scanner (no allocation step).
Scanning is never blocked only because allocation is missing. The shared pool is only enforced when you save an allocation that would exceed remaining pages.
Allocation Manager
Open Billing → Usage, or go to Agency Management at /workspace/agency.
You will see:
- Shared Crawl Pool total / Allocated / Available
- Website table: Website · Allocation · Pages Discovered · Status
- Allocation editor to change a website’s reserved budget (remaining pool updates when you save)
Prevent overallocation
If Remaining Pool is 3,000 and you try to allocate 5,000, save is blocked:
You can reduce the allocation, move pages from another website, or buy a +10,000 Monthly Crawl Page pack (Stripe Checkout).
Page add-ons
| Current pool | Purchase | New pool |
|---|---|---|
| 20,000 | +10,000 | 30,000 |
| 30,000 | +10,000 | 40,000 |
Buying a pack opens Stripe Checkout. After payment succeeds, the shared pool grows immediately. Existing website allocations stay unchanged — new capacity is available to assign.
Multi-website example
Agency pool = 20,000
- Allocate 2,000 → Client A (small brochure site)
- Allocate 5,000 → Client B (ecommerce)
- Allocate 8,000 → Client C (content hub)
- Leave 5,000 unallocated for a new client next month
- Need more headroom? Buy +10,000 → pool becomes 30,000; existing allocations unchanged
Where to manage it
| Surface | What you do |
|---|---|
| Add Website (Agency) | Discover pages with the Scanner’s discovery engine, then set the first allocation |
| Billing → Allocation Manager | Edit all website allocations and buy page packs |
Agency Management (/workspace/agency) | Shared pool overview, website allocations, allocation editor, usage, add-ons, and billing summary |
| Dashboard | Agency Overview widget (Agency plan) |
| Website settings / API | PATCH /api/websites/:siteId with monthlyAllocation |