This policy group functions uniquely in WAAP. Instead of challenging or blocking incoming traffic like other policy groups, it's designed to recognize and allow traffic from legitimate automated services, such as search engine crawlers, AI agents, social media preview fetchers, monitoring tools, and payment processors.
These services access a domain to gather information and are generally beneficial or necessary for normal web activity. For each bot, configure how WAAP treats its traffic once the bot has been recognized and verified.
ℹ️
This policy group is available in the Pro and Enterprise plans.
Bot categories
Known bots are grouped into intent-based categories to clarify each bot's purpose and help with policy decisions. Each bot belongs to exactly one category, with policies configured at the individual bot level.
Category
Description
AI Search
Bots indexing public content for AI-powered search and answers.
AI Training
Crawlers collecting public web data to train AI models.
User-triggered AI
Agents that fetch or browse on behalf of a specific user request.
Search Engines
Traditional search engine crawlers indexing pages for organic search.
Preview Bots
Fetchers that generate link previews and metadata for social platforms.
SEO and Data Crawlers
Third-party crawlers for SEO analysis, audience measurement, and content feeds.
Monitoring
Automated checks for uptime, performance, and infrastructure health.
Security Scanners
Crawlers performing security audits, certificate validation, and plugin checks.
Integrations and Services
Webhooks, payment callbacks, and infrastructure services making automated requests.
Bot actions
Each known bot can be set to one of two actions:
Allow — the request bypasses all WAAP processing layers, including DDoS protection, behavioral analytics, OWASP checks, and other handlers. Recommended only for fully trusted bots.
Policy-based — the request is inspected, but the bot-detection checks that only apply to unknown automation are skipped (see below). Signature-based WAF and OWASP protections, IP-reputation threat signals, and custom rules still apply.
Some bots are set to Policy-based by default; switch a bot to Allow to fully permit its traffic. AI Search and AI Training bots default to Allow, while User-triggered AI bots default to Policy-based. To restrict a bot — to enforce data-protection requirements, block an unused service, or protect paid content from scraping — adjust the action for that bot.
Policy-based mode and other protections
Most bot-detection checks are designed to identify unknown automated traffic. Once a request is matched to a verified known bot, running those same checks again produces guaranteed false positives rather than added security. Cloud-hosted AI fetchers would trip Traffic from hosting services, search crawlers would trip Anti-scraping (indexing pages is scraping by definition), and any high-volume crawler would trip Traffic anomaly.
For this reason, when a request is matched to a known bot set to Policy-based, the following checks are excluded — they do not evaluate, do not generate block events, do not increment detection counters, and do not appear as triggered in the event log:
These protections remain fully active for known-bot traffic:
Signature-based WAF inspection and OWASP protections
IP-reputation threat signals
Custom rules — a custom rule that targets a known-bot pattern is never suppressed
ℹ️
Exclusions apply only to traffic that passes verification. If a request carries a known-bot User-Agent but fails verification, it is not treated as a known bot and every check evaluates normally.
Bot verification
Identification does not rely on the User-Agent alone, which attackers can spoof. A request is treated as a known bot only when its User-Agent and its source match the identity published by the bot's vendor. Verification uses one or more of:
Official vendor IP ranges, refreshed regularly
Organization ownership of the source address
Signed requests, where the vendor supports them
Only traffic that passes verification is treated as a known bot and receives the handling described above. Traffic that fails verification is not classified as a known bot: it receives no exclusions and is inspected like any other request.
In the domain dropdown at the top of the page, select the needed domain.
Click the Known Bots tab to view bots grouped by category and adjust each bot's action.
Known Bots policies control how WAAP handles traffic from recognized automated services — search engine crawlers, AI agents, social media preview bots, monitoring tools, and payment processors. In the Customer Portal these bots are grouped into intent-based categories, but the API exposes them as a flat list of policies. Each bot can be set to Allow (bypass all WAAP processing) or Policy-based (inspected, with unknown-automation detection checks skipped).
ℹ️
An API token is required, along with the ID of a WAAP-protected domain and the Python or Go SDK installed for SDK examples. This policy group is available on the Pro and Enterprise WAAP plans.
The API returns all policies in the group. mode: true means Allow; mode: false means Policy-based.
ℹ️
mode: false (Policy-based) carries the detector-exclusion behavior described in the Customer Portal tab: unknown-automation detection checks are skipped for verified known-bot traffic, while signature-based WAF, OWASP protections, IP-reputation threat signals, and custom rules still apply.
Set bot mode
Sets a bot to Allow or Policy-based by specifying the target mode. Pass mode: true to set Allow and mode: false to set Policy-based. Use the View bot modes response to find the bot ID.
Parameter
Required
Description
bot_id
Yes
ID of the bot policy to configure.
domain_id
Yes
ID of the WAAP-protected domain.
mode
Yes
true for Allow, false for Policy-based.
import gcore
import os
client = gcore.CDB()
domain_id = int(os.environ["WAAP_DOMAIN_ID"])
# Find the bot ID by name
rule_sets = client.waap.domains.list_rule_sets(domain_id)
bot_set = next(
rs for rs in rule_sets if rs.resource_slug == "common-automated-services"
)
rules = client.waap.domains.policies.list(domain_id, rule_set_id=bot_set.id)
bot = next(r for r in rules if r.name == "Google Bot")
result = client.waap.domains.policies.update(bot.id, domain_id=domain_id, mode=True)
print(f"Google Bot is now {'Allow' if result.mode else 'Policy-based'}")
package main
import (
"context"
"fmt"
"os"
"strconv"
gcore "github.com/G-Core/gcore-go"
"github.com/G-Core/gcore-go/waap"
)
func main() {
client := gcore.NewClient()
domainID, _ := strconv.ParseInt(os.Getenv("WAAP_DOMAIN_ID"), 10, 64)
// Find the bot ID by name
ruleSets := client.Waap.Domains.ListRuleSetsAutoPaging(context.Background(), domainID, waap.DomainListRuleSetsParams{})
var botID string
for ruleSets.Next() {
rs := ruleSets.Current()
if rs.ResourceSlug == "common-automated-services" {
policies := client.Waap.Domains.Policies.ListAutoPaging(context.Background(), domainID, waap.DomainPolicyListParams{RuleSetID: gcore.Int(rs.ID)})
for policies.Next() {
bot := policies.Current()
if bot.Name == "Google Bot" {
botID = bot.ID
}
}
}
}
result, _ := client.Waap.Domains.Policies.Update(context.Background(), botID, waap.DomainPolicyUpdateParams{DomainID: domainID, WaapDomainPolicySettings: waap.WaapDomainPolicySettingsParam{Mode: true}})
mode := "Policy-based"
if result.Mode {
mode = "Allow"
}
fmt.Printf("Google Bot is now %s\n", mode)
}
# Set BOT_POLICY_ID to the bot ID from the View bot modes response
export BOT_POLICY_ID="{BOT_POLICY_ID}"
curl -X PATCH "https://api.cdb-staging.cdn.orange.com/waap/v1/domains/${WAAP_DOMAIN_ID}/policies/${BOT_POLICY_ID}" \
-H "Authorization: APIKey ${GCORE_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"mode": true}'
Response:
{"mode": true}
The API returns the updated mode. mode: true means Allow; mode: false means Policy-based.