Bot detection false positives: count legitimate outcomes

Measure bot detection false positives by separating an incorrect automation label from an incorrect restriction on legitimate activity. Include customers, useful crawlers, accessibility use and authorized machine clients in the controls. Report unavailable assessments and challenge abandonment alongside classification rates so the test reflects what the application actually does.

A low error percentage is difficult to interpret without its denominator and traffic mix. A login flow and a public catalog may tolerate different responses. This guide explains how to compare bot detection products using outcomes that engineering and product teams can investigate.

Label the activity before reading the score

Write down what the test is trying to detect. Is the positive class any automation, unauthorized automation or a particular abuse such as repeated account creation? These are different labels.

A search crawler is automated but may be permitted to read a catalog. An accessibility tool may change interaction patterns while serving a legitimate person. An internal CI job may be an authorized machine client that cannot run the same browser verification as a customer.

Do not define every allowed request as human. That hides the distinction between identity, automation and authorization. The AI-agent guide explains why a verified software client can still need a route-specific permission decision.

Build a control set around actual users

Google publishes crawler verification methods. Use the supported identity check for the actual client rather than allowing every request with a familiar user-agent string.

Record browser, route and collection conditions with each session. If a detector’s required SDK was absent, distinguish that implementation gap from the result of a fully supported evaluation.

Count four different outcomes

A useful evaluation keeps the labeled condition, usable result, returned classification and application response in separate fields. That makes it possible to investigate why a legitimate request failed.

• Classification false-positive rate: labeled negative sessions wrongly classified positive, divided by negative sessions receiving a usable classification.
• Miss rate: labeled positive sessions wrongly classified negative, divided by positive sessions receiving a usable classification.
• Unavailable-result rate: attempts without a usable assessment, divided by attempted checks.
• Legitimate-action failure rate: legitimate actions incorrectly prevented or abandoned because of protection, divided by evaluated legitimate actions.

State how unknown results are excluded from classification denominators. Show their counts beside the rate so a product cannot look better simply by classifying fewer sessions. When several requests belong to the same session or device, retain that grouping rather than treating every repeated request as independent evidence.

Do not confuse a bot score with a universal risk scale

Provider outputs can use different directions and subjects. Cloudflare exposes bot-related fields within its own products. Google reCAPTCHA scores interaction risk. HUMAN computes request risk for its enforcement workflow.

A threshold copied from one provider’s scale can mean something different in another. Verify the documented score, field entitlement and action association before comparing response policies.

ShieldLabs adds risk context for visitors, users, devices and IP with Trusted, Suspicious or Dangerous verdicts and ready-made account-level abuse detection. Device/account relationships can add context beyond an address blocklist, but legitimate outcomes still need to be evaluated on the supported collection path.

Investigate the response, not just the label

A correctly detected browser agent may be performing an allowed action. A genuine customer can be incorrectly challenged after a network change. A completed challenge may still precede an abusive submission. In each case, a label alone does not explain the product outcome.

Check which response applied, whether the user could complete it and whether recovery worked. Monitoring, rate limits, challenge, review and refusal have different costs. A lower challenge count is not a complete success measure if the remaining challenges disproportionately stop a legitimate cohort.

Keep the exact provider settings and product versions in the evaluation record. If the traffic mix changes after rollout, compare the affected groups before calling the difference a detector regression.

Review a production change in stages

Start with the routes and conditions the application supports. Compare current outcomes with the proposed response, inspect legitimate failures and keep a rollback path. Account-enabled enterprise pilots should include the same customer and machine-client controls as a self-serve API pilot.

The checker directory helps explain interfaces, while the complete guide connects that first look to deployment. A useful final report states which automation is detected, which legitimate activity is affected, what remains unavailable and how customers recover.

FAQ

Sources

• Google crawler verification.
• Cloudflare Bots documentation.
• Google: interpreting website assessments.
• HUMAN detection overview.
• ShieldLabs bot detection.