How to Measure the Performance of Batch AI Predict Scores?

Learn how to measure the real impact of Batch AI Predict scores using two test methods you can run directly in Batch. No external tooling required.

Introduction

Batch AI Predict scores are designed to target the right profiles, at the right time, with the right message. This guide covers two testing methods you can set up directly in Batch to quantify that impact:

Targeting Comparison Test (A/B/C): use this when you want to compare segmentation strategies (Batch AI Predict targeting vs. your existing business rules).
Content Personalization Test (A/B): use this when you want to compare message content (a message personalized using a predictive score vs. a generic version).

Both tests are self-contained and can be run without any external tooling. Which test should you run?

Use case

Recommended test

Proving that AI targeting outperforms manual rules

Targeting Comparison Test

Proving that personalized content generates more conversions

Content Personalization Test

Scores used: High Potential Client, Future LTV, Top Clients at Risk

Targeting Comparison Test

Scores used: Product Recommendation, Promotion Sensitivity

Content Personalization Test

Method 1 : Targeting Comparison Test

What it measures

This test answers the question: does Batch AI Predict identify better-performing profiles than your existing targeting logic?

Instead of simply replacing your current method with a predictive score, you run both targeting approaches in parallel (on the same campaign, at the same time) and compare results across three distinct groups:

Group A (Batch AI Predict only): profiles selected by the predictive score, but not by your business rules.
Group B (Business rules only): profiles selected by your existing logic, but not by the predictive score.
Group C (Common): profiles selected by both methods simultaneously.

Group C acts as a natural control reference: since both methods agree on these profiles, any performance difference between Groups A and B reflects the quality of the targeting logic, not the campaign itself.

Prerequisites

Before setting up this test, make sure you have:

At least one Batch AI Predict score activated and computed for your project.
A campaign you run regularly (newsletter, promotional send, retention push, etc.) for which you currently apply a targeting rule (recency, RFM, manual segment, etc.).
Access to Segments in your Batch dashboard.

Step-by-step setup

Step 1 : Create Segment A (Batch AI Predict targeting)

Go to Profiles > Segments and create a new segment. Apply a filter based on your Batch AI Predict score, for example :

Predictive score [Score name] is in top X%
Or use a threshold: Predictive score [Score name] ≥ 0.6

Name this segment clearly, for example: [Test] Batch AI Predict - Segment A.

Step 2 : Create Segment B (Business rules targeting)

Create a second segment that replicates your current targeting logic. For example:

Last purchase date is within the last 60 days
Or: Loyalty tier = Gold

Match the volume of Segment A as closely as possible (this is critical for a valid comparison). Name it: [Test] Business rules - Segment B.

Step 3 : Set up the three campaigns

Create three separate campaigns, each targeting one group. Use the same message content across all three (copy-paste the same template).

Group C (profiles present in both segments) cannot be isolated as a standalone segment; it is captured through cross-exclusions at the campaign level:

Campaign A: target Segment A, with Segment B added as an exclusion. This ensures only profiles selected by the predictive score but not by the business rules are included.

Campaign B: target Segment B, with Segment A added as an exclusion. This ensures only profiles selected by the business rules but not by the predictive score are included.
Campaign C: target Segment A and Segment B simultaneously (both included, no exclusion). This captures the profiles selected by both methods at the same time.

Critical conditions for a valid test:

All three campaigns must use the exact same message, sent on the same day, at the same time.
Do not apply different pressure rules to each campaign.

Important: Segments A and B must be as close in volume as possible. A significant size difference between the two groups would skew the comparison and invalidate the results.

Tip: The exclusion option is available in the targeting step of any campaign or automation. See the Targeting documentation for more details.

Step 4 : Set up a Conversion Goal

Before launching, configure a Conversion Goal on your campaign. This allows Batch to track business outcomes (purchases, subscriptions, etc.) for each group, not just engagement metrics.

In the campaign settings, define:

The conversion event (e.g., purchase, subscription_renewed).
The attribution window (check the default configured for your project).

How to read the results

Once your attribution window has elapsed, go to Analytics > Orchestration Analytics for the campaign. The attribution window duration is the one you configured in Step 4; wait for it to fully expire before reading results to ensure all conversions have been captured.

Compare Groups A and B on the following KPIs:

KPI

What it tells you

Click rate / Open rate

Whether Batch AI Predict targets more engaged profiles

Conversion rate

Whether targeted profiles actually convert

RPM (Revenue Per Mille)

The revenue generated per 1,000 messages sent (comparable even at different volumes)

Unsubscribe rate

Whether the targeting negatively impacts your audience

Tip: Focus your analysis on the comparison between Group A and Group B exclusively. Group C validates overlap but does not contribute to the core comparison.

Important: Do not compare this test campaign with previous campaigns. Differences in seasonality, template, or send day would invalidate the results. The only valid comparison is Group A vs. Group B within the same send.