> For the complete documentation index, see [llms.txt](https://doc.batch.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.batch.com/guides-and-best-practices/batch-ai/how-to-measure-the-performance-of-batch-ai-predict-scores.md).
# How to Measure the Performance of Batch AI Predict Scores?
## Introduction
Batch AI Predict scores are designed to target the right profiles, at the right time, with the right message. This guide covers two testing methods you can set up directly in Batch to quantify that impact:
* **Targeting Comparison Test (A/B/C)**: use this when you want to compare segmentation strategies (Batch AI Predict targeting vs. your existing business rules).
* **Content Personalization Test (A/B)**: use this when you want to compare message content (a message personalized using a predictive score vs. a generic version).
Both tests are self-contained and can be run without any external tooling. **Which test should you run?**
| Use case | Recommended test |
| ------------------------------------------------------------------- | ---------------------------- |
| Proving that AI targeting outperforms manual rules | Targeting Comparison Test |
| Proving that personalized content generates more conversions | Content Personalization Test |
| Scores used: High Potential Client, Future LTV, Top Clients at Risk | Targeting Comparison Test |
| Scores used: Product Recommendation, Promotion Sensitivity | Content Personalization Test |
## Method 1 : Targeting Comparison Test
### What it measures
This test answers the question: **does Batch AI Predict identify better-performing profiles than your existing targeting logic?**
Instead of simply replacing your current method with a predictive score, you run both targeting approaches in parallel (on the same campaign, at the same time) and compare results across three distinct groups:
* **Group A (Batch AI Predict only):** profiles selected by the predictive score, but not by your business rules.
* **Group B (Business rules only):** profiles selected by your existing logic, but not by the predictive score.
* **Group C (Common):** profiles selected by both methods simultaneously.
Group C acts as a natural control reference: since both methods agree on these profiles, any performance difference between Groups A and B reflects the quality of the targeting logic, not the campaign itself.
### Prerequisites
Before setting up this test, make sure you have:
* At least one Batch AI Predict score activated and computed for your project.
* A campaign you run regularly (newsletter, promotional send, retention push, etc.) for which you currently apply a targeting rule (recency, RFM, manual segment, etc.).
* Access to [Segments](https://doc.batch.com/getting-started/features/customer-engagement-platform/profiles/segments) in your Batch dashboard.
### Step-by-step setup
#### Step 1 : Create Segment A (Batch AI Predict targeting)
Go to **Profiles > Segments** and create a new segment. Apply a filter based on your Batch AI Predict score, for example :
* `Predictive score [Score name] is in top X%`
* Or use a threshold: `Predictive score [Score name] ≥ 0.6`
Name this segment clearly, for example: `[Test] Batch AI Predict - Segment A`.
#### Step 2 : Create Segment B (Business rules targeting)
Create a second segment that replicates your current targeting logic. For example:
* `Last purchase date is within the last 60 days`
* Or: `Loyalty tier = Gold`
Match the volume of Segment A as closely as possible (this is critical for a valid comparison). Name it: `[Test] Business rules - Segment B`.
#### Step 3 : Set up the three campaigns
Create **three separate campaigns**, each targeting one group. Use the same message content across all three (copy-paste the same template).
Group C (profiles present in both segments) cannot be isolated as a standalone segment; it is captured through cross-exclusions at the campaign level:
* **Campaign A:** target Segment A, with Segment B added as an **exclusion**. This ensures only profiles selected by the predictive score but not by the business rules are included.
* **Campaign B:** target Segment B, with Segment A added as an **exclusion**. This ensures only profiles selected by the business rules but not by the predictive score are included.
* **Campaign C:** target Segment A and Segment B simultaneously (both included, no exclusion). This captures the profiles selected by both methods at the same time.
**Critical conditions for a valid test:**
* All three campaigns must use the **exact same message**, sent on the **same day**, at the **same time**.
* Do not apply different pressure rules to each campaign.
{% hint style="warning" %}
**Important**: Segments A and B must be as close in volume as possible. A significant size difference between the two groups would skew the comparison and invalidate the results.
{% endhint %}
{% hint style="info" %}
**Tip**: The exclusion option is available in the targeting step of any campaign or automation. See the [Targeting documentation](https://doc.batch.com/getting-started/features/customer-engagement-platform/orchestration/targeting) for more details.
{% endhint %}
#### Step 4 : Set up a Conversion Goal
Before launching, configure a [Conversion Goal](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/orchestration-analytics#conversion-goal-dedicated-insights) on your campaign. This allows Batch to track business outcomes (purchases, subscriptions, etc.) for each group, not just engagement metrics.
In the campaign settings, define:
* **The conversion event** (e.g., `purchase`, `subscription_renewed`).
* **The attribution window** (check the default configured for your project).
### How to read the results
Once your attribution window has elapsed, go to **Analytics > Orchestration Analytics** for the campaign. The attribution window duration is the one you configured in Step 4; wait for it to fully expire before reading results to ensure all conversions have been captured.
Compare Groups A and B on the following KPIs:
| KPI | What it tells you |
| --------------------------- | ------------------------------------------------------------------------------------ |
| **Click rate / Open rate** | Whether Batch AI Predict targets more engaged profiles |
| **Conversion rate** | Whether targeted profiles actually convert |
| **RPM** (Revenue Per Mille) | The revenue generated per 1,000 messages sent (comparable even at different volumes) |
| **Unsubscribe rate** | Whether the targeting negatively impacts your audience |
{% hint style="info" %}
**Tip**: Focus your analysis on the comparison between **Group A** and **Group B** exclusively. Group C validates overlap but does not contribute to the core comparison.
{% endhint %}
{% hint style="warning" %}
**Important**: Do not compare this test campaign with previous campaigns. Differences in seasonality, template, or send day would invalidate the results. The only valid comparison is Group A vs. Group B within the same send.
{% endhint %}
***
## Method 2 : Content Personalization Test
### What it measures
This test answers the question: **does personalizing message content using a Batch AI Predict score generate better results than sending a generic message?**
Rather than comparing two targeting strategies, this test compares two **messages** sent to the same audience:
* **Variant A (personalized)**: message content is dynamically adapted using a predictive score (for example, a product recommendation block calculated by the model, or a promotional offer adjusted to the profile's price sensitivity).
* **Variant B (generic)**: same structure, same targeting, but static content (for example, a bestseller list identical for all recipients, or a fixed discount for everyone).
This approach is particularly relevant for scores such as `Products Recommendation`, `Promotion Sensitivity`, or `Promotion Recommendation`.
### Prerequisites
Before setting up this test, make sure you have:
* A Batch AI Predict score that feeds personalization attributes into your profiles (e.g., recommended product IDs, optimal discount level).
* A campaign or automation for which you currently send the same message to all recipients.
* Familiarity with [A/B testing in Batch](https://doc.batch.com/getting-started/features/customer-engagement-platform/message/overview#a-b-test).
### Step-by-step setup
#### Step 1 : Enable A/B Testing on your campaign or automation
In the message composer, toggle on **"A/B Testing"** at the top of the screen.
Batch will automatically split your audience equally and randomly between variants. With two variants, each receives 50% of the targeted profiles.
{% hint style="info" %}
**Tip**: An **Automatic Winner Selection** feature is available, but **do not enable it for this test**: it would stop the test early once a statistically significant winner is detected, preventing you from collecting data over the full test period. See [A/B Test documentation](https://doc.batch.com/getting-started/features/customer-engagement-platform/message/overview#a-b-test) for details.
{% endhint %}
#### Step 2 : Compose Variant A (personalized)
In Variant A, use personalization attributes fed by your Batch AI Predict score. For example:
* A dynamic product block using `{{ profile.custom_attributes.recommended_product_1 }}`
* A conditional section showing a promotional offer only if `{{ profile.custom_attributes.promo_sensitivity }}` exceeds a defined threshold.
The goal is that each profile receives content that is specific to them (not a generic version).
#### Step 3 : Compose Variant B (existing logic)
In Variant B, compose the same message structure but replace personalized blocks with your existing targeting logic. This can be static content (e.g., a fixed discount for everyone) or rule-based content (e.g., the 3 last viewed products, wishlist items, top bestsellers). The key is that this variant does not use any Batch AI Predict attribute.
#### Step 4 : Set up a Conversion Goal
As with the segmentation test, configure a [Conversion Goal](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/orchestration-analytics#conversion-goal-dedicated-insights) before launching. This is essential to compare business impact, not only click rates.
#### Step 5 : Launch and let it run
Launch the campaign or automation. For a recurring automation, avoid modifying the variants after launch; changes after sending can bias the results and are only recommended to correct critical errors.
{% hint style="warning" %}
**Important**: Do not manually adjust the audience split between variants. The random, equal distribution is what makes the test statistically valid.
{% endhint %}
### How to read the results
Go to **Analytics > Orchestration Analytics** for your campaign or automation. The A/B test results table displays key metrics per variant.
| KPI | What it tells you |
| -------------------- | -------------------------------------------------------------------------------- |
| **Click rate** | Whether the personalized content drives more engagement |
| **Conversion rate** | Whether personalized profiles are more likely to take action |
| **RPM** | Revenue generated per 1,000 messages (the most comparable metric across volumes) |
| **Unsubscribe rate** | Whether personalization creates a negative effect on any segment |
The best-performing metric per variant is highlighted in green in the results table.
`[SCREENSHOT : A/B test results table in Orchestration Analytics showing Variant A vs. Variant B metrics]`
{% hint style="info" %}
**Tip**: Repeat the test across several sends before drawing conclusions. Sending a single campaign rarely produces statistically significant results, especially on smaller audiences.
{% endhint %}
{% hint style="info" %}
**Tip**: To measure only the winner's performance after declaring a winner, use the date filter in Analytics and set the start date to the day the winner was selected. See [Orchestration Analytics](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/orchestration-analytics) for details.
{% endhint %}
***
## Related documentation
* [Segments](https://doc.batch.com/getting-started/features/customer-engagement-platform/profiles/segments)
* [A/B Testing : Message Overview](https://doc.batch.com/getting-started/features/customer-engagement-platform/message/overview#a-b-test)
* [Orchestration Analytics](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/orchestration-analytics)
* [Conversion Goal dedicated insights](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/orchestration-analytics#conversion-goal-dedicated-insights)
* [Business metrics : Key metrics](https://doc.batch.com/getting-started/features/customer-engagement-platform/analytics/key-metrics#business-metrics)
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://doc.batch.com/guides-and-best-practices/batch-ai/how-to-measure-the-performance-of-batch-ai-predict-scores.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.