Running A/B Tests for In-App Messages

A/B testing for In-App messages requires special consideration due to how they are delivered to mobile devices. Unlike email or push notifications, In-App messages with trigger rules are cached locally on the device only when the user meets predefined targeting conditions.

This caching behavior means audience segmentation must be resolved before the trigger event occurs. If segmentation is determined too late, the message may not be cached and won’t appear as expected.

Pre-Segmentation Settings

Build an automation workflow that randomly tags users as A or B.

1. Create an additional contact field to store the group’s value.
2. Build the workflow:
   • Add the **Split ** block
   • Connect each branch to the **Update custom fields ** blocks
   • In the Update custom fields blocks’ settings, specify the created field (e.g., A/B test) and values to update them (e.g., A/B)

You can trigger this workflow using the ApplicationInstalled, SubscriptionStarted, or another early-stage event. The matching In-App message will be cached once the group is assigned and the user meets the segment condition.

Triggering Settings

1. Create target dynamic segments and specify test groups in their conditions.

2. Create two messages:
   • Variant A → targeted at users in group A
   • Variant B → targeted at users in group B

Each message will be cached only for users who meet the segment condition in advance.

📘

Check out the guide on managing In-App campaigns to learn more about setting up triggering rules for various business goals

Troubleshooting: Button Tap Inside an In-App Message Doesn't Trigger the Action

If the In-App message itself displays correctly, but tapping a button inside it (for example, a button meant to open an external link) doesn't do anything, check what's on screen at the moment of the tap. If the app is currently showing one of its own screens or an internal dialog specific to that screen — rather than the app's home/main screen — the tap's action can fail to fire, even though the In-App message was visibly shown.

Recommendation: if you see this on a specific screen, avoid triggering the In-App message while that screen is active — for example, by excluding that screen from the trigger conditions, or adding a short delay so the message fires only once the user returns to the app's home/main screen.