Three levels of AI maturity in eCommerce: Why “close to the fire” experiments win

How often do you turn to AI when working on an email marketing strategy or preparing a promotional campaign? Let's be honest: it's no longer possible to imagine a marketer's work without the use of AI-powered tools. AI can write email copy, analyze customer behavior, generate product recommendations, and help marketers launch more experiments. But access to more AI tools does not automatically translate into better business results.

That was one of the central ideas of the “AI in eCommerce: Which technologies are worth investing in for 2027” panel at RARE 2026. Three experts approached the topic from different perspectives, but a common principle emerged: AI investments should start with a business problem, not a technology trend. 

For eCommerce and email marketing teams, that means choosing experiments that can influence meaningful KPIs, measuring both positive and negative effects, and building the processes needed to use AI reliably.

Let’s introduce the experts:

In this article, we discuss three levels of AI penetration in eСommerce and lessons to help you evaluate AI investments for 2027.

Key takeaways 

1. Email marketers should identify where AI can make a meaningful difference, then choose the level of automation that fits the task.
2. AI experiments should be tied to core business KPIs, with clear success criteria and guardrails to measure both positive results and potential harm. 
3. In 2027, the advantage will belong not to companies with the most AI tools, but to those that connect AI to business outcomes, measure its impact correctly, and know what not to automate. 

The three levels of AI maturity in eCommerce

AI adoption can range from using a chatbot for individual tasks to building systems that analyze business data and execute complex workflows. A useful way to understand this progression is to look at three levels: AI as a colleague, AI as an operating brain, and AI as a product or system.

Level 1: AI as a colleague

At this level, AI helps people complete individual tasks. An email marketer might use it to generate subject line options, draft campaign copy, summarize customer feedback, or translate content.

These applications are relatively easy to introduce because they fit into existing workflows. They can save time, but they do not necessarily change how the team operates.

For example, an AI assistant could prepare a weekly competitor analysis in Slack, summarize campaign results, or suggest several versions of a promotional email. The marketer still decides which insights matter, which copy fits the brand, and what action to take.

Level 2: AI as an operating brain

At this level, AI can access relevant business data, analyze it, and help answer questions that go beyond individual tasks.

Instead of asking AI to write a reactivation email, a marketer might ask why reactivation revenue has declined. The system could analyze campaign performance, customer behavior, and purchase history to identify potential causes and suggest areas for investigation.

The difference is context. AI becomes more useful when it can work with the data and business knowledge needed to address a specific question.

Giving a model access to more tools is not enough. It also needs relevant context, clear boundaries, and a well-defined objective. Without those foundations, even a powerful model may produce generic recommendations that do not reflect a company's customers, constraints, or business priorities.

Level 3: AI as a product or operating system

At the third level, AI becomes part of a system that can execute a process, evaluate results, and improve its next actions.

In email marketing, imagine a system that identifies a customer segment, proposes a campaign variation, runs an approved experiment, evaluates the outcome, and uses the findings to inform the next iteration. The team sets the goals and rules, while the system handles more of the execution.

This is a more ambitious level of adoption. It requires reliable data, clearly defined success criteria, and safeguards around the decisions AI can make.

AI maturity also does not have to be uniform across a company. A business might use advanced machine learning to select the best product offer for each customer while relying on AI only for administrative support in other departments. The appropriate level depends on the task, the available data, and the consequences of an incorrect decision.

Start AI experiments close to your core business KPI

Once you understand where AI might fit, the next question is what to test first.

A practical starting point is to choose experiments that are closely connected to the business outcome you want to improve. Rather than asking what AI can generate, ask where it can influence customer behavior and business performance.

For email marketers, this means looking beyond email volume or the time saved on copywriting. AI can help with tasks that have a more direct connection to conversion, retention, and revenue.

Potential starting points include:

• product recommendations: Test whether recommendations based on purchase history or browsing behavior increase conversion rate, average order value, or revenue per recipient;
• welcome emails: Explore whether content tailored to the signup source or customer interests improves first-purchase conversion;
• cross-sell and upsell campaigns: Test whether AI-selected products generate more incremental revenue without reducing margins;
• reactivation campaigns: Identify customer groups with distinct purchase patterns and test which messages or offers most effectively encourage them to return.

These are hypotheses, not guaranteed wins. The value comes from testing them against an appropriate control group and evaluating the results.

Research can help shape those tests. A 2025 study by Jean-Pierre Dubé and Ariel Xu, published in “Quantitative Marketing and Economics,” examined AI-generated email content for Wine Access through three randomized controlled trials. The researchers compared content written by salaried writers, generated by a large language model, and human-edited AI-generated content. In the first experiment, all three email approaches increased purchase incidence, revenue, and gross profit relative to a no-email control. 

However, the study did not find statistically significant differences in gross profit between the human-written and AI-assisted approaches across the experiments. The researchers' decision analysis favored an AI-based approach after accounting for labor and software costs. 

The lesson is that AI does not write better emails. Its business value depends on the alternative, the implementation cost, and the results you measure.

There is also an important counterpoint to starting with AI: first, check whether the core offer and customer experience are working. For example, changing a free-delivery threshold could have a substantial business impact without requiring a sophisticated AI solution. If an offer is uncompetitive or a campaign sends customers to an unavailable product, AI-generated copy may not solve the underlying problem.

Why 7,000 experiments don't guarantee better results

AI can make it easier to generate campaign variations and test more ideas. But running more experiments does not guarantee that a team will learn more or make better decisions.

A common mistake is to launch tests without a clear hypothesis, check results too early, or declare success based on a metric that does not reflect the intended business outcome. Reliable experimentation requires predefined acceptance criteria and a consistent way to interpret results.

For email marketing, this means defining what success looks like before launching a test.

Imagine that you are testing AI-powered product recommendations. More clicks may look promising, but they do not necessarily mean more profitable orders. Customers could click on lower-margin products, buy less expensive items, or return their purchases more often.

The primary metric should reflect the experiment's purpose, while guardrail metrics help reveal potential downsides.

In practice, use a measurement plan that includes:

1. A primary business metric. Choose the outcome the experiment is intended to improve, such as incremental revenue per recipient, repeat-purchase rate, or conversion rate.
2. A clear comparison. Use an appropriate control group to distinguish the effect of the intervention from other changes.
3. Guardrail metrics. Track possible negative effects, such as unsubscribe rate, average order value, gross margin, or customer complaints.
4. Predefined decision criteria. Decide in advance what result would justify adopting, adjusting, or stopping the experiment.
5. A record of what you learn. Document the hypothesis, audience, conditions, outcome, and limitations so the team can use the findings in future tests.

Reliability matters, too. A benchmark study, tau-bench, evaluated AI agents on simulated retail and airline customer service tasks. For the GPT-4o-based agent in a retail setting, the reported success rate was about 61% on a single attempt, while the probability of succeeding across all eight repeated attempts was below 25%. These results pertain to a specific benchmark and model generation, not to every AI tool available today. Still, they illustrate why a successful demo is not enough to prove that an agent can reliably handle a repeated business process.

Before delegating a task to AI, test it on representative cases, repeat the evaluation, and decide what level of error your business can tolerate. A tool that occasionally drafts an imperfect subject line may be useful with human review. A system that makes consequential pricing or customer-account decisions requires a different level of reliability and control.

Speed, scale, or performance: What should you invest in first?

AI can help teams move faster, produce more variations, and optimize results. But which benefit should take priority?

The answer depends on the bottleneck. If a workflow contains obvious time-consuming steps, efficiency may be the best starting point. If a team produces more campaigns or ads but struggles to generate profitable results, performance deserves attention first. If successful experiments are difficult to replicate, the priority may be to build a scalable process.

Speed alone can be misleading. Generating hundreds of advertising creatives is not valuable if customer acquisition costs and customer lifetime value do not support the business model. Automation can make an ineffective process faster without making it more profitable.

At the same time, scaling can become valuable once a team has a process worth repeating. AI agents can run sequences of tasks and iterate through workflows, potentially reducing the manual effort required to conduct and analyze experiments.

For email teams, the priority can be mapped to the problem:

 

Developing internal skills matters, too. Teams need to understand how to connect tools, define workflows, evaluate outputs, and improve the process over time. The objective is not to automate every task, but to build a system that produces better results with an appropriate level of effort and risk.

Before investing in AI to speed up email marketing, consider the time your team already spends on campaign production. If marketers spend hours coding emails or making routine design adjustments, they have less time to test new ideas, analyze results, and improve campaign performance.

AI can help generate ideas and analyze outcomes, while an efficient email production process helps teams put those ideas into practice faster. Together, they can make experimentation a more manageable part of everyday email marketing.

Build your AI strategy around what you won't do

With new AI features appearing across marketing platforms, the challenge shifts from finding something to try to deciding which options deserve attention.

AI labels can draw attention to a product, but they do not prove that its functionality is new or useful. Marketers need to distinguish essential capabilities from features that look impressive but do little to solve their actual problems.

Before investing in a new AI tool, ask practical questions:

1. Does it solve a problem we actually have?
2. Does it offer a meaningful improvement over our current workflow?
3. Can we measure its effect on a business outcome?
4. Does it integrate with our data, platforms, and approval processes?
5. Can we use it safely and reliably at the scale we need?
6. Is it better to buy an existing solution than to build our own?

Building a custom model or complex agent may make sense when AI is central to a company's product or when specific requirements justify the investment. For a common marketing task, an existing tool may be more practical.

The bigger challenge is deciding what not to do. More available options can make the strategy harder because teams can generate ideas and launch experiments faster than they can evaluate them.

For 2027, a useful AI strategy should define both priorities and boundaries. For example:

• don't adopt a tool solely because competitors are using it;
• don't scale an experiment before you understand its business impact;
• don't automate a decision simply because the technology can perform it;
• don't build a custom solution if an existing one meets your needs;
• don't use AI-generated content without the review or context required to protect quality and brand consistency.

Technical skills can help marketers make better choices, too. Understanding how tools connect, experimenting with lightweight scripts, and learning the basics of Python or GitHub can help teams build practical workflows. Not every marketer needs to become a developer, but greater technical literacy can make it easier to evaluate what is possible and what is worth maintaining.

Don't scale AI without guardrails

As AI systems gain access to more business data and operational tools, the potential benefits grow, but so do the risks.

A practical governance model separates company-wide rules from departmental business goals. The company defines which tools are allowed, how data should be protected, and which decisions AI must not make. Departments decide how to use AI to achieve their goals within those boundaries.

Access to data and APIs needs controls, especially when systems operate across multiple customers or business accounts.

Anonymization can reduce privacy risks, but it doesn't replace a broader security and governance process. Teams should assess what data an AI system can access, which actions it can take, how its outputs are reviewed, and whether a vendor's security practices meet their requirements.

For email marketing, this can mean limiting access to customer-level data, excluding unnecessary personal information from prompts, checking how a vendor processes and retains submitted data, and requiring human approval for sensitive decisions.

Governance also needs to reflect how people actually work. Overly restrictive policies may not reflect employees' real behavior, including their use of unofficial AI tools. Clear, practical rules and approved alternatives can help teams adopt AI without creating unnecessary security gaps.

The goal is to make experimentation safe enough to continue and successful enough to scale.

Wrapping up

The panel's discussion at the RARE conference offers a practical way to approach AI investment in 2027. Start with a business problem, choose a use case close to a meaningful KPI, and establish how you will measure both improvement and potential harm. Then evaluate whether the process is reliable, scalable, and what safeguards it needs.

Not every task needs an AI agent. Not every department needs the same level of automation. And not every promising experiment deserves more investment.

For email marketers, the opportunity is to use AI to make better decisions about customers, campaigns, and workflows, not simply to produce more content or launch more tests.

The most valuable AI strategy consistently identifies what works, learns from the evidence, and knows what not to do.