YOUR RECOMMENDATION ENGINE DOESN’T RECOMMEND. IT RETRIEVES.
In 2016, Paul Covington, Jay Adams, and Emre Sargin at Google published “Deep Neural Networks for YouTube Recommendations” [1]. The paper described a two-tower architecture: embed users into a vector space based on their interaction history, embed items into the same space, compute cosine similarity, return the top-N most similar items. It became the most influential recommendation systems paper of the decade, accumulating over 3,400 citations [2]. Spotify, Netflix, Amazon, and every personalization vendor in martech adopted some version of it.
The paper is brilliant. But it solved a specific problem: retrieval at YouTube scale, where the corpus changes every minute and 500 hours of video are uploaded every hour [3]. What happened next is that the entire industry mistook retrieval for recommendation, and retrieval for optimization. They are three different things. The cost of this confusion, measured in customer lifetime value, is 3-5x.
I hold a PhD in Physics and an Executive MBA from IE Business School. I taught machine learning and AI at Sofia University for ten years. I have spent four years building a platform that processes data for 250+ B2C brands across 15 countries. This article explains what recommendation engines actually do, why what they do is not what ecommerce needs, and what the architecture that works looks like in practice.
What retrieval actually does
The recommendation engine market is projected to reach $9.15 billion in 2025 and $38.18 billion by 2030 at a 33% CAGR [4]. Product recommendations account for just 7% of ecommerce traffic but generate 24% of orders and 26% of revenue [5]. Amazon attributes 35% of its purchases to its recommendation engine [6]. 92% of companies now use AI-driven personalization for customer engagement [7].
These numbers are real. But they obscure a critical distinction: the architecture behind nearly every one of these systems is retrieval, not recommendation, and certainly not optimization.
Here is what retrieval does, step by step:
Step 1: Embed products into a vector space based on co-occurrence patterns (which products appear together in browsing sessions, carts, and purchase histories). Step 2: Embed users into the same vector space based on their interaction history (views, clicks, purchases). Step 3: Compute cosine similarity between the user vector and all item vectors. Step 4: Return the top-N most similar items.
This is architecturally identical to search. It finds the nearest neighbor in embedding space. The model has learned “people who did X tend to do Y.” It has not learned “this specific customer should see Z because Z maximizes their predicted 12-month value.”
The Covington paper itself acknowledges this architecture explicitly. The system is split into two stages: candidate generation (retrieval) and ranking. The candidate generation stage narrows billions of videos to hundreds using the two-tower embedding approach. The ranking stage then scores those hundreds using a richer set of features to produce the final ordered list [1].
In martech, most personalization vendors implement only the first stage. They do retrieval. They call it recommendation. The ranking stage, where you could inject a value model, is either absent or optimizes for the wrong objective.
Three different problems, three different architectures
The recommendation systems research community distinguishes clearly between retrieval, ranking, and optimization [8]. A 2025 comprehensive survey on retrieval methods in recommender systems (Huang et al.) documents over 100 research papers on retrieval alone, noting that “multi-stage cascade ranking systems are widely used in the industry, with retrieval and ranking being two typical stages” [9].
Here is where each stage sits and what it optimizes:
Retrieval answers: What is most similar to this customer’s history? Method: Embedding similarity (cosine, dot product). Optimizes for: P(click | user, item). Result: More of the same. Discount buyers get more discounts. Bargain hunters see more bargains.
Ranking answers: Among similar items, which should be shown first? Method: Learning-to-rank models with richer features. Optimizes for: Weighted combination of engagement signals (click probability, dwell time, add-to-cart rate). Result: Better ordering. Still within the retrieval set.
Decisioning answers: What action maximizes this customer’s predicted lifetime value? Method: Trajectory optimization over predicted CLV. Optimizes for: E[LTV | action]. Result: Finds the most valuable next action, not the most similar product. May recommend something outside the customer’s existing pattern if the value model predicts it will shift their trajectory upward.
The distinction between ranking and decisioning is where the entire martech industry gets stuck. A system that ranks retrieved items by click probability is still optimizing for engagement. It cannot shape a customer’s trajectory because it has no concept of trajectory. It has no value model. It has no objective function beyond the current session.
Xavier Amatriain, former VP of Engineering at Netflix and now at LinkedIn, described the difference between exploitation and exploration in recommendation systems [10]. Exploitation is easy to measure: show users what they will click on and watch engagement metrics rise. Exploration is harder: show users something outside their pattern that might shift their behavior long-term. Every production system defaults to exploitation because the short-term metrics reward it.
A 2023 RecSys tutorial on “Customer Lifetime Value Prediction: Towards the Paradigm Shift of Recommender System Objectives” stated this explicitly: “Despite the success of current recommendation techniques in targeting user interest, optimizing long-term user engagement and platform revenue is still challenging due to the restriction of optimization objectives such as clicks, ratings, and dwell time” [11].
The paradigm shift the tutorial describes is precisely the shift from retrieval to decisioning.
The reinforcement loop: how retrieval destroys the value it measures
In martech, the retrieval failure mode is pervasive. A “personalization engine” that shows a price-sensitive customer cheaper products is not personalizing. It is reinforcing. The customer who might have bought a mid-range product, one that based on historical cohort data predicts a 3x higher repeat rate, never sees it. The system optimized for P(click) instead of E[LTV|action].
This creates a reinforcement loop:
- Customer buys a discounted item.
- System embeds this customer near other discount buyers.
- System recommends more discounted items (highest cosine similarity).
- Customer buys another discounted item (confirming the system’s prediction).
- System embeds this customer even deeper into the discount cluster.
- Customer never sees the mid-range product that would have triggered category expansion.
The system’s accuracy metrics improve. P(click) goes up because the system gets better at predicting what the customer will click on. But CLV goes down because the system is reinforcing the lowest-value behavioral pattern instead of shaping the highest-value trajectory.
This is not theoretical. A systematic review of value-aware recommender systems published in Expert Systems with Applications found that “the first explicit reference to the term value-aware is found in the work of Amatriain and Basilico (2016)” and documented that the majority of recommendation systems research optimizes for engagement rather than economic value [12]. The review identified reinforcement learning as the primary approach for systems that do optimize for lifetime value, noting that “more recent works have proposed directly optimizing the long-term performance of recommender systems” through RL algorithms [12].
The research on ad recommendation systems by Theocharous et al. (2015) demonstrated this gap empirically: “Even though Policy 2 has a lower CTR than Policy 1, it results in more revenue, as captured by the higher LTV. Hence, LTV is potentially a better metric than CTR for evaluating recommendation policies” [13]. The policy that maximizes clicks is not the policy that maximizes revenue. The policy that maximizes immediate revenue is not the policy that maximizes lifetime value. At every level of the optimization hierarchy, the optimal recommendation changes when you change the objective function.
What Amazon learned (and what martech ignored)
Amazon’s internal research has evolved toward what I would call value-aware decisioning. Their work on “Actions Speak Louder than Clicks” showed that optimizing for downstream purchase behavior rather than click-through rate produced different and more valuable recommendations [14]. The items that maximize clicks are not the items that maximize revenue. The items that maximize immediate revenue are not the items that maximize lifetime value.
Amazon attributes 35% of its total sales to its recommendation engine [6]. But the version of that engine that generates 35% of revenue is not the version that finds “similar products.” It is the version that predicts purchase probability weighted by basket value, informed by purchase history, category expansion patterns, and replenishment cycles.
Netflix’s recommendation system drives 80% of content discovery on the platform [15]. But Netflix does not optimize for clicks. It optimizes for hours watched, which is a closer proxy to lifetime value (subscriber retention) than click-through rate. This is why Netflix will sometimes surface content that a user has a lower probability of clicking on but a higher probability of watching to completion. The ranking objective is different from the retrieval objective.
The martech industry adopted the retrieval architecture from these systems without adopting the value-aware ranking layer that makes them work. Most product recommendation blocks in ecommerce run cosine similarity over a product embedding space and call the output “personalized recommendations.”
The missing layer: why personalization needs a value model
Here is the core insight, stated as precisely as I can: every personalization engine in martech is a function that maps (customer, context) to action. The question is what that function optimizes.
Retrieval systems optimize: argmax_{item} P(click | user, item, context) Find the item with the highest click probability given this user and context.
Decisioning systems optimize: argmax_{action} E[LTV | user, action, context] Find the action (which may be a product, a message, a channel, a timing, or a non-action) that maximizes the expected lifetime value of this customer.
The second formulation requires a value model: a predictive CLV model that sits upstream of every personalization decision. Not “what is most similar to their history” but “what intervention, shown at what moment, maximizes the predicted outcome for this specific customer’s trajectory.”
When I taught ML at Sofia University for ten years, I would frame it as the difference between argmax over a static distribution and optimization over a dynamic trajectory. The first finds the best item given the current state. The second finds the best action given the desired future state. Most of the martech stack is built on the first. The second requires a value layer that connects behavior to predicted revenue, not to predicted clicks.
This is what makes a decision intelligence platform different from a recommendation engine. The recommendation engine is a component. The decisioning platform is the architecture that tells every component, from product recommendations to search to email to push to ads to loyalty, what to optimize for.
Next-basket prediction: the capability retrieval cannot replicate
The most commercially valuable prediction in ecommerce is not “what product will this customer click next?” It is “what will this customer’s next basket contain, and when will they buy it?”
Next-basket prediction is a specific machine learning capability that models a customer’s purchase sequence over time and predicts the set of items they will buy in their next transaction [16]. Unlike collaborative filtering (which finds similar users) or content-based filtering (which finds similar products), next-basket prediction explicitly models temporal patterns: purchase cycles, category expansion trajectories, seasonal behaviors, and replenishment timing.
This capability transforms every downstream personalization decision:
Email timing becomes predictive. Send the browse abandonment email when the customer is 3 days before their predicted purchase window, not 24 hours after they left. The workflow triggers based on predicted behavior, not past behavior.
Product selection becomes trajectory-aware. Recommend the items that fit the customer’s predicted next basket, not just their last click. The product blocks on the homepage, category page, and product detail page each serve a different strategic objective.
Channel selection becomes optimized. Reach the customer through the channel they have historically engaged with at this point in their purchase cycle, whether that is email, SMS, push, or on-site banners.
Offer calibration becomes value-aware. A customer whose predicted basket is EUR200 should see a different promotion than one whose predicted basket is EUR50.
Retrieval cannot do any of this. Retrieval has no concept of time, no concept of trajectory, no concept of value. It has cosine similarity and a product catalog. That is not a recommendation engine. That is a search engine with a different name.
The convergence that confirms this gap
In January 2026, Gartner published its inaugural Magic Quadrant for Decision Intelligence Platforms [17]. In March 2026, Scott Brinker published a 33-page composable canvas framework with Databricks, positioning decisioning as Ring 4 of five concentric capabilities [18]. In April 2026, Brinker and Frans Riemersma published the 2026 State of Marketing Attribution report, declaring Attribution 1.0 dead and calling for attribution as a decisioning discipline [19].
Three independent publications. One convergent conclusion: decisioning is now a recognized, named, and budgeted enterprise capability. The systems that win will not be the ones that find the most similar product. They will be the ones that find the most valuable next action.
For a full analysis of how these frameworks map to the B2C decisioning gap, see our decision intelligence platform guide. For the five structural failures this architecture closes in the typical ecommerce stack, see our five blind spots analysis. For why this matters to your ad spend, see our guide on why you should stop advertising to your own customers.
What the numbers look like when you switch
Ivet, fashion retailer, 48,000+ SKUs, 10 EU countries. Before: Klaviyo for email with standard recommendation blocks (retrieval-based). After switching to value-optimized decisioning: 6.2% conversion rate on Releva-influenced traffic versus 2.7% uninfluenced, a 130% lift. Ad spend cut 50%. Repeat purchases up 2.5x. Releva became the #1 revenue source at EUR107K/mo. See the full case study.
Carsome, SE Asia’s largest car marketplace, $1.7B valuation. Before: MoEngage + Segment + Dynamic Yield (three platforms, none sharing an objective function). After: email opens from 1.2% to 18% (15x), click rates from 6.1% to 36% (6x), MYR 36.8M monthly attributed revenue, 82x annual platform cost.
The difference is not better algorithms. It is a different objective function. When every personalization touchpoint optimizes for predicted customer lifetime value instead of click probability, the compounding effect changes everything.
BCG’s Personalization Index shows that personalization leaders achieve revenue growth rates approximately 10 percentage points higher than laggards annually [20]. McKinsey data shows fast-growing companies derive 40% more revenue from personalization than slower-growing peers [21]. But these gains accrue only to systems that optimize for value, not engagement. A retrieval engine that optimizes for P(click) captures a fraction of the available value.
How to evaluate your current recommendation engine
Before investing in new tools, test your existing system against these five questions:
- Does your recommendation engine know what a customer is worth? If your product blocks show “similar products” without any reference to predicted CLV, you have a retrieval engine.
- Can your system recommend something the customer has never browsed? If recommendations are limited to items similar to past interactions, the system cannot explore. It can only exploit.
- Do your email recommendations differ from your on-site recommendations? If the email product blocks show the same items as the on-site blocks, there is no channel-aware decisioning. The system is running the same retrieval query everywhere.
- Does your system differentiate a EUR50 customer from a EUR5,000 customer? If both receive the same recommendation logic, the system has no value layer. Segments should combine behavioral, transactional, and predicted value data.
- Can you measure whether a recommendation changed a customer’s trajectory? If your measurement is limited to CTR and session conversion, you are measuring retrieval performance, not business impact. The right measurement is predicted CLV lift and RFM state transitions.
If your system fails three or more of these tests, the architecture needs to change. This connects directly to the five structural blind spots in most ecommerce marketing stacks: recommendation that reinforces patterns instead of shaping trajectories is Blind Spot 4 (The Unpredictable Value) manifesting at the product level.
Implementation: from retrieval to decisioning
You do not need to replace everything at once. The highest-impact sequence:
Week 1-2: Integrate and capture. Connect server-side tracking to capture 100% of visitor data. Turn on the standard product blocks, which come preconfigured with value-aware recommendation logic across industries.
Week 2-4: Activate behavioral flows. Launch three foundational workflows: abandoned cart (value-differentiated by cart amount), abandoned browse (with predicted-value product recommendations), and weekly personalized digest (AI-selected products matching each customer’s trajectory).
Week 4-8: Layer in prediction. Deploy the predictive CLV model. Connect it to segments so every workflow reads from predicted value. Configure the two-event CAPI architecture for Facebook and Google. For the full technical detail on server-side signal architecture, see our server-side tracking guide.
Week 8-12: Compound. Add loyalty integration, expand banner personalization, activate search personalization, and deploy cross-channel orchestration. For a deep dive into loyalty architecture, see our ecommerce loyalty program guide.
Most brands see initial measurable results within 30 days. Full ROI picture at 90 days. Book a demo to see how it works with your data.
FAQ
What is the difference between retrieval and recommendation? Retrieval finds the most similar items to a customer’s history by computing cosine similarity in an embedding space. True recommendation determines the most valuable next action for each customer’s trajectory by optimizing for predicted lifetime value. Most systems marketed as “recommendation engines” perform retrieval, not recommendation.
Why do recommendation engines reinforce low-value behavior? Because they optimize for P(click): the probability that a customer clicks on a recommended item. Since customers are most likely to click on items similar to what they already browsed, the system systematically shows discount buyers more discounts and bargain hunters more bargains. It reinforces the existing pattern instead of shaping a higher-value trajectory.
What is a value model in the context of recommendations? A value model is a predictive CLV model that sits upstream of every recommendation decision. Instead of asking “what will this customer click on?” it asks “what action maximizes this customer’s predicted 12-month value?” This changes every recommendation: product selection, timing, channel, and whether to recommend at all.
What paper created the modern recommendation engine architecture? Covington, Adams, and Sargin’s 2016 paper “Deep Neural Networks for YouTube Recommendations” described the two-tower embedding architecture that became the blueprint for nearly every recommendation system built since. The paper has accumulated over 3,400 citations and its architecture was adopted by Spotify, Netflix, Amazon, and virtually every martech personalization vendor.
How does next-basket prediction differ from collaborative filtering? Collaborative filtering finds similar users and recommends what those users bought. Next-basket prediction models a customer’s purchase sequence over time and predicts the specific items and timing of their next transaction. It explicitly models temporal patterns like purchase cycles, category expansion, and replenishment timing, which collaborative filtering cannot capture.
Can a recommendation engine optimize for customer lifetime value? A standard retrieval-based recommendation engine cannot, because it has no value model and no concept of trajectory. A decisioning system can, because it uses predicted CLV as its objective function and optimizes every recommendation toward the highest-value trajectory for each customer.
How long does it take to switch from retrieval to decisioning? Integration takes 3-5 days. Standard value-aware product blocks activate immediately. Predictive CLV modeling deploys within 4-8 weeks. Full cross-channel decisioning within 8-12 weeks. Most brands see measurable results within 30 days.
5. REFERENCES
[1] Covington, P., Adams, J. & Sargin, E. (2016). “Deep Neural Networks for YouTube Recommendations.” Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). pp. 191-198. https://dl.acm.org/doi/10.1145/2959100.2959190
[2] Semantic Scholar citation count for Covington et al. (2016). 3,445 citations as of 2026. https://www.semanticscholar.org/paper/Deep-Neural-Networks-for-YouTube-Recommendations-Covington-Adams/5e383584ccbc8b920eaf3cfce3869da646ff5550
[3] YouTube Creator Blog (2022). “500 hours of video uploaded every minute.” https://blog.youtube/
[4] Mordor Intelligence (2025). Product Recommendation Engine Market. “Market size $9.15B in 2025, projected $38.18B by 2030 at 33.06% CAGR.” https://www.mordorintelligence.com/industry-reports/recommendation-engine-market
[5] Clerk.io (2026). “Product recommendations account for just 7% of ecommerce traffic but generate 24% of orders and 26% of revenue.” https://www.clerk.io/blog/product-recommendations-statistics
[6] Amazon / McKinsey (widely cited). “Amazon attributes 35% of its purchases to its recommendation engine.” Multiple sources including Intellias (2025), EComposer (2025).
[7] WiserNotify (2025). “92% of companies now use AI-driven personalization for customer engagement.” https://wisernotify.com/
[8] Liang, S. & Zhang, Y. (2025). Generative Recommendation: A Survey of Models, Systems, and Industrial Advances. TechRxiv. “Three converging trends: (1) unification of retrieval and ranking under shared architectures; (2) integration of preference-aligned objectives; (3) rapid adoption of cross-domain foundation models.” https://www.techrxiv.org/doi/full/10.36227/techrxiv.176523089.94266134/v2
[9] Huang, J. et al. (2024/2025). “A Comprehensive Survey on Retrieval Methods in Recommender Systems.” ACM Transactions on Information Systems. “Multi-stage cascade ranking systems are widely used in the industry, with retrieval and ranking being two typical stages.” https://dl.acm.org/doi/10.1145/3771925
[10] Amatriain, X. Work on exploitation versus exploration tradeoffs in recommendation systems. Former VP of Engineering at Netflix, currently at LinkedIn.
[11] Li, K. et al. (2023). “Customer Lifetime Value Prediction: Towards the Paradigm Shift of Recommender System Objectives.” Tutorial at ACM RecSys 2023. https://dl.acm.org/doi/10.1145/3604915.3609499
[12] Ferraro, A. et al. (2023). “A Systematic Review of Value-Aware Recommender Systems.” Expert Systems with Applications. “The first explicit reference to the term value-aware is found in the work of Amatriain and Basilico (2016).” https://www.sciencedirect.com/science/article/pii/S0957417423006334
[13] Theocharous, G., Thomas, P.S. & Ghavamzadeh, M. (2015). “Personalized Ad Recommendation Systems for Life-Time Value Optimization with Guarantees.” IJCAI 2015. “Even though Policy 2 has a lower CTR than Policy 1, it results in more revenue.” https://dl.acm.org/doi/10.5555/2832415.2832500
[14] Amazon research, widely cited. “The items that maximize clicks are not the items that maximize revenue. The items that maximize immediate revenue are not the items that maximize lifetime value.”
[15] Netflix / EComposer (2025). “80% of Netflix content discovery comes from AI recommendations.” https://ecomposer.io/blogs/ecommerce/ai-in-ecommerce-statistics
[16] Epsilon Engineering Blog (2024). “Driving E-commerce Success through Next Basket Recommendation System.” https://medium.com/epsilon-engineering-blog/driving-e-commerce-success-through-next-basket-recommendation-system-f51cc3f45e54
[17] Gartner (2026). Magic Quadrant for Decision Intelligence Platforms. Pidsley, Idoine, Herschel, Quinn, Carlsson. January 26, 2026.
[18] Brinker, S. (2026). The New Martech “Stack” for the AI Age. Produced with Databricks. March 2026. https://www.databricks.com/resources/ebook/new-martech-stack-ai-age
[19] Brinker, S. & Riemersma, F. (2026). 2026 State of Marketing Attribution Report. “Attribution 1.0 is dead.”
[20] BCG (2025). Personalization Index. “Personalization leaders achieve compound annual growth rates 10+ percentage points higher than laggards.”
[21] McKinsey (2023). The Value of Getting Personalization Right (or Wrong) Is Multiplying. “Fast-growing companies derive 40% more revenue from personalization.” https://www.mckinsey.com/
[22] Envive (2026). “AI-driven experiences increase CLV by 33%. Product recommendations drive up to 31% of ecommerce revenues.” https://www.envive.ai/post/ai-personalization-in-ecommerce-lift-statistics
[23] Ringly (2026). “Sessions where shoppers engage with recommendations show a 369% increase in average order value.” https://www.ringly.io/blog/ecommerce-personalization-statistics-2026
[24] Salesforce / Intellias (2025). “Customers who interact with AI-powered product recommendations have a 26% higher average order value.” https://intellias.com/ecommerce-recommendation-engines/
[25] Verma, J. (2026). “RecSys After LLMs: Four Paradigms for What Comes Next.” “OneRec proposes replacing the cascade with a single end-to-end generative model.” https://januverma.substack.com/p/recsys-after-llms-four-paradigms
[26] Deng, J. et al. (2025). “OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment.” arXiv:2502.18965. https://arxiv.org/abs/2502.18965
[27] Brinker, S. (2026). “Could loyalty systems be killer context-as-a-service (CaaS) platforms?” Chiefmartec Newsletter. https://newsletter.chiefmartec.com/p/could-loyalty-systems-be-killer-context-as-a-service-caas-platforms
[28] EasyApps (2026). Ecommerce Personalization Statistics 2026. “5-8x ROI. 10-25% revenue increases. 15-30% higher conversion.” https://easyappsecom.com/guides/shopify-personalization-statistics-2026.html
[29] ecomrankd (2026). Ecommerce Statistics 2026. “AI-in-ecommerce market projected to reach $9.9 billion in 2026. Stores embedding AI report 15-25% revenue increases.” https://ecomrankd.com/ecommerce-statistics-2026/
[30] Elementor (2026). Ecommerce Statistics 2025-2026. “91% of consumers more likely to shop with brands providing personalized recommendations.” https://elementor.com/blog/ecommerce-statistics/
