Revolutionise your content delivery with AI-based recommendations
In this post, we discuss the advantages of a recommendation system for video streaming services and explore its positive impact on brand and content discoverability. Additionally, we gain insights into a new AI-powered recommendation engine by Magine Pro which will soon be released to our partners.
OTT streaming platforms offer an extensive range of content to their users. However, people often find it challenging to choose from numerous options, and may become quickly overwhelmed, resulting in poor choices or no decision at all.
According to consumer research at Netflix [1], a user typically loses interest in selecting content after browsing for just 60 to 90 seconds and reviewing 10 to 20 titles. If the user cannot find anything of interest, there is a substantial risk of them discontinuing the service.
Recommendation systems provide a solution by offering a personalised experience to users that make them feel as if the streaming service was created exclusively for them, which enhances organic engagement.
Where we started
During our last Hack Days at Magine Pro, we challenged ourselves to create a recommendation service for the platform. We brought together a cross-functional team of engineers from the Backend, Infrastructure, and Media teams to tackle this problem from different domain perspectives.
The aim was to create an end-to-end real-time recommendation system for a streaming service. Modern recommendation machine learning algorithms use datasets of users, items, and their interactions as inputs. This data has to be streamed, cleaned, transformed, and inserted into the model for the training.
Using modular programming, a technique of breaking down a product into smaller, self-contained units that can be developed and tested independently, we built separate components and went through all stages of the recommendation system creation process.
As a first proof of concept, we brought together the following services:
- Mux Analytics is used to examine clickstream data and get insights into video engagement and has an integration with AWS, our main cloud provider
- AWS Kinesis Data Streams provides the ability to stream large amounts of data in real-time
- AWS Lambda can be used to perform extract, transform, and load (ETL) jobs on the protobuf-encoded data we get from Mux
- AWS Personalize is a self-contained AI service that takes input data according to the schema and generates various types of recommendations
That worked like a charm! As a next step, our team tried to reduce the costs since using MUX/AWS integration and AWS Personalize was expensive. We also wanted to experiment with different recommendation engines to find the best suitable for our needs.
Minimising the costs
The team started to think about how we can make this feature less expensive for Magine Pro and our partners.
Here are the steps we performed to decrease the costs:
- Replaced the Mux Analytics module with the Apache Kafka, a distributed streaming platform, running in our AWS infrastructure setup
- Created a separate backend service for the ETL jobs, which listened to the Kafka topics, modified and enriched them with additional information, and provided the resulting data further
- Integrated Gorse, another recommendation engine, into our system, utilising Redis as our in-memory data store and a fully managed Postgres relational database, both running on AWS
The modularity of the system gives us flexibility in many ways. For example, we can introduce yet another recommendation engine and compare them to decide which one provides the best results. No need to rebuild the entire setup; only one component has to be modified.
Key concepts
Under the hood, Magine Pro uses a recommendation engine that automatically chooses the best machine learning technique for the most accurate results. The model processes real-time clickstream data from the platform, capturing diverse user behaviours. The following recommendation types are supported:
- Most popular
- Latest releases
- Because you watched X
- More like X
- Top picks for you
Magine Pro’s recommendation system includes a vast selection of diverse recommendation types which can be customised to align with your business requirements.
Recommendation algorithms
There are many algorithms that can be used in recommendation engines, let’s look closer at the most popular ones. Recommendation algorithms are categorised into non-personalised algorithms that provide identical content to all users, and personalised ones that recommend unique content to each user based on their clickstream data.
Latest Items algorithm shows the latest items to users according to timestamps, which allows a new item to be exposed to users in time.
Popular Items algorithm counts positive feedback within the specified window interval and chooses the most liked.
Item-based and user-based collaborative filtering algorithm discovers the similarities in the user’s past behaviour and makes predictions for the user based on a similar preference with other users.
It uses a simple yet effective approach – people often get the most helpful recommendation from people with similar tastes.
Matrix factorization algorithm identifies the relationship between items and users and returns recommended results.
Recommendation system by Magine Pro offers all these options, utilising a hybrid approach that incorporates a diverse range of rating and sorting algorithms.
The Cold Start Problem
Cold start problem refers to a situation where recommendation engines cannot provide optimal results due to suboptimal conditions, similar to a car engine having difficulty starting in cold weather. The two categories of cold start are item cold start and user cold start.
Cold start problem:
What to recommend to a new user U4?
Who to show the new movie I4?
Collaborative filtering algorithms can find items to recommend when the number of user actions on an item increases. However, they fall short in case of new users or items since the model cannot predict anything for them.
Our recommendation service provides suggestions of the most popular and latest items to new users to solve the cold start problem.
Evaluation
Model evaluation is an essential part of the work on recommendation engines and refers to estimating the performance of a recommender system.
The system automatically reserves a small part of the feedback to make an evaluation of it. The aim is to maximise the probability that the user likes recommended items so the positive feedback rate can be used as a metric.
In the recommendation engine, we use standard ML evaluation metrics to evaluate ratings and predictions: precision, area under curve (AUC), and recall, where 0 means the worst and 1 means we have the best-performing ML model.
As you can see on a screenshot from the recommendation engine dashboard, our model demonstrates impressive performance.
What’s next?
Currently, our team is going through the final steps of feature development, evaluating results to ensure that the system we have built is resilient, highly available, and provides the best possible results.
We are also looking into the potential of integrating a content promotion feature for our partners.
Furthermore, we have long-term plans, such as integrating another recommendation engine into our modular system and creating a scalable testing solution to select the most appropriate model that meets our partners’ needs.
Conclusion
This post has highlighted the significance of recommendation systems in OTT streaming services, explored various algorithms, and delved into the details of recommendation engines. Magine Pro is soon launching a new recommendation service that serves as an example to highlight common issues encountered in creating such services and to introduce frequently used performance metrics.
Don’t forget to subscribe to the Magine Pro e-newsletter to stay up-to-date with our latest news, partners, and products, and to find out which industry events we’ll be attending next.