Course Recommender System
Problem Statement
Designing a course recommender system that will recommend courses to prospective candidates in an e-learning platform.
Client is an e-learning based platform with courses on various topics including Web Development, Data Science, Mobile Development, Cloud Services etc.
The Client has a user base of 10000 users and offers more than 100 courses.
Data
The data consists of
1 -> Users Information.
2 -> Courses Information.
3 -> User – Course Interaction Activity.
Feature Building and Feature Store
Requirements
Download data stored in AWS and Create necessary features.
Design features and create a feature store.
Architecture
Explanation
Downloaded data from AWS bucket using Boto3.
Created a Mongo DB Database Collection to store courses data.
Extracted, cleaned, and validated users and interactions data to create features in correct format.
Used terraform to create Infrastructure required to store the Features in AWS ( Redshift and Dynamo DB ).
Created a feature store using Feast Feature Store and the AWS Infrastructure created.
Sync the feature store registries in AWS S3.
Building Docker File for the entire pipeline.
Model Training and Evaluation
Requirements
Create a model for recommending courses to students based on their interests and history of recorded activities.
Cater to new users with no recorded activities.
Scaling up as required.
Architecture
Explanation
Ingestion of features from feast feature store.
Split of Data into Train Validation Test sets.
Model Training using a hybrid recommender system.
Evaluation of the model and generating metrics.
Finding the best model whenever model is trained.
Upload of best model, metrics, artifacts to AWS S3 bucket.
Using a scheduler (Airflow) to schedule the pipeline.
Using Grafana and Prometheus to monitor statistics.
Building Docker file for the entire process.