Virtual Emotion Mirror

Reads your facial expression in the browser and suggests music and movies to match.

when: Jan – Mar 2025
where: Solo project
role: Solo: design and build
stack: React, NestJS, Python, TensorFlow, MongoDB, Spotify API, IMDB API
links: github · live demo

problem: Recommendations lean on likes and history, so they miss how you feel right now.
built: A web app that reads your expression from the webcam, classifies it in a Python service, and suggests music and movies to match, with a mood history over time.
result: A working end-to-end app, live on Vercel, with suggestions that follow your mood as it changes.

Contents

What it is

A web app that reads your facial expression through the webcam, works out your mood, and suggests music and movies that fit it. It also keeps a history, so you can see how your mood moves over days, weeks and months.

I wanted to see whether an expression could act as an implicit signal for personalization, and what a mood history would be like to look at.

Architecture

Face detection in the browser, classification on the server

Frontend: React with the webcam and a lightweight face detector that runs in the browser, so no video leaves the page and the server is not busy with detection.
Backend: a NestJS API gateway that handles login and talks to everything else, and a Python service with TensorFlow that classifies the expression.
Data: MongoDB, which stores timestamped predictions for the mood history.

System architecture: React app, NestJS gateway, Python inference service, Spotify and IMDB connectors, MongoDB — Fig. 1 System architecture.

Splitting detection (browser) from classification (server) keeps latency low while the model stays easy to swap.

Emotion pipeline

From a video frame to a stable mood

The browser detects the face in the live video.
Facial landmarks and expression features are extracted from the frame.
The TensorFlow model classifies it as happy, sad, angry, surprised or neutral, with a confidence score.
Predictions are averaged over a short window, so one noisy frame does not flip the result.

Emotion recognition pipeline from video stream to face detection, features, classification and smoothing — Fig. 2 From video frame to a stable emotion.

Suggestions

Mapping a mood to music and movies

The mood picks the content. Calmer or more uplifting music for stress or sadness, higher energy for happiness, and movies filtered by genre to match. Suggestions update as your mood changes, and results from Spotify and IMDB are cached so they load quickly.

Sequence diagram from login through face capture, emotion detection and recommendations — Fig. 3 One session, end to end: login, face capture, emotion detection, genre mapping, Spotify and IMDB, and the feedback loop.

Mood history

Each prediction is stored with a timestamp. The dashboard turns those into daily, weekly and monthly charts, so recurring patterns, mood cycles and stressful stretches are easy to spot.

Not done yet

Voice alongside the face, for a second signal.
Running the model on the device instead of the server.
Themes in the UI that follow your mood.
More careful pattern detection in the history.

< back to projects/