Training a model to understand what people are asking a chatbot
Japeto supplied 1,358 labelled messages from its website assistant across twenty business categories. I built a reproducible ML pipeline to test which modelling choices genuinely improved classification.
Role
Machine Learning Engineer
Organisation
Japeto
Industry
Conversational AI · Analytics
Focus
ML · Pipeline · Analytics
/ Problem
Japeto’s website assistant receives questions about services, billing, support, partnerships and other areas of the business.
The dataset contained 1,358 messages across twenty categories, but the classes were heavily imbalanced. The largest contained hundreds of examples while the smallest had only a handful.
There was another problem. Messages belonged to 742 conversation sessions.
A random train-test split could place different messages from the same conversation on both sides, making the model appear more accurate than it really was.
So the project became less about finding the highest score and more about answering a harder question:
How do I know one model is genuinely better than another?
/ Solution
I built the project as a repeatable experiment rather than a single training script.
Conversation sessions stay together so the same discussion cannot leak between training and testing.
I then compared text representations, classifiers, response-only inputs and context-enhanced inputs.
The original twenty labels were also heavily imbalanced, so I created a broader seven-category scheme and tested it against the original rather than assuming it would help.
It did.
The best twenty-category model reached about 70% accuracy and 0.672 macro F1. The strongest seven-category configuration reached 83.2% accuracy and 0.814 macro F1.
Including conversation context also consistently improved performance.
The biggest gain came from changing how the problem was represented, not simply changing algorithms.
/ What I learned
This project changed how I approach machine learning experiments.
It is easy to spend time swapping models while leaving the more important decisions untouched.
Here, reorganising the labels and supplying the right context mattered more than simply choosing a more sophisticated classifier.
Reproducibility also became essential. If every experiment uses different data boundaries or preprocessing, a better score tells you very little.
I wanted every improvement to have a reason behind it.