Evaluation and Error Analysis on a Multitask Review Triage Capstone

Read the analysis notebook

My graduate capstone at The George Washington University was a team project. We built a model that reads customer reviews and sorts them by how urgently they need a human. A support team can then find the safety complaint hiding inside ten thousand “great ride” reviews. The model was trained on 12,000 Uber reviews. It uses a BERT encoder with five auxiliary heads feeding a triage classifier of four classes. The architecture and labeling were my teammates’ work. My part was evaluation and error analysis. This meant measuring whether the model actually worked and determining where it fell short. The whole team recognized the imperfections that come with building an advanced model in eight weeks. We wanted to show exactly how good it is and where it breaks and not merely that we built a classifier.

The code and the full notebook are on GitHub.

Model Architecture

The shared encoder reads a review and hands its [CLS] embedding to five small heads. Each head predicts one facet of the review. Four of them cover sentiment and emotion and topic and intent. The fifth produces an urgency score. Those predictions are concatenated back onto the embedding and fed to a final head. That head assigns one of four priority tiers. The tiers range from “no triage needed” to “immediate.”

Results

I scored the model across every variant the team tried. These variants covered different loss strategies and encoder choices and feature fusion configurations and auxiliary head ablations. For each one I computed precision and recall and F1 for every class. I also reported macro and weighted F1 along with confusion matrices and the urgency error. The final model reached 82 percent accuracy and 0.83 weighted F1 on a held out set of 1,200 reviews. The gap between weighted and macro F1 shows the class imbalance. The “no triage” tier is nearly perfect while the crowded middle tiers carry most of the error.

The confusion matrix above is not a screenshot. The notebook rebuilds it as data and recomputes accuracy and F1 directly from it. It then checks those figures against the reported numbers and reproduces 0.8225 and 0.7028 and 0.8260 exactly. A result that proves its own arithmetic is worth more than one the reader is asked to trust.

Error Analysis

97 percent of the model’s misclassifications land a single tier apart. The model almost never confuses “no triage” with a review that needs immediate attention. The most costly direction of error is also the rarest. The adjacent tiers blur because they are written in the same way. Grouping reviews by tier and comparing their TF-IDF centroids shows that “urgent” and “immediate” sit at 0.87 cosine similarity while “no triage” is cleanly separated from everything above it. The confusion matrix and the language of the reviews tell the same story about where the boundary is hard.

Label Quality Audit

The model learned from labels generated in two different ways. One was a heuristic. The other was a separate pass from a language model. I treated the two as independent annotators and measured how often they agreed. They agreed only moderately at a Cohen’s kappa of about 0.33. The disagreement ran in one direction. The heuristic escalated far more aggressively than the model did. That noise in the training labels sets a real ceiling on how far the triage accuracy can be trusted. It is also the kind of issue worth catching before anyone puts weight on the result.

Availability

This project is an analysis notebook rather than a deployed app. There is no live link to launch. It runs on CPU from the public review data in a few seconds. The rendered version linked above shows every figure and number alongside the code that produced it. The source is on GitHub.