Data Challenge

Enhancing customer service

Evaluating American Air's Twitter customer service performance by comparing tweet sentiment and formality against competitors while providing a data-driven recommendations.

Overview Methodology Business Idea Findings Recommendation
Overview

The aim of the project

The aim of this project is to evaluate American Air and their Twitter Team's customer service performance. We compare them with competitors and run analysis on tweets from both the airline and their customers. We also provide advice on how they should improve their team.

Our goal is to find out whether American Air improves or worsens customer emotion within the conversation thread. Additionally, we chose to analyse the Twitter Team's formality in their tweets and whether this has an impact on customer emotion.

2M+ Tweets analysed
9 Airlines compared
78.8% Model accuracy
+0.227 Avg sentiment evolution
500 Hand-labelled tweets
Methodology

Data pipeline & approach

The project followed a structured pipeline from raw data through to business recommendations. Two parallel tracks ran simultaneously: sentiment analysis and formality analysis.

01
Data Cleaning
Removed tweets with no relevant information. Removed non-English tweets. Kept all relevant airline data.
02
Databases
An SQL based database stores data in a structured manner using tables that work well with large datasets of this scale.
03
Define Conversation
Before an analysis could be made about the effect of a conversation, a definition of what a conversation is had to be made. The definition we arrived to was an entirely english conversation between a minimum of two users longer than at least 2 tweets.
04
Analysing Conversations
A conversation is viewed as complete when a problem is solved or redirected. The difference between sentiment from the first to the last message is of importance.
05
Sentiment Analysis
Used language models to give a label for the general emotion of each tweet. To determine the quality of the model we compared its output to our manual labelling of 500 tweets.
06
Formality Analysis
A pretrained formality model rates tweet text on a scale of 0 (informal) to 1 (formal). Evaluated the same way as sentiment analysis — validated against 500 hand-labelled tweets. The model achieved a 78.7% accuracy.
Business Idea
Analyse formality effect
Analyse the effect the formality of tweets has on customer sentiment evolution. Research shows a link between formality and message effectiveness.
Model Evaluation
Confusion matrix validation
Both the sentiment and formality models were validated using confusion matrices comparing model output against manual labels of 500 tweets.
Output
Recommendation
All previous parts feed into a concrete recommendation for the customer service of American Air, grounded in empirical findings.
Business Idea Integration

Does formality drive sentiment?

Research has shown that there is a link between formality and the effectiveness of a message, and that there might be a level of formality which improves the tone. This gave us a reason to look into whether higher formality would lead to an improvement in customer sentiment evolution and if American Air can profit from an improvement in their tweet formality.

Scatter Plot
Shows sentiment evolution and formality scores for all airlines. Sentiments cluster at specific values (−2.0, −1.0, 0.0, 1.0, 2.0), while formality scores vary widely across the 0–1 range.
Box Plot
With the third lowest formality score, American Air's tweets are less formal than most competitors. This introduced the idea that increasing formality could improve sentiment evolution.
Formality Scale
Model rates tweet text on a scale of 0 (informal) to 1 (formal). Accuracy of 78.7% — an acceptable level, meaning we can rely on the outcome.
Key Findings

What the data revealed

Analysis of formality ranges and their corresponding average sentiment evolution scores yielded clear, actionable insights about the sweet spot for tweet formality.

Formality 0.0 – 0.2
Very low sentiment evolution score of 0.0576. Highly informal tweets correlate with barely any positive change in customer emotion.
Formality 0.6 – 0.8
Significantly higher sentiment evolution score of 0.2547. A moderate-to-high formality range shows meaningful improvement in customer sentiment.
Formality 0.8 – 1.0
Sentiment evolution at 0.2452 — diminishing returns. Overly formal tweets risk sounding robotic. American Air is advised not to go overboard.
Key Conclusion
Higher formality within tweets had a positive effect on sentiment evolution. The optimal range appears to be 0.6–0.8 — formal enough to be professional, natural enough to feel human.
American Air
Average sentiment evolution of 0.227. Overall slightly improved customer sentiment, but with 3rd lowest formality among airlines, there is substantial room to improve.
Recommendation

What American Air should do

Overall verdict

The American Air Twitter Team had an average sentiment evolution score of 0.227, meaning they overall slightly increased the sentiment of their customers. We also found that higher formality leads to a better score.

With American Air having the 3rd lowest formality there is plenty of room for improvement. Therefore, we believe the Twitter Team should be kept and guided towards a more consistently formal communication style — without tipping into robotic territory.

Twitter Team Improvements

Training and Development: Implement training programs to guide the Twitter Team on writing more formal responses to customers whilst ensuring it does not sound robotic. This will help achieve a higher sentiment evolution score.

Consistency in Communication: Maintaining a consistent level of formality across all interactions will help uphold a professional and reliable brand image.

Fly High Project Improvements

Time efficiency could have been improved by using a different database platform — putting conversations into tables took 16 hours. Running code locally instead of online for sentiment and formality analysis would also deliver results faster.

Accuracy of results could be improved by altering the model to be more suitable for the project. An accuracy of 78.8% has room for error and could hinder the analysis provided to the company.