Politeness classification, from SVMs to LoRA • Adriano Machado

Four-way politeness classifier on Intel's Polite Guard, from Word2Vec baselines to RoBERTa, PEFT and LLM prompting.

With Félix Martins and Francisco da Ana · Natural Language Processing, FEUP/FCUP

Intel's Polite Guard dataset labels customer-service text as polite, somewhat polite, neutral or impolite. Over two assignments we worked up from sparse features and SVMs through transformer fine-tuning, domain adaptation and parameter-efficient adapters, to LLM prompting. F1 went from 0.885 to 0.924.

The dataset

100k synthetic samples generated by Llama 3.1 8B, Gemma 2 9B and Mixtral 8x7B, plus 200 real annotated examples. Classes are balanced at 25% in every split. No missing or duplicated rows.

Two things showed up before any modelling:

Sentiment polarity tracks politeness monotonically, in the real data as well as the synthetic.
Text length also tracks politeness, but only in the synthetic half. The 200 real samples show no such pattern.

Box plots of sentiment polarity by politeness level, synthetic versus real data — Polarity rises with politeness in both halves of the dataset. The real samples (right) have much wider spread.

Box plots of text length by politeness level, synthetic versus real data — Text length by label. The median climbs steadily with politeness in the synthetic half (left); in the real samples (right) the ordering breaks.

Phase 1: traditional ML

Bag-of-words, TF-IDF and dense Word2Vec features, crossed with dimensionality reduction, digit handling and three classifiers: 42 feature/model configurations, each swept over dozens of hyper-parameters. Pre-processing (lowercase, strip non-alphanumerics, POS-tagged lemmatisation, optional stopword removal) cut the vocabulary from 10,943 to 6,811 tokens.

Word2Vec embeddings projected to 3D with UMAP in TensorBoard show the axis being learned. In the full projection (left) the colouring splits the cloud top to bottom, with the impolite-leaning words gathered in the lower half. Zooming in (right), "happy" and "unhappy" fall on opposite sides of that split: "happy" sits among query, avid, inquire, "unhappy" among complain, irresponsible, forget, frankly.

3D visualisation of Word2Vec embeddings — Word2Vec embeddings, coloured by politeness polarity.

Plotting the whole sweep against training time:

Validation accuracy against training time on a log scale for SVM, Naive Bayes and Logistic Regression — Every configuration in the sweep. Training time spans five orders of magnitude; accuracy spans eighteen points.

Best overall: SVM + Word2Vec, 89.6% validation accuracy at ~430 s (88.5% F1 on test).
Best under ten seconds: Logistic Regression + BoW bigrams, 88.4% at 5.0 s. Roughly 85× the compute for 1.2 extra points.
Naive Bayes trains in under a tenth of a second and plateaus near 79%.

Phase 2: transformers

Fine-tuned bert-base-uncased and roberta-base, sweeping learning rate (1e-6, 1e-5, 2e-5) and weight decay (0.0, 0.01, 0.1) over three epochs before training the winner for ten.

F1 score across nine hyper-parameter configurations for BERT and RoBERTa — The full sweep: all eighteen runs land between 0.895 and 0.929 validation F1.

The sweep is almost flat. Learning rate is the only parameter that moved anything. 1e-6 underfits within the epoch budget, and 1e-5 and 2e-5 are indistinguishable. Weight decay makes no difference at any value, and BERT and RoBERTa track each other within half a point. Winners: RoBERTa at lr 2e-5 / wd 0.1, BERT at lr 2e-5 / wd 0.01.

Domain adaptation, an intermediate masked-language-modelling pass over the Polite Guard corpus (mask probability 0.1, two epochs), pushed RoBERTa from 91.9% to 92.4% F1, matching the dataset's published state of the art. It reached 1.49 perplexity on that corpus.

PEFT saved training time; only LoRA held the accuracy.

Backbone	Adapter	Δ F1 vs full fine-tune	Δ training time
RoBERTa	LoRA	−1.91%	−25.3%
RoBERTa	IA3	−7.35%	−24.4%
BERT	LoRA	−2.25%	−26.2%
BERT	IA3	−15.82%	−26.0%

All four adapters save the same ~25%, so IA3 is strictly worse here. It also costs BERT seven times more F1 than RoBERTa, so the adapter choice depends on the backbone.

F1 score versus training time — F1 against training time for full fine-tuning, domain adaptation, LoRA and IA3.

Where the errors are

The classes are ordinal, so errors are worth reading as distances. Reordering the confusion matrix along the politeness spectrum (impolite, neutral, somewhat polite, polite) puts them near the diagonal.

Confusion matrices for SVM and for RoBERTa with domain adaptation, classes ordered along the politeness spectrum — SVM + Word2Vec (left) and RoBERTa + domain adaptation (right) on the same 10,200-sample test split.

Splitting the errors by how far they miss:

	SVM + Word2Vec	RoBERTa + DA	Change
Total errors	1,175	775	−34%
Off by one level	745	653	−12%
Off by two or more	430	122	−72%

Almost all of the gain sits in the bottom row. Both models miss by one level about as often. The difference is in the bigger misses, which RoBERTa cuts by 72%.

Among genuinely polite texts, the SVM labels 200 as neutral (two levels off) and 23 as impolite (three levels off). RoBERTa cuts those to 51 and 5. At the other end of the scale, the SVM calls eight impolite texts polite; RoBERTa never does.

Per-class F1:

Class	SVM + Word2Vec	RoBERTa + DA	Δ
Impolite	0.980	0.990	+0.010
Neutral	0.879	0.920	+0.041
Somewhat polite	0.859	0.870	+0.011
Polite	0.821	0.916	+0.095

Somewhat polite is the hardest class for both models and stays that way, gaining 0.011 from four extra orders of magnitude of compute. The residual error sits on the boundary that is fuzzy to begin with.

The SVM's failures are bag-of-words failures, on mixed-tone text where polite phrasing outweighs a harsh frame:

"Are you really trying to return a toy that's been opened and played with? Our return policy is clear - items must be in their original packaging with all tags attached. If you'd like to make a purchase, I'd be happy to help you find something else."

Labelled impolite; the SVM sees happy to help and find something else and predicts polite. RoBERTa's remaining errors are closer calls:

"It's a nice awareness that i learnt about this"

Labelled polite; predicted somewhat polite.

Phase 3: prompting LLMs

Llama 3 8B, Mistral 7B Instruct v0.3 and Gemini 2.0 Flash, over zero-shot, few-shot and few-shot chain-of-thought prompts, on an 800-sample subset of the test set.

Weighted F1 for three LLMs across three prompting strategies — Nine model/prompt combinations spanning 0.31 to 0.74 weighted F1. None reaches the SVM baseline.

Model	Zero-shot	Few-shot	Few-shot CoT
Mistral 7B Instruct v0.3	0.64	0.74	0.69
Llama 3 8B	0.42	0.34	0.67
Gemini 2.0 Flash	0.54	0.31	0.57

From that grid:

Few-shot examples hurt two of three models. Adding examples cost Gemini 23 points and Llama 8 against zero-shot, and few-shot is where the models spread widest, from 0.31 to 0.74. Chain-of-thought recovered both; on Mistral, the one model few-shot helped, CoT was slightly worse.
Prompt choice swings a single model by up to 33 points (Llama, 0.34 → 0.67), so comparing these models at one fixed prompt says very little.
Gemini 2.0 Flash, the strongest general-purpose model in the set, came last at two of three prompt styles. Mistral 7B won all three.

The best prompted result, Mistral few-shot at 0.74, is 18 points behind the fine-tuned RoBERTa and 14 behind the SVM.

Results

Model	Training method	F1
RoBERTa	Domain adaptation (MLM) + fine-tune	0.924
RoBERTa	Full fine-tuning	0.918
RoBERTa	LoRA	0.901
SVM	Word2Vec	0.885
Mistral 7B Instruct	Few-shot prompting	0.74

We cross-validated only the best configuration, given the cost, and redid the MLM pass independently inside every fold. Pre-training once over the full corpus and then splitting would have leaked validation text into the adapted weights.