Nicolaou, C;
Vaidya, A;
Dzogang, F;
Wardrope, D;
Konstantinidis, N;
(2019)
"Where is My Parcel?" Fast and Efficient Classifiers to Detect User Intent in Natural Language.
In:
Proceedings of the 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS).
(pp. pp. 351-356).
IEEE: Granada, Spain.
Preview |
Text
Konstantinidis_Intent_Classification_IEEE.pdf - Accepted Version Download (353kB) | Preview |
Abstract
We study the performance of customer intent classifiers designed to predict the most popular intent received through ASOS.com Customer Care Department, namely “Where is my order?”. These queries are characterised by the use of colloquialism, label noise and short message length. We conduct extensive experiments with twowell established classification models: logistic regression via n-grams to account for sequences in the dataand recurrent neural networks that perform the extraction of these sequential patterns automatically. Maintaining the embedding layer fixed to GloVe coordinates, a Mann-Whitney U test indicated that the F1 score on aheld out set of messages was lower for recurrent neural network classifiers than for linear n-grams classifiers (M1=0.828, M2=0.815; U=1,196, P=1.46e-20), unless all layers were jointly trained with all other network parameters (M1=0.831, M2=0.828, U=4,280, P=8.24e-4). This plain neural network produced top performance on a denoised set of labels (0.887 F1) matching with Human annotators (0.889 F1) and superior to linear classifiers (0.865 F1). Calibrating these models to achieveprecision levels above Human performance (0.93 Precision), our results indicate a small difference in Recall of 0.05 for the plain neural networks (training under 1hr), and 0.07 for the linear n-grams (training under 10min), revealing the latter as a judicious choice of model architecture in modern AI production systems.
| Type: | Proceedings paper |
|---|---|
| Title: | "Where is My Parcel?" Fast and Efficient Classifiers to Detect User Intent in Natural Language |
| Event: | 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS) |
| ISBN-13: | 978-1-7281-2946-4 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/SNAMS.2019.8931717 |
| Publisher version: | https://doi.org/10.1109/SNAMS.2019.8931717 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | text classification, customer intent, n-grams, conversational-AI, recurrent neural networks, word-embedding |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Physics and Astronomy |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10092817 |
Archive Staff Only
 |
View Item |
