Twitch Chat Analysis

Posted by Kalun Ho on 28.07.2021
Card Image

Overview

This project summarizes the chat messages of a live stream on Twitch by utilizing the current state-of-the-art Natural Language Processing (NLP) models. Specifically, the Twitch API is used to collect the chat messages from users first and the data are stored locally. Different pre-trained NLP models are used to create a visualization of the data in a way that similar sentences based on its semantics are closed to each others. Furthermore, a classification model assigns each message with one out of five categories, which is defined my me. An example is shown in Figure above on the bottom right corner.

Generate Dataset

There are in total three different models in use. The first model is a simple Naive Bayes classifier, pre-trained on the nps chat corpus. This model is then fine-tuned on a small, manually labeled dataset with approximately 500 labeled data, which are collected from Twitch. With this initial classifier, more than 11k real-world chat messages with labels are generated in order to fine-tune the second, better performing model. After cross-checking the generated dataset manually, a pre-trained BERT [1] (model 2) is fine-tuned, achieving a classification accuracy of 82% using a 90%/10% split.

Deploying Models

Card Image

New data from Twitch are feed forwarded into two different BERT models (model 2 and 3). Model 2 predicts the class of the chat message. A pre-trained Sentence BERT [2] (model 3) maps the message to a 768-dimensional vector (embedding). The dimensions are then reduced to 2-dimensions using Umap [3] in order to visualize it in a scatter plot (Demo 1). I also create a graph of frequent occurring words in a sentences based on the same dataset (Demo 2) :)

Literature

[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[2] Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
[3] McInnes, L., Healy, J., & Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.