F20AA Applied Text Analytics Coursework 1¶

UG_CwGroup 2¶

Authors:¶

Baber Jan - bj58¶

Gaurav Gosain - gg68¶

Muhammad Assad Khan - mk227¶

Ashab Uddin - mu15¶

Github Repository

Link to Demo Video

Link To this notebook as HTML

Link to this notebook uploaded on Colab

Github Repository for Sentimental Keywords

Environment Setup¶

Python environment should be Python 3.6

# !pip install -U pip setuptools wheel
# !pip install -U spacy
# !python -m spacy download en_core_web_sm
# !pip install textblob tweepy matplotlib nltk python-dotenv demoji pandas numpy tweet-preprocessor sklearn wordcloud seaborn tensorflow keras seaborn pyLDAvis
# !python -c "import nltk;nltk.download('vader_lexicon')"
# !pip install torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio===0.10.2+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
from textblob import TextBlob
import sys,os,time
import tweepy
import matplotlib.pyplot as plt
import json
from dotenv import dotenv_values
import demoji
import pandas as pd
import numpy as np
import preprocessor as pp
import re
import spacy
from spacy.tokenizer import Tokenizer
from spacy.lang.en import English
from nltk.stem import PorterStemmer
from nltk.stem.snowball import SnowballStemmer
from nltk.tokenize import word_tokenize
from IPython.display import clear_output
# from tqdm import tqdm
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
# Multinomial Naive Bayes
from sklearn.naive_bayes import MultinomialNB
# Gausian Naive Bayes
from sklearn.naive_bayes import GaussianNB
# Categorical Naive Bayes
from sklearn.naive_bayes import CategoricalNB
# Bernoulli Naive Bayes
from sklearn.naive_bayes import BernoulliNB
# K-Nearest Neighbors
from sklearn.neighbors import KNeighborsClassifier
# SVM
from sklearn.svm import SVC
# Linear Model
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier
# Grid Search
from sklearn.model_selection import GridSearchCV
# Random Forest
from sklearn.ensemble import RandomForestClassifier
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, f1_score, precision_score, recall_score
import seaborn as sns
from keras.models import Sequential
from keras.layers import Dense, Embedding, LSTM, SpatialDropout1D, Bidirectional
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.utils.np_utils import to_categorical
from IPython.display import display, Markdown
import threading
from sklearn.model_selection import StratifiedKFold
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from wordcloud import WordCloud, STOPWORDS
from PIL import Image
import warnings
import torch
import torch.nn.functional as F
import torch.nn as nn
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import pyLDAvis
import pyLDAvis.sklearn

pyLDAvis.enable_notebook()

# tqdm.pandas(desc='Progress')
warnings.filterwarnings("ignore")

def printmd(string):
    display(Markdown(string))

Data Collection¶

Our strategy for data collection from twitter was to use an inclusive search term, which comprises of obvious keywords such as #EXPO2020 and some specialised keywords to retrieve more unique tweets. Using an environment file to store our twitter API keys for security purposes and establishing a connection with the twitter API, we then used the BeautifulSoup library to scrape Wikipedia so we can retrieve a list of all pavillions being exhibited at the Expo.

We then found a Github Repository of sentiment keywords containing both positive and negative sentiments. The list of approximately 2000 words enabled us to better filter out spam and unrelated tweets. Apart from Expo2020, DubaiExpo, UAEEXPO, DubaiExpo2020 and UAEEXPO2020, our search term included the names of Expo Pavillions and the sentiment keywords. This search term would fetch tweets with the specific keywords, or fetch tweets with the names of Expo Pavillions which contained sentimental keywords. This approach would retrive tweets with the keywords either present in the text of the tweet or as a #, it also filtered mutiple unrelated, spam tweets and enabled tweets only in the english language to be retrieved.

The usage of the filter option enabled us to filter out retweets so we can get more unique tweets and the option of using 'extended' mode gave us the entire tweet instead of just the characters up until a certain limit. To ensure all filtering of media we would only store the full text and # of a tweet, thus non of our tweets contain any sort of media such as audios, videos, photos or gifs.. The issue however was that the twitter API would have a limit of 1000 tweets per 15 minutes so after fetching 1000 tweets, our code would halt for 15 minutes before continuing to fetch tweets and storing them in a .json file. In the end we have a corpus of 95,000 tweets which once filtered gave us a corpus of 3854 tweets to label.

The following code is used to import the Keys from the environment file required to authorize the api access by tweepy¶

# Importing the keys #
config = dotenv_values(".env")
consumerKey = config['API_KEY']
consumerSecret = config['API_KEY_SECRET']
accessToken = config['ACCESS_TOKEN']
accessTokenSecret = config['ACCESS_TOKEN_SECRET']

# Establish the connection with API #
auth = tweepy.OAuthHandler(consumerKey, consumerSecret)
auth.set_access_token(accessToken, accessTokenSecret)
api = tweepy.API(auth,wait_on_rate_limit=True)
NoOfTerms = 10000

'''
Code to fetch and process the names of the pavillions from the 
wikipedia page for Expo 2020.
'''

from bs4 import BeautifulSoup
import requests
import re

tables_data = []
html_content = requests.get("https://en.wikipedia.org/wiki/Expo_2020").text
soup = BeautifulSoup(html_content, "html")
tables = soup.findAll('table')
for table in tables:
    data = []
    table_body = table.find('tbody')

    rows = table_body.find_all('tr')
    for row in rows:
        cols = row.find_all('td')
        cols = [ele.text.strip() for ele in cols]
        cols = [re.sub(r'\[.*\]', "", e) for e in cols]
        if len(cols) > 0:
            data.append(cols[0])  # Get rid of empty values
    tables_data.append(data)
pavilions = tables_data[1]
pavilions = [ f"({pavilion} pavilion)" for pavilion in pavilions]

json.dump(pavilions, open('pavilions.json', 'w'))

# read the jumbled keywords from the file keywordsJumbled.json and save it into an array
with open('keywordsJumbled.json') as json_file:
    keywordsJumbled = json.load(json_file)

with open('pavilions.json') as json_file:
    pavilions = json.load(json_file)

# So we are getting 1000 tweets for 5 keywords at once
sIndex = 0
fileO = "tweets.json"
with open(fileO, 'a') as outfile:
    outfile.write("[")

counterForTweetsFetched = 0
for j in range(0, len(pavilions), 5):
    for i in range(3392, 5088, 5):  # index goes here
        searchTerm = f"(Expo2020 OR DubaiExpo OR UAEEXPO OR DubaiExpo2020 OR UAEEXPO2020 OR {' OR '.join(pavilions[j:j+5])}) ({' OR '.join(keywordsJumbled[i:i+5]).replace('-','')}) -filter:retweets"
        tweepyTweetsGen = tweepy.Cursor(
                api.search_tweets,
                q=searchTerm,
                tweet_mode='extended',
                lang="en").items(NoOfTerms)
        tweets = []
        while(1):
            if (counterForTweetsFetched == 1000):
                counterForTweetsFetched = 0
                time.sleep(960)
            try: 
                tweet = next(tweepyTweetsGen)
                counterForTweetsFetched += 1
                tweets.append({
                    "body": tweet.full_text,
                    "tags": keywordsJumbled[i:i+5],
                    "countries":pavilions[j:j+5]
                    })

            except tweepy.TooManyRequests:
                counterForTweetsFetched = 0
                time.sleep(960)
                continue
            except :
                break
        print(j,i,len(tweets))

        if(tweets):
            with open(fileO, 'a') as outfile:
                output = json.dumps(tweets)
                outfile.write(output[1:-1])
                outfile.write(",")

with open(fileO, 'a') as outfile:
    outfile.write("{}]")

Below we are showing the raw tweets we have collected and show it as a dataframe

# read final_tweets.json
with open('final_tweets.json', encoding="utf8") as f:
    data = json.load(f)

df_unlabeled = pd.DataFrame.from_dict(data)

df_unlabeled

df_unlabeled.shape

(3870, 3)

df_unlabeled.describe()

df_unlabeled.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3870 entries, 0 to 3869
Data columns (total 3 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   body       3870 non-null   object
 1   countries  2902 non-null   object
 2   tags       2556 non-null   object
dtypes: object(3)
memory usage: 90.8+ KB

Labelling of Corpus¶

For labelling of the tweets, a comparison of 3 different methodologies were subsequently used to guage on which approach would accurately label each tweet's sentiment. The labels used were Positive, Negative, Neutral, Spam and Hate which was then merged with our Negative class. Our original decision was to label approximately 20% to 30% of our corpus manually, from which we would then guage whether the labelling from TextBlob and Vader was accurate enough to automate the process. TextBlob returned an accuracy of 63.78% and Vader returned an accuracy of 66.17% on our manually labelled set of tweets. The results were deemed inadequet, hence our conclusion was that all tweets were to be labelled manually to ensure all labels were correct. To assist in labelling tweets manually, a website was built using the React JavaScript library and Firebase database to store our tweets. This helped us label tweets easily and in an efficient manner.

Click here to view our website

Once we finish labelling, we load our labelled tweets from our Firebase database in json format into a dataframe. We drop any tweets None type values leaving us with a total of 3854 tweets comprising of our 5 classes.

In the following cells we show our dataframe from tweets loaded from our completeLabelledTweets.json file which contains the tweet itself and the label.

# read final_tweets.json
with open('completeLabelledTweets.json', encoding="utf8") as f:
    data = json.load(f)

df = pd.DataFrame.from_dict(data)

df = df[['body', 'label']]

df.dropna(inplace=True)

df

df.shape

(3854, 2)

df.describe()

df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 3854 entries, 0 to 3853
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   body    3854 non-null   object
 1   label   3854 non-null   object
dtypes: object(2)
memory usage: 90.3+ KB

Corpus Visualisation and Analysis¶

Now we analyse our dataframe of tweets and its labels, initial findings show the vast discrepancy between labels, especially between Positve compared to Neutal, Negative or Hate. Spam tweets despite our best efforts to curb the amount of spam tweets, the remainings ones were manually removed, our corpus from 3854 tweets is now 2802 tweets, with the 1052 spam tweets discarded which made up 27.3% of our corpus.

Further investigation enhances the discrepancy of our corpus, revealing the degree of positive skewness, this gives an initial impression on people's opinion about Expo2020. An overwhelming majority of tweets reflect positively on Expo2020, this finding will be tested through subjectivity analysis while building multiple ML models. 56% of our corpus is comprised of Positive tweets, 34.4% is Negative tweets and 9.6% of Negative tweets.

df['label'].value_counts().plot(
    kind='pie', startangle=90, figsize=(20, 10), autopct='%1.1f%%',)
plt.show()

printmd('### {} spam tweets'.format(df[df['label'] == 'Spam'].shape[0]))

print('Size of data before removing spam', df.shape[0])

#  filter out the tweets with the label "spam"
df = df[df['label'] != 'Spam']

df['label'].value_counts().plot(kind='pie', startangle=90, figsize=(20, 10), autopct='%1.1f%%',)
plt.show()

# map the label to 1 = "positive" and -1 = "negative" and 0 = "neutral"
df.loc[:, 'label'] = df['label'].map(
    {'Positive': 1, 'Negative': -1, 'Neutral': 0, 'Hate': -1})

df.reset_index(drop=True, inplace=True)

print('Size of data after removing spam', df.shape[0])

Size of data before removing spam 3854

Size of data after removing spam 2802

manually_labeled_counts = [value for value in df['label'].value_counts()]

def percentage(part, whole):
    return 100 * float(part)/float(whole)

whole = len(df)

manually_labeled_labels = [f'Positive [{percentage(manually_labeled_counts[0], whole):.2f}%]',
                           f'Neutral [{percentage(manually_labeled_counts[1], whole):.2f}%]',
                           f'Negative [{percentage(manually_labeled_counts[2], whole):.2f}%]']

def plot_pie_distribution(labels, colors, counts, title):
    plt.figure(figsize=(10,10), facecolor='white')
    patches, texts = plt.pie(counts, colors=colors, startangle=90)
    plt.legend(patches, labels, loc="best")
    plt.title(title, fontsize=14)
    plt.axis('equal')
    # plt.tight_layout()
    plt.show()

# Create variables to hold the average polarity #

positive = 0
negative = 0
neutral = 0
polarity = 0

correct_labels = 0

for i in range(len(df)):
    analysis = TextBlob(df["body"][i])
    polarity += analysis.sentiment.polarity

    if(analysis.sentiment.polarity == 0):
        neutral += 1
        if (df["label"][i] == 0):correct_labels += 1

    elif(analysis.sentiment.polarity < 0.00):
        negative += 1
        if (df["label"][i] == -1):correct_labels += 1

    elif(analysis.sentiment.polarity > 0.00):
        positive += 1
        if (df["label"][i] == 1):correct_labels += 1

positive = format(percentage(positive, len(df)), '.2f')
negative = format(percentage(negative, len(df)), '.2f')
neutral = format(percentage(neutral, len(df)), '.2f')
polarity = percentage(polarity, len(df))

## Print the Pie Chart ##

labels = ['Positive ['+str(positive)+'%]',
          'Neutral ['+str(neutral)+'%]',
          'Negative ['+str(negative)+'%]']

sizes = [positive, neutral, negative]
colors = ['yellowgreen', 'gold', 'red']

plot_pie_distribution(manually_labeled_labels, colors, manually_labeled_counts, "Manually Labeled Data")
plot_pie_distribution(labels, colors, sizes, "TextBlob Labeled Data")

# Calculate the accuracy of textblob's labeling

printmd(f"# Accuracy of TextBlob's auto-generated labels compared to manually labeled data is __{percentage(correct_labels, len(df)):.2f}%__")

sid = SentimentIntensityAnalyzer()

# Create variables to hold the average polarity #
positive = 0
negative = 0
neutral = 0

correct_labels = 0

for i in (range(len(df))):
    analysis = sid.polarity_scores(df["body"][i])
    polarity = analysis['compound']

    if(polarity == 0):
        neutral += 1
        if (df["label"][i] == 0):correct_labels += 1

    elif(polarity < 0.00):
        negative += 1
        if (df["label"][i] == -1):correct_labels += 1

    elif(polarity > 0.00):
        positive += 1
        if (df["label"][i] == 1):correct_labels += 1

positive = format(percentage(positive, len(df)), '.2f')
negative = format(percentage(negative, len(df)), '.2f')
neutral = format(percentage(neutral, len(df)), '.2f')

## Print the Pie Chart ##

labels = ['Positive ['+str(positive)+'%]',
          'Neutral ['+str(neutral)+'%]',
          'Negative ['+str(negative)+'%]']

sizes = [positive, neutral, negative]
colors = ['yellowgreen', 'gold', 'red']

plot_pie_distribution(manually_labeled_labels, colors, manually_labeled_counts, "Manually Labeled Data")
plot_pie_distribution(labels, colors, sizes, "TextBlob Labeled Data")

# Calculate the accuracy of nltk's vaders labeling

printmd(
    f"# Accuracy of NLTK-Vaders' auto-generated labels compared to manually labeled data is __{percentage(correct_labels, len(df)):.2f}%__")

While using Vader to label our tweets, it could only correctly label 66.17% of the tweets, while better than TextBlob, it is insufficient which led to the conclusion of manually labelling our tweets using our website.

df

def clean_tweet(current_tweet):
    emoji_list = demoji.findall(current_tweet)
    cleaned_tweet = pp.clean(current_tweet.replace("#", ""))

    cleaned_tweet = re.sub(r'\W', ' ', cleaned_tweet)

    # removes words starting with @ and words within & and ; (html tags)
    cleaned_tweet = re.sub(
        r'((\b@[\w]*) | (&[\w]+;))', ' ', cleaned_tweet)

    # remove all non ascii characters
    cleaned_tweet = re.sub(r'[^\x00-\x7f]', r' ', cleaned_tweet)

    # remove words containing only numbers or starting with numbers
    # cleaned_tweet = re.sub(r'\b[0-9]+.*\b', '', cleaned_tweet)
    cleaned_tweet = re.sub(r'\b[0-9]+\b', ' ', cleaned_tweet)

    # replace emojis with their text form
    for emoji in emoji_list.keys():
        cleaned_tweet = cleaned_tweet.replace(
            emoji, f' {emoji_list[emoji].replace(" ", "")} ')

    # remove all single characters
    cleaned_tweet = re.sub(r'\s+[a-zA-Z]\s+', ' ', cleaned_tweet)

    # Substituting multiple spaces with single space
    cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

    cleaned_tweet = cleaned_tweet.lower()

    return cleaned_tweet

df['clean_body'] = df['body'].apply(clean_tweet)

def make_wordcloud(df, label):
    words = ' '.join(df[df['label'] == label]['clean_body'])

    mask=np.array(Image.open("mask.png"))

    wordcloud = WordCloud(width=mask.shape[1], height=mask.shape[0], background_color='white',
                          mask=mask, stopwords=STOPWORDS, min_font_size=5).generate(words)

    # plot the WordCloud image
    plt.figure(figsize=(15, 15), facecolor=None)
    plt.imshow(wordcloud)
    plt.axis("off")
    plt.tight_layout(pad=0)

    plt.show()


printmd('## Word Cloud for Positive Tweets')
make_wordcloud(df, 1)

printmd('## Word Cloud for Negative Tweets')
make_wordcloud(df, -1)

printmd('## Word Cloud for Neutral Tweets')
make_wordcloud(df, 0)

df.drop(['clean_body'], axis=1, inplace=True)

Pipeline Creation¶

Our sentiment analysis pipeline contains 3 stages, first stage is our Normaliser, second is the Vectoriser and the third is the Classifier after which each of the 1282 possible models are then critically evaluated on a different set of metrics to judge which model performs best, amongst which two are deep learning models. The reason for 1282 models is further explained , but due to the different range of N-Gram features from 1-4, different permutations of Lemmatization and Stemming and different corpus representations

Starting with the Normaliser, this stage of our pipeline begins by first cleaning tweets which takes place over multiple sub-stages. Firstly, all URLs included in the body of the tweet are removed using the preprocessor library. Next we remove the @ symbol infront of usernames which is followed by removing the # symbol from the hashtags but preserving the text. Then using Regex we find any numerical strings in the tweets or any strings which begin with a numerical value and remove them, this helps in filtering out mobile phone numbers or other random tweets. All HTML tags are also removed and then using Regex again, we remove any non ASCII characters as some tweets in english also contained strings from other languages which have must be removed. We decided on this preprocessing method after labelling our tweets and noticing patterns on how the # symbol was used not just for tags but also to give meaning in the tweets. Numerical strings in the tweets were used as phone numbers for promotional purposes such as registering for events or contacting a service regarding the expo or used in threads to number tweets in order, this was not subjectively important to the sentiment of the tweet and were also removed. Some tweets which contained emojis had to be mapped to their meaning/sentiment which was achieved using the demoji library, next we test 4 possible combinations of Stemming and Lemmatizing:

Stemming only
Lemmatizing only
Stemming and then Lemmatizing
Lemmatizing and then Stemming

Our hypothesis, as mentioned in the lectures, is that stemming would be better and this comparison would either verify or nullify our hypothesis. Once we have either conducted stemming or lemmatizing or a permutation of both, all stop words and single character words from the resulting text is removed and any double spaces is converted into a single space. This entire process normalises our corpus of tweets and preprocesses them, the resulting corpus is now ready for the representation or Vectoriser stage.

class Normalizer(BaseEstimator, TransformerMixin):
    def __init__(self, options):

        self.verbose = False

        if isinstance(options, tuple):
            options, self.verbose = options

        self.nlp = spacy.load("en_core_web_sm")
        pp.set_options(pp.OPT.URL)
        if 'l' not in options and 's' not in options:
            print("Options: ls or sl or s or l")
            raise Error

        char_map = {'l': 'Lemmatization', 's': 'Stemming'}

        if self.verbose: printmd("## Using " + " + ".join([f"{char_map[option]}" for option in options]))

        self.options = options
        self.stemmer = SnowballStemmer(language='english')

    def lemmatize_tweet(self, current_tweet):
        lemmatized_tweet_tweet = []

        if self.verbose:
            printmd("## Lemmatizing Tweet")

        if type(current_tweet) != str:
            current_tweet = " ".join(current_tweet)
        doc = self.nlp(current_tweet)
        for token in doc:
            lemmatized_tweet_tweet.append(token.lemma_)
        
        if self.verbose:
            
            printmd(f'''
| **Tweet** | **Lemmatized Tweet** |
| --- | -- |
| {current_tweet} | {lemmatized_tweet_tweet} |
''')

        return lemmatized_tweet_tweet

    def stemitize_tweet(self, current_tweet):

        if self.verbose:
            printmd("## Stemming Tweet")
        
        stemitized_tweet_tweet = []
        if type(current_tweet) == str:
            current_tweet = current_tweet.split()
        for token in current_tweet:
            stemitized_tweet_tweet.append(self.stemmer.stem(token))

        if self.verbose:

            printmd(f'''
| **Tweet** | **Tweet after Stemming** |
| --- | -- |
| {current_tweet} | {stemitized_tweet_tweet} |
''')
        
        return stemitized_tweet_tweet

    def remove_stopwords(self, current_tweet):
        stopwords_removed_tweet = []
        for word in current_tweet:
            if word not in self.nlp.Defaults.stop_words:
                stopwords_removed_tweet.append(word)
        return stopwords_removed_tweet

    def clean_tweet(self, current_tweet):

        titles = []
        tweet_progress = []

        if(self.verbose):
            titles.append("Tweet before cleaning")
            tweet_progress.append(current_tweet.replace("\n", " "))
            printmd("## Cleaning Tweet")

        emoji_list = demoji.findall(current_tweet)

        if(self.verbose):
            printmd(f"### Found {len(emoji_list)} Emoji/(s) in the tweet")
            if len(emoji_list) > 0:
                printmd(f"### Emoji/s: {emoji_list}")       

        cleaned_tweet = pp.clean(current_tweet.replace("#", " "))

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after removing the \"#\" symbol and any links from the tweet")
            tweet_progress.append(cleaned_tweet)

        # removes words starting with @ and words within & and ; (html tags)
        cleaned_tweet = re.sub(
            r'((@[\w]*) | (&[\w]+;))', ' ', cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append(
                "Tweet after removing all words starting with @ and words within & and ; (html tags)")
            tweet_progress.append(cleaned_tweet)

            
        # replace emojis with their text form
        for emoji in emoji_list.keys():
            cleaned_tweet = cleaned_tweet.replace(
                emoji, f' {emoji_list[emoji].replace(" ", "")} ')

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after replacing emojis with their text form")
            tweet_progress.append(cleaned_tweet)    
        cleaned_tweet = re.sub(r'\W', ' ', cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after removing punctuation")
            tweet_progress.append(cleaned_tweet)

        # remove all non ascii characters
        cleaned_tweet = re.sub(r'[^\x00-\x7f]', r' ', cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after removing all non ascii characters")
            tweet_progress.append(cleaned_tweet)

        # remove words containing only numbers or starting with numbers
        # cleaned_tweet = re.sub(r'\b[0-9]+.*\b', '', cleaned_tweet)

        # remove words containing only numbers
        cleaned_tweet = re.sub(r'\b[0-9]+\b', ' ', cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after removing words containing only numbers")
            tweet_progress.append(cleaned_tweet)

        # remove all single characters
        cleaned_tweet = re.sub(r'\s+[a-zA-Z]\s+', ' ', cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after removing all single characters")
            tweet_progress.append(cleaned_tweet)

        # Substituting multiple spaces with single space
        cleaned_tweet = re.sub(r'\s+', ' ', cleaned_tweet, flags=re.I)

        if(self.verbose):
            titles.append("Tweet after substituting multiple spaces with single space")
            tweet_progress.append(cleaned_tweet)

        cleaned_tweet = cleaned_tweet.lower()

        if(self.verbose):
            titles.append("Tweet after converting to lower case")
            tweet_progress.append(cleaned_tweet)

        s = "| Action | Tweet (Result) |\n| --- | --- |\n"

        if(self.verbose):
            for i in range(len(titles)):
                s += f"| **{titles[i]}** | {tweet_progress[i]} |\n"

            printmd(s)


        return cleaned_tweet

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        lemmatize_tweet_list = []
        for datum in X.iterrows():
            # Clean the tweet
            cleaned_tweet = self.clean_tweet(current_tweet=datum[1]['body'])

            # Check for the options
            for c in self.options:
                if c == "l":
                    # Lematize the tweet
                    cleaned_tweet = self.lemmatize_tweet(
                        current_tweet=cleaned_tweet)
                elif c == 's':
                    cleaned_tweet = self.stemitize_tweet(
                        current_tweet=cleaned_tweet)

            # Remove stop words
            tweet_Without_Stop_Words = self.remove_stopwords(cleaned_tweet)
            # As this is a list, join to make a string again
            normalized_Tweet = " ".join(tweet_Without_Stop_Words)
            # Append tweet to the lematize_tweet_list
            lemmatize_tweet_list.append(normalized_Tweet)

            if self.verbose:
                printmd(f"## Tweet after Normalization \n### {normalized_Tweet}\n---\n")

        X = lemmatize_tweet_list
        return X


# List of Machine Learning Models to be trained and tested
modelAlgo = [
    (MultinomialNB, {}),
    (BernoulliNB, {}),
    (KNeighborsClassifier, {"algorithm": 'auto', "leaf_size": 20,
     "n_neighbors": 5, "p": 2, "weights": 'uniform', "n_jobs": -1}),
    (SVC, {"gamma": 'scale', "kernel": 'sigmoid', "tol": 0.1}),
    (LogisticRegression, {"max_iter": 1000, "penalty": 'none',
     "solver": 'lbfgs', "tol": 0.01, "n_jobs": -1}),
    (SGDClassifier, {"alpha": 0.0001, "loss": 'modified_huber',
     "penalty": 'l1', "tol": 0.0001, "n_jobs": -1}),
    (RandomForestClassifier, {"n_jobs": -1, "n_estimators": 125, "min_samples_split": 4,
     "min_samples_leaf": 2, "max_features": None, "max_depth": 35, "criterion": "entropy"}),
    (MLPClassifier, {"solver": "adam", "max_iter": 600, "learning_rate_init": 0.001,
     "hidden_layer_sizes": (50, 150, 200, 25), "activation": "relu", "batch_size": 250}),
]

# Vectorization Techniques
options = [
    ('l', 'Lemmatization'),
    ('s', 'Stemming'),
    ('ls', 'Lemmatization followed by Stemming'),
    ('sl', 'Stemming followed by Lemmatization')
]
# Number of words taken as a token for the vectorizer
nGramsList = range(1, 5)
modelsDataFrame = pd.DataFrame(
    columns=['Algorithim', 'Ngram Range', 'Vectorizer', 'Normalizing Technique'])
for algo, arguments in modelAlgo:
    for m in nGramsList:
        for n in range(m, 5):
            for option, option_description in options:
                for vectorizer in [CountVectorizer, TfidfVectorizer]:
                    modelsDataFrame.loc[len(modelsDataFrame)] = [
                        algo.__name__, f'{m}-{n}', vectorizer.__name__, option_description]
display(modelsDataFrame)

# _ = Normalizer(('ls', True)).fit_transform(df.iloc[20:21])
_ = Normalizer(('ls', True)).fit_transform(df.iloc[333:334])

Vectoriser is the second stage of our pipeline in which we will be comparing 2 representations, the TFIDF representation using Sklearn TfidfVectoriser, the second representation is Bag of words (BoW) using Sklearn CountVectoriser. TFIDF being the gold standard in information retrieval as it weighs importance of each term in the corpus BoW is the simplest representation of text for machine learning yet highly effecient. A comparison of both will tell us which representation is optimal for our corpus.

The third stage of the pipeline is the Classifier which is a series of different Discriminative Models which will find the boundaries between our classes to correctly label them. The models include:

K-nearest Neighbour
Support Vector Classifier
Logistic Regression
Stochastic Gradient Descent
Random Forest Classifier
Multi-Layer Perceptron Classifier
Long Short-Term Memory (LSTM)

In Generative Models we will be using:

Multinomial Naive Bayes
Bernoulli Naive Bayes

Each model will undergo extensive hyperparameter optimisation to fine tune each them to have the most optimal version of each model. From then each model will be compared with each other on the basis of multiple metrics such as F1 score, Precision, Recall and Accuracy.

Each model will run on each combination of representations (TFIDF and Bag of Words) and normalisation (Stemming and Lemmitizing). Each model will also run on 2 variations of our corpus. The first variation is of having a stratified data set where each label in the test and train split is in the same proportion as in the original data set, this will result in an unbalanced dataset as the corpus is positively skewed. The second variation is ensuring all labels are in equal amount so our model is trained on an equal set for each class, this would result in a small subset of the corpus to be used but the model will not have any inherent bias unlike the previous as this is a balanced data set.

Each of these models will also use a range of N-grams from Uni-gram to 6-gram and all combinations in between, thus generating 960 models in total and with 2 variations of our corpus, we will have 1920 models in total each with 10 fold cross validation to find the optimal model.

Model Optimisation¶

Before finding the best model, first we must tune each model for optimum perfomance, thus first we must optimse each model's hyperparameters for which we used Sklearn GridSearchCV which will test every possible combination of hyperparameters in a defined hyperparameter search space, to search for a globally optimal solution. Due to being very computationally expensive, this was conducted on University system's.

To be able to have an unbiased comparison between models while ensuring each model performance to its optimal potential, thus first we must optimse each model's hyperparameters for which we used Sklearn GridSearchCV which will test every possible combination of hyperparameters in a defined hyperparameter search space, to search for a globally optimal solution. GridSearchCV will search for each individual model, their optimal hyperparameters for our corpus, to optimise each model's performance and validation which was tested with 10 fold cross validation for each possible combination of hyperparameters for each model. Each model was judged on the basis of their F1 score .Due to the massive amount of computational power required, the GridSearchCV algorithm was conducted on University systems with multi-threading to ensure efficiency while retrieving results for each of our models.

param_grid = {'kernel': ['poly', 'rbf', 'sigmoid'], 'gamma': [
    'scale', 'auto'], 'tol': [1e-2, 1e-3, 1e-4, 1e-1, 1e-5]}

gridcv = GridSearchCV(SVC(), param_grid, scoring='f1_macro', n_jobs=-1, cv=10)

gridcv.fit(result, df['label'])

print("SVC")

print(gridcv.best_params_)

param_grid = {'penalty':['l1', 'l2', 'elasticnet', 'none'], 'solver': ['netwon-cg', 'sag', 'saga', 'lbfgs'], 'tol':[1e-2, 1e-3, 1e-4, 1e-1, 1e-5], 'max_iter':[1000]}

gridcv = GridSearchCV(LogisticRegression(), param_grid, scoring='f1_macro', n_jobs=-1, cv=10)

gridcv.fit(result, df['label'])

print('LogisticRegression')

print(gridcv.best_params_)

param_grid = {'loss': ['hinge', 'modified_huber', 'squared_hinge', 'perceptron', 'huber'], 'penalty': [
    'l1', 'l2', 'elasticnet'], 'alpha': [0.0001, 0.00001, 0.001, 0.01, 0.1], 'tol': [1e-2, 1e-3, 1e-4, 1e-1, 1e-5]}

gridcv = GridSearchCV(SGDClassifier(), param_grid, scoring='f1_macro', n_jobs=-1, cv=10)

gridcv.fit(result, df['label'])

print('SGDClassifier')

print(gridcv.best_params_)

param_grid = {'n_neighbors': list(range(3, 10)), 'weights': ['uniform', 'distance'], 'algorithm': ['auto'], 'p': [1, 2], 'leaf_size': list(range(20,40)), }

gridcv = GridSearchCV(KNeighborsClassifier(), param_grid,
                      scoring='f1_macro', n_jobs=-1, cv=10)

gridcv.fit(result, df['label'])

print('KNeighborsClassifier')

print(gridcv.best_params_)


param_grid = {'n_estimators': list(range(100, 300, 25)), 'criterion': ['gini', 'entropy'], 'max_depth': list(
    range(15, 40)), 'min_samples_split': [2, 3, 4, 5, 6], 'min_samples_leaf': [1, 2, 3], 'max_features': ['auto', 'log2', None]}

gridcv = GridSearchCV(RandomForestClassifier(), param_grid,
                      scoring='f1_macro', n_jobs=-1, cv=10)

gridcv.fit(result, df['label'])

print('RandomForestClassifier')

print(gridcv.best_params_)

param_grid = {'activation': ['relu', 'tanh'], 'solver': ['sgd', 'adam'], 'batch_size': [
    'auto', 50, 100, 250, 300], 'learning_rate_init': [0.01, 0.001, 0.0001, 0.00001], 'max_iter': [200, 400, 600, 800], 
    'hidden_layer_sizes': [(50, 100, 150), (50, 150, 200, 25), (200, 300, 75, 30), (75, 75, 300, 600, 100), (200, 500, 1000, 750, 300, 100)]}


gridcv = GridSearchCV(MLPClassifier(), param_grid,
                      scoring='f1_macro', n_jobs=-1, cv=10)


gridcv.fit(result, df['label'])


print('MLPClassifier')


print(gridcv.best_params_)

Evaluation Methodology¶

For model evaluation, we would be generating a confusion matrix to find out the number of True Positives, True Negatives, False Positives and False Negatives. From this we will calculate Recall score, Precision score and F1 score which is the most important evaluation metric in an unbalanced data set. The F1 for the stratified version of training we will be using F1 score with macro average due to the skewed nature of our corpus and each sample being equally important while for the variation with equal number of samples per class we will use micro average as our corpus is no longer unbalanced. Accuracy is also an important metric for our balanced version of the corpus since will be no inherrent biases while training the model.

The best performing model will be the model with the overall best accuracy and F1 score, and the preprocessing steps and hyperparameters for that model would verify or nullify our hypotheses regarding representation and normalisation. These steps and hyperparameter optimisation will gaurantee the most optimal model for our corpus given its heavily positively skewed nature.

Creating, Training and Testing different Machine Learning Models on the unblanced data set with various n-gram and vectorization techniques.¶

Using multi-threading, we run our models with their optimised hyperparameters so each model runs on an individual thread. We store print out the results and keep track of the best model which gave the best F1 score and accuracy. The same process is repeated for the balanced data set.

# Doing Multi threading because it will take alot of time
MAX_THREADS = 10
highest_f1_score_model = {"model":None, "score":0,"lock":threading.Lock()}
highest_accuracy_model = {"model":None, "score":0,"lock":threading.Lock()}

# Do the training and testing
# Creating the list to store the result of ewach model
result = {"list":[],"lock":threading.Lock()}

def trainingAndTesting(model,df,highest_f1_score_model,highest_accuracy_model,result,numberOFThreadsCreated,kfold):
    totalFScore = 0
    totalAccuracy = 0
    totalConfusion_matrix = None
    best_model = None
    local_highest_f1_score = 0
    for train_index, test_index in kfold.split(df[['body']], df['label']):
        X_train, X_test = df.iloc[train_index][['body']], df.iloc[test_index][['body']]
        y_train, y_test = df.iloc[train_index]['label'], df.iloc[test_index]['label']
        model['pipeline'].fit(X_train, y_train)
        y_pred = model['pipeline'].predict(X_test)
        totalAccuracy += accuracy_score(y_test, y_pred)
        totalFScore += f1_score(y_test, y_pred, average='macro')
        totalConfusion_matrix = totalConfusion_matrix + confusion_matrix(y_test, y_pred) if totalConfusion_matrix is not None else confusion_matrix(y_test, y_pred)
        if f1_score(y_test, y_pred, average='macro') > local_highest_f1_score:
            local_highest_f1_score = f1_score(y_test, y_pred, average='macro')
            best_model = model

    
    fscore = totalFScore/kfold.get_n_splits()
    acc_score = totalAccuracy/kfold.get_n_splits()
    confusion_matrix_result = totalConfusion_matrix/kfold.get_n_splits()
    highest_f1_score_model["lock"].acquire()
    if fscore > highest_f1_score_model.get("score"):
        highest_f1_score_model["score"] = fscore
        highest_f1_score_model["model"] = best_model
    highest_f1_score_model["lock"].release()
    
    highest_accuracy_model["lock"].acquire()
    if acc_score > highest_accuracy_model.get("score"):
        highest_accuracy_model["score"] = acc_score
        highest_accuracy_model["model"] = best_model
    highest_accuracy_model["lock"].release()
    
    result["lock"].acquire()
    best_model["pipeline"]=None
    result["list"].append({"model": best_model, "accuracy": acc_score,
                          "f1_score": fscore, "confusion_matrix": confusion_matrix_result})
    result["lock"].release()
    numberOFThreadsCreated["lock"].acquire()
    numberOFThreadsCreated["num"] -=  1
    numberOFThreadsCreated["lock"].release()
    
    
# 10 fold cross validation for each model
numberOfFolds = 5
kfold = StratifiedKFold(n_splits=numberOfFolds, shuffle=True, random_state=7)
numberOFThreadsCreated = {"num":0,"lock":threading.Lock()}
threads= []
models =0
for algo, arguments in (modelAlgo):
    for m in nGramsList:
        for n in range(m, 5):
            for option, option_description in options:
                for vectorizer in [CountVectorizer, TfidfVectorizer]:
                    pipeline = Pipeline([
                        ("normalizer", Normalizer(options=option)),
                        ("vectorizer", vectorizer(ngram_range=(m, n))),
                        ("classifier", algo(**arguments))
                    ])
                    model = {"pipeline": pipeline, "minNrange": m, "maxNrange": n,
                             "machineLearingAlgo": algo.__name__, "vectorizer": vectorizer.__name__, "options": option_description}
                    models+=1
                    while(True):
                        numberOFThreadsCreated["lock"].acquire()
                        if(numberOFThreadsCreated["num"] < MAX_THREADS):
                            numberOFThreadsCreated["lock"].release()
                            break
                        numberOFThreadsCreated["lock"].release()
                        time.sleep(10)
                    thread = threading.Thread(target=trainingAndTesting, args=(model,df,highest_f1_score_model,highest_accuracy_model,result,numberOFThreadsCreated,kfold))
                    numberOFThreadsCreated["lock"].acquire()
                    numberOFThreadsCreated["num"] += 1
                    thread.start()
                    numberOFThreadsCreated["lock"].release()
                    threads.append(thread)

# Wait for each model to finish to start printing
for thread in threads:
    thread.join()

printmd(
    f'We had {models} models which were trained and tested in parallel with {MAX_THREADS} threads.')
# Now will do printing
def prettyPrintModels(model):
    modelAlgoName = model["model"]["machineLearingAlgo"]
    vectorizerName = model["model"]["vectorizer"]
    optionsName = model["model"]["options"]
    minNrange = model["model"]["minNrange"]
    maxNrange = model["model"]["maxNrange"]
    if minNrange==maxNrange:
        nGramString = f'{minNrange} {"word" if minNrange==1 else "words as a feature for vectorization"}'
    else:
        nGramString = f'{minNrange}-{maxNrange} words as a feature for vectorization'
    printmd("## Trained and Tested  Model: " + modelAlgoName + 
            "\n\t - using " + optionsName + " for tokenization" +
            "\n\t - with " + vectorizerName + " as a vectorizer taking " + nGramString +
            "\n\t - without stratification on an unbalanced dataset")
    printmd("--"*10+"Results" + "--"*10)
    printmd(f"- Average Accuracy of {modelAlgoName} across {numberOfFolds}-folds = {model['accuracy']}")
    printmd(f"- Average F1-Score of {modelAlgoName} across {numberOfFolds}-folds = {model['f1_score']}")
    printmd(f"- Average Confustion Matrix of {modelAlgoName} across {numberOfFolds}-folds:")
    # print(model['confusion_matrix'])
    sns.heatmap(model['confusion_matrix'], annot=True)
    plt.show()


# Fist will sort the models based on the model's minimum ngram range and maximum ngram range
result["lock"].acquire()
result["list"].sort(key=lambda x: (x["model"]["minNrange"],x["model"]["maxNrange"]))
for model in result["list"]:
    prettyPrintModels(model)

# Print the best model 

printmd('---'*10)
result["lock"].release()

# Load all the json files in the directory
MAX_THREADS = 10
highest_f1_score_model = {"model":None, "score":0,"lock":threading.Lock()}
highest_accuracy_model = {"model":None, "score":0,"lock":threading.Lock()}


# Do the training and testing
# Creating the list to store the result of each model
result = {"list": [], "lock": threading.Lock()}
numberOfFolds =5
def prettyPrintModels(model):
    modelAlgoName = model["model"]["machineLearingAlgo"]
    vectorizerName = model["model"]["vectorizer"]
    optionsName = model["model"]["options"]
    minNrange = model["model"]["minNrange"]
    maxNrange = model["model"]["maxNrange"]
    if minNrange == maxNrange:
        nGramString = f'{minNrange} {"word" if minNrange==1 else "words as a feature for vectorization"}'
    else:
        nGramString = f'{minNrange}-{maxNrange} words as a feature for vectorization'
    printmd("## Trained and Tested  Model: " + modelAlgoName +
            "\n\t - using " + optionsName + " for tokenization" +
            "\n\t - with " + vectorizerName + " as a vectorizer taking " + nGramString +
            "\n\t - without stratification on an unbalanced dataset")
    printmd("--"*10+"Results" + "--"*10)
    printmd(
        f"- Average Accuracy of {modelAlgoName} across {numberOfFolds}-folds = {model['accuracy']}")
    printmd(
        f"- Average F1-Score of {modelAlgoName} across {numberOfFolds}-folds = {model['f1_score']}")
    printmd(
        f"- Average Confustion Matrix of {modelAlgoName} across {numberOfFolds}-folds:")
    sns.heatmap(model["confusion_matrix"], annot=True)
    plt.show()


outputs = []
for file in os.listdir("./Outputs"):
    if file.endswith(".json"):
        # Load the json file
        with open(os.path.join("./Outputs", file), "r") as f:
            model = json.load(f)
            with open(os.path.join("./Outputs", f"{file[:-5]}_confusion_matrix.npy"), "rb") as fc:
                #Load the confusion matrix
                model["confusion_matrix"] = np.load(fc)
            highest_accuracy_model["lock"].acquire()
            if model["accuracy"] > highest_accuracy_model.get("score"):
                highest_accuracy_model["score"] = model["accuracy"]
                highest_accuracy_model["model"] = model['model']
            highest_accuracy_model["lock"].release()

            highest_f1_score_model["lock"].acquire()
            if model["f1_score"] > highest_f1_score_model.get("score"):
                highest_f1_score_model["score"] = model["f1_score"]
                highest_f1_score_model["model"] = model['model']
            highest_f1_score_model["lock"].release()
            outputs.append(model)
outputs.sort(key=lambda x: (x["model"]["minNrange"],x["model"]["maxNrange"]))
result["lock"].acquire()
result["list"] = outputs
result["lock"].release()
for model in outputs:
    prettyPrintModels(model)
printmd('---'*10)

 - using Lemmatization followed by Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming followed by Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization followed by Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming followed by Lemmatization for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization followed by Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

result["lock"].acquire()
# result_balanced["list"].sort(key=lambda x: (
#     x["model"]["machineLearingAlgo"], x["model"]["minNrange"], x["model"]["maxNrange"]))
result["list"].sort(key=lambda x: (
    x["f1_score"]))
resultsDataFrameUnBalanced = pd.DataFrame(
    columns=['Algorithim', 'Ngram Range', 'Vectorizer', 'Normalizing Technique', 'Accuracy', 'F1-Score'])
for model in reversed(result["list"]):
    resultsDataFrameUnBalanced.loc[len(resultsDataFrameUnBalanced)] = [model["model"]["machineLearingAlgo"],
                                                                       f'{model["model"]["minNrange"]}-{model["model"]["maxNrange"]}',
                                                                       model["model"]["vectorizer"],
                                                                       model["model"]["options"],
                                                                       model["accuracy"],
                                                                       model["f1_score"]]
result["lock"].release()
resultsDataFrameUnBalanced

Creating, Training and Testing different Machine Learning Models on the stratified balanced data set with various n-gram and vectorization techniques.¶

# Stratifying

'''
The following line of code is used to stratify the dataset.

It keeps the same proportion of each class in the dataset. To achieve this it uses the minimum count value of the
classes in the dataset.
'''
df_balanced = df.groupby('label').apply(
    lambda x: x.sample(n=df['label'].value_counts().min()))


# Doing Multi threading because it will take alot of time
MAX_THREADS = 10
highest_f1_score_model_BalancedDataSet = {
    "model": None, "score": 0, "lock": threading.Lock()}
highest_accuracy_model_BalancedDataSet = {
    "model": None, "score": 0, "lock": threading.Lock()}



# Do the training and testing
# Creating the list to store the result of each model
result_balanced = {"list": [], "lock": threading.Lock()}


def trainingAndTesting(model, df, highest_f1_score_model_BalancedDataSet, highest_accuracy_model_BalancedDataSet, result_balanced, numberOFThreadsCreated, kfold):
    totalFScore = 0
    totalAccuracy = 0
    totalConfusion_matrix = None
    local_highest_accuracy_score = 0
    best_model = None
    for train_index, test_index in kfold.split(df[['body']], df['label']):
        X_train, X_test = df.iloc[train_index][[
            'body']], df.iloc[test_index][['body']]
        y_train, y_test = df.iloc[train_index]['label'], df.iloc[test_index]['label']
        model['pipeline'].fit(X_train, y_train)
        y_pred = model['pipeline'].predict(X_test)
        totalAccuracy += accuracy_score(y_test, y_pred)
        totalFScore += f1_score(y_test, y_pred, average='macro')
        totalConfusion_matrix = totalConfusion_matrix + confusion_matrix(
            y_test, y_pred) if totalConfusion_matrix is not None else confusion_matrix(y_test, y_pred)
        if local_highest_accuracy_score < accuracy_score(y_test, y_pred):
            local_highest_accuracy_score = accuracy_score(y_test, y_pred)
            best_model = model

    fscore = totalFScore/kfold.get_n_splits()
    acc_score = totalAccuracy/kfold.get_n_splits()
    confusion_matrix_result = totalConfusion_matrix/kfold.get_n_splits()
    highest_f1_score_model_BalancedDataSet["lock"].acquire()
    if fscore > highest_f1_score_model_BalancedDataSet.get("score"):
        highest_f1_score_model_BalancedDataSet["score"] = fscore
        highest_f1_score_model_BalancedDataSet["model"] = best_model
    highest_f1_score_model_BalancedDataSet["lock"].release()

    highest_accuracy_model_BalancedDataSet["lock"].acquire()
    if acc_score > highest_accuracy_model_BalancedDataSet.get("score"):
        highest_accuracy_model_BalancedDataSet["score"] = acc_score
        highest_accuracy_model_BalancedDataSet["model"] = best_model
    highest_accuracy_model_BalancedDataSet["lock"].release()

    result_balanced["lock"].acquire()
    result_balanced["list"].append({"model": best_model, "accuracy": acc_score,
                                    "f1_score": fscore, "confusion_matrix": confusion_matrix_result})
    result_balanced["lock"].release()
    numberOFThreadsCreated["lock"].acquire()
    numberOFThreadsCreated["num"] -= 1
    numberOFThreadsCreated["lock"].release()


# 10 fold cross validation for each model
numberOfFolds = 5
kfold = StratifiedKFold(n_splits=numberOfFolds, shuffle=True, random_state=7)
numberOFThreadsCreated = {"num": 0, "lock": threading.Lock()}
threads = []
models = 0
for algo, arguments in (modelAlgo):
    for m in nGramsList:
        for n in range(m, 5):
            for option, option_description in options:
                for vectorizer in [CountVectorizer, TfidfVectorizer]:
                    pipeline = Pipeline([
                        ("normalizer", Normalizer(options=option)),
                        ("vectorizer", vectorizer(ngram_range=(m, n))),
                        ("classifier", algo(**arguments))
                    ])
                    model = {"pipeline": pipeline, "minNrange": m, "maxNrange": n,
                             "machineLearingAlgo": algo.__name__, "vectorizer": vectorizer.__name__, "options": option_description}
                    models += 1
                    while(True):
                        numberOFThreadsCreated["lock"].acquire()
                        if(numberOFThreadsCreated["num"] < MAX_THREADS):
                            numberOFThreadsCreated["lock"].release()
                            break
                        numberOFThreadsCreated["lock"].release()
                        time.sleep(10)
                    thread = threading.Thread(target=trainingAndTesting, args=(
                        model, df_balanced, highest_f1_score_model_BalancedDataSet, highest_accuracy_model_BalancedDataSet, result_balanced, numberOFThreadsCreated, kfold))
                    numberOFThreadsCreated["lock"].acquire()
                    numberOFThreadsCreated["num"] += 1
                    thread.start()
                    numberOFThreadsCreated["lock"].release()
                    threads.append(thread)

# Wait for each model to finish to start printing
for thread in threads:
    thread.join()

printmd(
    f'We had {models} models which were trained and tested in parallel with {MAX_THREADS} threads.')

# Now will do printing


def prettyPrintModels(model):
    modelAlgoName = model["model"]["machineLearingAlgo"]
    vectorizerName = model["model"]["vectorizer"]
    optionsName = model["model"]["options"]
    minNrange = model["model"]["minNrange"]
    maxNrange = model["model"]["maxNrange"]
    if minNrange == maxNrange:
        nGramString = f'{minNrange} {"word" if minNrange==1 else "words as a feature for vectorization"}'
    else:
        nGramString = f'{minNrange}-{maxNrange} words as a feature for vectorization'
    printmd("## Trained and Tested  Model: " + modelAlgoName +
            "\n\t - using " + optionsName + " for tokenization" +
            "\n\t - with " + vectorizerName + " as a vectorizer taking " + nGramString +
            "\n\t - without stratification on an unbalanced dataset")
    printmd("--"*10+"Results" + "--"*10)
    printmd(
        f"- Average Accuracy of {modelAlgoName} across {numberOfFolds}-folds = {model['accuracy']}")
    printmd(
        f"- Average F1-Score of {modelAlgoName} across {numberOfFolds}-folds = {model['f1_score']}")
    printmd(
        f"- Average Confustion Matrix of {modelAlgoName} across {numberOfFolds}-folds:")
    sns.heatmap(model["confusion_matrix"], annot=True)
    plt.show()


# Fist will sort the models based on the model's minimum ngram range and maximum ngram range
result_balanced["lock"].acquire()
result_balanced["list"].sort(key=lambda x: (
    x["model"]["minNrange"], x["model"]["maxNrange"]))
for model in result_balanced["list"]:
    prettyPrintModels(model)

# Print the best model

printmd('---'*10)
result_balanced["lock"].release()

# Load all the json files in the directory
MAX_THREADS = 10
highest_accuracy_model_BalancedDataSet = {"model":None, "score":0,"lock":threading.Lock()}
highest_f1_score_model_BalancedDataSet = {"model":None, "score":0,"lock":threading.Lock()}


# Do the training and testing
# Creating the list to store the result of each model
result_balanced = {"list": [], "lock": threading.Lock()}
numberOfFolds =5
def prettyPrintModels(model):
    modelAlgoName = model["model"]["machineLearingAlgo"]
    vectorizerName = model["model"]["vectorizer"]
    optionsName = model["model"]["options"]
    minNrange = model["model"]["minNrange"]
    maxNrange = model["model"]["maxNrange"]
    if minNrange == maxNrange:
        nGramString = f'{minNrange} {"word" if minNrange==1 else "words as a feature for vectorization"}'
    else:
        nGramString = f'{minNrange}-{maxNrange} words as a feature for vectorization'
    printmd("## Trained and Tested  Model: " + modelAlgoName +
            "\n\t - using " + optionsName + " for tokenization" +
            "\n\t - with " + vectorizerName + " as a vectorizer taking " + nGramString +
            "\n\t - without stratification on an unbalanced dataset")
    printmd("--"*10+"Results" + "--"*10)
    printmd(
        f"- Average Accuracy of {modelAlgoName} across {numberOfFolds}-folds = {model['accuracy']}")
    printmd(
        f"- Average F1-Score of {modelAlgoName} across {numberOfFolds}-folds = {model['f1_score']}")
    printmd(
        f"- Average Confustion Matrix of {modelAlgoName} across {numberOfFolds}-folds:")
    sns.heatmap(model["confusion_matrix"], annot=True)
    plt.show()


outputs = []
for file in os.listdir("./OutputsBalanced"):
    if file.endswith(".json"):
        # Load the json file
        with open(os.path.join("./OutputsBalanced", file), "r") as f:
            model = json.load(f)
            with open(os.path.join("./OutputsBalanced", f"{file[:-5]}_confusion_matrix.npy"), "rb") as fc:
                #Load the confusion matrix
                model["confusion_matrix"] = np.load(fc)
            highest_accuracy_model_BalancedDataSet["lock"].acquire()
            if model["accuracy"] > highest_accuracy_model_BalancedDataSet.get("score"):
                highest_accuracy_model_BalancedDataSet["score"] = model["accuracy"]
                highest_accuracy_model_BalancedDataSet["model"] = model['model']
            highest_accuracy_model_BalancedDataSet["lock"].release()

            highest_f1_score_model_BalancedDataSet["lock"].acquire()
            if model["f1_score"] > highest_f1_score_model_BalancedDataSet.get("score"):
                highest_f1_score_model_BalancedDataSet["score"] = model["f1_score"]
                highest_f1_score_model_BalancedDataSet["model"] = model['model']
            highest_f1_score_model_BalancedDataSet["lock"].release()
            outputs.append(model)
outputs.sort(key=lambda x: (x["model"]["minNrange"],x["model"]["maxNrange"]))
result_balanced["lock"].acquire()
result_balanced["list"] = outputs
result_balanced["lock"].release()
for model in outputs:
    prettyPrintModels(model)
printmd('---'*10)

 - using Lemmatization followed by Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming followed by Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization followed by Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming followed by Lemmatization for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization followed by Stemming for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

 - using Lemmatization for tokenization
 - with CountVectorizer as a vectorizer taking 1 word
 - without stratification on an unbalanced dataset

result_balanced["lock"].acquire()
# result_balanced["list"].sort(key=lambda x: (
#     x["model"]["machineLearingAlgo"], x["model"]["minNrange"], x["model"]["maxNrange"]))
result_balanced["list"].sort(key=lambda x: (
    x["accuracy"]))
resultsDataFrameBalanced = pd.DataFrame(
    columns=['Algorithim', 'Ngram Range', 'Vectorizer', 'Normalizing Technique', 'Accuracy', 'F1-Score'])
for model in reversed(result_balanced["list"]):
    resultsDataFrameBalanced.loc[len(resultsDataFrameBalanced)] = [model["model"]["machineLearingAlgo"],
                                                                   f'{model["model"]["minNrange"]}-{model["model"]["maxNrange"]}',
                                                                   model["model"]["vectorizer"],
                                                                   model["model"]["options"],
                                                                   model["accuracy"],
                                                                   model["f1_score"]]
result_balanced["lock"].release()
pd.set_option("display.max_rows", None, "display.max_columns", None)
display(resultsDataFrameBalanced)

df_balanced_results = pd.read_json('resultsDataFrameBalanced.json')
df_unbalanced_results = pd.read_json('resultsDataFrameUnbalanced.json')

# plot the results
def plot_accuracy(df, title):
    plt.figure(figsize=(20, 10))
    # make a bar chart using plt
    sns.barplot(x="Algorithim", y="Accuracy", data=df)
    plt.title(title)
    plt.show()


plot_accuracy(df_balanced_results,
              "Accuracy of the best model on balanced dataset")
plot_accuracy(df_unbalanced_results,
            "Accuracy of the best model on unbalanced dataset")

def plot_f1_score(df, title):
    plt.figure(figsize=(20, 10))
    sns.barplot(x="Algorithim", y="F1-Score", data=df)
    plt.title(title)
    plt.show()

plot_f1_score(df_balanced_results,
                "F1-Score of the best model on balanced dataset")
plot_f1_score(df_unbalanced_results,
                "F1-Score of the best model on unbalanced dataset")

plt.Figure(figsize=(20, 10))

sns.catplot(x="Algorithim", y="F1-Score",
            data=df_balanced_results, kind="point", label='Balanced', size=5, aspect=4)
plt.title("Accuracy of the best model on balanced dataset")
plt.show()


sns.catplot(x="Algorithim", y="F1-Score",
            data=df_unbalanced_results, kind="point", label='Unbalanced', size=5, aspect=4)
plt.title("Accuracy of the best model on unbalanced dataset")
plt.show()

r = sns.kdeplot(df_balanced_results["Accuracy"], shade=True, label='Balanced', color='r')
b = sns.kdeplot(df_unbalanced_results["Accuracy"], shade=True, label='Unbalanced', color='b')
r.figure.set_size_inches(20, 10)
plt.title("Accuracy of the best model on both the datasets")
plt.legend()
plt.show()

r = sns.kdeplot(df_balanced_results["F1-Score"], 
                shade=True, label='Balanced', color='r')
b = sns.kdeplot(df_unbalanced_results["F1-Score"],
                shade=True, label='Unbalanced', color='b')
r.figure.set_size_inches(20, 10)
plt.title("F1-Score of the best model on both the datasets")
plt.legend()
plt.show()

g = sns.violinplot(x="Algorithim", y="Accuracy", data=df_balanced_results)
g.set_xticklabels(g.get_xticklabels(), rotation=45)
g.figure.set_size_inches(20, 10)
plt.title("Violin plot for the balanced dataset")
plt.show()

g = sns.violinplot(x="Algorithim", y="F1-Score", data=df_unbalanced_results)
g.set_xticklabels(g.get_xticklabels(), rotation=45)
g.figure.set_size_inches(20, 10)
plt.title("Violin plot for the unbalanced dataset")
plt.show()

s = sns.displot(x="Algorithim", y="Accuracy", data=df_balanced_results)
s.figure.set_size_inches(15, 15)
plt.title("Displot for the balanced dataset (Accuracy)")
plt.show()

s = sns.displot(x="Algorithim", y="F1-Score", data=df_balanced_results)
s.figure.set_size_inches(15, 15)
plt.title("Displot for the balanced dataset (F1-Score)")
plt.show()

s = sns.displot(x="Algorithim", y="Accuracy", data=df_unbalanced_results)
s.figure.set_size_inches(15, 15)
plt.title("Displot for the unbalanced dataset (Accuracy)")

s = sns.displot(x="Algorithim", y="F1-Score", data=df_unbalanced_results)
s.figure.set_size_inches(15, 15)
plt.title("Displot for the unbalanced dataset")
plt.show()

Evaluation¶

For model evaluation, we would be generating a confusion matrix to find out the number of True Positives, True Negatives, False Positives and False Negatives. From this we will calculate Recall score, Precision score and F1 score which is the most important evaluation metric in an unbalanced data set. The F1 for the stratified version of training we will be using F1 score with macro average due to the skewed nature of our corpus and each sample being equally important while for the variation with equal number of samples per class we will use micro average as our corpus is no longer unbalanced. Accuracy is also an important metric for our second variation since the corpus is now balanced and there will be no inherrent biases while training the model.

The best performing model will be the model with the overall best accuracy and F1 score, and the preprocessing steps and hyperparameters for that model would verify or nullify our hypotheses regarding representation and normalisation. These steps and hyperparameter optimisation will gaurantee the most optimal model for our corpus given its heavily positively skewed nature.

printmd("---"*10+ "Evaluation for Un-balanced Dataset" + "---"*10)
printmd("## Model with the best accuracy on the unbalanced data set is: " + highest_accuracy_model["model"]["machineLearingAlgo"] +
        f'\n\t - with Accuracy = {highest_accuracy_model_BalancedDataSet["score"]}'+
        "\n\t - using" + highest_accuracy_model["model"]["options"] + " for tokenization" +
        "\n\t - with " + highest_accuracy_model["model"]["vectorizer"] + " as a vectorizer taking " +
        f'{highest_accuracy_model["model"]["minNrange"]}-{highest_accuracy_model["model"]["maxNrange"]} words as a feature for vectorization' +
        "\n\t - without stratification on an unbalanced dataset")

printmd("## Model with the best f1-score on the unbalanced data set is: " + highest_f1_score_model["model"]["machineLearingAlgo"] +
        f'\n\t - with F1 score = {highest_accuracy_model_BalancedDataSet["score"]}'+
        "\n\t - using" + highest_f1_score_model["model"]["options"] + " for tokenization" +
        "\n\t - with " + highest_f1_score_model["model"]["vectorizer"] + " as a vectorizer taking " +
        f'{highest_f1_score_model["model"]["minNrange"]}-{highest_f1_score_model["model"]["maxNrange"]} words as a feature for vectorization' +
        "\n\t - without stratification on an unbalanced dataset")
printmd("---"*10+ "Evaluation for Balanced Dataset" + "---"*10)

printmd("## Model with the best accuracy on the startified balanced data set is: " + highest_accuracy_model_BalancedDataSet["model"]["machineLearingAlgo"] +
        f'\n\t - with Accuracy = {highest_accuracy_model_BalancedDataSet["score"]}'+
        "\n\t - using " + highest_accuracy_model_BalancedDataSet["model"]["options"] + " for tokenization" +
        "\n\t - with " + highest_accuracy_model_BalancedDataSet["model"]["vectorizer"] + " as a vectorizer taking " +
        f'{highest_accuracy_model_BalancedDataSet["model"]["minNrange"]}-{highest_accuracy_model_BalancedDataSet["model"]["maxNrange"]} words as a feature for vectorization' +
        "\n\t - with stratification on an balanced dataset")

printmd("## Model with the best f1-score on the startified balanced data set is: " + highest_f1_score_model_BalancedDataSet["model"]["machineLearingAlgo"] +
        f'\n\t - with F1 score = {highest_accuracy_model_BalancedDataSet["score"]}'+
        "\n\t - using " + highest_f1_score_model_BalancedDataSet["model"]["options"] + " for tokenization" +
        "\n\t - with " + highest_f1_score_model_BalancedDataSet["model"]["vectorizer"] + " as a vectorizer taking " +
        f'{highest_f1_score_model_BalancedDataSet["model"]["minNrange"]}-{highest_f1_score_model_BalancedDataSet["model"]["maxNrange"]} words as a feature for vectorization' +
        "\n\t - with stratification on an balanced dataset")

 - with Accuracy = 0.7159054760281753
 - usingStemming followed by Lemmatization for tokenization
 - with TfidfVectorizer as a vectorizer taking 1-3 words as a feature for vectorization
 - without stratification on an unbalanced dataset

 - with F1 score = 0.7159054760281753
 - usingLemmatization followed by Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1-1 words as a feature for vectorization
 - without stratification on an unbalanced dataset

 - with Accuracy = 0.7159054760281753
 - using Lemmatization followed by Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1-1 words as a feature for vectorization
 - with stratification on an balanced dataset

 - with F1 score = 0.7159054760281753
 - using Lemmatization followed by Stemming for tokenization
 - with TfidfVectorizer as a vectorizer taking 1-1 words as a feature for vectorization
 - with stratification on an balanced dataset

Deep Learning Pipeline¶

Creation of a BiLSTM pipeline for Sentiment Analysis¶

For our Deep Learning pipeline, we will be using a Recurrent Neural Network (RNN) known as Long Short-Term Memory (LSTM). The biggest factor in using LSTM is that it does not use a bag of word representation or TF-IDF, instead it remembers the order of words in a tweet during vectorisation. The first step in the pipeline is the same as our previous models where we first Normalise our tweets and Lemmatise them. Then we pass the normalised tweets to our LSTM model where it first tokenises the tweet using keras.preprocessing and then in the vectorisation stage, we use a pretrained model called Glove trained with 60 billion parameters with 50 dimensions (between 1 to 50 character words). We use Glove to assign a value to each token and represent each sentence as a vector. We kept a maximum limit of 50 words as twitter has a limit of 140 characters per tweet hence it is impossible for a tweet to have more words, if a tweet has less words then we use padding to assign the remaining length of the vector to 0 so each vector has the same length. The benefit of using this representation is that Glove assigns the value to a token in its corresponding position with reference to its context, preserving the order of words and the value assigned is the value represented in Glove's vocabulary. Hence, the vector containing the tokens also preserve their order. This representation is then fed into our LSTM model which has 2 hidden layers and the output layer using softmax activation function to give us the probability of a tweet belonging to each class and the class with the highest probability is the class assigned to the tweet. We stop training of our LSTM when our validation accuracy decreases and loss increases in our graph, signalling overfitting, LSTM is tested through 5 fold cross validation.

class BiLSTM(nn.Module):

    def __init__(self, le, embedding_matrix):
        super(BiLSTM, self).__init__()
        self.hidden_size = 50
        drp = 0.1
        n_classes = len(le.classes_)
        self.embedding = nn.Embedding(max_features, embed_size)
        self.embedding.weight = nn.Parameter(
            torch.tensor(embedding_matrix, dtype=torch.float32))
        self.embedding.weight.requires_grad = False
        self.lstm = nn.LSTM(embed_size, self.hidden_size,
                            bidirectional=True, batch_first=True)
        self.linear = nn.Linear(self.hidden_size*4, 80)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(drp)
        self.out = nn.Linear(80, n_classes)

    def forward(self, x):
        # rint(x.size())
        h_embedding = self.embedding(x)
        #_embedding = torch.squeeze(torch.unsqueeze(h_embedding, 0))
        h_lstm, _ = self.lstm(h_embedding)
        avg_pool = torch.mean(h_lstm, 1)
        max_pool, _ = torch.max(h_lstm, 1)
        conc = torch.cat((avg_pool, max_pool), 1)
        conc = self.relu(self.linear(conc))
        conc = self.dropout(conc)
        out = self.out(conc)
        return out


class LstmModelPytorch(BaseEstimator, TransformerMixin):
    def __init__(self, max_features, n_epochs, batch_size, maxlen, embed_size, kFolds, debug):
        # Reproducing same results
        self.max_features = max_features
        self.le = LabelEncoder()
        self.n_epochs = n_epochs
        self.loss_fn = nn.CrossEntropyLoss(reduction='mean')

        self.batch_size = batch_size
        self.maxlen = maxlen
        self.embed_size = embed_size
        self.kFolds = kFolds
        self.debug = debug

    def load_glove(self, word_index, embed_size):
        EMBEDDING_FILE = 'glove.6B/glove.6B.50d.txt'
        def get_coefs(word, *arr): return word, np.asarray(arr,
                                                           dtype='float32')[:300]
        embeddings_index = dict(get_coefs(*o.split(" "))
                                for o in open(EMBEDDING_FILE, encoding="utf8"))

        all_embs = np.stack(embeddings_index.values())
        emb_mean, emb_std = -0.005838499, 0.48782197
        embed_size = all_embs.shape[1]

        nb_words = min(max_features, len(word_index)+1)
        embedding_matrix = np.random.normal(
            emb_mean, emb_std, (nb_words, embed_size))
        for word, i in word_index.items():
            if i >= max_features:
                continue
            embedding_vector = embeddings_index.get(word)
            if embedding_vector is not None:
                embedding_matrix[i] = embedding_vector
            else:
                embedding_vector = embeddings_index.get(word.capitalize())
                if embedding_vector is not None:
                    embedding_matrix[i] = embedding_vector
        return embedding_matrix

    def plot_graph(self, epochs, train_loss, val_loss):
        fig = plt.figure(figsize=(12, 12))
        plt.title("Train/Validation Loss")
        plt.plot(list(np.arange(epochs) + 1), train_loss, label='train')
        plt.plot(list(np.arange(epochs) + 1), val_loss, label='validation')
        plt.xlabel('num_epochs', fontsize=12)
        plt.ylabel('loss', fontsize=12)
        plt.legend(loc='best')
        plt.show()

    def fit(self, X, y=None):
        xDf = pd.DataFrame(X, columns=['body'])
        xDf['label'] = y.to_list()
        average_trainingLoss = []
        average_validationLoss = []
        totalFScore = 0
        totalAccuracy = 0
        totalConfusion_matrix = None
        kfold = StratifiedKFold(n_splits=self.kFolds,
                                shuffle=True, random_state=7)
        foldCounter = 0
        bestModel = None
        bestValidationF1 = 0
        for train_index, test_index in kfold.split(df[['body']], df['label']):
            foldCounter += 1
            train_X, test_X = df.iloc[train_index][[
                'body']], df.iloc[test_index][['body']]
            train_y, test_y = df.iloc[train_index]['label'], df.iloc[test_index]['label']

            self.tokenizer = Tokenizer(num_words=self.max_features)
            self.tokenizer.fit_on_texts(list(train_X['body']))
            train_X = self.tokenizer.texts_to_sequences(train_X['body'])

            test_X = self.tokenizer.texts_to_sequences(test_X['body'])

            if self.debug:
                self.embedding_matrix = np.random.randn(120000, 300)
            else:
                self.embedding_matrix = self.load_glove(
                    self.tokenizer.word_index, self.embed_size)

            # Pad the sentences
            train_X = pad_sequences(train_X, maxlen=self.maxlen)
            test_X = pad_sequences(test_X, maxlen=self.maxlen)

            train_y = self.le.fit_transform(train_y.values)
            test_y = self.le.transform(test_y.values)
            # Load train and test in CUDA Memory
            x_train = torch.tensor(train_X, dtype=torch.long).cuda()
            y_train = torch.tensor(train_y, dtype=torch.long).cuda()
            x_cv = torch.tensor(test_X, dtype=torch.long).cuda()
            y_cv = torch.tensor(test_y, dtype=torch.long).cuda()

            # Create Torch datasets
            train = torch.utils.data.TensorDataset(x_train, y_train)
            valid = torch.utils.data.TensorDataset(x_cv, y_cv)

            self.model = BiLSTM(self.le, self.embedding_matrix)
            self.optimizer = torch.optim.Adam(
                filter(lambda p: p.requires_grad, self.model.parameters()), lr=0.001)
            self.model.cuda()

            # Create Data Loaders
            train_loader = torch.utils.data.DataLoader(
                train, batch_size=batch_size, shuffle=True)
            valid_loader = torch.utils.data.DataLoader(
                valid, batch_size=batch_size, shuffle=False)

            train_loss = []
            valid_loss = []

            for epoch in range(self.n_epochs):
                start_time = time.time()
                # Set model to train configuration
                self.model.train()
                avg_loss = 0.
                for i, (x_batch, y_batch) in enumerate(train_loader):
                    # Predict/Forward Pass
                    y_pred = self.model(x_batch)
                    # Compute loss
                    self.loss = self.loss_fn(y_pred, y_batch)
                    self.optimizer.zero_grad()
                    self.loss.backward()
                    self.optimizer.step()
                    avg_loss += self.loss.item() / len(train_loader)

                # Set model to validation configuration -Doesn't get trained here
                self.model.eval()
                avg_val_loss = 0.
                val_preds = np.zeros((len(x_cv), len(self.le.classes_)))

                for i, (x_batch, y_batch) in enumerate(valid_loader):
                    y_pred = self.model(x_batch).detach()
                    avg_val_loss += self.loss_fn(y_pred,
                                                 y_batch).item() / len(valid_loader)
                    # keep/store predictions
                    val_preds[i * batch_size:(i+1) *
                              batch_size] = F.softmax(y_pred).cpu().numpy()

                # Check Accuracy
                val_accuracy = sum(val_preds.argmax(
                    axis=1) == test_y)/len(test_y)
                train_loss.append(avg_loss)
                valid_loss.append(avg_val_loss)
                elapsed_time = time.time() - start_time
                print('Epoch {}/{} at {} fold: \t loss={:.4f} \t val_loss={:.4f}  \t val_acc={:.4f}  \t time={:.2f}s'.format(
                    epoch + 1, self.n_epochs, foldCounter, avg_loss, avg_val_loss, val_accuracy, elapsed_time))
            average_trainingLoss.append(train_loss)
            average_validationLoss.append(valid_loss)
            y_true = [self.le.classes_[x] for x in test_y]
            y_pred = [self.le.classes_[x] for x in val_preds.argmax(axis=1)]
            fSc = f1_score(y_true, y_pred, average='weighted')
            if fSc > bestValidationF1:
                bestValidationF1 = fSc
                bestModel = self.model
            totalAccuracy += accuracy_score(y_true, y_pred)
            totalFScore += fSc
            totalConfusion_matrix = totalConfusion_matrix + confusion_matrix(
                y_true, y_pred) if totalConfusion_matrix is not None else confusion_matrix(y_true, y_pred)

        torch.save(bestModel, 'bilstm_model')
        # Element wise sum the average training and validation loss
        average_trainingLoss = np.array(average_trainingLoss).sum(axis=0)
        average_validationLoss = np.array(average_validationLoss).sum(axis=0)
        self.plot_graph(self.n_epochs, average_trainingLoss,
                        average_validationLoss)

        printmd("## Trained and Tested  Model: BiLSTM" +
              "\n\t - using lemmitization for tokenization" +
              "\n\t - with Glove Embeddings for vectorization" +
              f"\n\t - {'without stratification on an unbalanced dataset'if len(X)>2000 else 'on a statified balanced dataset'}")
        printmd("--"*10+"Results" + "--"*10)
        printmd(
            f"- Average Accuracy of BiLSTM across {self.kFolds}-folds = {totalAccuracy/self.kFolds}")
        printmd(
            f"- Average F1-Score of BiLSTM across {self.kFolds}-folds = {totalFScore/self.kFolds}")
        printmd(
            f"- Average Confustion Matrix of BiLSTM across {self.kFolds}-folds:")
        sns.heatmap(totalConfusion_matrix/self.kFolds, annot=True)
        plt.show()

    def predict(self, X):
        self.model = torch.load('bilstm_model')
        x = X[0]
        # tokenize
        x = self.tokenizer.texts_to_sequences([x])
        # pad
        x = pad_sequences(x, maxlen=maxlen)
        # create dataset
        x = torch.tensor(x, dtype=torch.long).cuda()

        pred = self.model(x).detach()
        pred = F.softmax(pred).cpu().numpy()

        pred = pred.argmax(axis=1)

        pred = self.le.classes_[pred]
        return pred[0]


embed_size = 50  # how big is each word vector
# how many unique words to use (i.e num rows in embedding vector)
max_features = 120000
maxlen = 50  # max number of words in a tweet to use
batch_size = 512  # how many samples to process at once
n_epochs = 30  # how many times to iterate over all samples
n_splits = 5  # Number of K-fold Splits
debug = 0

Training and Testing Deep learning Model: Bi-LSTM on the unbalanced data set using Lematization for Tokenization with Glove for Vectorization¶

lstmPipelineUnBalanced = Pipeline([
    ('normalizer', Normalizer('l')),
    ('estimator', LstmModelPytorch(max_features=max_features,
     n_epochs=n_epochs, batch_size=batch_size, maxlen=maxlen, embed_size=embed_size, kFolds=n_splits, debug=debug))
])
_ = lstmPipelineUnBalanced.fit(df[['body']], df['label'])

Epoch 1/30 at 1 fold: 	 loss=1.0576 	 val_loss=0.9809  	 val_acc=0.5597  	 time=1.11s
Epoch 2/30 at 1 fold: 	 loss=0.9702 	 val_loss=0.8880  	 val_acc=0.5597  	 time=0.07s
Epoch 3/30 at 1 fold: 	 loss=0.9160 	 val_loss=0.8581  	 val_acc=0.5597  	 time=0.07s
Epoch 4/30 at 1 fold: 	 loss=0.9234 	 val_loss=0.8639  	 val_acc=0.5597  	 time=0.06s
Epoch 5/30 at 1 fold: 	 loss=0.9126 	 val_loss=0.8652  	 val_acc=0.5936  	 time=0.07s
Epoch 6/30 at 1 fold: 	 loss=0.8976 	 val_loss=0.8459  	 val_acc=0.5686  	 time=0.06s
Epoch 7/30 at 1 fold: 	 loss=0.8803 	 val_loss=0.8321  	 val_acc=0.5668  	 time=0.06s
Epoch 8/30 at 1 fold: 	 loss=0.8741 	 val_loss=0.8188  	 val_acc=0.5829  	 time=0.06s
Epoch 9/30 at 1 fold: 	 loss=0.8656 	 val_loss=0.8115  	 val_acc=0.6025  	 time=0.07s
Epoch 10/30 at 1 fold: 	 loss=0.8441 	 val_loss=0.7711  	 val_acc=0.6275  	 time=0.06s
Epoch 11/30 at 1 fold: 	 loss=0.8070 	 val_loss=0.7576  	 val_acc=0.6364  	 time=0.07s
Epoch 12/30 at 1 fold: 	 loss=0.7866 	 val_loss=0.7508  	 val_acc=0.6613  	 time=0.06s
Epoch 13/30 at 1 fold: 	 loss=0.7428 	 val_loss=0.6956  	 val_acc=0.6649  	 time=0.07s
Epoch 14/30 at 1 fold: 	 loss=0.7384 	 val_loss=0.7248  	 val_acc=0.6631  	 time=0.06s
Epoch 15/30 at 1 fold: 	 loss=0.7293 	 val_loss=0.6779  	 val_acc=0.6934  	 time=0.07s
Epoch 16/30 at 1 fold: 	 loss=0.6881 	 val_loss=0.6671  	 val_acc=0.6916  	 time=0.06s
Epoch 17/30 at 1 fold: 	 loss=0.6831 	 val_loss=0.6973  	 val_acc=0.6560  	 time=0.07s
Epoch 18/30 at 1 fold: 	 loss=0.6862 	 val_loss=0.6459  	 val_acc=0.6988  	 time=0.07s
Epoch 19/30 at 1 fold: 	 loss=0.6585 	 val_loss=0.6700  	 val_acc=0.6952  	 time=0.07s
Epoch 20/30 at 1 fold: 	 loss=0.6435 	 val_loss=0.6359  	 val_acc=0.7023  	 time=0.06s
Epoch 21/30 at 1 fold: 	 loss=0.6210 	 val_loss=0.6403  	 val_acc=0.7077  	 time=0.07s
Epoch 22/30 at 1 fold: 	 loss=0.6176 	 val_loss=0.6191  	 val_acc=0.6952  	 time=0.07s
Epoch 23/30 at 1 fold: 	 loss=0.6110 	 val_loss=0.6148  	 val_acc=0.7166  	 time=0.07s
Epoch 24/30 at 1 fold: 	 loss=0.5853 	 val_loss=0.6060  	 val_acc=0.6988  	 time=0.08s
Epoch 25/30 at 1 fold: 	 loss=0.5799 	 val_loss=0.6276  	 val_acc=0.6952  	 time=0.08s
Epoch 26/30 at 1 fold: 	 loss=0.5908 	 val_loss=0.5864  	 val_acc=0.7184  	 time=0.07s
Epoch 27/30 at 1 fold: 	 loss=0.5601 	 val_loss=0.5858  	 val_acc=0.7148  	 time=0.08s
Epoch 28/30 at 1 fold: 	 loss=0.5477 	 val_loss=0.5880  	 val_acc=0.7237  	 time=0.07s
Epoch 29/30 at 1 fold: 	 loss=0.5519 	 val_loss=0.5865  	 val_acc=0.7291  	 time=0.07s
Epoch 30/30 at 1 fold: 	 loss=0.5291 	 val_loss=0.5719  	 val_acc=0.7255  	 time=0.19s
Epoch 1/30 at 2 fold: 	 loss=1.1179 	 val_loss=1.0494  	 val_acc=0.5597  	 time=0.28s
Epoch 2/30 at 2 fold: 	 loss=1.0237 	 val_loss=0.9472  	 val_acc=0.5597  	 time=0.11s
Epoch 3/30 at 2 fold: 	 loss=0.9434 	 val_loss=0.8754  	 val_acc=0.5597  	 time=0.07s
Epoch 4/30 at 2 fold: 	 loss=0.9100 	 val_loss=0.8596  	 val_acc=0.5597  	 time=0.06s
Epoch 5/30 at 2 fold: 	 loss=0.9208 	 val_loss=0.8589  	 val_acc=0.5651  	 time=0.07s
Epoch 6/30 at 2 fold: 	 loss=0.9056 	 val_loss=0.8543  	 val_acc=0.5722  	 time=0.07s
Epoch 7/30 at 2 fold: 	 loss=0.8961 	 val_loss=0.8468  	 val_acc=0.5651  	 time=0.06s
Epoch 8/30 at 2 fold: 	 loss=0.8920 	 val_loss=0.8417  	 val_acc=0.5651  	 time=0.07s
Epoch 9/30 at 2 fold: 	 loss=0.8785 	 val_loss=0.8311  	 val_acc=0.5775  	 time=0.07s
Epoch 10/30 at 2 fold: 	 loss=0.8618 	 val_loss=0.8150  	 val_acc=0.5918  	 time=0.07s
Epoch 11/30 at 2 fold: 	 loss=0.8577 	 val_loss=0.7890  	 val_acc=0.6114  	 time=0.07s
Epoch 12/30 at 2 fold: 	 loss=0.8294 	 val_loss=0.7649  	 val_acc=0.6310  	 time=0.07s
Epoch 13/30 at 2 fold: 	 loss=0.8066 	 val_loss=0.7310  	 val_acc=0.6702  	 time=0.07s
Epoch 14/30 at 2 fold: 	 loss=0.7825 	 val_loss=0.7056  	 val_acc=0.6756  	 time=0.08s
Epoch 15/30 at 2 fold: 	 loss=0.7624 	 val_loss=0.6913  	 val_acc=0.6970  	 time=0.08s
Epoch 16/30 at 2 fold: 	 loss=0.7438 	 val_loss=0.6921  	 val_acc=0.6684  	 time=0.08s
Epoch 17/30 at 2 fold: 	 loss=0.7232 	 val_loss=0.6588  	 val_acc=0.6898  	 time=0.08s
Epoch 18/30 at 2 fold: 	 loss=0.7018 	 val_loss=0.6519  	 val_acc=0.6881  	 time=0.07s
Epoch 19/30 at 2 fold: 	 loss=0.6791 	 val_loss=0.6420  	 val_acc=0.7059  	 time=0.07s
Epoch 20/30 at 2 fold: 	 loss=0.6786 	 val_loss=0.6409  	 val_acc=0.7184  	 time=0.07s
Epoch 21/30 at 2 fold: 	 loss=0.6667 	 val_loss=0.6349  	 val_acc=0.7041  	 time=0.07s
Epoch 22/30 at 2 fold: 	 loss=0.6428 	 val_loss=0.6274  	 val_acc=0.7201  	 time=0.06s
Epoch 23/30 at 2 fold: 	 loss=0.6461 	 val_loss=0.6302  	 val_acc=0.7255  	 time=0.07s
Epoch 24/30 at 2 fold: 	 loss=0.6260 	 val_loss=0.6337  	 val_acc=0.7148  	 time=0.07s
Epoch 25/30 at 2 fold: 	 loss=0.6223 	 val_loss=0.6243  	 val_acc=0.7237  	 time=0.07s
Epoch 26/30 at 2 fold: 	 loss=0.6171 	 val_loss=0.6186  	 val_acc=0.7433  	 time=0.07s
Epoch 27/30 at 2 fold: 	 loss=0.6142 	 val_loss=0.6085  	 val_acc=0.7451  	 time=0.07s
Epoch 28/30 at 2 fold: 	 loss=0.5736 	 val_loss=0.5897  	 val_acc=0.7469  	 time=0.07s
Epoch 29/30 at 2 fold: 	 loss=0.5692 	 val_loss=0.5861  	 val_acc=0.7504  	 time=0.07s
Epoch 30/30 at 2 fold: 	 loss=0.5637 	 val_loss=0.5877  	 val_acc=0.7344  	 time=0.06s
Epoch 1/30 at 3 fold: 	 loss=1.0393 	 val_loss=0.9484  	 val_acc=0.5607  	 time=0.29s
Epoch 2/30 at 3 fold: 	 loss=0.9430 	 val_loss=0.8743  	 val_acc=0.5607  	 time=0.09s
Epoch 3/30 at 3 fold: 	 loss=0.9126 	 val_loss=0.8637  	 val_acc=0.5607  	 time=0.06s
Epoch 4/30 at 3 fold: 	 loss=0.9034 	 val_loss=0.8767  	 val_acc=0.5714  	 time=0.06s
Epoch 5/30 at 3 fold: 	 loss=0.8995 	 val_loss=0.8679  	 val_acc=0.5696  	 time=0.07s
Epoch 6/30 at 3 fold: 	 loss=0.8844 	 val_loss=0.8577  	 val_acc=0.5750  	 time=0.06s
Epoch 7/30 at 3 fold: 	 loss=0.8785 	 val_loss=0.8495  	 val_acc=0.5643  	 time=0.06s
Epoch 8/30 at 3 fold: 	 loss=0.8572 	 val_loss=0.8400  	 val_acc=0.5893  	 time=0.06s
Epoch 9/30 at 3 fold: 	 loss=0.8341 	 val_loss=0.8167  	 val_acc=0.5893  	 time=0.07s
Epoch 10/30 at 3 fold: 	 loss=0.8144 	 val_loss=0.7985  	 val_acc=0.6232  	 time=0.07s
Epoch 11/30 at 3 fold: 	 loss=0.7829 	 val_loss=0.7672  	 val_acc=0.6304  	 time=0.06s
Epoch 12/30 at 3 fold: 	 loss=0.7633 	 val_loss=0.7538  	 val_acc=0.6464  	 time=0.06s
Epoch 13/30 at 3 fold: 	 loss=0.7225 	 val_loss=0.7231  	 val_acc=0.6411  	 time=0.06s
Epoch 14/30 at 3 fold: 	 loss=0.6924 	 val_loss=0.7139  	 val_acc=0.6536  	 time=0.07s
Epoch 15/30 at 3 fold: 	 loss=0.6766 	 val_loss=0.6944  	 val_acc=0.6446  	 time=0.07s
Epoch 16/30 at 3 fold: 	 loss=0.6636 	 val_loss=0.6765  	 val_acc=0.6821  	 time=0.07s
Epoch 17/30 at 3 fold: 	 loss=0.6208 	 val_loss=0.6810  	 val_acc=0.6839  	 time=0.07s
Epoch 18/30 at 3 fold: 	 loss=0.6204 	 val_loss=0.6940  	 val_acc=0.6893  	 time=0.07s
Epoch 19/30 at 3 fold: 	 loss=0.6011 	 val_loss=0.7018  	 val_acc=0.6732  	 time=0.07s
Epoch 20/30 at 3 fold: 	 loss=0.5825 	 val_loss=0.6797  	 val_acc=0.6982  	 time=0.06s
Epoch 21/30 at 3 fold: 	 loss=0.5858 	 val_loss=0.6921  	 val_acc=0.6911  	 time=0.07s
Epoch 22/30 at 3 fold: 	 loss=0.5588 	 val_loss=0.7093  	 val_acc=0.6893  	 time=0.07s
Epoch 23/30 at 3 fold: 	 loss=0.5654 	 val_loss=0.7118  	 val_acc=0.7107  	 time=0.06s
Epoch 24/30 at 3 fold: 	 loss=0.5474 	 val_loss=0.6757  	 val_acc=0.6964  	 time=0.06s
Epoch 25/30 at 3 fold: 	 loss=0.5339 	 val_loss=0.6918  	 val_acc=0.7107  	 time=0.06s
Epoch 26/30 at 3 fold: 	 loss=0.5141 	 val_loss=0.7025  	 val_acc=0.7071  	 time=0.06s
Epoch 27/30 at 3 fold: 	 loss=0.5109 	 val_loss=0.7319  	 val_acc=0.6982  	 time=0.06s
Epoch 28/30 at 3 fold: 	 loss=0.5166 	 val_loss=0.7044  	 val_acc=0.7054  	 time=0.06s
Epoch 29/30 at 3 fold: 	 loss=0.5007 	 val_loss=0.7018  	 val_acc=0.7089  	 time=0.07s
Epoch 30/30 at 3 fold: 	 loss=0.4887 	 val_loss=0.7144  	 val_acc=0.7036  	 time=0.07s
Epoch 1/30 at 4 fold: 	 loss=1.0415 	 val_loss=0.9567  	 val_acc=0.5589  	 time=0.23s
Epoch 2/30 at 4 fold: 	 loss=0.9530 	 val_loss=0.8578  	 val_acc=0.5589  	 time=0.07s
Epoch 3/30 at 4 fold: 	 loss=0.9220 	 val_loss=0.8143  	 val_acc=0.5589  	 time=0.07s
Epoch 4/30 at 4 fold: 	 loss=0.9098 	 val_loss=0.8145  	 val_acc=0.5589  	 time=0.07s
Epoch 5/30 at 4 fold: 	 loss=0.9024 	 val_loss=0.8143  	 val_acc=0.5607  	 time=0.07s
Epoch 6/30 at 4 fold: 	 loss=0.8953 	 val_loss=0.8101  	 val_acc=0.5625  	 time=0.08s
Epoch 7/30 at 4 fold: 	 loss=0.8857 	 val_loss=0.8042  	 val_acc=0.5714  	 time=0.08s
Epoch 8/30 at 4 fold: 	 loss=0.8748 	 val_loss=0.7933  	 val_acc=0.5821  	 time=0.07s
Epoch 9/30 at 4 fold: 	 loss=0.8554 	 val_loss=0.7727  	 val_acc=0.6018  	 time=0.06s
Epoch 10/30 at 4 fold: 	 loss=0.8470 	 val_loss=0.7471  	 val_acc=0.6250  	 time=0.07s
Epoch 11/30 at 4 fold: 	 loss=0.8173 	 val_loss=0.7332  	 val_acc=0.6393  	 time=0.07s
Epoch 12/30 at 4 fold: 	 loss=0.7751 	 val_loss=0.6989  	 val_acc=0.6446  	 time=0.08s
Epoch 13/30 at 4 fold: 	 loss=0.7467 	 val_loss=0.6937  	 val_acc=0.6536  	 time=0.07s
Epoch 14/30 at 4 fold: 	 loss=0.7402 	 val_loss=0.6831  	 val_acc=0.6607  	 time=0.07s
Epoch 15/30 at 4 fold: 	 loss=0.7117 	 val_loss=0.6687  	 val_acc=0.6804  	 time=0.07s
Epoch 16/30 at 4 fold: 	 loss=0.6918 	 val_loss=0.6682  	 val_acc=0.6714  	 time=0.07s
Epoch 17/30 at 4 fold: 	 loss=0.6675 	 val_loss=0.6320  	 val_acc=0.6929  	 time=0.08s
Epoch 18/30 at 4 fold: 	 loss=0.6493 	 val_loss=0.6786  	 val_acc=0.6696  	 time=0.07s
Epoch 19/30 at 4 fold: 	 loss=0.6270 	 val_loss=0.6266  	 val_acc=0.7000  	 time=0.07s
Epoch 20/30 at 4 fold: 	 loss=0.6144 	 val_loss=0.6310  	 val_acc=0.7018  	 time=0.08s
Epoch 21/30 at 4 fold: 	 loss=0.6039 	 val_loss=0.6080  	 val_acc=0.6964  	 time=0.08s
Epoch 22/30 at 4 fold: 	 loss=0.5962 	 val_loss=0.6096  	 val_acc=0.7000  	 time=0.07s
Epoch 23/30 at 4 fold: 	 loss=0.5859 	 val_loss=0.6217  	 val_acc=0.6946  	 time=0.08s
Epoch 24/30 at 4 fold: 	 loss=0.5677 	 val_loss=0.5909  	 val_acc=0.7054  	 time=0.08s
Epoch 25/30 at 4 fold: 	 loss=0.5548 	 val_loss=0.6501  	 val_acc=0.6946  	 time=0.07s
Epoch 26/30 at 4 fold: 	 loss=0.5579 	 val_loss=0.5996  	 val_acc=0.7000  	 time=0.07s
Epoch 27/30 at 4 fold: 	 loss=0.5422 	 val_loss=0.6056  	 val_acc=0.7071  	 time=0.08s
Epoch 28/30 at 4 fold: 	 loss=0.5353 	 val_loss=0.6036  	 val_acc=0.7089  	 time=0.07s
Epoch 29/30 at 4 fold: 	 loss=0.5291 	 val_loss=0.5953  	 val_acc=0.7018  	 time=0.06s
Epoch 30/30 at 4 fold: 	 loss=0.5161 	 val_loss=0.5877  	 val_acc=0.7000  	 time=0.07s
Epoch 1/30 at 5 fold: 	 loss=1.0277 	 val_loss=0.9944  	 val_acc=0.5589  	 time=0.22s
Epoch 2/30 at 5 fold: 	 loss=0.9526 	 val_loss=0.9546  	 val_acc=0.5589  	 time=0.07s
Epoch 3/30 at 5 fold: 	 loss=0.9222 	 val_loss=0.9542  	 val_acc=0.5589  	 time=0.07s
Epoch 4/30 at 5 fold: 	 loss=0.9181 	 val_loss=0.9412  	 val_acc=0.5589  	 time=0.06s
Epoch 5/30 at 5 fold: 	 loss=0.9080 	 val_loss=0.9274  	 val_acc=0.5589  	 time=0.08s
Epoch 6/30 at 5 fold: 	 loss=0.8947 	 val_loss=0.9173  	 val_acc=0.5589  	 time=0.07s
Epoch 7/30 at 5 fold: 	 loss=0.8795 	 val_loss=0.9103  	 val_acc=0.5589  	 time=0.06s
Epoch 8/30 at 5 fold: 	 loss=0.8697 	 val_loss=0.8953  	 val_acc=0.5732  	 time=0.07s
Epoch 9/30 at 5 fold: 	 loss=0.8532 	 val_loss=0.8722  	 val_acc=0.6411  	 time=0.07s
Epoch 10/30 at 5 fold: 	 loss=0.8448 	 val_loss=0.8524  	 val_acc=0.6464  	 time=0.07s
Epoch 11/30 at 5 fold: 	 loss=0.8038 	 val_loss=0.8316  	 val_acc=0.6768  	 time=0.06s
Epoch 12/30 at 5 fold: 	 loss=0.7789 	 val_loss=0.8059  	 val_acc=0.6661  	 time=0.06s
Epoch 13/30 at 5 fold: 	 loss=0.7403 	 val_loss=0.7734  	 val_acc=0.6571  	 time=0.07s
Epoch 14/30 at 5 fold: 	 loss=0.7313 	 val_loss=0.7457  	 val_acc=0.6768  	 time=0.06s
Epoch 15/30 at 5 fold: 	 loss=0.6982 	 val_loss=0.7639  	 val_acc=0.6804  	 time=0.07s
Epoch 16/30 at 5 fold: 	 loss=0.6849 	 val_loss=0.7167  	 val_acc=0.6643  	 time=0.07s
Epoch 17/30 at 5 fold: 	 loss=0.6771 	 val_loss=0.7039  	 val_acc=0.7036  	 time=0.07s
Epoch 18/30 at 5 fold: 	 loss=0.6436 	 val_loss=0.7036  	 val_acc=0.7268  	 time=0.07s
Epoch 19/30 at 5 fold: 	 loss=0.6291 	 val_loss=0.6964  	 val_acc=0.7089  	 time=0.06s
Epoch 20/30 at 5 fold: 	 loss=0.6151 	 val_loss=0.6988  	 val_acc=0.7071  	 time=0.07s
Epoch 21/30 at 5 fold: 	 loss=0.6011 	 val_loss=0.6997  	 val_acc=0.7036  	 time=0.07s
Epoch 22/30 at 5 fold: 	 loss=0.5953 	 val_loss=0.6987  	 val_acc=0.7161  	 time=0.07s
Epoch 23/30 at 5 fold: 	 loss=0.5880 	 val_loss=0.6828  	 val_acc=0.7143  	 time=0.07s
Epoch 24/30 at 5 fold: 	 loss=0.6027 	 val_loss=0.6573  	 val_acc=0.7232  	 time=0.06s
Epoch 25/30 at 5 fold: 	 loss=0.5907 	 val_loss=0.6688  	 val_acc=0.7232  	 time=0.08s
Epoch 26/30 at 5 fold: 	 loss=0.5816 	 val_loss=0.6720  	 val_acc=0.7286  	 time=0.07s
Epoch 27/30 at 5 fold: 	 loss=0.5662 	 val_loss=0.6579  	 val_acc=0.7268  	 time=0.06s
Epoch 28/30 at 5 fold: 	 loss=0.5482 	 val_loss=0.6461  	 val_acc=0.7089  	 time=0.17s
Epoch 29/30 at 5 fold: 	 loss=0.5319 	 val_loss=0.6426  	 val_acc=0.7250  	 time=0.07s
Epoch 30/30 at 5 fold: 	 loss=0.5292 	 val_loss=0.6377  	 val_acc=0.7232  	 time=0.06s

 - using lemmitization for tokenization
 - with Glove Embeddings for vectorization
 - without stratification on an unbalanced dataset

Training and Testing Deep learning Model :Bi-LSTM on the startified balanced data set using Lematization for Tokenization with Glove for Vectorization¶

lstmPipelineBalanced = Pipeline([
    ('normalizer', Normalizer('l')),
    ('estimator', LstmModelPytorch(max_features=max_features,
     n_epochs=n_epochs, batch_size=batch_size, maxlen=maxlen, embed_size=embed_size, kFolds=n_splits, debug=debug))
])

# Make the dataset balanced with stratification
df_balanced = df.groupby('label').apply(
    lambda x: x.sample(n=df['label'].value_counts().min()))
df_balanced = df_balanced.reset_index(drop=True)

lstmPipelineBalanced.fit(df[['body']], df['label'])

Epoch 1/30 at 1 fold: 	 loss=1.0942 	 val_loss=1.0307  	 val_acc=0.5597  	 time=0.29s
Epoch 2/30 at 1 fold: 	 loss=1.0097 	 val_loss=0.9425  	 val_acc=0.5597  	 time=0.06s
Epoch 3/30 at 1 fold: 	 loss=0.9398 	 val_loss=0.8697  	 val_acc=0.5597  	 time=0.07s
Epoch 4/30 at 1 fold: 	 loss=0.9094 	 val_loss=0.8551  	 val_acc=0.5597  	 time=0.06s
Epoch 5/30 at 1 fold: 	 loss=0.9119 	 val_loss=0.8577  	 val_acc=0.5597  	 time=0.07s
Epoch 6/30 at 1 fold: 	 loss=0.8915 	 val_loss=0.8474  	 val_acc=0.5775  	 time=0.07s
Epoch 7/30 at 1 fold: 	 loss=0.8905 	 val_loss=0.8321  	 val_acc=0.5704  	 time=0.07s
Epoch 8/30 at 1 fold: 	 loss=0.8692 	 val_loss=0.8274  	 val_acc=0.5882  	 time=0.07s
Epoch 9/30 at 1 fold: 	 loss=0.8571 	 val_loss=0.8080  	 val_acc=0.5989  	 time=0.07s
Epoch 10/30 at 1 fold: 	 loss=0.8308 	 val_loss=0.7824  	 val_acc=0.6221  	 time=0.07s
Epoch 11/30 at 1 fold: 	 loss=0.8018 	 val_loss=0.7604  	 val_acc=0.6613  	 time=0.07s
Epoch 12/30 at 1 fold: 	 loss=0.7847 	 val_loss=0.7415  	 val_acc=0.6471  	 time=0.07s
Epoch 13/30 at 1 fold: 	 loss=0.7634 	 val_loss=0.7370  	 val_acc=0.6524  	 time=0.07s
Epoch 14/30 at 1 fold: 	 loss=0.7313 	 val_loss=0.7543  	 val_acc=0.6453  	 time=0.07s
Epoch 15/30 at 1 fold: 	 loss=0.7172 	 val_loss=0.7138  	 val_acc=0.6613  	 time=0.07s
Epoch 16/30 at 1 fold: 	 loss=0.6923 	 val_loss=0.6903  	 val_acc=0.6774  	 time=0.07s
Epoch 17/30 at 1 fold: 	 loss=0.6833 	 val_loss=0.6911  	 val_acc=0.6756  	 time=0.07s
Epoch 18/30 at 1 fold: 	 loss=0.6541 	 val_loss=0.6865  	 val_acc=0.6649  	 time=0.07s
Epoch 19/30 at 1 fold: 	 loss=0.6506 	 val_loss=0.7002  	 val_acc=0.6560  	 time=0.07s
Epoch 20/30 at 1 fold: 	 loss=0.6371 	 val_loss=0.6883  	 val_acc=0.6791  	 time=0.07s
Epoch 21/30 at 1 fold: 	 loss=0.6374 	 val_loss=0.6348  	 val_acc=0.7023  	 time=0.07s
Epoch 22/30 at 1 fold: 	 loss=0.6174 	 val_loss=0.6836  	 val_acc=0.6667  	 time=0.07s
Epoch 23/30 at 1 fold: 	 loss=0.6152 	 val_loss=0.6025  	 val_acc=0.7148  	 time=0.08s
Epoch 24/30 at 1 fold: 	 loss=0.5895 	 val_loss=0.6092  	 val_acc=0.7166  	 time=0.08s
Epoch 25/30 at 1 fold: 	 loss=0.5787 	 val_loss=0.6028  	 val_acc=0.7184  	 time=0.07s
Epoch 26/30 at 1 fold: 	 loss=0.5694 	 val_loss=0.6153  	 val_acc=0.7148  	 time=0.07s
Epoch 27/30 at 1 fold: 	 loss=0.5541 	 val_loss=0.5908  	 val_acc=0.7326  	 time=0.07s
Epoch 28/30 at 1 fold: 	 loss=0.5304 	 val_loss=0.6037  	 val_acc=0.7077  	 time=0.08s
Epoch 29/30 at 1 fold: 	 loss=0.5316 	 val_loss=0.5875  	 val_acc=0.7255  	 time=0.08s
Epoch 30/30 at 1 fold: 	 loss=0.5208 	 val_loss=0.5678  	 val_acc=0.7344  	 time=0.07s
Epoch 1/30 at 2 fold: 	 loss=1.0995 	 val_loss=1.0248  	 val_acc=0.5597  	 time=0.14s
Epoch 2/30 at 2 fold: 	 loss=1.0013 	 val_loss=0.9300  	 val_acc=0.5597  	 time=0.12s
Epoch 3/30 at 2 fold: 	 loss=0.9335 	 val_loss=0.8716  	 val_acc=0.5597  	 time=0.08s
Epoch 4/30 at 2 fold: 	 loss=0.9218 	 val_loss=0.8548  	 val_acc=0.5597  	 time=0.07s
Epoch 5/30 at 2 fold: 	 loss=0.9075 	 val_loss=0.8549  	 val_acc=0.5597  	 time=0.07s
Epoch 6/30 at 2 fold: 	 loss=0.8919 	 val_loss=0.8430  	 val_acc=0.5633  	 time=0.06s
Epoch 7/30 at 2 fold: 	 loss=0.8933 	 val_loss=0.8319  	 val_acc=0.5615  	 time=0.06s
Epoch 8/30 at 2 fold: 	 loss=0.8780 	 val_loss=0.8217  	 val_acc=0.5740  	 time=0.06s
Epoch 9/30 at 2 fold: 	 loss=0.8599 	 val_loss=0.8073  	 val_acc=0.6078  	 time=0.06s
Epoch 10/30 at 2 fold: 	 loss=0.8379 	 val_loss=0.7755  	 val_acc=0.6203  	 time=0.06s
Epoch 11/30 at 2 fold: 	 loss=0.8227 	 val_loss=0.7450  	 val_acc=0.6613  	 time=0.07s
Epoch 12/30 at 2 fold: 	 loss=0.7852 	 val_loss=0.7055  	 val_acc=0.6774  	 time=0.07s
Epoch 13/30 at 2 fold: 	 loss=0.7618 	 val_loss=0.6807  	 val_acc=0.6916  	 time=0.07s
Epoch 14/30 at 2 fold: 	 loss=0.7398 	 val_loss=0.6745  	 val_acc=0.6738  	 time=0.07s
Epoch 15/30 at 2 fold: 	 loss=0.7223 	 val_loss=0.6672  	 val_acc=0.6774  	 time=0.07s
Epoch 16/30 at 2 fold: 	 loss=0.7045 	 val_loss=0.6771  	 val_acc=0.6809  	 time=0.07s
Epoch 17/30 at 2 fold: 	 loss=0.6967 	 val_loss=0.6531  	 val_acc=0.6952  	 time=0.06s
Epoch 18/30 at 2 fold: 	 loss=0.6646 	 val_loss=0.6581  	 val_acc=0.7041  	 time=0.07s
Epoch 19/30 at 2 fold: 	 loss=0.6470 	 val_loss=0.6410  	 val_acc=0.7023  	 time=0.06s
Epoch 20/30 at 2 fold: 	 loss=0.6459 	 val_loss=0.6330  	 val_acc=0.7059  	 time=0.07s
Epoch 21/30 at 2 fold: 	 loss=0.6343 	 val_loss=0.6369  	 val_acc=0.7005  	 time=0.07s
Epoch 22/30 at 2 fold: 	 loss=0.6149 	 val_loss=0.6333  	 val_acc=0.7094  	 time=0.07s
Epoch 23/30 at 2 fold: 	 loss=0.6289 	 val_loss=0.6265  	 val_acc=0.7184  	 time=0.07s
Epoch 24/30 at 2 fold: 	 loss=0.6160 	 val_loss=0.6349  	 val_acc=0.7166  	 time=0.07s
Epoch 25/30 at 2 fold: 	 loss=0.5970 	 val_loss=0.6396  	 val_acc=0.7148  	 time=0.07s
Epoch 26/30 at 2 fold: 	 loss=0.5906 	 val_loss=0.6348  	 val_acc=0.7201  	 time=0.07s
Epoch 27/30 at 2 fold: 	 loss=0.5875 	 val_loss=0.6225  	 val_acc=0.7255  	 time=0.08s
Epoch 28/30 at 2 fold: 	 loss=0.6018 	 val_loss=0.6247  	 val_acc=0.7130  	 time=0.06s
Epoch 29/30 at 2 fold: 	 loss=0.5698 	 val_loss=0.6140  	 val_acc=0.7415  	 time=0.06s
Epoch 30/30 at 2 fold: 	 loss=0.5712 	 val_loss=0.6261  	 val_acc=0.7130  	 time=0.07s
Epoch 1/30 at 3 fold: 	 loss=1.0692 	 val_loss=1.0260  	 val_acc=0.4554  	 time=0.21s
Epoch 2/30 at 3 fold: 	 loss=1.0009 	 val_loss=0.9472  	 val_acc=0.5768  	 time=0.07s
Epoch 3/30 at 3 fold: 	 loss=0.9404 	 val_loss=0.8764  	 val_acc=0.5607  	 time=0.06s
Epoch 4/30 at 3 fold: 	 loss=0.9104 	 val_loss=0.8639  	 val_acc=0.5607  	 time=0.07s
Epoch 5/30 at 3 fold: 	 loss=0.9052 	 val_loss=0.8724  	 val_acc=0.5714  	 time=0.07s
Epoch 6/30 at 3 fold: 	 loss=0.8991 	 val_loss=0.8710  	 val_acc=0.5696  	 time=0.07s
Epoch 7/30 at 3 fold: 	 loss=0.8974 	 val_loss=0.8565  	 val_acc=0.5804  	 time=0.06s
Epoch 8/30 at 3 fold: 	 loss=0.8822 	 val_loss=0.8527  	 val_acc=0.5750  	 time=0.07s
Epoch 9/30 at 3 fold: 	 loss=0.8794 	 val_loss=0.8437  	 val_acc=0.5875  	 time=0.07s
Epoch 10/30 at 3 fold: 	 loss=0.8455 	 val_loss=0.8257  	 val_acc=0.5982  	 time=0.06s
Epoch 11/30 at 3 fold: 	 loss=0.8292 	 val_loss=0.8017  	 val_acc=0.6214  	 time=0.07s
Epoch 12/30 at 3 fold: 	 loss=0.8016 	 val_loss=0.7776  	 val_acc=0.6446  	 time=0.07s
Epoch 13/30 at 3 fold: 	 loss=0.7756 	 val_loss=0.7630  	 val_acc=0.6643  	 time=0.07s
Epoch 14/30 at 3 fold: 	 loss=0.7608 	 val_loss=0.7420  	 val_acc=0.6571  	 time=0.07s
Epoch 15/30 at 3 fold: 	 loss=0.7224 	 val_loss=0.7271  	 val_acc=0.6589  	 time=0.06s
Epoch 16/30 at 3 fold: 	 loss=0.7000 	 val_loss=0.7382  	 val_acc=0.6625  	 time=0.07s
Epoch 17/30 at 3 fold: 	 loss=0.6781 	 val_loss=0.7313  	 val_acc=0.6696  	 time=0.07s
Epoch 18/30 at 3 fold: 	 loss=0.6655 	 val_loss=0.7258  	 val_acc=0.6696  	 time=0.18s
Epoch 19/30 at 3 fold: 	 loss=0.6527 	 val_loss=0.7434  	 val_acc=0.6768  	 time=0.07s
Epoch 20/30 at 3 fold: 	 loss=0.6427 	 val_loss=0.7143  	 val_acc=0.7036  	 time=0.07s
Epoch 21/30 at 3 fold: 	 loss=0.6208 	 val_loss=0.7326  	 val_acc=0.6964  	 time=0.06s
Epoch 22/30 at 3 fold: 	 loss=0.6074 	 val_loss=0.7308  	 val_acc=0.6911  	 time=0.07s
Epoch 23/30 at 3 fold: 	 loss=0.5905 	 val_loss=0.7319  	 val_acc=0.7089  	 time=0.08s
Epoch 24/30 at 3 fold: 	 loss=0.5874 	 val_loss=0.7467  	 val_acc=0.6929  	 time=0.07s
Epoch 25/30 at 3 fold: 	 loss=0.5683 	 val_loss=0.7097  	 val_acc=0.7089  	 time=0.07s
Epoch 26/30 at 3 fold: 	 loss=0.5573 	 val_loss=0.7223  	 val_acc=0.7000  	 time=0.06s
Epoch 27/30 at 3 fold: 	 loss=0.5354 	 val_loss=0.7272  	 val_acc=0.7089  	 time=0.07s
Epoch 28/30 at 3 fold: 	 loss=0.5324 	 val_loss=0.7247  	 val_acc=0.7071  	 time=0.07s
Epoch 29/30 at 3 fold: 	 loss=0.5295 	 val_loss=0.7371  	 val_acc=0.7018  	 time=0.07s
Epoch 30/30 at 3 fold: 	 loss=0.5172 	 val_loss=0.7279  	 val_acc=0.7125  	 time=0.07s
Epoch 1/30 at 4 fold: 	 loss=1.0287 	 val_loss=0.9609  	 val_acc=0.5589  	 time=0.29s
Epoch 2/30 at 4 fold: 	 loss=0.9581 	 val_loss=0.8665  	 val_acc=0.5589  	 time=0.11s
Epoch 3/30 at 4 fold: 	 loss=0.9100 	 val_loss=0.8145  	 val_acc=0.5589  	 time=0.07s
Epoch 4/30 at 4 fold: 	 loss=0.9243 	 val_loss=0.8126  	 val_acc=0.5589  	 time=0.06s
Epoch 5/30 at 4 fold: 	 loss=0.9064 	 val_loss=0.8205  	 val_acc=0.5589  	 time=0.06s
Epoch 6/30 at 4 fold: 	 loss=0.8967 	 val_loss=0.8077  	 val_acc=0.5589  	 time=0.06s
Epoch 7/30 at 4 fold: 	 loss=0.8887 	 val_loss=0.7986  	 val_acc=0.5589  	 time=0.06s
Epoch 8/30 at 4 fold: 	 loss=0.8800 	 val_loss=0.7895  	 val_acc=0.5571  	 time=0.07s
Epoch 9/30 at 4 fold: 	 loss=0.8621 	 val_loss=0.7779  	 val_acc=0.5857  	 time=0.06s
Epoch 10/30 at 4 fold: 	 loss=0.8433 	 val_loss=0.7429  	 val_acc=0.6054  	 time=0.07s
Epoch 11/30 at 4 fold: 	 loss=0.8137 	 val_loss=0.7196  	 val_acc=0.6464  	 time=0.07s
Epoch 12/30 at 4 fold: 	 loss=0.7951 	 val_loss=0.6810  	 val_acc=0.6589  	 time=0.06s
Epoch 13/30 at 4 fold: 	 loss=0.7696 	 val_loss=0.6670  	 val_acc=0.6661  	 time=0.07s
Epoch 14/30 at 4 fold: 	 loss=0.7442 	 val_loss=0.6625  	 val_acc=0.6714  	 time=0.07s
Epoch 15/30 at 4 fold: 	 loss=0.7275 	 val_loss=0.6772  	 val_acc=0.6643  	 time=0.07s
Epoch 16/30 at 4 fold: 	 loss=0.6974 	 val_loss=0.6391  	 val_acc=0.6696  	 time=0.06s
Epoch 17/30 at 4 fold: 	 loss=0.6749 	 val_loss=0.6237  	 val_acc=0.6714  	 time=0.07s
Epoch 18/30 at 4 fold: 	 loss=0.6671 	 val_loss=0.6286  	 val_acc=0.6714  	 time=0.07s
Epoch 19/30 at 4 fold: 	 loss=0.6479 	 val_loss=0.6781  	 val_acc=0.6804  	 time=0.06s
Epoch 20/30 at 4 fold: 	 loss=0.6416 	 val_loss=0.6122  	 val_acc=0.6982  	 time=0.06s
Epoch 21/30 at 4 fold: 	 loss=0.6267 	 val_loss=0.6365  	 val_acc=0.6982  	 time=0.07s
Epoch 22/30 at 4 fold: 	 loss=0.6022 	 val_loss=0.6045  	 val_acc=0.7143  	 time=0.07s
Epoch 23/30 at 4 fold: 	 loss=0.6043 	 val_loss=0.6326  	 val_acc=0.7036  	 time=0.06s
Epoch 24/30 at 4 fold: 	 loss=0.5854 	 val_loss=0.6222  	 val_acc=0.7196  	 time=0.06s
Epoch 25/30 at 4 fold: 	 loss=0.5614 	 val_loss=0.6130  	 val_acc=0.7214  	 time=0.06s
Epoch 26/30 at 4 fold: 	 loss=0.5519 	 val_loss=0.6026  	 val_acc=0.7179  	 time=0.07s
Epoch 27/30 at 4 fold: 	 loss=0.5488 	 val_loss=0.6124  	 val_acc=0.7196  	 time=0.06s
Epoch 28/30 at 4 fold: 	 loss=0.5345 	 val_loss=0.5871  	 val_acc=0.7286  	 time=0.06s
Epoch 29/30 at 4 fold: 	 loss=0.5320 	 val_loss=0.6126  	 val_acc=0.7179  	 time=0.06s
Epoch 30/30 at 4 fold: 	 loss=0.5201 	 val_loss=0.5777  	 val_acc=0.7232  	 time=0.07s
Epoch 1/30 at 5 fold: 	 loss=1.0816 	 val_loss=1.0231  	 val_acc=0.5589  	 time=0.28s
Epoch 2/30 at 5 fold: 	 loss=0.9796 	 val_loss=0.9612  	 val_acc=0.5589  	 time=0.08s
Epoch 3/30 at 5 fold: 	 loss=0.9155 	 val_loss=0.9556  	 val_acc=0.5589  	 time=0.06s
Epoch 4/30 at 5 fold: 	 loss=0.9265 	 val_loss=0.9465  	 val_acc=0.5625  	 time=0.06s
Epoch 5/30 at 5 fold: 	 loss=0.9158 	 val_loss=0.9312  	 val_acc=0.5804  	 time=0.07s
Epoch 6/30 at 5 fold: 	 loss=0.9030 	 val_loss=0.9270  	 val_acc=0.5643  	 time=0.07s
Epoch 7/30 at 5 fold: 	 loss=0.8868 	 val_loss=0.9235  	 val_acc=0.5661  	 time=0.06s
Epoch 8/30 at 5 fold: 	 loss=0.8818 	 val_loss=0.9064  	 val_acc=0.5964  	 time=0.06s
Epoch 9/30 at 5 fold: 	 loss=0.8701 	 val_loss=0.8954  	 val_acc=0.6071  	 time=0.07s
Epoch 10/30 at 5 fold: 	 loss=0.8536 	 val_loss=0.8778  	 val_acc=0.6268  	 time=0.07s
Epoch 11/30 at 5 fold: 	 loss=0.8229 	 val_loss=0.8545  	 val_acc=0.6643  	 time=0.06s
Epoch 12/30 at 5 fold: 	 loss=0.7909 	 val_loss=0.8380  	 val_acc=0.6714  	 time=0.06s
Epoch 13/30 at 5 fold: 	 loss=0.7703 	 val_loss=0.8172  	 val_acc=0.6571  	 time=0.07s
Epoch 14/30 at 5 fold: 	 loss=0.7346 	 val_loss=0.8085  	 val_acc=0.6893  	 time=0.07s
Epoch 15/30 at 5 fold: 	 loss=0.7469 	 val_loss=0.7997  	 val_acc=0.6911  	 time=0.06s
Epoch 16/30 at 5 fold: 	 loss=0.7119 	 val_loss=0.7927  	 val_acc=0.6857  	 time=0.06s
Epoch 17/30 at 5 fold: 	 loss=0.6947 	 val_loss=0.7758  	 val_acc=0.6929  	 time=0.07s
Epoch 18/30 at 5 fold: 	 loss=0.6734 	 val_loss=0.7754  	 val_acc=0.6821  	 time=0.06s
Epoch 19/30 at 5 fold: 	 loss=0.6571 	 val_loss=0.7501  	 val_acc=0.7000  	 time=0.06s
Epoch 20/30 at 5 fold: 	 loss=0.6377 	 val_loss=0.7423  	 val_acc=0.7036  	 time=0.07s
Epoch 21/30 at 5 fold: 	 loss=0.6345 	 val_loss=0.7206  	 val_acc=0.7179  	 time=0.06s
Epoch 22/30 at 5 fold: 	 loss=0.6021 	 val_loss=0.7005  	 val_acc=0.7143  	 time=0.06s
Epoch 23/30 at 5 fold: 	 loss=0.5979 	 val_loss=0.7306  	 val_acc=0.7339  	 time=0.07s
Epoch 24/30 at 5 fold: 	 loss=0.5711 	 val_loss=0.7305  	 val_acc=0.7411  	 time=0.07s
Epoch 25/30 at 5 fold: 	 loss=0.5674 	 val_loss=0.7186  	 val_acc=0.7411  	 time=0.07s
Epoch 26/30 at 5 fold: 	 loss=0.5571 	 val_loss=0.7175  	 val_acc=0.7429  	 time=0.07s
Epoch 27/30 at 5 fold: 	 loss=0.5491 	 val_loss=0.7110  	 val_acc=0.7321  	 time=0.06s
Epoch 28/30 at 5 fold: 	 loss=0.5390 	 val_loss=0.7521  	 val_acc=0.7321  	 time=0.07s
Epoch 29/30 at 5 fold: 	 loss=0.5414 	 val_loss=0.6999  	 val_acc=0.7446  	 time=0.07s
Epoch 30/30 at 5 fold: 	 loss=0.5206 	 val_loss=0.7110  	 val_acc=0.7500  	 time=0.06s

 - using lemmitization for tokenization
 - with Glove Embeddings for vectorization
 - without stratification on an unbalanced dataset

Pipeline(steps=[('normalizer', Normalizer(options='l')),
                ('estimator',
                 LstmModelPytorch(batch_size=512, debug=0, embed_size=50,
                                  kFolds=5, max_features=120000, maxlen=50,
                                  n_epochs=30))])

Using our LSTM Model¶

For both versions of our LSTM model (trained on balanced data set and unbalanced data set), we made a predict function where we enter a tweet and the model predicts whether it is Positive (1), Neutral (0) or Negative (-1).

lstmPipelineBalanced.predict(pd.DataFrame(['Expo 2020 was not fun'], columns=['body']))

1

pyLDAvis¶

tfidf_vectorizer = TfidfVectorizer()
normalized_df = Normalizer('l').transform(df)
df_tfidf = tfidf_vectorizer.fit_transform(normalized_df)

11 topics¶

lda_tfidf = LatentDirichletAllocation(n_components=11, random_state=0)
lda_tfidf.fit(df_tfidf)
pyLDAvis.sklearn.prepare(lda_tfidf, df_tfidf, tfidf_vectorizer)

20 topics¶

lda_tfidf = LatentDirichletAllocation(n_components=20, random_state=0)
lda_tfidf.fit(df_tfidf)
pyLDAvis.sklearn.prepare(lda_tfidf, df_tfidf, tfidf_vectorizer)

35 topics¶

lda_tfidf = LatentDirichletAllocation(n_components=35, random_state=0)
lda_tfidf.fit(df_tfidf)
pyLDAvis.sklearn.prepare(lda_tfidf, df_tfidf, tfidf_vectorizer)

Observation from Topic Modelling¶

We observed the topic modelling with 11, 20 and 35 number of topics as parameters and with 0.42 as the relevance metric. In all the parameters we find that, most topics are overlapping each other in terms of the intertopic distance. There are few outliers for all the cases. For all the 3 observation, the general rule given below are followed:

There is an outlier with violent speech, like war, conflict, fatal and uae not being safe. This may be because our twitter data collection was done around the time of UAE and Yemen conflict
There is an outlier based on trading stocks, crypto and technologies on crypto currency.
The Overlapping topics describes different pavillions, and events. This is consistent on any number of topics larger than 8

Below we are describing the common observation found from 11, 20 and 35 number of topics as parameters.

Slovenia pavilion is described to display forestry. To investigate further, our research on the slovenia pavillion showed that its decorated with forestry and displays the beautiful natures in slovenia
Celebration of Rwanda national day along with investment opportunity in Rwanda.
One of the topics are comprised of names of the sheikhs in UAE
Healthcare and wellness
Described of Saudi Arabia's pavillion with words: virtual, aroma, coffee, flavor
Topic on GCC countries airplane crews.

Evaluation and Conclusions¶

The key take aways from our model is that it does not fully grasp the nuances of a tweet, such as sarcasm or any implicit meanings, this is mainly due to the limited corpus size we are using, as research papers such as [1] use more than 10k tweets while our corpus was limited to only 2084. The quality of tweets also limited the use of negation such as using "Not good", as our corpus did not contain such tweets. This is verfied as each tweet was manually labelled, more so other papers such as [1] would remove negation, hence their model was not trained for negation tweets. Some aspects such as removing URLs strengthened our arguments for our pre-processing methodology by these research papers, especially emojis [2]. However, new research paper was found with the same corpus or the same topic for which we conducted opinion mining hence a proper comparison cannot be made with research papers.

More so, since our ML models have relativey miniscule data, they are under trained and have not seen the majority of possible ways to convey similar sentiments, unlike as previously discussed other ML models which are trained such as GPT-3, which uses 175 billion parameters. Other key takeaways include the heavily skewed corpus in the real world as compared to a closed environment where all the data is perfectly balanced, in our case, the heavily skewed dataset meant our ML models would mostly learn only positive sentiments and even a simple tweet such as 'Pavillion #EXPO2020' would return a positive sentiment. Another example in our case is the tweets regarding "South Africa Pavillion" as almost 90% of the tweets related to that specific pavillion were negative, hence our model associates it with negative sentiments. Due to the lack of neutral and negative tweets, our model has mainly learnt negative sentiments.

The issues in retrieving or creating a well balanced and a densely populated data set is due to non-linearity of retrieving tweets involving subjective opinions of people which makes opinion mining a complex problem to solve. The real world like our data set gave a heavily skewed dataset which is almost always the case and due to the nature of tweets and the way they are written, multiple stages are involved in the normalisation of each tweet. Also due to a lot of twitter bots active and spamming #, a lot of tweets are irrelevant/spam tweets which is why our initial corpus of 95k is down to 2082 tweets.

References¶

[1] Z. Jianqiang and G. Xiaolin, "Comparison Research on Text Pre-processing Methods on Twitter Sentiment Analysis," in IEEE Access, vol. 5, pp. 2870-2879, 2017, doi: 10.1109/ACCESS.2017.2672677.

[2] A. Sarlan, C. Nadam and S. Basri, "Twitter sentiment analysis," Proceedings of the 6th International Conference on Information Technology and Multimedia, 2014, pp. 212-216, doi: 10.1109/ICIMU.2014.7066632.

Action	Tweet (Result)
Tweet before cleaning	A huge worldwide THANK YOU to the Unsung Heroes, our vital frontline Health Workers 👩‍⚕️ 👨‍⚕️ We couldn’t be where we’re at right now without every single one of YOU 💙 @FrontlineUAE #Expo2020 #Dubai https://t.co/7ips0I1gja
Tweet after removing the "#" symbol and any links from the tweet	A huge worldwide THANK YOU to the Unsung Heroes, our vital frontline Health Workers 👩‍⚕️ 👨‍⚕️ We couldn’t be where we’re at right now without every single one of YOU 💙 @FrontlineUAE Expo2020 Dubai
Tweet after removing all words starting with @ and words within & and ; (html tags)	A huge worldwide THANK YOU to the Unsung Heroes, our vital frontline Health Workers 👩‍⚕️ 👨‍⚕️ We couldn’t be where we’re at right now without every single one of YOU 💙 Expo2020 Dubai
Tweet after replacing emojis with their text form	A huge worldwide THANK YOU to the Unsung Heroes, our vital frontline Health Workers womanhealthworker manhealthworker We couldn’t be where we’re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after removing punctuation	A huge worldwide THANK YOU to the Unsung Heroes our vital frontline Health Workers womanhealthworker manhealthworker We couldn t be where we re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after removing all non ascii characters	A huge worldwide THANK YOU to the Unsung Heroes our vital frontline Health Workers womanhealthworker manhealthworker We couldn t be where we re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after removing words containing only numbers	A huge worldwide THANK YOU to the Unsung Heroes our vital frontline Health Workers womanhealthworker manhealthworker We couldn t be where we re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after removing all single characters	A huge worldwide THANK YOU to the Unsung Heroes our vital frontline Health Workers womanhealthworker manhealthworker We couldn be where we re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after substituting multiple spaces with single space	A huge worldwide THANK YOU to the Unsung Heroes our vital frontline Health Workers womanhealthworker manhealthworker We couldn be where we re at right now without every single one of YOU blueheart Expo2020 Dubai
Tweet after converting to lower case	a huge worldwide thank you to the unsung heroes our vital frontline health workers womanhealthworker manhealthworker we couldn be where we re at right now without every single one of you blueheart expo2020 dubai

	body	countries	tags
count	3870	2902	2556
unique	3870	39	394
top	VIDEO:\nPrime Minister, @EdNgirente officiates...	[(Tunisia pavilion), (Turkey pavilion), (Turkm...	[a+, abound, abounds, abundance, abundant]
freq	1	808	440

	body	label
count	3854	3854
unique	3854	5
top	VIDEO:\nPrime Minister, @EdNgirente officiates...	Positive
freq	1	1568

	body	countries	tags
0	Wow, this gonna be an awesome performance. \n#...	NaN	NaN
1	We are excited to welcome @issfjo as a communi...	NaN	NaN
2	#DubaiExpo2020 \nVisit 🇿🇼 #zimpavilion #expo2...	NaN	NaN
3	Recognition is priceless! #reemalhashimy #expo...	NaN	NaN
4	Mr. Parag Ghosh, Founder & CEO of Auspice ...	NaN	NaN
5	Will be sharing my thoughts at the Rwanda Busi...	NaN	NaN
6	#IweWosvora\n\n#Zimbabwe’s healthcare system h...	NaN	NaN
7	“We built a city, and then we lent it to @expo...	NaN	NaN
8	15 Places You Must Visit In the World 🌎 \| BBI ...	NaN	NaN
9	At GTR MENA 2022, @FABConnects @OxfordEconomic...	NaN	NaN
10	Catch a recap on https://t.co/iKOHLUidUv and j...	NaN	NaN
11	We had such a wonderful time seeing all of you...	NaN	NaN
12	@Annamartling at @karolinskainst and Ebba Hall...	NaN	NaN
13	Certainly not to be missed if you are part of ...	NaN	NaN
14	The Malaysian Rubber Council is showcasing mad...	[(Zambia pavilion), (Zimbabwe pavilion)]	NaN
15	District 2020 - the planned legacy of resident...	NaN	NaN
16	After a great Rwanda National day at #expo2020...	NaN	NaN
17	Come and Join us on saturday 5th february at 7...	NaN	NaN
18	Genomics Medicine Conference \nBreakthroughs &...	[(Zambia pavilion), (Zimbabwe pavilion)]	NaN
19	#DubaiInteriors supply gorgeous and luxurious ...	NaN	NaN
20	At #RisalaFurniture, you will find widest rang...	NaN	NaN
21	What an incredible January with various meetin...	NaN	NaN
22	Read more: https://t.co/OUBGbXRe0q\n\n#MTC #Ma...	NaN	NaN
23	CEO Clubs Network is proud to announce its Cou...	[(Zambia pavilion), (Zimbabwe pavilion)]	NaN
24	#CarpetsDubai provide best quality #vinyl #Ski...	NaN	NaN
25	At #InteriorDubai #Kazak #Rugs are highly affo...	NaN	NaN
26	New Dubai Vlog Check it out here 👇\n\n#dubai #...	NaN	NaN
27	At #VinylFlooring , #Vinyl #CarpetTiles Dubai ...	NaN	NaN
28	Basically a fancy spaza shop with no aircon th...	NaN	NaN
29	“Champions have a way of making things happen ...	NaN	NaN
30	#ParquetFlooring supply appealing and noticeab...	NaN	NaN
31	There are countless experiences across this la...	NaN	NaN
32	Expo 2020 Dubai top events \n\n#إكسبو2020 \n#E...	NaN	NaN
33	India’s beautiful oral music tradition lives o...	NaN	NaN
34	It is always your next move!\n#businessadvisor...	NaN	NaN
35	🔵⚪ From 28 February, the enfant terrible of fa...	NaN	NaN
36	VIP entrance at the Morocco pavilion at Expo 2...	[(Zambia pavilion), (Zimbabwe pavilion)]	NaN
37	🚇 Until 21 February, dive into the pharaonic #...	NaN	NaN
38	🍳 From 10 to 22 February, meet on the esplanad...	[(Zambia pavilion), (Zimbabwe pavilion)]	NaN
39	The spaces of the future must be designed with...	NaN	NaN

	body	label
0	Wow, this gonna be an awesome performance. \n#...	Positive
1	We are excited to welcome @issfjo as a communi...	Positive
2	Catch a recap on https://t.co/iKOHLUidUv and j...	Spam
3	Are you wondering what the Dubai Expo is about...	Neutral
4	Come to #Expo2020 with your family and get mes...	Positive
5	Expo 2020’s UK pavilion showcases the first pr...	Neutral
6	South African 🇿🇦 Rapper \nrecording his new si...	Spam
7	South African 🇿🇦 Rapper \nrecording his new si...	Neutral
8	South African 🇿🇦 Rapper \nrecording his new si...	Spam
9	Dubai Expo 2020\n\n"Connecting Minds, Creating...	Neutral
10	We can make your dreams come true. #Belarus #I...	Spam
11	Let's take the first step together. #Uzbekista...	Neutral
12	Dubai ruler tours the pavilion of Germany at t...	Neutral
13	Discover Azerbaijan with Frisaga. #Ukraine #Uz...	Neutral
14	Rwanda National Day at #Expo2020Dubai \n\n#Her...	Positive
15	.\n\nThe fractional ownership investment at SL...	Spam
16	A scale model of Hyperloop is at the Spain Pav...	Positive
17	It was an honor inviting our friends from USA ...	Positive
18	Al Ali Yacht Celebrating #50th #nationaldayuae...	Spam
19	@AliZafarsays thank u for this... It was su h ...	Positive
20	#ExperienceIndia at the Nakheel Mall in Palm J...	Negative
21	Zimbabwe Deputy Minister of Health and Child C...	Neutral
22	Passionate dancers, romantic songs and delicio...	Positive
23	Expo 2020 Dubai’s Pakistan pavilion welcomes a...	Positive
24	"Breaking Barriers Through Digital Medicine" b...	Positive
25	ADPHC participated in 2 events held at #Expo20...	Neutral
26	Leading figure in Indipop and the Bollywood in...	Positive
27	Really great time in Dubai with customers and ...	Positive
28	Register for AED 100 at https://t.co/gH7N3bOrP...	Spam
29	Look: #Dubai gets Dh13-million ambulance respo...	Positive
30	Discover ideas and innovations for a more sust...	Positive
31	Self Storage Dubai provides flexible and conve...	Spam
32	Our world and our wellbeing are interconnected...	Positive
33	Expo 2020 Dubai hosts football legend Cristian...	Neutral
34	Look: #Dubai gets Dh13-million ambulance respo...	Positive
35	Dubai reveals the world’s fastest and most exp...	Positive
36	Get combos now. Pls log on https://t.co/kmmQQo...	Spam
37	Golf meets @EXPO2020Dubai 👋\n\n@Collin_Morikaw...	Positive
38	Our exhibition is presented in a tour format a...	Neutral
39	They are talking about Asiwaju traveling abroa...	Spam

	body	label
0	Wow, this gonna be an awesome performance. \n#...	1
1	We are excited to welcome @issfjo as a communi...	1
2	Are you wondering what the Dubai Expo is about...	0
3	Come to #Expo2020 with your family and get mes...	1
4	Expo 2020’s UK pavilion showcases the first pr...	0
5	South African 🇿🇦 Rapper \nrecording his new si...	0
6	Dubai Expo 2020\n\n"Connecting Minds, Creating...	0
7	Let's take the first step together. #Uzbekista...	0
8	Dubai ruler tours the pavilion of Germany at t...	0
9	Discover Azerbaijan with Frisaga. #Ukraine #Uz...	0
10	Rwanda National Day at #Expo2020Dubai \n\n#Her...	1
11	A scale model of Hyperloop is at the Spain Pav...	1
12	It was an honor inviting our friends from USA ...	1
13	@AliZafarsays thank u for this... It was su h ...	1
14	#ExperienceIndia at the Nakheel Mall in Palm J...	-1
15	Zimbabwe Deputy Minister of Health and Child C...	0
16	Passionate dancers, romantic songs and delicio...	1
17	Expo 2020 Dubai’s Pakistan pavilion welcomes a...	1
18	"Breaking Barriers Through Digital Medicine" b...	1
19	ADPHC participated in 2 events held at #Expo20...	0
20	Leading figure in Indipop and the Bollywood in...	1
21	Really great time in Dubai with customers and ...	1
22	Look: #Dubai gets Dh13-million ambulance respo...	1
23	Discover ideas and innovations for a more sust...	1
24	Our world and our wellbeing are interconnected...	1
25	Expo 2020 Dubai hosts football legend Cristian...	0
26	Look: #Dubai gets Dh13-million ambulance respo...	1
27	Dubai reveals the world’s fastest and most exp...	1
28	Golf meets @EXPO2020Dubai 👋\n\n@Collin_Morikaw...	1
29	Our exhibition is presented in a tour format a...	0
30	Expo 2020 Dubai top events \n\n#إكسبو2020 \n#E...	1
31	At 10am we're ready to welcome you. Book ahead...	1
32	Chairman of Abu Dhabi Executive Office visits ...	0
33	To all the explorers, wanderers and travelers ...	1
34	Full Video Link : https://t.co/91DaOYmxfd\nCri...	0
35	The view from the Morocco Pavilion #Expo2020Du...	0
36	Highlights from Rwanda National Day at Dubai E...	1
37	📽️ The moment Cristiano Ronaldo (@Cristiano) ...	0
38	Join us at #expo2020 Dubai for a unique opport...	1
39	Cristiano Ronaldo was given a warm welcome at ...	1

F20AA Applied Text Analytics Coursework 1¶

UG_CwGroup 2¶

Authors:¶

Baber Jan - bj58¶

Gaurav Gosain - gg68¶

Muhammad Assad Khan - mk227¶

Ashab Uddin - mu15¶

Environment Setup¶

Data Collection¶

The following code is used to import the Keys from the environment file required to authorize the api access by tweepy¶

Labelling of Corpus¶

Corpus Visualisation and Analysis¶

1052 spam tweets¶

Accuracy of TextBlob's auto-generated labels compared to manually labeled data is 63.78%¶

Accuracy of NLTK-Vaders' auto-generated labels compared to manually labeled data is 66.17%¶

Word Cloud for Positive Tweets¶

Word Cloud for Negative Tweets¶

Word Cloud for Neutral Tweets¶

Pipeline Creation¶

Using Lemmatization + Stemming¶

Cleaning Tweet¶

Found 3 Emoji/(s) in the tweet¶

Emoji/s: {'💙': 'blue heart', '👩\u200d⚕️': 'woman health worker', '👨\u200d⚕️': 'man health worker'}¶

Lemmatizing Tweet¶

Stemming Tweet¶

Tweet after Normalization¶

huge worldwid thank unsung hero vital frontlin health worker womanhealthwork manhealthwork couldn right everi singl blueheart expo2020 dubai¶

Model Optimisation¶

Evaluation Methodology¶

Creating, Training and Testing different Machine Learning Models on the unblanced data set with various n-gram and vectorization techniques.¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: BernoulliNB¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: KNeighborsClassifier¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: LogisticRegression¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MLPClassifier¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: MultinomialNB¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: RandomForestClassifier¶

Trained and Tested Model: SGDClassifier¶

Trained and Tested Model: SGDClassifier¶