Motivation:

The General Elections in India are scheduled to be held between April and May. The volume of data generated through news and social media is huge. What an intriguing project to take up as a data scientist!

Data Science offers algorithms to analyze those text, audio, and video data to infer the nation’s sentiment before, during, and after the elections. The statistical analysis of information available from all sources can provide a deep insight into:

• Party, party leaders and their actions since the last election was held
• Polarized tweets and public sentiments
• Variation of polarized tweets over a time-frame of 2.5 months and party wise comparisons
• Identifying names of party leaders
• Special events from the frequency of polarized tweets and comparing them between parties
• Actionable items of different parties, party leaders during election campaigns and promotions
• The sentiment of one party towards another as well as sentiment between contending leaders and coalition parties
• Election campaigns, its popularity and how it affects the general public
• Location-based insights of tweets, like assembly seats, won or lost by parties at different locations
• Drift or sudden change in moods due to announcement of poll results for a single state or multiple states
• Elected leaders, their promises and actions
• Introduction of completely new constitutional amendments
• Economic, social and political changes that a nation undergoes between successive terms and how it affects the country’s GDP and foreign relations

This blog is part 2 of my previous article on Text-based classification of election sentiments. In this blog we’ll cover:

• Other text classification techniques and methods to identify tweets based on defined categories using Named Entity Recognition (NER)
• Text based classifiers like Decision Tree and other classifiers SGD (Stochastic Gradient Descent) , SVM (Kernel -Linear and RBF)
• Polarized tweet classification and Parts-Of-Speech Tagging

Let’s dive in!

Decision Tree Classifier

Decision tree classifier is composed of decision nodes and leaf nodes, where the decision of validating feature values and creating branches are assigned to decision nodes, while labels are assigned to leaf nodes.

Decision trees grow by selecting the best decision stumps of highest accuracy during classification process. Leaf nodes with error below acceptable margin are replaced by new decision stumps on subset of training data that excludes the path from root of the subtree to the leaf.

The best decision stump can be chosen by computing the information gain or entropy of the system. Information gain serves as an estimation of the degree of disorganization for the input values due to same or different labels assigned to them.

Mathematically, entropy is defined as the sum of the probability of each label times the log probability of that same label.

Hence, in situations where there are different labels of varying frequency assigned to the input, both the values P(l) or log2P(l) range from medium to high, causing an increase to the entropy of the system.

Results using Decision Tree Classifier with input features : Word frequencies of n-grams (1/2/3 gram words), positive, negative, compound scores obtained from SentimentAnalyzer, tweet polarity and retweet count.

RESULTS

BJP
Accuracy Decision Tree Classifier : 0.47
Confusion Matrix from Decision Tree Classifier
  |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <.> .  3  .  .  .  .  . |
1 |  . <1> 2  .  .  4  3  . |
2 |  .  2<11> 1  1 13  2  . |
3 |  .  .  . <1> .  4  .  . |
4 |  .  .  2  . <.> 3  .  . |
5 |  2  .  3  1  1<27> 2  . |
6 |  .  .  1  .  .  . <7> . |
7 |  .  .  2  .  .  1  . <.>|
--+-------------------------+
Decision Tree Precision 0.46
Decision Tree Recall 0.47
Decision Tree F_Score 0.44
Congress
Accuracy Decision Tree Classifier : 0.94
Confusion Matrix from using Decision Tree Classifier
  |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <2> .  1  .  .  .  .  . |
1 |  . <8> .  .  .  .  2  . |
2 |  .  .<29> .  .  1  .  . |
3 |  .  .  . <5> .  .  .  . |
4 |  .  .  .  . <5> .  .  . |
5 |  .  .  .  .  .<35> 1  . |
6 |  .  .  1  .  .  . <7> . |
7 |  .  .  .  .  .  .  . <3>|
--+-------------------------+
(row = reference; col = test)
Decision Tree Precision 0.97
Decision Tree Recall 0.97
Decision Tree F_Score 0.97

As the moods comprise of imbalanced classes, the model performance is plotted using precision-recall curve. The precision-recall curve for multi-class mood classification can be plotted as:

CODE

def plot_precision_recall(precision, recall, f1_score, text):
    num_classes = 8
    colors = ['blue', 'green', 'red', 'cyan', 'magenta', 'yellow', 'black', 'gray']
    legends = ['anger', 'arousal', 'dominance', 'faith',  'fear', 'joy', 'neutral', 'sadness']
    plt.figure(figsize=(7, 8))
    f_scores = np.linspace(0.2, 0.8, num=4)
    lines = []
    labels = []
    count = 0
    for f_score in f1_score:
        x = np.linspace(0.01, 1)
        y = f_score * x / (2 * x - f_score)
        l, = plt.plot(x[y >= 0], y[y >= 0], color=colors[count], alpha=0.2)
        plt.annotate('f1={0:0.1f}'.format(f_score), xy=(0.9, y[45] + 0.02))
        count = count + 1

    for i in range(num_classes):
        l, = plt.plot(recall[i], precision[i], color=colors[i], lw=2)
        lines.append(l)

    fig = plt.gcf()
    fig.subplots_adjust(bottom=0.25)
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Precision-Recall curve of Multi-class Moods for ' + text + ' using FastText')
    plt.legend(lines, legends, loc=1, prop=dict(size=6))
    plt.show()

Analyzing the prediction models of BJP and Congress, it is seen that model accuracy (precision, recall and F-Score) of Congress out-performs BJP by a large margin (over 45%)

Classifiers from Sklearn

a. Stochastic Gradient Descent (SGD) is one of the simple and efficient approach to discriminative learning of linear classifiers under convex loss functions. SGD has been known to handle large-scale, sparse ML problems for text-classification and NLP in a convenient manner.

sgd = SklearnClassifier(SGDClassifier(loss='log'))

RESULTS

Results BJP
------------

Accuracy by using SGDClassifier: 0.5
SGD Precision : 0.55
SGD Recall 0.58
SGD F_Score 0.53
SGD Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <.> .  2  .  .  1  .  . |
1 |  . <2> 3  .  .  4  1  . |
2 |  .  .<22> .  .  8  .  . |
3 |  .  .  1 <.> .  4  .  . |
4 |  .  .  1  . <.> 4  .  . |
5 |  .  .  8  .  .<27> 1  . |
6 |  .  .  .  .  .  2 <6> . |
7 |  .  .  1  .  .  1  . <1>|
--+-------------------------+
(row = reference; col = test)


Results Congress
----------------

Accuracy by using SGDClassifier: 0.51
SGD Precision : 0.79
SGD Recall : 0.76
SGD F_Score : 0.73
SGD Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <2> .  .  .  .  1  .  . |
1 |  . <4> 3  .  .  3  .  . |
2 |  .  .<28> .  .  2  .  . |
3 |  .  1  . <1> .  3  .  . |
4 |  .  .  1  . <1> 3  .  . |
5 |  .  .  4  .  .<31> 1  . |
6 |  .  .  .  .  .  1 <7> . |
7 |  .  .  1  .  .  .  . <2>|
--+-------------------------+
(row = reference; col = test)

b. Support vector Machines (SVM) has both linear and RBF kernel to determine the best decision boundary for text classification. However SVMs find restricted usage to classifying huge corpus of text due to their training complexity.

classifier = SklearnClassifier(SVC(kernel='linear')).train(trainset)classifier = SklearnClassifier(SVC(kernel='rbf')).train(trainset)

RESULTS

BJP Results Linear SVM
-----------------------

Accuracy by using Linear SVM: 0.58
Linear SVM Precision : 0.57
Linear SVM Recall : 0.58
Linear SVM F_Score : 0.54
Linear SVM Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <2> .  .  .  .  1  .  . |
1 |  2 <2> 2  .  .  3  1  . |
2 |  .  1<15> .  . 12  1  1 |
3 |  .  .  . <1> .  4  .  . |
4 |  .  1  3  . <.> 1  .  . |
5 |  .  .  4  .  1<29> 2  . |
6 |  .  .  1  .  .  . <7> . |
7 |  .  .  1  .  .  .  . <2>|
--+-------------------------+
(row = reference; col = test)


Congress Results Linear SVM
-----------------------------

Accuracy by using Linear SVM: 0.65
Linear SVM Precision : 0.65
Linear SVM Recall : 0.65
Linear SVM F_Score : 0.64
Linear SVM Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <2> .  .  .  .  1  .  . |
1 |  1 <4> 2  .  .  2  1  . |
2 |  .  1<17> .  1  9  1  1 |
3 |  .  .  . <3> .  2  .  . |
4 |  .  1  2  . <1> 1  .  . |
5 |  .  .  5  .  .<29> 2  . |
6 |  .  .  1  .  .  . <7> . |
7 |  .  .  1  .  .  .  . <2>|
--+-------------------------+
(row = reference; col = test)


BJP Results RBF SVM
--------------------
Accuracy by using RBF SVM: 0.58
RBF SVM Precision : 0.55
RBF SVM Recall 0.58
RBF SVM F_Score 0.53
RBF SVM Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <.> .  2  .  .  1  .  . |
1 |  . <2> 3  .  .  4  1  . |
2 |  .  .<22> .  .  8  .  . |
3 |  .  .  1 <.> .  4  .  . |
4 |  .  .  1  . <.> 4  .  . |
5 |  .  .  8  .  .<27> 1  . |
6 |  .  .  .  .  .  2 <6> . |
7 |  .  .  1  .  .  1  . <1>|
--+-------------------------+
(row = reference; col = test)


Congress Results RBF SVM
------------------------

Accuracy by using RBF SVM: 0.76
RBF SVM Precision : 0.79
RBF SVM Recall : 0.76
RBF SVM F_Score : 0.73
RBF SVM Confusion Matrix   |  0  1  2  3  4  5  6  7 |
--+-------------------------+
0 | <2> .  .  .  .  1  .  . |
1 |  . <4> 3  .  .  3  .  . |
2 |  .  .<28> .  .  2  .  . |
3 |  .  1  . <1> .  3  .  . |
4 |  .  .  1  . <1> 3  .  . |
5 |  .  .  4  .  .<31> 1  . |
6 |  .  .  .  .  .  1 <7> . |
7 |  .  .  1  .  .  .  . <2>|
--+-------------------------+
(row = reference; col = test)

From the comparative results between BJP and Congress, its seen that for both Linear and RBF kernel Congress mood prediction gives better accuracy, precision, recall and F-score. Moreover, BJP predictions does not change much between Linear and RBF kernel, whereas for Congress we see 10%–11% increase in accuracy results for RBF kernel. In the next section, let’s study how sensitive and polarized tweet classification and how it differs for different models.

In the next section I explain Parts of Speech Tagger and see illustrate a comparative study of the most prominent parties over last 2.5 months.

Parts of Speech Tagger:

Parts-of-speech classifier comes with the functionality of extracting and detecting features by considering words in the context of neighborhood words. The identification of previous words, the parts of speech it belongs to, acts as a feature to the current word for evaluating which part of speech-tag should be assigned to it. This also improves the performance of part-of-speech classifier. In certain cases the classifier is able to learn a noun, when it follows an adjective.

For example, for the following tweet “It was BJP’s loss while the Congress gained big in the state elections. What impact will this have on. “ The classifier is able to guess a noun will follow the adjective “big”, but there could be in-between articles or pronouns (in the) before the noun (state elections). Another tweet for example “The Nationalist Congress Party and the Congress have reached a seat-sharing agreement on 45 of the 48 seats in Maharashtra’s kitty”identifies proper noun (party names and location names marked in bold) as follows:

[(‘The’, ‘DT’), (‘Nationalist’, ‘NNP’), (‘Congress’, ‘NNP’), (‘Party’, ‘NNP’), (‘and’, ‘CC’), (‘the’, ‘DT’), (‘Congress’, ‘NNP’), (‘have’, ‘VBP’), (‘reached’, ‘VBN’), (‘a’, ‘DT’), (‘seat-sharing’, ‘JJ’), (‘agreement’, ‘NN’), (‘on’, ‘IN’), (‘45’, ‘CD’), (‘of’, ‘IN’), (‘the’, ‘DT’), (‘48’, ‘CD’), (‘seats’, ‘NNS’), (‘in’, ‘IN’), (‘Maharashtra’, ‘NNP’), (‘’’, ‘NNP’), (‘s’, ‘VBD’), (‘kitty’, ‘NNS’)]

It also identifies numbers with tag “CD” (45 and 48), that are more useful during counting votes and announcement of results.

We can tag each word in the tweet using Parts-of-Speech tagger as follows:

CODE

import nltk
from nltk.tokenize import *
from nltk.util import ngrams
from nltk.classify import *
from sklearn.preprocessing import LabelEncoder

data = combined_df.ix[:, ['tweet', 'mood', 'compound', 'pos', 'neg', 'neu', 'polarity', 'retweet_count']]
data_sentiment = label_encode(combined_df['mood'])
data['tokenized_tweet'] = data.apply(lambda row: nltk.word_tokenize(row['tweet']), axis=1)

for i, row in data.iterrows():
     s = word_tokenize(row[0])
     s = nltk.pos_tag(s)
     data.set_value(i, 'tweet', s)
     data.set_value(i, 'mood', data_sentiment[i])

As we tag each word in a sentence, we can further extract n-gram words with specific tags for classifying election moods and predicting them.

Classification of Highly Polarized Tweets : Adjectives and Adverbs

The n-gram model is created by taking into consideration adverbs and adjectives from highly polarized tweets. NLTK python package uses part-of-speech classifier to determine the context in which a word appears and labels the word accordingly to the appropriate part of speech tag. Discovering word context and feeding the contextual features improves the accuracy of classifier, as it classifies tweets with more adjectives and adverbs.

For example, one such tweet which uses part-of-speech tagger for classification is : “What a message as though its a great achievement. @BJP4India u appointed state incharges too but no hullaballoo” has adjective “great” , adverb “too” and noun “achievement” . As the tweet is classified with POS, the words as tagged as follows:

[(‘What’, ‘WP’), (‘a’, ‘DT’), (‘message’, ‘NN’), (‘as’, ‘IN’), (‘though’, ‘IN’), (‘its’, ‘PRP$’), (‘a’, ‘DT’), (‘great’, ‘JJ’), (‘achievement’, ‘NN’), (‘BJP4India’, ‘NNP’), (‘u’, ‘NN’), (‘appointed’, ‘VBD’), (‘state’, ‘NN’), (‘incharges’, ‘NNS’), (‘too’, ‘RB’), (‘but’, ‘CC’), (‘no’, ‘DT’), (‘hullaballoo’, ‘NN’)]

Such polarized sentiment classification are more useful, when parties promote campaigns, leaders organize meetings, or the election results of a location are published. Such an illustration is given below.

The following code and figures show frequencies of Adjectives — JJ, JJR (adjective, comparative) , JJS (adjective, superlative), Adverbs — RB , RBR (adverb, comparative) and RBS (adverb, superlative) and Proper Nouns (NNP and NNPS) for tweets belonging to BJP, Congress and both parties. Its clearly visible that on same dates frequency of adjectives and adverbs increases showing strong public sentiments in the tweets.

CODE

labels = ['BJP', 'Congress', 'BJP-Congress']
titles = ['Parts of Speech (Adjectives/Adverbs/Proper Nouns) Tagging for BJP', 'Parts of Speech (Adjectives/Adverbs/Proper Nouns) Tagging for Congress', 'Parts of Speech (Adjectives/Adverbs/Proper Nouns) Tagging for BJP and Congres']

for fcount in range(len(files)):
    if(files[fcount].endswith('.csv')):
        combined_df = pd.read_csv(path + files[fcount])
        data = combined_df.ix[:, ['tweet', 'mood', 'created_at']]
        data_sentiment = label_encode(combined_df['mood'])
        data_date = pd.to_datetime(combined_df['created_at'])
        n = len(data)

        data['tokenized_tweet'] = data.apply(lambda row: nltk.word_tokenize(row['tweet']), axis=1)


        for i, row in data.iterrows():
            s = word_tokenize(row[0])
            s = nltk.pos_tag(s)
            data.set_value(i, 'tweet', s)
            data.set_value(i, 'mood', data_sentiment[i])
            data.set_value(i, 'created_at', data_date[i])

        dateMap = {}
        for index, row in data.iterrows():
            adj = 0
            adv = 0
            pn = 0

            for i in range(len(row[0])):
                time = row['created_at']._date_repr
                if row[0][i][1] == 'RB' or row[0][i][1] == 'RBR' or row[0][i][1] == 'RBS':
                    adv = adv + 1
                elif row[0][i][1] == 'JJ' or row[0][i][1] == 'JJR' or row[0][i][1] == 'JJS':
                    adj = adj + 1
                elif row[0][i][1] == 'NNP' or row[0][i][1] == 'NNPS':
                    pn = pn + 1
            if(time in dateMap):
                oldData = dateMap[time]
                dateMap[time] = [adv+oldData[0], adj+oldData[1], pn+oldData[2]]
            else:
                dateMap[time] = [adv, adj, pn]



        sortedDate = sorted(dateMap.items())
        ndf = pd.DataFrame(0, index=range(len(sortedDate)),
                          columns=['date', 'adv', 'adj', 'nn'])
        count = 0

        for item in sortedDate:
            date_ = item[0]
            vals = item[1]
            ndf.set_value(count, 'date', pd.to_datetime(date_))
            ndf.set_value(count, 'adv', vals[0])
            ndf.set_value(count, 'adj', vals[1])
            ndf.set_value(count, 'nn', vals[2])
            count = count+1

        mplt.plot(ndf['date'], ndf['adv'],'r', label='Adverbs')
        mplt.plot(ndf['date'], ndf['adj'], 'b', label='Adjectives')
        mplt.plot(ndf['date'], ndf['nn'], 'g', label = 'Proper Noun')
        mplt.xlabel('Date')
        mplt.ylabel('Frequency')
        mplt.title(titles[fcount])
        mplt.legend()
        mplt.show()

The comparative results which take into account both BJP and Congress in the same tweets, show high frequencies on dates 2018–12–07, 2019–01–04, 2019–01–22, 2019–02–12. The events of those days can be summarized as follows which likely to have caused higher number of sensitive tweets between party members.

2018–12–07 :

• Exit poll prediction results have been announced, where Congress is likely to win Rajasthan and Chhattisgarh.

2019–01–04 :

• Rafale deal was discussed in Parliament for long time.
• Announced thatPM will launch BJP poll campaign in North-East.

2019–01–22 :

• Amit Shah addresses the first rally on January 22 in Malda.

2019–02–12 :

• It was celebrated as hug day where Congress party’s chief Rahul Gandhi hugged Prime Minister Narendra Modi and many tweets got exchanged in twitter to spread the message of hugging.

Analysis of Precision and Recall

High precision in generic tweet classification as well as polarized tweet classification using POS, shows that in both cases the selected models does not have high false positives. It also implies the proportion of the data points the model predicts as respective moods as relevant, were actually relevant. The generic tweet classification also shows high recall, implying less false negatives and the ability to find all relevant instances in the classified dataset. Its high F-Score also shows its an accurate and robust classifier. On the other hand, polarized tweet classification though exhibits a high precision and accuracy, it has a low-recall signifying the model failed to identify all the relevant cases.

Classification and Prediction results with part-of-speech tagger

Improvements with Sequence Classifier

To capture dependencies between related classified moods, different joint classifier/sequence classifier models can be applied. Joint classifiers can improve the accuracy further as they aid in tweet classification by jointly selecting an appropriate label or part-of-speech tag for each word or collection of words in the tweet. Further, already tagged or words left to the current words contribute to feature history during classification. Models which do such sequence classification by scoring previous and current tags are Hidden Markov Models, Maximum Entropy Markov Models and Linear-Chain Conditional Random Field Models.

Conclusion

We see that of the NLTK based classifiers, Decision Tree performs the best followed by Naive Bayes and MaxEnt classifier. Of the classifiers from sklearn, plugged in with NLTK, SVM’s (linear and RBF kernel) and SGD’s performance exceeds that of MaxEnt. We also see that the accuracy of polarized tweet classification and prediction is high, but it has low precision (for Congress) and recall (both Congress and BJP). This is due to the imbalanced dataset and some sort of under-sampling or over-sampling techniques would have improved the accuracy of the minority class. Also building a word corpus using a specific theme based polarized tweets on the election context, it can further boost the accuracy.

Text can also be categorized using CountVector, Term Frequency (Tf), and Inverse Document Frequency (Idf), where each word frequency and word context in the tweet helps to define features for mood classification. In the next post, will discuss about other supervised ML algorithms like Boosting, Bagging and Ensembling that can use CountVector, Tf, Idf to classify and predict election sentiments.

References:

1. https://www.nltk.org/book/ch06.html
2. https://medium.com/analytics-vidhya/twitter-sentiment-analysis-for-the-2019-election-8f7d52af1887
3. https://medium.com/analytics-vidhya/sentiment-classification-for-2019-elections-using-text-based-classifiers-217f86b05124

Please let me know if there were any mistakes, suggestions feedbacks are welcome. Follow me at linkedin.

Disclaimer Statement

The work analyses tweet of 2 prominent parties for the upcoming election. The author has no intention to create controversy in people’s mind or hurt anybody’s feelings or incite feelings of anger or hatred. Its purely done for academic, research and information purposes and somebody else might get different results on application of other techniques of analysis. Its an unbiased and impartial summary and does not discriminate/differentiate any individual or group