What is sentiment analysis?
Sentiment analysis is a procedure for classifying whether a block of text or speech is to be evaluated as positive, negative or neutral. But does it only focus on polarity? Not at all. With the help of algorithms, it facilitates insights into expressed emotions such as joy and anger.
In software development, sentiment analysis is one of the three central pillars of deep data analysis, along with topic modelling and named entity recognition. Unfortunately, there is no generally accepted definition.
Since the late 1970s, linguists and social scientists have been working on the analysis of tonalities in texts. More recently, the increasing development of neural networks has led to completely new fields of application and approaches to sentiment analysis.
Also known as “opinion mining” or “emotion AI”, sentiment analysis can be used, among other things, for a better understanding of customer needs, analyses of reviews and decision-making processes, as well as for referral and recommendation programmes.
“With the help of AI sentiment analysis we want to help customers better understand the emotions expressed in conversations.”
Tucan.ai is also using sentiment analysis to improve the conversation analysis capabilities of its software toolkit. “Currently, we are still in the standardised training phase. Soon, however, our users will be able to change the sentiments read from a paragraph or text part themselves. This will allow us to train the AI specifically with customer-specific data so that it can make increasingly precise predictions,” says Mohammed Aymen Ben Slimen who has been supporting the machine learning team of Tucan.ai since the beginning of this year.
“With the help of AI sentiment analysis we want to help customers better understand the emotions expressed in conversations. For example, if a speaker agrees with a topic, we show a happy emoji, and if someone complains, this is marked by a sad emoji,” explains Aymen.
How does sentiment analysis work?
Sentiment analysis is a subfield of natural language processing, i.e. the algorithmic processing and evaluation of natural language in the form of text or speech data. Corresponding AI systems try to automatically capture the opinion of the sender. In essence, three central aspects in the process have been evaluated:
- Content: What central topics are speakers talking about?
- Polarity: Which positive and negative opinions are expressed by whom?
- Opinion leadership: Who represents which opinion on which topic?
At its core, sentiment analysis is a straightforward process: a text is first broken down into components such as sentences, phrases or small word parts. Then the individual emotional components are identified. These are then rated, for example with “-1” and “+1”. In a multi-layer sentiment analysis, these ratings are also combined.
Types of sentiment analysis
The most common type of sentiment analysis focuses on opposites or polarity, but others can also identify specific feelings and emotions. The most common forms are:
- Standard sentiment analysis
- Emotion recognition
- Fine-grained sentiment analysis
- Multilingual sentiment analysis
- Aspect-based sentiment analysis (ABSA)
- Intention detection
Standard sentiment analysis
Standard sentiment analysis identifies the degree of an opinion and classifies it as positive, negative or neutral. Here is an example:
- Positive: “I really love this new theatre.”
- Neutral: “I’m unsure. I liked the play, but I didn’t enjoy the ambience.”
- Negative: “The play was terrible and so was the overall vibe and service.”
Emotion recognition makes it possible to recognise the underlying emotions of a text, such as joy, anger and frustration. In machine learning using lexicons, this type is particularly challenging. Using a lexicon can define emotions of clients, but people often express emotions in very different ways. For example, words that have a negative connotation can also have a positive meaning (e.g. sick).
Multilingual sentiment analysis
Multilingual sentiment analysis is very challenging and comparatively tricky. It involves the classification and processing of multiple languages. Alternatively, language classifiers can be used to train and adapt sentiment analysis to needs, e.g. preferred language.
Fine-grained sentiment analysis
This method, also called graded sentiment analysis, allows you to add a few more categories in addition to positive, negative or neutral ones. It is very similar to 5-star ratings and is divided into five segments: very positive, positive, neutral, negative and very negative.
As the name suggests, intent detection analyses a text to understand the intent behind a particular opinion. It recognises valuable customer opinions for solving a problem or improving a product or service. Intention recognition can also predict whether a customer intends to use a product by observing and creating a pattern, which is useful for advertising and marketing.
Aspect-based sentiment analysis
Aspect-based sentiment analysis focuses on identifying features or aspects of an entity or opinion, such as product reviews. For example, reviews are often composed of multiple opinions about product features such as user interface, price, mobile versions, integrations, to name a few. In other words, it is a more granular approach to analysing reviews.
Sentiment analysis techniques
Over time, a variety of techniques and technologies have evolved around sentiment analysis. In particular, recent advances in machine learning and neural networks have led to rapid improvements in NLP in general and sentiment analysis in particular.
In the case of Tucan.ai, the first step is a deep-learning based model that can predict the sentiment of a particular paragraph or part of a text (positive, negative or neutral). “After we create the transcript,” says Aymen, “we pass it to an NLP sentiment analysis model that predicts sentiments and displays emojis related to the result.”
Rule-based sentiment analysis
In this, simpler approach to text analysis, no training or ML models are used. Here, the software classifies parts of text based on sophisticated linguistic rules. These rules are also called lexicons. Therefore, the rule-based approach is also called lexicon-based approach. Widely used rule-based approaches are TextBlob, VADER and SentiWordNet.
In contrast to rule-based systems, no rules are given to the machine learning algorithm, but are learned by the system itself. This requires a training set of data where the input (sentence, paragraph, text) is assigned a tag (negative, positive, neutral). In order for the algorithm to work, the text is converted into a numerical re-presentation. This representation is usually a vector that holds information about the text, in a bag of words approach e.g. the frequency of a term. Recently, “word embeddings” – where semantic information can be stored in vectors to words – have become very popular.
Deep learning allows data to be processed in a much more complex way. An LSTM (Long Short-Term Memory) model is a type of recurrent neural network (RNN) used to process temporal data. Since we assume that the order of features (words) in a sentence is important, we use this neural network architecture. Deep Learning is computationally intensive and is not suitable for high-dimensional sparse vectors (poor performance and slow convergence).
When we extract features from the original text for model training, we need to represent them as dense vectors. In such a technique, each text is converted into a sequence of numbers, where each number is assigned to a word in the vocabulary. Taking this further, we need to assign words that have similar usage/meaning to similar real number vectors (rather than an index) by accessing the embeddings already mentioned above. Without them, the model would misinterpret the index number of words as meaning. With “word embeddings”, all words are embedded in a multi-dimensional vector space so that their similarities can be measured by distance.
Application of sentiment analysis
About 80 per cent of all the data that can be collected in a sentiment analysis – whether by human or computer – is unstructured and eludes classical approaches to analysis. The targeted use of this data can mean an immense competitive advantage for companies or organisations. However, the volume of data usually exceeds the possibilities of human analysis. The ability to process huge amounts of data in a short time is a powerful argument for using automated systems for sentiment analysis.
Using machine learning, sentiment analysis transforms all the unstructured data collected from transcripts, chatbots, social media, surveys, etc. into meaningful information. It is a powerful AI resource that can be used to improve decision-making in a sustainable way, as demonstrated by the following exemplary use cases:
Increase meeting efficiency
Each meeting topic can be analysed through sentiment analysis based on what each participant says to document how they feel about it emotionally. This makes it easier to avoid misunderstandings, duplications, repetitions and ambiguities.
Coaching – Sentiment analysis in sales talks
Sentiment analysis also helps in the documentation and evaluation of sales calls as well as in the coaching of call agents and consultants. In the operational area, it is often used as a performance measurement tool to evaluate the empathy or emotional intelligence of sales staff through interactions with customers. Mood analysis also plays an important role in coaching salespeople to improve their conversational skills.
Lead scoring in sentiment analysis
Sentiment analysis can also measure the interest of potential customers through lead scoring. Scores are presented on a scale of -1 and 1, with the lower end indicating negative responses and the upper end indicating positive responses.
Customer feedback and online conversations are important aspects of brand monitoring. Apart from social media, conversations can also take place in news, on websites, on review blogs or in forums. Listening to customers’ voices using text analytics and sentiment extraction can help to better understand their attitudes towards a product or service.
Social media monitoring
Did you know that around 500,000 tweets go online every minute and these can contain valuable insights about brands, products and services? Twitter sentiment analysis allows companies to extract the emotions underlying conversations on social media. It helps to understand how people talk about a topic and why.
Monitoring brands and social media gives us great insights into customer sentiment. But did you know that sentiment analysis in phone calls can be used to target marketing, sales and customer success? This is useful and common in call centres or customer care teams, for example. Because as we all know: Good customer service usually means higher customer retention.
Market research through Twitter sentiment analysis
By analysing the sentiment of tweets related to the product, you can systematically find out what people think about your product. Do they find it useful? Does it actually fulfil its purpose? Perhaps they are dissatisfied with the price or wish for a new function.
Such insights can help companies transform their business and take different strategic actions to, for example:
- Attract new customers;
- become more competitive;
- reduce customer service;
- make the brand more profitable;
- sell more services;
- retain existing clients;
- and improve marketing campaigns.