Sentiment Benchmarks

econob’s text analytics technology outshines those of Microsoft, Google und IBM

In 2017, we dealt intensively with the competition and compared the performance of “LingRep” with that of other providers in a specially developed benchmarking tool. For this purpose, the interfaces of the leading providers Google Cloud, Microsoft Azure and IBM Alchemy were integrated and measurements were carried out with regard to speed and quality.

Our procedure is as transparent and neutral as possible. In order to arrive at fair comparative values, three officially available test sets were used – none of the texts we prepared. These test sets provide manual, human-rated texts that are compared with automated reviews from Google, Microsoft, IBM and econob in our benchmarking tool.

Extraordinary promising results

With the following benchmarking results, we can claim to be competitive in international comparison. With the analysis of German texts, we clearly deliver the best results in terms of evaluation performance and precision. In this case, only IBM can be used as a reference since the analysis services of Google and Microsoft do not support the German language. Our language-independent “LingRep” technology is able to process even more complex speech patterns.

The following three graphs illustrate the relationship between valuation quality and performance. The diagrams speak for themselves:

 

Figure 1: German Tweets

German Tweets
Test set created by: University of Zurich
Outcome
econob
IBM
Speed
98%
2%
Quality
88%
65%
conob outperforms IBM especially regarding computational speed and also shows significantly better results in terms of quality.

Figure 2: German Product Reviews

German Product Reviews
Test set created by: University of Pennsylvania
Ergebnis
Outcome
IBM
Speed
83%
17%
Quality
88%
55%
The precision of LingRep is significantly higher, especially in the German language. With complex language structures such as those of the German language, IBM is unable to keep pace with speed or quality.

Figure 3: English Product Reviews

English Product Reviews
Test set created by: University of Pennsylvania
Outcome
econob
Google
IBM
Microsoft
Speed
97%
3%
57%
54%
Quality
82%
80%
57%
73%
Even in the English product reviews, econob achieved the best overall result with 82%. Google delivers similarly good results in terms of quality, but this is much slower.
Manually or with conventional working methods, companies are no longer able to meet digital requirements and evaluate big data. It is therefore necessary to use automated systems that evaluate and analyse unstructured information such as texts in real time. This is exactly the core competence of econob GmbH.

Automated recognition – the “mood” underlying a text – is called sentiment analysis. The tonality of a text can be determined; whether it can be categorized as positive, negative or neutral. The text is assigned a number in the range of -1 (extremely negative) to +1 (extremely positive) according to an international, standardized scale.

An example with a very positive result value of 0.75: “econob’s text analysis system performs best in the analysis of German-language texts”.