Hate Speech and AI: Issues in Detection

Hate speech is a form of expression which attacks someone mostly based on their race, gender, ethnicity and sexual orientation. The history of hate speech dates back long time ago; however, with the expansion of the internet and social media, it had its most accelerated form. Now, 41% of the American population have experienced a form of online harassment as Pew Research Center’s report suggests. Also, the high correlation between suicide rates and verbal harrasment in migrant groups shows the crucial importance of detecting and preventing the spread of hate speech. Additonally as an instance from recent years, after the mass murder that happened in Pittsburg synagoge it has seen that the murderer was posting hated messages to jews constantly before the incident.



Retrieved from: https://www.kqed.org/news/11702239/why-its-so-hard-to-scrub-hate-speech-off-social-media


Furthermore, the Pew Research Center’s report also suggests that 79% of the American population thinks that the detection of hate speech/online harassment is in the responsibility of online service providers. Hence, many online service providers are aware of the importance of the issue and have close relationships with AI engineers while solving it.

When it comes to the logic of hate speech detection, there are many complex points. Firstly, such complexity comes from the current AI technologies’ limitations on understanding the contexts of human language. For instance, current technologies fail to detect hate speech or give false positives when there are contextual differences. As such, researchers from Carnegie Mellon University suggested that the toxicity of the speech may differ with the race, gender and ethnic characteristics of the people. Hence, to increase the quality of the data and detection; it is important to identify the characteristics of the author while identifying the hate speech and its toxicity rate according to the researchers. Also, such identification can also reduce the current bias the algorithms have.

Retrieved from: https://www.pewresearch.org/internet/2017/07/11/online-harassment-2017/pi_2017-07-11_online-harassment_0-01/


However, current AI technologies have difficulties in detecting such characteristics. Firstly, it’s difficult to identify the demographics and characteristics of the authors’; since in most of the cases such information is not available on the internet. So, the process of distinguishing hate speech becomes harder. Secondly, even if the author clearly indicates such information; sometimes the detection process becomes more difficult due to the cultural insights of the given context. The dynamics of the countries or even the regions in countries is changeable and is really related to their culture and language. Such differences and ongoing changing factors are also crucial points for the outcomes of the processes; some outcomes may fail to detect or detect false positives due to non-statistical cultural differences.



Language is one of the most complicated and most significant functions of the humankind. There are many different ways and contexts of communicating with language which even neuroscientists could not fully map yet. However, with artificial intelligence scientists are also one step forward in describing the patterns and mechanisms of language. In such sense, the crucially important subject in the age of the internet, hate speech detection, also has an advantage since it is much easier to detect online harassment with machine learning algorithms. Nevertheless, there is no way for humans to get out of the detection cycle in today’s technology with the issues faced in detection processes. 







WordCloud and Sentiment Analysis with Python

One of the most popular concepts of our day is the word cloud and the work done on it. People use the nltk library to experiment with the word cloud. The aim here is to process the processes before the natural language processing phases. Since the Python programming language reaches a wider audience every day, the variety of projects made with nltk is increasing. Beginners often analyze tweets posted on Twitter on any topic and make visualizations, analyzes and inferences from it. While creating a word cloud, one of the key points is the number of repetitions. The more the word repeats, the more prominent it becomes in the word cloud. I tried to explain your research with the image below. Here the part indicated with q is the part of the word or phrase you are looking for. To do this, you must first have a Twitter Developers account. In addition, you will be providing the link here as we will pull it from here. Through the link, you can pull tweets to your local workplace and take action on them.

As can be seen in the photo above, the most critical part of the job is to set up the def structure well and decide on which variables to label. You can save the token and key values ​​somewhere and withdraw from there. Here, I did not show the key values ​​due to privacy, but once you have a Twitter Developer account, you can access these values ​​yourself and make the necessary assignments. If you want to access specific values ​​while analyzing on Twitter, you can make custom searches. I will put the necessary documentation in the Resources section, you can review it according to your request and integrate it into your own code and analysis. If I need to give extra information, there are several methods in nltk we will use. One of the ones I use individually is “stopwords” and the other is “wordnet”. You can change the English option according to the language you want to work on. It is a comprehensive English language in terms of word strength and effectiveness. If you are working on your own language and have a great collection of words, you can specify it in the comments tab. Thus, we can keep the interaction high and increase the yield rate. You can observe the part I explained here in the image below.

I chose the word ‘samsung’ for the Word Cloud study. By entering the abbreviation of the language option in the Lang section, you can pull the hashtag data you have chosen to your work area. At first we add the necessary libraries, and then we can change the background color we will work with according to your personal wishes. In addition, if you set the plt.axis value to “on”, you can observe the frequency of repeating words. Since I did this with the highlighting method myself, I found it unnecessary to show the axes as extra. What I do here is to set up a basic wordcloud structure and to gain something even at your entry level. If you have a career planning for natural language processing, you can see it as a start and continue to improve by following the necessary notes. The word cloud structure is the bottom tab of these jobs. While I teach you as a junior, I also continue to work in this field myself. The natural language processing career is an area where you need to stay up-to-date on what others are doing, how they code, and what projects are done on platforms such as large-scale resource browsing and github.

I’ll show you an entry-level sentiment analysis, which I will mention in the last part. Before starting what I will explain here, when writing code in Python, you need to set up the def patterns well and feed your code into it here. Accelerating functional processes and writing an understandable, scalable code for people who will come after you while working in the future will benefit everyone. By writing clean code with def structures, you can easily transfer what you do there to the person who comes for you. Going back to the sentiment analysis, here we can already do this scoring work via the textblob library. TextBlob classifies it as a result of the content of tweets sent on Twitter and the positive – negative words of this content. After this classification, it gives you a ready-made column for analysis. You can analyze this according to your wishes and try different studies. For example, you can chart it, observe the number of repetitive words and take the differences with these values ​​and integrate the background into a picture you have edited yourself.