When we look at linguistics and the evolution of human language, it is apparent that even though we have grammatical rules to dictate how a language should be read or written, it is by no means static.
Language evolves due to many reasons. The most typical reason for this is that we are constantly finding and establishing better and more concise vocabulary to further reflect our culture and society. In addition, as languages evolve worldwide, current technology is advancing exponentially, especially within the last few decades or so.
As language evolves, it is crucial to adjust technology to new vocabulary to deliver catered results to the individual. In this article, we will discover how and if Artificial Intelligence can comprehend slang vocabulary and how Natural Language Processing (NLP) works overall.
For more information surrounding NLP, check out these AITJ articles: Conversational AI on Social Media Marketing: Must Need to Know and Conversational AI In Healthcare.
History of NLP
Natural Language Processing, otherwise known as NLP, is considered the “automatic manipulation of natural language, like speech and text, by software.” In a basic sense, this is how technology like AI can comprehend human conversations and language inputs to understand data sets beyond numerical values.
NLP, at a glance, seems to be a relatively new concept, connecting computers and humans like never before. However, this technology is not new whatsoever and surprisingly dates back to the 1950s. In 1954, Georgetown University and IBM collaborated on a scientific experiment known as the George-IBM Experiment. This test was revolutionary at the time, and machine translation was utilized to translate 60 Russian sentences into English successfully. This became a very momentous event unlike anything before and eventually encouraged various governments to invest in computer linguistics as a result.
In the 1980s, NLP became more successful and evolved to the point that researchers began using Machine Learning algorithms to assist in these translations and comprehensions. In the early 2010s, this approach shifted from machine learning to using deep neural network technology to drive this language processing.
How Slang Words Originate and Evolve
Slang vocabulary is a very unique part of linguistics. Researchers suggest that many slang words we see today originate from various sources, a big culprit being media and the internet. Often, informal words and phrases, such as “LOL,” spread around after being used on social media, to the point where dictionaries officially recognize them as a part of our defined lexicon.
Julie Coleman, the author of The Life of Slang, once suggested that it’s not as simple as language changing on its own but rather that “technologies have developed and they allow the transmission of slang terms to pass from one group to another much more quickly.” With these developed technologies rapidly expanding and becoming more available to the public to create their own slang, this raises an interesting question in the world of Artificial Intelligence: How can NLP recognize and keep up with the rapid expansion of informal language that may not be included in standardized dictionaries or follow grammatical expectations?
How AI Can Utilize and Comprehend Slang Vocabulary
The internet, a place where many slang words originate, is a widely vast environment. For researchers and data analysts to recognize the informal language on the internet, they need to utilize a massive amount of data points, consisting of arguably millions of social media posts and billions of individual words. Once this data set is used in Machine Learning, the machine separates a complete sentence into individual words. This process is called tokenization.
Once tokenization is completed, two processes called stemming and lemmatization occur, where the technology then reduces the separated words into their basic root form (such as the English word “bringing” or “brought” to “bring”). Once this occurs, the researchers use NLP to train software to recognize word pairings to obtain a better overall sense of the language, (“follow” and “unfollow” are examples of word pairings).
Then utilizing deep learning, neural networks are created to undergo sentiment analysis on texts. This is where the connection between slang and standardized grammatical words becomes intertwined, as the machine can eventually pair words together and recognize correlations.
In the grand scheme of things, we need an extensive data set to draw conclusions from for technology to understand slang vocabulary. But again, as language evolves rapidly with the internet and other forms of media, this can become challenging for AI to keep up with. In a sense, AI may not be able to understand 100 percent of a given language due to rapid changes. Although some efforts have been deemed successful in language processing, it still might not be as easy for AI at this moment to completely recognize complex structures as our human languages. However, technology has impressed us before, so who is to say this will not become possible in the future? It very well might.
