NLP Series: What is Natural Language Processing?

Finally, we’ll conclude the chapter with an overview of the rest of the topics in the book. Figure 1-1 shows a preview of the organization of the chapters in terms of various NLP tasks and applications. Government agencies are bombarded with text-based data, including digital and paper documents. Word sense disambiguation (WSD) refers to identifying the correct meaning of a word based on the context it’s used in. Like sentiment analysis, NLP models use machine learning or rule-based approaches to improve their context identification. Natural language interaction is the seventh level of natural language processing.

For example, using this technology will allow you to extract the sentiment behind a text. A subfield of NLP called natural language understanding (NLU) has begun to rise in popularity because of its potential in cognitive and AI applications. NLU goes beyond the structural understanding of language to interpret intent, resolve context and word ambiguity, and even generate well-formed human language on its own.

Neural networks: what can artificial neural networks do?

NLP is an important component in a wide range of software applications that we use in our daily lives. In this section, we’ll introduce some key applications and also take a look at some common tasks that you’ll see across different NLP applications. This section reinforces the applications we showed you in Figure 1-1, which you’ll see in more detail throughout the book. An effective user interface broadens access to natural language processing tools, rather than requiring specialist skills to use them (e.g. programming expertise, command line access, scripting).

Similar to other early AI systems, early attempts at designing NLP systems were based on building rules for the task at hand. This required that the developers had some expertise in the domain https://www.metadialog.com/ to formulate rules that could be incorporated into a program. Such systems also required resources like dictionaries and thesauruses, typically compiled and digitized over a period of time.

Unlocking the potential of natural language processing: Opportunities and challenges

There are many different ways to analyze language for natural language processing. Some techniques include syntactical analyses like parsing and stemming or semantic analyses like sentiment analysis. Because of their complexity, generally it takes a lot of data to train a deep neural network, and processing it takes a lot of compute power and time. Modern deep neural network NLP models are trained from a diverse array of sources, examples of natural language processing such as all of Wikipedia and data scraped from the web. The training data might be on the order of 10 GB or more in size, and it might take a week or more on a high-performance cluster to train the deep neural network. (Researchers find that training even deeper models from even larger datasets have even higher performance, so currently there is a race to train bigger and bigger models from larger and larger datasets).