Tokenization Using Spacy

Last Updated : 12 Apr, 2025

Before we get into tokenization, let's first take a look at what spaCy is. spaCy is a popular library used in Natural Language Processing (NLP). It's an object-oriented library that helps with processing and analyzing text. We can use spaCy to clean and prepare text, break it into sentences and words and even extract useful information from the text using its various tools and functions. This makes spaCy a great tool for tasks like tokenization, part-of-speech tagging and named entity recognition.

What is Tokenization?

Tokenization is the process of splitting a text or a sentence into segments, which are called tokens. These tokens can be individual words, phrases, or characters depending on the tokenization method used. It is the first step of text preprocessing and is used as input for subsequent processes like text classification, lemmatization and part-of-speech tagging. This step is essential for converting unstructured text into a structured format that can be processed further for tasks such as sentiment analysis, named entity recognition and translation.

Example of Tokenization

This is the sentence: "I love natural language processing!"

After tokenization: ["I", "love", "natural", "language", "processing", "!"]

Each token here represents a word or punctuation mark, making it easier for algorithms to process and analyze the text.

Implementation of Tokenization using Spacy Library

Python

import spacy  # Creating blank language object then # tokenizing words of the sentence nlp = spacy.blank("en")  doc = nlp("GeeksforGeeks is a one stop\ learning destination for geeks.")  for token in doc:     print(token)

Output:

GeeksforGeeks
is
a
one
stop
learning
destination
for
geeks
.

We can also add functionality in tokens by adding other modules in the pipeline using spacy.load().

Python

nlp = spacy.load("en_core_web_sm") nlp.pipe_names

Output:

['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']

Here is an example to show what other functionalities can be enhanced by adding modules to the pipeline.

Python

import spacy  # loading modules to the pipeline. nlp = spacy.load("en_core_web_sm")  # Initialising doc with a sentence. doc = nlp("If you want to be an excellent programmer \ , be consistent to practice daily on GFG.")  # Using properties of token i.e. Part of Speech and Lemmatization for token in doc:     print(token, " | ",           spacy.explain(token.pos_),           " | ", token.lemma_)

Output:

If  |  subordinating conjunction  |  if
you  |  pronoun  |  you
want  |  verb  |  want
to  |  particle  |  to
be  |  auxiliary  |  be
an  |  determiner  |  an
excellent  |  adjective  |  excellent
programmer  |  noun  |  programmer
,  |  punctuation  |  ,
be  |  auxiliary  |  be
consistent  |  adjective  |  consistent
to  |  particle  |  to
practice  |  verb  |  practice
daily  |  adverb  |  daily
on  |  adposition  |  on
GFG  |  proper noun  |  GFG
.  |  punctuation  |  .

In the example above, we utilized part-of-speech (POS) tagging and lemmatization through the spaCy NLP modules. This allowed us to obtain the POS for each word and convert each token to its base form through lemmatization. Prior to loading the NLP model with "en_core_web_sm", we would not have had access to this functionality. The en_core_web_sm model is essential as it provides the necessary linguistic features, such as tokenization, POS tagging and lemmatization, enabling these advanced NLP capabilities.

Read More:
Sentiment Analysis using VADER
Text Generation using Recurrent Long Short Term Memory Network
Text Preprocessing in Python

Tokenization Using Spacy

apsingh123

Improve

Article Tags :

Tokenization Using Spacy

What is Tokenization?

Example of Tokenization

Implementation of Tokenization using Spacy Library

Similar Reads