Tokenizing Strings in List of Strings – Python

Last Updated : 04 Feb, 2025

The task of tokenizing strings in a list of strings in Python involves splitting each string into smaller units, known as tokens, based on specific delimiters. For example, given the list a = [‘Geeks for Geeks’, ‘is’, ‘best computer science portal’], the goal is to break each string into individual words or tokens, resulting in a list of lists: [[‘Geeks’, ‘for’, ‘Geeks’], [‘is’], [‘best’, ‘computer’, ‘science’, ‘portal’]].

Using list comprehension

List comprehension is a concise way of creating lists. It allows for looping over an iterable and applying operations or expressions to generate new lists. When combined with split() this provides a very efficient way to tokenize strings.

Python

a = ['Geeks for Geeks', 'is', 'best computer science portal'] res = [sub.split() for sub in a] print(res)

Output

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: list comprehension iterates over each string in a, applying the split() method to split each string into a list of words.

Using resplit()

For more complex tokenization, where delimiters are not just spaces, re.split() from the re module can be used. It allows us to split strings based on regular expressions, making it suitable for handling multiple delimiters, such as spaces, punctuation, and other special characters.

Python

import re a = ['Geeks for Geeks', 'is', 'best computer science portal']  res = [re.split(r'\s+', sub) for sub in a] print(res)

Output

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: list comprehension splits each string in a into words using the re.split() with the pattern r’\s+’, which matches one or more whitespace characters.

Using map()

map() applies a given function to all items in an iterable . It’s a functional programming tool that is often used in Python to avoid explicit loops. By combining map() with split() , we can tokenize strings efficiently in a functional programming style.

Python

a = ['Geeks for Geeks', 'is', 'best computer science portal'] res = list(map(str.split, a)) print(res)

Output

[['Geeks', 'for', 'Geeks'], ['is'], ['best', 'computer', 'science', 'portal']]

Explanation: map() applies split() to each string in a, splitting them into words and then the result is converted back to a list.

Python | Splitting string list by strings

manjeet_04

Improve

Article Tags :

Practice Tags :

python

Similar Reads

List of strings in Python

A list of strings in Python stores multiple strings together. In this article, weâ€™ll explore how to create, modify and work with lists of strings using simple examples. Creating a List of StringsWe can use square brackets [] and separate each string with a comma to create a list of strings. [GFGTABS

2 min read

Python | Splitting string list by strings

Sometimes, while working with Python strings, we might have a problem in which we need to perform a split on a string. But we can have a more complex problem of having a front and rear string and need to perform a split on them. This can be multiple pairs for split. Let's discuss certain way to solv

3 min read

Python - Substring presence in Strings List

Given list of substrings and list of string, check for each substring, if they are present in any of strings in List. Input : test_list1 = ["Gfg", "is", "best"], test_list2 = ["I love Gfg", "Its Best for Geeks", "Gfg means CS"] Output : [True, False, False] Explanation : Only Gfg is present as subst

5 min read

Python | Convert List of String List to String List

Sometimes while working in Python, we can have problems of the interconversion of data. This article talks about the conversion of list of List Strings to joined string list. Let's discuss certain ways in which this task can be performed. Method #1 : Using map() + generator expression + join() + isd

6 min read

Python | Delimited String List to String Matrix

Sometimes, while working with Python strings, we can have problem in which we need to convert String list which have strings that are joined by deliminator to String Matrix by separation by deliminator. Lets discuss certain ways in which this task can be performed. Method #1 : Using loop + split() T

5 min read

Splitting String to List of Characters - Python

We are given a string, and our task is to split it into a list where each element is an individual character. For example, if the input string is "hello", the output should be ['h', 'e', 'l', 'l', 'o']. Let's discuss various ways to do this in Python. Using list()The simplest way to split a string i

2 min read

Splitting String to List of Characters - Python

The task of splitting a string into a list of characters in Python involves breaking down a string into its individual components, where each character becomes an element in a list. For example, given the string s = "GeeksforGeeks", the task is to split the string, resulting in a list like this: ['G

3 min read

Python | List of tuples to String

Many times we can have a problem in which we need to perform interconversion between strings and in those cases, we can have a problem in which we need to convert a tuple list to raw, comma separated string. Let's discuss certain ways in which this task can be performed. Method #1: Using str() + str

8 min read

Extract List of Substrings in List of Strings in Python

Working with strings is a fundamental aspect of programming, and Python provides a plethora of methods to manipulate and extract substrings efficiently. When dealing with a list of strings, extracting specific substrings can be a common requirement. In this article, we will explore five simple and c

3 min read

Python program to split a string by the given list of strings

Given a list of strings. The task is to split the string by the given list of strings. Input : test_str = 'geekforgeeksbestforgeeks', sub_list = ["best"] Output : ['geekforgeeks', 'best', 'forgeeks'] Explanation : "best" is extracted as different list element. Input : test_str = 'geekforgeeksbestfor

4 min read