Java StringTokenizer Class
Last Updated : 10 Jan, 2025
StringTokenizer class in Java is used to break a string into tokens based on delimiters. A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.
- A token is returned by taking a substring of the string that was used to create the StringTokenizer object.
- It provides the first step in the parsing process often called lexer or scanner.
- It implements the
Enumeration interface. - To perform Java String Tokenization, we need to specify an input string and a set of delimiters.
- A delimiter is a character or set of characters that separate tokens in the string.
Note: StringTokenizer is a legacy class, and the split() method is preferred for modern applications.
Example: Below is a simple example that explains the use of Java StringTokenizer to split a space-separated string into tokens:
Java // Demonstration of Java StringTokenizer import java.util.StringTokenizer; public class Geeks { public static void main(String[] args) { // Input string String s = "Hello Geeks how are you"; // Create a StringTokenizer object // with space as the delimiter StringTokenizer st = new StringTokenizer(s, " "); // Tokenize the string and print each token while (st.hasMoreTokens()) { System.out.println(st.nextToken()); } } } OutputHello Geeks how are you
Explanation: In the above example, we have created a StringTokenizer object by passing the string and a space ” ” as the delimiter. The hasMoreTokens() method checks there are more tokens available to process or not. The nextToken() method get the next token (substring).
Below is the representation of the process, which we defined in the above example:
Constructors of StringTokenizer Class
The StringTokenizer class provides three constructors to tokenize strings in different ways.
Constructors | Description |
|---|
StringTokenizer(String str) | Creates a tokenizer for the specified string. Uses default delimiters (whitespace, tabs, etc.). |
|---|
StringTokenizer(String str, String delim) | Creates a tokenizer for the specified string using the given delimiters. |
|---|
StringTokenizer(String str, String delim, boolean returnDelims) | Creates a tokenizer for the specified string using the given delimiters and specifies whether the delimiters should be returned as tokens. |
|---|
Note:
- Default Delimiters: When no delimiter is specified, whitespace is used.
- returnDelims: If true, the delimiters are treated as tokens themselves.
Below is a concise explanation of how each constructor works, along with a code example in the combined way.
Cases of StringTokenizer Constructors
1. If the returnDelims is false, delimiter characters serve to separate tokens.
Example:
Input: if string –> “hello geeks” and Delimiter is ” “, then
Output: tokens are “hello” and “geeks”.
2. If the returnDelims is true, delimiter characters are considered to be tokens.
Example:
Input: String –> is “hello geeks” and Delimiter is ” “, then
Output: Tokens –> “hello”, ” ” and “geeks”.
3. Multiple delimiters can be chosen for a single string.
Example:
Syntax: StringTokenizer st1 = new StringTokenizer( “2+3-1*8/4”, “+*-/”);
Input: String –> is “2+3-1*8/4” and Delimiters are +,*,-,/
Output: Tokens –> “2”,”3″,”1″,”8″,”4″.
Example:
Java // Demonstration of String Tokenizer Constructors import java.util.*; class Geeks { public static void main(String[] args) { // Example with Constructor 1 System.out.println("Using StringTokenizer Constructor 1: "); // Using StringTokenizer to split the string into // tokens using space (" ") as the delimiter StringTokenizer st1 = new StringTokenizer( "Geeks fo Geeks", " "); // Iterate through tokens while // there are more tokens available while (st1.hasMoreTokens()) // Getting and printing the next token System.out.println(st1.nextToken()); // Example with Constructor 2 System.out.println("Using StringTokenizer Constructor 2: "); // Using StringTokenizer to split the string // using ":" as the delimiter StringTokenizer st2 = new StringTokenizer( "java : Code : String : Tokenizer", " :"); // Iterate through tokens and print them while (st2.hasMoreTokens()) System.out.println(st2.nextToken()); // Example with Constructor 3 System.out.println("Using StringTokenizer Constructor 3: "); // Using StringTokenizer with returnDelims = true // to include delimiters as tokens StringTokenizer st3 = new StringTokenizer( "java : Code", " :", true); // Iterate through tokens (including delimiters) // and print them while (st3.hasMoreTokens()) System.out.println(st3.nextToken()); } } OutputUsing StringTokenizer Constructor 1: Geeks fo Geeks Using StringTokenizer Constructor 2: java Code String Tokenizer Using StringTokenizer Constructor 3: java : Code
Methods Of StringTokenizer Class
Below are some commonly used methods of StringTokenizer class along with a combined code example demonstrating some of these methods.
Example:
Java // Demonstration of StringTokenizer Methods import java.util.*; class Geeks { public static void main(String[] args) { // Creating a StringTokenizer StringTokenizer st = new StringTokenizer( "Welcome to GeeksforGeeks"); StringTokenizer st1 = new StringTokenizer(""); // countTokens Method int c = st.countTokens(); System.out.println(c); // hasMoreTokens Methods System.out.println("Welcome to GeeksforGeeks: "+ st.hasMoreTokens()); System.out.println("(Empty String) : "+ st1.hasMoreTokens()); // nextElement() Method System.out.println("\nTraversing the String:"); while(st.hasMoreTokens()){ System.out.println(st.nextElement()); } } } Output3 Welcome to GeeksforGeeks: true (Empty String) : false Traversing the String: Welcome to GeeksforGeeks
Similar Reads
Java StringBuilder Class
In Java, the StringBuilder class is a part of the java.lang package that provides a mutable sequence of characters. Unlike String (which is immutable), StringBuilder allows in-place modifications, making it memory-efficient and faster for frequent string operations. Declaration: StringBuilder sb = n
7 min read
String Class in Java
A string is a sequence of characters. In Java, objects of the String class are immutable, which means they cannot be changed once created. In this article, we are going to learn about the String class in Java. Example of String Class in Java: [GFGTABS] Java // Java Program to Create a String import
7 min read
Java Reader Class
Reader class in Java is an abstract class used for reading character streams. It serves as the base class for various subclasses like FileReader, BufferedReader, CharArrayReader, and others, which provide more efficient implementations of the read() method. To work with the Reader class, we must ext
7 min read
Java Pattern Class
The Pattern class in Java is used for defining regular expressions (regex) to perform pattern matching on strings. It is part of the java.util.regex package and it plays a key role in searching, replacing, and manipulating strings based on patterns. The Matcher class works together with Pattern to p
3 min read
Java.io.StreamTokenizer Class in Java | Set 2
StringTokenizer Class in Java | Set 1 Methods: parseNumbers() : java.io.StreamTokenizer.parseNumbers() specifies that the number in StreamTokenizer is parsed, so that each character - " 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 " has a numeric attribute. When the parser encounters a word token that has the forma
8 min read
Java.io.StreamTokenizer Class in Java | Set 1
Java.io.StreamTokenizer class parses input stream into "tokens".It allows to read one token at a time. Stream Tokenizer can recognize numbers, quoted strings, and various comment styles. Declaration : public class StreamTokenizer extends Object Constructor : StreamTokenizer(Reader arg) : Creates a t
9 min read
Java.io.StringReader class in Java
StringReader class in Java is a character stream class whose source is a string. It inherits Reader Class. Closing the StringReader is not necessary, it is because system resources like network sockets and files are not used. Let us check more points about StringReader Class in Java. Declare StringR
4 min read
Java PushbackReader Class
The PushbackReader class in Java is part of the java.io.package, and is used for reading characters from a stream. This class lets us push characters back into the stream. Features of PushbackReader Class: This class uses a buffer that allows us to push characters back into the stream.This class is
6 min read
StringTokenizer hasMoreTokens() Method in Java with Examples
The hasMoreTokens() method of StringTokenizer class checks whether there are any more tokens available with this StringTokenizer. Syntax: public boolean hasMoreTokens() Parameters: The method does not take any parameters. Return Value: The method returns boolean True if the availability of at least
2 min read
Matcher Class in Java
In Java, Matcher is a class that is implemented by the MatchResult interface, that performs match operations on a character sequence by interpreting a Pattern. Below, we can see the declaration of java.util.regex.Matcher in java.lang.Object Class: public final class Matcher extends Object implements
4 min read