StringTokenizer Class (Legacy)

Before String.split existed (it arrived in Java 1.4), the way to break a line like "Java,is;fun" into words was StringTokenizer from java.util. You'll still meet it in older code and in fast-input templates used for competitive programming, so it's worth recognizing. For new code, though, split is clearer, and the Javadoc itself says StringTokenizer is kept only for compatibility.

Tokenizing a Line

import java.util.Arrays;
import java.util.StringTokenizer;
 
public class Tokens {
    public static void main(String[] args) {
        StringTokenizer st = new StringTokenizer("Java,is;fun", ",;");
        System.out.println(st.countTokens() + " tokens");
        while (st.hasMoreTokens()) {
            System.out.println(st.nextToken());
        }
 
        String line = "Navika,,Srihitha,";
        System.out.println(Arrays.toString(line.split(",")));
        StringTokenizer st2 = new StringTokenizer(line, ",");
        System.out.println(st2.countTokens() + " tokens from StringTokenizer");
    }
}

3 tokens
Java
is
fun
[Navika, , Srihitha]
2 tokens from StringTokenizer

How it works

1. The second argument, ",;", is a set of delimiter characters: any comma or semicolon splits.
2. hasMoreTokens() and nextToken() walk through the pieces one at a time, like an iterator.
3. Empty pieces are skipped. In "Navika,,Srihitha,", StringTokenizer sees only 2 tokens, while split keeps the empty piece between the two commas.

StringTokenizer Compared with split

Common Mistakes

Expecting empty fields from StringTokenizer. CSV-style data with blank fields loses them; use split(",", -1).

Passing a multi-character delimiter like ", ". It's treated as two separate delimiter characters, , and space, not as the pair.

Using it in new code. split is clearer and handles patterns.

Practice

1. Easy: Use StringTokenizer to count the words in "Advitha loves Java", then do the same with split("\\s+") after strip().
2. Harder: Parse "Pune:12;Delhi:30;Chennai:24" into city and temperature pairs, first with StringTokenizer and then with two split calls.

Key Takeaways