Example of StreamTokenizer in Java

This is a simple example which you can use to find out the number of words, numbers in a file using StreamTokenizer.

Open CMD and type javap java.io.StreamTokenizer to know about the StreamTokenizer class.

StreamTokenizer defines four int constants: TT_EOF, TT_EOL, TT_NUMBER, and TT_WORD. There are three instance variables. nval is a public double used to hold the values of numbers as they are recognized. sval is a public String used to hold the value of any words as they are recognized. ttype is a public int indicating the type of token that has just been read by the nextToken( ) method. If the token is a word, ttype equals TT_WORD. If the token is a number, ttype equals TT_NUMBER. If the token is a single character, ttype contains its value. If an end-of-line condition has been encountered, ttype equals TT_EOL. (This assumes that eolIsSignificant( ) was invoked with a true argument.) If the end of the stream has been encountered, ttype equals TT_EOF.

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;


class STN {
    

    public static void main(String[] args) {
                
        StreamTokenizer sTokenizer = null;
        int wordCount = 0, numberCount = 0;
        
        try {
            
            sTokenizer = new StreamTokenizer(new FileReader(args[0]));
            
            while (sTokenizer.nextToken() != StreamTokenizer.TT_EOF) {
                
                if (sTokenizer.ttype == StreamTokenizer.TT_WORD)
                    wordCount++;
                else if (sTokenizer.ttype == StreamTokenizer.TT_NUMBER)
                    numberCount++;
            }
            
            System.out.println("Number of words in file: " + wordCount);
            System.out.println("Number of numbers in file: " + numberCount);
            
        } catch (FileNotFoundException ex) {
            ex.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }
}