Back to DFS's Workshop Page
Back to Agenda Page

Problem: Word Frequencies

This problem will introduce you to text processing.

Consider these questions:

1. How many times does a particular word, e.g., rabbit, appear in Alice's Adventures in Wonderland?
2. What word appears most frequently?
3. What are the ten most frequently used words?
4. How many words are used only once?
5. Can we tell anything about the subject matter of the text by investigating words with a certain range of frequencies?

As an introduction, we will take a sentence from the old days of typing classes and determine the order of first occurrence of each of the words. Here is the code for use from the Command Line.

$line = "Now is the time for all good men to come to the aid of their party.";
$lineofwords = explode(" ", $line);
$ooo = array();
foreach($lineofwords as $word )
{
   if( !in_array($word, $ooo) )
      $ooo[] = $word;
}
echo " # Word\n";
foreach($ooo as $n => $w)
{
   printf("%2d %s\n", $n + 1, $w);
}

The subscripts, which automatically start at 0, will indicate the order in which they are encountered in the string. The output generated will be the following.

 # Word
 1 Now
 2 is
 3 the
 4 time
 5 for
 6 all
 7 good
 8 men
 9 to
10 come
11 aid
12 of
13 their
14 party.

To turn this into a useful program, we would like to get information about a file such as

• How many words are in the text
• How many different words are in the text
• What the order of first occurrence of the words is
• What the frequency of the words is, in alphabetical order
• What the frequency of the words is, in numerical order

You are now ready to attempt using the computer to help you actually analyze text files.

The first four of the original questions can easily be answered if we sort the data, i.e., put it in alphabetical or numerical order. The last question requires analysis and would probably entail comparing information gleaned from using your program on a number of different texts.

This then is your task: Give the user the opportunity to selectively view organized data about a text file.

1. Finding the File
   • Use a recursive script to navigate through directories on the hard drive to the file desired
     See the Navigating Directories Problem
   • Allow the user to paste a URL into a text box
     The URL passed can be accessed by fopen()
2. Specifying the Data Display to Be Done
   • Listing by Order of First Occurrence
   • Alphabetical Listing of Frequencies
   • Numerical Listing of Frequencies

To see how this could work, view this possible solution.

Notes

• Use fgetss() to input lines of text, stripping any and all HTML tags.
• Be sure to trim() extraneous characters off the words.
• Since you are using one-dimensional arrays, you can make use of the built-in sort functions.