CS 539 Spring 2007

Computer Science Department

CS539 Machine Learning - Spring 2007
Project 8 - Rule Learning

PROF. CAROLINA RUIZ

Due Date: Thursday, April 12th 2007. Slides are due at 3:00 pm and the written report is due at 4:00 pm.

Project Description
Project Assignment
Report Submission and Due Date

PROJECT DESCRIPTION

Use sequential covering algorithms to construct sets of classification rules for each of the following problems:

Predicting the class attribute in the census-income dataset.
The sample problems that come with the FOIL release 6 system:
- ackermann.d
- crx.d
- member.d
- ncm.d
- qs44.d
- sort.d

PROJECT ASSIGNMENT

Read Chapter 10 of the textbook on Rule Learning and the FOIL paper distributed in class New Generation Computing '95. in great detail.
Solve Exercise 10.2 of your textbook (page 303). Include your solution in your written report (and not in your oral report).
The following are guidelines for the construction of your set of rules:
- Code: Use both of the following two implementations of sequential-covering methods (or write your own implementations of those methods if you prefer):
  - The Prism algorithm implemented in the Weka system.
    Read the code of the Prism classifier in Weka in great detail. You can find a description of this algorithm in "Data Mining: Practical Machine Learning Tools and Techniques (Second Edition)". I.H. Witten and E. Frank. Morgan Kaufmann Publishers. 2005.
  - The FOIL system (release 6) developed by Quinlan. This and other versions of FOIL are available online at: Quinlan's Webpage. Quinlan's webpage contains several copies of his papers describing the techniques employed in the system (the New Generation Computing '95 was the paper distributed in class).
- Objectives of the Learning Experiments: Before you start running experiments, look at the raw data in detail. Figure out 3 to 5 specific, interesting questions about the domain that you want to answer with your Rule Learning experiments. Try to choose questions that are about the domain and not about the Rule Learning or the experimental parameters, but that would particularly benefit from using rules to answer them. These questions may be phrased as conjectures that you want to confirm/refute with your experimental results.
- Training and Testing Instances:
  1. For the Census Data, select a stratified subset of instances of a size that both Prism and FOIL can handle (I would guess that this number is somewhere between 100 and 1000 instances). To use this dataset with FOIL, you'll need to format it appropriately.
  2. Use each and every sample dataset that comes with FOIL release 6: ackermann.d, crx.d, member.d, ncm.d, qs44.d, and sort.d. In order to use Prism over those datasets, you'll need to format them appropriatedly.
- Preprocessing of the Data: You should apply relevant filters to your dataset as needed before doing the mining and/or using the results of previous mining tasks. For instance, you may decide to remove apparently irrelevant attributes, replace missing values if any, discretize attributes in a different way, etc. Your report should contained a detailed description of the preprocessing of your dataset and justifications of the steps you followed. If Weka does not provide the functionality you need to preprocess your data as you need to obtain useful patterns, preprocess the data yourself either by writing the necessary filters (you can incorporate them in Weka if you wish).
- Evaluation and Testing: If at all possible, use n-fold crossvalidation.

REPORT AND DUE DATE

Written Report.
Your report should contain the following sections with the corresponding discussions:
1. Code Description: Describe the Prism algorithm that you used from Weka and the FOIL algorithm IN YOUR OWN WORDS. Explain the algorithm underlying the code in terms of the input it receives and the output it produces, AND the main steps it follows to construct rules. Make sure to describe the correspondence between the algorithm you describe and the parts of the code that implement it.
2. Data: Describe the dataset that you selected in terms of the attributes present in the data, the number of instances, missing values, and other relevant characteristics.
  Provide a detail description of the preprocessing of your data. Justify the preprocessing you apply and why the resulting data is the appropriate one for mining rules from it.
3. Experiments: For each dataset used and for each experiment you ran describe:
  - Which of your 3-5 specific questions/conjectures about the dataset domain you aim to answer/validate with your experiments.
  - Data: What data did you use to construct and test your classifier?
  - Any additional pre or post processing done to the data or the classifier's output in order to improve the accuracy of your classifier.
  - Accuracy of the resulting classifier.
  - Readability of the resulting classifier. Describe the set of rules obtained.
  - Compare the results obtained with Prism and with FOIL.
  - Discuss how this accuracy compares with that of your most accurate ZeroR experiment, decision trees, neural nets, Bayesian methods, instance-based learning, regression methods, and genetic algorithms from the previous assignments.
4. Summary of Results
  - What was the accuracy of the most accurate hypothesis (set of rules) you obtained?
  - Discuss how this accuracy compares with that of your most accurate results from the previous assignments.
  - Readability of the resulting classifier. Describe the set of rules obtained.
  - Include the most accurate hypothesis you obtained in your report.
  - Discuss the strengths and the weaknesses of your project.
Oral Report. We will discuss the results from the individual projects during the class on April 12th. Your oral report should summarize the different sections of your written report as described above. Each of you will have 5 minutes to explain your results and to discuss your project in class. Be prepared!
Submission and Due Date.
1. Please submit the following file by email by Thursday April 12th at 3:00 pm.
  [your-lastname]_proj8_slides.[ext] containing your slides for your oral report. This file should be either a PDF file (ext=pdf) or a PowerPoint file (ext=ppt). Please use only lower case letters in the name file. For instance my file would be named ruiz_proj8_slides.ppt
2. Please bring a hardcopy of your report to class on April 12th, 2007. This written report is due at 4:00 pm that day.

CS539 Machine Learning - Spring 2007 Project 8 - Rule Learning

PROF. CAROLINA RUIZ

PROJECT DESCRIPTION

PROJECT ASSIGNMENT

REPORT AND DUE DATE

CS539 Machine Learning - Spring 2007
Project 8 - Rule Learning