CS539 Machine Learning - Fall 2014 
Project 8 - Rule Learning & Comparison of ML Methods

PROF. CAROLINA RUIZ 

This is an individual project
Slides are due on Fri. Dec. 12th at 10:00 am.
Written Report is due on Tu. Dec. 16th at the beginning of class. 

Project Assignment:

• Read in great detail Sections 10.1-10.5 of the textbook on Rule Learning.

• Read the Quinlan's "New Generation Computing '95" paper on FOIL.

• *** The project description below specifies the structure of the written report and the page limit for each section (no template will be provided this time). The font size should be no smaller than 11pts.

• [200 points] Part I.

  • Machine Learning Technique(s): Use sequential covering algorithms to construct sets of classification rules.

    • The JRip algorithm implemented in the Weka system. Read the code of the JRip classifier in Weka in detail.

    • The FOIL system (release 6) developed by Quinlan. This and other versions of FOIL are available online at Quinlan's Webpage.

      See basic instructions for installing FOIL (thanks, Ermal and Nitish!). Also, see the =README= and the MANUAL files that come with FOIL.

  • Dataset(s): For Part I of this project, we will use the following datasets:

    • The sample problems that come with the FOIL release 6 system:
      • ackermann.d
      • member.d
      • ncm.d
      • sort.d

    • The Flags dataset available at the UCI Machine Learning Repository. Use religion as the target attribute.

      Try to add basic relationships to this dataset that FOIL can use as building blocks to construct meaningful rules.
      See the crx.d dataset, which comes with FOIL, for an example of a tabular dataset that has been transformed into the .d format. It is a subset of the Credit Approval dataset available at the UCI Machine Learning Repository.

  • For each of the five datasets above:
    1. Run experiments with both JRip and with FOIL on the dataset.
       The Part I section of your oral and written reports should contain at most one page per dataset providing:
       1. [5 points] A brief description of the dataset.
       2. [5 points] A description of the experiments you ran with JRip and with FOIL on the dataset.
       3. [10 points] A description of the results of the experiments.
       4. [10 points] The interpretation of the FOIL rules in plain English.
       5. [10 points] Analysis of the results.
       6. Note: No need to describe the JRip or FOIL code.

  ---

• [210 points] Part II. Complete the following table comparing the machine learning techniques used on the course projects.

  • Dataset:
    Use a dataset of your choice. This dataset can consist of data that you use for your own research or work, a dataset taken from a public data repository (see for example my list of additional Machine Learning, AI, Data Mining, Statistics, Databases, Data Sets and other online resources), or data that you collect from other public data sources. This dataset should contain a large enough number of instances, and a combination of nominal and numeric attributes. Pick a dataset that is well suited for using the machine learning techniques on the table below.
  • Experiments:
    Run experiments so that you can report results using the same evaluation approach (if at all possible 10-fold cross-validation, if not 4-fold cross-validation, or percentage split), the same training and testing datasets, etc. If at all possible, use R only. If not, use Weka only (in that case, I suggest you use the Weka Experimenter to run experiments under the same conditions, and obtain paired T-test comparisons). But there is no need to use both R and Weka, unless you need to do so for some reason.

  Technique	DecisionTrees	NeuralNetworks	NaiveBayes/BayesNets	Instance-Based IB1/IBk/LR/LWR	GeneticAlgorithms	RuleLearning JRip/Foil
  Code Used. If at all possible R, if not Weka
  Dataset (name):
  Accuracy (or error metrics). List metrics used.
  Statistically significantly better than: (list methods) with p ≤ 0.05
  Size of the model
  How readable is the model?
  Interesting patterns in the model
  Number of attributes used
  Num. of training instances
  Num. of test instances
  Missing values included?(y/n)
  Pre-processing
  Evaluation method used (n-fold cross val with n=?)
  Training Time
  Testing Time
  Strengths and Weaknesses of the method

  For those methods for which two or more alternatives are listed on the table (e.g., Naive Bayes / Bayesian Nets), provide the required information for each of the alternative listed, in the order they are listed, separated by "/"s (e.g. "78% / 81%", under accuracy if the accuracy of your best Naive Bayes model was 78% and the accuracy of your best Bayesian Net was 81% on the dataset analyzed).

  • Written and Oral Reports: Part II of your reports should contain:
    1. [10 points] A description of your dataset (at most 1/2 page).
    2. [170 points] Your complete table (as many pages as it takes).
    3. [30 points] Overall analysis of your table and visualizations (at most 1 page).

  ---

• [45 points] Part III. Advanced Topic.
  Pick one of the machine learning methods from the table, and investigate in great depth (BOTH theoretically AND experimentally) a topic of your choice that is related to the machine learning method and that was not covered already in the course or your previous projects. This topic might be something that was described or mentioned in the textbook or in class, or that comes from your own research, or that is related to your interests. But this advanced topic MUST be related to a machine learning method studied in this course.

  • Written and Oral Reports: Part III of your reports should contain:
    1. [5 points] The machine learning method selected from the table, the name of the advanced topic, and an explanation of how the advanced topic relates to the machine learning method (at most 1/3 page).
    2. [20 points] An indepth description of the advanced topic (at most 2/3 page).
    3. [10 points] Detailed description of the experiments ran applying the advanced topic to the dataset used for Part II, and their results (at most 1/2 page).
    4. [10 points] Analysis of these results and visualizations (at most 1/2 page).