CS539 Machine Learning. Fall 2014 
Project 1 - Data Preprocessing

PROF. CAROLINA RUIZ 

DUE DATE: Tuesday, Sept 16th 2014:

• Slides are due at 10:00 am (by email to Prof. Ruiz), and
• a hardcopy of the written report is due at 2:00 pm (beginning of class).

 

• Project Goals
• Project Assignment
• Oral and Written Report Submission and Due Date

PROJECT GOALS

• to gain experience with data preprocessing including data cleaning, attribute discretization, and dimensionality reduction;

• to gain familiarity with R's and Weka's preprocessing functionality.

PROJECT ASSIGNMENT

---

Instructions:

• You must work on this project individually. That is, your project solutions must be your own. Help or assistance from classmates, other people, or online resources are NOT allowed.
• You must use the Project 1 report template for your written report, not exceeding the page limits stated in the template nor decreasing the font size.
• Follow the directions under "Oral and Written Report Submission and Due Date" below to prepare and submit your slides and written report.

---

• Download and install R and Weka as described on the course webpage.

  Regarding Weka:

  • Find the Weka code in a file called "weka-src.jar", which should be located in the directory where Weka was installed. You might need to unzip and/or use jar utilities this file to extract its contents. Inside, you will find the .java files that implement Weka.
  • Consult the "README" file, the "documentation" webpage, and the "WekaManual" provided with the Weka system (in the same directory where Weka was downloaded). Browse through the "Package Documentation" to become familiar with it.
  • When needed, use the following command to increase the amount of main memory used by Weka. Here, I'm increasing the amount of main memory used by Weka to 768m, but you can specify any other size instead of 768 if more memory is needed/available:

    java -Xmx768m -jar weka.jar

  Regarding R:
  

  • Although the R package RWeka provides an R/Weka interface, you are required to use R primitives and R packages directly without invoking Weka to complete the portions of the project that are intended to be done with R.

• Dataset: Consider the following dataset: The Diabetes 130-US hospitals for years 1999-2008 Data Set available from the Univ. of California Irvine (UCI) Data Repository.
  

  • Browse addional information provided about this dataset in the paper cited on the dataset website:

    Beata Strack, Jonathan P. DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, and John N. Clore, .Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records,. BioMed Research International, vol. 2014, Article ID 781670, 11 pages, 2014. .

    In particular, carefully read the description of the attributes provided in Table 1 of this paper. Note that even though some attributes are codified using numeric values, they are nominal (not numeric) attributes. Use the type of each attribute (numeric or nominal) as provided in Table 1 in the remainder of this project.

  • When using supervised techniques, assume that "readmitted" is the target attribute.

  • Convert this data to the arff format. For this you can either use any tools provided by Weka, or you can make the conversion outside the Weka system using other tools (e.g., a word editor, Excel, etc.).

  • Modify this data as needed so that it can be uploaded onto R. Consider representing it using R's data frames. See my notes on R. [Throughout this project, take advantage of R's superb plotting capabilities to show your results graphically.]

• Experiments:

  1. Data Exploration.

     1. Start by familiarizing yourself with the dataset. Carefully look at the data directly (for this use Excel or a file editor, as well as Weka's and R's funcionality to explore and to visualize the data). Describe in your report your observations about what is good about this data (mention at least 3 different good things), and what is problematic about this data (mention at least 3 different bad things). If appropriate, include visualizations of those good/bad things.
     2. Based on the above exploration, identify all of the attributes that you would remove from the dataset from the start and explain why.

     ---

  2. Data Preprocessing. We will preprocess the dataset attributes using Weka's filters and R functions as described below.

     1. Discretization. Explore different ways of discretizing the continuous attribute "num_lab_procedures". That is, convert this numeric attribute into a nominal one by binning it into intervals.
        1. In Weka's Explorer:
           1. Use the Discretize filter under the Supervised Filters. Play with the filter and its parameters. Read the Java code implementing it, and briefly describe this code in your written report.
           2. Use the Discretize filter under the Unsupervised Filters. Play with the filter and its parameters. Read the Java code implementing it, and briefly describe this code in your written report.
        2. In R, find out what R functions can be used to bin numeric attributes into nominal ones. Include a short description of these functions in your report, as well as a description of the results you obtained by using them. If needed, write your own R code for discretization.

     2. Missing Values. Consider the following data attributes which contain mising values: "weight", "medical_specialty", and "race". We will explore different ways of removing missing values in each of these 3 attributes.
        1. In Weka, missing values in arff files are represented with the character "?". See the weka.filter.ReplaceMissingValuesFilter in Weka. Play with the filter and read the Java code implementing it.
        2. In R, find out what R functions can be used to replace missing values. Include a short description of these functions in your report, as well as a description of the results you obtained by using them.
        3. For each of the 3 attributes, discuss whether or not any of the approaches above was a sensible way to deal with missing values. If not, propose a better approach to do so, implement it in R, and discuss the results you obtained with your implementation.

     3. Attribute/Feature Selection.
        1. Use only R for this part.
           1. Using the original set of numeric/continuous attributes (without discretization), calculate the correlation matrix of these attributes.
           2. If you had to remove 3 continuous attributes from the dataset based on the correlation matrix, which attributes would you remove and why? Explain your answer.
        2. Use only Weka for this part.
           1. Apply Correlation Based Feature Selection to the original dataset. For this, use Weka's CfsSubsetEval available under the Select attributes tab with default parameters. Include in your report which attributes were selected by this method.
              See Chapter 7 of Witten's, Frank's, and Hall's Weka textbook (available online from the WPI Library, and Witten's and Frank's textbook slides - Chapter 7 (Slides 5-6) for a description of this method.
           2. What can you observe about these selected attributes with respect to the correlation matrix you computed with R above? Were the 3 attributes you chose to remove above kept or removed by CfsSubsetEval?
        3. In R, find out what R functions can be used for feature selection. Include a short description of these functions in your report, as well as a description of the results you obtained by using them.

     4. Attribute/Feature Extraction.
        1. Using Weka, apply Principal Components Analysis to the dataset. Use "varianceCovered"=0.95. Discuss your results in your report.
        2. Using R, apply Principal Components Analysis to the dataset. Discuss your results in your report.
        3. Compare the results obtained with each of the two tools. Are they the same? if not, how are they different?

     ---

  3. Model Construction: ZeroR and OneR. In this part we will use both Weka and R to construct "ZeroR" and "OneR" classifiers over the dataset. In Weka, these classifiers appear under the "Classify" tab. In R, look to see if these classifiers are implemented, if not implement them yourself and include your R code in your report.

     Experiment with these two machine learning techniques varying the following settings:

     • Testing method: Use different ways of testing your models. That is, explore the following alternatives:
       • Testing the model over the training data.
       • Using percentage split (i.e., splitting the input dataset into two parts one for training and one for testing).
       • Using n-fold crossvalidation. Play with different values for n.
     • Data preprocessing: Run experiments without any data preprocessing first. Then experiment with preprocessing (e.g., attribute discretization, and/or replacing missing values, and/or dimensionality reduction, ...), and see how the preprocessing used alters the obtained models.

     Include in your report a summary of the ZeroR and OneR results obtained modifying the settings as described above. Analyze your results. In particular, pay attention to the models constructed, the accuracy (percentage of correctly classified instances), the error (percentage of incorrectly classified instances), the confusion matrices, and any other part of the output that you find interesting.

ORAL AND WRITTEN REPORTS AND DUE DATE

• Written Report. Please hand in a hardcopy of your report at the beginning of class when the project is due.

• Oral Report. We will discuss the results from the individual projects during the class when the project is due. Each of you will have approximately 3 minutes to present your project. Prepare SLIDES with the results of your experiments. Your slides should be a good summary of your project work. Be ready to show your results and to discuss your project in class within the time allowed. Given the time constraints, focus your presentation on the most relevant, unique, or creative parts of your project.

  Email your slides to the professor by the deadline stated at the top of this webpage. The name of the file containing your slides should be:

   [your-lastname]_proj1_slides.[ext]  

  This file should be either a PDF file (ext=pdf) or a PowerPoint file (ext=ppt). Please use only lower case letters in the name file. For instance, the file with my slides for Project 1 would be named ruiz_proj1_slides.ppt