Guidelines for Projects and for Written Reports

• General Guidelines for the Experiments

  • Objectives of the Data Mining Experiments: The overall objective of the project is to obtain the best models you can. The meaning of "best" might be the most accurate, or most readable, or the most accurate and readable, or another "goodness" criterion described in the challenges provided.

  • Data and Domain Knowledge: Before you start running experiments, make sure to understand the raw data (based on the knowledge gained in Project 1) very well, to learn as much as you can about the domain, and to research approaches used by others on this dataset to the extent possible.

  • Performance metric(s): Classification accuracy is typically the metric that we want to maximize. But it is important to also pay attention to other metrics relavant to the data mining method under consideration. Include in your report the size of the models you construct, their readability, and how much time it took Weka to construct them.

  • Preprocessing of the Data: For each experiment, you should pre-process your data based on the knowledge you have of the data and the domain, as well as based on the results of previous experiments. For instance, you may decide to remove apparently irrelevant attributes, replace missing values if any, discretize attributes in a different way, etc. If Weka does not provide the functionality that you need to preprocess your data to obtain useful patterns, preprocess the data yourself either by writing the necessary filters (you can incorporate them in Weka if you wish), or by using a separate tool.

  • Training and Testing Instances: Use 10-fold cross-validation (unless the combined execution time is prohibiting, cross-validation is not consistent with the performance metric(s) chosen, or the particular project description asks you to use a different experimental protocol).

• Specific Guidelines for the Experiments and Written Report

  Your written report should follow the structure described below:

  • Page 1 should contain each of the team members' names, and authorship - that is, a description of who did what on this project.

  • [10 points] Algorithms and Code: (maximum 1 page)
    Page 2 should provide a description (in very high level pseudo-code) of the Weka code that implements the data mining technique(s) covered by the project. In your written report, describe the algorithm underlying the code IN YOUR OWN WORDS, using high level pseudo-code. Briefly explain the algorithm in terms of the inputs it receives and the outputs it produces, AND the main steps it follows to construct the model (in pseudo-code). Make sure that the algorithms you described are the ones implemented in the Weka code, which are not necessarily the same as the ones described in the textbook or in class.

  • Experiments and Results: (maximum 3 pages per challenge)
    The next pages should contain descriptions of your experiments and answers to the provided challenges. Maximum 2 pages per challenge. For each challenge:
    • [15 points per challenge] Challenge's page 1:
      Use tables to summarize the data, pre-processing, settings, postprocessing, and results of the experiments you run. You should run a sufficient number of coherent experiments (that is, the results of an experiment should motivate the next experiment(s)). Each row in the table corresponds to an experiment and the columns are as following:
      1. Data: What data did you use to construct and test your model?
      2. Pre-Processing: What pre-processing was done to the data in order to improve the model's performance.
      3. Post-Processing: What post-processing was done to the model in order to improve the model's performance.
      4. Experimental Protocol: Only if for some reason it is different from 10-fold cross-validation.
      5. Resulting model: Describe the size and readability of the resulting model. Include any relavant observations about the model.
      6. Performance of the resulting model:
         • State what the performance of the model is.
         • If applicable, elaborate on the confusion matrix and/or other relevant performance indicators.
         • How long did it take Weka to construct this model?

    • [10 points per challenge] Challenge's Pages 2 and 3: Best Performing Model.
      
      • The first of these two page should provide a detailed description of the experiment that yielded the best performing model you were able to construct for this challenge. Include detailed descriptions of:
        1. pre-processing done,
        2. advanced techiques used,
        3. clever ideas used during the contruction of the model Provide both motivation and justification for running this experiment in the way that you did (an important aspect of this part of the report is story-telling). Provide any useful graphs and visualizations.
        4. answers to the specific questions asked in the challenge description.
      • The second of these two pages should contain the best performing model (e.g., decision tree).

  • [10 points] Conclusions and Lessons Learned. The last page of your report should include a discussion of:
    • How the performance of the experiments in this project compare with that of your best performing results from previous projects on the same dataset.
    • How well the particular data mining methods used in this project worked on this dataset.
    • What combination of pre-processing, parameter values, and/or post-processing yielded particularly good results?
    • What lessons you learned in this project and how you would use these lessons on the next project.
    • Strengths and weaknesses of your project.

Guidelines for Oral Reports and Slides

• [10 points] Your oral report should summarize the most important parts of your written report and should elaborate only on the most significant or more unique parts of your work. Each group will have 4 minutes to present their project in class. Given the time constraint, your presentation should consist of 3-4 slides (and no more!). Once again, an important aspect of both your written and oral reports is the "story-telling" aspect. Try to tell the story of what experiments you ran and why, how each experiment shed lights on what experiment(s) to run next, and what you learned with them. Be prepared and use your presentation time wisely!
• [5 points] Class participation. Your attention and class participation during project presentations will be graded.

Submission and Due Dates

1. Please submit the following files by email to cs4445-staff@cs.wpi.edu BY 9:00 AM the day the project is due (submissions received after 9:00 am won't be accepted):
   1. Your group's oral presentation: Submit the following file with the slides for your (group's) oral report:

      [your-lastname]_[your-partner's_lastname]_proj[n]_slides.[ext]

   where: [n] is the project number; [ext] is pdf, ppt, or pptx. Please use only lower case letters in the name file, and sort the lastnames in alphabetical order. For instance, the file with my slides for Project 1 would be named ruiz_proj1_slides.ppt (if I worked on the project alone) or ruiz_smith_proj1_slides.ppt (if I worked on the project in a group with Joe Smith).

2. In addition, each groupd should submit:
   1. a hardcopy of the individual solutions (one per student) to the homework assignment, AND
   2. a hardcopy of your group written report
   by the beginning of class the day that the project is due.

Grading Criteria