Before you start running experiments, make sure to understand the raw data very well, to learn as much as you can about the domain, and to research approaches used by others on this dataset to the extent possible.

• Dataset Description:
  In your written report, describe the dataset in terms of the attributes present in the data, the number of instances, missing values, and other relevant characteristics. Describe briefly what pre-processing (if any) was used before any experiments were ran.

• For each data mining technique used in the project:

  1. Study the corresponding chapters/sections of the textbook specified in the Course Schedule.

  2. Algorithms and Code: Read the Weka code and the Matlab, RapidMiner, and SPSS documentation corresponding to the data mining technique(s) covered by the project. Be prepared to describe the algorithm underlying the code IN YOUR OWN WORDS. Be prepared to explain the algorithm in terms of the inputs it receives and the outputs it produces, AND the main steps it follows to construct the model, using high-level pseudo-code. Make sure to be able to describe the correspondence between the algorithm you describe and the parts of the code that implement it (which are not necessarily the same as the ones described in the textbook or in class). Detailed questions about the algorithm and the code will be included in the in-class test.

  3. Guiding Questions - Objectives of the Data Mining Experiments: Before you start running experiments, look at the raw data in detail. Figure out 3 specific, interesting questions about the dataset domain (e.g., diabetis, weather, labor contracts, ...) that you want to answer with your data mining experiments. Choose questions that are about the domain (not about the data mining method or the experimental parameters!) that would particularly benefit from using the data mining method under study. These questions may be phrased as conjectures that you want to confirm or refute with your experimental results, or as plain questions.

  4. Performance metric(s): Use the performance metrics specified in the project description. If these are not given, explain what performance metric(s) you will use to evaluate the models you construct (e.g., accuracy, error rate, size of the model, readability of the model, ...) and why.

  5. Preprocessing of the Data: Try to keep the initial pre-processing to a minimum at first. You should apply only necessary filters to your dataset before starting mining the data, and then introduce additional pre-processing as needed based on the results of experiments you run. Your report should contained a detailed description of the preprocessing of your dataset and justifications of the steps you followed. If Weka/Matlab/RapidMiner/SPSS do not provide the functionality that you need to preprocess your data to obtain useful patterns, preprocess the data either by writing your own scripts and filters, or by using a separate tool.

  6. Training and Testing Instances: Use 10-fold cross-validation, unless the data is insufficient, the combined execution time is prohibiting, cross-validation is not consistent with the performance metric(s) chosen, or the particular project description asks you to use a different experimental protocol. Justify your choice if different from that requested by the project description.

  7. Experiments:

     You must run a sufficiently large and coherent set of experiments. Start with a basic experiment with default parameters (if possible), and design new experiments varying the settings (i.e., pre-processing, parameters, and/or post-processing, ideally varying one setting at a time) based on the results that you obtain in your experiments. Each experiment should be motivated by a previous experiment, and by the guiding questions.

     • For each experiment you ran describe:
       1. Objectives: Which of your 3 specific guiding questions/conjectures about the dataset domain you aim to answer/validate with this experiment. Describe also any additional objectives for this experiment that might have been motivated by your previous experiments.
       2. Data: What data did you use to construct and test your model?
       3. Parameters and Settings: Describe what parameter values and other settings you used and why.
       4. Additional Pre or Post Processing: Any additional pre or post processing done to the data or the model in order to improve the model's performance, as measured by the performance metric(s) chosen.
       5. Resulting model: Describe the resulting model (e.g., size of the model, readability). If the model is readable summarize in your own words what the model says, focusing on the most interesting/relevant patterns. Elaborate on if and how the model answers the objectives of this experiment.
       6. Performance of the resulting model:
          • State what the performance of the model is. If applicable, elaborate on the confusion matrix and/or other relevant performance indicators.
          • How long it took Weka/Matlab to construct this model?
          • Compare the performace of this model with that of other models constructed in this project for this dataset.

  8. Analysis of Results
     • Provide insightful observations and comments on the results of the experiments.
     • What general observations can you draw regarding the quality/performance of the models as you varied the settings (i.e., pre-processing, parameters, post-processing) of the experiments?
     • What was/were the model/models with the highest performance? Consider both quantitative (e.g., model accuracy, time taken to built the model, ...) as well as qualitative (e.g., model size, readability, ...) evaluation criteria.
     • Discuss how this performance compares with that of your best performing results from previous projects (if any) on the same dataset, including ZeroR and OneR.
     • Include the model (or at least a representative part of the model if too large to include full model) in your report.
     • Elaborate on whether or not your experiments helped you answer your initial 3 objectives, and what answers you obtained for these guiding questions.
     • Discuss how well this particular data mining method worked on this dataset. What combination of parameters yielded particularly good results?
     • Overall project conclusion: Discuss the strengths and the weaknesses of your project.