WPI Worcester Polytechnic Institute

Computer Science Department
------------------------------------------

CS 533

Modeling and Performance Evaluation of Network and Computer Systems

Spring 2004

This course studies methods and concepts of computer and communication network modeling and system performance evaluation. Topics include some of stochastic processes, measurement techniques, monitoring tools, statistical analysis of performance experiments, simulation models, analytic modeling and queuing theory, work load characterization and performance evaluation problems. The required background is CS 504, EE 502 or an equivalent background in probability and an interest in learning about computer performance evaluation.

Index


Course Information

Class information:
day: Th
time: 6:00pm - 8:50pm
place: SL123 (Salisbury Labs)

Professor: Mark Claypool
email: claypool at cs.wpi.edu
office hours: M 12-1, F 10-11 and by appointment
place: FL-138
phone: x5409

Class email alias: cs533-all at cs.wpi.edu

Required Textbooks:

Other Reference Textbooks that look interesting:


Grading Policy

Final grades will be computed as follows:

Final grades will reflect the extent to which you have demonstrated understanding of the material, and completed the assigned projects. The base level grade will be a "B" which indicates that the basic objectives on assignments and exams have been met. A grade of "A" will indicate significant achievement beyond the basic objectives and a grade of "C" will indicate not all basic objectives were met, but work was satisfactory for credit. No incomplete grades will be assigned unless there exist exceptional, extenuating circumstances. Similarly, no makeup exams will be given unless there exist exceptional, extenuating circumstances.

Exams

There will be two in-class exams. The first is roughly mid-way through the semester and the second is during the last week of the semester. Exams will generally be closed book and closed notes, with the exception of a single sheet of paper (front and back) on which you can write (hand-written, not typed) whatever notes you wish. The majority of each exam will cover basic ideas and objectives of the class with a few questions testing additional understanding and insight.

Late Policy

Late assignments will be be penalized 10% of total assignment value per day (with the weekend counting as one day) or partial day, and no assignments will be accepted after seven days beyond the due date. All assignments are due as indicated when handed out. Exceptions to these rules can only be made beforehand.


Objectives

Here is the list of objectives you should be able to achieve after this course:

You should check out the assignment timeline to help you plan for doing projects and exams.


Slides

Slides from the in-class lectures will be available shortly after they are presented, depending upon how things go. Here is what we have so far:

Admin pdf ppt
Introduction (Ch 1-2) pdf ppt
Statistics (Ch 12-15) pdf ppt
Selection of Techniques and Metrics (Ch 3) pdf ppt
Types of Workloads (Ch 4) pdf ppt
Workload Characterization Techniques (Ch 6) pdf ppt
Data Presentation (Ch 10-11) pdf ppt
Monitors (Ch 7) pdf ppt
Capacity Planning and Benchmarking (Ch 9) pdf ppt
Experimental Design (Ch 16-17) pdf ppt
Simulation (Ch 24-25) pdf ppt
Queuing Theory (Ch 30-31) pdf ppt
Experiments pdf ppt
Group Work - ppt

[GS95] John Gustafson and Quinn Snell. HINT: A New Way to Measure Computer Performance, In Proceedings of the Fifth Annual Hawaii International Conference on System Sciences (HICSS), Honolulu, Hawaii, USA, 1995. (slides: ppt, pdf)

[Hen00] John L. Henning. SPEC CPU2000: Measuring CPU Performance in the Net Millennium, IEEE Computer, July 2000. (slides: ppt, pdf)

[LHPT95] P. Lukowicz, E. Heinz, L. Prechelt, W. Tichy. "Experimental Evaluation in Computer Science: A Quantitative Study", Journal of Systems and Software, 28(1):9-18, January 1995. (slides: ppt, pdf)

[WS00] A. Watson and M. A. Sasse. "The Good, the Bad, and the Muffled: the Impact of Different Degradations on Internet Speech", In Proceedings of ACM Multimedia, Los Angeles, CA, USA, November 2000. (slides: ppt, pdf)


Projects

There will be 4 projects related to performance evaluation. You must work individually for the projects unless otherwise noted. Projects will be posted here as they are developed. Here is what we have so far:


Samples

In this section are any code samples discussed in class, practice exams or any other demonstration-type class materials.

Final exam stuff:

Mid-term exam stuff:


Hotlinks

Professional Societies

There are two professional societies dedicated to performance evaluation. They are:

  1. Computer Measurement Group (CMG) - A not-for-profit industry group dedicated to measuring and managing computers.
  2. SIGMETRICS - An ACM special interest group for performance. More "academic" than CMG.

Benchmarks

There are numerous benchmarks available for computer system performance evaluation. A few are:

  1. SPEC
  2. HINT
  3. Perfect

Traces

Traces of actual computer and network behavior can be invaluable for driving simulations and building analytic models. Here are a few:

  1. National Trace Database - Mostly memory references with a few Ethernet traces
  2. BYU Trace Distribution Center - Processor traces
  3. MPEG-4 and H.263 Video Traces - Several sixty minute video traces showing frame types and sizes
  4. Web Traces and Logs - Table of known publicly available traces and logs
  5. Network game packet traces


[Return to the WPI Homepage] [Return to the CS Homepage] [Return to Mark Claypool's Homepage]


Mark Claypool (claypool at cs.wpi.edu)