As the number of active Internet users continues to grow and streaming media applications become more commonplace, the number of users and volume of data traversing the Internet is increasing in explosive proportions. The sheer number of possible users and applications at any point in time raises the probability of streaming multimedia flows encountering congestion.
To overcome short-term congestion and avoid long-term congestion collapse, there is a growing consensus that all Internet applications must be TCP-Friendly. In fact, there are proposed approaches to detect and restrict the bandwidth of non-TCP friendly flows. Thus, many new streaming media protocols are TCP-friendly, but do not use TCP's retransmissions to avoid excessive delay and delay jitter, leaving the media flow susceptible to quality degradation from packet loss.
While multimedia applications can tolerate some data loss, excessive packet loss during congestion yields unacceptable media quality. Since video encoding involves interframe dependencies to achieve high compression rates, the random dropping of packets by routers can seriously degrade video quality. In MPEG, for example, dropping packets from an independently encoded I frame causes the following dependent P and B frames to not be fully decodable. In practice, interframe dependencies have been shown to cause a 3% packet loss rate to result in a 30% frame loss rate.
Streaming media flows often utilize lower latency repair approaches, such as Forward Error Correction (FEC), in conjunction with TCP-Friendly protocols to deliver streaming applications over the Internet. However, FEC requires additional repair data to be added to the original video data. If a streaming video is to operate within TCP-Friendly bandwidth limits, the additional FEC data will reduce the effective transmission rate of the original video content.
Selecting the best distribution of FEC packets within video frames with inherent interframe encoding dependencies can be cast as a constrained optimization problem that attempts to optimize the quality of the video stream. Current approaches use either apriori, static FEC choices or adapt FEC to perceived packet loss on the network without regard to TCP-Friendly data rate constraints.
To preserve real-time streaming media playout, multimedia servers must scale back their streaming data rate to match the TCP-Friendly data rate, an approach called media scaling. Temporal scaling is a widely used form of media scaling whereby the multimedia server selectively discards frames prior to transmission. Armed with knowledge about the relative importance of specific frame types and interframe dependencies, a multimedia application can discard the least significant packets with respect to perceived quality, while a congested router can only randomly drop packets.
We derive an analytic model that characterizes the performance of temporally scaled MPEG video with Forward Error Correction in the presence of packet loss. Given parameters to represent network loss, and MPEG frame types and sizes, our model allows specification of the number of FEC packets per MPEG frame type and temporal scaling pattern and computes the total playable frame rate. We represent a presentation layer network protocol by using our model to exhaustively search all possible combinations of FEC and temporal scaling patterns to find the combination of FEC and temporal scaling that yields the maximum playable frame rate under the TCP-Friendly bandwidth constraint. The analytic calculations required by the search can be done in real-time, making the determination of optimal choices for adaptive FEC feasible for most streaming multimedia connections.
For details on our approach, including extensive evaluation, see the following papers:
Huahui Wu, Mark Claypool, and Robert Kinicki. A Model for MPEG with Forward Error Correction and TCP-Friendly Bandwidth, Proceedings of Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), Monterey, California, USA, June 2003.
Huahui Wu, Mark Claypool, and Robert Kinicki. Adjusting Forward Error Correction with Temporal Scaling for TCP-Friendly Streaming MPEG, Technical Report WPI-CS-TR-03-10, Computer Science Department, Worcester Polytechnic Institute, April 2003.
To provide a brief look at the possible benefits of our approach, we compare the playable frame rate for four FEC choices:
In all cases, the total bandwidth used by the MPEG video plus FEC is temporally scaled to meet TCP-friendly bandwidth constraints.
a. Comparison of Playable Frame Rates (Small Frame Sizes) |
b. Comparison of Playable Frame Rates (Medium Frame Sizes) |
c. Comparison of Playable Frame Rates (Large Frame Sizes) |
|
|
The figures depict the playable frame rates for each FEC choice. For all figures, the x-axes are the packet loss probabilities, and the y-axes are the playable frame rates. For frame rate targets: 24-30 frames per second is full-motion video, 15 frames per second can approximate full motion video for some video content, 7 frames per second appears choppy, and at 3 frames per second or under a video becomes a series of still pictures.
From the Figure, adjusted FEC provides the highest playable frame rate under all network and video conditions. For the typical video size (b), the benefits of adjusted FEC over non-FEC is substantial, more than doubling the frame rate at 1% loss, and still surpassing the minimum 3 fps at 4% loss. The two fixed FEC approaches usually improve playable frame rates over non-FEC video, and FEC(4/2/1) even achieves the playable frame rate provided by adjusted FEC for a few loss rates, such as 2.5%.
For the smaller video frame sizes (a), created by halving the frame sizes and doubling the round-trip time in order to provide an available bandwidth that allows for visual comparison between graphs, FEC(1/0/0) does substantially better, coming closer to the maximum frame rate achieved by adjusted FEC. FEC(4/2/1) does worse even than non-FEC, since the fixed number of FEC packets added is a larger fraction of overhead for the smaller video frames.
For the larger video frame sizes (c), created by doubling the frame sizes and halving the round-trip time, FEC(4/2/1) does substantially better, providing close to the maximum frame rate achieved by adjusted FEC. FEC(1/0/0) does significantly worse since it does not provide enough protection for the larger frame sizes, with frame rates well below that of adjusted FEC, but still provides improvement over non-FEC.
Additional analysis and evaluation of our adjusted FEC approach and other fixed FEC schemes can be found in the papers listed at the end of the introduction.
| Non-FEC | 2% loss (8.4 fps) | 3% loss (4.0 fps) | 4% loss (1.6 fps) |
| Fixed FEC (1,0,0) | 2% loss (12.5 fps) | 3% loss (6.6 fps) | 4% loss (3.1 fps) |
| Fixed FEC (4,2,1) | 2% loss (12.4 fps) | 3% loss (7.5 fps) | 4% loss (2.5 fps) |
| Ajusted FEC | 2% loss (16.6 fps) | 3% loss (9.5 fps) | 4% loss (3.9 fps) |
For questions or comments, please send email to: