Metrics

Metrics quantify the performance of congestion control algorithms. They are captured while experiments are running and stored in a database. This page gives an overview about the metrics that are used in ccperf.

Metric Types

Metrics can be categorized into three types:

  • Flow metrics capture the performance of an individual flow. They are measured at the endpoints of a network path at either the source, the receiver, or both.

  • Link metrics are recorded at the network links of interest, typically at bottlenecks. They include metrics that measure queue states.

  • Network metrics assess the entire network as a whole by aggregating other metrics, e.g., the aggregated throughput of all flows.

Sample Types

There are three ways how metrics are sampled:

  • A sample metric is sampled \(n\) times to give the vector (sequence) of metric samples \([x_1, x_2, \ldots, x_n] \in \mathbb{R}^n\). When the sampling is done within one experiment at different measurement (sampling) times \(t = 1, 2, \ldots, T\), we refer to the sample metric as a time series \(x_t = [x_1, x_2, \ldots, x_T]\).

  • A singleton metric \(x \in \mathbb{R}\) is a single value that quantifies some measurable property. For example, the flow completion time quantifies the lifetime of a flow in an experiment.

  • A statistical metric \(x \in \mathbb{R}\) condenses one or more other metrics to a single real number. Common statistical metrics in ccperf are the arithmetic mean mean_x, sample variance var_x, coefficient of variation cov_x = std_x / mean_x, or accumulation agg_x \(= \sum_{i}^{n} x_{i}\) of a sample metric \([x_1, x_2, \ldots, x_n]\). For example, mean_throughput averages the time series of a flow’s throughput over an experiment.

Flow Metrics

Metric Type Layer Unit Description
in_flight Sample Transport Segments Amount of unacknowledged segments in flight
cwnd Sample Transport Segments Congestion window
throughput Sample * Mbps Throughput (data rate) measured at the destination
sending_rate Sample * Mbps Sending rate at source
srtt Sample Transport ms Smoothed RTT estimate
owd Sample Application ms One-way delay
recovery_time Sample Transport ms Duration spent in loss recovery
network_power Sample * Mbps/ms Kleinrock’s network_power \(= \frac{\texttt{throughput}}{\texttt{srtt}}\)
utilization_bw Sample * Bandwidth utilization \(u = \frac{\texttt{throughput}}{\texttt{bandwidth}}\)
utilization_bdp Sample Transport BDP utilization \(u = \frac{\texttt{in_flight}}{\texttt{BDP}}\)
utility_x_pf Sample * Proportional fairness \(U(x) = log(x)\)
utility_x_mpdf Sample * Minimum potential delay fairness \(U(x) = \frac{1}{x}\)
flow_completion_time Singleton Link s Duration between the first sent byte and last received byte
total_retransmissions Statistical Transport Number of retransmitted segments
total_rtos Statistical Transport Number of retransmission timeouts (RTOs)
Metric Type Layer Unit Description
qdisc_delay Sample TC ms Queueing delay (Sojourn time) of qdisc
qdisc_length Sample TC packets Number of packets in qdisc
qdisc_drops Sample TC packets Number of dropped packets by qdisc
total_qdisc_drops Statistical TC packets Number of dropped packets by qdisc in total

Network Metrics

In the context of network metrics, an aggregated flow metric of type \(x\) accumulates indivual flow metrics with agg_x \(= \sum_{k=1}^K x_k\), where \(x_k\) is the flow metric of the \(k\)-th flow. The dimension over which \(x\) is accumulated is not explicitly specified in the name agg_x for brevity. Likewise, time series can be aggregated and averaged, e.g., mean_agg_throughput \(\,= \frac{1}{T} \sum_{t=1}^T \sum_{k=1}^K x_{t,k}\). A network metric mean_agg_x is averaged over time and accumulated over flows. Moreover, the following network metrics are measured as well:

Metric Type Layer Unit Description
jains_fairness_x * * Jain’s index of an allocation \(\boldsymbol{x}\) given by \(f(\boldsymbol{x}) = \frac{\left(\sum_{k = 1}^K x_k \right)^2}{K \sum_{k = 1}^K x_k^2}\)
entropy_fairness_x * * Entropy fairness of an allocation \(\boldsymbol{x}\) given by \(f(\boldsymbol{x}) = \exp\left(H\left(\frac{\boldsymbol{x}}{\sum_{k=1}^{K} x_k}\right)\right)\), where \(H\) is the Shannon entropy function
product_fairness_x * * Product fairness of an allocation \(\boldsymbol{x}\) given by \(f(\boldsymbol{x}) = \prod_{k=1}^K \frac{x_k}{\sum_{k=1}^K x_k}\)

Flow Scores

Scores are metrics with two key properties: they are normalized to a range of [0, 1] and higher values indicate better performance. The following flow scores summarize the performance of a flow in an experiment. Note that some scores use a squashing function \(\phi(x; k) = 1 - \mathrm{e}^{-kx}\) to map values to the range [0, 1].

Name Description Formula
score_tput The mean bandwidth utilization (mean_utilization_bw) \(\frac{1}{T}\sum_{t=1}^{T} \frac{\text{throughput}_t}{\text{btl_rate}_t}\)
score_delay The mean normalized queueing delay (\(\text{max_qdisc_delay}\) is the max. possible queueing delay) \(1 - \frac{1}{T}\sum_{t=1}^{T} \frac{\text{qdisc_delay}_t}{\text{max_qdisc_delay}}\)
score_fct The normalized flow completion time (\(T\) is the total duration of the experiment) \(1 - \frac{\text{flow_completion_time}}{T}\)
score_retransmissions The (squashed) number of retransmissions \(1 - \phi(\text{retransmissions}; k=qsize)\)
score_rtos The (squashed) number of retransmission timeouts \(1 - \phi(\text{rtos}; k=2)\)