Queues and Scheduling at Core – Facing Interfaces
The routing model so far is that all the traffic sourced from PE X to PE Y travels inside a single LSP established between the two, and the maximum number of hops crossed is four, which includes both the source and destination PE routers. If the primary path fails, the secondary path is used, which raises the maximum number of hops crossed to five.
Figure 9.15 Interface bandwidth and buffer division across the classes of service
Later in this case study, we analyze the pros and cons of having multiple LSPs with bandwidth reservations between two PE routers.
In terms of granularity, a core-facing interface is only class of service aware, so all customer traffic that belongs to a certain class of service is queued together, because it all requires the same behavior. Also, control traffic on the core interface needs to be accounted for.
How a queuing and scheduling mechanism operates depends on three parameters associated with each queue: the transmit rate, the length, and its priority. We have already specified the priority to be assigned to each queue in Table 9.2 and have explained the reasons for these values, so we now focus on the other two parameters.
To handle resource competition, an interface has a maximum bandwidth value associated with it, which can either be a physical value or artificially limited, and a total amount of buffer. Both of these are divided across the queues that are present on that interface, as illustrated in Figure 9.15.
Choosing a queue transmit rate depends on the expected amount of traffic of the class of service that is mapped to that specific queue, so it is a parameter tied to network planning and growth. Such dimensioning usually also takes into account possible failure scenarios, so that if there is a failure, some traffic is rerouted and ideally other links have enough free bandwidth to cope with the rerouted traffic.However, it is becoming clearer that there is always a close tie between a QOS deployment and other active processes in the network, such as routing or the link bandwidth planning.
Regarding the amount of buffer allocated to each queue, the RT class of service is special because of its rigid delay constraint of a maximum of 75 milliseconds (ms). When RT traffic crosses a router, the maximum delay inserted equals the queue length in which that traffic is placed. So the value for the queue length to be configured at each node can be determined by dividing the 75 ms value by the maximum number of hops crossed. However, such division should take into account not just the scenario of the traffic using the primary path, which has four hops, but also possible failure scenarios that cause the traffic to use the secondary path, which has five hops. Not accounting for failure of the primary path may cause traffic using the secondary path to violate the maximum delay constraint. So, dividing the maximum delay constraint of 75 ms by five gives an RT queue length of 15 ms.
This dimensioning rule applies to both core- and customer-facing interfaces, because the maximum number of hops crossed also needs to take into account the egress PE interface, where traffic is delivered to the customer.
After the queue length for RT has been established to comply with the delay requirement, an interesting question is what assurances such dimensioning can offer regarding the maximum jitter value. From a purely mathematical point of view, if the delay varies between 0 ms and 75 ms, the maximum delay variation is 75 ms. However, the jitter phenomenon is far more complex than this.
When two consecutive packets from the same flow are queued, two factors affect the value of jitter that is introduced. First, the queue fill level can vary, which means that each packet reaches the queue head in a different amount of time. Second, there are multiple queues and the scheduler is jumping between them. Each time the scheduler jumps from serving the RT queue to serving the other queues, the speed of removal from the queue varies, which introduces jitter. Obviously, the impact of such scheduler jumps is minimized as the number of queues decreases. Another approach to reduce jitter is to give the RT queue a huge weight in terms of the PW-DWRR algorithm, to ensure that in – contract traffic from other classes of service does not hold up the scheduler too often and for too much time. However, a balance must be found regarding how much the RT traffic should benefit at the expense of impacting the other traffic types.
So in a nutshell, there are tactics to minimize the presence of jitter, but making an accurate prediction requires input from the network operation, because the average queue fill levels depend on the network and its traffic patterns.
The guideline for the other queues is that the amount of interface buffer allocated to the queue length should be proportional to the value of the assigned transmit rate. So, e.g., if the BU queue has a transmit rate of 30% of the interface bandwidth, the BU queue length should be equal to 30% of the total amount of buffer available on the interface. This logic can be applied to queues that carry customer traffic, as well as to queues that carry the network internal traffic such as the CNTR traffic, as illustrated in Figure 9.16. Here, x, x1, x2, and x3 are variables that are defined according to the amount of traffic of each class of service that is expected on the interface to which the queuing and scheduling are applied.
Figure 9.16 Queuing and scheduling at core-facing interfaces
The DA queue has both green and yellow traffic, so a WRED profile is applied to protect the queuing resources for green traffic.
A special case is the BE queue, because its transmit rate increases and decreases over time, depending on the amount of bandwidth available when other classes of service are not transmitting. Because of the possible variations in its transmit rate, the BE queue length should be as high as possible, to allow it to store the maximum amount of traffic. The obvious implication is a possible high delay value inserted into BE traffic. However, such a parameter is considered irrelevant for such types of traffic.
Queues and Scheduling at Customer-Facing Interfaces
Some of the dimensioning for queuing and scheduling at a customer-facing interface is the same as for core-facing interfaces. However, there are a few crucial differences.
In terms of granularity, customer-facing interfaces are aware both of the class of service and the customer. For example, on a physical interface with multiple hub and spoke VLANs, the queuing and scheduling need to be done on a per-VLAN basis, because each VLAN represents a different customer.
Another difference is the dimensioning of the transmit rates, because at this point the focus is not the expected traffic load of each class of service as it was on a core-facing interface, but rather the agreement established with the customer regarding how much traffic is hired for each class of service. For the bandwidth sharing, the value to be split across the configured queues is not the interface maximum bandwidth, but is the value of the defined shaping rate. Because the customer-facing interface is the point at which traffic is delivered to the customer, a shaper is established to allow absorption of traffic bursts.In a certain way, the shaper can be seen as guarding the exit of the scheduler.
Also, a customer-facing interface does not handle any CNTR traffic, so queue three is not present.
All other points previously discussed in the core-facing interfaces section, including queue priorities, queue length dimensioning, and applying WRED on the DA queue, are still valid.
Tracing a Packet Through the Network
The previous sections of this topic have presented all the QOS elements that are active in this case study. Now let us demonstrate how they are combined by tracing a packet across the network.
Figure 9.17 Tracing a packet across the network
Figure 9.18 Packet walkthrough across the ingress PE router
We follow the path taken by three packets named RT1, DA1, and DA2 and a fourth packet that arrives with an unknown marking in its User Priority field, which we call UNK. All these packets belong to a hub-and-spoke VLAN, and they cross the network inside a primary LSP established between two PE routers, as illustrated in Figure 9.17.
The four packets are transmitted by the customer and arrive at the ingress PE customer-facing interface, where they are subject to admission control and classification based on their arrival rate and the markings in their User Priority field.
At the ingress PE, we assume that packet RT1 arrives below the maximum rate agreed for RT traffic, so it is admitted into the RT class of service. For the DA packets, when DA2 arrives, it falls into the interval between CIR and PIR so it is colored yellow, and DA1 is colored green. Packet UNK arrives with a marking of 111 in its User Priority field, and because this is outside the range established in Table 9.3, it is classified as BE. All four packets are now classified, so we can move to the core-facing egress interface.
The core-facing interface of the ingress PE router has five queues, four of which carry customer traffic belonging to this and other customers, and the last queue, queue three, carries the network’s own control traffic, as illustrated in Figure 9.18, which again shows CNTR packets as black triangles. Also, we are assuming that there is no BU traffic, so queue two is empty.
The RT packet is placed in queue four, both DA1 and DA2 are placed in queue one, and UNK is placed in queue zero, following the class of service-to-queue mapping established in Table 9.2 . How long it takes for the packets to leave their queues depends on the operation of the PB- DWRR algorithm, which considers the queue priorities and transmit rates.
Figure 9.19 Packet walkthrough across the P router
Figure 9.20 WRED operation
For packet forwarding at this PE router, the customer traffic is encapsulated inside MPLS and placed inside an LSP, which crosses several P routers in the middle of the network and terminates at the egress PE.
An EXP rewrite rule is applied to ensure that traffic leaving this PE router has the correct EXP marking, as defined in Table 9.3, because those same EXP markings are the parameters evaluated by the ingress core-facing interface on the next P router crossed by the LSP.
At any of the P routers between the ingress and egress PE routers, the behavior is identical. At ingress, the classifier inspects the packets’ EXP markings and assigns the packets to the proper classes of service. It should be pointed out that P routers understand the difference in terms of color between packets DA1 and DA2, because these packets arrive with different EXP values, another goal accomplished by using the EXP rewrite rule. The network control traffic is classified based on its DSCP field, as illustrated in Figure 9.19 . The P router’s egress interface needs to have queuing and scheduling and an EXP rewrite rule.
As previously mentioned, WRED is active on queue one, to protect queuing resources for green traffic. Let us now illustrate how that resources protection works by considering the scenario illustrated in Figure 9.20.
In Figure 9.20, when the fill level of queue one goes above 50%, the drop probability for any newly arrived DA yellow packet (represented as inverted triangles) as established by the WRED profile is 100, meaning that they are all dropped. This implies that queue one can be seen as having a greater length for green packets than for yellow packets. Following this logic, queue one is effectively double the size for green packets than for yellow ones. So on any router crossed by packets DA1 and DA2, if the fill level of queue number one is more than 50%, DA2 is discarded. This packet drop should not be viewed negatively; it is the price to pay for complying with the CIR rate.
Figure 9.21 Packet passage through the egress PE router
The UNK packet is placed in queue zero, because it was considered as BE by the ingress PE, but it contains the same marking of 111 in the User Priority field that it had when it arrived at the ingress PE. This has no impact because all the QOS decisions are made based on the EXP field, so the value in the User Priority field of the packet is irrelevant.
The final network point to analyze is the egress PE router. At this router, traffic arrives at the ingress core-facing interface encapsulated as MPLS and is delivered to the customer on an Ethernet VLAN, as illustrated in Figure 9.21.
Because it is a customer-facing interface, there is no CNTR traffic, so queue three is not present.
Packet DA2 is not present at the egress PE router, because it was dropped by the WRED profile operation on one of the routers previously crossed.
When the customer traffic is delivered on an Ethernet VLAN, the markings present in the User Priority field once again become relevant. A rewrite rule on the customer-facing egress interface must be present to ensure that the User Priority field in the UNK packet is rewritten to zero. This step indicates to the customer network the fact that the ingress PE router placed this packet in the BE class of service.
The previous figures have shown packets arriving and departing simultaneously. This was done for ease of understanding. However, it is not what happens in reality because it ignores the scheduling operation, which benefits some queues at the expense of penalizing others. The time difference between packets arriving at an ingress interface depends on the PB-DWRR operation preformed at the previous egress interface. Because of the queue characteristics, we can assume that packet RT1 arrives first, followed by DA1 and DA2, and only then does packet UNK arrive.
Although both queues one and zero have the same priority value of low, it is expected that packets in queue one are dequeued faster, because this queue has an assured transmit rate. Queue zero has a transmit rate that varies according to the amount of free bandwidth available when other queues are not transmitting.
Adding More Services
So far, this case study has considered a mapping of one type of service per each class of service, which has made the case study easier to follow. However, as discussed throughout this topic, this is not a scalable solution. The key to a scalable QOS deployment is minimizing the behavior aggregates implemented inside the network and to have traffic belonging to several services reuse them.
So let us analyze the impact of adding a new service that we call L3, which is delivered over a Layer 3 VPN. We consider that its requirements are equal to those previously defined for the RT class of service. Implementing this new service requires a change in the technology used to provide connectivity between customer sites, which has several implications for network routing and VPN setup. However, the pertinent question is what of the QOS deployment presented so far needs to change.
From the perspective of a P router, the changes are very minimal. Packets belonging to RT and L3 arrive at the P router ingress core interface inside MPLS tunnels. So as long as the EXP markings of packets belonging to L3 and RT are the same, the behavior applied to both is also the same. The EXP markings could also be different and still be mapped to the same queue. However, there is no gain in doing this, and in fact it would be a disadvantage, because we would be consuming two out of the eight possible different EXP markings to identify two types of traffic that require the same behavior.
As a side note, the only possible change that might be required is adjusting the transmit rate of queue four, because this new service may lead to more traffic belonging to that class of service crossing this router.
On the PE router, the set of tools used remains the same. The changes happen in whichever fields are evaluated by the QOS tools, such as the classifier. As previously discussed, traffic belonging to the RT class of service is identified by its User Priority marking. So if L3 traffic is identified, for example, by its DSCP field, the only change is to apply a different classifier according to the type of service, as illustrated in Figure 9.22 .
fn this scenario, the integration of the new service is straightforward, because the requirements can be mapped to a behavior aggregate that is already implemented.
However, let us now assume that the new service has requirements for which no exact match can be found in the behavior aggregates that already exist in the network.
The obvious easy answer for such scenarios is to simply create a new behavior aggregate each time a service with new requirements needs to be introduced. However, this approach is a not scalable solution and causes various problems in the long run, such as exhaustion of EXP values and an increase in jitter because the scheduler has to jump between a greater number of queues.
Figure 9.22 Adding a new L3VPN service
There are several possible solutions. One example is use a behavior aggregate that has better QOS assurances. Another is to mix traffic with different but similar requirements in the same queue and then differentiate between them inside the queue by using the QOS toolkit.
Previously in this case study, we illustrated that using WRED profiles inside queue one allows DA green and DA yellow traffic to have access to different queue lengths, even though both are mapped to the same queue.
The same type of differentiation can be made for the queue transmit rate. For example, we can limit the amount of the queue transmit rate that is available for DA yellow traffic. This can be achieved by applying a policer in the egress direction and before the queuing stage.
A key point that cannot be underestimated is the need for network planning for the services that need to be supported both currently and in the future. It is only with such analyses that synergies between different services can be found that allow the creation of the minimal possible number of behavior aggregates within the network.
Multicast Traffic
Services that use multicast traffic have increased in the past few years, both in the enterprise and in the home user space, and the expectation is for such growth to continue in the future. So let us analyze the implications of introducing multicast to the QOS deployment presented so far, by assuming that the BU class of service carries multicast traffic.
The set of QOS tools that are applied to BU traffic are effectively blind regarding whether a packet is unicast or multicast, or even broadcast. The only relevant information are the packets’ QOS markings and their arrival rate at the ingress PE router. However, the total amount of traffic that is present in the BU class of service affects the dimensioning of the length and transmit rate of queue number two, into which the BU traffic is placed. The result is that there is a trade-off between the queuing resources required by queue number two and the effectiveness of how multicast traffic is delivered across the network.
Figure 9.23 Delivery of multicast traffic using P2MP LSPs
As a side note, it is possible for classifiers to have a granularity level that allows the determination of whether a packet is unicast or multicast, so that different classification rules can be applied to each traffic type, as needed.
Let us now analyze how to effectively deliver multicast traffic across the network in this case study. Within the MPLS realm, the most effective way to deliver multicast traffic is by using a point-to-multipoint (P2MP) LSP, which has one source and multiple destinations. The efficiency gain is achieved by the fact that if a packet is sent to multiple sources, only a single copy of the packet is placed in the wire. Figure 9.23 illustrates this behavior. In this scenario, a source connected to PE1 wishes to send multicast traffic towards two destinations, D1 and D2, that are connected to PE2 and PE3.
The source sends a single multicast packet X, and a single copy of that packet is placed on the wire inside the P2MP LSP. When a branching node is reached, represented by router B in Figure 9.23 . a copy of packet X is sent to PE2 and another copy is sent to PE3. However, only one copy of packet X is ever present on the links between any two network routers. Achieving such efficiency in the delivery of multicast traffic translates into having no impact on the QOS deployment presented thus far. In a nutshell, the better the delivery of multicast traffic across the network is, the less impact it has on the QOS deployment.
If P2MPLSP were not used, the source would have to send two copies of packet X to PE1, which would double the bandwidth utilization. As such, the transmit rate requirements of queue two would increase and it would need a larger queue length. As a side note, P2MP LSPs are not the only possible way to deliver multicast traffic. It was chosen for this example because of its effectiveness and simplicity.
Using Bandwidth Reservations
So far, we have considered one LSP carrying all the traffic between two PE routers. We now expand this scenario by adding bandwidth reservations and considering that the LSPs are aware of the class of service.
Figure 9.24 Using LSPs with bandwidth reservations and ERO
In this new setup, traffic belonging to each specific class of service is mapped onto a unique LSP created between the two PE routers on which the service end points are located, and these LSPs contain a bandwidth reservation, as illustrated in Figure 9.24 . Also, the LSPs maintain 100% path control because they use strict ERO, as mentioned previously in this case study.
Establishing LSPs with bandwidth assurance narrows the competition for bandwidth resources, because the bandwidth value reserved in an LSP is accessible only to the class of service traffic that is mapped onto that LSP. However, this design still requires admission control to ensure that at the ingress PE router, the amount of traffic placed into that LSP does not exceed the bandwidth reservation value.
Bandwidth reservation in combination with the ERO allows traffic to follow different paths inside the network, depending on their class of service. For example, an LSP carrying RT traffic can avoid a satellite link, but one carrying BE traffic can use the satellite link.
So using bandwidth reservations offers significant gains for granularity and control. But let us now consider the drawbacks. First, there is an obvious increase in the complexity or network management and operation. More LSPs are required, and managing the link bandwidth to accommodate the reservations may become a challenge on its own.
Typically, networks links have a certain level of oversubscription, following the logic that not all customers transmit their services at full rate all at the same time. This can conflict with the need to set a strict value for the LSP’s bandwidth.
Also, how to provision and plan the link bandwidth to account for possible link failure scenarios becomes more complex, because instead of monitoring links utilization, it is necessary to plan the different LSP’s priorities in terms of setup and preemption. So as with many things inside the QOS realm, the pros and cons need to be weighed up to determine whether the gains in control and granularity are worth the effort of establishing bandwidth reservations.
Conclusion
This case study has illustrated an end-to-end QOS deployment and how it interacts with other network processes such as routing. The directions we have decided to take at each step should not be seen as mandatory; indeed, they are one of many possible directions. As we have highlighted, the choice of which features to use depends on the end goal, and each feature has pros and cons that must be weighed. For example, using LSP with bandwidth reservations has a set of advantages, but also some inherent drawbacks.
