I. Introduction

Since the very beginnings of the telecommunications industry, Inter-provider communication has been a necessity for every telecommunication provider throughout the world. The importance of connecting users located in any nation or region forces service providers to be interconnected in such a way that any user can communicate with another.

This paper aims to present a traffic model for internetwork and inter-provider MPLS (Multi-Protocol Label Switching) networks, using their characteristics and advantages. MPLS technology requires agreements to be made between providers that cover conditions and necessities for any specific situation ^[1,2].

MPLS and its extensions for traffic engineering, MPLS-TE ^[3,4], include the following features and requirements that network designers must deal with:

Section 3 focuses on bandwidth and queue management for different Classes of Traffic (items 1, 3 and 4 above) because these features of MPLS are the most closely related to traffic.

Guichard et al. ^[14] presented a hypothetical example of a global operator providing Internet, voice, data, layer 3 VPN, and other services. This operator, known as Globenet, provides telecommunication services in several regions of the world through the interconnection of its Points of Peering (POP).

Globenet made agreements of interconnection with other regional providers, partly aimed at reaching more customers with less investment in infrastructure, but mostly to make communication possible between clients in separate regions (for instance, from Europe to Africa). In Africa, Globenet previously had only two POP: one in Algeria and the other in South Africa; thus, to connect customers in Europe with those in South Africa, they used a leased line from Lisbon to Johannesburg. However, this was expensive and not particularly efficient, and as a result, Globenet made an agreement with the African Provider ATC, involving using ATC's network to reach its customers in different parts of Africa. Globenet then installed a Virtual POP (VPOP) in Johannesburg, with the equipment located in ATC's buildings. ATC also has POPs in Paris and Frankfurt to facilitate interconnection with Globenet in Europe (Fig. 1).

Fig. 1 Globenet and ATC interconnection ^[14]. 

In ^[14], for means of illustration, the authors start from the premise that ATC does not use its EF queues, instead disposing of them and their assigned bandwidth for the transportation of Globenet's traffic as a Virtual Leased Line (VLL). One very important aspect of this agreement is that Globenet EF, AF and DF traffic is transported by ATC in its network as EF traffic; Globenet's traffic must therefore be marked in such a way that it is delivered to its VPOP in Johannesburg classified just as it was before entering ATC's network. In this way, customers can enjoy the standards outlined in the Globenet QoS policy.

Considering all of the above information, the following questions must be answered: It is uncommon for a provider to choose not to use its EF queues when disposing of its network for another provider's traffic? What type of queue management must this operator thus perform in order to use its EF queues for transporting, both its own traffic and that of another provider? What must this operator do if it decides to transport both sets of traffic in the AF or DF queues?

Several studies have defined optimization models and algorithms employed for different interconnection purposes. In addition to that discussed earlier in this section, it is also important to mention the work carried out in ^[15], which presents an interconnection model that uses a PCE-based interlayer technique to compute shortest paths based on GMPLS Traffic Engineering methods. Although it does not focus on path computation between providers or domains, the model considers path computation between layers rather than network nodes, which is especially useful for systems that combine many different layer technologies.

The model presented in the following section is based on work outlined in ^[16], a flow-based model designed for queue balancing in LSR. This model can adapt to traffic assignment when QoS applies, and was also employed in the model discussed in Section II by taking into account traffic from another provider borrowing the network. In ^[17], a similar model for queue balancing is proposed, but instead it is applied over wireless mesh networks. Simulation results for this model revealed stable queue sizes at low congestion levels, after applying the proposed optimization mechanisms.

Reference ^[18] presents a queuing network model developed primarily for inter-area PCE based on shortest path computation, but which can also be adapted to inter-domain shortest path computation. This work is significant because it focuses on traffic generated by request-response messages for path computation. More recent work presented in ^[19] includes a procedure that allows the multiple discovery of inter-domain sequences and path computation over each sequence. Although its perspective is similar to that of ^[18,19] focuses on proposing more efficient and short path algorithms, and inter-domain loop avoidance, rather than on a mathematical model as is the case in the present work. Similar results are outlined in ^[20], in which a tree schema is defined for the computation of the shortest point-to-multipoint (P2MP) paths. Computation starts in a PCE in the destination domain, with each PCE determining the number of paths downstream, and providing information to upstream PCE, until some PCE in the initial domain complete the whole tree. A very similar approach is undertaken in ^[18], in which Virtual Shortest Path Trees (VSPT) are shared between PCEs in computing end to end TE-LSPs, without sharing topology information between domains. Reference ^[20] is also not model-focused, but is centered on mechanisms and algorithms aimed at optimal shortest path computation. The same is true for ^[21], in which different procedures are proposed with the same aim of optimizing inter-domain path computation.

The mathematical model described in Section 3 represents a reference starting point for the elaboration of routing mechanisms and optimization algorithms as in ^[19-21].

Because the models presented in ^[16] were not validated, Section 3 also includes the results of model validation carried out using computational tools, starting from the configuration specifications defined for Globenet in ^[14].

II. Traffic model for interprovider agreements

A. Flow-based model for dynamic queue balancing

Due to the sharing of bandwidth in the borrowed network by both operators, queues must fulfill certain conditions such that link and node (servers) congestion avoidance may be accomplished.

In an MPLS context, nodes are referenced as Label Switched Routers or LSRs, and are responsible for packet forwarding between ingress and egress routers. For a TE-LSP computed between a head-end LSR and a tail-end LSR, all of the LSRs in the path are responsible for serving the queues.

A few modifications must be made to the work detailed in ^[16] in order to achieve the conditions outlined above regarding the Globenet study case.

Let fi (i=1,..., M) represent the i-th type of traffic intensity of provider 1 (user of the borrowed network) incoming into a node, and gj (j=1,., N) represent the j-th type of traffic intensity of provider 2 (owner of the borrowed network) incoming into the same node as fi. Let rj (j=1,..., N) be the bandwidth assigned to the j-th queue, and r the bandwidth assigned to the network segment in which the node is located.

The following two conditions must be fulfilled:

Condition (1) assures that queue's assigned bandwidth is never greater than the total outgoing bandwidth of the node, thereby avoiding overload. Condition (2) assures that the traffic intensity generated by both providers is never greater than the bandwidth assigned to all traffic (i.e., all queues).

Normally, RED and WRED drop packets when (2) is not achieved, as these algorithms are designed for congestion avoidance. The bandwidth assigned r is taken by RED and WRED to be the maximum threshold, above which all packets are dropped.

In the case of Globenet, five Classes of Service (CoS), which are transported in five queues, are defined. Let us suppose that the African operator ATC has four queues. Globenet's traffic must still be transported self-defined in the queues of ATC (in all four queues or a selection), labeled in such a way that traffic arrives well classified at the VPOP. This is the reason why sub index j is used for both the traffic types of provider 2 (g_j), and for the assigned queue bandwidth (r_j).

In addition, according to ^[16], it is necessary to define a control variable vij for dynamic queue management:

This function must be a matrix of ones and zeros, with the ones placed in the column corresponding to the queue for each Globenet CoS; consequently, the following condition must be fulfilled:

Then, the traffic intensity can be defined as follows:

Condition (4) relates to the prevention of packet loss, while condition (5) is aimed at preventing the traffic intensity for both providers in any queue from being less than the bandwidth assigned to that queue. The left-hand side of (5) is equivalent to the average rate of arrivals ( in the j-th queue. For the EF, AF4 and AF3 queues, the traffic intensity may be greater than the respective assigned bandwidth, because in these cases part of the total bandwidth will be used (Russian Doll Model for Bandwidth Constraints) to assure that these queues' traffic arrives complete at their destination, due to their importance. For the DF queue, the traffic intensity must be less than or equal to (never bigger) the assigned bandwidth, otherwise dropping mechanisms (RED) will have to be actioned.

Finally, according to ^[16], the mean value of waiting arrivals (i.e., queue size) must be lower than the incoming buffer (queue maximum size before packet dropping starts).

After some operations on both sides of (6), the formula can be expressed in terms of maximum delay time (applying classic teletraffic theory ^[7] for an M/M/1 system):

In this way, an expression of maximum time delay that represents a fundamental performance metric is found. This expression is a function of queue size, server utilization and traffic intensity arriving from both providers to the node, and it is very similar to one defined in ^[16], with the difference being that the former considers traffic from another provider.

It is important to remember that RED computes the average queue size periodically from the behavior of the arriving traffic, using the following expression defined in ^[22]:

III. Model validation

In order to validate the presented model, the Wolfram Mathematica 10.1 Student Version software tool was used, in particular to check the raised equations, based on network configurations outlined in ^[14].

In the study case presented in Section 1, Globenet defines five Classes of Service (CoS) that are carried in five queues (EF, AF4, AF3, AF2 and DF) in case of congestion, with the African provider ATC defining four CoS (EF, AF4, AF3 and DF). Although all traffic must be carried in the four African CoS, Globenet data must be labeled such that when this traffic arrives at the VPOP, it can be differentiated in terms of the five queues defined by Globenet. This is why in expressions (1) through (7), index j is used for both r (bandwidth) and g (traffic of another provider).

It is important that Globenet's traffic be carried by the African provider in queues, which ensure the former's QoS. For example, although Globenet's EF traffic cannot be carried in ATC's AF or DF queues, the opposite is valid (i.e., Globenet's AF or DF traffic can be carried in ATC's EF queue). Therefore, in the best case, ATC must carry Globenet's EF traffic in its own EF queues.

To achieve a smaller Utilization factor, the network provider must first increase the serving rate in P routers, such that Utilization remains below 80 %, and the Average Queue Time varies at close to 50 milliseconds.

The provider's network designers must also define the maximum time delay in router P, such that the addition of hop-by-hop delay does not increase point-to-point delay in a way that affects QoS.

In the case of the EF queue, Table 1 shows the queuing system after raising the Service Rate to 80.3 Mbps (38.44 %), and passing random flows through it.

Table 1 Metric results after raising service rate 

Queue length variation during one minute of system operation is shown in Fig. 2:

Fig. 2 Queue Length variation during a period of one minute with Utilization p=76.46 %. 

Figure 2 shows a substantial improvement in Queue Length behavior over time. The maximum length is around 25 Mb, but the average value is much lower at around 5 Mb. This will be reflected in a maximum time delay of 50 milliseconds.

Figure 3 displays the Time Delay variation over 10 samples after increasing the Service Rate in P routers. As indicated in Section 2, the proposed model is applied by defining a maximum value for Time Delay, testing several traffic flows, and verifying expressions (5) and (7). When a successful result is not achieved, congestion control mechanisms (Priority Queueing PQ or Class-Based Weighted Fair Queueing for EF and AF traffic), and dropping mechanisms (RED or WRED for DF queues) are activated.

Fig. 3 Improvement in Time Delay after increasing the Service Rate in P routers. 

IV. Discussion

Neither ^[16] nor ^[18] include performance testing of the presented models. Furthermore, the authors of ^[16] did not carry out any simulation testing either; instead they aimed to present hypothetical cases, and thus defined limits to the values of Globenet's network attributes.

For instance, Globenet ^[14] defines limits for both Waiting Queue Time and Queue Length in P routers aimed at avoiding buffer hogging by one queue. Specifically, a maximum Queue Length is set of up to 3060 packets, with a packet length of 76 bytes (about 1.86 Mbits), and maximum System Time of 30 milliseconds.

For non-EF queues, the maximum queue length is 3875 packets, with a packet length of 250 bytes (around 7.75 Mbits). As mentioned at the end of Section 4, the definition of such limits is important for successfully applying the proposed model. In order to achieve the limits set by Globenet, utilization must decrease to below approximately 60 %, which implies an increase in the Service Rate of P routers to above 115 Mbps. Although this Service Rate is high, CISCO has developed technology such as TYPHOON, TOMAHAWK and TRIDENT with higher processing capacities that decrease the Service Time sufficiently to reduce the System Time almost to zero.

It is important to bear in mind that the Mathematica 10.1 software tool only demonstrates the equation behavior when variables take some random values. However, this is not a simulation, and therefore, the results obtained using simulation tools such as OPNET or NS3 may differ when certain details of network components' operations are taking into account. Nevertheless, although the results presented here are limited by the general computation methods of the employed mathematical tool, they are useful for the initial analysis design, and the applicability of the presented model.

The model also represents a starting point for users aiming to size the Point-to-Point delay. Section 2.1 reviews only Time Delay for a P router, not Point-to-Point, which could minimally raise the simple addition of delay in every router used in TE-LSP. Similarly, Section 2.2 presents an expression for the Time Delay produced by the interexchange of RSVP messages when computing the shortest path, but not that produced by traffic flowing through the previously established TE-LSP. Future work should thus involve modeling the Pointto-Point delay generated during the borrowing of one provider's network by another. In such situations, Traffic Engineering techniques other than DS-TE should be employed to solve the associated problems, but using methods such as bandwidth reservation inside TE-LSP, and preemption values (preemption is an attribute of MPLS-TE that gives priority to traffic according to their preemption value ^[4].

Several researchers have developed on the above idea. Although ^[23] and ^[24] do not evaluate Time Delay behavior, they combine class-based bandwidth constraints with the dynamic assignation of preemption values.

The bandwidth management model presented in ^[23] differs from RDM (Russian Doll Model) and MAM (Maximum Allocation Model) methods in not only defining a maximum bandwidth value per CoS, but also a minimum bandwidth value for certain CoS, such that other service traffic has no effect on the increase or decrease in the overall traffic intensity. Shan and Yang ^[23] also proposed an algorithm to manage allocated bandwidth based on preemption values.

Although the work detailed in ^[24] is similar to that mentioned above, the algorithm employed for bandwidth management differs in that it sizes both IP Transfer Delay (IPTD) and IP Delay Variation (IPDV), and considerably delays that increase when links utilization is above 80 % -in this paper, link utilization is almost 100 %-. The work presented in ^[24] also extends its algorithm to an inter-domain environment (without simulation results), defining a "Bandwidth Broker (BB)" as an entity that manages internal and external information regarding available resources both in its domain and in adjacent ones, interchanging it with another domain's BBs.

Finally, the authors of ^[25] proposed an algorithm for queue management (mainly queue length control), based on prediction of changes (i.e., an increase or decrease) in the packet arrival rate. This algorithm can achieve great queue length stability around a desired value, despite the increase in traffic or even in the Round Trip Propagation Time (RTPT), and as such it could be used in combination with the model presented in Section 2 to control the waiting time of packets in routers.

V. Conclusions

In this paper, a traffic model for provider interconnection that models the queue management necessary for one provider to carry another's traffic was presented. Additionally, the mechanisms that must be applied if this process generates a congestion condition were described, based on values of Queue Length and Time Delay, following ^[7]. After defining a maximum value of Time Delay, congestion control mechanisms manipulate the queue length and other attributes to assure compliance with ^[7]. An M/M/1 queueing system was implemented in Wolfram Mathematica 10.1 Study Version, following (5) and (7), in which the server is a P router acting as an LSR.

The initial results show that Time Delay and Queue Length increase to unacceptable values for QoS requirements when Utilization is around 98 %. In order to reduce the Time Delay, the provider must increase the Service Rate of P routers sufficiently to set Utilization below 60 %, and thereby satisfy Globenet's constraints regarding Queue Length and Time Delay.

Future work should involve validating the presented model via simulation or experimentation, as well as extending the model to the other aspects of MPLS-TE listed in Section II.A that are not covered in the present paper.