A Practical Navigator for the Internet Economy

A double issue on


Economics of IP Network Interconnection: Peering, Transit, and IXs:




Changing Role of Peering and Transit in IP Network Interconnection Economics

Development of Bandwidth Cost Management Capability May Be Critical to ISP Survival

Coping with the Internet Core Oligopoly Demands Innovation A Synthetic Path Analysis Examination of Netflow Data . . . pp. 1 - 5

Contributors to this Issue . . . p. 6


Why Peering and Transit Is Badly Broken at the Global Tier One Level

How the Process Evolved into the Oligopoly of the I-Core by Farooq Hussain . . . pp. 7 - 10


Bill Woodcock Explains New Framework on Which to Build Peering and Transit

Two Interviews. . . pp. 11 - 27


A Mini Encyclopedia of the Economics, Politics and Technology of Internet Interconnections -- Our Experts Discuss Architecture, Traffic Flows, Transit and Bandwidth Costs, as Well as Market Economics

Participant's Introductions . . . . p. 28

Use of Looking Glasses and Other Peering - Traffic Related Strategy . . . p. 37

Exchanges . . . p. 39

LINX . . . p. 40

Transit in Wales . . . p. 42

Complexities of Peering . . . p. 42

Transit Pricing . . . p. 43

The Netflow Methodology . . . p. 44

Global Peering and Transit Issues . . . p. 46

Woodcock's Synthetic Path Analysis . . . p. 48

Economy of Scale . . . p. 47

Quality of Service . . . p. 49

Exchange Economics for Colorado Springs . . . p. 50

Andrew Odlyzko's Evaluation of the Peering World, . . . p. 51

Economics of Maintaining Capacity . . . p. 52

Does Peering Save Money over Transit? . . . p. 53

Costs Move from the Center to the Edges . . . p. 56

Tier 1 Versus Transit Free . . . p. 57

Economics of Level 3's European Connectivity . . . p. 59

Traffic Patterns - How Much is Local? . . . p. 60

Transit Accounting and Billing Issues . . . p. 63

Pricing and Backbones - Where's the Industry Headed? . . . p. 64

One Hundred Dollar Pricing Floors? . . . p. 64

Who Dominates - Cable or DSL? . . . p. 65

Adlex and Other Tools . . . p. 68

Traffic and Backbone Quality . . . p. 69

Avoiding Backbones? . . . p. 70

National Fiber Backbone? . . . p. 70

Economics and Policy Issues of Commercial Backbones . . . p. 72

Is There a Backbone Problem? . . . p. 74

A New Fiber Project in Holland . . . p. 78

Do ATM based Exchanges Make Sense Anymore? - Bill Norton on NANOG . . . p. 80- 83


Lack of Broadband Infrastructure Now a Bottleneck Holding Back IT industry

13 Micron Technology as Part of a Hardware Revolution that Brings PC Economics to Telco Switching . . . pp. 84- 88


David Reed - Its the Latency Stupid . . . p. 89

ICANN's Season of Delusions: Attempt to Spin Court Defeat is Rebuffed in IETF List

Froomkin Drives Sims, Cohen and Cerf To Distraction Summary of Froomkin's "Form and Substance" Paper Froomkin and Malamud Reaction to .Org Decision . . . pp. 90 - 100


RIAA Runs Amok, Sues Four Backbones Over Chinese Site and then Sues Verizon

Old Time Net Architect Makes Mistake with Affidavit in Support of First RIAA Suit . . . pp. 101 - 105



. . . pp. 106-116

Executive Summary

. . . pp. 117-118


Changing Role of Peering and Transit in IP Network Interconnection Economics


Development of Bandwidth Cost Management Capability May Be Critical to ISP Survival


Coping with the Internet Core Oligopoly Demands Innovation

A Synthetic Path Analysis Examination of Netflow Data

..... pp. 1 - 5



This issue of the COOK Report explores IP network interconnection (peering, transit and exchanges) for the first time since about 1999.

While a lot has changed, a lot remains the same. Peering and transit is just as obscure and complex, even to other network professionals, as it ever was. "That is such a black art," said Richard Shockey co-chair of the IETF ENUM working group, when we told him what we were doing. Nevertheless, though a definitive treatise on the state-of-the-art in peering and transit is very likely impossible, we believe that we have thrown considerable light on the subject with the information that we publish in this combined issue.

We have found that the technology, politics, and economics of IP network interconnection differ significantly -- depending on the size of network and geography involved. In North America the state of interconnection is both more evolved and more complex than it is in other global regions. While our contributors to this issue cover the globe, we shall focus in most detail on the structure and evolution in North America.

In North America the Tier 1's oligopoly of peering only with themselves is still well entrenched. Farooq Hussain has written for this issue a remarkably candid summary of the evolution of the Tier 1's peering policy. They are, he says, the Internet Core Networks that announced anonymously on December 5, 2001 their decision to move their peering to Equinix Exchanges. He identifies them as UUNET, Sprint, Cable and Wireless, Genuity, Level 3, Qwest, and AT&T. He also finds their peering requirements to be arbitrary beyond reason. For example, interconnection at OC48 is one thing, but to be forced to do so at 15 locations around the United States is something else again.

Estimates of the capacity utilization of the Tier 1 backbones show them to be lightly utilized at about 15 to 20%. Given this situation Sprint, for example, is undoubtedly quite happy to have SBC buying nine OC-48s. Because the ISP and backbone industry is unregulated, what knowledge we have is sketchy and largely subject to the willingness of folk who both know, and will take the risks of speaking up. Given the state of the industry such folk are few and far between.

Over the past eight weeks, to generate the material for this issue, we have had conversations on a private mail list with more than 30 people who are closely involved in Internet exchanges, peering and transit. Some of these folk have suggested in voice conversation that the oligopoly is engaging in behavior that could blow up ways that would be very embarrassing for the industry. [For the moment however this is only speculation. See for example the sidebar exchange between Miles Fidelman and Sean Donelan on page 78 below.] Whatever happens, peering and transit arrangements have major impacts on the economics of ISP operation. Real money is involved and with real money come power struggles. Certainly, with all seven loosing money and four of the seven (UUNET, Genuity, Qwest and Level 3) either in bankruptcy or in dire financial difficulty, behavior at the Internet Core is not likely to become customer friendly.

Folk associated with the carriers complain that long haul transit backbones are being forced to sell transit essentially below their cost and suggest that something about the system has to change. Of course since almost everything is kept under strict non-disclosure and since these seven players are also either telephone companies, or associated with telephone companies, the picture becomes even more murky due to opportunities for subsidizing operations from the telco side of the house.

In the midst of this uncertainty, the United States government has defined the Internet as a critical telecommunications infrastructure. Yet, one of many ironies of the current situation is that virtually no one in the federal government nor in the financial community knows with accuracy how strained the financial position of each company is. In Europe with the demise of KPN Qwest this spring we saw how quickly a major player can disappear.

But there are mitigating factors to be found in the uncertainty. Thousands of small ISPs survive by dint of hard work and because of what we contend are, for them, favorable economies of scale. They are defining analytical tools that we discuss in detail in this issue. They are weaving their positions at the edge of the network into a mesh of cooperation that is likely to provide resiliency for the internet as a whole as shockwaves from the collapse at the center propagate.

Consequently, the lack of detailed knowledge about the economic condition players at the center may not be as problematic as we had believed. Discussion with the 25 contributors to this issue (listed on page 6) has shown us that by the time the US and other governments collected the kinds of data that the telcos are required to submit data on ISP interconnection and transit economics would be out of date.

Ascendancy of Current Tier 1s Being Challenged

The tectonic plates of network traffic and power are shifting with the economic uncertainty brought on by the industry crash and the increase of cable modem and DSL traffic. Given the extraordinarily low cost of bandwidth and the existing investments of US carriers in some of the fiber players, we can expect very soon to see a build out on the part of these carriers into peering at Asian and European exchanges. In this sense a lot of effort will be put by large players into moves to enable them to avoid paying transit fees to the currently seven largest global backbones (Tier 1). In doing so, the likely outcome is that these new comers will eventually either replace or join the Tier 1 oligopoly.

They are, in effect, climbing a peering "ladder" where as their bandwidth increases and they peer with each other and can get peering with larger players, they are likely to depeer with smaller players whom they feel they now no longer need and believe they can sell transit to. Thus although the plates of peering are shifting, the fundamental premise is likely to remain one where players peer only if their aggregate traffic is approximately equal. However peering is motivated as much by politics as economics. Therefore, it would be a mistake to think that these premises apply with equal force the world over.

Bill Woodcock has shown that a case can be made for the position that it makes sense for a larger network to accept traffic from a smaller network that terminates on the larger network. However the larger players are still firmly of the opinion that size differences in network traffic are there to be exploited by the larger and presumably more powerful network. Not surprisingly this view is firmly rejected by the smaller players. Therefore as the new broadband based networks move to extend their peering infrastructure around the US and across oceans, they are likely to act increasingly like the Tier 1 oligopoly they want to replace and seek to sell transit to rather than peer with others who haven't grown as fast.

What is unknown is how good a job how many smaller players can do of extending peering with other small players through use of the approaches and methodology outlined by Woodcock in the long interview in this issue. At an abstract level Woodcock's views that peering is a good thing seem to be understood and accepted by the large players as well. The problem for them seems to be one of "good for whom?" along with the belief that, once you get to a certain size (and we might add business model), they don't scale.

Therefore as the new broadband based networks move to extend their peering infrastructure around the US and across oceans, they are likely to act increasingly like the Tier 1 oligopoly they want to replace and seek to sell transit to rather than peer with others who haven't grown as fast.

Although the make up of the Tier 1 backbones may change, the existence of a handful of global networks at the top of global IP transit food chain is not likely to change in a serious way. Perhaps the most significant unknowns are how large and sustainable the Tier 2 doughnut around the Tier 1 transit providers can become. Also it is uncertain whether the benefits derived from joining the Tier 1 club will be great enough for the new entrants to find that the results actually pay back the efforts and expense in infrastructure building necessary to accomplish their goals.

Ren Nowlin reminded us that peering terms are not set in granite. "Future forward crystal balls are vague. Never forget depeering is a variable in business plans. Some peers are migrated for free and others for a fee when networks evolve and move out of IXs depending on perceived value at that stage by the two parties. SBC does not plan to depeer today but Level 3 didn't either at this stage of network development. Peering agreements, and terms, are not static."

Players like SBC and Level 3 have to contend with billions of dollars of debt. Consequently it seems very likely that whenever management believes that depeering would be likely to produce additional income, depeering will be likely to occur.

It's the Economics of Networks, Stupid!

With the industry collapse, the tightly run economic aspects of one's business matter in a way that they did not during the bubble when money flowed freely. In this context, some important new tools and concepts are being developed in the peering and transit arena. In addition to Farooq's examination of the current global climate at the Tier 1 level, this issue looks at the development of some tools and methodologies to manage, much more cost effectively, an ISP's interconnection costs.

Since about 1998 Cisco (and now Juniper) routers have had the capability of giving the users what is called Netflow data. Use of the Netflow data can give significant information about where a network's traffic is going, including what autonomous systems the ISP's traffic flows through to reach its customers. Various efforts are underway. A small handful of ISPs of differing sizes are beginning to use tools by companies like Route Science and SockEye to do load balancing of their up stream connections in real time. Some folk are also beginning to build tools using Netflow data to help them most cost effectively analyze how to do their transit and peering in the first place. One such effort is by Jeffrey Pappen, Peering Coordinator for Yahoo. It is called TUNDRA, The Ultimate Netflow Data Realtime Analysis, and was presented by Pappen at NANOG in October 2001. See www.nanog.org/mtg-0110/ppt/tundra.ppt Another is by Martin van den Nieuwelaar and is called Network Intelligence. See http://www.networkintelligence.biz Stephen Stuart at PAIX has looked at Netflow approaches and applied some of them there.

Another, and likely the most important, effort has been developed by Bill Woodcock of Zocalo.net, and Packet Clearing House and Alex Tudor at Agilent technologies. This is discussed in great detail in a 16,000 word interview with Bill Woodcock on pages 12 to 27of this issue. We note that Woodcock's approach is the subject of a patent application. In the language of the legal department of Agilent. "Some of technology described in this article [our interviews] has been claimed in one or more pending patent applications that are owned jointly by Zocalo and Agilent Technologies, Inc., each of whom is free to license."

On September 2 Bill explained the patent issue more clearly: "The deal between me and Agilent was that I would do a "technology transfer" and tell them what code to write. They wrote the code at my instruction, and we jointly filed a patent application to preclude someone else's making proprietary claims against it. The content of the patent disclosure and application are open source, and free for public use. Again, the purpose of the patent application, is to protect open-source use of the algorithms against future claims that they're someone's proprietary invention."

Finally we note that from, an economic point of view, the most important component of what Woodcock is doing is what he calls Synthetic Path Analysis. That analysis is publicly discussed below for the first time. It has not yet been tested in production use over a period of six months to a year's time. This should be done. Until it is, we can say only that from an intuitive point of view it makes strong sense. Certainly ISPs should be looking at experimenting with it.

We think that what Woodcock says is extremely significant. Among other things, he points out that the Tier Ones, by peering in their tight oligopoly, may have rendered themselves irrelevant. Why? Because the smaller networks with rich peering are beginning to build such a well connected donut around the Tier 1's that one may be on the verge of being able to deliver one's traffic to all of the destinations to which it needs to go without relying on the Tier 1's for transit.

In short, ideas and methodology are evolving. The new topology looks more like a multifaceted geodesic dome than like a dozen global backbones with networks hung off them tree-branching fashion. Critical components of this new topology, in addition to peering and transit circuits, are the hubs into which the circuits are attached. These are the more than 300 exchange points around the world that facilitate network interconnection. There it seems likely that newly cost-conscious ISPs will increasingly deploy Netflow data methodology to model their traffic and decide from the results where and with whom to interconnect. When we described to the technology director of a large CLEC what they do, he said: "that sounds pretty much like how we model and plan our long distance circuit interconnections with the PSTN. Do it right and you are profitable. Do it wrong and you are history."

Of course what is right and what is wrong is likely to be a function of where one is in the industry. Below the Internet Core oligopoly one has a group of very large players who are themselves under stress and therefore must find strategies for survival. This group includes AOL, SBC, Yahoo, Shaw Cable, France Telecom, Equant, Verizon, Bell South and many foreign carriers like Telstra and NTT. See for example http://www.sbcbackbone.net/peering/ They have their own ideas about peering and given cooperation by them we may be able to begin to explore them in the future. (No promises because such cooperation is not yet firmly in place.) Beneath this group are the several thousand smaller ISPs with which Woodcock works. We believe that both these groups could profit from Woodcock's understanding.

Future Direction

Vertically integrated local phone companies are forced to have a business model that squeezes every drop of money from every drop of traffic if they are to get enough income to pay the interest on their bonds. Forced into this procrustean bed, they will find that the economics of an industry concentrated on broadband with user control at the edges and less and less concern with extracting rent from the content are capable of benefiting from the declining costs of bandwidth and the hardware necessary to support it. The local companies will either adapt to the new model through bankruptcy or find through political intervention a way to bail out their debt ridden foolishness and go merrily on as before promoting bandwidth scarcity and ensuring that the telecom industry in the US in the 21st century shares the fate of the steel industry in the 20th century.

The questions facing us are no longer just ones of technology. They are ones of economics and policy. How we answer them will determine whether our information technology industry prospers or stagnates. One thing that we find fascinating about Woodcock's peering and transit methodology is that it may begin to provide an answer to the question of how asset based telecom and the fringes of the network can stitch themselves together to begin to replace the long haul carrier networks.

We begin this issue with Farooq Hussain's overview of global peering and transit. There follows the two part interview with Bill Woodcock. Bill suggests a very interesting world view where ISPs that are clueful and careful can make themselves competitive by avoiding interconnection with their ILEC and backhaul circuits to appropriate exchange points. They should peer as much as possible at an exchange where the cost of interconnection is as cheap as possible. Reliability is not critical because, should peering sessions fail, traffic can be delivered through the transit providers. Since transit is critical one should connect to at least two transit providers ideally at different exchanges. One figures out in each specific instance how to do this using the path analysis tools and then one simply does it.

Certainly, for the multiply-homed ISP, this is a viable model. However, when one gets to an entity the size of Verizon, SBC, Bell South or one of the MSOs with tens of thousands of cable modem customers one hears a different tune. There it is a huge difference in scale including national operation and perhaps a gigabit or even multiple gigabits per second in bandwidth requirements that prevents reliance on the ability to cutover quickly, when a problem arises, a large amount of peering bandwidth to a transit provider. The amount is such it is said that no transit provider could be counted on to accept instantaneous cut over without itself being swamped.

Yet these big up and coming want-to-be Tier 1 networks do peer. In fact most peer very assiduously. The ILECs in general still have only one transit provider. SBC uses Sprint. Verizon relies on Genuity and BellSouth on UUNET. We are told this is because of FCC inter lata long distance restrictions. As more and more are granted long distance authorization in their respective states, we are told this will change and the LECs will very likely begin building their own backbones. Given SBC's investment in Williams, BellSouth's in Level 3 and Verizon's in Qwest, building backbone infrastructure in the US and extending light waves to Europe and Asia for peering purposes would be trivial.

We don't claim to have special insight into the LEC or MSO operators peering mindset. Even were the LECs of the persuasion to do as much peering and little transit as possible, we suspect that such an attitude wouldn't stand up very well in the face of the LECs need to maintaining revenue and pay down debt. The same would go for their cable brethren. Despite the fact that transit bandwidth is by all standards exceedingly cheap having fallen in cost by anywhere from 80 to 90% in about two years, the LECs appear to be ready to make capital expenditures that go beyond transit costs to buy infrastructure. They say it is to have greater reliability and flexibility within their own networks. Many smaller independent ISPs feel the real motivation is to bury them.

The overarching problem looks to be that the Stupid Network doesn't yet seem to have a sustainable business model. If costs are indeed focused mainly at the edges what we have seen in the willingness to invest in user owned assets at the edge is reassuring. The problem is that there seems to be an unfundable unsustainable vacuum in the middle that no one has figured out how to deal with. But figure it out we better for as Roxane Googin points out until we solve "The Paradox of the Perfect Network," the telecom and IT industries will remain flat on their back. The result will be a permanent recession and an end to growth.


Contributors to Peering & Transit Symposium

Dileep Argawaal, Founder and CEO of Worldlink, Kathmandu Nepal

John Brown, CEO Chargres Technologies

David Diaz, Peering Advisor to BellSouth.Net, former CTO NetRail

Ralph Doncaster, founder I-Stop.com Network

Pedro Ferreira, Doctoral Student (transit & peering), Carnegie Mellon

Avi Freedman, founder of Netaxs and Chief Network Scientist at Akamai

Dan Golding, Effective Sept 27 Peering Manager AOL.

Roxane Googin, Telecom and IT Equities Analyst

Mike Hughes, Network Architect, LINX

Farooq Hussain, Partner at Network Conceptions and former Sprint NAP PI

Joe Klein, Peering Coordinator Adelphia

Kurtis Lindqvist, CEO Netnod, former network Architect Peering Coordinator, KPNQwest

Francois Menard, Project Manager, IMS Conseils

George McLaughlin, Director AARNet, Australia's Academic and Research Network

Keith Mitchell, founder and CTO Xchange Point

Lauren Nowlin, Peering Coordinator, SBC

Andrew Odlyzko, Director, Digital Technology Center, Univeristy of Minnesota

Jere Retzer, Senior Manager, Next Generation Networks, OHSU & Co-founder Northwest Access Exchange

Philip Smith, Consulting Engineer, Office of the CTO, Cisco Systems

James Spenceley, Network Architect, COMindico. Australia

Stephen Stuart, VP Engineering, PAIX

Martin van Nieuwelaar, Network Intelligence Software

Wouter van Hulten, Principal, Interxion.com (European exchange point operator)

Alexander Tudor, Researcher Agilent Laboratories and Collaborator with Bill Woodcock

Phil Weller, CTO of FastNet

Bill Woodcock, Research Director of Packet Clearing House and founder of Zocalo Networks