Building the World’s Largest Optical Network to Enable Global Science

An Examination of the Accomplishments of Harvey Newman and His Caltech Group

The April 2009 issue features the first part of a long two-part interview with a high energy physicist Harvey Newman.  Harvey explains the origins of his involvement in data networks and other fields of information technology, along with some highlights of his career at the frontiers of modern science. His founding of international networking for his scientific field started with the need to be working with European particle colliders, first in Germany and then at CERN while raising a family, teaching and running a research group at Caltech in Pasadena.  

Harvey explains how during the 1980s and then into the 1990s, he worked with the early data network protocols and beginning with X.25 adopting varied protocols like DECnet SNA in TCP/IP as the 80s turned into the 90s -- eventually switching to a reliance on TCP/IP during the mid-1990s.   The data output of large particle accelerators continually pressed against the state of the art of information technology and networks.  Deregulation in Europe in 1998 made it possible for Harvey to move into  the optical hierarchy for his transatlantic links.   With the floodgates open and the data needs of the large hadron collider truly unprecedented,  Harvey began to build a team that was able to take on the demands of what has become a global collaboration.

He has focused on innovative problem solving in several network-related technologies, developed a new class of global-scale systems to support the science, and delved into areas of science policy as well as telecommunications policy.   As an example of this we may point to his design work on the data dissemination structure for the LHC. In this case he has focused not only on issues of technology, but also on how to structure and operate his data dissemination architecture so as to encourage cooperation amongst universities and laboratories throughout the world, while also working in many countries to bring along those science groups with less-advanced national infrastructures, and economies, towards equality with their peers.

A critical skill is his ability to conceive, fashion and implement an architecture that enables scientists located at hundreds of sites in many nations to successfully carry out their work in a decentralized, cooperative way, while remaining within the orbits both of their local universities, all the while connected via networks to their experiments at the LHC and especiallly their colleagues working there are the CERN laboratory in Switzerland and France. The decentralization of the architecture created an attractive feedback loop that enabled the widely decentralized communities, and their funding agencies, to participate in a project of global importance while building local and national capabilities.

Another outcome made possible by carrier competition was the ability  to work independently of carrier technology and protocols.  Harvey made a critical point when he said: "What has made the wide distribution of data to and from many switches and also many servers at one and now 10 Gbps feasible was the abandonment of the old “carrier class routers” and the adoption of 10 gigabit Ethernet (10GE), in full-scale switches where a 10GE port costs on the order $ 10k."  Being able to gain access to basic fiber infrastructure in the form of lightwaves enabled Harvey’s global science collaboration to function relatively free of telco business model constraints.

No Progress Until the Telco Model Becomes Technology Driven

Newman:  “When it comes to wide area networks, the world is mostly politically and policy driven, and not technology driven; unlike some other areas of information technology.   Somebody mentioned today on your list that the carriers seem to be waiting to see public demand. However this is not true for storage and not true for processors in that the makers of these devices just keep increasing their capacity, challenging users to create and develop new applications and new skills.   Unfortunately the deployment of bandwidth is not technology driven.  The vendors are looking at markets.

We seem to have a kind of conundrum with bandwidth: you do not seem to have an elastic demand.  If the producers put out 10 times the capacity, the public seems to be unaware that it’s there and unsure of exactly what they should do with it.  With bandwidth neither awareness nor a real cost model is driving an elastic demand.”

Indeed the telecommunications driven market gives us billing for every bit, carrier class routers at $ 1.5 million a pop and nothing more innovative than triple play.  Harvey’s world unfortunately is a much too well kept secret.   As we shall see in part two next month he is using the needs of his global science collaboration to inspire, or press, political authorities to be innovative in building new infrastructure.  As we cling to the last vestiges of our market model other countries – less restrictive – will reap the benefits on a large scale - not only in science, but also in education technology and economic competitiveness.

Harvey Newman: It’s a different strategy based on dark fiber infrastructures which have been deployed by national research and education networks (NRENs) in a growing number of countries outside the US, including economies like those in central Europe (Czech Republic, Slovakia and Romania for example) that are orders of magnitude smaller than the US. Just put things out there.  And find out what people want to use. Create conditions where the costs are low so that mass-produced technologies are able to use the infrastructure fully, and the methods to use it effectively are created rapidly on a large scale, and evolve as the infrastructure evolves.

Cook Report:  the problem is this won’t happen as long as national backbones are powered by ridiculously expensive carrier class routers supported by customer billing systems designed for the days of analog technology.

The final half of the first part of the interview with Harvey explains the technical workings of the collider itself and lays the groundwork for understanding the LHC network and its unique accompanying telepresence system that will be the subject of next months conclusion.

IPv4 Market Risks, p. 33

As with new registry policies IPv4 blocks become property, the question is raised as to whether there will emerge a business model focusing on the aggregation of owned IP blocks into routable chunks.  David Conrad responded The fact that speculators will have locked up the allocated-but-unused address space merely maintains the status quo...

Vest: Actually, the status quo is likely to change in one huge way, with multiple far-reaching effects, regardless of whether the market "works" and liberates lots of additional IPv4, or absolutely fails to motivate current "reserve address holders" (i.e., hoarders), or is fatally distorted by speculators, etc.
The next time a lawsuit arises over control or use of IP address resources (it has happened several times in the past), at least one side is going to have recourse to the protections afforded by whatever property law is in force in the relevant jurisdiction. Before transfers, resource users were not considered to "possess" IP number resources in the way that makes the phrase "possession is nine-tenths of the law" such a perennial favorite. Once transfers start happening, and lawyers and accountants are forced to reckon with the implications for both transfer transaction participants and everybody else, this will no longer be true.

The first likely impact will be to render moot any policy-based requirements or restrictions on IP address transfers themselves. The really big impact will come when this starts to undermine participation in (any) shared public registration database -- which to date is the only thing that has preserved the presumption of uniqueness that puts the "public" in "public IP addresses". If that goes, the only things that might be able to put it back will be property law and/or public regulation -- which will have to be reconciled with other laws about privacy, cross-border trade and direct investment, etc.

Editor: What is being abandoned according to Tom Vest is “That canonical RIR-era arrangement -- hierarchical routing and prefix-length filtering for the big operators plus a neutral, open allocation mechanism for new entrants -- created an excellent, conflict-minimizing industrial environment for growing the Internet.”  snip “Now, however, that bargain has been abandoned -- and it seems highly likely that the party that lost out (i.e., the new entrants) will ultimately react the same way that they do in other industries that are shaped by a concentration of market power/control over critical bottleneck inputs.

So, in the end, we may get to run the market advocates' experiment and see exactly what the routing table looks like when there are no durable limits on demands for finite routing system capacity.”

And later Vest: The demand for confidentiality or non-transparency, and for freedom from "bureaucracy" is common if not universal, and quite understandable in most cases. Whenever transacting parties can get away with non-disclosure, they usually do, for better or worse (e.g., in the banking sector, better yields right to individual privacy, worse yields global financial collapse). In this particular case, the complete absence of any countervailing enforcement mechanisms is sure to encourage more people to press that demand. But as a result, the registration database that is the only mechanism that assures the uniqueness of IP number resources (and hence their basic utility) will cease to be sustainable.

KPN Joins Amsterdam in
Extending its Fiber Build  p. 43

Paul Budde on Feb 4: The City of Amsterdam announcement to now move to the next stage of their FttH project - with another roll out covering 100,000 connections - is a clear indication that the concept of open access FttH networks is a valid one. This will have large-scale implications for countries around the world that are looking at using open network based telecoms infrastructure projects to stimulate their economies.

BuddeComm has been involved in industry policy discussions with experts on three continents about the future of telecoms.  We have developed a scenario. The vision naturally has a range of strategies attached, but in simple terms it works as follows: 1. Telecoms infrastructure is of national economic and social importance; 2. For our societies to profit from the digital economy infrastructure must be based on the principle of open networks; 3. This allows us to multiply the benefits this infrastructure has to offer to other sectors such as healthcare, education, energy, environment, media and communications; 4. Once open networks and the access tariffs are established the national telecoms operator will be in the best position to run this network, thus avoiding the necessity for wasteful duplication. 5. Infrastructure and digital applications will need to be developed parallel with each other and this requires trans-sectoral thinking from the government and the industry - not the current silo thinking.

While it is great to put such visions in front of the policymakers in the end the question is whether this is achievable - can such a vision be implemented? Again the Netherlands is paving the way. Amsterdam was one of the first to identify the multiplier effect and the need for trans-sectoral thinking. Unfortunately the incumbent KPN first preferred to roll out fibre nationally  [Snip]

Editor: The Open network should become a magnet for all providers and the supplier that keeps it open gets the maximum return on the cost of the build by attracting many new businesses.  Think in terms of the shopping mall that grows because of all the other places that one can do business with on a one stop basis.

Vincent Deker: If you allow all your competitors on your network, all services will come on your net, and that results in the lowest cost possible per service. Which in turn attracts more customers for those services, so your network grows much faster. An open network is not charity from our side, in the long run it simply works best for everybody.'

IT, Internet and the Power Grid  p. 47

A discussion about the use of the internet for energy saving and CO2 reduction shows the complexity of needed planning.

Bill St Arnaud: Most people think today's Smart Grids will reduce consumption- where in fact they largely only displace consumption from peak demand. In the several studies done s far on smart grids and smart meters, the savings have been very small for consumers.

Some early studies indicate that providing feedback to consumers on energy consumption will reduce demand. But these studies are very limited scope and the sample population is usually made of dedicated and concerned homeowners. We will need longer-term studies with larger populations to see if feedback meters will have a meaningful impact.

The biggest beneficiary of smart grids is not the consumer, but the utility who does not have to build more power plants for peak load. Indirectly this marginally does reduce CO2 emissions because base load is usually nuclear while peak load is usually gas powered plants

Jeff Sterling offers an excellent summary of three different ways of looking at he concept of a smart Grid.

Smart Grid is a Vision. As we speak, smart grid doesn't really exist and it's probably an evolutionary process. The question is whether smart grid is a transformative conversation with transectoral implications that goes beyond the boundaries of the IEEE or other private industry group that are looking to optimize the electric utility system. I would argue that smart grid is the connective tissue we need to enable the new energy sector.

Smart Grid is an operating system
. The bottom line for any successful implementation of smart grid is software based. There are firms out there that want to be the Microsoft of the Grid. Google, Cisco, IBM, Microsoft, and a host of startups are all trying to figure out what to do. What is the model? I would hope for a open standards approach that would create a development platform for localized energy projects.

Smart Grid is a community network
. Smart grid is the ultimate opportunity for transectoral integration at a community level. Unless you include the human factor into the system you are reinforcing the silo mentality that is ossifying innovation in community infrastructure.

Progresive Lebanon Regulatory Agenda  p.58

Kamal Shehadi is the chairman of Lebanon's Telecommunications Regulatory Authority (established April '07).

Shehadi: We are using the fact that telco assets are still public assets to push for infrastructure sharing for passive elements and to provide access to existing conduits and ducts to the new entrants to lower the Capex and speed up deployment. It is easier to claim that these assets, since they are still publicly-owned, should be used in a way that benefits consumers, new entrants, etc., and not just the incumbent's market share...

Van der Berg: Good luck to you then. It seems like you have some of the right ideas already.

Goldstein: It seems to me that since there is not yet a private "incumbent", and it's still in the ministry's hands, you have an opportunity to structure the privatization in a manner more befitting of the national interest than many earlier privatizations.

Infrastructure sharing is necessary, of course, because it is a natural monopoly -- requiring competitors to put in their own ducts and conduits is economically not feasible, and harms the streets in the process. But the natural monopoly in many cases (probably to all buildings except the largest) really extends to the entire loop plant as well. So why not define the whole outside plant (conduits, poles, copper and glass loops) as the infrastructure, and put it in a separate entity from the one that provides network services (which gets the switches and existing PSTN customer accounts)? This is sort of how BT structured OpenReach, and a lot of us have been advocating this for the US.

James Seng: Please do not make the same mistake in Singapore during privatization. Privatize the service but keep the infrastructure. It will save you a lot of headache later.

Verwayen Takes Pro Incumbent View of Network Neutrality at Alcatel Lucent  p. 60

Goldstein: Rather than craft a set of consumer-friendly solutions that might actually protect ISPs against such things, it has become a battle of extremes. Verwaayen seems to be firmly in the IMS camp, wanting to replace the Internet (a business model, not a protocol suite!) with IMS, an equipment-vendor-friendly approach where he and his carrier-customers can sell individual applications. Just like 1982's Compuserve, but with video.

Rood: I think this is a crisp summary of Verwaayen's mindset. He sees the Internet business model as Fred describes it as flawed, as he cannot imagine any end user desiring the freedom in choice of ISP as a business model, because he is thinking about end users in terms of only desiring applications and content provisioning and then the IMS model is the only one that makes sense to him.


Building the World’s Largest Optical Network to Enable Global Science Collaboration

An Examination of the Accomplishments of Harvey Newman and His Caltech Group

Editor’s Introduction                                                             p. 1
Why Networks?                                                                     p. 4
Evolution of Network Infrastructure in the 80s                     p. 5
Ushered into the 1990s by Al Weis and IBM                          p. 6
Deregulation Opened the Floodgates                                    p. 7
No Progress Until the Telco Model Becomes
Technology Driven                                                                p. 9
The New LHC Network                                                         p. 11
Drinking from the Collider Firehose                                     p. 13
What We Are Doing with the Collider                                   p. 14
A Synergy Between High Energy Physics and
Cosmology                                                                           p. 16
The Collider’s Detection Systems                                        p. 21
State of the Higgs in 2008                                                   p. 25
Supersymmetry                                                                  p. 27
String Theory                                                                     p. 28
Global Collaboration Mandates Inventive Solutions
to Resource Constraint Problems                                       p. 30

Symposium Discussion January 19 - February 18 2009

What to Do With Fiber Optic Bandwidth?
Next Steps for Leadership
Or “Dad, I Want a HyperRoom!!” 
                p. 31

More About Risks of Emerging
IPv4 Markets           
                                    p. 33
But Aren’t the RIRs Restricting Transfers to 24s?                      p. 38
RFC 1744 – Predictions of the Market to Come                          p. 39

KPN Joins Amsterdam in Extending its
Open FttH Network                                     
p. 43

The Official Amsterdam Announcement                                    p. 44
Historic Step                                                                              p. 45

IT, Internet, and the Power Grid                  
p. 47

It all Depends                                                                             p. 54

Kamal Shehadi, Chairman of Lebanese
Regulatory Authority Discusses Forward
Looking Goals    
 p. 58

Verwayen on Net Neutrality
Not What one Would Have Hoped for from the
Former CEO of BT   
                                     p. 61

Executive Summary                             
             p. 65