Clouds, Data Centers, and Network Programability

This issue explores some of the very significant consequences of vast increases in memory, processing cycles, and storage that make the network far less of a bottleneck than it was ten or twenty years ago.  Computational cycles, along with both memory and data storage, are now plentiful and cheap.  The ability to send streams of photons at nearly the speed of light down strands of glass means that the components of computing can be spread out physically in a manner such the location of clumps of processing power or storage becomes relatively unimportant. Location is unimportant because they can be connected to form a network in such a way that the fiber optics transferring information from one point to another makes it possible to treat the whole as a giant computational complex that can be organized on the fly.  The time needed for the components to communicate is so minute that it is possible for widely separated devices to be organized by software and to work as though they were one device.

As capabilities grow, what used to be a static network, becomes dynamic and the configuration of attached devices and CPUs can be instantaneously changed, divided, reorganized and so on to make resources available to end users on an as-needed and on-demand basis.  The entire complex can be “virtualized” by changing the way data is sent through the network. Virtualization means that the owner of large resources can have a network brain or command center that acts in a manner analogous to the CPU in a desktop or cluster computer telling the network wide collection of resources what to do.  In other words the network has its own brain that can direct its resources in a multiplexed manner to perform a wider range of services for end users.  As a result we have the cloud and we have the provisioning of software and many other things “as a service.”

The cloud or over all “network” itself becomes programmable.  This issue describes the approaches that have to be taken to bring this new capability to bear.  Here the primary element is the new Layer 2 communications protocol called OpenFlow that gives access to the forwarding plane of a network switch or router remotely over the network. This separation of the control from the forwarding allows for more sophisticated traffic management.  OpenFlow can move packet routing capability from the proprietary operating systems of very expensive routers down to the layer two level of very cheap commoditized switches that can run ultra high bandwidth Carrier Ethernet. OpenFlow is considered to be an enabler of the next subject of this issue  Software Defined Networking.

According to Wikipedia Software defined networking (SDN) is an emerging architecture for computer networking separating the control plane from the data plane in network routers and switches. Under SDN, the control plane is implemented via software in a sever or servers separate from the network equipment and the data plane is implemented in commodity network equipment. As a result SDN allows for quick experimenting and optimization of switching/routing policies.  Also SDN can give external access to the innards of switches and routers that formerly were closed and proprietary.

With the internet having grown to the status of a mission critical global communications system, concerns have been raised that the resulting inflexibility of the implementation of the commercial internet.  In effect if you want to explore any changes, you must have an alternative network infrastructure necessary before anyone can experiment with different and presumably better protocols for transporting data.  Consequently the third subject of this issue is the NSF-funded GENI or Global Environment for Network Innovation.  GENI members use OpenFlow enabled GENI “racks” to use circuit-switched Carrier Ethernet channels to form virtual networks that can run alongside ordinary production networks.

As a result of this, the old idea of “the network is the computer” is becoming more of a reality as opposed to the earlier metaphor coined by John Gage at Sun Microsystems.  With OpenFlow and SDN, collections of networked linked resources can be set up and torn down to establish a huge, and globally spread out array of “Virtual machines.”

To explain what is happening I have interviewed five different participants in these important developments.  I begin with Jennifer Rexford at Princeton University who explains how some of the business concerns of managing AT&T’s backbone in 2003-2004 led her group to explore a very early effort to separate the network routing mechanism from its decentralized distribution into routers and instantiate the function into a single network “controller-computer”.  This had benefits of making it much easier for AT&T to do more fine grained treatment of customer routing needs.  Jennifer worked with GENI but now is focused on development of a programming capability called Frenetic to make fine-grained control of an SDN easier at the hypervisor or application layer where a hierarchy of services is enabled within the cloud.

The explosion of capability in hardware and software at low cost is enabling the cost-effective application of enterprise computing for many new activities.  As a result, the amount of digital data generated by enterprise business process is exploding.  The design of data centers to handle the result is of critical importance to companies like Ciena that produce switches and routers for optical networks.

Consequently I have interviewed Chris Janz of Ciena’s Enterprise Strategy Group to learn how Ciena uses OpenFlow and SDN capabilities to enable commercial data centers to grow  and support cloud services much more cost effectively.  Here one of the prime examples is Google’s g-scale network connecting a dozen mega data centers around the world that handle Google’s backend needs while a separate Google optical network connects to it and then to the global internet. The evolution of these technologies (OpenFlow and SDN) is making further evolution of cloud services available.  An entire family of cloud based services is emerging.

This wikipedia article on cloud computing points out that there re many types of public cloud services. There are many types of public cloud computing:[1]
Infrastructure as a service (IaaS)
Platform as a service (PaaS)
Software as a service (SaaS)
Storage as a service (STaaS)
Security as a service (SECaaS)
Data as a service (DaaS)
Test environment as a service (TEaaS)
Desktop as a service (DaaS)
API as a service (APIaaS)
The business model, IT as a service (ITaaS), is used by in-house, enterprise IT organizations that offer any or all of the above services.

New “as-a-service” flavors are popping up frequently. For example on April 1, 2012 the NSF granted the Computation Institute of the University of Chicago and Argonne National laboratory a 2.4 million dollar award for SciDaaS – Data Management as a Service.  The ward would underwrite the cost of developing a suite of innovative research data management services for the NSF community: the SciDaaS project. These services, to be accessible at, will allow research laboratories to outsource a range of time-consuming research data management functions, including storage and movement, publication, and metadata management.

Enterprise customers share a desire to bring down the capital and operational cost-at-scale curve for their networks.  Looking at these operational trends, they conclude that computing will increasingly be concentrated in large data centers and increasingly they will be in ones that they do not own. For them SDN helps address the question of how do you make more network resources more affordable?

Next Inder Monga of the Energy Sciences Network described for me how he uses OpenFlow to take the dynamic light paths of data from, say, a radio telescope and deliver it across global networks to juts to the border of a university network but all the way to a scientist’s work station or grid computing cluster.  Through the refined use of headers, large carrier Ethernet flows can be designed to transit through a network and arrive with precision and unimpeded at a desired end point or end points.  However, we must note that this unimpeded service from one end user to another must also be balanced within the network so that large scale, end-to-end users don’t soak up all the network’s capability.

GENI is a program that has been underway for about five years and is designed to develop an architecture for network experimentation that can run independently of the commercial internet.  It has figured out that the most effective way to do this is to use open flow switches to do virtual network overlays on the backbones of the major R and E networks in the United States.  One of the uses in the early stages has been to develop GENI Racks composed of OpenFlow switches that are really just generic switches from companies like HP where the switch becomes OpenFlow capable by means of a simple firmware upgrade delivered over the network.

GENI was in design from about 2004-2007 and in 2007 the NSF awarded an agreement to BBN to oversee their building of a prototype system consisting of GENI racks installed in many university network centers such that these racks could be used to establish overlaid virtual networks.  The bid established the GENI Program Office at BBN in Boston to organize the establishment and use of the GENI infrastructure.  Chip Elliot, whom I interviewed on August 3rd, was named as the Program Office director.

Chip deals with the conceptual difficulty of explaining GENI that while researchers in some areas of science build a new tool to use the network for their particular area of science, with GENI the concept is of basic change to the network itself.  As he explained it -- “computer science researchers are always fiddling with their computers, making systems that try new things: unix, the internet. GENI is in that category: a large scale system being built by the researchers themselves that will open up space for experimentation.”  He continues: “We’ve been running the GENI project for the last five years by issuing our own solicitations to researchers, to the research teams that come up with their own proposals to design and build parts of GENI. The winners are designing and building GENI.”

I have found out from spending a fair amount of time with GENI and the GENI Wiki that the basic idea presented is to say OK:  here is a foundation of Carrier Ethernet.  Build what you want to run on top of it.  At the end of 2012 the basic foundation for running virtual overlays is there and is being enlarged to cover more campuses on the foot print of the R and E networks.  However precisely what will be built to use separate slices of the network is not yet clear to me. 

The idea seems a good one, however my final interviewee, Glenn Ricart has come up with his own idea of US Ignite.  Ignite is going to function as a proving ground for new applications. It will do so in a very interesting way that uses GENI technology…basically the racks that are quite generic and not terribly expensive to stitch together “slices” that will interconnect a few cities with their own high speed fiber networks and cities without city wide fiber networks but one with area like the zone at Case Western in Cleveland, Ohio to Lafayette, Louisiana; Chattanooga, Tennessee, and the semi-rural chain of towns in Utah known as Utopia and stimulus-built middle mile networks.  The focus on new applications is interesting while the willingness to be open to new constituencies is intriguing.  Would not it be wonderful if we could take R and E oriented projects and connect them into an alternate infrastructure that might catalyze a way of doing things that would be different from the predatory operation and mindless entertainment that is the product of the the carriers and MSOs?

As we go to press:

OpenFlow and SDN continue to be very topical.  At RIPE 65 in Amsterdam on September 24-28 Ivan Pepelnjak, Chief Technology Advisor, NIL Data Communications, a Slovenian company, presented a slide deck called OpenFlow and SDN: hype, useful tools or panacea? His comments were that for the mega data centers of Google Microsoft and Amazon, this new technology was indeed good but for use by lesser giants, the case was not entirely made.  The deck is worth a look and may be viewed here.

In Light Reading we have Internet2 Readies Its SDN Launch.    September 25, 2012 -- Craig Matsumoto:  “Internet2 is ready to launch its first try at software-defined networking (SDN), a little Layer 2 bonus to go with its new 100Gbit/s backbone. The SDN platform should be completed sometime this month . . . As of two weeks ago, the Layer 2 gear from Brocade -- the part that would provide SDN support -- was still being installed; Vietzke expected that buildout to be completed by Internet2's next member meeting, next week in Philadelphia.” The Mozilla Foundation also announced the first winners of the US Ignite Gigabit applications competition.

And finally the New York Times embarrassed itself with a two part investigative report called Power, Pollution and the Internet.    The power required by data centers is indeed a problem as readers either already know or will see. However for his common place errors,  James Glanz, the author was ripped to shreds on arch-econ.  “The NYT article . . . .was terrible.  Sensationalistic, errors all over, and what look like intentionally misleading assertions based on selected segments of of real research.  I would expect this kind of journalism in a backwater blog, but NYT?!!!!” - wrote Ken Miller whose professional work whose professional work is in data center design and capacity planning.

“Frank and I have been in agreement on the efficiencies available through fiber and the ignorance around the power consumption in the IT closets for network and communications for years. There are a great deal of energy inefficiencies spread all over the enterprise, why does the NYT focus just on data center? I agree with Terra that there are opportunities for LARGE power reductions (maybe 60-80% across the entire compute facility?); many of them aren't IT equipment, but include inefficiencies of mechanical cooling equipment, electrical equipment, or design decisions which are present in all commercial, manufacturing, and residential facilities.  Why the focus on data center?  If I run the numbers on energy consumption and waste compared to other industries, it is a small niche player.  I could only conclude that Mr. Glanz had a theory and set out to write an article to prove it."

"They go out of their way to state that NYT did research for a year.  On WHAT?, all the data they reported came from UpTime, Koomey, McKinsey and other previously published reports (some very old). NYT didnt say anything that hasnt been published for almost a decade.   I have worked with Ken Brill and UpTime for years .  I supplied data to Ph.D. Koomey in support of his research when he was at LBL documenting old system energy consumtion.  (Met him through Ken Brill).  I have the first McKinsey & Company report on data center efficiency in 2007 (or '08) and some of their subsequent research, none of which seems to be commissioned by the NYT.  I'm happy to forward that report to members on the list that want to see it (it's on visualization and utilization rates inside the enterprise data center and how that translates to power use).  These people do very real and valuable scientific research, to have their data thrown around like this devalues the real work they perform.”



Executive Summary p.  4

Introduction Innovation, Geni, and Software Defined Networks        p.  9

Innovation not Stupidity in the Middle                  p. 10

Enabling a Separate Network Control Plane
Jennifer Rexford on the “Why” of OpenFlow  
Enabling a Network Control Plan                      p. 13
Enabling Innovation at the Network Level       p. 15
The Commoditization of Routers                      p. 19
OpenFlow, GENI and Network Virtualization     p. 20
Frenetic                                                             p. 22
Tightly Coupled Systems                                      p. 23

A Vendor Instantiation of OpenFlow
Ciena’s Chris Janz                             p. 25

The Cloud, the Data Center and IT-as-a-Service   p. 25
Content and Compute:  New Players, Machine-to-Machine Traffic and Unpredictable Bursts        p. 26
Leading Use Case for SDN “Dynamically Multiplex” Instead of Over-Provision                                   p. 27
Beyond Data Center Use Cases – SDN & Carriers
p. 33

Using OpenFlow to Carry Lambda Traffic from Network Border to Scientist’s Work Station
Inder Monga                                                p.34
Using Open Flow to Direct Packets Where you Want Them to Go                                           p.37
Make Network into Virtual Switch for End User Control                                                    p. 40
Increasing the Capability of Dynamic Lightpaths p. 44

GENI or Global Environment for Network Innovation Chip Elliott
What Is GENI?                                                   p. 46
The Slice                                                         p. 48
Speak OpenFlow – Join GENI                            p. 51
OpenFlow Used in Setting GENI Slices to Run
over Campus Ethernet                                      p. 51
GENI as a Highly Flexible Global Data Utility     p. 55

Real World Use of GENI
Glenn Ricart and US Ignite                                 p. 66
US Ignite                                                             p. 72
Three Key Technologies                                     p. 74

Appendix: Welcome to the GENI Wiki p.80

Connecting with the GENI Community                p. 81
Experimentation with GENI                                 p. 82
Welcome GENI Experimenters!                            p. 86
GENI Experiment Repository                               p. 87
Welcome to GENI Operations                               p. 93
GENI Racks                                                          p. 94
Huawei adopts SDN                                             p. 95