08.10.17 Ninth Lecture in the Australian American Leadership Dialogue Scholar Tour University of Sydney Title: Coupling Australia’s Researchers to the Global Innovation Economy Sydney, Australia

1. “Coupling Australia’s Researchers
to the Global Innovation Economy”
Ninth Lecture in the
Australian American Leadership Dialogue Scholar Tour
University of Sydney
Sydney, Australia
October 17, 2008
Dr. Larry Smarr
Director, California Institute for Telecommunications and
Information Technology
Harry E. Gruber Professor,
Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD

2. Abstract
An innovation economy begins with the “pull toward the future” provided by a robust public
research sector. While the shared Internet has been rapidly diminishing Australia’s “tyranny of
distance,” the 21st Century global competition, driven by public research innovation, requires
Australia to have high performance connectivity second to none for its researchers.
A major step toward this goal has been achieved during the last year through the Australian
American Leadership Dialogue (AALD) Project Link, establishing a 1 Gigabit/sec dedicated end-to-
end connection between a 100 megapixel OptIPortal at the University of Melbourne and Calit2@UC
San Diego over AARNet, Australia's National Research and Education Network.
From October 2-17 Larry Smarr, as the 2008 Leadership Dialogue Scholar, is visiting Australian
universities from Perth to Brisbane in order to oversee the launching of the next phase of the
Leadership Dialogue’s Project Link—the linking of Australia’s major research intensive universities
and the CSIRO to each other and to innovation centres around the world with AARNet’s new 10
Gbps access product.
At each university Dr. Smarr will facilitate discussions on what is needed in the local campus
infrastructure to make this ultra-broadband available to data intensive researchers. With this
unprecedented bandwidth, Australia will be able to join emerging global collaborative research—
across disciplines as diverse as climate change, coral reefs, bush fires, biotechnology, and health
care—bringing the best minds on the planet to bear on issues critical to Australia’s future.

3. “To ensure a competitive
economy for the 21st century,
the Australian Government
should set a goal of making
Australia the pre-eminent
location to attract the best
researchers and be a preferred
partner for international research
institutions, businesses and
national governments.”

4. The OptIPuter Creates an OptIPlanet Collaboratory:
Enabling Data-Intensive e-Research
GIST, Korea www.evl.uic.edu/cavern/sage Michigan
KISTI, Korea SARA, Netherlands “OptIPlanet: The OptIPuter
Chicago Global Collaboratory” –
SAGE software, developed by UIC/EVL for OptIPuter, Special Section of
Supports Global Collaboration. Five Sites Streaming Future Generations
Computer Systems,
Compressed HD Video (~600mb Per Stream)
Volume 25, Issue 2,
Using “SAGE Visualcasting” to Replicate Streams February 2009
Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI
Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST
Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent

5. OptIPuter Step I:
From Shared Internet to Dedicated Lightpaths

6. Shared Internet Bandwidth:
Unpredictable, Widely Varying, Jitter, Asymmetric
10000 12 Minutes
100-1000x
Stanford Server Limit
Computers In: 1000 Normal
Time to Move UCSD Internet!
Australia a Terabyte
100
Outbound (Mbps)
Canada
Czech Rep.
India Data Intensive
10 10 Days Sciences
Japan
Korea Require
Mexico Australia Fast
1
Moorea Predictable
Netherlands Bandwidth
Poland 0.1
Taiwan
United States
0.01
0.01 0.1 1 10 100 1000 10000
Source: Larry Smarr and Friends
Inbound (Mbps)
Measured Bandwidth from User Computer
to Stanford Gigabit Server in Megabits/sec
http://netspeed.stanford.edu/

7. Two New Calit2 Buildings Provide
New Laboratories for “Living in the Future”
• “Convergence” Laboratory Facilities
– Nanotech, BioMEMS, Chips, Radio, Photonics
– Virtual Reality, Digital Cinema, HDTV, Gaming
• Over 1000 Researchers in Two Buildings
– Linked via Dedicated Optical Networks
UC San Diego
www.calit2.net
Preparing for a World in Which
Distance is Eliminated…

8. Discovering New Applications and Services
Enabled by 1-10 Gbps Lambdas
Maxine Brown, Tom DeFanti, Co-Chairs
i Grid 2005
www.igrid2005.org
THE GLOBAL LAMBDA INTEGRATED FACILITY
September 26-30, 2005
Calit2 @ University of California, San Diego
California Institute for Telecommunications and Information Technology
21 Countries Driving 50 Demonstrations
Using 1 or 10Gbps Lightpaths
Sept 2005

9. The Large Hadron Collider
Uses a Global Fiber Infrastructure To Connect Its Users
• The grid relies on optical fiber networks to distribute data from
CERN to 11 major computer centers in Europe, North America,
and Asia
• The grid is capable of routinely processing 250,000 jobs a day
• The data flow will be ~6 Gigabits/sec or 15 million gigabytes a
year for 10 to 15 years

10. EXPReS-Oz eVLBI Using 1 Gbps Lightpaths
October 2007
Image Credit: Paul Boven,
Data Streamed at 512 Mbps
Image created by Paul Boven, JIVE
Satellite image: Blue Marble Next Generation, courtesy of NASA Visible Earth

11. Next Great Planetary Instrument:
The Square Kilometer Array Requires Dedicated Fiber
www.skatelescope.org
Transfers Of
1 TByte Images
World-wide
Will Be Needed
Every Minute!

12. Dedicated 10Gbps Lightpaths Tie Together
State and Regional Fiber Infrastructure
Interconnects
Two Dozen
State and Regional
Internet2 Dynamic Optical Networks
Circuit Network
Under Development
NLR 40 x 10Gb Wavelengths
Expanding with Darkstrand to 80

13. Global Lambda Integrated Facility
1 to 10G Dedicated Lambda Infrastructure
Interconnects Global
Public Research Innovation Centers
Source: Maxine Brown, UIC and Robert Patterson, NCSA

14. AARNet Provides the National and Global Bandwidth
Required Between Campuses
25 Gbps to US
60 Gbps Brisbrane - Sydney - Melbourne
30 Gbps Melbourne - Adelaide
10 Gbps Adelaide - Perth

15. OptIPuter Step II:
From User Analysis on PCs to OptIPortals

16. My OptIPortalTM – Affordable
Termination Device for the OptIPuter Global Backplane
• 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000
• 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC!
• Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC
Source: Phil Papadopoulos SDSC, Calit2

17. OptIPuter Scalable Displays
Are Used for Multi-Scale Biomedical Imaging
200 Megapixels!
Source:
Mark
Ellisman,
David
Green: Purkinje Cells Lee,
Red: Glial Cells Jason
Light Blue: Nuclear DNA Leigh
Two-Photon Laser Confocal Microscope Montage of
40x36=1440 Images in 3 Channels of a Mid-Sagittal Section
of Rat Cerebellum Acquired Over an 8-hour Period

18. Scalable Displays Allow Both
Global Content and Fine Detail

19. Allows for Interactive Zooming
from Cerebellum to Individual Neurons

20. Prototyping the PC of 2015:
Two Hundred Million Pixels Connected at 10Gbps
Data from the Transdisciplinary Imaging Genetics Center
50 Apple
30”
Cinema
Displays
Driven by
25 Dual-
Processor
G5s
Source: Falko Kuester, Calit2@UCI
NSF Infrastructure Grant

21. World’s Largest OptIPortal –
1/3 Billion Pixels
NASA Earth Satellite Images
Bushfires October 2007 San Diego

22. Very Large Images Can be Viewed
Using CGLX’s TiffViewer
Spitzer Space Telescope (Infrared)
Hubble Space Telescope (Optical)
Source: Falko Kuester, Calit2@UCSD

23. Challenge—How to Bring This Visualization Capability
to the Supercomputer End User?
2004
35Mpixel EVEREST Display ORNL

24. The Livermore Lightcone: 8 Large AMR Simulations
Covering 10 Billion Years “Look Back Time”
• 1.5 M SU on LLNL Thunder
• Generated 200 TB Data
• 0.4 M SU Allocated on
SDSC DataStar for
Data Analysis Alone
5123 Base Grid, 7
Levels of Adaptive
Refinement65,000
Spatial Dynamic
Range
Livermore Lightcone Tile 8
Source: Michael Norman, SDSC, UCSD

25. Using OptIPortals
to Analyze Supercomputer Simulations
Mike Norman, SDSC
October 10, 2008
Two 64K
Images
From a
Cosmological
log of gas temperature log of gas density
Simulation
of Galaxy
Cluster
Formation
Each Side:
2 Billion
Light Years

26. Students Learn Case Studies
in the Context of Diverse Medical Evidence
UIC Anatomy Class
electronic visualization laboratory, university of illinois at chicago

27. Using HIPerWall OptIPortals
for Humanities and Social Sciences
Software Studies
Initiative, Calti2@UCSD
Interface Designs for
Cultural Analytics
Research Environment
Jeremy Douglass (top)
& Lev Manovich
Calit2@UCI (bottom)
200 Mpixel
HIPerWall Second Annual
Meeting of the
Humanities, Arts,
Science, and
Technology
Advanced
Collaboratory
(HASTAC II)
UC Irvine May 23, 2008

28. On-Line Resources
Help You Build Your Own OptIPuter
www.optiputer.net
http://wiki.optiputer.net/optiportal
www.evl.uic.edu/cavern/sage
http://vis.ucsd.edu/~cglx/

29. OptIPuter Step III:
From YouTube to Digital Cinema Streaming Video

30. AARNet Pioneered Uncompressed HD VTC with
UWashington Research Channel--Supercomputing 2004
Canberra Pittsburgh

31. e-Science Collaboratory Without Walls
Enabled by iHDTV Uncompressed HD Telepresence
1500 Mbits/sec Calit2 to UW Research Channel Over NLR
May 23, 2007
John Delaney, PI LOOKING, Neptune
Photo: Harry Ammons, SDSC

32. OptIPlanet Collaboratory Persistent Infrastructure
Between Calit2 and U Washington
Photo Credit: Alan Decker Feb. 29, 2008
Ginger
Armbrust’s
Diatoms:
Micrographs,
Chromosomes,
Genetic
Assembly
iHDTV: 1500 Mbits/sec Calit2 to
UW Research Channel Over NLR
UW’s Research Channel
Michael Wellings

33. Telepresence Meeting
Using Digital Cinema 4k Streams
4k = 4000x2000 Pixels = 4xHD Streaming 4k
100 Times with JPEG
the Resolution 2000
of YouTube! Compression
½ Gbit/sec
Lays
Technical
Basis for
Global
Keio University Digital
President Anzai Cinema
Sony
UCSD NTT
Chancellor Fox SGI
Calit2@UCSD Auditorium

34. OptIPuter Step IV:
Integration of Lightpaths, OptIPortals, and Streaming Media

35. New Year’s Challenge: Streaming Underwater Video
From Taiwan’s Kenting Reef to Calit2’s OptIPortal
My next plan is
to stream stable Remote Videos Local Images
and quality
underwater
images
to Calit2,
hopefully by
PRAGMA 14. --
Fang-Pang to LS
Jan. 1, 2008
March 6, 2008
Plan
Accomplished!
March 26, 2008
UCSD: Rajvikram Singh, Sameer Tilak, Jurgen Schulze, Tony Fountain, Peter Arzberger
NCHC : Ebbe Strandell, Sun-In Lin, Yao-Tsung Wang, Fang-Pang Lin

36. The Calit2 OptIPortals at UCSD and UCI
Are Now a Gbit/s HD Collaboratory
NASA Ames Visit Feb. 29, 2008
HiPerVerse:
First ½ Gigapixel
Distributed
OptIPortal-
124 Tiles
Sept. 15, 2008
Calit2@ UCI wall
Calit2@ UCSD wall
UCSD cluster: 15 x Quad core Dell XPS with Dual nVIDIA 5600s
UCI cluster: 25 x Dual Core Apple G5

37. Command and Control:
Live Session with JPL and Mars Rover from Calit2
Source: Falko Kuester, Calit2; Michael Sims, NASA

38. U Michigan Virtual Space Interaction Testbed (VISIT)
Instrumenting OptIPortals for Social Science Research
• Using Cameras Embedded in
the Seams of Tiled Displays
and Computer Vision
Techniques, we can
Understand how People
Interact with OptIPortals
– Classify Attention, Expression,
Gaze
– Initial Implementation Based on
Attention Interaction Design
Toolkit (J. Lee, MIT)
• Close to Producing Usable
Eye/Nose Tracking Data using
OpenCV
Leading U.S.
Researchers on the
Social Aspects of
Collaboration
Source: Erik Hofer, UMich, School of Information

39. Calit2 Microbial Metagenomics Cluster-
Next Generation Optically Linked Science Data Server
Source: Phil Papadopoulos, SDSC, Calit2
512 Processors ~200TB
~5 Teraflops Sun
1GbE X4500
~ 200 Terabytes Storage and Storage
10GbE
Switched 10GbE
/ Routed
Core

40. CAMERA’s
Global Microbial Metagenomics CyberCommunity
2200 Registered Users From Over 50 Countries

41. OptIPuter Step V:
The Campus Last Mile

42. CENIC’s New “Hybrid Network” - Traditional Routed IP
and the New Switched Ethernet and Optical Services
~ $14M
Invested
in
Upgrade
Now
Campuses
Need to
Upgrade
Source: Jim Dolgonas, CENIC

43. AARNet 10Gbps Access Product is Here!!!
• HD and Other High Bandwidth Applications Combined
with “Big Research” Pushing Large Data Sets Means
1 Gbps is No Longer Adequate for All Users
• AARNet Helps Connect Campus Users or Remote Instruments
• Will Permit Researchers to Exchange Large Amounts of
Data within Australia, and Internationally via SXTransPORT
© 2008, AARNet Pty Ltd 43
Slide From Chris Hancock, CEO AARNet

44. The “Golden Spike” UCSD Experimental Optical Core:
Ready to Couple Users to CENIC L1, L2, L3 Services
Quartzite Communications
To 10GigE cluster Goals by 2008:
Core Year 3
node interfaces
>= 60 endpoints at 10 GigE CENIC L1, L2
>= 30 Packet switched Wavelength
Quartzite Selective
Services
Corewavelengths
.....
Switch
>= 30 Switched
>= 400 Connected endpoints
Lucent To 10GigE cluster
node interfaces and
other switches
To cluster nodes
.....
Glimmerglass
Approximately 0.5 Tbps .....
To cluster nodes
GigE Switch with
Dual 10GigE Upliks
Arrive at the “Optical”
Production
OOO
Switch
To cluster nodes
Center of Hybrid Campus
32 10GigE
.....
Switch GigE Switch with
Dual 10GigE Upliks
Force10
...
To Packet Switch CalREN-HPR
GigE Switch with
Dual 10GigE Upliks other Research
nodes
Cloud
GigE
Funded by
10GigE
NSF MRI Campus Research
4 GigE
4 pair fiber Grant Cloud
Cisco 6509
Juniper T320
OptIPuter Border Router
Source: Phil Papadopoulos, SDSC/Calit2
(Quartzite PI, OptIPuter co-PI)

45. Use Campus Investment in Fiber and Networks
to Physically Connect Campus Resources
HPC System
Cluster Condo
PetaScale
Data Analysis UCSD Storage
Facility
UC Grid Pilot
Digital
Collections Research
Manager Cluster OptIPortal
Research
10Gbps
Instrument
Source:Phil Papadopoulos,
SDSC/Calit2

46. Green
Initiative:
Can Optical
Fiber Replace
Airline Travel
for Continuing
Collaborations
?
Source: Maxine Brown, OptIPuter Project Manager

47. OptIPortals
Are Being Adopted Globally
U Melbourne Today--
And
New Zealand’s First OptlPortal!
AIST-Japan Osaka U-Japan KISTI-Korea CNIC-China
NCHC-Taiwan
Russian
Academy
Sciences
Moscow SARA- Netherlands Brno-Czech Republic
Last Week
Monash University Two Days Ago ANU
EVL@UIC Calit2@UCSD Calit2@UCI CICESE, Mexico
Canberra
U Queensland CSIRO Discovery Center

48. “Using the Link to Build the Link”
Calit2 and Univ. Melbourne Technology Teams
January 15, 2007
No Calit2 Person Physically Flew to Australia to Bring This Up!
www.calit2.net/newsroom/release.php?id=1219

49. UM Professor Graeme Jackson Planning
Brain Surgery for Severe Epilepsy
www.calit2.net/newsroom/release.php?id=1219

50. HD MURPA Talk
to Monash University from Calit2
July 30, 2008
July 31, 2008

51. Smarr American Australian Leadership Dialogue
OptIPlanet Collaboratory Lecture Tour October 2008
AARNet National Network
• Oct 2—University of Adelaide
• Oct 6—Univ of Western Australia
• Oct 8—Monash Univ.; Swinburne
Univ.
• Oct 9—Univ. of Melbourne
• Oct 10—Univ. of Queensland
• Oct 13—Univ. of Technology
Sydney
• Oct 14—Univ. of New South Wales
• Oct 15—ANU; AARNet;
Leadership Dialogue Scholar
Oration, Canberra
• Oct 16—CSIRO, Canberra
• Oct 17—Sydney Univ.

52. AARNet’s “EN4R” –
Experimental Network For Researchers
• For Researchers
• Free Access for
up to 12 months
• 2 Circuits Reserved
for EN4R on Each
Optical Backbone
Segment
• Access to North
America via.
SXTransPORT
52
Source: Chris Hancock, AARNet

53. EVL’s SAGE OptIPortal VisualCasting
Multi-Site OptIPuter Collaboratory
CENIC CalREN-XD Workshop Sept. 15, 2008
EVL-UI Chicago
At Supercomputing 2008 Austin, Texas
November, 2008 Streaming 4k
SC08 Bandwidth Challenge Entry
On site: Remote:
SARA (Amsterdam)
U Michigan
U of Michigan
GIST / KISTI (Korea) UIC/EVL
Osaka Univ. (Japan) U of Queensland
Masaryk Univ. (CZ), Russian Academy of Science
Calit2
Requires 10 Gbps Lightpath to Each Site
Uncompressed High Definition Video From Each Site
Source: Jason Leigh, Luc Renambot, EVL, UI Chicago

54. AARNet’s Roadmap Towards 2012
Source: Chris Hancock, AARNet

55. 21st Century Australian Information Infrastructure:
Joining the Global Data-Intensive Collaboratory
• All Data-Intensive Australian:
– Researchers
– Scientific Instruments
– Data Repositories
• Should Have Best-of-Breed End-End Connectivity
• Today, that Means 10Gbps Lightpaths
• This Requires a Spirited Partnership:
– Federal
– State
– Universities and CSIRO
– AARnet
The Mutuality Principle at Work!