The University of Edinburgh is undergoing a large project to reprocure its campus networking infrastructure. The existing network, which has grown organically over many years, contains equipment that is up to 20 years old and no longer meets the university's needs. After an internal review in 2014 recommended a new network be procured, the university embarked on a multi-stage competitive dialogue procurement process that is still ongoing. The process involves pre-market engagement, shortlisting bidders, and multiple rounds of dialogue and evaluation to refine solutions before selecting a final vendor. The procurement has proven to be a large undertaking but may result in a network solution tailored to the university's unique requirements.
3. IPv6 @ the STFC
Philip Garrad,
Science and Technology
Facilities Council (STFC)
4. • What is STFC
• What IPv6 work had we completed before Sept 2016
• What happened in September 2016
• How we deployed IPv6 in 6 months
• What have we learnt from this
Overview
5. One of Europe’s largest multi-disciplinary scientific research organisations
Science and Technology Facilities Council
World class research, innovation and skills
• Broad range of physical, life and computational sciences
• Around 1,700 scientists in particle and nuclear physics, and astronomy
• Access for 7,500 scientists to world-leading, large-scale facilities
• Science and Innovation Campuses at Daresbury and Harwell
• Globally-recognised capabilities and expertise in technology R&D
• Inspiring young people to undertake STEM
6. One of Europe’s largest multi-disciplinary scientific research organisations
Science and Technology Facilities Council
UK Astronomy Technology Centre
Edinburgh, Scotland
Polaris House
Swindon, Wiltshire
Chilbolton Observatory
Stockbridge, Hampshire
Daresbury Laboratory
Sci-tech Daresbury Campus, Liverpool City Region
Rutherford Appleton Laboratory
Harwell Didcot, Oxfordshire
Boulby Underground
Laboratory
North Yorkshire
7. Large Hadron Collider @CERN
Science and Technology Facilities Council
Credit: CERN
Recreating the conditions a billionth of a second after the Big Bang
LHC research also delivers wide benefits for society through new
technologies, including cancer therapies, medical and industrial
imaging and manufacturing processes
8. IPv6 Address plan, devised by Robin Tasker for STFC.
STFC’s IPv6 address plan
IPv6 Status in Sept 2016
2001 0630 0058 a b c d 0000 0000 0000 0000: : : : : : :
a = STFC Address plan version (16 chances)
b = Network Type
c = Network Subtype
d = Locally assigned
NETWORK HOST
JANET RAL a b c d 0000 0000 0000 0000: : : : : :
9. • “Cheap” IPv6 firewall connected to border router
• 3x IPv6 hosts connected directly to the firewall
• Provided connectivity to allow interested parties
access to an IPv6 network.
• Take up was limited to the RAL Tier 1 group.
Small IPv6 Test installation
IPv6 Status in Sept 2016
10. • 20th September 2016 - WLCG management board
accepted the HEPiX IPv6 working group proposal to
allow sites to deploy IPv6 only CPU by April 2017.
• This mandated LHC Tier 1s to provide some services
as dual stack by April 2017
• STFC needed to deploy production quality IPv6
network at RAL within 6 months.
Worldwide Large Hadron Collider Computing Group (WLCG) meeting.
What happened in Sept 2016?
11. • Step 1 – Wait 2 months…
• Allowed for some scheduled network upgrades to complete
• Various pieces of non IPv6 capable hardware replaced
• Step 2 – Realise Christmas is coming
• Don’t make significant network change - site shuts for 10 days
• Prepare by upgrading the firmware on all devices
• Step 3 – Plan how to deploy IPv6 in 3 months
How to deploy IPv6 in 6 months?
12. Starting Point – September 2016
Starting point – Limited IPv6 in the RAL network Philip Garrad - STFC
23. • LHC Tier 1 @ RAL have been running IPv6 for a year
• IPv6 is being deployed to LHC Tier 2 @ RAL now
• STFC Network group need to run their own IPv6
subnet to gain better understanding of IPv6
• Auto configuration, Security, logging, auditing.
March 2018
Still to do… (Quite a lot)
24. • Don’t get hung up on a numbering scheme
• Clearly it would have been less stressful to deploy
IPv6 to our timetable
• Moving through the network top to bottom simplified
the problem
• Starting small made things easier
• Even without an IPv6 capable IPAM system you can
deploy IPv6 – use the IPv4 address
Lessons Learnt
30. > The University of Edinburgh has a large computer network
> The network has grown organically over the years
– We’ve tried to be economical with equipment
– “Technical debt” - network needs to be refreshed
> We’re part way through the reprocurement
– This may be the most frustrating talk you
hear today
Summary
32. Oops! That was 2017.
In 2018 we're 23rd=.
A slide from our Bidders Conference
Perhaps we should
buy a new network or
something.
33. >…apparently this kind of marketing makes network
providers give you things cheap. Who knew?
I’m a techy, but…
34. > It’s not all about technology
> “Digital Transformation” – everything is digital
– Data networking is fundamental
> Data Driven Innovation
– High capacity
– Rapid change
> Student Experience
– High quality WiFi
– Stability and performance
Business drivers
36. > Plan installation of LAN across campus
> 10 Mbps Ethernet, 100 Mbps FDDI, 2 Mbps
serial
– Thick and thin ethernet
– Extensive fibre installation
> 1000 hosts, growing to 3000 hosts over
~5 years
> One of first two UK Universities to buy
Cisco routers
> IP(v4), IPX, AppleTalk, DECNet Phase IV,
bridging (for Pink Book)
Indulge me – 1988-1990
37. > 2014: New director of IT Infrastructure
– First task: infrastructure review
– Some kit up to 10 (20?) years old
– No budget for renewal
> Network has grown to >100K outlets
– Integral to University business, operations
and strategy
– Phones, WiFi, BMS, CCTV, Things…
> In the interim network budget -> Estates
– Building refurb cycle (10-20 yr) != network
replacement cycle (3-5 yr)
Fast forward to 2014
Oops!
39. We’re a University; we have committees and projects and such
Well maybe not quite yet...
2014: internal infrastructure review
2016: PTS Consultants – consider network strategy
Nov 2016: PTS report – “buy a new network”
“we need a strategy” – set up a project
May 2017: Budget approved
40. > 5 main campuses
– University estate approx. 4 x 10 miles (or 10 x 10)
– 220+(ish) buildings
– Mainly owned or leased fibre
> Over 100,000 outlets
– 30K+ active ports
– 10M, 100M, 1G, 10G, 40G
– 2700 switches, 2700 WAPs
– 600+ racks
– ~13 core and distribution routers
> Constant churn
– 10 year building programme
Scale
41. > Wired and wireless networks
> Data centre firewalls
> Wireless firewall
> Other security features
> Management and monitoring
> Automation/Orchestration
> Future expansion
> Professional services
> 5+2+2+2+2 year contract
Out of scope
Scope
In scope
> Data centre networking
> Upgrade or replacement of
cabling unless essential for
solution
> Wireless p2p links
> Student residences
> (Perimeter firewall)
42. > Budget: [REDACTED]
> Project Board: senior representatives across UoE
> Project Team: ~7 core members
– Director ITI
– 2 x Communications Infrastructure Section
– Procurement Officer
– Project Manager
– 2 x PTS Consultants
> SMEs: Security, Academic computing, Lawyer (~10)
Now can we start the procurement, Daddy?
43. > Multiple rounds of dialogue – not just an ITT
> “Down select” at each round to reduce field
– Kind of knockout stages
> Drive down price
> Co-design?
– collaborate on what’s possible/available
– range of potential solutions
– beware of steering bidder and then rejecting
Procurement by Competitive Dialogue
44. > There are things I can’t tell you:
– because we’re only part way through the
process and we don’t know yet;
– because they’re confidential –
commercially sensitive or NDA;
– because we’ve decided not to tell anyone
– like the budget
Frustrations
45. > Premarket engagement
> OJEU Contract Notice
> ESPD
– Like PQQ – select top 10
> ITPD
– 10 provide outline solution – select 5
> ITCD
– Lengthy dialogue
– 5 provide developed solution – select 3
> ITCD 2
– Select 2
> ITT
– Select 1
Programme
YOU ARE
HERE!
46. > June-July 2017: Premarket engagement
– 6 manufacturers selected by consultants
– Alcatel Lucent, Cisco, Dell EMC, Extreme, HPE, Huawei
– ½ day presentations
– Own research
> What's on the market?
– Pure L2 or pure L3 backbone? Y
– VLANs or no VLANs? Y
– Overlays, tunnels, data/control plane...? Y
– NAC for security or configuration? Y
– Automation and orchestration? Y
– SDN (some or all of the above)? Y
– Firewalls and security features? Y
– You name it…
Pre-market engagement
?!
47. > September 2017:
– Backward looking scored Qs
– Includes draft ITPD document so they know what's coming
> About 20 responses
> Mid October: Choose top 10 to receive ITPD
– Formal scoring
> No, I can’t tell you who any of them were
Contract Notice and ESPD
49. > Responses received early December
– 10 x ((10 + 14 pages) + MRs + pricing spreadsheet)
> ½ day meetings with bidders to present their outline solutions
and answer questions
– Includes manufacturers
– NOT scored
> Answers scored
> Sorry letters and ITCD invitations issued 22 Dec
– We spoiled everyone’s Christmas
> You think that sounds like a lot of work?
ITPD Responses
50. > Mid-Jan to late Feb 2018
> 2 + 1 day with each of 5 bidders
– Discuss outline solution
– Questions arising from solution
– (Some) ways that the University is different from the model
– Bidders may change original solution and costings
> Generate ITCD document with scored Qs
– Expanded from ITPD brief and Qs
– But ask original Qs again - make sure the answers
haven't changed
> Responses required
– 10 page developed solution
– 30 page scored questions (8+22)
– Pricing spreadsheet
> Issued 21 Feb 2018
> Responses 14 March 2018
ITCD
51. > ITCD 2 (3->2) – March-June 2018
> ITT (2->1) – July-September 2018
> Selection and award – September-November 2018
> Implementation – November 2018 for 18 months
> Contract – 5+2+2+2+2 years
And next...?
52. > Budget: [REDACTED]
> 20 initial responses: [REDACTED]
> 10 ITPD bidders: [REDACTED]
> 5 ITCD bidders: [REDACTED]
> 3 ITCD bidders: [REDACTED]
> Solutions we’ve been offered? [REDACTED]
> Leaning toward any particular outcome? Not redacted - because
we have to judge all bids on their merits.
The frustrating bit summarised
54. > Competitive dialogue is a **lot** of work
– ITCD had 7 staff almost full time for 6 weeks, not
including scoring
– Also for the bidders
> We might have too many stages
– Each stage takes a lot of effort (info pack, dialogue,
evaluation, down selection)
– 10 - 4 - 2 - 1 rather than 10 - 5 - 3 - 2 - 1?
> The process can change your views
– I can’t tell you how. Sorry
Is there anything we'd do differently next time?
55. End of part 1
(If you want to see part 2 you’ll have to invite us back
next year)
57. > ESPD: European Single Procurement Document
– Like PQQ – backward looking “have you done this before?”
> ITPD: Invitation To Participate in Dialogue
– Outline solution based on model design
> ITCD: Invitation To Continue in Dialogue
– After down selection
> ITT: Invitation To Tender
Alphabet soup
Introduce yourself – Philip Garrad – Network Technical Architect at STFC
A background slide from our Bidders Conference. A population of 50,000 makes us comparable with most large towns in Scotland, approximately 1/10 of the population of Edinburgh.
Digital services, digital educators, digital students, evidence based decision making, digital skills for doing all of that.
The planning team was Scott Currie and me.
We may have been the first UK Uni to actually *buy* Cisco routers. Image credit: pinterest.com.
List of protocols included for nostalgia.
The new director is Tony Weir
In the interim someone though it would be a wizard wheeze to get network installation off the Computing Service’s budget and integrated into Estates.
This was basically a good idea. It stopped the two-stage issue of new build and refurbished buildings being planned and then the occupants saying, “ah, but we’d like a network now”. Integrate the network at the design stage and make it infrastructure.
Unfortunately network kit should be replaced on a 3-5 year cycle (or whatever) but buildings may not be refurbed for 20 years.
At the time of writing we have 10 3Com 3300 switches in service; I can’t find a date for them but the latest documentation says copyright 1999. We also have one length of thick Ethernet.
All these numbers are in constant churn.
Map is about 12 miles across
Perimeter firewall is out of scope but we might be persuaded.
This where it starts to get frustrating.
ESPD: European Single Procurement Document
Like PQQ – backward looking “have you done this before?”
ITPD: Invitation To Participate in Dialogue
Outline solution based on model design
ITCD: Invitation To Continue in Dialogue
After down selection
ITT: Invitation To Tender
Picture credit <https://en.wikipedia.org/wiki/Pyramid#/media/File:Upside_down_Pyramid,_Bratislava_02.jpg>
Another Bidders Conference slide. I’ve included it because…
Another slide from the Bidders Conference. Note the vision statement – responsive, resilient, adaptive, consistent, security, flexibility…