A look at using aggregation as a first class construct within operating systems to enable scaling applications and services.

1. Systems Support for Many Task Computing Holistic Aggregate Resource Environment Eric Van Hensbergen (IBM) and Ron Minnich (Sandia National Labs)

2. Motivation

5. Distribute system services among compute nodes

6. Leverage aggregation as a first-class systems construct to help manage complexity and provide a foundation for scalability, reliability, and efficiency.

8. Compute Node Linux

9. ZeptoOS

10. Kittyhawk

12. Pleasantly portable

14. Local & remote resources accessed via simple API

15. Each thread can dynamically organize local and remote resources via dynamic private namespace

16. Everything Represented as File Systems Console, Audio, Etc. Wiki, Authentication, and Service Control Process Control, Debug, Etc. Hardware Devices System Services Application Services Disk Network TCP/IP Stack DNS GUI /dev/eth0 /net /arp /udp /tcp /clone /stats /0 /1 /ctl /data /listen /local /remote /status /net /cs /dns /win /clone /0 /1 /ctl /data /refresh /2 /dev/hda1 /dev/hda2

17. Plan 9 Networks Internet High Bandwidth (10 GB/s) Network LAN (1 GB/s) Network Wifi/Edge Cable/DSL Content Addressable Storage File Server CPU Servers CPU Servers PDA Smartphone Term Term Term Term Set Top Box Screen Phone )‏ )‏ )‏

18. An Issue of Scale Chip BG/p – 4 way System 72 Racks Node Card (4x4x2) 32 compute 0-2 IO cards Compute Card 2 chips Rack 32 Node Cards

19. Aggregation as a First Class Concept Local Service Aggregate Service Remote Service Proxy Service Remote Service Remote Service

20. Issues of Topology

22. Local cache mode

23. Collaborative cache mode

24. Designated cache server(s)

25. Integrate replication and redundancy

27. Leverage natural topology of machine where possible.

30. Allow for localized monitoring and control as well as high-level global reporting and control

31. Explore both push and pull methods of modeling

32. Based on experiences with supermon system.

34. Allows scheduling of new work from within the cluster, using localized as well as global scheduling controls.

35. Can allow for more organic growth of workloads as well as top-down and bottom-up models.

36. Can be extended to allow direct access from end-user workstations.

37. Based on experiences with Xcpu mechanism.

40. CNK emulation environment

43. Charles Forsyth (Vitanuova)

44. Jim McKie (Bell Labs)

45. Ron Minnich (Sandia National Labs)

46. Eric Van Hensbergen (IBM Research)

48. Questions? Discussion?

55. Networking tightly tied to sockets & TCP/IP w/long call-path

56. Typical installations extremely overweight and noisy

59. Few Linux projects were even invited to submit final proposals

60. FTQ on BG/L IO Node running Linux

61. FTQ on BG/L IO Node Running Plan 9

65. Easy to get 512 and 1024 node configurations

66. Up to 8192 nodes available upon request internally

68. Variety of interconnects allows exploration of alternatives

69. Embedded core design provides simple architecture that is quick to port to and doesn't require heavy weight systems software management, device drivers, or firmware

72. Virtual cut-through hardware routing

73. 1.4Gb/s on all 12 node links (2.1 GB/s per node)

74. 1 µs latency between nearest neighbors, 5 µs to the farthest

75. 4 µs latency for one hop with MPI, 10 µs to the farthest

76. Communications backbone for computations

78. Reduction operations functionality

79. 2.8 Gb/s of bandwidth per link

80. Latency of one way tree traversal 2.5 µs

81. ~23TB/s total binary tree bandwidth (64k machine)

83. Active in the I/O nodes (1:64)