1. CS 704DAdvanced Operating System(Distributed Computing) Debasis Das

2. Distributed Computing Environment(DCE) MIT CS704D Advanced OS Class of 2011 2

4. Set of services & tools that run on existing systems

5. Provide s the base for distributed applications

6. OS on which the DCVE can be ported easily include

7. OSF/1, AIX

8. Domain OS, ULTRIX

9. HP-UX, SINIX

11. TCP/IP, X.25 etc11/21/2009 IT703D: Distributed Computing Debasis Das

12. DCE Applications DCE software Operating systems & Networking 11/21/2009 IT703D: Distributed Computing Debasis Das DCE Structure

14. Academic institutions were asked to contribute working code; almost entirely in C

15. OSF checked and integrated the code as a single package

16. First release 199211/21/2009 IT703D: Distributed Computing Debasis Das

18. Programming model for concurrent applications

19. Create & controls threads within an application

20. Synchronization of global resource access.

21. Remote Procedure call (RPC) facility

22. Client-server communications

23. Network and protocol independent

24. Automatic data conversions

25. Distributed Time Service (DTS)

26. Clock synchronizing mechanisms, NIST& Other sources11/21/2009 IT703D: Distributed Computing Debasis Das

28. Cell directory services (CDS)

29. Global Directory Services (GDS)

30. Global directory Agent (GDA)

31. Servers, files, devices can be uniquely named and found without being location dependent

32. Distributed File Systems (DFS)

33. System-wide file system

34. Location transparency, high performance and high availability

35. File services to clients of other file services

36. Security Service

37. Authentication & authorization to protect against illegitimate access11/21/2009 IT703D: Distributed Computing Debasis Das

38. Component Components using the service Threads None RPC Threads, name, security DTS Threads, RPC, name, security Name Threads, RPC, DTS, security DFS Threads, RPC, DTS, name, security Security Threats, RPC, DTS, name 11/21/2009 IT703D: Distributed Computing Debasis Das DCE Components Interdependence

40. Concept cells helps make it manageable and scalable

41. A cell is

42. A group of users, machines and other resources

43. Common purpose, share common DCE services

44. Min cell configuration requires

45. A cell directory service, a security server, a distributed time server and 1 or more client machines

46. Each client must have client processes for security service, cell directory service, distributed time service, RPC facility and threads facility

47. Will have a distributed client process if a distributed file server exists within cell11/21/2009 IT703D: Distributed Computing Debasis Das

49. Administration

50. Security

51. Overhead11/21/2009 IT703D: Distributed Computing Debasis Das

53. Product oriented or function oriented

54. Product oriented: one cell for one product

55. Function oriented: one function for all products in a cell. Design dept for a product company for example 11/21/2009 IT703D: Distributed Computing Debasis Das

57. Administrator needs to know resources in the cell, users

58. Cell may be decided by the administrative reach manageable by an administrator11/21/2009 IT703D: Distributed Computing Debasis Das

60. Cell boundaries can act as a firewall

61. Require additional authentication/authorization for distant cell resources11/21/2009 IT703D: Distributed Computing Debasis Das

63. These overheads cause time and performance penalties

64. So cell boundary should encompass most of the resources used by the particular group of users11/21/2009 IT703D: Distributed Computing Debasis Das

65. Message Passing 11/21/2009 IT703D: Distributed Computing Debasis Das

67. Copy sharing or message passing approach11/21/2009 IT703D: Distributed Computing Debasis Das

68. Shared Memory area P1 P2 Shared Data P2 P1 Message Passing 11/21/2009 IT703D: Distributed Computing Debasis Das

70. Message passing favored

71. Simple primitives

72. Send

73. Receive11/21/2009 IT703D: Distributed Computing Debasis Das

75. Uniform Semantics

76. Efficiency

77. Reliability

78. Correctness

79. Flexibility

80. Security

81. Portability11/21/2009 IT703D: Distributed Computing Debasis Das

83. Straightforward to construct new applications communicating via primitives

84. Send messages to distributed parts of an application without worrying about the network & other complexities

85. Clean & simple semantics11/21/2009 IT703D: Distributed Computing Debasis Das

87. Local communication

88. Remote communication

89. Very important that these modes be exactly similar or close11/21/2009 IT703D: Distributed Computing Debasis Das

91. Avoid the cost of establishing and the closing connections for every message

92. Minimizing the cost of maintaining connections

93. Piggybacking of acknowledgement for previous message through the next message from the receiver11/21/2009 IT703D: Distributed Computing Debasis Das

95. Must be able to guarantee delivery, guard against lost packets through ack and retransmission (lost packets)

96. Duplicate packets handling, via sequence numbers etc.11/21/2009 IT703D: Distributed Computing Debasis Das

98. Sender to multiple receivers

99. One receiver’s ability to receive from multiple senders

100. Properties required

101. Atomicity: delivered to all of the receivers or none

102. Ordered delivery: delivered in proper order

103. Survivability: delivered despite partial failures of processes, machines and links11/21/2009 IT703D: Distributed Computing Debasis Das

105. Types & levels of correctness required can be flexible

106. Flexibility to permit any kind of control flow; including synchronous and asynchronous send/receive primitives11/21/2009 IT703D: Distributed Computing Debasis Das

108. Authentication of receiver by sender

109. Encryption of message before transmission over the network11/21/2009 IT703D: Distributed Computing Debasis Das

111. The application that uses the IPC primitives should be portable. IPC will need to take care of differences due to heterogeneity of systems. The higher level message passing primitives need to hide the differences.11/21/2009 IT703D: Distributed Computing Debasis Das

113. Address

114. Sending and receiving process addresses

115. Sequence number

116. Message id, useful to detect lost packets or duplicates

117. Structural information

118. Type & length of message (internal or external)11/21/2009 IT703D: Distributed Computing Debasis Das

119. Structural Information Actual data/ Pointer to data Addresses Sequence Number/ Message id Type No. of Bytes/ elements Receive Process addr Send Process addr 11/21/2009 IT703D: Distributed Computing Debasis Das Typical Message Structure

121. Who’s the receiver?

122. One or more receivers?

123. Is the message guaranteed to have been accepted by the receiver?

124. Should the send wait for a reply?

125. Action required in case of a catastrophic failure of node/link

126. What action if receiver not ready? Buffer? What if buffer is full?

127. What if more than one message is outstanding? Can receiver accept messages out of sequence?11/21/2009 IT703D: Distributed Computing Debasis Das

129. Blocking

130. Process gets blocked/suspended after executing send or receive primitives

131. Non blocking

132. Process can continue even after executing a send/receive primitive

133. Question How does the blocked receiver know when a message has arrived?

134. Polling

135. Interrupts11/21/2009 IT703D: Distributed Computing Debasis Das

137. Return the message if available or

138. Indicate that there are no messages11/21/2009 IT703D: Distributed Computing Debasis Das

139. Synchronous Communication with Blocking Send/Receive Primitives Receive process message Reply Sending process Blocked here Unblocked here 11/21/2009 IT703D: Distributed Computing Debasis Das

141. Assured reliability

142. No backward error recovery is necessary if a message is lost or undelivered11/21/2009 IT703D: Distributed Computing Debasis Das

144. Blocking receive can get blocked if the message is never received11/21/2009 IT703D: Distributed Computing Debasis Das

146. Synchronous communication does not need a buffer

147. Asynchronous communication would need unbounded set of buffers

148. Something in between

149. Single message buffer

150. Multiple messages or bounded buffer11/21/2009 IT703D: Distributed Computing Debasis Das

151. Null Buffer Single buffer 1 2 3 Receiving processes Sending processes Multiple buffer n 11/21/2009 IT703D: Distributed Computing Debasis Das Buffering Mechanisms

153. Alternative 2: send message, start a time out. Send again on time out. Retry for n number of times.

154. Single copy from send process address space to receive process address space.11/21/2009 IT703D: Distributed Computing Debasis Das

156. Avoids delays in a networked/distributed system

157. In synchronous mode at most one message can be outstanding

158. Either in kernel or application address space11/21/2009 IT703D: Distributed Computing Debasis Das

160. Unbounded buffers required to ensure all the messages sent are received, even if several may be outstanding11/21/2009 IT703D: Distributed Computing Debasis Das

162. Strategy to manage

163. Unsuccessful communication

164. Return an error message when buffer is full

165. Makes message passing unreliable

166. Flow controlled communication

167. Sender is blocked until some messages are accepted by the receiver

168. True asynchronous nature is lost

169. Can cause unpredictable deadlocks11/21/2009 IT703D: Distributed Computing Debasis Das

171. Create-buffer like system calls can be used

172. Creating buffers in kernel space may limit the buffers

173. Creating buffers in the application space could be more flexible. Memory allocation process will allocate memory

174. Better concurrency & flexibility but difficult to design11/21/2009 IT703D: Distributed Computing Debasis Das

176. Any message larger than a MTU will have to multiple pieces or multiple datagram

177. There’s an additional process of assembling the message

178. Many a time ca introduce a reliability issue complicating communication11/21/2009 IT703D: Distributed Computing Debasis Das

180. Absolute pointers lose meaning. For example a tree object with all the node values and information about structure is required. The sender and receiver may be different architectures. Receiver needs to reconstruct

181. Program objects have different sizes. Receiver needs to know the object type

182. So you need encoding at the sender and decoding at the receiver.11/21/2009 IT703D: Distributed Computing Debasis Das

184. A tag along with the data item, describes the data item

185. Example: ASN.1 CCITT standard, Mach distributed OS

186. Untagged representation

187. Receiver knows the structure so that it can reconstruct the data

188. Example: SUN XDR eXternal Data Representation11/21/2009 IT703D: Distributed Computing Debasis Das

190. Process id used as a parameter in the communication primitive

191. Send(process_id, message)

192. Receive(process_id, message

193. Implicit address

194. Send_any(service_id, message) functional addressing

195. Receive_any(process_id, message)11/21/2009 IT703D: Distributed Computing Debasis Das

197. 32 bit IP address as machine_id, 16 bit local process id

198. No global co-ordination required

199. Process cannot be transferred for load balancing or any other reason11/21/2009 IT703D: Distributed Computing Debasis Das

201. The third parameter is the latest known location of the process

202. A link that has original machine id and the new machine id is left on the machine.

203. On the new machine a mapping table points to the new local_id against the old id.11/21/2009 IT703D: Distributed Computing Debasis Das

205. The chain could be broken and process not locatable, if an intermediate node is down.

206. Location transparency failsProblem of poor reliability, scalability 11/21/2009 IT703D: Distributed Computing Debasis Das

208. Use name servers

209. Has table with high level id and corresponding low level id

210. High level id is some ASCII string

211. Low level id is machine_id@local_id format

212. Problems of poor reliability and scalability11/21/2009 IT703D: Distributed Computing Debasis Das

214. But, it is prone to partial failures

215. Node crashes

216. Communication link failuresLeads to following problems 11/21/2009 IT703D: Distributed Computing Debasis Das

218. Link failure, receiver node down

219. Loss of response message

220. Link failure, sender node down

221. Unsuccessful execution of the request

222. Receive node crashed during processing11/21/2009 IT703D: Distributed Computing Debasis Das

223. Request Ack Reply Ack Client Server 11/21/2009 IT703D: Distributed Computing Debasis Das 4 Message IPC Protocol

224. Request Reply Ack Client Server 11/21/2009 IT703D: Distributed Computing Debasis Das 3 Message IPC Protocol

225. Request Reply Client Server 11/21/2009 IT703D: Distributed Computing Debasis Das 2 Message IPC Protocol

227. Non-indempotent routines can cause problems

228. Avoid operation of non-indempotent routine if the request is a duplicate

229. Number the requests, cache the replies

230. When request received, check if a reply already exists

231. Send the earlier reply again if it exists11/21/2009 IT703D: Distributed Computing Debasis Das

233. One is for size of full message

234. A bitmap indicating position of the packet in the sequence

235. Receiver sends a bitmap indicating packets not received after timeout

236. Sender sends the missing packets in “selective repeat”

237. When reception completed, receiver sends an ack for the message11/21/2009 IT703D: Distributed Computing Debasis Das

239. Many to one (multiple senders, one receiver)

240. Many to many (multiple senders, multiple receivers)11/21/2009 IT703D: Distributed Computing Debasis Das

242. Broadcast- multicast to ALL receivers

243. Issues

244. Group Management, and addressing

245. Message delivery to receiver processes

246. Buffered and un-buffered multicast

247. Send to all and bulletin board semantics

248. Flexible reliability in multicast communication

249. Atomic Multicast

250. Group Communication primitives11/21/2009 IT703D: Distributed Computing Debasis Das

252. Open and closed groups

253. Flexibility required to accommodate both

254. Group creation/deletion, membership of groups

255. Group servers can manage that

256. Poor reliability & scalability

257. Replication is possible but consistency maintenance is an issue/overhead11/21/2009 IT703D: Distributed Computing Debasis Das

259. Top level is a ASCII

260. Lower level quite often underlying hardware dependent

261. On networks with multicast addresses, use the same

262. Or use broadcast address

263. If machines are not on same LAN, one to one messages will have to be sent. Lower level address then is a list of machines running processes in the group11/21/2009 IT703D: Distributed Computing Debasis Das

265. Sender kernel contacts group server that has lower level id and the process ids of members of the group

266. List of process ids inserted into message

267. If the low level id is a multicast/broadcast address, one message goes to the address.

268. However if it is a list of id, the message is sent to each machine

269. If in a broadcast group, a machine may find message irrelevant, discards it

270. Sender unaware of group size11/21/2009 IT703D: Distributed Computing Debasis Das

272. It is unrealistic to expect the sender to be blocked until all the group members have received the message

273. Sending process is unaware of how many members are there

274. Un-buffered

275. Receivers not ready, lose the message

276. Buffered

277. Message is buffered, eventually everyone will receive the message11/21/2009 IT703D: Distributed Computing Debasis Das

279. Bulletin board: send message to a channel. Receivers copy off the message and act on it

280. Avoids problems as follows

281. Receiver would read and act only when it is in an appropriate state

282. Sender will receive responses only when it is in right state.

283. Message may be withdrawn when service is no longer required11/21/2009 IT703D: Distributed Computing Debasis Das

285. 0-reliable: No response required, e.g. multicast of time info

286. 1-reliable: exactly one response is required, e.g. a bid situation

287. M-out-of-n reliability: m that is greater than 1and less than n response required, e.g. majority voting for consistency check

288. All-reliable: all responses required, e.g. files copied to file servers11/21/2009 IT703D: Distributed Computing Debasis Das

290. Usual time out repeats until all have been acknowledged

291. Fails if either the sender or one or more receivers fail during the process

292. One way around (Tannenbaum) is for every receiver to do an atomic multicast of the message received

293. Eventually all will receive message. Use in rare cases11/21/2009 IT703D: Distributed Computing Debasis Das

295. However, to find the lower level address there could be confusion. What should be contacted name server or a group server?

296. Separate primitives like send and send_group could allow send_group to have parameters such as level of reliability and atomicity11/21/2009 IT703D: Distributed Computing Debasis Das

298. Non-determinism is an issue

299. Not known which of the sender may have the right information for the receiver first

300. Sometimes it is useful to have a group of senders to be controlled dynamically

301. For example allowing producers to write only when buffer is not full

302. More of a programming issue rather than a OS issue11/21/2009 IT703D: Distributed Computing Debasis Das

304. Besides ordered delivery of messages is important in many to many case

305. Message sequencing is required. One to many case, this is trivial.

306. In many to one the receiver has to order the messages received

307. In many to many message can arrive at different times at different nodes, due to different delays seen over LAN/WAN11/21/2009 IT703D: Distributed Computing Debasis Das

309. Consistent Ordering

310. Causal Ordering11/21/2009 IT703D: Distributed Computing Debasis Das

312. Keep the clocks at every node synchronized

313. Use a time window to deliver messages from queue

314. A message with lower time stamp may arrive later

315. The time window can take care of that11/21/2009 IT703D: Distributed Computing Debasis Das

317. One method of many to many is to break up the scheme in may to one and one to many parts.

318. Senders send messages to one receiver that sequences the messages(called sequencer)

319. The sequencer multicasts to all the other designated receivers

320. Each receiver accepts the messages in a buffer. Delivers only if there are no gaps in the sequence, else waits for the other messages to arrive11/21/2009 IT703D: Distributed Computing Debasis Das

322. ABCAST protocol is better, it is a distributed algorithm and hence not subject to a single point failure

323. Sender assigns a sequence number greater than any assigned earlier, a simple counter will do. Send to all members of the group.

324. All the members return a proposed sequence number based on max(Fmax, Pmax)+1+i/N, i is the member no., N is the total no. of members, Fmax is the largest sequence no. proposed by the group, Pmax is the max no. proposed by the member

325. Sender choose the largest proposed no. and lets everyone know as the committ no.

326. All members use this final committed no as the sequence no. Delivery to applications use this no.11/21/2009 IT703D: Distributed Computing Debasis Das

328. If there is an “happened-before” relationship the messages must be delivered in order, otherwise order need not be maintained11/21/2009 IT703D: Distributed Computing Debasis Das

330. A vector of n is maintained, value at each position is the sequence no. of last message in sequence received from that node

331. When sending message, increment own element in vector and send the vector along with message

332. Message buffered in runtime system and tests done

333. Delivered to receive process if it passes both tests

334. S[i]=R[i]+1, S[j] is less or eq to R[j] all j not eq is11/21/2009 IT703D: Distributed Computing Debasis Das

335. D B C A 3 2 5 1 2 2 5 1 3 2 4 1 3 2 5 1 Fails first test A[1]=C[1] + 1 Fails second test at A[3] <= D[3] 11/21/2009 IT703D: Distributed Computing Debasis Das Causal Ordering- Example