This document discusses real-time data processing and streaming technologies. It provides links to resources about real-time computing, streaming data, and Apache streaming frameworks like Storm and Spark Streaming. Key topics covered include exactly-once processing, distributed fault tolerance, and building real-time systems that can respond within targeted response times using streaming data pipelines.

1. 7FWF R F W HW
(). (, 0

2. http://readme.skplanet.com/?p=12465

3. https://voltdb.com/sites/default/files/real-time-data-report.pdf 2015 Real-time Data Report by VoltDB
b 효과 같은 a
b 능력 다른 a

4. b A FP WN
b f4
r w 4
마감 시간(Deadline)
) y4 (( y4
https://www.soasta.com/blog/23-stats-mobile-web-performance-monitoring/
https://en.wikipedia.org/wiki/Real-time_computing
b ? F A FP WN 4
F I
N
BS W

5. 4 4
BW F 6FWHM
Unbound Bound
Data Data
X X
v
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

6. >NH S GFWHM
Unbound Data h e
작은 배치 처리
끊임없이

7. 5B5 d 실시간으로 l l
” 목표 응답 시간
b s
고객을 잃기 전까지
실시간 1 목표
스트림 1 방식
http://bigdatapage.com/4-really-real-meanings-of-real-time/

8. https://tsicilian.wordpress.com/2015/02/16/streaming-big-data-storm-spark-and-samza/
BTF n m
A8 = A FI 8YFP NRW =SST m
E5A?
e
BHFPF
Exactly-onceb 목표 10초 이내

9. 사용 4
좋아 4
https://twitter.com/jaykreps/status/568499167686995969
https://twitter.com/miguno/status/568503505113247744
Storm vs. Spark Streaming
좋아 4
사용 4 w x

10. https://databaseline.wordpress.com/2016/03/12/an-overview-of-apache-streaming-technologies/
s o (). (

11. 분산 g w
장애 !
m t
u l

12. 분산 장애 부분 전송 성공
At-most-once 1
At-least-once 1
Exactly-once 1

13. 8 FHWP SRH
)
8 FHWP SRH
https://twitter.com/mathiasverraes/status/632260618599403520
8 FHWP SRH
M F SRP W S MF I T SGP NR IN W NG W I W 1
8 FHWP SRH I PNY
) : F FRW I S I S FL
8 FHWP SRH I PNY

14. 8 FHWP SRH
m t
u l
Idempotent exactly-once
1 in d
2
2 5W P F W SRH 8 FHWP SRH
Transactional exactly-once
1 “ SPP GFH
“ 76 s

15. YARN
HDFS
Rake
Server
Query
Cache
AppServer SQL
thin
JDBC client
Apache
Kafka
Rake
Library
Rake
API
Logs
LogAgent
HTTPS
Phoenix
JDBC client
(select)
JDBC client
(upsert)
Kafka
Consumer
Kafka Producer
Collector
6=8

16. 6=8
) - g , g

17. YARN
HDFS
Web Server HQL
Apache
Kafka
Kafka
Consumer
Kafka Producer
Kafka
Producer
Collector
YARN
Hive
Tez
)) ⇔

18. )) ⇔
D G
) - g
>SGNP
/ g
D G ..((g
>SGNP .,((g
)) D3>( >3D
Web
Mobile

19. 수신, 처리, 저장 k
부분 8 FHWP SRH
전체 8 FHWP SRH
데이터 원천 비즈니스 행동 i t
목표 응답 시간 내 반응
실시간 시스템 구축 s c
데이터 파이프라인 배치 처리 pb
l 스트림 처리 p

20. Appendix

21. https://lobste.rs/s/ecjfcm/why_is_exactly-once_messaging_not_possible_in_a_distributed_queue/
8 FHWP SRH delivery N impossible.
8 FHWP SRH processing of messages N possible
N WM T SH NRL HFR G FI idempotent.
8 FHWP SRH
8 FHWP SRH
NI TSW RW
8 FHWP SRH

22. http://readme.skplanet.com/?p=10170