The document discusses using a collaborative approach and distributed event processing platform called Agilis to detect stealthy port scans across multiple organizations. It describes how a stealthy scan works and how collaborating organizations can share network traffic data in a "semantic room" to identify scanners that target only a small number of ports at each location. The Agilis platform is able to process large amounts of real-time data in parallel to detect such attacks with low latency even when the workload varies over time. A demonstration of the system detected a stealthy scan within 700 seconds using traffic from 8 machines simulated to represent different collaborators.
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
AGILIS: An On-Line Map Reduce Environment for Collaborative Cyber Security
1. AGILIS: an on-line map reduce environmentfor collaborative security MIDLAB Middleware Laboratory Sapienza Università di Roma Dipartimento di Informatica e Sistemistica Roberto Baldoni UniversitàdegliStudidi Roma “La Sapienza” baldoni@dis.uniroma1.it, http://www.dis.uniroma1.it/~baldoni/ Prin Meeting - San Vito diCadore Joint Work with IBM Haifa in the contextofCoMiFin EU Project 14/2/2011
2. Middleware Laboratory MIDLAB Focus and structure of the talk Requirements coming from the financial context; Collaborative event processing for Cyber Security Edge vs centralized event processing over the internet Agilis Esper Roberto Baldoni
3. MIDLAB Middleware Laboratory Sapienza Università di Roma Dipartimento di Informatica e Sistemistica The case of the Financial Critical Infrastructure
4. Middleware Laboratory MIDLAB The case of Collaborative Cyber Security in Financial Ecosystem "webification" of critical financial services, such as home banking, online trading, remote payments; Cross-domain interactions, spanning different organization boundaries are in place in financial contexts; Heterogeneous infrastructure systems such as telecommunication supply, banking, and credit card companies working on heterogeneous data; Roberto Baldoni
5. Middleware Laboratory MIDLAB The case of Collaborative Cyber Security in Financial Ecosystem A payment card fraud (2008) 100 compromised payment cards used by a network of coordinated attackers retrieving cash from 130 different ATMs in 49 countries worldwide, totaling 9 million of US dollars. High degree of coordination, half an hour to be executed evade all the local monitoring techniques used for detecting anomalies in payment card usage patterns. The fraud has been detected only later, after aggregating all the information gathered locally by each financial institution involved in the payment card scam Roberto Baldoni
6. Middleware Laboratory MIDLAB The case of Collaborative Cyber Security in Financial Ecosystem Distributed Denial Of Service Attack (2007, Northern Europe) render web-based financial services unreachable from legitimate users. DDoS attack targeted a credit card company and two DNS. Internet restored only after several trial-and-error activities carried out manually by network administrators of the attacked systems and of their Internet Service Providers (ISPs). Long preparation time (days), short attack time (seconds) Roberto Baldoni
7.
8. Use of Botnets (rented now with a credit card in a few minutes)
12. Iran (in progress!). Stuxnet worm invaded Iran’s Supervisory Control and Data Acquisition systemsMcAfee report 2010 “in the crossfire: criticalinfrastructures in the ageof cyber war “ Roberto Baldoni
16. one out of five DDos attacks is accompanied with an extorsionMcAfee report 2010 “in the crossfire: criticalinfrastructures in the ageof cyber war “ Roberto Baldoni
17. Middleware Laboratory MIDLAB The case of Collaborative Cyber Security in Financial Ecosystem Both previous attacks cannot be detected quickly through information available at the IT infrastructure of a single financial player (i.e., using local monitoring) Need of Information Sharing Exchange non-sensitive status information Set up of agreements Advantages of a global monitoring system Damage mitigation Quick reaction Roberto Baldoni
21. Legal IssuesLLYODS France Telecom UBS Internet AT&T SWIFT Unicredit EDF Events warnings Roberto Baldoni
22. MIDLAB Middleware Laboratory Sapienza Università di Roma Dipartimento di Informatica e Sistemistica Collaborative event Processing for cyber security: The CoMiFin Project ApplicationLevel CollaborationLevel Internet level
23. Middleware Laboratory MIDLAB Collaborative Cyber Security Platform Monitoring and reaction to threats (MitM, Stealty Scan , Phishing, …) Black/white lists distribution (for credit reputation, trust level, …) Anti-terrorism lists (with name check VAS) Anti money laundering monitoring Risk management support Some Requirements on the platform uneven workload along the time High throughput high computational power Large storage capabilities Timeliness Roberto Baldoni
24.
25. set of processing and data sharing services provided by the SR along with the data protection, privacy, isolation, trust, security, dependability, performance requirements.
26. The contractalsocontainsthe hardware and software requirements a member has to provision in order to be admitted into the SR.
30. highly flexible to accommodate the use of different technologies for the implementation of the processing and sharing within the SR (i.e., the implementation of the SR logic or functionality).Roberto Baldoni The notion of semantic room
38. Middleware Laboratory MIDLAB Data Management problems in the semantic room Jurisdiction and regulation (Where and how will data be governed?) Ownership of Data (Who owns the data in the semantic room?) Data Portability Data anonymization Data Retention/Permanence (What happens to data over time?) Security and Privacy (How is data secure and protected?) Reliability, Liability and Quality of Service of the partner of the semantic room Government Surveillance (How much data can the government get from a semantic room?) …………………. Roberto Baldoni
40. Middleware Laboratory MIDLAB IBM System S [ICDCS 06] high cost of ownership Centralized data management No cooperative approach Cooperative Intrusion Detection Systems (e.g. Dshiels) Correlation among local warnings High cost of ownership Obscure data management Related work Roberto Baldoni
41. MIDLAB Middleware Laboratory Sapienza Università di Roma Dipartimento di Informatica e Sistemistica Preventing Stealthy Scan Through centralized processing ApplicationLevel CollaborationLevel Internet level
42. Middleware Laboratory MIDLAB Collaborative Stealthy scan Attacker performs port scanning simultaneously at multiple sites trying to identify TCP/UDP ports that have been left open. Those ports can then be used as the attack vectors Added value of collaboration: Ability to identify an attacker trying to conceal his/her activity by accessing only a small number of ports within each individual domain Action taken: black list IP addresses update historical records Roberto Baldoni
46. Use of real trace (e.g., ITOC US Army) Roberto Baldoni
47.
48. Analyze the sequence of SYN, ACK, RST packets in the three-way TCP handshake. Specifically, in normal activities the following sequence is verified (i) SYN, (ii) SYN-ACK, (iii) ACK.
49. In the presence of a SYN port scan, the connection looks like the following: (i) SYN, (ii) SYN-ACK, (iii) RST (or nothing)
50. For a given IP address, if the number of incomplete connections is higher than a certain threshold T, we can conclude that the IP address is likely carrying out malicious port scanning activities. Roberto Baldoni
51.
52. Example of semantic room for stealthy scan: Ingredients EPL Query EPL Query EPL Query EPL Query EPL Query Subscriber Middleware Laboratory MIDLAB Branch j Branch 1 Esper CEP Engine Gateway POJOs I/O socket I/O socket adapter Input Streams sniffer Main Engine ... Output Streams Branch N POJOs Gateway I/O socket suspected IPs adapter Scanner list sniffer Roberto Baldoni
56. MIDLAB Middleware Laboratory Sapienza Università di Roma Dipartimento di Informatica e Sistemistica Preventing Stealthy Scan through edge processing ApplicationLevel CollaborationLevel Internet level
61. Agilis consists of a distributed network of processing and storage elements hosted on a cluster of machines (also geographycally dispersed) Roberto Baldoni
65. Middleware Laboratory MIDLAB Demo: Done at Haifa IBM Research LAB (2009) Simple and homemade attacks artificial traces Simple stealty scan detection algorithm 8 Linux Machines on a LAN, each of which with 2GB of RAM and 20GB of disk space One machine was hosting all the management processes (JT, XS Catalogue) Each of the remaining 7 hosts modeled a single SR participant DMZ web server under attack TT and XS data server Scenarios: Single intruding host that generated a series of TCP/SYN requests targeting a fixed set of 300 unique ports on each the 7 attacked servers requests injected at constant rate of 10, 20, and 30 req/server/sec ratio of attack to legitimate traffic 1:5 blacklisting threshold: 20,000 requests and 1000 unique port processing window: 4 minutes Results: No overload Detection latency 700 sec, 430 sec, 330 sec Roberto Baldoni
66.
67. Real TCP dumpsJoint work with Giorgia Lodi and Leonardo Aniello Roberto Baldoni