1. Incident
reporting
S. Freitag, F.
Feldhaus
Incident reporting
Before you
report GridKa Summer School 2010
Incident
Scenarios
Incident
handling Stefan Freitag, Florian Feldhaus
Robotics Research Institute
TU Dortmund
September 10, 2010

2. Contents
Incident
reporting
S. Freitag, F.
Feldhaus
Before you
report 1 Before you report
Incident
Scenarios
Incident
handling
2 Incident Scenarios
3 Incident handling

3. Do you know....?
Incident
reporting
Security Incident Response Policy1
S. Freitag, F.
Feldhaus
objective: ensure that all incidents are investigated as fully
Before you
report as possible and that sites promptly report intrusions.
Incident
Scenarios
As a grid participant, you agree to
Incident
report suspected security incidents that have impact or
handling relationship to grid resources, services, or identities
respond to and investigate incident reports regarding
resources, services, or identities for which you are
responsible
perform appropriate investigations and forensics and share
the results with the incident coordinator
follow the incident response procedure
Next question: what is the incident response procedure?
1
https://edms.cern.ch/document/428035/7

4. EGEE incident response procedure2
Incident
reporting
S. Freitag, F.
Feldhaus
Audience
Before you
report
grid site security officers and site administrators
Incident
Scenarios
Incident Definition of security incident
handling
The act of violating an explicit or implied security policy
Definition of actions for the case of a security incident
More on this in a few minutes . . .
2
https://edms.cern.ch/document/867454

5. Security incident - scenario A (2009)
Incident
reporting
S. Freitag, F. Some grid sites allow gsissh-based access to VoBoxes (e.g.
Feldhaus
for VO software managers)
Before you
report On a VoBox Grid users are mapped to local accounts
Incident
Scenarios
Initial step for an attacker
Incident
handling
gain access to user credentials (certificate or proxy)
What happens next ?
Connect to VoBox using stolen credentials
Running e.g. a kernel exploit to gain root privileges

6. Security incident - scenario A (2009)
Incident
reporting
S. Freitag, F.
Feldhaus
# s h −x w u n d e r b a r e m p o r i u m . s h
Before you
report [...]
Incident
[+] got r i n g 0 !
Scenarios [+] d e t e c t e d 2.6 s t y l e 4k s t a c k s
Incident [ + ] D i s a b l e d s e c u r i t y o f : n o t h i n g , what an
handling
i n s e c u r e machine !
[ + ] Got r o o t !
sh −3.00# i d
u i d =0( r o o t ) g i d =0( r o o t ) g r o u p s =64004( hepcg )
c o n t e x t=u s e r u : s y s t e m r : i n i t r c t

7. Security incident - scenario B (2010)
Incident
reporting
Department A The Grid
S. Freitag, F.
Feldhaus
Before you
report
Incident
Scenarios
Incident
handling

8. Security incident - scenario B (2010)
Incident
reporting
Department A The Grid
S. Freitag, F.
Feldhaus
CERTIFICATE
X.509
Before you
report
CERTIFICATE
Incident X.509
Scenarios
Incident
handling

9. Security incident - scenario B (2010)
Incident
reporting
S. Freitag, F.
Department A The Grid
Feldhaus
CERTIFICATE
Before you X.509
report
Incident
Scenarios CERTIFICATE
X.509
Incident
handling
Alien
attacker

10. Security incident - scenario B (2010)
Incident
reporting
S. Freitag, F.
Feldhaus The Grid
Before you
report
Incident
Scenarios
stolen
Incident CERTIFICATE
X.509
handling
Alien
attacker

11. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus For the next slides please keep in mind:
Before you
report
Incident The red block describes actions required by the EGEE Incident
Scenarios
Response Procedure document
Incident
handling
The blue block contains information about actions carried out
during a security incident at the Grid resource in Dortmund
Down here you will find additional information, e.g. max.
response times

12. Incident handling
Incident
reporting
First action
S. Freitag, F.
Feldhaus Inform immediately your local security team and your ROC
Before you Security Contact
report
Incident
Scenarios Action
Incident
handling Sent E-Mail to Ursula Epting
Read Incident response procedure
Informed 2nd site security officer and local security team
max. 4 hours or

13. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus
Before you Response procedure
report
Incident In case no support is shortly available [...] try to contain the
Scenarios
incident. For instance by unplugging the network cable
Incident
handling connected to the host. Do NOT reboot or power off the host.
Action
Disconnected affected workernodes from network

14. Incident handling
Incident
reporting Response procedure
S. Freitag, F.
Feldhaus Assist your local security team and your ROC Security Contact
to confirm and investigate the incident. Announce the incident
Before you
report to all the sites.
Incident
Scenarios
Actions
Incident
handling
Send a heads-up e-mail (template: next slide)
Arranged meeting with local security team
Network guys were asked to check logs
max. 4 hours (Announcement)

15. Heads-up E-mail
Incident
reporting
S. Freitag, F.
Feldhaus
Before you
report
** PLEASE DO NOT REDISTRIBUTE ** EGEE-<DATE> (ex: EGEE-20090531)
Incident ** This message is sent to the EGEE CSIRTs and must NOT be publicly archived **
Scenarios Dear CSIRTs,
It seems a security incident has been detected at <your site>.
Incident Summary of the information available so far:
handling
Ex: A malicious SSH connection was detected from XXXXX. The extent of the
incident is unclear for now, and more information will be published in the coming
hours as forensics are progressing at our site. However, all sites should check for
successful SSH connection from XXXXX as a precautionary measure.

16. Incident handling
Incident
reporting Response procedure
S. Freitag, F.
Feldhaus Report a downtime for the affected hosts on the GOCDB
Before you
report
→ Send an EGEE broadcast announcing the downtime for
Incident
the affected hosts Use ”Security operations in progress” as
Scenarios the reason with no additional detail both for the broadcast
Incident
handling
and the GOCDB.
Actions
Created downtime for possibly affected hosts udo-ce01/
udo-dcache01
max. 1 day after discovery

17. Incident handling
Incident
reporting
S. Freitag, F. Response procedure
Feldhaus
Perform appropriate forensics and take necessary actions to
Before you
report prevent further damage
Incident
Scenarios
Identify and kill suspicious process(es) as appropriate, but
Incident
aim at preserving the information they could have
handling
generated
If it is suspected that some grid credentials have been
abused or compromised, you MUST ensure the relevant
accounts become suspended
If it is suspected that some grid credentials have been
abused, you MUST ensure that the relevant VO
manager(s) have been informed.

18. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus
Before you
Response procedure
report
Incident
Perform appropriate forensics and take necessary actions to
Scenarios prevent further damage
Incident
handling If it is suspected that some grid credentials have been
compromised, you MUST ensure that the relevant
certification authority gets informed.
If needed, seek for help from your local security team or
from your ROC Security Contact

19. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus
Before you
report Action
Incident
Scenarios Banned affected users on our compute elements by adding
Incident their DN to the blacklist in
handling
/opt/glite/etc/lcas/ban users.db
E-Mail to VO manager regarding compromised user
Contacted the certification authority

20. Incident handling
Incident
reporting
S. Freitag, F. Response procedure
Feldhaus
As part of the security incident resolution process, sites are
Before you
report expected to report the following information:
Incident
Scenarios affected hosts and hosts used as entry point to the site
Incident remote IP address(es) of the attacker
handling
evidence of the compromise, including timestamps
what was lost, details of the attack
list of other sites possibly affected (if available)
possible vulnerabilities exploited by the attacker (if
available)
actions taken to resolve the incident

21. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus
Before you
Response procedure
report
Incident
Scenarios
Tracked down the UI that was used by the attacker for job
Incident
submission (checking logs of batchsystem, Compute
handling Element, . . . )
Analyzed netflow to/fro affected workernode
Analyzed executables deployed by the attacker
Updated incident report regularly

22. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus Response procedure
Before you Coordinate with your local security team and your ROC
report
Security Contact to send an incident closure report including
Incident
Scenarios lessons learnt and measures taken to prevent future incidents.
Incident
handling
Actions
Preparation and submission of final report
max. 1 months

23. Incident handling
Incident
reporting
S. Freitag, F.
Feldhaus
Response procedure
Before you
report
Restore the service, and if needed, send an EGEE broadcast,
Incident
Scenarios update the GOCDB, service documentation and procedures to
Incident prevent recurrence as necessary
handling
Actions
Re-installation of affected workernode
Safety tuning

24. Incident
reporting
S. Freitag, F.
Feldhaus
Before you
report
Incident
Scenarios
Incident
handling
Thanks for your attention!