The UNC School of Medicine suffered a security breach last summer that required notification of over 100,000 patients that their information had been exposed. This presentation will talk about the scope of damage that is caused by a breach of this
magnitude and the many steps that are necessary for damage control and recovery.
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
After the Breach
1. After the BreachAfter the Breach
Dennis SchmidtDennis Schmidt
Director, Office of Information SystemsDirector, Office of Information Systems
HIPAA Security OfficerHIPAA Security Officer
UNC School of MedicineUNC School of Medicine
2. OMG, We have a breach!OMG, We have a breach!
In late July, 2009, UNC Information Technology
employees discovered that a server which
contained sensitive information on 180,000
research subjects, including 114,000 Social
Security Numbers, had been the target of a
computer hack in 2007. The compromised
server was taken down and the data on the
server were removed.
3. Incident DiscoveryIncident Discovery
OIS receives call from departmental serverOIS receives call from departmental server
admin reporting that a server would not rebootadmin reporting that a server would not reboot
after power failure.after power failure.
OIS technician suspects virus and performs fullOIS technician suspects virus and performs full
virus scan on machine. Virus detected.virus scan on machine. Virus detected.
Technician is told by department that server mayTechnician is told by department that server may
contain sensitive information.contain sensitive information.
Server turned over to OIS Information SecurityServer turned over to OIS Information Security
for forensic analysis.for forensic analysis.
4. Forensic AnalysisForensic Analysis
-- A Long, Painful Process ---- A Long, Painful Process --
Verification – Verify the incident occurred
Interview the SysAdmins and other users involved
Examine system and application logs (Snort, Tipping
Point, etc.)
Check volatile information using forensic tools
System Description
Physical observation, forensic tools
Interview SysAdmins and users, determine use
Hardware and software system characteristics
Hard disk geometry
5. Forensic Analysis (cont.)Forensic Analysis (cont.)
Evidence Collection
All available computer information (volatile and
non-volatile) is collected and transferred to external
media or forensic workstation to perform analysis
tasks.
Data must be collected in order of volatility and data
integrity safeguarded by hash signature, MD5
6. Forensic Analysis (cont.)Forensic Analysis (cont.)
Timeline Creation & Analysis – Use time-stamps
from internal and external sources to correlate
into timeline that traces back the system activity.
Media Analysis – Thorough examination of the
media layers (physical, data, metadata, file system
and file name) searching for evidence.
7. Forensic Analysis (cont.)Forensic Analysis (cont.)
Data Recovery – extracting unallocated data in
order to recover any deleted files. File
fragments could represent a critical piece of
information relevant to the case
String Search – searching for specific strings or
keywords contained inside files to reveal useful
information relevant to the case.
Reporting -- detailed report(s) of the forensic
process explaining the evidence found, together
with the techniques and methodology used.
8. Houston, We have a problem!Houston, We have a problem!
Virus/worm/trojan infection for 2 yearsVirus/worm/trojan infection for 2 years
26 files containing over 500,000 records26 files containing over 500,000 records
180,000 unique research subjects180,000 unique research subjects
114,000 Social Security Numbers114,000 Social Security Numbers
10. But, did they get anything?But, did they get anything?
When did compromise occur? Is it still active?When did compromise occur? Is it still active?
When were the sensitive files put on theWhen were the sensitive files put on the
machine? When were they last accessed?machine? When were they last accessed?
Was it during the compromise window?Was it during the compromise window?
Is there any corroborating evidence on theIs there any corroborating evidence on the
network of file downloads from the server?network of file downloads from the server?
11. The Antivirus DilemmaThe Antivirus Dilemma
Full virus scan changes the last accessed time onFull virus scan changes the last accessed time on
everyevery file.file.
It now becomes impossible to determine if theIt now becomes impossible to determine if the
malware actually accessed specific files.malware actually accessed specific files.
e.g., If compromise occurred one week ago, and laste.g., If compromise occurred one week ago, and last
access of sensitive file was one month ago, you knowaccess of sensitive file was one month ago, you know
the data was not likely accessed by the malware.the data was not likely accessed by the malware.
If virus scan was done yesterday, you no longerIf virus scan was done yesterday, you no longer
know when the file was last accessed.know when the file was last accessed.
12. No Smoking GunNo Smoking Gun
There was no way to prove that data on theThere was no way to prove that data on the
server was accessed inappropriately.server was accessed inappropriately.
And… there was no way to prove that data onAnd… there was no way to prove that data on
the server wasthe server was notnot accessed inappropriately.accessed inappropriately.
The doors were unlocked and people were in theThe doors were unlocked and people were in the
house, but we couldn’t prove that they stolehouse, but we couldn’t prove that they stole
anything.anything.
13. Second OpinionSecond Opinion
Magnitude of potential breach warrantedMagnitude of potential breach warranted
additional opinionsadditional opinions
ITS Security conducted parallel investigation toITS Security conducted parallel investigation to
verify or refute initial findingsverify or refute initial findings
Additional corroborating data searchedAdditional corroborating data searched
Network traffic logs (only last 90 days)Network traffic logs (only last 90 days)
14. Notification is not an IT DecisionNotification is not an IT Decision
University Counsel makes final recommendationUniversity Counsel makes final recommendation
based on inputs from:based on inputs from:
IT Security (OIS & ITS)IT Security (OIS & ITS)
University RelationsUniversity Relations
UNC Health Care Communications/MarketingUNC Health Care Communications/Marketing
UNC Health Care CounselUNC Health Care Counsel
HIPAA Privacy and HIPAA Security OfficersHIPAA Privacy and HIPAA Security Officers
15. How do we notify 180,000 people?How do we notify 180,000 people?
Is their address current? Do we have anIs their address current? Do we have an
address?address?
Are they still alive?Are they still alive?
Who writes the letters?Who writes the letters?
Who addresses the envelopes? Licks theWho addresses the envelopes? Licks the
stamps?stamps?
Who handles phone calls from concernedWho handles phone calls from concerned
recipients?recipients?
16. The Notification ProcessThe Notification Process
UNC Hired Rust Consulting to assistUNC Hired Rust Consulting to assist
Consultation servicesConsultation services
Mailed notification lettersMailed notification letters
Established and staffed Call CenterEstablished and staffed Call Center
Responded to calls; referred problem calls to UNCResponded to calls; referred problem calls to UNC
Received 4,144 callsReceived 4,144 calls
450 calls referred to UNC450 calls referred to UNC
17. Technical ResponseTechnical Response
Major concern: Uncontrolled serverMajor concern: Uncontrolled server
proliferationproliferation
Determine scope of problemDetermine scope of problem
Protect high risk machines firstProtect high risk machines first
Develop long term strategy to mitigate riskDevelop long term strategy to mitigate risk
18. The Scope of the ProblemThe Scope of the Problem
500+ machines with server OS’s on SOM500+ machines with server OS’s on SOM
networknetwork
2200 machines running a service2200 machines running a service
2068 File Server / File Services2068 File Server / File Services
1989 Remote Access / Remote Management1989 Remote Access / Remote Management
762 Web Servers762 Web Servers
194 Database Servers194 Database Servers
19. Manual Data CollectionManual Data Collection
Mandatory self reporting of serversMandatory self reporting of servers
433 servers reported433 servers reported
98 server admins98 server admins
47 different OS flavors and versions47 different OS flavors and versions
Qualys scans on all servers reporting sensitiveQualys scans on all servers reporting sensitive
information (200 machines)information (200 machines)
20. Long Range StrategyLong Range Strategy
IT Simplification and Security RFP (Dell)IT Simplification and Security RFP (Dell)
Develop Plan for streamlining IT resources in SOMDevelop Plan for streamlining IT resources in SOM
Develop strategic virtualization architectureDevelop strategic virtualization architecture
Develop enterprise storage architectureDevelop enterprise storage architecture
Develop security umbrella to cover centralizedDevelop security umbrella to cover centralized
operationoperation
Goal: Provide robust centralGoal: Provide robust central servicesservices that willthat will
get end users out ofget end users out of serverserver businessbusiness
21. Recovery from the breachRecovery from the breach
Moved data to centrally managed serversMoved data to centrally managed servers
Database encrypted behind hardware firewallDatabase encrypted behind hardware firewall
All working files encrypted with PGP Net ShareAll working files encrypted with PGP Net Share
All machines, including desktops, scanned with QualysAll machines, including desktops, scanned with Qualys
Well defined procedures documented, approved by IRBWell defined procedures documented, approved by IRB
Two person rule for manual movement of data filesTwo person rule for manual movement of data files
Update software to automate processesUpdate software to automate processes
22. How much did it cost?How much did it cost?
Average breach reportedly costs $204 per nameAverage breach reportedly costs $204 per name
$204 X 180,000 = $36.7 Million!$204 X 180,000 = $36.7 Million!
Other references state that a major breach costsOther references state that a major breach costs
an organization aan organization a minimumminimum of $1 Million.of $1 Million.
Postage alone cost $75,000.Postage alone cost $75,000.
Rust Consulting cost $260,000Rust Consulting cost $260,000
Thousands of person hours spent on the projectThousands of person hours spent on the project
OIS Security, ITS Security, OUC, P&A, HIPAAOIS Security, ITS Security, OUC, P&A, HIPAA
Privacy, senior leadership, etc. etc. etc.Privacy, senior leadership, etc. etc. etc.
23. Lessons LearnedLessons Learned
Implementation of IT Governance is criticalImplementation of IT Governance is critical
Decentralized server environment is high riskDecentralized server environment is high risk
New procedures for virus investigationsNew procedures for virus investigations
involving sensitive datainvolving sensitive data
Disconnect from networkDisconnect from network
Do not shut downDo not shut down
Do not perform virus scanDo not perform virus scan
Notify IT SecurityNotify IT Security