In Big Data we focus on the 4 V's: Volume, Velocity, Varity and Veracity. But another important topic is often not in the focus: Privacy and Security. Yet as important and if not considered from the beginning it might put your Big Data project at risk. Learn about most important Privacy and Security fundamentals in Big Data, you should take into account in your next Big Data project.
5. Big Data Definition (4 Vs)
+ Time to action ? – Big Data + Real-Time = Stream Processing
Characteristics of Big Data: Its Volume, Velocity
and Variety in combination
09.09.2016 TE 09.2016 - BigData Privacy & Security Fundamentals5
6. Data
Acquisition
Data
Sources
Governance
Organisation
Information
Provisioning Consumer
Data
Management
Trivadis Architecture Canvas for Analytical Applications
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Legal ComplianceQuality & Accountability Security & PrivacyMetadata Management Master Data Management
IT Operations Business StakeholdersBI Competence Center
Un-/Semi- structured
Data
Structured
Data
Master & Reference
Data
Machine Data
Content
Services (Push)Connectors (Pull)
StreamBatch/Bulk
IncrementalFull
Raw Data at Rest
Standardized Data at Rest
Optimized Data at Rest
Data Lab (Sandbox)
Data Refinery/Factory
Virtualization
Raw Data in Motion
Standardized Data in Motion
Optimized Data in Motion
Query
Service / API
Search
Information Services
Data Science
Tools
Dashboard
Prebuild &
AdHoc BI Assets
Advanced Analysis
Tools
6
7. Big Data Ecosystem – many choices ….
09.09.2016 TE 09.2016 - BigData Privacy & Security Fundamentals7
8. Top 8 Laws of Big Data
1. The faster you analyze your data, the greater its predictive value
2. Maintain one copy of your data, not dozens
3. Use more diverse data, not just more data
4. Data has value far beyond what you originally anticipate
5. Plan for exponential growth
6. Solve a real pain point
7. Put data and humans together to get the most insight
8. Big Data is transforming business the same way IT did
09.09.2016 TE 09.2016 - BigData Privacy & Security Fundamentals
Source: thebigdatagroup.com
8
9. Data Breaches
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
http://www.Conjur.net/breache
9
10. Data Breaches
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Verizon Data Breache Investigation Report
89% of breaches had a financial or
espionage motive
No locale, industry or organization is
bulletproof when it comes to the
compromise of data
New vulnerabilities come out every day
63% of confirmed data breaches involved
weak, default or stolen passwords.
http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/
10
11. Data Breaches
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Verizon Data Breache Investigation Report
http://www.verizonenterprise.com/verizon-insights-lab/dbir/2016/
11
12. Motivation for Privacy & Security in BigData
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
The bigger your data, the bigger the target
Data theft is a rampant and growing area of crime
Stricter Data Protection bushed by regulations
The only real way to save money and keep security costs low is to take preventive
steps to avoid common vulnerabilities and to minimize their impact.
care must be taken at every step of a big data project to ensure you don’t stumble
into pitfalls which could lead to wasted time and money, or even legal trouble.
12
13. Top Ten Big Data Security & Privacy Challenges (CSA)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
1. Secure computations in distributed
programming frameworks
2. Security best practices for non-
relational data stores
3. Secure data storage and
transactions logs
4. End-point input validation/filtering
5. Real-Time Security Monitoring
6. Scalable and composable privacy-
preserving data mining and
analytics
7. Cryptographically enforced data
centric security
8. Granular access control
9. Granular audits
10.Data Provenance
13
14. TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Top Ten Big Data Security & Privacy Challenges (CSA)
https://cloudsecurityalliance.org/media/news/csa-releases-the-expanded-top-ten-big-data-security-privacy-challenges/
14
15. TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Privacy
&
Data Protection Regulations
15
16. „Privacy“ vs “Data Protection”?
BD-PSF - BigData Privacy & Security Fundamentals20.06.2016
Is there a Difference?
Yes:
Country specific (US=Privacy ¦ EU = Data Protection)
Data Protection: Protect against unauthorised access
Data Privacy: authorized Access
Tecnical vs Legal
17. when does „Privacy“ apply?
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Whenever data is:
Collected
Processed
Stored
Which...
… relates to a living individual person who can be identified by that data.
In “Data Protection” Regulations:
“personal identifiable information” (PII)
“sensitive personal information” (SPI)
17
18. Personally Identifiable Information (PII)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
… means data which relate to a living individual who can be identified
from those data, or
from those data and other information which is in the possession of the data
controller,
and includes any expression of opinion about the individual and any indication
of the intentions of the data controller or any other person in respect of the
individual.
18
19. “Sensitive Personal Information” (SPI)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
… is PII data, consisting of Information as to:
the racial or ethnic origin of the data subject,
his political opinions,
his religious beliefs or other beliefs of a similar nature,
whether he is a member of a trade union (within the meaning of the Trade Union and
Labour Relations (Consolidation) Act 1992),
his physical or mental health or condition,
his sexual life,
the commission or alleged commission by him of any offence
19
20. National Data Protection Regulations
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
DE, AT and CH have similar national Data Protection regulations
(BDSG / DSG)
Regulates protection of the persons privacy
Data protection principles must be met
Transfer to 3rd Party only with legal contract regulating the use of PII Data.
Fines are up to 300000 EUR, if not comply with law
20
21. National Data Protection Regulations
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Data protection principles
Fair and lawful
Purposes
Adequacy not excessively
Accuracy
Retention
Rights of the Person
Security (Technical & Organisational Measures - TOM)
Transfer only with adequate level of protection
https://ico.org.uk/for-organisations/guide-to-data-protection/data-protection-principles/
21
22. EU GDPR – General Data Protection Regulation
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
A single law, the General Data Protection Regulation shall unify data protection
within the European Union.
As a regulation it directly imposes a uniform data security law on all EU members.
The regulation aims to enhance privacy and strengthen data protection rights for
EU citizens.
Agreed on may 2016 – Affective Mid 2018
22
23. EU GDPR – Key facts
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Businesses not in EU still
have to comply if data from
EU Citizen is processed
Appointment of a DPO will be
mandatory
Mandatory Privacy Risk
impact assessment (PIA)
Data Breach Notification
requirements
Data Minimization
(right to erasure)
Data security
(integrity & confidentiality)
Data Processors (Provider) have
direct legal obligations)
Privacy by design
(compliance with the principals of
data protection)
Must “implement appropriate
technical and organisational
measures” to ensure
GDPR compliance
Fines up to 20.000.000 EUR or
4% of companies annual turnover
23
24. TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Privacy by Design (enisa)
24
25. Privacy by Design (enisa)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.201625
26. Privacy by Design (enisa)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
https://www.enisa.europa.eu/publications/big-data-protection
26
27. Is there not a conflict?
TE 09.2016 - BigData Privacy & Security Fundamentals27 09.09.2016
8 Laws of Big Data
1. Faster Analyzation
2. Maintain one copy, not dozens
3. more diverse data
4. Data has value far beyond…
5. Plan for exponential growth
6. Solve a real pain point
7. Put data and humans together to
get the most insight
8. Big Data is transforming business
Privacy by design
1. Minimize
2. Hide
3. Separate
4. Aggregate
5. Inform
6. Control
7. Enforce
8. Demonstrate
28. Is there not a conflict?
TE 09.2016 - BigData Privacy & Security Fundamentals28 09.09.2016
8 Laws of Big Data
1. Faster Analyzation
2. Maintain one copy, not dozens
3. more diverse data
4. Data has value far beyond…
5. Plan for exponential growth
6. Solve a real pain point
7. Put data and humans together to
get the most insight
8. Big Data is transforming business
Privacy by design
1. Minimize
2. Hide
3. Separate
4. Aggregate
5. Inform
6. Control
7. Enforce
8. Demonstrate
29. Is there not a conflict?
TE 09.2016 - BigData Privacy & Security Fundamentals29 09.09.2016
8 Laws of Big Data
1. Faster Analyzation
2. Maintain one copy, not dozens
3. more diverse data
4. Data has value far beyond…
5. Plan for exponential growth
6. Solve a real pain point
7. Put data and humans together to
get the most insight
8. Big Data is transforming business
Privacy by design
1. Minimize
2. Hide
3. Separate
4. Aggregate
5. Inform
6. Control
7. Enforce
8. Demonstrate
30. Is there not a conflict?
TE 09.2016 - BigData Privacy & Security Fundamentals30 09.09.2016
8 Laws of Big Data
1. Faster Analyzation
2. Maintain one copy, not dozens
3. more diverse data
4. Data has value far beyond…
5. Plan for exponential growth
6. Solve a real pain point
7. Put data and humans together to
get the most insight
8. Big Data is transforming business
Privacy by design
1. Minimize
2. Hide
3. Separate
4. Aggregate
5. Inform
6. Control
7. Enforce
8. Demonstrate
32. Security controls
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Top 10 best practices to enhance security and privacy of BigData (CSA):
1. Authorize access to files by predefined security policy
2. Protect data by data encryption while at rest
3. Implement Policy Based Encryption System (PBES)
4. Use antivirus and malware protection systems at endpoints
5. Use big data analytics to detect anomalous connections to cluster
6. Implement privacy preserving analytics
7. Consider use of partial homomorphic encryption schemes
8. Implement fine grained access controls
9. Provide timely access to audit information
10.Provide infrastructure authentication mechanisms
https://downloads.cloudsecurityalliance.org/initiatives/bdwg/Comment_on_Big_Data_Future_of_Privacy.pdf
32
33. Mitigation measures and good practices (ensia)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Strong and scalable encryption
Encrypt data in transit and at rest, to ensure data confidentiality and integrity.
Ensure proper encryption key management solution, considering the vast amount of
devices to cover.
Consider the timeframe for which the data should be kept - data protection regulation
might require that you dispose of some data, due to its nature after certain period of
time.
Design databases with confidentiality in mind – for example, any confidential data
could be contained in separate fields, so that they can be easily filtered out and/or
encrypted.
33
34. Mitigation measures and good practices (ensia)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Application security
Use regular security testing procedures to re-assure the level of security, specially
after patches or functionality changes.
Ensure tamper resistant devices to avoid misuse.
Ensure internal security testing procedures for new and updated components are
carried out regularly; if it is not possible third party evaluations, audits and
certification are key elements for the confidence and trust in products and actors.
Ensure procurement policies cover purchasing from authentic suppliers.
34
35. Mitigation measures and good practices (ensia)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Standards and Certification
Use devices which comply with desired security standards.
Ensure obtained certification relates to the use of Big Data.
Secure use of Cloud in Big Data
Ensure Big Data is included in the risk assessment for Cloud.
Ensure proper Service Level Agreements have been adopted.
Ensure proper resource isolation and exit strategies have been negotiated
35
36. Mitigation measures and good practices (ensia)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Source filtering
Use devices with authentication capabilities to ensure that validation of endpoint
sources is possible
Assign confidence levels on the endpoint sources
Re-evaluate confidence levels of the endpoints regularly, specially after patches
or changes in firmware
If confidence in endpoint source
36
37. Mitigation measures and good practices (ensia)
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Access control and authentication
Use authentication and authorization to ensure that Big Data queries are executed
by authorized users and entities only
Use components in the Big Data system that follow same security standards to
maintain the desired level of security
Big Data monitoring and logging
Enable logging on nodes participating in the Big Data computation
Enable logging on databases (relational or not) , as well as Big Data applications
Detect and prevent modification of logs
Regularly test the restoration of Big Data backups considering the vast amount of
data being used in the system
37
38. TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Putting it together
38
39. Putting it Together
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Privacy & Security an important
subject
Each BigData Project has to
take Security into account
As earlier as better – later
changes are costive
New EU-GDPR changes
importance significant
(and also the risk not to comply)
Traditional security controls apply
also to BigData, but might be
challenging
Security Standards for BigData are
slowly getting established
We have to look closely to
technology vendors and their
functionalities…
compliance requirements might
affect the vendor selection
39
40. TE 09.2016 - BigData Privacy & Security Fundamentals09.09.201640
41. Big Data &
Data Science
TE 09.2016 - BigData Privacy & Security Fundamentals09.09.2016
Advanced Analytics
§ Data Mining
§ Semantic Web
§ Visualisierung
Big Data & Data Scientist
Trainings
Big Data Consulting &
Managed Services
Large & Speedy Data
§ Hadoop Ecosystem
§ NoSQL DBs
§ Event Hubs & Streaming Analytics
§ Unified Query (RDBMS ó Big Data)
§ DWH Archive
§ Internet of Things
Big I Data I Warehouse
§ Konvergenz BI & Big Data
§ LDW Logical Data Warehouse
Big Data Privacy & Security
41
42. Session Feedback – now
TE 09.2016 - BigData Privacy & Security Fundamentals42 09.09.2016
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as “svv”)
– Password: sent by e-mail...