Big Data meetup of Open Tech Week by Sigma Software, Lviv 2019

1. Tales from fire depot:
how to do BigData and survive

2. About me:
●
Denis Pyshev, Twitter: @gemelen
●
Senior Software Engineer, Sigma Software
●
Software development in general, FinTech, AdTech, 8+ years
●
Data Platform / Datawarehouse team, Verizon Media Group
(Oath/AoL+Yahoo), 3+ years
●
Lviv Scala User Group

3. Team challenges
●
Know (and trust) your team
●
Do not throw bus factor under the bus
●
Beware of your platform
●
Learn your product
●
Secure your stuff

4. Know (and trust) your team
●
Your team members are not code monkeys (I hope so)
●
They could know better
●
At least try to listen to them
●
Let your engineers make things they would like (it will save money too)

5. Know (and trust) your team: case of GDPR

6. Do not throw bus factor under the bus
●
People go to vacation from time to time (do they?)
●
Shit happens, no exceptions
●
Shit would probably happen when person in charge is out of town

7. Do not throw bus factor under the bus:
laptop stays home

8. Beware of your platform
●
Collect platform knowledge
●
Experiment and share results (at least within the team)
●
Check if default values are viable in your setup

9. Beware of your platform: case of compression

10. Learn your product
●
Domain knowledge does worth to get
●
Collected data is worthless without its consumption
●
Your product doesn’t fly in free space: monitor ins and outs
●
Create your own monitoring tools on top of monitoring services
●
We did so: check github.com/verizonAdPlatforms/bandar-log

11. Learn your product: case of [object Object]

12. Secure your stuff
●
Never ever store any credentials in repositories, no matter how private
they are – someday they would leak
●
“The response to the credentials leakage was absolutely overwhelming
for us. Within less than one-minute foreign parties tried to take advantage
of the leaked credentials. Within the 6 minutes in which the credentials
were valid we received connections from different Anonymizing Networks
and from China.” - ict.swisscom.ch
●
Use infosec/devops techniques to separate code, configuration and
credentials
●
Use 2FA

13. Secure your stuff

14. Q&A