FeedHenry & MongoDB

•

0 likes•664 views

Cian Clarke

Presentation given to the Dublin Mongo User Group, December 2012 on our use of MongoDB in FeedHenry

Technology Business

Cian

Clarke
www.cianclarke.com

@cianclarke

github.com/cianclarke

cian.clarke@feedhenry.com

oh sh!t you guys we need
Data Storage
like right now ~ Somebody, Aug 2011

© Copyright FeedHenry Ltd. 2012
Partners
Clients

But why did
choose ?
Why not
or or ….
, or stick with

Product
Catalog

•  400,000
records

•  Parsed,
then
dumped
into
DB
all
at
once

•  Fell
over
:-‐(

•  Node
+
Streams
=
Awesome

•  Streaming
CSV
parser
-‐>
Chunks
-‐>
Into
DB

?@feedhenry | @cianclarke

feedhenry | cianclarke
feedhenry.com | cianclarke.com
me

www

What's hot

GraphQL & Relay

Viacheslav Slinko

Better APIs with GraphQL

Josh Price

Pycon Talk

Abhijit Gadgil

How web works and browser works ? (behind the scenes)

Vibhor Grover

GraphQL, Redux, and React

Keon Kim

Introduction to GraphQL at API days

yann_s

Cloud Native Predictive Data Pipelines (micro talk)

Sid Anand

React, GraphQL и Relay - вполне себе нормальный компонентный подход (nodkz)

Pavel Chertorogov

GraphQL With Relay Part Deux

Brad Pillow

We’ll start with the broad overview of how the New Relic platform can be extended and customized to manage your distributed cloud architectures, drive immersive customer experience and provide real-time insights into your business. We’ll dive in and MongoDB will show us how they integrated database monitoring and analytics into the New Relic platform via multiple New Relic APIs. You’ll leave empowered to try existing integrations, excited to build new ones, and motivated to make the New Relic platform your own.

Extending the New Relic Platform

New Relic

GraphQL

Cédric GILLET

Scaling the guardian

Michael Brunton-Spall

GraphQL IndyJS April 2016

Brad Pillow

The guardian and app engine

Michael Brunton-Spall

GraphQL is a powerful way to bridge the gap between frontend and backend. Providing a typed API with introspection. This can be used for code generation or code completion. Because of some misconceptions it can be hard to combine GraphQL with Kafka, but it doesn’t need to be hard. After handling misconceptions like 'Kafka is just a message bus' and 'GraphQL is only for Graph databases' I will have a few demo's how both can be combined. It can be a clear way to make your event sourcing backend available to the frontend. GraphQL can be used to quickly setup a typed API, if you already use ksqlDB.

Kafka and GraphQL: Misconceptions and Connections | Gerard Klijs, Open Web

HostedbyConfluent

Scaling small apps

Michael Brunton-Spall

The war between DBAs and developers has been raging since the dawn of relational databases. One reason for disagreement comes from developers who want to store their data in JSON because it is fast, standard, and flexible. DBAs cringe when they hear of long text strings being stored in their SQL databases; they cry with concern, “No data validation? No schema binding?”. Is there any hope for these two warring factions to see eye-to-eye? This session will explore the new JSON functionality introduced in SQL Server 2016. We will use T-SQL examples to learn how these functions can be used to parse, create, and modify JSON data. More importantly, we will discuss how to optimize performance when using these functions. By the end of this session DBAs and developers will know how to efficiently work with JSON in SQL Server 2016 and 2017. It will also usher in an era of peace between DBAs and developers… … at least until someone brings up the topics of cursors, NOLOCKs, or Entity Framework.

DBAs vs Developers - JSON in SQL Server

Bert Wagner

DBAs vs Developers: JSON in SQL Server - CBusPASS

Bert Wagner

Cloud Native Data Pipelines (GoTo Chicago 2017)

Sid Anand

GraphQL is an application layer query language from Facebook. With GraphQL, you can define your backend as a well-defined graph-based schema. Then client applications can query your dataset as they are needed. GraphQL’s power comes from a simple idea — instead of defining the structure of responses on the server, the flexibility is given to the client. Will GraphQL do to REST what REST did to SOAP?

GraphQL in an Age of REST

Yos Riady

What's hot (20)

GraphQL & Relay

Better APIs with GraphQL

Pycon Talk

How web works and browser works ? (behind the scenes)

GraphQL, Redux, and React

Introduction to GraphQL at API days

Cloud Native Predictive Data Pipelines (micro talk)

React, GraphQL и Relay - вполне себе нормальный компонентный подход (nodkz)

GraphQL With Relay Part Deux

Extending the New Relic Platform

GraphQL

Scaling the guardian

GraphQL IndyJS April 2016

The guardian and app engine

Kafka and GraphQL: Misconceptions and Connections | Gerard Klijs, Open Web

Scaling small apps

DBAs vs Developers - JSON in SQL Server

DBAs vs Developers: JSON in SQL Server - CBusPASS

Cloud Native Data Pipelines (GoTo Chicago 2017)

GraphQL in an Age of REST

Similar to FeedHenry & MongoDB

Netflix Big Data Paris 2017

Jason Flittner

Traveloka's journey to no ops streaming analytics

Rendy Bambang Junior

12-Step Program for Scaling Web Applications on PostgreSQL

Konstantin Gredeskoul

At Stitch Fix we have a lot of Data Scientists. Around eighty at last count. One reason why I think we have so many, is that we do things differently. To get their work done, Data Scientists have access to whatever resources they need (within reason), because they’re end to end responsible for their work; they collaborate with their business partners on objectives and then prototype, iterate, productionize, monitor and debug everything and anything required to get the output desired. They’re full data-stack data scientists! The teams in the organization do a variety of different tasks: - Clothing recommendations for clients. - Clothes reordering recommendations. - Time series analysis & forecasting of inventory, client segments, etc. - Warehouse worker path routing. - NLP. … and more! They’re also quite prolific at what they do -- we are approaching 4500 job definitions at last count. So one might be wondering now, how have we enabled them to get their jobs done without getting in the way of each other? This is where the Data Platform teams comes into play. With the goal of lowering the cognitive overhead and engineering effort required on part of the Data Scientist, the Data Platform team tries to provide abstractions and infrastructure to help the Data Scientists. The relationship is a collaborative partnership, where the Data Scientist is free to make their own decisions and thus choose they way they do their work, and the onus then falls on the Data Platform team to convince Data Scientists to use their tools; the easiest way to do that is by designing the tools well. In regard to scaling Data Science, the Data Platform team has helped establish some patterns and infrastructure that help alleviate contention. Contention on: Access to Data Access to Compute Resources: Ad-hoc compute (think prototype, iterate, workspace) Production compute (think where things are executed once they’re needed regularly) For the talk (and this post) I only focused on how we reduced contention on Access to Data, & Access to Ad-hoc Compute to enable Data Science to scale at Stitch Fix. With that I invite you to take a look through the slides.

Data Day Seattle 2017: Scaling Data Science at Stitch Fix

Stefan Krawczyk

This session will take you on a journey of real world scenario of Office 365 tenant to tenant content migration encompassing SharePoint Online site collections, Office 365 Groups, OneDrive For Business and Office 365 Video with aggressive timeline and deadline for a global retail organisation. We will cover: - Examining the pitfalls of migrating to a new Office 365 tenant - Introducing new standards for consistent structure - Phases of project and how they determined what each phase would involve - Highlighting technical difficulties encountered and how they were overcome

Ensuring Successful Office 365 Tenant to Tenant Migration SPS Cambridge 2017...

Chirag Patel

Ensuring Successful Office 365 Tenant to Tenant Migration SharePoint Saturday...

Chirag Patel

Which Freaking Database Should I Use?

Great Wide Open

ASTQB washington-sept-2015

Dan Boutin

Data Day Texas 2017: Scaling Data Science at Stitch Fix

Stefan Krawczyk

Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.

What's New in Apache Hive 3.0?

DataWorks Summit

What's New in Apache Hive 3.0 - Tokyo

DataWorks Summit

SharePoint Databases: What you need to know (201509)

Alan Eardley

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

State management with GraphQL [Angular Minsk, Online, 13.06.20]

Mikhail Asavkin

Jeremy Engle's slides from Redshift / Big Data meetup on July 13, 2017

AWS Chicago

As Hadoop applications move into cloud deployments, object stores become more and more the source and destination of data. But object stores are not filesystems: sometimes they are slower; security is different, What are the secret settings to get maximum performance from queries against data living in cloud object stores? That's at the filesystem client, the file format and the query engine layers? It's even how you lay out the files —the directory structure and the names you give them. We know these things, from our work in all these layers, from the benchmarking we've done —and the support calls we get when people have problems. And now: we'll show you. This talk will start from the ground up "why isn't an object store a filesystem?" issue, showing how that breaks fundamental assumptions in code, and so causes performance issues which you don't get when working with HDFS. We'll look at the ways to get Apache Hive and Spark to work better, looking at optimizations which have been done to enable this —and what work is ongoing. Finally, we'll consider what your own code needs to do in order to adapt to cloud execution.

Dancing Elephants - Efficiently Working with Object Stories from Apache Spark...

DataWorks Summit/Hadoop Summit

Dancing Elephants: Working with Object Storage in Apache Spark and Hive

Steve Loughran

GraphQL Munich Meetup #1 - How We Use GraphQL At Commercetools

Nicola Molinari

Presentation to the NASA Cloud Community of Interest. Shows the evolution of simple webapps lacking resilience and scalability up to cloud-native apps that tolerate server faults and availability zone outages. Describes how images.nasa.gov leverages S3, EC2, ELB, DynamoDB, CloudSearch, ElasticTranscoder and more to provide a modern, scalable, mobile-friendly site showcasing the best of NASA's images, video and audio.

Not Your Father’s Web App: The Cloud-Native Architecture of images.nasa.gov

Chris Shenton

English will follow the Korean version. 패스트캠퍼스가 주최한 "2019 퓨처 컨퍼런스"에서 "지속적으로 확장 가능한 빅데이터 플랫폼을 향한 여정"이라는 제목으로 발표한 슬라이드를 공유합니다. 참고로 데이터에 포커싱된 세미나가 아니라서, 일반적인 청중분들도 최대한 이해할 수 있게 슬라이드를 준비했습니다. [Translated] Here is the deck that I gave a talk in 2019 Future last Friday. FYI, I prepared this deck for general attendees to understand it easily because the conference doesn't focus on Data engineering. https://www.fastcampus.co.kr/2019future_tech/

[Coupang] Journey to the Continuous and Scalable Big Data Platform : 지속적으로 확장...

Matthew (정재화)

Similar to FeedHenry & MongoDB (20)