Crawl and scrape thousands of bidding contest from hundreds of sources globally every day

1. Crawlers and Web UI
with full stack JS

2. What is Oppex?
• World’s largest source of public sector
bidding contests
• Crawl and scrape thousands of bidding
contest from hundreds of sources globally
every day

3. Stack
NodeJS
Solr MongoDB
React Flux
NodeJS + Express
Oppex Engine
Oppex UI

4. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure

5. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure
• Clustered Node.js workers
• Kue + Redis for job queues
• Pre-defined crawler
strategies
• Reverse html templates

6. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure
• Translation via Google
Translate API & Yandex
Translate API
• Geocoding Google Places API
& MapQuest API
• Currency parsing, email &
phone extraction etc.

7. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure
• MongoDB for raw data
storage
• Solr for search indexing

8. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure
• User-saved Solr queries ran
at predefined intervals
• Mandrill for emails
• Socket.io

9. Oppex Engine
• Crawlers
• Postprocessors
• Data storage
• Notifications
• Infrastructure
• Docker virtualization
• Backups to AWS S3 + Glacier
• Logstash for logging
• Elasticsearch + Kibana for
monitoring
• Automated deployment
scripts

10. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus

11. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus
• Great for large and
complex apps
• Stop fiddling with DOM
• Encourages modular design
• Ensures data integrity across
the app

12. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus
• Turns ES6 code into ES5
• Imports
• Classes
• Arrow functions
• Promises

13. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus
• Isomorphic apps
• CommonJS or Bower even on
client
• Hot reload for client code or
stylesheets

14. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus
• Never copy objects/
collections ever again
• Deep trees with easy access
to nodes
• Lazy Seq for chaining
collection methods without
intermediate stage

15. Oppex UI
• React + Flux
• Babel
• Webpack
• Immutable.js
• Stylus
• The best css preprocessor
• More logic processing than
Less
• Pure Node, no Ruby required
unlike Sass

16. Thank you!
• Mikko Lehmuskoski
• mikko@oppex.com
• +358 40 750 8307