Cubes – pluggable model explained
•
5 gefällt mir•17,179 views
Description of the new pluggable model in Cubes – lightweight Python OLAP framework.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Cubes – pluggable model explained
Ähnlich wie Cubes – pluggable model explained (20)
Mehr von Stefan Urbanek
Mehr von Stefan Urbanek (15)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Cubes – pluggable model explained
- 1. data brewery Pluggable Model Cubes Analytical Workspace Redesign Stefan Urbanek – @Stiivi February 2014
- 2. Original Cubes Cubes before 1.0
- 3. Model ■ single JSON or a model bundle ■ contains all model objects ■ full description required
- 4. one file or one directory bundle ✂ model browser formatters http backends one per serving: server workspace [workspace] backend=sql url=postgresql://localhost/database modules
- 5. Browser Aggregation Browser SQL Snowflake Browser SQL Denormalized Browser MongoDB Browser Some HTTP Data Service Browser ? multiple backends available
- 6. Backend ■ implemented as python module with an entry point create_workspace() ■ provides Workspace and Browser workspace represents data storage ■ only one Workspace per serving only one kind of storage per serving
- 7. Requirements
- 8. Model ■ composed of multiple parts ■ external model definition provided from external source, such as analytical service ■ shared dimension descriptions only one dimension description is necessary per composed model
- 9. Backend ■ heterogenous storage multiple data stores, different types of data stores ■ different schemas in same store ■ multiple environments dev, test, production, ...
- 10. Redesign
- 11. Backend ■ “backend” are multiple objects: ! | Provider Browser ■ better plug-in system instead of Python module ■ more flexible composition Store
- 12. Backend Objects ■ Browser – performs aggregated browsing ■ Store – maintains database connection ■ Model Provider – provides model Note: not every kind has to be implemented
- 13. Logical Physical ∑ create model connect aggregate | Provider Browser Store model cubes model physical data store (database or API) dimensions backend objects
- 14. Browser
- 15. Browser ■ depends on the logical model ■ implements aggregation aggregate(), values(), … ■ gets data from associated store
- 16. Logical Physical ∑ aggregate | Browser Store model physical data store (database or API) browser
- 18. Store
- 19. Store * ■ provides database or API connection ■ might provide a model ■ slicer tool actions physical mapping validation, model from schema generation, schema from model generation, schema conversions and optimization, ... *former backend’s “Workspace” object
- 20. Logical Physical connect | Browser Store physical data store (database or API) store
- 21. Store Methods Store is not required to implement any methods at this time. Future: ■ validate(cube) – does logical map to physical? ■ create(object) – create physical structure
- 22. Model Provider
- 23. Model Provider ■ creates model from external source ■ might suggest store to be used
- 25. Provider Methods ■ dimension_metadata(name,temps,locale) ■ cube_metadata(name,locale) or ■ dimension(name,temps,locale) ■ cube(name,locale)
- 26. SQL Backend Mongo Backend | | Snowflake Browser Mongo Browser SQL Store Google Analytics Backend | Mongo Store GA Model Provider example backends GA Browser GA Store
- 27. from cubes import | AggregateBrowser, Store ! class SQLStore( Store): | default_browser_name = “sql_snowflake” ! def __init__(self, **options): # initialize the store here ! def validate_cube(self, cube): return True # if valid ! ! class | SQLSnowflakeBrowser(| Browser): def __init__(self, model, locale): # initialize the browser ! def features(self): # return list of browser features def aggregate(self, cell, ...): # return aggregation of the cell from slicer.ini
- 28. New Workspace * ■ global object at library level ■ provides appropriate browser ■ contains run-time configuration ■ might have state persistence *former backend Workspace is now Store
- 29. Future Workspace ■ caching ■ cube composition ■ …?
- 31. Workspace Model Providers API Model Provider Static Model Provider Cubes sales churn activations events BI Data 2 (Mongo) Events (API) Stores BI Data (Postgres) heterogenous environment
- 32. Workspace [workspace] models_path: /var/lib/cubes/models Model Providers crm sales ! Static Model Provider events [models] crm: crm.cubesmodel sales: sales.cubesmodel events: events.cubesmodel Cubes ! sales churn activations events [datastore_bidata] type: sql url: postgresql://localhost/crm ! Stores BI Data (Postgres) BI Data 2 (Mongo) [datastore_bidata2] type: mongo host: localhost collection: events
- 33. Conclusion
- 34. Conclusion ■ heterogenous pluggable environment ■ externally provided models ■ easier backend implementation