This document discusses MapSherpa's process for aggregating and standardizing financial transactions from multiple marketplaces into a single database. MapSherpa sells print maps through various channels, including their own platform, Amazon, and MapTrove. They needed a way to collect sales data from these disparate sources and calculate additional costs like printing, shipping, and royalties in a standardized format. The solution involves using FME to extract and transform data from CouchDB, CSV files, Amazon MWS API, and Google Sheets into normalized tables in PostgreSQL. Reports are then generated from these tables in Google Data Studio for accounting and business decisions.
2. The MapSherpa platform connects
map sellers with content from
publishers around the world
Itâs a catalog of digital versions of print maps
3. Todayâs treatment
1. What is MapSherpa
2. How we sell maps
3. What we need
4. Standardizing our sales data
5. The workflow
6. Output reports
Data Detox
4. Who are we and what do we do?
The MapSherpa system is a digital
repository of maps intended for print
â Custom maps: users can search by
place name and select the page size,
orientation and map extent
â Print on Demand maps: users can
search for static maps by keyword
5. âDigitalâ Sales
MapSherpa Kiosk is our front-end tool
for brick-and-mortar map stores
â Our retail partners order maps for their
clients
â MapSherpa provides them with
download links to PDFs for printing the
maps
â Our retail partners can also integrate
the MapSherpa API into their web
stores
6.
7. Print Sales
We sell print copies of maps directly to
consumers too
â Amazon Canada, USA and UK
â MapTrove.ca and MapTrove.com
Sales from these five sites have additional
costs we need to calculate/store
â Printing costs
â Shipping costs
â Inventory costs
â ...
8. All Sales-related Data in One Place
As our sales volume has increased, it has become increasingly necessary
to have an automated process to collect and standardize all related data
This solution is needed to drive
â Retailer partner invoicing
â Supplier billing (map publishers, printers)
â Corporate accounting and decision-making
9. What We Need
1. Collect transactions from the three sources
a. MapSherpa
b. Amazon
c. MapTrove
2. Standardize sales/returns from all three
sources in the same format
3. Calculate printing costs and data royalties
4. Collect all sales related fees (transaction fees,
marketplace fees, etc)
5. Display this data in useful ways
11. MapSherpa Records - CouchDB
1. Sales/returns for digital sales (our retail
partners)
2. Partial records for merchant-fulfilled Amazon
orders
3. Partial records for maps sent into Fulfilled by
Amazon
4. Partial records for MapTrove orders
13. MapTrove Sales - CSV
CSV export of sales data from Magento database
1. E-commerce sales/returns for MapTrove
2. The additional part of MapTrove records from
CouchDB
PayPal API to get PayPal fees for each transaction
15. Amazon Sales - CA, US, UK - MWS API
Retrieve data from the Amazon Marketplace Web
Service (MWS) API
1. Finances - get all individual financial
transactions
2. Orders - financial transactions donât contain
all info we need (e.g. shipping province for tax
purposes)
3. Reports - orders donât contain all the
additional info we need (e.g. warehouse
location for Fulfilled by Amazon orders)
17. Google Sheets - Pricing and Printing Costs
Two additional data sources
1. One Google Sheets document per currency
(CAD, USD, GPB) with print retail prices
2. One Google Sheets document with printing
cost calculation rules
19. Output to PostgreSQL Tables
Four tables contain all of the data we currently need
1. Transactions - records for sales, returns, marketplace fees
2. Printers - records for printing costs for each sale
transaction record
3. Licensors - records for data royalties for each sale
transaction record
4. Shipping - records for âAmazon Buy Shippingâ costs
These tables are combined into various views for report creation
22. MWSCaller Custom Transformer
â Currently tested with three Amazon MWS API paths:
â Finances
â Orders
â Reports
â Outputs XML or plain text (when possible - i.e. for some reports)
â No one place has all of the information we need. We make calls to
the three API paths and combine the data together for each sale
23.
24. Google Data Studio
Reports can be created, accessed and
modified for accounting and business
decision making
25. Google Data Studio
Several database views have been
created from the four base tables to
satisfy report requirements
Allows us to optimise for each report
without having multiple places to update
the base data
26. In combining all our sales and
production data in one place we can
now more efficiently do billing,
invoicing and planning
We have relationships with publisher from around the world and are licensed to provide their maps to our retailer network.
Custom maps: UK Ordnance Survey 1:25,000 and 1:50,000 topo maps, Canada/USA streets made from TomTom data, Lucidmap city/province maps
Print on Demand maps: National Geographic historical back catalog, 10th Atlas of the World, Reference series; Lucidmap Canadian cities/provinces, Backroad MapBooks Topo sheets, Twisted Spirit Mythical Monsters Maps
Our retailers all have accounts in MapSherpa where they can search for custom and print on demand maps for their customers. They are able to order and download PDFs to be printed. These orders have data royalties owed to the map publisher calculated when they are submitted to the system.
MapSherpa OnDemand API allows customizing and ordering of all of the custom maps and print on demand maps available to the retailer doing the integration. There are methods for listing available custom/POD maps, bringing up the customize dialog, setting page size, orientation, title, etc.
We have agreements in place with many of our publishing partners to sell their maps on Amazon and on our own e-commerce site. The royalty model is different from the one calculated automatically on digital sales, so this needs to be handled when we process the financial information each month. We also have to calculate the production costs based on printing rules established with our printing partners.
When sales volume was low, it was feasible to accumulate and calculate all required information manually, though this was prone to error. Now we are at a point where it is no longer practical, and for the last year we have been working on refining a database solution containing all relevant data and a front end solution of reports.
First Click: MapSherpa records
Second Click: Amazon records
Third Click: MapTrove/PayPal records
Fourth click: writing to postgresql
MapSherpaâs back-end database isCouchDB. We store all sales and returns of digital (retail partner) sales in the MapSherpa database, along with orders created to retrieve the PDFs for print sales (amazon, maptrove). We also send maps into Amazonâs warehouses, and we need to store and calculate appropriate data for these maps as inventory.
Make sure the view containing the appropriate couchdb documents exists in the database, request all records for the desired time frame, and do a little clean up.
Inventory orders in the MapSherpa database need to have the âprint retail priceâ set, which is different from the content access fee that MapSherpa stores by default for digital sales.
MapTrove integrates the MapSherpa API, so when a map is ordered, the appropriate sale is recorded in the MapSherpa database. Each month we use a CSV sales report to get the sales revenue, shipping revenue, taxes, etc. and join them to the MapSherpa records.
MapTrove uses PayPal as its payment processor, so we also need to retrieve the paypal fees for each transaction and store them along with the sales information.
Top: get appropriate data from the PayPal API, format it and filter by sales and returns
Bottom: read in the sales report CSV, format the records, join to paypal records, join to MapSherpa records, set attributes accordingly
We use three parts of the MWS API to retrieve the information we need for each sale/refund transaction.
Basic workflow:
Get financial transactions (XML), filter them so we know what kind of transaction they are and format the attributes
Get orders for all sales and returns in the current time period being processed and join them to the financial transactions
More attribute formatting, filtering seller and amazon fulfilled transactions
Join seller fulfilled transactions with mapsherpa records
Get all fees and charges for sales and returns
Get amazon fulfilled orders report so we can join to amazon fulfilled transactions so we know what warehouse the maps came from
We store the current retail price of all maps sold in Amazon and MapTrove in a google sheet as it is quick to update prices based on production cost changes in a spreadsheet. Once the sheets are updated, changes are applied to the marketplaces
We have one sheet per region with printing rules by finish, page size, etc. that is used to calculate printing costs in FME
Printing cost rules are also stored in a Google Sheets document and are retrieved for calculating printing costs
All records end up here to have data licensing fees and printing costs calculated as appropriate. Finally everything is saved to the database, and the materialized views build from main tables are updated.
The custom transformer itself is pretty basic with just a handful of transformers, but the main part is a python caller that makes the appropriate requests to the MWS API
The current endpoint of all of this is a collection of Google Data Studio reports. This allows us to quickly put together the reports we need and make changes as necessary, which occasionally requires changes to the views in the database.