Come explore with us CartoDB - Ext JS components (www.cartodb.com). These new components will allow you, as developers, to visualize and interact with geospatial data using up to a billion data points in real time. We will show you how easy it is to enable visualizations, filter dynamically, create time-lapse animations, and explore large location datasets at unprecedented scale. Come learn how to use these new open source components to build interactive geospatial visualizations that deliver solutions, value, and insights to your customers.
2. What is CARTO?
• Built on Open Source Technologies
• Runs on top of PostGIS Database
• SaaS, On-Premise, DIY
• Generates Raster Tiles and Geospatial Data
• Full SQL API, JavaScript, and 3rd party libraries
• Great User Interfaces to design and publish maps
3. Geospatial Data & Maps
BY 2020, APPROXIMATELY 25 BILLION DEVICES WILL BE
PERMANENTLY CONNECTED THROUGH THE INTERNET,
ALL OF WHICH WILL GENERATE LOCATION DATA.
SOURCE: GARTNER
4.
5. BUSINESS PRESSURES
• Massive sensorisation due to IoT
• Consumerization of data & maps
• Data storage is more accessible
than ever
• Mobility demands location awareness
• Empowerment of data scientists & business professionals
THE POWER OF PLACE OPTIMIZES
OPERATIONAL PERFORMANCE
6. MARKET DRIVERS
• Continuous need for deeper knowledge
on business drivers: location is crucial
• Use location insights to increase your
operational efficiencies
• Limit risk by fully understanding the
impact of location on your decisions
THE MORE DIGITAL WE BECOME,
THE MORE LOCATION MATTERS
7. Data Sizes
• < 1,000 Data
Points
• 1k to 1M
• 1M to 10M
• 10M+
17. CARTO Map Component
• Center on any country using ISO3166
country code
• Provide Array [lat, lng] format
• Setting a default zoom level
items: [{
xtype: “cartomap”,
center: “jp”
}]
items: [{
xtype: “cartomap”,
center: [30, -90]
}]
items: [{
xtype: “cartomap”,
center: [30, -90],
defaultMapZoom: 9
}]
http://rawgit.com/CrestoneDigital/extjs-
carto/master/carto/examples/basicMap/basicMap-example2.html
18. items: [{
xtype: “cartomap”,
center: “us”,
layers: [{
subLayers: [{
store: {
autoLoad: true,
proxy: {
username: 'crestonedigital',
table: 'petroleum_refineries'
}
}
}]
}]
}]
CARTO Map Component
• Easily add a basic layer to your map
• Map has layers
• CARTO layers have multiple subLayers
http://rawgit.com/CrestoneDigital/extjs-
carto/master/carto/examples/basicMap/basicMap-example3.html
19. CARTO Map Component + CARTO Store
• More advanced map with a grid binding (map to grid)
http://rawgit.com/CrestoneDigital/extjs-carto/master/carto/examples/gridMap/gridMap-example1.html
21. CARTO Map Component + CARTO Store
• More advanced map with a grid binding (grid to map)
http://rawgit.com/CrestoneDigital/extjs-carto/master/carto/examples/gridMap/gridMap-example2.html
23. Carto Map Component
• More advanced map with grid locked to map extents
http://rawgit.com/CrestoneDigital/extjs-carto/master/carto/examples/gridMap/gridMap-example3.html
25. CARTO Store
• Handles all the building of CARTO SQL API Queries to retrieve data
• Can be used to access data for GridPanels, DataViews, Charts
• Store Filters, Sorters, Limit, Paging are all supported
• GroupBy: ‘value’ turns a store into a Facet that includes the name, count,
avg, min, and max values. Most commonly used for ComboBoxes to filter
data
• Can be used for Autocomplete ComboBoxes
• Filters, Limits can be used with GroupBy for sub selections
• Flags to multipurpose store to retrieve Tables and Table Fields
28. CARTO Explorer
• Browse any CARTO account
• List and view all Public tables
• Change base layers
• Filter any field
• Edit CartoCSS
• Explore statistics
• Bind statistics to viewport
• Styles outliers using STDEV
http://rawgit.com/CrestoneDigital/extjs-carto/master/carto/examples/playMap/playMap-example1.html
Good morning, thank you so much for being here. I want to thank Sencha for inviting me to speak today. My name is Michael and really happy to see everyone show up for this session today. I know there other great talks going on and hope that you will enjoy seeing some maps come to life in ExtJS.
A little bit about myself before we begin. I am Michael Giddens the CEO of Crestone Digital which is a software development and consulting firm located in Denver. I have been working in the data science, open data and the software development field for 17 years as both a developer and a solutions architect. I have been writing applications in ExtJS 3, 4, 5, and 6 and really happy to see the direction that Sencha is going with the framework. For the last decade I have been working in both the scientific and commercial scene working with big data mainly in the geospatial field.
What I am really excited about is sharing what we at Crestone have been doing is to bring geospatial components directly into ExtJS and share where we are at and how the components work and how you can start using these today.
CARTO is an open source geosaptial and mapping company. All of their products, libraries, and code are available in GitHub. Their database is Postgres, with PostGIS and their own libraries added on top. Their business model comes in by providing software as a service, on-premise and support installations. There is and has always been a free tier that is available for anyone to start using the geospatial platform.
CARTO is a company that focuses on their stack and in turn works with other companies like Crestone to work with clients that do not have development teams or expertise in location intelligence. Being that we love ExtJS and CARTO we felt that this was a great time to bring the two products together by creating native open source components.
Now that we know what CARTO is, what does CARTO do? On the high level it is a "Bring your own data", optionally mashing it up with their demographic service that can turn your data or address information into a comprehensive and engaging data product. It will then take that data that you have provided and generate raster tiles much like you would see in Google Maps, Bing and other traditional map engines. The same data can also be exported in GeoJSON, Shapefiles, CSV and a few other useful formats.
What really makes this platform powerful is the SQL REST API. This means you can take any query and do basic filters or even more complex spatial joins, intersections and so on to produce the data incites needed for your data product. Now if you were just using their interface they have a lot of quick start wizards to do bubble maps, heat maps, time-lapse and more. Even if you are not making maps right now I would really encourage you to go check out their blogs on what is trending right now in the geospatial world.
It is estimated that by 2020 about 25 billion devices will be permanently connected and online, all which will be generating location data. What business owners and decision makers need to know is, what are the opportunities and things that need to be done to grow the business and make the right decisions. It is the people that are caring around the phones that are producing the data, it is the tweets that people are creating. It's your social media interactions, credit card transactions, and everything is being store somewhere now. And in most cases it is including geocoded location data. From what we have seen with our customers and talking with others is that most companies are not really taking that information into account when making decisions.
So what we do as developers is to help build data products that help these decisions makers evaluate and understand this information, to look for value in the information. With ExtJS and CARTO combined it is a great fit for these types of location intelligent applications. For the things we work on at Crestone it is a lot of data management systems, generating reports, exploring data in various formats so adding interactive maps has become essential. What we call the new “informative maps” is really Location Intelligence, and really is a new emerging field in the industry and right now CARTO leading this industry. Towards the end after I go through the components we will look at a sample project that let us see and explore real world data.
Map engines that you see most often are Google Maps, Bing along with other search engines. You have probably even used them to find this hotel, check for restaurants and driving directions for those fortunate enough to live close by. Maybe your company or government agency is using ArcGIS to visualize different data layers. But really viewing layers on a map is only good if you can ask it questions. These new ExtJS Components will let you generate maps and put point, lines, and polygons on maps, and I am going to show you how to use the same data from those maps for populate grid, dataviews, graphs and tie them all together.
Businesses are starting to get pressure about the decisions they make now that the world is becoming more and more globally connected. We have a growing number of Internet of things coming online and more and more users expect maps to be real-time and interactive, data storage is becoming cheaper and even governments are working towards sharing more open data and transparency.
There are even market drivers that are pushing for Location Intelligence. We have seen a need for deeper knowledge on what drives customers and seen that businesses are not taking advantage of location insights. If we can limit our risks for our marketing departments and sales engineers it will help to run a more efficient company. As developers, while it is not our job to be an expert in all of these roles, it is important to understand what they do to hopefully build a product that fits their needs.
As a map developer it is also important to know what types of limitations exist in building maps. Anyone that has used or worked with Google Maps or other client side markers knows that the performance of the application is relative to the quality of the computer, the version of the browser and memory available. While browsers are continuing to improve and ExtJS is continually getting more efficient for larger enterprise applications we still deal with limitations.
[Click]
When Google Maps starting to set a new standard for online mapping back in 2005 it was truly cutting edge. Typically we call these interactive maps as slippy maps. But putting more than a 1,000 markers on a map was really just way too much. Fast forward 11 years, the browsers and computers have greatly improved but placing 1,000 or more markers on a map is still a challenge. So most slippy map engines use raster tiles, 256 or 512 pixel images that are generated on a server in the cloud and delivered to the browser to give us that immersive map experience. There are various base layers from satellite imagery to your more traditional road map and all that gets generated from data. To give you an example. Open street map which is all the data from the world is 720GB of uncompressed XML data containing around 3.6 Trillion points of interests and 375 million road segments. So in cases like these, base layers tiles are cached and pre generated due to the time and complexity to render these tiles.
Where CARTO comes in is a way to generated live tiles from your data and overlay them on top of these base layers. Their servers also cache tiles on a Content delivery network when the data has not changed to provide even better performance when serving tiles with larger amounts of data. For databases up to 10 million point are great use cases for live interactive maps.
[Click]
Projects that have more than 10 million it is best to understand what is needed and architect a hybrid of performance vs data exploration. As you can see here with this last example this dataset was beyond the 10 million marker and at global zoom levels like this, Paris for example only takes up about 10x10 pixels so no sense in analyzing millions every time if we can normalize the data at these levels then use dynamic mapping at lower levels when we are only looking at a smaller geographical boundary.
I would like to show you a few use case examples of how CARTO is being used now and hopefully might align with the types of applications you are making and give you some ideas of things you can do with location data.
http://bbvatourism.vizzuality.com/
BBVA wanted to see minute by minute of how and where transactions were being made by different nationalities and types of purchases. BBVA has been analyzing territorial dynamics based on anonymized data of credit card transactions for more than 4 years. The data visualization was crafted by the BBVA Data & Analytics team together with Vizzuality to allow visitors to unveil the where, when and what; ever wondered when the French buy their food, or which places the Germans flock to on their holidays? In this example we can see the dynamics of spending in Spain.
In this project Cash Atlas was developed with the CARTO platform to provide a tool that can be used in advocacy to promote considerations and appropriate use of Cash Transfer Programs in humanitarian response situations. The Cash Atlas is an interactive global mapping tool aiming to visually represent cash transfer programmes at a global level.
The next is a exploration of the busiest airports in the United States. That simple question led another CARTO partner Geographica on a path to insights and analyses of airports and flights all around the globe. Using CARTO, they developed a data-driven interactive visualization to derive insights on thousands of data points.
Visonomy has been working on an interactive tool for climate risk called their Vizonomy climate risk terminal which assesses flood risk and planning for sea level rising using real-time data and dynamic mapping.
With the access to more open data and CARTO having their home base in New York we see a lot of interesting visualizations in that area. One that seems interesting is how people are using their homes in New York City for rental property using Air B&B. This interactive map visualized the median days by neighborhood needed to make rent. In most cases it is around 4-7 days. Due to these type of studies New York is creating new laws regulating New York residents from using their homes as rental properly and is a hot topic right now. You can see how one map can shed light on a very controversial topic and get people to start asking more informed questions.
Finally a quick one on social data. In this case it is visualizing all the twitter data during Super Bowl 50 and looking at the categorical tweets of teams and players. This was an interactive time-lapse that shows how post marketing analysis can be used to see impacts and social trends.
So all of these examples were not ExtJS and has started to become a problem when we want to bring CARTO into our clients applications. So we decided to change that and today we are happy to introduce a suite of components to now make maps like these now possible easily in ExtJS.
We have been working on these components for a few months now and testing them in real world projects. We are partnering with CARTO and these will be the official CARTO components that will stay open source and continued to be improved and supported. It is our hope that you will be as excited about these are we are and provide us feedback on how we can improve these to do even more.
At this time we are happy to introduce CARTO map component, proxy and store to easily connect and work with CARTO tables in ExtJS 6. It supports MVVM style development, Binding and has the feel of other ExtJS components.
These components integrate natively in ExtJS and works directly with filters, sorters, Grid Panels, ComoBoxes, Charts, and Data Views. Even paging tool bars. Basically anything that uses a store can work with the CARTO store. There are even a few extra things our components do to leverage your CARTO tables.
So lets dive into some code and see how it works. Lets start with the simplest of components and build up the features from there. To start with lets just add a base map with no data. To do this we only need to include the library and use the xtype "cartomap". This will use all the default settings and show a map that will fit and layout as any normal Ext.Component.
Now I know that is not very exciting but for us it is a great start and maybe you just want to zoom into an area relevant to some other information on your app. CARTO provides a choice of many predefined base layers to choose from and is easy to set the basemap from a predefined list. Now if you have your own base layers or use 3rd party ones like Open Street Map or Map Box for those more familiar with base layers services. These all can be set by providing the url and optional attribution. Basemap is bindable and can easily be connected to a combobox.
The starting location on the map is also important when setting up your map. To do that we have two attributes called center and defaultMapZoom. We have two ways of defining a starting center point. The first is by using the ISO3166 country codes and the second is in lat/lng array notation. The map zoom tells at which level the map should be rendered. Those who are not familiar with zoom levels they typically range from 0 which is the world and 18 being around street level. Again both these values are bindable so great if you had a combobox of countries and zoom levels for each.
So now that we have a map lets try making use of your CARTO tables. To do this we have a property called "layers" much like components have items. This can accept a list of arrays each representing a layer on the map. CARTO is a little special when it comes to maps and has something called a subLayer. What makes these subLayers different than traditional layers are that these are merged together server side on CARTO and returned as one tile layer. This is great if you are trying to show 5 layers of data and do not want to return 5 * the number of tiles on the screen.
So when setting up a sublayer you need to provide a store. This store assumes a CARTO store and does not work with any other type of ExtJS stores. In that store we need to also define a few values for our CARTO proxy. The username is the actual CARTO account and the table is the name of your dataset. And that's it. The autoLoad will add the sublayer to the map at the time of initialization as you would expect with any store. The orange dots you see are the default style but can be changed as you will see in a bit.
[Show Example]
So at this point you can load any data as quickly as that to your project in just a matter of minutes. But really we want to do more. So in this next example we are adding a few more things to the example and hooking a grid up to one of the subLayer stores.
[Show Example]
So you will see a few new things introduced. The first is we have styled what is called the CartoCSS. CartoCSS is a language created by the Mapbox company to create styles for raster tiles. It is very versatile, and as the name suggests, similar to CSS with cascading style rules. We have also changed the baselayer and added in a hover tooltip. Even though we are returning tiles in the form of png images there is an interactivity mode that makes a json request to the CARTO server and retrieves a hashkey of all the pixels on that tile. When you mouse over the corresponding pixel it will retrieve the related metadata to that record and fire an onHover event with that item record. Finally we have added in a normal grid panel and bound the click.
So what does that look like?
What you see in green are the new additions. On the left you will see that we set the selection to be bindable. To make the base layer dark we used the Dark Matter base layer provided by CARTO. To enable the heatmap style we have a few predefined types already setup and the one we are using in this example is "intensity". This takes the markers that overlap and applies a "multiply" much like you would see in those familiar with Photoshop.
We bind the subLayer to the 'layer' store using the default bind property. Similarly to the grid, if the bind config is given a string, the subLayer assumes that it is referencing the store.
The layer store is defined in the viewModel's stores config. It is a carto store, and it's proxy is given the necessary username and table.
Finally for the mouse over tooltips we have to enable the interactivity settings. By setting enable to true it will activate the listeners for click, onHover, onHoverOut. Setting the tooltip enable to true activates the tooltip when you mouse over that sublayer of points, lines, or polygons. This interactivity object also tells which fields to pull from the table to reduce the amount of overhead in situations where you may have many columns or large values not needed for display.
Now with the Grid Panel this should look very familiar since we are just binding the selection to the same 'selectedValue' as well as the store to the 'layer' store.
Since 'selection' is two-way bindable on both the grid and the map, selecting one sets the selection on the other and vice versa.
You will see in the grid column model now that we have the store attached to the maps sublayer, you can point the dataIndex to those same fields. And as simple as that we have a map and grid all coming from a geospatial table in CARTO.
So lets do some more binding of map components and kinda go backwards now.
[Show example]
In this example we do not show the data on the map. What we want is to show a marker in the place that the record is associated to and center to that location on the map. We also want to show a "More details" off to the right which is a normal dataView / xTemplate. So how does this work.
Going off of what we have learned so far we have added a few new things. The first is the selectedAction property which takes an array of values that defines how the map behaves when the maps's selected value changes.
We then bind to the layer store in the viewModel for the map, grid and xtemplate. Finally we have to activate an enableLatLng property in the proxy to allow the selectedActions to listen to the features location.
Adding onto the features and building blocks of these components is the ability to automatically filter the store based on the extents of the viewport of the map. We have a quick little demo to show how this works.
[Show example]
Right now we have a few records throughout the US and the associated store to that subLayer attached to the grid panel. By turning on the Lock to Map you will see that the store results automatically get updated to the view port extents. When I zoom in and around the map those results get updated in real time.
So lets see what this looks like in the code.
So in our view Model we add a storeId so that we can reference this later in our map. We also add a reference in the checkbox to bind that value of the mapLock property. By enabling and disabling the mapLock property will tell the store whether or not to add the bounding box to the proxy's queries that are happening behind the scenes.
The other new property is the storesToLock. This is what registers to the mapLock to dispatch the updates to all stores in that array. By adding and removing stores to that property will allow you as a developer to give control over which stores are updated when the map moves.
So the great thing about the components we have developed is you never need to use the map even if you have your data stored in CARTO. CARTO is designed with a RESTful API and this whole time we are just obfuscating the SQL behind the scenes in the proxy to retrieve the data and map that back to the store. So by using a store directly or in a view model like shown in the past examples you can easily set things up to work with GridPanels, dataViews, and Charts. It also works with Sorters, Filters, Limit, Paging and all the normal grid plugins and features most of you are used to using.
We Have added a new GroupBy value which turns the normal fields store into aggregation features that are really powerful for generating stats. Any PostGres stats aggregation will work with this feature. Additionally filters and limits can be used with the group by for sub selections.
Finally we do have 2 unique flags that repurpose the store to retrieve a list of tables from your CARTO account and a list of column fields from a table. We do that by setting the mode to tables or columns in the proxy.
We also have options to aggregate the data and have examples on GitHub to show how these work.
So we have put together a little kitchen sink that lets you explore your CARTO account and data to see how it all works together. In this examples we can explore the tables, add filters, bind to the viewport and edit the CSS. Lets take a look at how this works.
[Show Example]
By picking a table it is adding that layer to the map and dynamically generating a list of filters from the column names store. Each of these filters is using the group by to give us the ability to go right in and filter the data.
We can then come over to the statistics and then pick a column to get the group by and count of those records in the database. we can sum up any numerical value. As you can see we have the store bound so any changes automatically update and take into account the filters being set in the other tab.
Now we adding in some other stats on the tweets to se some min max. We have a simple transform in the reader of the proxy that is looking at all the data and calculating some standard deviation steps and adding those as new fields to the store. We then us a getRowClass in our view config to determine which css class to set for that row. The nice part is as we change the data the standard deviation steps are recalculated to only the results being returned.
Now this is for all data and filters. We can go one step further and turn on the bounding box constraint and at that point as we move the map around all the values are updated dynamically.
Finally I would like to show you a more real world data product that you might see this being used. In this examples we have tried to make use of all the different features shown today to build a pretty complex project. This example the data was loaded, designed and build in just over one day by one developer.
[Show Example]
What you are seeing is a map of 700 thousand data points. Keep in mind that this data is being generated serverside on a non dedicated CARTO account and are also limited to the internet connection here. So on the first screen we see the map with all the points. We can color code them by natual and human related files. Right away I see that most of the natural fires are only in the wester side of the united states.
Now if I go to my filters and look at say the state of COlorado it will update the map but also all the related stores get updated. So the summary is now showing 36k fires and the statistics have calculated the total acres and average acres for each year burned. We can switch over to standarddeviation to the number of files and learn about other outliers. But what interested me is that over the history of this data there have been 2 years that really are over 3 steps from the average.
If I go do a quick search in google for "list of Colorado wildfire" I see right away the Hayman Fire in 2002 and Hire Park fire in 2012 cause an enormous amount of data. These are things I would not have easily found just by looking at the original CSV file.
We can take our exploration one step further in this demo by turning on the bounding box and zooming into the map. As we do so you will again see the stats changing as we move around. All without ever writing a query.
The library is available today on GitHub, free and open source and if you are interested in our work or would like to contribute please feel free to get in touch.
That is all I have for you today and thank you for joining me.