Caching is one of the most popular ways to optimize the performance of web applications. Magento is not far from this trend and the primary approach to speed up slow code from the core team is to cache it. This approach works well for small projects where the rate of inventory changes is shallow. But as soon as you enter sales season or receive regular product updates from the external system, your cache hit ratio plummets. In this talk, you will learn how to ensure that your system’s uncached response times are as light as possible when you get a cache miss by minimizing the I/O of your application with intelligent batch data preloading.

1. Optimizing Magento
By Preloading Data

2. Disclaimer
This talk is not about cache optimization or pre-warming

3. Cache Pitfalls
● Combination of variables that invalidate it
● Slow updates in the backend
● Cache memory usage grows with the catalog size
● Caching onion nightmares

4. But do we need caching?

5. How many page parts are cached?

6. How many page parts are cached?

7. What’s problem that cache solve?

8. Top Menu
On each menu item Magento does a call to obtain url of category

9. Product List
On each configurable product all assigned simple
products are loaded to its calculate price

10. Load in the Loop (N+1 problem)

11. Why does it exist in Magento core?
- Introduction of excessive abstractions
- Promotion of singular service access
- System is designed from the bottom up

12. Can we fix it without modifying core?
- Observe load events on collections of interest
- Load data for related items in the collection
- Create a plugin around a service that triggers
N+1

13. There is a module for that!

14. https://github.com/EcomDev/magento2-product-preloader

15. 1. Implement DataLoader interface on your custom service with:
○ isApplicable(string $type) method for deciding if loaded
○ load(ScopeFilter $filter, ProductWrapper[] $products) for loading
related data for a list of product at once
2. Register custom loader as a dependency in di.xml for
LoaderService
3. Use LoadService for accessing data in your plugin with:
○ get(int $productId, string $type) method for accessing data
○ has(int $productId, string $type) method for checking when preload was
done for this product
How to use

16. Example with configurable products

17. Target Goal
● Remove load in the loop for the price
retrieval
● Remove load in the loop for the inventory
check
● Achieve below 90 queries on category page with
configurables
● Simple product status is ignored in the
example for simplicity

18. Loader For Simple Product Prices in Configurable
class ConfigurablePrice implements DataLoader
{
public const DATA_KEY = 'configurable_price';
private ResourceConnection $resourceConnection;
public function __construct(ResourceConnection $resourceConnection)
{
$this->resourceConnection = $resourceConnection;
}
public function load(ScopeFilter $filter, array $products): array
{
/*...*/
}
public function isApplicable(string $type): bool
{
return $type === self::TYPE_LIST;
}
}

19. Loader For Simple Product Prices in Configurable
public function load(ScopeFilter $filter, array $products): array
{
$configurableProductIds = [];
foreach ($products as $product) {
if ($product->isType(Configurable::TYPE_CODE)) {
$configurableProductIds[] = $product->getId();
}
}
if (!$configurableProductIds) {
return [];
}
$connection = $this->resourceConnection->getConnection('catalog');
$priceIndexTable = $this->resourceConnection->getTableName('catalog_product_index_price','catalog');
$relationTable = $this->resourceConnection->getTableName('catalog_product_super_link','catalog');
/*...*/
}

20. Loader For Simple Product Prices in Configurable
public function load(ScopeFilter $filter, array $products): array
{
/*...*/
$select = $connection->select()
->from(['index' => $priceIndexTable], [])
->join(['relation' => $relationTable], 'relation.product_id = index.entity_id', [])
->columns([
'entity_id' => 'relation.parent_id',
'price' => 'MIN(index.price)',
'final_price' => 'MIN(index.final_price)',
])
->group('relation.parent_id')
->where('relation.parent_id IN(?)', $configurableProductIds)
->where('index.customer_group_id = ?', $filter->getCustomerGroupId())
->where('index.website_id = ?', $filter->getWebsiteId())
;
return $connection->fetchAssoc($select);
}

21. Plugin on Price Resolver
class ConfigurablePriceResolverPlugin
{
private LoadService $loadService;
public function __construct(LoadService $loadService)
{
$this->loadService = $loadService;
}
public function aroundResolvePrice(ConfigurablePriceResolver $subject, callable $proceed, SaleableInterface $product)
{
$productId = (int)$product->getId();
if ($this->loadService->has($productId, ConfigurablePrice::DATA_KEY)) {
return (float)($this->loadService->get($productId, ConfigurablePrice::DATA_KEY)['final_price'] ?? 0);
}
return $proceed($product);
}
}

22. DI configuration
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
<type name="EcomDevProductDataPreLoaderDataServiceLoadService">
<arguments>
<argument name="loaders" xsi:type="array">
<item name="configurable_price" xsi:type="object">EcomDevConfigurableProductPreloaderLoaderConfigurablePrice</item>
</argument>
</arguments>
</type>
<type name="MagentoConfigurableProductPricingPriceConfigurablePriceResolver">
<plugin
name="use_preloaded_final_price_of_the_product"
type="EcomDevConfigurableProductPreloaderPluginConfigurablePriceResolverPlugin"
/>
</type>
</config>

23. See rest of the loaders in the link at the end

24. So what’s the result?
https://blackfire.io/profiles/compare/6285d4fd-f239-4845-a72a-f596e9d035dc/graph

25. How about under load?

26. Key takeaways
- Avoid using cache for data within cached page
- Use single query per set of data
- Disable block_html cache
- Use profiler to find bottlenecks

27. Future plans
- Make it a plug-and-play module
- Create common preloaders of product data
- MSI qty retrieval for shopping cart and checkout
- FPC identities retrieval for composite products
- Create preloader module for categories
- Rewrite setup:di:compile with higher performance
solution

28. QA & Links
Example in the slides:
https://github.com/EcomDev/example-preloading-configurable-product
Preloader Library:
https://github.com/EcomDev/magento2-product-preloader
Contacts:
https://ivanchepurnyi.github.io/