How to import 1 million SKUs in under 10 minutes

How to import 1 million SKUs
in under 10 minutes
1/26

Building an import correctly is hard
2/26

2008
it all started with
DataFlow
3/26

DataFlow
Stores CSV
records into
database
4/26

DataFlow
Stores CSV
records into
database
Imports product
by product via
AJAX call
4/26

DataFlow
Stores CSV
records into
database
Imports product
by product via
AJAX call
Uses product
model to save
data
4/26

DataFlow
Stores CSV
records into
database
Imports product
by product via
AJAX call
Uses product
model to save
data
Closing the
browser window
stops the whole
process
4/26

Speed
2-3 products per second
5/26

Speed
2-3 products per second
~ 20 minutes for 5k products
5/26

2011
Import/Export saved us all
6/26

ImportExport
Stored batches of
data into the
database during
validation
7/26

ImportExport
Stored batches of
data into the
database during
validation
Processes stored
data in one HTTP
request
7/26

ImportExport
Stored batches of
data into the
database during
validation
Processes stored
data in one HTTP
request
Validates product
data without using
product model
7/26

ImportExport
Stored batches of
data into the
database during
validation
Processes stored
data in one HTTP
request
Validates product
data without using
product model
Uses multi-row
inserts to
populate tables
7/26

ImportExport
Stored batches of
data into the
database during
validation
Processes stored
data in one HTTP
request
Validates product
data without using
product model
Uses multi-row
inserts to
populate tables
Does not run
indexers
7/26

Speed
41 product per second
8/26

Speed
41 product per second
~ 2 minutes for 5k products
8/26

But there are some
drawbacks
9/26

High memory usage on large datasets
But there are some
drawbacks
9/26

High memory usage on large datasets
Slow in generating primary keys for new products
But there are some
drawbacks
9/26

2015
Magento 2.x
Import Export
10/26

Same base functionality as in M1
ImportExport M2
11/26

More complex ﬁle format to edit and parse
ImportExport M2
11/26

Slower on complex product data
ImportExport M2
11/26

Slower on complex product data
Adds additional single statement inserts
ImportExport M2
11/26

2019
I got an idea and a project to implement it on
12/26

Separate Feeds
Main entity (sku, type, set)
13/26

Separate Feeds
Attributes (sku, attribute, store, value)
13/26

Separate Feeds
Category (sku, category slug, position)
13/26

Separate Feeds
Conﬁgurable Options (sku, attribute, label)
13/26

Separate Feeds
Images (sku, image)
13/26

Separate Feeds
Images (sku, image)
...
13/26

Lazy Entity Resolving
Reduce memory requirements of the import
14/26

Cleaner and more readable feed processing
14/26

Cleaner and more readable feed processing
Possibility of acquiring entity ids in batches
automatically
14/26

$resolver = $this->resolverFactory->createSingleValueResolver(
'catalog_product_entity', 'sku', 'entity_id'
);

$insert = InsertOnDuplicate::create(
'catalog_product_entity_varchar',
['entity_id', 'attribute_id', 'store_id', 'value']
)->withResolver($resolver);

$insert
->withRow($resolver->unresolved('sku1'), 1, 0, 'some value')
->withRow($resolver->unresolved('sku2'), 1, 0, 'some value1')
->withRow($resolver->unresolved('sku3'), 1, 0, 'some value2');
15/26

Conﬁgure table lookup information
);


$insert
15/26

Pass resolver into insert builder
);

$insert
15/26

Use resolver to create identiﬁer containers
);


$insert
15/26

Insert on duplicate will skip any unresolved entries
);


$insert
15/26

Batch auto-increment generation
START TRANSACTION;

INSERT INTO catalog_product_entity (sku)
VALUES
('sku1'),
('sku2'),
('sku3'),
('sku4');

SELECT entity_id, sku
FROM catalog_product_entity
WHERE
sku IN ('sku1', 'sku2', 'sku3', 'sku4');

ROLLBACK;
17/26

Start a transaction
START TRANSACTION;

VALUES
('sku1'),
('sku2'),
('sku3'),
('sku4');

WHERE

ROLLBACK;
17/26

Populate table with un-resolved keys

VALUES
('sku1'),
('sku2'),
('sku3'),
('sku4');

START TRANSACTION;
WHERE

ROLLBACK;
17/26

Retrieve new identiﬁers
WHERE

START TRANSACTION;

VALUES
('sku1'),
('sku2'),
('sku3'),
('sku4');

ROLLBACK;
17/26

Rollback transaction
ROLLBACK;
START TRANSACTION;

VALUES
('sku1'),
('sku2'),
('sku3'),
('sku4');

WHERE

17/26

Compile query for constant batch size
Prepared Statements
18/26

Send only data instead of generating new queries
Prepared Statements
18/26

Send only data instead of generating new queries
Reduces query processing on MySQL side by half
Prepared Statements
18/26

Speed
450 products per second
19/26

Speed
450 products per second
~ 11 seconds for 5k products
19/26

But it was still not good enough
20/26

But it was still not good enough
45 minutes to import 1 million SKUs
20/26

Sure, because it's a sequential process...
21/26

So I made it asynchronous as PoC
22/26

Under the hood
Each target tabletarget table receives a separate connection
to MySQL
23/26

Under the hood
to MySQL
Identity resolver is attached to the source table
connection
23/26

Under the hood
to MySQL
connection
Each feed is processed concurrently by using
round robinround robin strategy
23/26

Under the hood
to MySQL
connection
Each feed is processed concurrently by using
round robinround robin strategy
During MySQL query execution PHP prepares the
next batch
23/26

Speed
1,850 products per second
24/26

Speed
1,850 products per second
~ 9 minutes for 1m products
24/26

It is coming this fall as an open source tool for
everyone!
25/26

Questions
ivan@ecomdev.org
IvanChepurnyi.GitHub.io
26/26

How to import 1 million SKUs in under 10 minutes

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie How to import 1 million SKUs in under 10 minutes

Ähnlich wie How to import 1 million SKUs in under 10 minutes (20)

Mehr von Ivan Chepurnyi

Mehr von Ivan Chepurnyi (8)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

How to import 1 million SKUs in under 10 minutes