Speeding up custom feed processes

We use custom scripts to upload product data (new and updated) into Magento. Several months back we were trying to process a “full” feed and update all 7800 products each with around 20-30 attributes. The process was taking 15+ hours and was crippling our site. We needed to find a solution and quickly.

About the same time, we had a Magento core developer working with us on an enablement engagement (one of the benefits of being one of the largest implementations of Magento, at least in the US). The developer reviewed our feed process and discovered an amazingly simple solution. Due to the EAV structure that Magento uses, they make use of an indexer. The indexer can be run manually to refresh everything but it also runs with every save of product data. This is an expensive process. The solution to our problem was to lock (disable) the indexer while our feeds were running. When our feed finish, we were running a full reindex anyway. So there was no need to have the extra overhead of the indexer running during each product save.

Here is how you “disable” the indexer:

Disable the indexer

<span class="line"><span class="x">Mage::getSingleton('index/indexer')->lockIndexer();</span>

And here is how you re-enable the indexer when done:

Re-enable the indexer

<span class="line"><span class="x">Mage::getSingleton('index/indexer')->unlockIndexer();</span>

Finally, to run a full reindex, use:

Full reindex

<span class="line"><span class="x">$processes = Mage::getSingleton('index/indexer')->getProcessesCollection();</span>
</span><span class="line"><span class="x">foreach ($processes as $process)</span>
</span><span class="line"><span class="x">{</span>
</span><span class="line"><span class="x">    echo 'Processing: ' . $process->getIndexerCode() . "n";</span>
</span><span class="line"><span class="x">    $process->reindexEverything();</span>
</span><span class="line"><span class="x">}</span>

Hopefully this will save someone a bit of a headache. It saved us about 13 hours off our run time and left the site responsive even during the less than 2 hours it took to process all 7800 products.

Comments !