Laravel. Chunk Large Queries

When you work with large queries, you may find yourself in position when meory consumption increases rapidly, which may cause your server to run out of memory. To circumvent it, the large queries are split into smaller chunks and then processed separately to keep the memory footprint down.

The Laravel Eloquent has a nice method, which is called chunk, and is supposed to work really nicely. However, it appears it has some kind of bug, which makes it inconsistent, and you will find it usually stops halfway. Here is the snippet, if you want to give it a try.

$currentPosition = 0;
Product::whereRaw('Updated  > Uploaded')->chunk(5, function($products) use ($arg1,$arg2) {
    foreach ($products as $product) {
        $currentPosition = ($currentPosition + 1);
        $this->productUpload($product);
    }
});

The final solution, involves a bit of manual intervention, but it works reliably, very readable, and does not use closures.

$query = Product::whereRaw('Updated  > Uploaded');

$totalToUpload = $query->count();
$perChunk = 5;
$numChunks = ceil($totalToUpload / $perChunk);
$currentPosition = 0;

for ($i = 0; $i < $numChunks; $i++) {
    $limitStart = $i * $perChunk;
    $limitEnd = $perChunk;
    $products = $query->skip($limitStart)->take($limitEnd)->get();
    foreach ($products as $product) {
        $currentPosition = ($currentPosition + 1);
        $this->productUpload($product);
    }
}

Happy coding!