Memory leaks Symfony2 Doctrine2 / exceed memory limit

I have a lot of trouble with the combination of symfony2 and doctrine2. I have to deal with huge datasets (around 2-3 million write and read) and have to do a lot of additional effort to avoid running out of memory.

I figgured out 2 main points, that "leak"ing memory (they are actually not really leaking, but allocating a lot).

1. The Entitymanager entity storage (I don't know the real name of this one) it seems like it keeps all processed entities and you have to clear this storage regularly with

   $entityManager->clear()

2. The Doctrine QueryCache - it caches all used Queries and the only configuration I found was, that you are able to decide what kind of Cache you wanna use. I didn't find a global disable neither a useful flag for each query to disable it. So usually I disable it for every query object with the function

    $qb = $repository->createQueryBuilder($a);
    $query = $qb->getQuery();
    $query->useQueryCache(false);
    $query->execute();
    

So.. that's all I figured out right now.. My questions are:

Is there a easy way to deny some objects from the Entitymanagerstorage? Is there a way to set the querycache use in the entitymanager? Can I configure this caching behaviors somewhere in the Symfony/doctrine configuration?

Would be very cool if someone has some nice tips for me.. otherwise this may help some rookie..

cya

As stated by the Doctrine Configuration Reference by default logging of the SQL connection is set to the value of kernel.debug, so if you have instantiated AppKernel with debug set to true the SQL commands get stored in memory for each iteration.

You should either instantiate AppKernel to false, set logging to false in you config YML, or either set the SQLLogger manually to null before using the EntityManager

$em->getConnection()->getConfiguration()->setSQLLogger(null);

Try running your command with --no-debug. In debug mode the profiler retains informations about every single query in memory.

1. Turn off logging and profiling in app/config/config.yml

doctrine:
    dbal:
        driver: ...
        ...
        logging: false
        profiling: false

or in code

$this->entityManager->getConnection()->getConfiguration()->setSQLLogger(null);

2. Force garbage collector. If you actively use CPU then garbage collector waits and you can find yourself with no memory soon.

At first enable manual garbage collection managing. Run gc_enable() anywhere in the code. Then run gc_collect_cycles() to force garbage collector.

Example

public function execute(InputInterface $input, OutputInterface $output)
{
    gc_enable();

    // I'm initing $this->entityManager in __construct using DependencyInjection
    $customers = $this->entityManager->getRepository(Customer::class)->findAll();

    $counter = 0;
    foreach ($customers as $customer) {
        // process customer - some logic here, $this->em->persist and so on

        if (++$counter % 100 == 0) {
            $this->entityManager->flush(); // save unsaved changes
            $this->entityManager->clear(); // clear doctrine managed entities
            gc_collect_cycles(); // PHP garbage collect

            // Note that $this->entityManager->clear() detaches all managed entities,
            // may be you need some; reinit them here
        }
    }

    // don't forget to flush in the end
    $this->entityManager->flush();
    $this->entityManager->clear();
    gc_collect_cycles();
}

If your table is very large, don't use findAll. Use iterator - http://doctrine-orm.readthedocs.org/projects/doctrine-orm/en/latest/reference/batch-processing.html#iterating-results

1. Set SQL logger to null

$em->getConnection()->getConfiguration()->setSQLLogger(null);

2. Manually call function gc_collect_cycles() after $em->clear()

$em->clear(); gc_collect_cycles();

Don't forget to set zend.enable_gc to 1, or manually call gc_enable() before use gc_collect_cycles()

3. Add --no-debug option if you run command from console.