-
- Downloads
[FIX] mail: speed up filtering out blocked emails
For large lists of blocked emails, the time taken to load the records in cache becomes prohibitive. For instance on a sample blocklist of 600k entries, the `search([])` to load them could take 80-90s to run! And this toll is taken every time the mail composer processes an email batch in mass-mailing mode. Technically, most of the time is spent iterating on the list of ids many time, in order to prefetch fields into cache by determining what's missing. Loading the contents of the list in raw SQL takes a fraction of that time (400ms vs 90s). Subsequently using a set for lookups in that blocklist is also much faster (average time complexity O(1) vs O(n)). Example: looking up an item in a 600k-entry blocklist is easily 5 orders of magnitude faster! ``` In [1]: blocklist = set(x[0] for x in self._cr.fetchall()) In [2]: len(blocklist) Out[2]: 634610 In [3]: self._cr.execute("SELECT email from mail_blacklist") In [4]: blocklist = set(x[0] for x in self._cr.fetchall()) In [5]: %timeit "hello@hello.com" in blocklist The slowest run took 35.29 times longer than the fastest. This could mean that an intermediate result is being cached. 10000000 loops, best of 3: 31.6 ns per loop In [6]: self._cr.execute("SELECT email from mail_blacklist") In [7]: blocklist = [x[0] for x in self._cr.fetchall()] In [8]: %timeit "hello@hello.com" in blocklist 100 loops, best of 3: 2.24 ms per loop ``` closes odoo/odoo#57128 X-original-commit: 3b9a8575 Signed-off-by:Olivier Dony (odo) <odo@openerp.com>
Loading
Please register or sign in to comment