Skip to content
Snippets Groups Projects
Commit 2ccf0bd0 authored by Olivier Dony's avatar Olivier Dony
Browse files

[FIX] mail: speed up filtering out blocked emails


For large lists of blocked emails, the time taken to load the records in
cache becomes prohibitive. For instance on a sample blocklist of 600k
entries, the `search([])` to load them could take 80-90s to run!
And this toll is taken every time the mail composer processes an email
batch in mass-mailing mode. Technically, most of the time is spent
iterating on the list of ids many time, in order to prefetch fields
into cache by determining what's missing.

Loading the contents of the list in raw SQL takes a fraction of that
time (400ms vs 90s).

Subsequently using a set for lookups in that blocklist is also much
faster (average time complexity O(1) vs O(n)).

Example: looking up an item in a 600k-entry blocklist is easily 5
orders of magnitude faster!

```
In [1]: blocklist = set(x[0] for x in self._cr.fetchall())

In [2]: len(blocklist)

Out[2]: 634610

In [3]: self._cr.execute("SELECT email from mail_blacklist")

In [4]: blocklist = set(x[0] for x in self._cr.fetchall())

In [5]: %timeit "hello@hello.com" in blocklist
The slowest run took 35.29 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 31.6 ns per loop

In [6]: self._cr.execute("SELECT email from mail_blacklist")

In [7]: blocklist = [x[0] for x in self._cr.fetchall()]

In [8]: %timeit "hello@hello.com" in blocklist
100 loops, best of 3: 2.24 ms per loop
```

closes odoo/odoo#57128

X-original-commit: 3b9a8575
Signed-off-by: default avatarOlivier Dony (odo) <odo@openerp.com>
parent 017d1c9d
Branches
Tags
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment