On Debian based systems, the `tzdata` package is maintained to reflect changes
in timezones and there is no need to upgrade the `python3-tz` package.
On the other hand, for those who are using `pip` and thus our `requirements.txt`,
the package needs to be up to date. By unpinning it in the requirements.txt:

- new installations based on pip will be up to date
- older installations based on pip can easily upgrade
- debian based installations have to maintain the tzdata package
- mixed installs like on runbot will rely on Debian tzdata

closes odoo/odoo#117527

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

lxml 4.6.1 has issues with python 3.10. So we can safely mimic 15.0 as
it's Focal based too.

Error encountered while importing etree from lxml:
etree.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

Part-of: odoo/odoo#98081

We have our own html2plaintext, already used in lot of use cases instead of
just a few for the html2txt library.

Notably for emails: most emails going through Odoo stack use our simple
html2plaintext to format the body alternative. When no body alternative
is given to ``build_email`` an alternative is built using the library to
remove. Using our own parser allows to have the same results compared to
using ``MailMail.send()``. Difference lies in spaces and new lines as well
as markdown. Our html2plaintext is a bit simple and does not try to generate
Markdown but generates a simple plaintext version.

This also helps solving some issues with depending on that library.

Task-2702034

closes odoo/odoo#82330

X-original-commit: odoo/odoo@9b5f86e8fa5bef341ea7392b86987092d6f6933a
Related: odoo/enterprise#23292
Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>

- Up to odoo 10.0, feedparser dependency was optionally used in the cli of vendored html2text.py
  (see https://github.com/odoo/odoo/blob/10.0/addons/mail/models/html2text.py#L437)
- Since 11.0 (67c17cb3), vendored html2text.py has been removed in favor of maintained package
- Turns out the feedparser part in html2text was dead code for a long time anyway
  (see https://github.com/Alir3z4/html2text/issues/220

)
- So we can safely drop this dependency

closes odoo/odoo#76273

X-original-commit: 271b9468
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.0.1 to 8.1.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/8.0.1...8.1.1

)

Multiple security issues in earlier versions, though generally not
exploitable through Odoo.

Backports are made in Debian/Ubuntu, but for pip users we should bump to
the latest "safe" version.

Courtesy of @dependabot

closes odoo/odoo#68169

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

As Fedora 32 was the current release when Odoo 14.0 was released, this
should be the supported version.

Also, a few  old libs were still in mentioned in the packaging files.
They flew under the radar because they never broke the packaging.
This is not the case anymore, those libs disappeared from the Fedora
repos.

It seems that pyparsing is not used anymore since 5a1c06a1 and thus can
be safely removed from `requirements.txt` too.

pychart is not used anymore since 3425752e.

While at it, remove mix of tabs and spaces in package.dffedora, also add
missing packages to avoid installation at test time.

Now that I started down the slippery slope, also removed some `-dev`
packages in package.dfsrc as wheel's are available.

Finally, the rpm install script now detects the python ABI version in
order to avoid update this file at each ABI change in Fedora.

Fixes #63719

closes odoo/odoo#65224

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

When Odoo is started in worker mode with Python 3.8.5, and
gevent/greenlet installed from requirements, the following error message
is thrown:

`RuntimeWarning: greenlet.greenlet size changed, may indicate binary
incompatibility`

As a gevent developper stated [1] that gevent 1.5 is not compatible with
Python 3.8, this commit bumps the version to 20.9.0 (current version for
the next Debian and Ubuntu releases [2] [3])

This commit should not impact those who use the Debian/Ubuntu packages
of gevent and greenlet. The error does not appear with those versions.

In Ubuntu Focal, the packaged version is 1.4.0 [4] but the problem was
not reported with this version and python 3.8.

For reference, it was bumped to 1.5.0 for Python 3.7 in [5].
And greenlet was bumped too for issues with Python 3.8 and 3.9 in [6].

As a result, the requirements for greenlet/gevents gains even more
complexity and should be cleaned when python 3.6 support will be
dropped.

[1] https://github.com/gevent/gevent/issues/1260
[2] https://packages.debian.org/bullseye/python3-gevent
[3] https://packages.ubuntu.com/hirsute/python3-gevent
[4] https://packages.ubuntu.com/focal/python3-gevent


[5] odoo/odoo@bb0b32bd1a3fc0c047d0787b10e8c1c8d696daa5
[6] odoo/odoo/@648635de

Fixes #64106

closes odoo/odoo#65012

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

Only updates outdated requirements which actively cause issues:

* freezegun broken in 3.8 (removal of time.clock)
* xlrd broken in 3.8 (removal of time.clock)
* also monkeypatches xlrd.xlsx for 3.9 (removal of
  Element.getiterator, breaks because of defusedxml)
* jinja triggers DeprecationWarning in 3.8
* pillow triggers warning in 3.9
* lxml, greenlet don't compile in 3.9
* reportlab doesn't work in 3.9

New versions try to match those of Debian Bullseye.

Also adds a script to more easily compare dependency versions between
the requirements files and what's in various distributions (currently
supports checking against debian and ubuntu).

Furthermore updates warnings filtering:

* removes xlrd (mischeck was monkeypatched as noted above)
* removes setuptools (was for older versions, one would hope this
  isn't an issue anymore)
* adds babel: python-babel/babel#684 fixes the deprecation warning but
  is not part of any release yet
* ignores error related to `random.sample` on a set, this is a
  diagnostics bug because recordsets implement both Sequence and Set,
  and the stdlib checks for Set first (bpo-42470)

See #59980
Closes #61103

closes odoo/odoo#62380

Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>

Fixes #62214

closes odoo/odoo#62475

X-original-commit: 45afa3a2
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

Since the usage of gevent 1.3.7 with python 3.6 the CPU usage exploded
on runbot running builds.

Before a better solution is found, I revert to 1.1.2 as before.

closes odoo/odoo#57280

X-original-commit: 9b48876b
Signed-off-by: Xavier Dollé (xdo) <xdo@odoo.com>

The following commit fixes >=py3.7.4 compatibility:
 - https://github.com/gevent/gevent/commit/9d27d269ed01a7e752966caa7a6f85d773780a1a
It was released in stable version gevent==1.5.0 on April 10, 2020:
 - https://pypi.org/project/gevent/1.5.0/

gevent==1.3.4 was released on June 20, 2018
 - https://github.com/gevent/gevent/releases/tag/1.3.4

And python3.7.0 was released June 27, 2018
 - https://www.python.org/downloads/release/python-370/

So, the current pinned version 1.3.4 is not optimized for py3.7

It could be a possible reason to reproduce the following error:
 - https://github.com/odoo/odoo/pull/50861

This change upgrades the pinned version to gevent==1.5.0
in order to get an optimized version for py3.7

Bump version to greenlet==0.4.14 for py3.7 since that it is the
version defined in the sha of release of gevent==1.5.0
 - https://github.com/gevent/gevent/commit/a1a72cb9
 - https://github.com/gevent/gevent/blob/a1a72cb9/setup.py#L188



closes odoo/odoo#57179

X-original-commit: 42391761
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

closes odoo/odoo#54227

Related: odoo/enterprise#11742
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>

It has been a recurrent request from customers to be able to send email
messages to email addresses containing non-ascii characters. [IDNA] is a
domain extension to allow unicode characters in domain names. [SMTPUTF8]
is a SMTP extension to allow unicode in any header.

IDNA defines the [punycode] encoding which translates unicode to an
ascii representation. This encoding MUST be used to encode domains.

SMTPUTF8 is an SMTP extension that allow utf-8 in all headers on the
envelope.

[IDNA] https://tools.ietf.org/html/rfc5890
[SMTPUTF8] https://tools.ietf.org/html/rfc6531
[punycode] https://tools.ietf.org/html/rfc3492



Task: 2116928
opw-2229906
opw-2248251

closes odoo/odoo#47709

Signed-off-by: Raphael Collet (rco) <rco@openerp.com>

allows having DeprecationWarnings pop up on runbot

Before this commit, a lot of leftover import shims existed in the
codebase for py2-py3 compatibility, these are no longer needed since
Odoo 13.0+ doesn't support Python 2 anymore and is (finally) in EOL.

With this commit, these shims are dropped, making the code cleaner,
easier to read and with one less dependency.

Queue -> queue -> py2-py3 compatibility
xmlrpclib -> xmlrpc.client -> py2-py3 compatibility
ConfigParser -> configparser -> py2-py3 compatibility
itertools.izip_longest -> itertools.zip_longest -> py2-py3 compatibility
urllib -> urllib.request -> py2-py3 compatibility
__builtins__ -> builtins -> py2-py3 compatibility
_winreg -> winreg -> py2-py3 compatibility

mock -> unittest.mock -> merged into CPython

The debian/fedora packages and requirements.txt have been updated accordingly

closes odoo/odoo#44601

Related: odoo/enterprise#8141
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>

Bumps [psutil](https://github.com/giampaolo/psutil) from 5.5.1 to 5.6.6.
- [Release notes](https://github.com/giampaolo/psutil/releases)
- [Changelog](https://github.com/giampaolo/psutil/blob/master/HISTORY.rst)
- [Commits](https://github.com/giampaolo/psutil/compare/release-5.5.1...release-5.6.6)

The only API-incompatible change in 5.6 is the removal of memory_maps() on
OSX, which we aren't relying on at this point:
  https://github.com/giampaolo/psutil/blob/master/HISTORY.rst#560



closes odoo/odoo#47632

Original-signed-off-by: dependabot[bot] <support@github.com>
X-original-commit: a6026a4f
Signed-off-by: Olivier Dony (odo) <odo@openerp.com>

Python module vatnumber doesn't seem maintained anymore. Therefore, we should:
    - call directly stdnum (which is maintained and mostly used everywhere in vatnumber)

Also improve stdnum import, vat fix method and vat expected formats

task-1915371

closes odoo/odoo#36978

Signed-off-by: Quentin De Paoli (qdp) <qdp@openerp.com>

As pdfminer does not have a Debian package in Ubuntu Bionic, it cannot
be declared as a strong requirement.

With this commit, a warning is logged if the library is not installed.
It does not prevent to index other types of documents.

closes odoo/odoo#44327

Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

PyPDF performs badly on many types of PDF documents.
We add a text extraction with pdfminer, which is designed for this task.
Because pdf content extraction was so flaky, it was completely
deactivated by 1b753b0d. We revert that :-)

closes odoo/odoo#38508

Task: 2152494
Signed-off-by: Sébastien Theys (seb) <seb@odoo.com>

psycopg2 2.7 not be installed on python 3.8, needs at least psycopg2 2.8
use the same version as windows to avoid complicated rules if windows
AND python 3.8
Note that psycopg2 3.8.4 is currently the only one released after the
release of python 3.8 but reported compatibilty issued seems to be
fixed since 3.8 at psycopg/psycopg2#854

Fixes odoo/odoo#42660

closes odoo/odoo#44143

X-original-commit: f6158264
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>

At 795c7b0a the external dependencies was changed from trying
to import 'ldap' to checking than 'pyldap' package was installed.
The problem is that pyldap is a unmaintained library that should no
longer be used, as explained on the package page:
https://pypi.org/project/pyldap/


"The pyldap fork was merged back into python-ldap, and released as
 python-ldap 3.0.0."

Having pyldap version >= 3.0 installs python-ldap automatically and
will not cause any issue.

The Debian control file package name is adapted to use the latest.

The "ldap" externalm dependency defined in __manifest__.py will cause
pkg_resources.get_distribution() to fail in both case ("python-lap" or
"pyldap"), but the "import" fallback will succeed. For that reason, the
log warning is turned into a log info.

closes odoo/odoo#43769

Note: This library should be replaced by the pure python "ldap3" library.
X-original-commit: 1afd0ccf
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

Some library versions are outdated since the release of Debian Buster.

With this commit the required libraries versions will match as close as
possible the versions available in the current Debian stable release
(Buster).

Also, the requirements were tested against a Windows Python 3.7 to
ensure that a "pip install -r" can be used without the need of a CPP
compiler.

As Babel format_time now returns 'HNE' (Heure Normale de l'EST) for Fr
locale instead of the zone offset, the test is adapted.

Finally the babel.dates is explicitely imported, otherwise the proper
import of this submodule is relying on a side effect.

closes odoo/odoo#43106

X-original-commit: 32e455bf
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

When installing requirements on MS Windows platform with Python 3.8, the
Pillow requirement is defined two times. This leads to a pip crash.

With this commit, the Pillow requirement is only defined once.

Fixes #40080

closes odoo/odoo#40272

X-original-commit: cce9660c
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

- adding gevent for win32 and python >= 3.7

- improving Pillow selection in different python versions

- adding psutil installation for win32

- adding psycopg2 installation for win32

closes odoo/odoo#38489

X-original-commit: 6f887a97
Signed-off-by: Christophe Monniez (moc) <moc@odoo.com>

https://packages.debian.org/buster/python-pil


As of today, python-pillow is at 5.4 in the latest debian release

This allows to deprecate the older versions progessively

Newer versions have new features like the exif_transpose method in 6.0
as discussed at #37448

closes odoo/odoo#38245

X-original-commit: 4ad7a99d
Signed-off-by: Martin Trigaux (mat) <mat@odoo.com>

Repeat #22988 before v13 is released.

Until #35085 is fixed and we can use 0.15.x, at least with this patch people installing Odoo v13 with these requirements will no longer hit #18052.

closes odoo/odoo#36553

Signed-off-by: Olivier Dony (odo) <odo@openerp.com>

Use polib library that handles this correctly
The complexity of the parser is moved to the library

Recommended by GitHub's repository alerts.

We normally stick as close as possible to the version we depend
on in the official DEB packages. This in turn depends on the version of
Debian stable at the time of release - for 10.0 that would be Debian 8
(jessie) and thus Jinja 2.7.3 (albeit with security backports).

However Jinja2 before 2.10.1 suffers from a few issues that could lead
to crashes of Odoo processes.

It seems it's worth an exception to our rule for pip users, similarly to
previous bump up at d2605bcc.

closes odoo/odoo#32602

Signed-off-by: Christophe Simonis <chs@odoo.com>

Recommended by GitHub's repository alerts.

We normally stick as close as possible to the version we depend
on in the official DEB packages. This in turn depends on the version of
Debian stable at the time of release - for 11.0 that would be Debian 9
(stretch) and thus Jinja 2.8 (with security backports).

However Jinja2 before 2.10.1 suffers from a few issues that could lead
to crashes of Odoo processes.

It seems it's worth an exception to our rule for pip users, similarly to
previous bump up at d2605bcc.

closes odoo/odoo#32601

Signed-off-by: Christophe Simonis <chs@odoo.com>

Oversight of e554ce34

closes odoo/odoo#33322

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>

Recommended by GitHub's repository alerts.

We normally stick as close as possible to the version we depend
on in the official DEB packages. This in turn depends on the version of
Debian stable at the time of release - for 11.0 that would be Debian 9
(stretch) and thus Jinja 2.8 (with security backports).

However Jinja2 before 2.10.1 suffers from a few issues that could lead
to crashes of Odoo processes.

It seems it's worth an exception to our rule for pip users, similarly to
previous bump up at d2605bcc.

closes odoo/odoo#32601

Signed-off-by: Christophe Simonis <chs@odoo.com>

- Added new library zeep for soap request.
- Remove old suds library.

Task ID: 41696 Closes #26934

closes odoo/odoo#25982

In light of CVE-2018-18074, using python-requests 2.20.0 is recommended,
or any distribution-specific version that was patched.

See also https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-18074

Commit cf853a78 removed all yml tests
and the yml import engine from Odoo forever, however PyYAML remains a
dependency even though it's not used anymore.

This commit removes any reference to this lib that could be found.

closes odoo/odoo#27563

Continuation from https://github.com/odoo/odoo/commit/27c017ad98b4c9ac5fdcd28c9fd84d3690993118
PyYAML was missed in the list
Closes #26587 

Recommended by GitHub's repository alerts.

We normally stick as close as possible to the version we depend
on in the official DEB packages. This in turn depends on the version of
Debian stable at the time of release - for 9.0 that would be Debian 8
(jessie) and thus Pillow 2.6.1.

However Pillow versions before 3.3.2 and Jinja2 before 2.8.1 suffer
from a few issues that could lead to crashes of Odoo workers.
The bugfixes have been backported in the DEB packages for Pillow,
so users of Debian/Ubuntu LTS versions won't be affected if they are
keeping their systems updated.

However it's worth an exception to our rule for pip users.

Since af9d6b86 and
5a57c29f , python3.7 can be used to run
Odoo. This commit ensures that python3.7 compatibles versions of gevent,
greenlet and lxml will be installed without breaking compatibility with
Debian Stretch packages versions when using the Debian packaged Python.

Closes #25841

Task 40692

Various changes to import/export (mainly) UIs:

* default to excel & "full" (non-import-compatible) export
* auto-detect encoding of CSV using chardet
* remember column -> field mapping after having imported a file (useful
   for repeated imports where auto-matching failed)
* better handle localised booleans & column names
* automatically select source list view's fields when exporting
* better integrate import templates feature and add a number of templates

Include libsass instead of node-less

Thank you to @d-fence for testing this.