PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

As of rfc5322 section 3.4.1 local-part is case-sensitive. However most main
providers do consider the local-part as case insensitive. With the introduction
of smtp-utf8 within odoo, this assumption is certain to fall short for
international emails. We now consider that

  * if local part is ascii: normalize still 'lower' ;
  * else: use as it, SMTP-UF8 is made for non-ascii local parts;

Concerning domain part of the address, as of v14 international domain (IDNA)
are handled fine. The domain is always lowercase, lowering it is fine as it
is probably an error. With the introduction of IDNA, there is an encoding
that allow non-ascii characters to be encoded to ascii ones, using 'idna.encode'.

Also remove usage of 'email_re' in mailing email check. It is too restrictive
compared to real formatting we support (or try to). Valid outgoing emails
were directly canceled, notably when containing unicode.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS: MAIL COMPOSER IN MAILING

When using the composer with a mailing, it currently skips recipients whose
email is a multi-email due to the strict usage of 'email_normalize'.

We can improve multi-email support by effectively checking for the first
email found, using the "less strict" mode of normalize. It means more emails
are detected as valid, and therefore sent.

Due to lower support of multi-emails when sending emails, this even allows
to send multiple emails as all emails are mailed.

SPECIFICATIONS: DEFAULT RECIPIENTS

Mailings are generally done using default recipients, aka using a model method
that returns the people to mail: customers ('partner_id'), customer emails
('email_from'), specific implementation, ...

This is implementation using '_message_get_default_recipients' that returns
'partner_ids', 'email_to' and 'email_cc' that are then used in the mail
composer to generate final recipients.

In this commit we better handle the content of email fields to avoid issues
with multi-emails. For that purpose we correctly split the content of those
fields. We now have several 'email_to' for records having multi-emails instead
of a single badly-formatted 'email_to'.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

Post-message hook is used in various apps to link records to a newly created
created partner. This is the case notably when used with a template as it
creates partner on the fly based on emails to always handle partners.

We now check either the complete 'email', either the normalized version of
it to avoid comparison issues with multi emails and formatted emails. As
'email_normalized' now supports multi-emails by storing the first found one
it helps finding the partner.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

Tool method '_mail_find_partner_from_emails' that searches for partners based
on email is improved to support multi-emails input. Instead of skipping
multi-email (current behavior of 'email_normalize') we now consider first
found email in the input.

It is used notably to match incoming email / partners or suggest recipients
in Chatter.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

When having multi-emails input in an email field, 'email_normalized' field is
currently 'False', as they expect the field to contain a single email. This
has several drawbacks

  * searching partners or fetching information based on emails does not work as
    most tool methods use 'email_normalized' which is False (see e.g.
    '_message_partner_info_from_emails', '_mail_find_partner_from_emails'
    or 'find_or_create');
  * blacklist is not available as it is based on 'email_normalized';
  * mass_mailing wrongly considers those emails as invalid and cancel their
    mail and related trace, as it tries to skip sending emails to invalid
    emails;

Be more defensive and use first found email in case of multi-emails field.
Other emails are ignored. It is already an improvement that does not break
flows in stable and allow more emails to be sent.

  before
  -> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
  -> email_normalized: False
  after
  -> email: '"Raoul" <raoul1@raoul.fr>, raoul2@raoul.fr'
  -> email_normalized: raoul1@raoul.fr

A side effect is that it helps finding back some partners, as indicated in
tests where less phantom partners are created.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

When building the final 'from' of outgoing emails using 'formataddr' we
have issues if email contains multi emails or formatted email. Having a
wrongly formatted email in 'email_from' leads to issues as it is badly
recognized by email providers, could be considered as being phishing and
also breaks reply_to mechanism.

Main fix of this commit is to extract emails and rebuild the 'email_from'
based on found emails. 'from' of sent emails is now the first found email
in 'email_from' field of related <mail.mail> record like

  -> before: email_from: '"Raoul" <raoul@raoul.fr>, raoul2@raoul.fr'
  -> after: email_from: '"Raoul" <raoul@raoul.fr>' and raoul2 is ignored

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

When possible, use 'email_formatted' field on partner, instead of calling
'format_addr'. That way management of corner case input (multi emails and
double encapsulation) is managed by the computed field itself.

When 'format_addr' has to be used, ensure email part is normalized to avoid
formatting issues.

Also use tools to extract emails instead of 'email_re' regex when possible.
That way all code extracting and managing emails go through the same stack.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

When building the final 'email_to' of outgoing emails using 'formataddr' we
have issues if email contains multi emails or formatted email. Main fix of
this commit is to extract emails and rebuild the 'email_to' list based on
found emails.

E.g. partner Raoul - email: "Raoul" <raoul@raoul.fr>
 -> before: to: "Raoul" <"Raoul" <raoul@raoul.fr>> (double format)
 -> after: to: "Raoul" <raoul@raoul.fr>

E.g. partner Raoul - email: raoul1@raoul.fr, raoul2@raoul.fr
 -> before: to: "Raoul" <raoul1@raoul.fr, raoul2@raoul.fr>
    single email with multiple emails, depends on server fault tolerance)
 -> after: to: "Raoul" <raoul1@raoul.fr>, "Raoul" <raoul2@raoul.fr>
    multi emails

Fix that computation by using all normalized emails found in 'email' fields
and rebuilding a formatted email based on name + those emails. We do not
use `email_formatted` as it is not really multi-enabled. We prefer a local
defensive approach to be as tolerant as possible with respect to user inputs.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

SPECIFICATIONS

Main fix in this commit: fix multiple nested formatting in 'email_formatted'
computation for <res.partner>. Other use cases are mainly left untouched as
we let users deal with their input. In summary :

  * double format: if email already holds a formatted email, we should not use
    it to compute email_formatted, like

      name: Name / email: 'Format' <email@domain.com>
      -> before '"Name" <"Name" <email@domain.com>>"
      -> after '"Name" <email@domain.com>''

  * multi emails: sometimes this field is used to hold several addresses
    like email1@domain.com, email2@domain.com. We currently let this value
    globally untouched by extracting emails and joining them, as we do not
    expect email_formatted to be a list of emails. Extractin emails allows
    to filter out extra text stored in email field, like

      name: Name / email: text, email1@domain.com, email2@domain.com
      -> before: "Name" <text, email1@domain.com, email2@domain.com>
      -> after: "Name" <email1@domain.com,email2@domain.com>

  * invalid email: if something is wrong, better keep it in email_formatted
    than harcoding "False". Indeed this eases management and understanding
    of failures at mail.mail, mail.notification and mailing.trace level. This
    behavior does not change as it was already implemented like that even if
    not sure it was intended;

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

This method is still not really tested, except its override of 'mail.thread.cc'.
So better have a test, especially in CRM that has some specific overrides.
Formatted email and multi-email input are tested, to check their current
support. Next commits will try to improve that, notably avoid formatting
issues.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

PURPOSE

Be defensive when dealing with email fields, notably when having multi-emails
or email field containing an already-formatted email.

RATIONALE

Add tests related to not standard usage of email field. Two main use cases
are tested here

  * formatted emails: `"Full Name" <email@domain.com>` stored into the 'email'
    field;
  * multi emails: `email1@domain.com, email2@domain.com` stored into a single
    'email' field;

Additional tests: tests with unicode / ascii / case / wrong formatting are also
added to check the support in normalize and format methods.

IMPLICATION

Email field is generally managed as "containing a valid email". This means
it is sometimes used as it in 'formataddr' as well as to perform searches or
identification checks. Example of issue: partner 'Raoul' has a formatted email
like "Raoul" <raoul@raoul.fr>. Using 'formataddr' in email_from leads to

  from: "Raoul" <"Raoul" <raoul@raoul.fr>>
  -> which is incorrect (but often dynamically corrected by email servers);

Email field holding multi-emails are not normalized, as current normalize
is done only if the field holds a single email. It means

  * no easy finding based on 'email_normalized', e.g. various tools like
    '_mail_find_partner_from_emails' or 'find_or_create' do not find partners
    based on this email;
  * no exclusion list management;
  * issue with formatting, like

  to: "Raoul" <raoul@raoul.fr,raoul.other@raoul.fr>
  -> which is incorrect (but often dynamically corrected by email servers);

USAGE: OUTGOING EMAILS

Those use cases currently generate faulty outgoing emails. This is valid for
recipients ('email_cc', 'email_to') as well as author ('email_from').

For formatted emails: `email_to` is formatted again based on name and email
which leads to sending emails to `"Full Name" <"Other"<email@domain.com>>`.

Note that multi emails without formatting may work as it leads to email_to
`"Full name" <email1@domain.com,email2@domain.com>`. Some outgoing email
servers correctly send multiple emails. It depends on their fault
tolerance.

USAGE: FIND BASED ON EMAIL (NORMALIZED)

When searching for partners (e.g. using '_mail_find_partner_from_emails' or
'find_or_create') normalized version of input is used.

In case of multi emails sanitize is 'False', as normalization expects a single
email in the field. Therefore no partner is found. In processes that do a
"search or create" (e.g. using a template on a record) this leads to creating
a new partner (or several partners in case of multi emails) each time.

USAGE: OTHER FLOWS

Other flows are build on top of '_mail_find_partner_from_emails' / 'create'
of outgoing emails and are impacted by formatted email / multi email usage.
Those include notably

  * mass_mailing: '_message_get_default_recipients' should be defensive to
    give correct values when creating mailing emails;
  * mass_mailing: faulty emails is based on normalize and multi-emails are
    considered as faulty and ignored;
  * after post hook: '_message_post_after_hook' tries to link messages without
    author (but email_from) with newly-created partners, when partners are
    created from chatter. It is therefore impacted by those corner cases;
  * marketing_automation: built on top of mass_mailing and suffers from the
    same issues;

USAGE: UNICODE

Unicode in emails should be supported. 'formataddr' and IrMailServer notably
received fixes to support unicode. Some check performed on email addresses
fail when unicode is involved, which leads to some emails not being sent
while they could.

SPECIFICATIONS

Add tests related to those corner cases. Also add tests for computation of
`email_formatted` field of Partner model. It currently generates wrong email
values for the same corner cases (multi emails, formatted emails).

Tests are also added for the computation of `email_normalized` field used
notably for blacklists. It is not computed currently when being in multi
email mode which prevents from any blacklist mechanism as well as make
email finding harder. `_mail_find_partner_from_emails` tool method is also
tested with multi email as it uses the same heuristic as normalized email
field.

Tests are also added for mass mailing, when having to mail documents that
have a partner with formatted emails / multi-emails, or that have an email
field with formatted emails / multi-emails.

Also restore a test removed at odoo/odoo@afcb7349083c669dbda18b9b1955eabbe14ea675 while it should have been
updated to state that email addresses containing non-ascii characters are
supported.

Add some tests for tools methods used in various email processing flows.
Unicode tests are also added.

In future commits we will try to make email usage a bit more defensive to
try to lessen issues with that kind of use cases.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

Add 'id' in partner ordering, to be sure searching on duplicated partners is
deterministic.

In mail, use standard ordering when searching on users to be deterministic.
As ordering on users is based on name then login which is unique, this is a
safe ordering and can be used as it.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

MailTemplate model has a 'partner_to' field is dynamically rendered to contain
partners being recipients. After rendering it should hold a comma-separated
list of partner IDs.

However as we are unsure it was correctly written better be defensive. We now
check that we can effectively transform items to IDs using 'isdigit()'.

Task-2612945 (Mail: Defensive email formatting)

Part-of: odoo/odoo#74474

If leads are created using live chat, users without CRM rights cannot
access the visitors list anymore because the lead/opportunity
information is not available to them.

This commit limits the display of `lead_count` to the
`sales_team.group_sale_salesman` group.

In 14.0, the column still appeared without the `lead_count` value
displayed, but clicking on it raised a traceback.

Steps to reproduce:
- Install `website_crm_livechat`
- Login as Mitchell Admin
- Send a message in the live chat
- Go to Discuss
- Answer the livechat message with `/lead New`
- Go to Settings / Users / Marc Demo
- Remove the Sales rights
- Logout
- Login as Marc Demo
- Go to the Website / Reporting / Visitors page

=> The page could not be reached and an access right error message was
generated.

opw-3475301

closes odoo/odoo#133581

Signed-off-by: Thibault Delavallee (tde) <tde@openerp.com>

Apparently `lxml.html.document_fromstring` (and possibly other
`lxml.html` loaders) parses byte-strings as latin1 regardless of their
actual encoding, maybe because python2, maybe because there's a super
legacy html4 parser underlying it.

Either way that means ever since loading
`static/description/index.html` files was added 10 years
ago (4bf6a7ea) `_get_desc` has been
loading these files in latin1 rather than the utf8 most people would
expect.

Add an explicit decoding phase to try and load html description files
in UTF8. Fall back to latin1 in case there are description files which
are genuinely in latin1, or even just some random-ass broken stuff
which very much isn't utf8 (the extended-ascii encodings -- of which
latin1 is one -- will happily accept and mangle any input as every
byte value is valid, utf8 is a lot more structured).

Closes #127846

closes odoo/odoo#133731

X-original-commit: 4dbc3b00
Signed-off-by: Xavier Morel (xmo) <xmo@odoo.com>

In Saudi Arabia, the InvoiceLine/ID should not be greater than 6 digits.
Using the move.line_id, this limit can be exceeded.

Simply count the invoice line ids starting from 1 instead.

In master, add a parameter `line_id` to `_get_invoice_line_vals`.

closes odoo/odoo#133590

Signed-off-by: Josse Colpaert <jco@odoo.com>

Bug:
When printing DDT documents for a delivery with a pricelist applied, the
total value shown comes from the product's original price and not the
modified pricelist price.

Setup:
- install `sale_management` and `l10n_it_stock_ddt`
- have a product P with a price A
- create a pricelist that where P has a price B

Steps to reproduce:
- activate DDT report printing
- create a quotation set the pricelist you created and the product P
- validate the quotation
- go to the associated delivery and validate it
- print the DDT report

You should see that the price mentioned on the DDT report does not
account for the pricelist. (price A is shown, instead of B)

opw-3171295

closes odoo/odoo#121424

Signed-off-by: Josse Colpaert <jco@odoo.com>

For some reason, “New” has been translated as “Nieuw” in Bulgarian, but
this is not Bulgarian, this is Dutch.

This commit replaces all occurrences of “Nieuw” with “Нов”, which is the
correct translation for “New” in Bulgarian.

closes odoo/odoo#133582

Related: odoo/enterprise#46542
Signed-off-by: Louis Wicket (wil) <wil@odoo.com>

The test `test_ddt_flow` was failing because the picking types were created
without a ddt sequence. This is because the picking types were created
at the moment the company was created. However, the country of the company
was set after the setup of the CoA. This means that when we were creating
the picking types, the company's country was not Italy, therefore no
DDT sequence was created.

This commit fixes this by overriding the `setup_company_data` method to
add the country while creating the company and its CoA.

Fixes runbot error 24396 & 24397

closes odoo/odoo#133434

Signed-off-by: Josse Colpaert <jco@odoo.com>

Don't do it only at install.

closes odoo/odoo#132062

Signed-off-by: Nicolas Seinlet (nse) <nse@odoo.com>

A check was added to prevent importing records with prefixes of existing
modules: https://github.com/odoo/odoo/pull/130825

 . This queries the
known modules, but non-admin users don't have access to that by default,
causing the import to fail for them. Allow the module query regardless
of access rights.

closes odoo/odoo#133300

Signed-off-by: Raphael Collet <rco@odoo.com>

After this commit, we prevent the slideshow from appearing when clicking
on an "Images Wall" snippet image that has been set as a link.

Steps to reproduce the bug:

- Drag and drop an "Images Wall" snippet onto the page.
- Add some images in the snippet.
- Click on an image within the snippet.
- In the editor text toolbar, click on the "Link" button.
- Set the URL of the link to "/contactus".
- Click on the "Save" button in the modal.
- Save the page.
- Click on the image with the link.
- Bug: The slideshow briefly appears before redirecting to the "contact
us" page.

task-3425624

closes odoo/odoo#128885

Signed-off-by: Colin Louis (loco) <loco@odoo.com>

The condition to create is wrongly evaluated and does not take into account
the country code and the picking type code. This commit fixes this by
reordering the conditions.

closes odoo/odoo#133146

Signed-off-by: Josse Colpaert <jco@odoo.com>

When a model selection record is going to be deleted, a _process_ondelete method is called in order to delete all the records of the corresponding model that have that selection. These records are obtained by calling _get_records, which uses a query that needs a table. Thus, we should avoid cases for non-abstract models that have _auto = False.

closes odoo/odoo#132751

Signed-off-by: Raphael Collet <rco@odoo.com>

It's possible to specify an existing module as the xml id prefix when
importing records from a file. This will result in `ir_module_data`
records being created with `noupdate` set to False. When the module is
upgraded, these records will be deleted.

This can lead to undesired side effects like journal items for
accounts imported this way being deleted.

This change prevents creating new records linked to existing modules
when importing from a file. Instead, the user should either use no
prefix or the name of a non-existent module, like `__import__`.

opw-3231987

closes odoo/odoo#130825

Signed-off-by: Merel Geens <mege@odoo.com>

closes odoo/odoo#68375

Signed-off-by: Quentin De Paoli <qdp@odoo.com>

Part-of: odoo/odoo#68375

This reverts commit 95468505.

The commits break the preview, as docs is not defined in preview (while done in real printing).
Not sure what we will do, with a check on the presence of the field or not, as it was working fine before, only a wrong modification of a view made it visible.
Linked to runbot error 24276

closes odoo/odoo#132639

Signed-off-by: Florian Gilbert (flg) <flg@odoo.com>

This PR odoo/odoo#125747 missed the account 973, which is still not in Ukrainian.
This account should be translated as well.

closes odoo/odoo#132866

Signed-off-by: Josse Colpaert <jco@odoo.com>

The overriding of the placeholder does not work.
It's done to be different than the standard translation (Ciudad).
Modified it to be standard way of overriding placeholders.
Linked to runbot error 6175

closes odoo/odoo#132833

Signed-off-by: Habib Ayob (ayh) <ayh@odoo.com>

steps to reproduce it:
1. Create a storable product, add a Purchase vendor with
 start date = 20/6/2022 and delay = 56
2. Create another storable product, add the same Purchase vendor with
 start date = 11/01/2023 and delay = 21
3. For both products, add Routes "MTO".
4. Create a SO with those 2 products, and confirm it
5. As you can see, the price of the first product is correctly set
but not the second product

BUG:
This is due of "Order Deadline" (date_order) being set to an old date,
which is before the second product date_order.

FIX:
set max PO date as today

opw-3167094

closes odoo/odoo#124448

Related: odoo/enterprise#44391
Signed-off-by: William Henrotin (whe) <whe@odoo.com>

There was inconsistency in the translation for the words "price unit", with this
 commit all the "price unit" are translated the same.

closes odoo/odoo#121413

Task-id: 3262408
Signed-off-by: Brice Bartoletti (bib) <bib@odoo.com>

This PR solve two things:
- Preview: due to the hardcoded width the preview didn't take all the page and
was push on the left.
- PDF: Weird stuff happened with the header, he was hiding information below
 the header.

Task-id: 3262408
Part-of: odoo/odoo#121413

Before this PR, when there is no shipping address, the pdf show an empty
"Shipping Address" header. By changing the colspan dynamically we can manage to
keep the layout like it was and remove the useless section.

Task-id: 3262408
Part-of: odoo/odoo#121413

When the user tries to import the invoice file with space on both sides
of the date value the error will be generated.

Steps to reproduce:
1. Install the `account_edi_ubl_cii` module.
2. Generate one `factur-x.xml` file or import this file
https://drive.google.com/file/d/1RCqLT37j2g6LPkauf75LpOzZt64XHd_L/view?usp=drive_link
3. Go to the invoice app and click the `upload` button.
4. Select the `factur-x.xml` file and an error will be generated at the backend
and nothing to be imported.

This commit fixes the issue by removing the extra spaces in the date value,
if user gives the spaces in the invoice file.

see-
https://github.com/odoo/odoo/blob/a0ea6302a574624604cad7d0903711b30a4802a6/addons/account_edi_ubl_cii/models/account_edi_xml_cii_facturx.py#L315



sentry-4331059603

closes odoo/odoo#129374

Signed-off-by: Laurent Smet (las) <las@odoo.com>

1) The aim of this commit is to make the tax template more consistent with
the instanciated tax.
Indeed a tax wouldn't pass the constrains if it hadn't all its
repartition line.

2) Make the refund consistent with the rest of the taxes.

closes odoo/odoo#130706

Task-id: None
Signed-off-by: Florian Gilbert (flg) <flg@odoo.com>

When the user configures the 'external_layout_din5008' layout template in the
general settings and tries to any QWeb reports, that case generates the
traceback.

Steps to reproduce:
- Install the 'l10n_de' and 'sale_timesheet' modules.
- Settings > General Settings
- Search for 'Document Layout' and configure the 'external_layout_din5008'
  layout.
- Settings > Technical > User Interface > Views
- Search the 'external_layout_din5008'  QWeb template view.
- Search  '<span t-if="not o and not docs">
<t t-esc="company.l10n_de_document_title"/></span>'
  line and replace by,
'<span t-if="company"><t t-esc="company.l10n_de_document_title"/></span>'
- Go to the Sales menu and print any reports from the print menu.
After that, a traceback was generated.
Error:
AttributeError: 'res.company' object has no attribute 'l10n_de_document_title'
Template: l10n_de.external_layout_din5008

The 'l10n_de_document_title' field does not exist in the 'res.company'
object and also this field used in 'l10n_de.external_layout_din5008'
template, and this template is configured as layout.  So, remove this line from
this template.
Code reference:
https://github.com/odoo/odoo/blob/14.0/addons/l10n_de/report/din5008_report.xml#L109



Sentry-4283510443

closes odoo/odoo#128559

Signed-off-by: Florian Gilbert (flg) <flg@odoo.com>

Turkey has adapted its standard VAT from 18% up to 20%
This commit sets the 18% VAT taxes and its tax group to inactive and
introduces the respective 20% taxes and a 20% tax group.

closes odoo/odoo#132340

Task-id: None
Signed-off-by: Brice Bartoletti (bib) <bib@odoo.com>
Co-authored-by: Ramazan Subaşı <subasiramazan@gmail.com>