Mail junked by Outlook

kjpetrie · ‎19-12-2010

For the last six weeks I've been trying to send an important Application form to British Gas. Whatever I do they don't receive it. Today I had a long conversation with one of their staff during which he sent me an e-mail so I could attach my files one at a time to a reply (thus ensuring I was sending to the right address). He received none of my e-mails. I told him I'd try sending it directly from my own mail server rather than through PN, So I reconfigured my Postfix to send mail direct rather than relaying through PN. It still didn't get through, but at least I had a DSN (2.6.0) and knew where it had been sent.

BG apparently use Outlook.com to receive their incoming mail, so I have now pointed out to them Outlook receives my mail but doesn't forward it to the right department. Their complaints dept now has my log entry to forward to their technical team so they can raise the problem with Microsoft. I then reset postfix to relay through PN, on the assumption receiving servers are more likely to talk to an ISP's server than a consumer's one.

As it happens, I also have an Outlook account which I set up during a recent house move to ensure I could still get incoming mail while PN was disconnected, so I decided to resend my form to that to see whether I got it. It finished up in Junk.

Here are the headers:

Received-SPF: TempError (protection.outlook.com: error in processing during
 lookup of redacted: DNS Timeout)
Date: Tue, 28 Nov 2023 16:55:06 +0000
From: "K.J. Petrie" <redacted>
To: Ken Petrie <redacted@outlook.com>
Subject: SEG Application
Message-ID: <20231128165506.3ca6f777@redacted>
Content-Type: multipart/mixed; boundary="MP_/BRle1nuPW=AuQY5.D78xFKd"
X-MS-Exchange-Organization-ExpirationStartTime: 28 Nov 2023 16:55:13.7020
 (UTC)
X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit
X-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000
X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit
X-MS-Exchange-Organization-Network-Message-Id:
 dcb71d21-a431-432e-8def-08dbf032ca5a
X-EOPAttributedMessage: 0
X-EOPTenantAttributedMessage: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa:0
X-MS-Exchange-Organization-MessageDirectionality: Incoming
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic:
 HE1EUR01FT023:EE_|CWXP123MB3384:EE_|CWXP123MB6269:EE_
X-MS-Exchange-Organization-AuthSource:
 HE1EUR01FT023.eop-EUR01.prod.protection.outlook.com
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-UserLastLogonTime: 11/28/2023 3:27:08 PM
X-MS-Office365-Filtering-Correlation-Id: dcb71d21-a431-432e-8def-08dbf032ca5a
X-MS-Exchange-EOPDirect: true
X-Sender-IP: 84.93.230.250
X-SID-PRA: redacted
X-SID-Result: NONE
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-SCL: 5
X-Microsoft-Antispam: BCL:0;
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2023 16:55:13.5457
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: dcb71d21-a431-432e-8def-08dbf032ca5a
X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-CrossTenant-AuthSource:
 HE1EUR01FT023.eop-EUR01.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg:
 00000000-0000-0000-0000-000000000000
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CWXP123MB3384
X-MS-Exchange-Transport-EndToEndLatency: 00:00:10.3994666
X-MS-Exchange-Processed-By-BccFoldering: 15.20.7025.020
X-Microsoft-Antispam-Mailbox-Delivery:
	ucf:0;jmr:0;ex:0;psp:0;auth:0;dest:J;OFR:SpamFilterAuthJ;ENG:(5062000305)(920221119095)(90000117)(920221120095)(90013020)(91025020)(91040095)(9050020)(9075021)(9100341)(944500132)(2008001134)(4810010)(4910033)(9610028)(9560006)(10180021)(9320005)(9245025)(120001);RF:JunkEmail;
X-Message-Info:
	6hMotsjLow8h2u6w9D6EhSXbH5e2qrRuktKGzoMpZ/DPZHdEDNAKuuBxHyl1eaUW5NR2g5clva9SnC2dYaIipZDq30vv5aSp37sc+qHJnu4g6Z4X9vtCjDBqcSk6s2Y7pdt8nWkV88zQu1dci28ywAI+etWrdZakMj/v6joM/OH3+mk+eVk9NeDKNju0sSMQU4h3SgjFXfXf8uRKDLzmyw==
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0xO0Q9MjtHRD0xO1NDTD02
X-Microsoft-Antispam-Message-Info:
	=?us-ascii?Q?J1lxFxN+Aa4su+lPqwKk2ZlR9BItMH0AWJRbQJIRFE1IU86O807PTUuZEgJ6?=
 =?us-ascii?Q?sUACE0bPab5/eNXJggUUiZwevSY+v2o/p1XwJhIhpaR33nfhYDpNyr1FgfhW?=
 =?us-ascii?Q?r14dA6aK3zCYtYUSFCCNvOG5Xs7Ntto4aa1JCIp1h+ZfKdRnMTbVF7NtB09g?=
 =?us-ascii?Q?JQNUuxFfb7J7LhCnEkItTQpEZWOsZrQrJSG+udZaibakzjuG/+p5tmQNmSP7?=
 =?us-ascii?Q?AQvMl+YSXDEVNW1UHcLiUYx94w46LDaP4a3r17QYu1r22xWFjvf3/cBnS9ho?=
 =?us-ascii?Q?L9ShV3PRzj0bdsqPr9XkCMaYdtP2UuseUgYtMvi2oGWftRq/YrxaJmUOwX1e?=
 =?us-ascii?Q?iMDi3EcfABxsdVK97oHvVIPXa6s4bhWFzlePI4iQyrKyK+3glKYzYlJZLEKE?=
 =?us-ascii?Q?4GcwAGoBkfz6StkkVnArlpey5+dEksRVC6YdDKYbRqJ9Ll4D2hFdeeuRalM6?=
 =?us-ascii?Q?58CtDsfI+xFySfCvp3TG4F9OA9Nw/My+c3ZiUol8iSEFFMLx+e99uMuKCQw8?=
 =?us-ascii?Q?+AlEuNwFQ4008EwgJSQ2GFW+2TV9d2oSF250iW+MbYucQ/0GP9PM/Fv4lIaK?=
 =?us-ascii?Q?cMwWpBRZVTYAOyK8lISiOD0wjZo0c6H6aUnG4TjO+JmGfkt3m1mKCVtSTYDC?=
 =?us-ascii?Q?Cs66MxLfY/X42I083Oc3RS+9BvWcwXTXbsNGrqCQGI+D8BLmL00RfV2R7w6f?=
 =?us-ascii?Q?6YrcXxMEcVuMCFKsW+F+82OBYGPuNar1rwOOS56hA7cM9LwRpVFI8zyvQIay?=
 =?us-ascii?Q?yeIdShPTAp5buXZVOCKvEg5dWqKsOP5id6O2DcEeoPKDWvqGT0Odf3uCJeKS?=
 =?us-ascii?Q?ixA4eH6qD/MsByaqbsR698qwQ5aOFvvMgSio0tCQJWH+1RAwzeSM00BHKpBQ?=
 =?us-ascii?Q?Z68o+oMeyzVZ6lDaS4Ep/6mdHKZ/zb2ODFdhvIZ94M1Id336Fe2BLfnMqUP8?=
 =?us-ascii?Q?Mo2jozRxpN/0UJLcup9u73kkqEEcOjReLBYk+X/Yt4TKTIeRCeW75pf2CwsG?=
 =?us-ascii?Q?CNtG5TUAAVZ6WfsMuqGPQqqmW6Ii4gdpuclQBPCdinjRKVedzN3IVvjVc26c?=
 =?us-ascii?Q?rRy9Zvf7C++k/ewUbY4sYxBYYlUR4loOoSD8/8pffenUqcDbBpZNv1+KX09t?=
 =?us-ascii?Q?/jf3gX35O24uqgMOw/O0uNNZapa2VkwXbCXfrAYdX5dlVlVMGtmWnYUHnpqd?=
 =?us-ascii?Q?jSFa5swkPxTUeDnM3LHzRdyzikA1fGaypOq1jHelqF3tt62EQFfT/M44q08q?=
 =?us-ascii?Q?871pVFdZKQBdlqrkmiitXOhmLQuILraGlyfMRIHoVQ5D1R7yOt2AcHiaJEXU?=
 =?us-ascii?Q?i/AzoFAp9EBNT93jEi+d7I0sbGVYKYnoc6bo9m+rZV2EWJsk3w5/MDT8XSS5?=
 =?us-ascii?Q?KqE0G8hDb9qw0NoObhCli/rS8ZH24W+WAjIAZfMy9H6BLlIfrgUA21+J90CY?=
 =?us-ascii?Q?lmSGRjZr+r01kwzbkyroPNrOXuZiVHeuk0CR9jBWTN/iuwquPME14/oDZe/o?=
 =?us-ascii?Q?UiIO1VZfy7+/W+ts/bTeDAJoQn6Eg0AUMDF7VwnMvBtb2oBFF/cqlB5WUr3c?=
 =?us-ascii?Q?dr/YQTrXpJhl5zYkySo3PiBfi3rKkC/SAI5ndlHy8+n5BVKueYmlYTJB+zz4?=
 =?us-ascii?Q?GfX8zdjLcKLuxZj2wgm8VUsJhJKTXElaacgJ8sEWInhDOcpKn6PraZWyvzfS?=
 =?us-ascii?Q?SaLus7VfgE5+XXISRb87D8h5L7zdW6OBoRHRjIZIL4wQJU3ZzX7Cm4dSKbki?=
 =?us-ascii?Q?XGhWYzbgiRjZOp8WPV6T5iMCPMJJJtieZbpZXVt6HrRyQ5AAxScWvgheMkZ8?=
 =?us-ascii?Q?EbGDh6xnbUFxrhOwak3D86mfS2IiCygEJ1iYtcL2tqGprsYBSC+BCtoLz1FH?=
 =?us-ascii?Q?H8zJnFU3ZAS0IroPM+VVlbXwNceoP6mVSSUJ4OIatOfqXX/3kTLNE2fGLiDc?=
 =?us-ascii?Q?WruzLNmKpc6PP/nG0wJhRg=3D=3D?=
MIME-Version: 1.0

So it seems MS treat legitimate mail coming from PN as spam. There was nothing in my message or the all-important attachments which would look like spam and I had tried sending files individually and none get through.

Can you do anything to persuade these large providers they need to let their customers see the messages sent through your system, especially when they advertise e-mail as the preferred method to contact them?

MisterW · ‎30-07-2007

Looks like you are probably sending from a domain email address, and you have no SPF record ( in fact a DNS lookup is failing for your domain ) or DKIM. Hence outlook.com is thus treating it as spam.

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

kjpetrie · ‎19-12-2010

I have an SPF record. They appear to have a DNS lookup problem, or perhaps too short a time out.

MisterW · ‎30-07-2007

Seems its not an uncommon problem https://techcommunity.microsoft.com/t5/outlook/received-spf-temperror-protection-outlook-com-error-i...

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

kjpetrie · ‎19-12-2010

I have now abandoned using PN's relay to send mail and set up my own servers with DKIM and DMARC verification in addition to the SPF I used previously.

That works, though I have to use a separate server for each of my three domains to pass DMARC. Aren't virtual machines wonderful things!

I had previously used the PN relay on the basis an ISP's server would be more reputable than a private one on a customer's line. It seems that has now changed with the requirement for evermore rigid verification schemes.

Sad that things have got so complicated. There must be hundreds of less technically aware people whose attempts to contact utilities and other online businesses are silently blocked by this sort of thing.

Townman · ‎22-08-2007

Hi Ken,

@kjpetrie wrote:

I had previously used the PN relay on the basis an ISP's server would be more reputable than a private one on a customer's line. It seems that has now changed with the requirement for evermore rigid verification schemes.

Is that really the right conclusion here? From your headers (which alludes to the linked to article) ...

Received-SPF: TempError (protection.outlook.com: error in processing during
 lookup of redacted: DNS Timeout)

Is this really anything to do with an ISP's server's reputation or more probably just failures within Microsoft's infrastructure?

DNS timeouts are nothing to do with the ISP's server's reputation, but everything to do with the receiving service's infrastructure and configuration. There is a possibility that MS's DNS services were (are ?) subject to DOS attacks which can render unpredictable DNS performance.

The linked to article references a large number of .au TLDs being blighted by this issue - I trust that you are not suggesting that all those services in .au land are less "reputable" that a private one?

One of the posts on there suggested that different outcomes were being experienced depending on if the sending server was using IPv6 or IPv4, the latter being more "reliable". Running your own service on a Plusnet line assures that you will be seen as IPv4 ... could that contribute to the better experience?

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

kjpetrie · ‎19-12-2010

The SPF lookup timeout proved to be a red herring. The real problem as I have established by experimentation sending mail to my own Outlook account, is that Outlook seems to treat any message without valid DKIM as spam and dumps it in the recipient's Junk folder. Since I was sending from my own domain through a third party's (PN's) server I could not control DKIM and therefore could not pass DKIM. The fact PN's server as a known provider should be more reputable than one on a customer's IP (as most spambots would be) has insufficient weight to overcome that action and the only way I could set up DKIM for my own domains was to use my own servers.

In fact, in order to achieve a DMARC pass for all three domains I have had to set up three separate instances of Postfix to get the Envelope From to match the Header From because Postfix seems to set the former at start-up rather than submission. That's arguably correct if we think of the Envelope From as equivalent to a Postmark and the Header From as the sender's letterhead.

My mail now passes both DKIM and DMARC in addition to SPF and goes to the user's Inbox rather than Junk so companies like BG and Ofgem are getting my mail again.

That's what matters to me as the sender.

Townman · ‎22-08-2007

One of DKIM or SPF should be sufficient.

@account.plus.com has DKIM
@yourowndomain.co.uk hosted on Plusnet (NS with Plusnet) can have SPF - there is a means of injecting the TXT record documented around this forum
@yourowndomain.co.uk using a DNS elsewhere but sending via Plusnet's SMTP server can have a SPF record set up as documented around this forum

Out of your experience, are you of the opinion that Outlook has now become totally draconian requiring BOTH DKIM and SPF?

I just ran a test sending from my hosted domain to my Outlook mailbox and encountered no issues - SPF=pass, DKIM=none (fail).

Authentication-Results: spf=pass (sender IP is 84.93.230.244)
 smtp.mailfrom=<redacted>.me.uk; dkim=none (message not signed)
 header.d=none;dmarc=bestguesspass action=none
 header.from=<redacted>.me.uk;compauth=pass reason=109
Received-SPF: Pass (protection.outlook.com: domain of <redacted>.me.uk
 designates 84.93.230.244 as permitted sender)
 receiver=protection.outlook.com; client-ip=84.93.230.244;
 helo=avasout-ptp-003.plus.net; pr=C

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

kjpetrie · ‎19-12-2010

Without DKIM I would get:

Authentication-Results: spf=pass (sender IP is 84.92.47.176)
 smtp.mailfrom=<redacted>.org; dkim=none (message not signed)
 header.d=none;dmarc=permerror action=none
 header.from=<redacted>.co.uk;compauth=fail reason=001
Received-SPF: Pass (protection.outlook.com: domain of <redacted>.org designates
 84.92.47.176 as permitted sender) receiver=protection.outlook.com;
 client-ip=84.92.47.176; helo=<redacted>.co.uk; pr=C

.... many irrelevant lines...

X-Microsoft-Antispam-Mailbox-Delivery:
    ucf:0;jmr:0;ex:0;psp:0;auth:0;dest:J;OFR:SpamFilterAuthJ;ENG:(5062000305)(920221119095)(90000117)(920221120095)(90005022)(91005020)(91035115)(9050020)(9100341)(944500132)(2008001134)(2008121020)(4810010)(4910033)(9610028)(9560006)(10180021)(9320005)(9245025)(120001);RF:JunkEmail;

The body was:

"Hallo,

Just testing, so sending this message of a sensible length as a test to
see whether all goes well. I do not expect it to reach my inbox but we
shall see."

So SPF wasn't enough in that case.

Townman · ‎22-08-2007

Does the following difference offer any insight?

You...

 header.d=none;dmarc=permerror action=none
 header.from=<redacted>.co.uk;compauth=fail reason=001

Me...

 header.d=none;dmarc=bestguesspass action=none
 header.from=<redacted>.me.uk;compauth=pass reason=109

TBH I do not know where DMARC enters into the equation, but I get "bestguesspass" whereas you get "permerror". Also I get compauth pass, whereas you get a fail.

Is your connection to the Plusnet SMTP server authenticated? It is not essential if you are connected to the network, but if you are, it adds weight to being a proven user.

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

kjpetrie · ‎19-12-2010

I have no idea why the dmarc was different. I had no DMARC record at the time that was sent; I didn't even have a DKIM record.

I suspect there's an element of machine learning involved - the more it's used to accepting mail from a particular source the more it is likely to do so, and vice versa, but who knows? I did run another test a few minutes ago sending through the PN server with exactly the same configuration and this time it was accepted, so I think the fact I've set things up to pass more tests and it's accepted lots of my mail as legitimate has got it into the habit of accepting what it previously was in the habit of dumping into Junk, even when those credentials are not provided.

Perhaps it's a bit like a credit rating - if you've never had a loan it's hard to get one because you have no record.

In fact, MS do advise their users to mark wanted mail as "not spam" to teach the system, so that does suggest it learns what to do. Of course, that assumes users check their Junk folders, which businesses redistributing the Inbox to various departments won't do.

what's really going wrong here, I think, is that tools developed for the consumer/personal market are now being adopted by business to save having their own IT people setting up servers, and they're not designed for business needs.

MisterW · ‎30-07-2007

This looks like it might give some clue https://support.clickdimensions.com/hc/en-us/articles/360042239312-Implicit-Authentication-for-Micro...

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

Townman · ‎22-08-2007

Interesting! That leads to soup to nuts here - Anti-spam message headers | Microsoft Learn

Reason Codes

001: The message failed implicit authentication (compauth=fail). This result means that the sending domain didn't have email authentication records published, or if they did, they had a weaker failure policy (SPF ~all or ?all, or a DMARC policy of p=none).
1xx or 7xx: The message passed authentication (compauth=pass). The last two digits are internal codes used by Microsoft 365.

It is not clear to me how I get Implicit Authentication, but @kjpetrie does not. The following is my SPF configuration:

"v=spf1 a mx include:_spf-internal.plus.net include:_spf-internal2.plus.net ~all"

As I read the above, that SPF should get me a 001 [as it is "a weaker failure policy (SPF ~all ..."] not a 109 pass, due to soft fail and no DMARC record (p=none).

However, this seems murky to me...

bestguesspass: Indicates that no DMARC TXT record exists for the domain exists. If the domain had a DMARC TXT record, the DMARC check for the message would have passed.

I wonder what criteria beings about the assessment that something which does not exist, if it did exist would have created a good outcome. Is this the email / spam management implementation of Schrödinger's cat syndrome (the DMARC record is not there, but be deemed valid if it were regardless of its absence)?

Black arts space this, but I am not complaining that it works!

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

kjpetrie · ‎19-12-2010

It looks as if MS have quietly merged Outlook.com with Office 365, thus extending the policy to all mail they receive, but there does seem to be a bit of heuristics involved as well, just to fuzz the edges a bit. I do now have a DMARC policy, but it's currently none, though it looks as if I need to up it to reject if things are to keep working in the future.

Good find, MisterW. That explains something of what's going on and confirms that in general DKIM verification is needed, but DKIM can only be set up for a domain if the domain owner also controls the MTA and can configure it to sign mail with the domain's key, so private people with their own domain can't use it if they use a third party server, and DMARC requires Envelope and Header Froms to match, so can't be used with a third party server at all.

The key differences between personal and corporate recipients is people generally have a good idea who their friends are and usually don't want mail from people they don't know, whereas companies want the general public to contact them because they're prospective customers. They don't want people blocked because they're unknown because that's blocking initial contact. They would want other techniques to reduce spam, like content analysis or origins on blocklists. So tools designed for one won't work for the other.

I wonder how long it will be before they realise their 'cheap' solution to e-mail is actually losing them business?

As for establishing whether DMARC would pass, that's easy: DMARC passes if SPF and DKIM both pass and Header and Envelope Senders match. Since Outlook already has that information it can calculate the result and set the policy to its default without needing the DMARC record. The DMARC record exists to suggest other policies for failures, which the recipient is free to use or ignore.

Townman · ‎22-08-2007

"As for establishing whether DMARC would pass, that's easy: DMARC passes if SPF and DKIM both pass and Header and Envelope Senders match. Since Outlook already has that information it can calculate the result and set the policy to its default without needing the DMARC record. The DMARC record exists to suggest other policies for failures, which the recipient is free to use or ignore."

That then makes me wonder how you@account.plus.com or me@mydomain.co.uk can every get stuff into Outlook from relay.plus.net without a DMARC record:

you@youraccount.plus.com has DKIM but not SPF so both cannot pass
me@mydomain.co.uk as hosted on Plusnet has (can have) SPF but not DKIM so similarly cannot pass both

Therefore the presumption stated above is not capable of being fulfilled to reach a best guess pass - the best that might be achieved is SPF or DKIM pass with header and Envelope Senders matching. The later part of that leads me to think about testing "send on behalf of" aka an alternate identity!

The option of the receiver to ignore DMARC then is very similar to SPF which if even checked by the receiver might still be ignored.

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.