Hello,
just fyi: I was unsubscribed from the list since 2921-95-26 (where I received a last posting), without notice.
Over the past weeks I thought, it's so quiet - nice, everyone enjoys vacation ;)
Anyone else had this issue?
vy 73, - Thomas dl9sau
Hi Thomas,
I can see that the mailing list software put you on moderation due to excessive bounces, that’s the reason.
Kind Regards, Chris - G1FEF
On 29 Jul 2021, at 20:44, Thomas Osterried via 44Net 44net@mailman.ampr.org wrote:
Hello,
just fyi: I was unsubscribed from the list since 2921-95-26 (where I received a last posting), without notice.
Over the past weeks I thought, it's so quiet - nice, everyone enjoys vacation ;)
Anyone else had this issue?
vy 73,
- Thomas dl9sau
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
Hello Chris,
Am 29.07.2021 um 22:13 schrieb G1FEF via 44Net 44net@mailman.ampr.org:
Hi Thomas,
I can see that the mailing list software put you on moderation due to excessive bounces, that’s the reason.
thank you for your info.
But I havn't any e-mail problem. Unfortunately, two months are a too long time for looking at logs. Do you have such a bounce mail?
vy 73, - Thomas dl9sau
Kind Regards, Chris - G1FEF
On 29 Jul 2021, at 20:44, Thomas Osterried via 44Net 44net@mailman.ampr.org wrote:
Hello,
just fyi: I was unsubscribed from the list since 2921-95-26 (where I received a last posting), without notice.
Over the past weeks I thought, it's so quiet - nice, everyone enjoys vacation ;)
Anyone else had this issue?
vy 73,
- Thomas dl9sau
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
I think one reason for bounces could be that the mailman mailing list does not handle DKIM properly. When a user sends a mail to the list, it passes through the DKIM signature, but it also modifies the mail (adding a standard footer as the mimimum, probably more) As a result, the DKIM signature of messages received via the list is invalid, and maybe some mail servers see it as a reason to reject the mail (as if it is spoofed). In my mail system, most mails in this list get a red color because that is what some rule processing here does to get my attention to falsified mails.
In fact this should be solved by mailman, but some googling seems to indicate that the mailman people know that, but have not yet come into action. So likely there is no setting in the admin panel to easily solve this. Maybe some extra software or feature of the existing software could be used to work around this. That would at the minimum have to delete any DKIM-Signature: header on incoming mail (unfortunately this is a multiline header so it will require more than passing through some simple grep or sed), and additionally it could add a new DKIM signature afer all mailman processing, for the mailman.ampr.org domain. But the latter is optional.
Rob
On 7/29/21 10:27 PM, Thomas Osterried via 44Net wrote:
Hello Chris,
Am 29.07.2021 um 22:13 schrieb G1FEF via 44Net 44net@mailman.ampr.org:
Hi Thomas,
I can see that the mailing list software put you on moderation due to excessive bounces, that’s the reason.
thank you for your info.
But I havn't any e-mail problem. Unfortunately, two months are a too long time for looking at logs. Do you have such a bounce mail?
vy 73,
- Thomas dl9sau
Hi Rob,
Configuring a mailing list with regards to DMARC, SKIM/SPF is indeed a problem!
The issue stems from that fact that if someone sends an email from a domain that has a published DMARC policy, and is DKIM signed, then in order for that DMARC policy to be passed by the recipient, the From address, Subject and Body of the message must not be altered, otherwise the DKIM signing fails.
The 44Net mailing list changes the Subject by pre-pending "[44net]”, it also alters the body by appending the list information block of text. So right away DKIM will always fail.
The mailman authors added an option to get around this by munging the From address header. So if I send an email from: chris@ardc.net this gets changed to: 44net@mailman.ampr.org mailto:44net@mailman.ampr.org
My address gets added on as a CC: field (unless I specifically switch that off in my personal settings) so generally you can see who originally sent the email, which gives you the option to reply direct if you wish to.
However, there are better ways of doing this, without “munging” headers that are not supposed to get touched, like the From header, which was designed to never change as it identifies the original source of the email. One way of doing this would be to leave the From, Subject and Body alone and instead add a Sender header with the 44net@mailman.ampr.org mailto:44net@mailman.ampr.org address in. The Sender header was designed to identify someone sending an email on behalf of someone else, which is effectively what a mailing list is doing.
TBH, I haven’t touched the mailing list server (except to keep it updated) since Brian set it up. How Brian set it up was the best option at the time, but things change, technology moves on, often quite quickly in the internet world!
The munging of the From header does generally work, although, as you correctly pointed out, it does cause issues with recipients that check DMARC policies and DKIM signatures. Also, the mailing list server has never been setup to sign outgoing emails, there is an SPF record in place, but no DKIM for the mailing list server. We do have DMARC, DKIM & SPF for the top domain ampr.org http://ampr.org/ and all outgoing mails from @ampr.org will be signed and there should be alignment with DKIM and SPF, but we could never apply a policy to subdomains of ampr.org http://ampr.org/ as all the membership use it, e.g. I can setup chris@g1fef.ampr.org mailto:chris@g1fef.ampr.org etc. As the mailing list server uses a subdomain: mailman.ampr.org http://mailman.ampr.org/ this also created a problem.
This is one of the reasons ARDC moved to @ardc.net for emails, as we can fully control the domain without worrying about subdomains. Even though there is a mechanism for setting the policy on the top domain and subdomains separately, a lot of MTA’s still don’t honour these and we got an awful lot of issues with email deliverability when we used @ampr.org now we are on @ardc.net the problems have mostly gone away.
Perhaps we should move the mailing list to ardc.net http://ardc.net/ and set it up to interact better with today’s email environment? Just a thought.
Chris - G1FEF
On 1 Aug 2021, at 10:30, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
I think one reason for bounces could be that the mailman mailing list does not handle DKIM properly. When a user sends a mail to the list, it passes through the DKIM signature, but it also modifies the mail (adding a standard footer as the mimimum, probably more) As a result, the DKIM signature of messages received via the list is invalid, and maybe some mail servers see it as a reason to reject the mail (as if it is spoofed). In my mail system, most mails in this list get a red color because that is what some rule processing here does to get my attention to falsified mails.
In fact this should be solved by mailman, but some googling seems to indicate that the mailman people know that, but have not yet come into action. So likely there is no setting in the admin panel to easily solve this. Maybe some extra software or feature of the existing software could be used to work around this. That would at the minimum have to delete any DKIM-Signature: header on incoming mail (unfortunately this is a multiline header so it will require more than passing through some simple grep or sed), and additionally it could add a new DKIM signature afer all mailman processing, for the mailman.ampr.org domain. But the latter is optional.
Rob
Chris,
Yes it is a complicated situation, mailing list operators have often objected to DMARC/DKIM/SPF because of this. And, as you correctly describe, it also depends on the configuration of the mailing list. I think in the current setup (mail is "From 44net@mailman.ampr.org") the correct procedure is to delete the DKIM-Signature header (after its validation on the incoming side which is likely done by other software) so that it does not appear in the outgoing mails from the list. The receiving server sees the mailman.ampr.org sender so it has no reason to lookup the DMARC policy for the original sender, and there is no invalid DKIM signature so no problems with that either.
The mailman people recognize that solution but I think it has not been implemented yet. But maybe you have some other processing between the receiving mailserver and mailman (like procmail) where you could delete the header.
Rob
On 8/1/21 1:11 PM, Chris Smith via 44Net wrote:
Hi Rob,
Configuring a mailing list with regards to DMARC, SKIM/SPF is indeed a problem!
The issue stems from that fact that if someone sends an email from a domain that has a published DMARC policy, and is DKIM signed, then in order for that DMARC policy to be passed by the recipient, the From address, Subject and Body of the message must not be altered, otherwise the DKIM signing fails.
The 44Net mailing list changes the Subject by pre-pending "[44net]”, it also alters the body by appending the list information block of text. So right away DKIM will always fail.
The mailman authors added an option to get around this by munging the From address header. So if I send an email from: chris@ardc.net this gets changed to: 44net@mailman.ampr.org mailto:44net@mailman.ampr.org
My address gets added on as a CC: field (unless I specifically switch that off in my personal settings) so generally you can see who originally sent the email, which gives you the option to reply direct if you wish to.
However, there are better ways of doing this, without “munging” headers that are not supposed to get touched, like the From header, which was designed to never change as it identifies the original source of the email. One way of doing this would be to leave the From, Subject and Body alone and instead add a Sender header with the 44net@mailman.ampr.org mailto:44net@mailman.ampr.org address in. The Sender header was designed to identify someone sending an email on behalf of someone else, which is effectively what a mailing list is doing.
TBH, I haven’t touched the mailing list server (except to keep it updated) since Brian set it up. How Brian set it up was the best option at the time, but things change, technology moves on, often quite quickly in the internet world!
The munging of the From header does generally work, although, as you correctly pointed out, it does cause issues with recipients that check DMARC policies and DKIM signatures. Also, the mailing list server has never been setup to sign outgoing emails, there is an SPF record in place, but no DKIM for the mailing list server. We do have DMARC, DKIM & SPF for the top domain ampr.org http://ampr.org/ and all outgoing mails from @ampr.org will be signed and there should be alignment with DKIM and SPF, but we could never apply a policy to subdomains of ampr.org http://ampr.org/ as all the membership use it, e.g. I can setup chris@g1fef.ampr.org mailto:chris@g1fef.ampr.org etc. As the mailing list server uses a subdomain: mailman.ampr.org http://mailman.ampr.org/ this also created a problem.
This is one of the reasons ARDC moved to @ardc.net for emails, as we can fully control the domain without worrying about subdomains. Even though there is a mechanism for setting the policy on the top domain and subdomains separately, a lot of MTA’s still don’t honour these and we got an awful lot of issues with email deliverability when we used @ampr.org now we are on @ardc.net the problems have mostly gone away.
Perhaps we should move the mailing list to ardc.net http://ardc.net/ and set it up to interact better with today’s email environment? Just a thought.
Chris - G1FEF
On 1 Aug 2021, at 10:30, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
I think one reason for bounces could be that the mailman mailing list does not handle DKIM properly. When a user sends a mail to the list, it passes through the DKIM signature, but it also modifies the mail (adding a standard footer as the mimimum, probably more) As a result, the DKIM signature of messages received via the list is invalid, and maybe some mail servers see it as a reason to reject the mail (as if it is spoofed). In my mail system, most mails in this list get a red color because that is what some rule processing here does to get my attention to falsified mails.
In fact this should be solved by mailman, but some googling seems to indicate that the mailman people know that, but have not yet come into action. So likely there is no setting in the admin panel to easily solve this. Maybe some extra software or feature of the existing software could be used to work around this. That would at the minimum have to delete any DKIM-Signature: header on incoming mail (unfortunately this is a multiline header so it will require more than passing through some simple grep or sed), and additionally it could add a new DKIM signature afer all mailman processing, for the mailman.ampr.org domain. But the latter is optional.
Rob
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
On Sun, 2021-08-01 at 13:30 +0200, Rob PE1CHL via 44Net wrote:
the correct procedure is to delete the DKIM-Signature header
Rob is correct. This can easily be achieved by setting REMOVE_DKIM_HEADERS=3 in mm_cfg.py. You can see all the available options in Mailman/Defaults.py.
-Jim P.
Yes, that’s imperative for any mailing list these days too.
On Aug 1, 2021, at 7:50 AM, Jim Popovitch via 44Net 44net@mailman.ampr.org wrote:
On Sun, 2021-08-01 at 13:30 +0200, Rob PE1CHL via 44Net wrote:
the correct procedure is to delete the DKIM-Signature header
Rob is correct. This can easily be achieved by setting REMOVE_DKIM_HEADERS=3 in mm_cfg.py. You can see all the available options in Mailman/Defaults.py.
-Jim P.
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
Since moving to Mailman3, I haven’t see any DKIM/DMARC issues. We replace From and make sure any footer added is an attachment. Mailman2 has long been in maintenance mode.
Jason
On Aug 1, 2021, at 7:31 AM, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
Chris,
Yes it is a complicated situation, mailing list operators have often objected to DMARC/DKIM/SPF because of this. And, as you correctly describe, it also depends on the configuration of the mailing list. I think in the current setup (mail is "From 44net@mailman.ampr.org") the correct procedure is to delete the DKIM-Signature header (after its validation on the incoming side which is likely done by other software) so that it does not appear in the outgoing mails from the list. The receiving server sees the mailman.ampr.org sender so it has no reason to lookup the DMARC policy for the original sender, and there is no invalid DKIM signature so no problems with that either.
The mailman people recognize that solution but I think it has not been implemented yet. But maybe you have some other processing between the receiving mailserver and mailman (like procmail) where you could delete the header.
Rob
On 8/1/21 1:11 PM, Chris Smith via 44Net wrote: Hi Rob,
Configuring a mailing list with regards to DMARC, SKIM/SPF is indeed a problem!
The issue stems from that fact that if someone sends an email from a domain that has a published DMARC policy, and is DKIM signed, then in order for that DMARC policy to be passed by the recipient, the From address, Subject and Body of the message must not be altered, otherwise the DKIM signing fails.
The 44Net mailing list changes the Subject by pre-pending "[44net]”, it also alters the body by appending the list information block of text. So right away DKIM will always fail.
The mailman authors added an option to get around this by munging the From address header. So if I send an email from: chris@ardc.net this gets changed to: 44net@mailman.ampr.org mailto:44net@mailman.ampr.org
My address gets added on as a CC: field (unless I specifically switch that off in my personal settings) so generally you can see who originally sent the email, which gives you the option to reply direct if you wish to.
However, there are better ways of doing this, without “munging” headers that are not supposed to get touched, like the From header, which was designed to never change as it identifies the original source of the email. One way of doing this would be to leave the From, Subject and Body alone and instead add a Sender header with the 44net@mailman.ampr.org mailto:44net@mailman.ampr.org address in. The Sender header was designed to identify someone sending an email on behalf of someone else, which is effectively what a mailing list is doing.
TBH, I haven’t touched the mailing list server (except to keep it updated) since Brian set it up. How Brian set it up was the best option at the time, but things change, technology moves on, often quite quickly in the internet world!
The munging of the From header does generally work, although, as you correctly pointed out, it does cause issues with recipients that check DMARC policies and DKIM signatures. Also, the mailing list server has never been setup to sign outgoing emails, there is an SPF record in place, but no DKIM for the mailing list server. We do have DMARC, DKIM & SPF for the top domain ampr.org http://ampr.org/ and all outgoing mails from @ampr.org will be signed and there should be alignment with DKIM and SPF, but we could never apply a policy to subdomains of ampr.org http://ampr.org/ as all the membership use it, e.g. I can setup chris@g1fef.ampr.org mailto:chris@g1fef.ampr.org etc. As the mailing list server uses a subdomain: mailman.ampr.org http://mailman.ampr.org/ this also created a problem.
This is one of the reasons ARDC moved to @ardc.net for emails, as we can fully control the domain without worrying about subdomains. Even though there is a mechanism for setting the policy on the top domain and subdomains separately, a lot of MTA’s still don’t honour these and we got an awful lot of issues with email deliverability when we used @ampr.org now we are on @ardc.net the problems have mostly gone away.
Perhaps we should move the mailing list to ardc.net http://ardc.net/ and set it up to interact better with today’s email environment? Just a thought.
Chris - G1FEF
On 1 Aug 2021, at 10:30, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
I think one reason for bounces could be that the mailman mailing list does not handle DKIM properly. When a user sends a mail to the list, it passes through the DKIM signature, but it also modifies the mail (adding a standard footer as the mimimum, probably more) As a result, the DKIM signature of messages received via the list is invalid, and maybe some mail servers see it as a reason to reject the mail (as if it is spoofed). In my mail system, most mails in this list get a red color because that is what some rule processing here does to get my attention to falsified mails.
In fact this should be solved by mailman, but some googling seems to indicate that the mailman people know that, but have not yet come into action. So likely there is no setting in the admin panel to easily solve this. Maybe some extra software or feature of the existing software could be used to work around this. That would at the minimum have to delete any DKIM-Signature: header on incoming mail (unfortunately this is a multiline header so it will require more than passing through some simple grep or sed), and additionally it could add a new DKIM signature afer all mailman processing, for the mailman.ampr.org domain. But the latter is optional.
Rob
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
On 1 Aug 2021, at 12:30, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
Chris,
Yes it is a complicated situation, mailing list operators have often objected to DMARC/DKIM/SPF because of this. And, as you correctly describe, it also depends on the configuration of the mailing list. I think in the current setup (mail is "From 44net@mailman.ampr.org") the correct procedure is to delete the DKIM-Signature header (after its validation on the incoming side which is likely done by other software) so that it does not appear in the outgoing mails from the list.
That would work of course, but it means writing a hack to fix the issue, the problem I have found with doing that, is some software package gets updated and the hack breaks everything. I prefer to wait for the proper fix from the folks that wrote the software.
The receiving server sees the mailman.ampr.org sender so it has no reason to lookup the DMARC policy for the original sender, and there is no invalid DKIM signature so no problems with that either. The mailman people recognize that solution but I think it has not been implemented yet. But maybe you have some other processing between the receiving mailserver and mailman (like procmail) where you could delete the header.
The “correct” solution is probably RFC8617 (http://arc-spec.org http://arc-spec.org/) there is a patch available for mailman, but I don't really want to mess with the current setup as it mostly works, and there’s always a risk of breaking something. As I said before, we should probably move the mailing list to ardc.net http://ardc.net/ so we can control DMARC properly, if we were to setup a new mailman instance we can also get all these things in place and working first.
I’m happy to look into this if the need is there… Are there a lot of folk affected by this?
73, Chris - G1FEF
Rob
On 8/1/21 1:11 PM, Chris Smith via 44Net wrote:
Hi Rob,
Configuring a mailing list with regards to DMARC, SKIM/SPF is indeed a problem!
The issue stems from that fact that if someone sends an email from a domain that has a published DMARC policy, and is DKIM signed, then in order for that DMARC policy to be passed by the recipient, the From address, Subject and Body of the message must not be altered, otherwise the DKIM signing fails.
The 44Net mailing list changes the Subject by pre-pending "[44net]”, it also alters the body by appending the list information block of text. So right away DKIM will always fail.
The mailman authors added an option to get around this by munging the From address header. So if I send an email from: chris@ardc.net this gets changed to: 44net@mailman.ampr.org mailto:44net@mailman.ampr.org
My address gets added on as a CC: field (unless I specifically switch that off in my personal settings) so generally you can see who originally sent the email, which gives you the option to reply direct if you wish to.
However, there are better ways of doing this, without “munging” headers that are not supposed to get touched, like the From header, which was designed to never change as it identifies the original source of the email. One way of doing this would be to leave the From, Subject and Body alone and instead add a Sender header with the 44net@mailman.ampr.org mailto:44net@mailman.ampr.org address in. The Sender header was designed to identify someone sending an email on behalf of someone else, which is effectively what a mailing list is doing.
TBH, I haven’t touched the mailing list server (except to keep it updated) since Brian set it up. How Brian set it up was the best option at the time, but things change, technology moves on, often quite quickly in the internet world!
The munging of the From header does generally work, although, as you correctly pointed out, it does cause issues with recipients that check DMARC policies and DKIM signatures. Also, the mailing list server has never been setup to sign outgoing emails, there is an SPF record in place, but no DKIM for the mailing list server. We do have DMARC, DKIM & SPF for the top domain ampr.org http://ampr.org/ and all outgoing mails from @ampr.org will be signed and there should be alignment with DKIM and SPF, but we could never apply a policy to subdomains of ampr.org http://ampr.org/ as all the membership use it, e.g. I can setup chris@g1fef.ampr.org mailto:chris@g1fef.ampr.org etc. As the mailing list server uses a subdomain: mailman.ampr.org http://mailman.ampr.org/ this also created a problem.
This is one of the reasons ARDC moved to @ardc.net for emails, as we can fully control the domain without worrying about subdomains. Even though there is a mechanism for setting the policy on the top domain and subdomains separately, a lot of MTA’s still don’t honour these and we got an awful lot of issues with email deliverability when we used @ampr.org now we are on @ardc.net the problems have mostly gone away.
Perhaps we should move the mailing list to ardc.net http://ardc.net/ and set it up to interact better with today’s email environment? Just a thought.
Chris - G1FEF
On 1 Aug 2021, at 10:30, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
I think one reason for bounces could be that the mailman mailing list does not handle DKIM properly. When a user sends a mail to the list, it passes through the DKIM signature, but it also modifies the mail (adding a standard footer as the mimimum, probably more) As a result, the DKIM signature of messages received via the list is invalid, and maybe some mail servers see it as a reason to reject the mail (as if it is spoofed). In my mail system, most mails in this list get a red color because that is what some rule processing here does to get my attention to falsified mails.
In fact this should be solved by mailman, but some googling seems to indicate that the mailman people know that, but have not yet come into action. So likely there is no setting in the admin panel to easily solve this. Maybe some extra software or feature of the existing software could be used to work around this. That would at the minimum have to delete any DKIM-Signature: header on incoming mail (unfortunately this is a multiline header so it will require more than passing through some simple grep or sed), and additionally it could add a new DKIM signature afer all mailman processing, for the mailman.ampr.org domain. But the latter is optional.
Rob
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
Well, I think "affected" is almost everyone on the list, but I cannot see how many users were fatally affected (unsubscribed), that is something you should be able to see in logs. I just heard from a local user he got unsubscribed as well.
Rob
On 8/1/21 1:57 PM, Chris Smith via 44Net wrote:
I’m happy to look into this if the need is there… Are there a lot of folk affected by this?
On 1 Aug 2021, at 13:04, Rob PE1CHL via 44Net 44net@mailman.ampr.org wrote:
Well, I think "affected" is almost everyone on the list, but I cannot see how many users were fatally affected (unsubscribed), that is something you should be able to see in logs.
I get 1 or 2 month, so not a huge number. Most of them are genuine bounces, e.g. the user’s mailbox is full, or no longer exists.
Ok, I will add this to my todo list and look into the various options available to us.
Chris
I just heard from a local user he got unsubscribed as well.
Rob
On 8/1/21 1:57 PM, Chris Smith via 44Net wrote:
I’m happy to look into this if the need is there… Are there a lot of folk affected by this?
44Net mailing list 44Net@mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net
-
Chris Smith -
G1FEF -
Jason McCormick -
Jim Popovitch -
Rob PE1CHL -
Thomas Osterried