Difference between revisions of "2024/05/10"

From HypertWiki
Jump to navigation Jump to search
(Created page with "{{page/date|Woozle's journal}} category:journal category:story category:tech ''writing should commence later today; just posting images for now......")
 
(template shenanigans)
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{page/date|Woozle's [[Woozle/blarg|journal]]}}
{{woozle/page/journal}}
[[category:journal]]
[[category:story]]
[[category:story]]
[[category:tech]]
[[category:tech]]
''writing should commence later today; just posting images for now...''
[[category:Hetzner]]
<gallery>
[[category:Digital Ocean]]
File:2024-05-10.screen.01.png
{| align=right
File:2024-05-10.screen.02.png
|-
File:2024-05-10.screen.03.png
|'''Discuss''': [https://toot.cat/@woozle/112422665268341073 APub fedi (Mastodon)]
File:2024-05-10.screen.04.png
|}
File:2024-05-10.screen.05.png
{| width=100%
File:2024-05-10.screen.06.png
|-
File:2024-05-10.screen.07.png
| __TOC__
File:2024-05-10.screen.08.png
| align=right |
File:2024-05-10.screen.09.png
{| width=50%
File:2024-05-10.screen.10.png
|-
File:2024-05-10.screen.11.png
|
File:2024-05-10.screen.12.png
==Non-Technical Summary==
</gallery>
* Gmail is blocking messages from my email server.
* It turns out the address was somehow retroactively placed on 3 blocklists, from an incident 3+ years go, before I was using it.
* I've managed to get off 2 of them.
* The 3rd one is more tricky -- it's for a large block of addresses, and the only ways to get off are (1) get the provider (Digital Ocean, in this case) to do it, (2) pay a not-terrible amount of money to a service called Whitelisted, (3) move to a server that isn't in a blocklist zone.
* I'm considering which combination of 2 and 3 to go with, because fat chance of #1 happening. I have found a server-center that isn't blocked -- it's in Finland :D
|}
|}
==Where It Starts==
Two days ago (May 8), I noticed that Gmail is now outright blocking my emails.
<woozalia@gmail.com>: host gmail-smtp-in.l.google.com[172.253.115.27] said:
    550-5.7.1 [68.183.140.54      18] Gmail has detected that this message is
    likely 550-5.7.1 suspicious due to the very low reputation of the sending
    IP address. 550-5.7.1 To best protect our users from spam, the message has
    been blocked. 550-5.7.1 For more information, go to 550 5.7.1
    https://support.google.com/mail/answer/188131
 
I was [https://toot.cat/@woozle/112407604197312764 rather frightfully put out] by this, having already spent many hours/days configuring SPF and DKIM and I don't know what else in order to prevent being spamcanned -- and the message seemed to suggest that there was no remedy or appeals process.
 
It had been working fine as recently as May 5, so whatever changed was obviously very recent.
[[File:2024-05-10.screen.13.png|400px|thumb|cloud1 (which uses the blocked IP address) first spun up on 2020/10/21.]]
'''Long story short''':
* Actually, no, it happened in August of 2020 -- definitely ''not'' sometime between May 5 and May 8, not recent, and in fact over a month before I started using that IP address.
* There are 3 IP blocklists which might be causing this. I've managed to get off two of them; the third involves more effort and possibly some money.
 
==Long Story Not Short==
Upon following the "more information" link, I found information which suggested that Gmail was blocking me because I'd ended up on a blocklist somehow, and that this had happened because I was running an open SMTP relay on that server (which is something you just don't do, in this era, because spammers ''will'' abuse it).
===finding the problem===
The first thing I did (yesterday afternoon) was check [https://mxtoolbox.com MX Toolbox] to make sure I hadn't inadvertently opened a relay (or that my system had been hacked and one had been installed) -- nope, all good:
[[File:2024-05-10.screen.02.png]]
 
Next thing was to see if I could find out what blocklists I'm on, and hopefully why.
[[File:2024-05-10.screen.01.png]]
 
Note that this list goes on for about 3-4x that long, and all the rest are green/OK.
===getting off the lists===
====List the First====
The next step, then, is how to appeal these listings -- especially given that they're all based on an incident which happened before I was assigned that IP address, and which was in fact over 3 years ago.
 
SORBS seemed to be telling me that I basically can't right now because they're dealing with a DDoS attack... except that "Re-testing is currently ENABLED", so...?
 
[[File:2024-05-10.screen.03.png]]
 
I was able to create an account and at that point it seemed willing to let me start into the de-listing process -- at which point I first found out just how far back they were reaching:
 
[[File:2024-05-10.screen.04.png]]
 
In any case, I was able to request a de-listing -- which succeeded, and somehow ended me up at Spamhaus (not SORBS), so maybe that "can't do it now [yes we can]" warning was about something else?
 
[[File:2024-05-10.screen.05.png]]
====List the Second====
So that's one down. It was bedtime, so I started in on the next one the next day, i.e. this morning: "SORBS SPAM" -- huh, SORBS again? Didn't I just do them? Well, apparently not...
 
[[File:2024-05-10.screen.06.png|thumb|I saw the SORBS information here first, but the other screen seems rather more readable.]]
[[File:2024-05-10.screen.07.png]]
 
I tried to submit a removal request from there, but it said this:
 
[[File:2024-05-10.screen.08.png]]
 
...which turned out to mean that I had to be actually accessing the web site ''from'' the blocked IP address.
 
Now, that machine is not a desktop. It doesn't have a desktop installed on it. There's Linux trickery<ref>If you connect with <code>ssh -X</code>, you can run GUI applications remotely.</ref> you can use in order to run GUI applications -- such as a web browser -- on a remote machine, but this tends to be slow and also adds more computing burden to the machine (lots of libraries need to be installed, and it may cause additional processes to always be run at startup; I don't actually know), so I try to avoid it -- and web browsers tend to be particularly CPU-hungry.
 
My first thought was to install a very basic browser, since the SORBS site doesn't seem likely to expect a lot of finicky JavaScript stuff to work right -- but then I remembered the existence of  {{l/wp|Lynx (web browser)}}, a web browser that runs entirely in a text-terminal... and after some stumbling around trying to figure out how to navigate web-pages and click on things without a mouse, success was attained:
 
[[File:2024-05-10.screen.08a.png]]
[[File:2024-05-10.screen.09.png]]
====List the Third====
And now we come to the final blocklist on the list of listed blocks, "UCEPROTECTL3":
 
{|
|-
| [[File:2024-05-10.screen.10.png|x400px]]
| [[File:2024-05-10.screen.11.png|x400px]]
|}
 
The second screen in particular contained some text which seems to throw some light on the situation:
<blockquote>
Further information which seems potentially illuminating:
'''As you should know now: It is not you, it is your complete provider which got UCEPROTECT-Level 3 listed.'''
<br>Your IP 68.183.140.54 was NOT part of abusive action, but you are the one that has freely chosen your provider.
By tolerating or ignoring that your provider doesn't care about abusers you are indirectly also supporting the global spam with your money.
Seen from this point of view, you really shouldn't wonder about the consequences.
'''Therefore we recommend:'''
<br>Please send a complaint to your provider and request they fix this problem immediatly.
Think about this: You pay them so that you can use the Internet without problems;
 
If they are ignoring your complaint or claiming they can't do anything, you should consider changing your provider.
There are currently about 105,000 providers worldwide, but only a few hundred make it to get listed into UCEPROTECT-Level 3.
According to the statistics measured against the mailflow of several national authorities in Germany, Austria and Switzerland, those few providers which often end up in our Level 3 are responsible for 50 - 75% of all global spam, while almost no real mail came from their networks and ranges.
</blockquote>
...which seems maybe a tad harsh (especially the first paragraph) -- but reading it in Autistic Non-Implicatory Mode, it actually seems like a pretty legit point.
[[File:2024-05-10.screen.14.png|thumb|Not On Any of the Lists!]]
 
So apparently my primary recourse is to get DigitalOcean to get ''their'' house in order. Do I have any confidence that they can or will? Kind of not. However, there are at least two other options:
# Find a servlet (VPS) host whose IP range is ''not'' blocked for bad behavior.
# Get on WhiteListed:
 
[[File:2024-05-10.screen.12.png]]
 
This, unfortunately, costs money (although the price goes down steeply the further ahead you pay). (Note: CHF - Swiss Francs - are apparently approximately $1.)
 
I'm thinking I will want to use some combination of the two: pay for a certain amount of WhiteListed time, and also move to another host -- because, of the two available Hetzlets<ref>like a "droplet" ({{l/wp|Virtual private server|VPS}}) at Digital Ocean, except at Hetzner</ref> I currently have set up, although one of them (in the US) is on UCE's blocklist, the other one -- in Finland -- is not (see sidebar, because for some reason I wanted to include the entire listing).
 
Moral of the story: before setting up an email host, check the blocklists for an address that isn't on any.
==5/12 followup: unblocked!==
Well, Gmail is now accepting my messages again -- so I guess I don't actually need to get off that 3rd list, or at least not so urgently. I'll still be moving my email server to a block that isn't listed there, eventually, but I don't have to rush about it (or pay money to Whitelisted). So ''that's'' good.
==5/13 followup: MS careth not==
<pre>
<woozalia@outlook.com>: host
    outlook-com.olc.protection.outlook.com[52.101.73.18] said: 550 5.7.1
    Unfortunately, messages from [68.183.140.54] weren't sent. Please contact
    your Internet service provider since part of their network is on our block
    list (S3140). You can also refer your provider to
    http://mail.live.com/mail/troubleshooting.aspx#errors.
    [AMS1EPF0000004A.eurprd04.prod.outlook.com 2024-05-13T14:22:20.493Z
    08DC723349903703] (in reply to MAIL FROM command)
</pre>
I have to think that maybe the remaining blocklist is the issue here. I need to try emailing myself from Hetz1... but I need to set up a proper server and webmail, first, so I can check off the "Delivery status notification" box (which I'm sure is a header thing that I could do ''without'' a webmail client, but then I also need a way to view returned messages and would prefer not to spend an hour figuring out what I'm doing and then getting ambiguous results).
 
==Footnote==
<references>
</references>

Latest revision as of 20:30, 13 May 2024

Friday, May 10, 2024 (#131)
Woozle's Journal
Thursday Friday Saturday
Exact day: category (1) This month: category (6) / page
Other years: category (3) This year: category (6) / page
Discuss: APub fedi (Mastodon)

Non-Technical Summary

  • Gmail is blocking messages from my email server.
  • It turns out the address was somehow retroactively placed on 3 blocklists, from an incident 3+ years go, before I was using it.
  • I've managed to get off 2 of them.
  • The 3rd one is more tricky -- it's for a large block of addresses, and the only ways to get off are (1) get the provider (Digital Ocean, in this case) to do it, (2) pay a not-terrible amount of money to a service called Whitelisted, (3) move to a server that isn't in a blocklist zone.
  • I'm considering which combination of 2 and 3 to go with, because fat chance of #1 happening. I have found a server-center that isn't blocked -- it's in Finland :D

Where It Starts

Two days ago (May 8), I noticed that Gmail is now outright blocking my emails.

<woozalia@gmail.com>: host gmail-smtp-in.l.google.com[172.253.115.27] said:
   550-5.7.1 [68.183.140.54      18] Gmail has detected that this message is
   likely 550-5.7.1 suspicious due to the very low reputation of the sending
   IP address. 550-5.7.1 To best protect our users from spam, the message has
   been blocked. 550-5.7.1 For more information, go to 550 5.7.1
   https://support.google.com/mail/answer/188131

I was rather frightfully put out by this, having already spent many hours/days configuring SPF and DKIM and I don't know what else in order to prevent being spamcanned -- and the message seemed to suggest that there was no remedy or appeals process.

It had been working fine as recently as May 5, so whatever changed was obviously very recent.

cloud1 (which uses the blocked IP address) first spun up on 2020/10/21.

Long story short:

  • Actually, no, it happened in August of 2020 -- definitely not sometime between May 5 and May 8, not recent, and in fact over a month before I started using that IP address.
  • There are 3 IP blocklists which might be causing this. I've managed to get off two of them; the third involves more effort and possibly some money.

Long Story Not Short

Upon following the "more information" link, I found information which suggested that Gmail was blocking me because I'd ended up on a blocklist somehow, and that this had happened because I was running an open SMTP relay on that server (which is something you just don't do, in this era, because spammers will abuse it).

finding the problem

The first thing I did (yesterday afternoon) was check MX Toolbox to make sure I hadn't inadvertently opened a relay (or that my system had been hacked and one had been installed) -- nope, all good: 2024-05-10.screen.02.png

Next thing was to see if I could find out what blocklists I'm on, and hopefully why. 2024-05-10.screen.01.png

Note that this list goes on for about 3-4x that long, and all the rest are green/OK.

getting off the lists

List the First

The next step, then, is how to appeal these listings -- especially given that they're all based on an incident which happened before I was assigned that IP address, and which was in fact over 3 years ago.

SORBS seemed to be telling me that I basically can't right now because they're dealing with a DDoS attack... except that "Re-testing is currently ENABLED", so...?

2024-05-10.screen.03.png

I was able to create an account and at that point it seemed willing to let me start into the de-listing process -- at which point I first found out just how far back they were reaching:

2024-05-10.screen.04.png

In any case, I was able to request a de-listing -- which succeeded, and somehow ended me up at Spamhaus (not SORBS), so maybe that "can't do it now [yes we can]" warning was about something else?

2024-05-10.screen.05.png

List the Second

So that's one down. It was bedtime, so I started in on the next one the next day, i.e. this morning: "SORBS SPAM" -- huh, SORBS again? Didn't I just do them? Well, apparently not...

I saw the SORBS information here first, but the other screen seems rather more readable.

2024-05-10.screen.07.png

I tried to submit a removal request from there, but it said this:

2024-05-10.screen.08.png

...which turned out to mean that I had to be actually accessing the web site from the blocked IP address.

Now, that machine is not a desktop. It doesn't have a desktop installed on it. There's Linux trickery[1] you can use in order to run GUI applications -- such as a web browser -- on a remote machine, but this tends to be slow and also adds more computing burden to the machine (lots of libraries need to be installed, and it may cause additional processes to always be run at startup; I don't actually know), so I try to avoid it -- and web browsers tend to be particularly CPU-hungry.

My first thought was to install a very basic browser, since the SORBS site doesn't seem likely to expect a lot of finicky JavaScript stuff to work right -- but then I remembered the existence of Lynx (web browser), a web browser that runs entirely in a text-terminal... and after some stumbling around trying to figure out how to navigate web-pages and click on things without a mouse, success was attained:

2024-05-10.screen.08a.png 2024-05-10.screen.09.png

List the Third

And now we come to the final blocklist on the list of listed blocks, "UCEPROTECTL3":

2024-05-10.screen.10.png 2024-05-10.screen.11.png

The second screen in particular contained some text which seems to throw some light on the situation:

Further information which seems potentially illuminating: As you should know now: It is not you, it is your complete provider which got UCEPROTECT-Level 3 listed.
Your IP 68.183.140.54 was NOT part of abusive action, but you are the one that has freely chosen your provider. By tolerating or ignoring that your provider doesn't care about abusers you are indirectly also supporting the global spam with your money. Seen from this point of view, you really shouldn't wonder about the consequences.

Therefore we recommend:
Please send a complaint to your provider and request they fix this problem immediatly. Think about this: You pay them so that you can use the Internet without problems;

If they are ignoring your complaint or claiming they can't do anything, you should consider changing your provider. There are currently about 105,000 providers worldwide, but only a few hundred make it to get listed into UCEPROTECT-Level 3.

According to the statistics measured against the mailflow of several national authorities in Germany, Austria and Switzerland, those few providers which often end up in our Level 3 are responsible for 50 - 75% of all global spam, while almost no real mail came from their networks and ranges.

...which seems maybe a tad harsh (especially the first paragraph) -- but reading it in Autistic Non-Implicatory Mode, it actually seems like a pretty legit point.

Not On Any of the Lists!

So apparently my primary recourse is to get DigitalOcean to get their house in order. Do I have any confidence that they can or will? Kind of not. However, there are at least two other options:

  1. Find a servlet (VPS) host whose IP range is not blocked for bad behavior.
  2. Get on WhiteListed:

2024-05-10.screen.12.png

This, unfortunately, costs money (although the price goes down steeply the further ahead you pay). (Note: CHF - Swiss Francs - are apparently approximately $1.)

I'm thinking I will want to use some combination of the two: pay for a certain amount of WhiteListed time, and also move to another host -- because, of the two available Hetzlets[2] I currently have set up, although one of them (in the US) is on UCE's blocklist, the other one -- in Finland -- is not (see sidebar, because for some reason I wanted to include the entire listing).

Moral of the story: before setting up an email host, check the blocklists for an address that isn't on any.

5/12 followup: unblocked!

Well, Gmail is now accepting my messages again -- so I guess I don't actually need to get off that 3rd list, or at least not so urgently. I'll still be moving my email server to a block that isn't listed there, eventually, but I don't have to rush about it (or pay money to Whitelisted). So that's good.

5/13 followup: MS careth not

<woozalia@outlook.com>: host
    outlook-com.olc.protection.outlook.com[52.101.73.18] said: 550 5.7.1
    Unfortunately, messages from [68.183.140.54] weren't sent. Please contact
    your Internet service provider since part of their network is on our block
    list (S3140). You can also refer your provider to
    http://mail.live.com/mail/troubleshooting.aspx#errors.
    [AMS1EPF0000004A.eurprd04.prod.outlook.com 2024-05-13T14:22:20.493Z
    08DC723349903703] (in reply to MAIL FROM command)

I have to think that maybe the remaining blocklist is the issue here. I need to try emailing myself from Hetz1... but I need to set up a proper server and webmail, first, so I can check off the "Delivery status notification" box (which I'm sure is a header thing that I could do without a webmail client, but then I also need a way to view returned messages and would prefer not to spend an hour figuring out what I'm doing and then getting ambiguous results).

Footnote

  1. If you connect with ssh -X, you can run GUI applications remotely.
  2. like a "droplet" (VPS) at Digital Ocean, except at Hetzner