Subject: Re: Spam

From: Andrew Shore <[email-masked]>
Date: Fri, 30 May 2003 11:55:23 +0100

The following is part of a message posted to my ISP's announce NG. I 
found it interesting, specifically the claim that there are 180 entities 
creating nearly all the spam... it would explain the recurrence of 
certain seemingly unlikely patterns across the board, but is still hard 
to credit even with that in mind.

-----

Subject:     Unsolicited Commercial / Bulk Email Spam
Newsgroups:  demon.announce
From:        Malcolm S. Muir <[email-masked]>
Reply-To:    [email-masked]
Date:        Fri, 30 May 2003 10:41:48 +0100
Message-ID:  <[email-masked]>

The following is a summary of advice we have available on our web
site for customers having trouble with unsolicited email.

The full text can be found at:

    http://www.demon.net/helpdesk/spam

[snip]
Where does UCE come from?

    Some 'spam' is sent by companies that are new to the Internet
    and do not understand how unwelcome this material is. However
    recent reports suggest that 90% of all the material currently
    being sent originates from as few as 180 individuals or 'spam
    gangs'. These groups make a business out of promoting unsavoury
    (and sometimes illegal) material. They hide the true origin of
    the material by relaying their email via insecure mail systems
    and machines. Although in the past they have targeted incorrectly
    configured machines at ISPs and large companies, they now
    regularly exploit end-user ('customer') machines.
[snip]

Malcolm Muir
Demon Internet

-----

Andrew.
-- 
Andrew Shore.

---

From: Paul Vixie <[email-masked]>
Date: 30 May 2003 18:39:57 +0000

[email-masked] (Andrew Shore) writes:

> ... I found it interesting, specifically the claim that there are 180
> entities creating nearly all the spam... it would explain the recurrence
> of certain seemingly unlikely patterns across the board, but is still
> hard to credit even with that in mind.

it's absolutely true.  see the background at www.spamhaus.org for details.

however, it's also irrelevant.  bad spam has currently driven-low gray spam,
but if not for those ~200 spam gangs giving unsolicited e-mail a bad reputation
there would be 50X more unsolicited bulk e-mail coming to each of us, from
companies who don't want to be evil but who don't think our privacy is worth
as much as their revenues.  the spectre of "gray spam" from so-called legit
companies is more frightening to me even then the Windows Open Proxy Platform.

if we could put those ~200 spam gangs in jail then they would be replaced in
a month.  if we could make spamganging so illegal that they were eventually
not replaced, then their traffic would be replaced by bulk e-mail from every
customer of every CRM (customer relationship management) company in the world.

not that it's practical to even do away with the spam gangs we have now, but
what would replace them is actually worse, so let's reconsider that goal set.

as i've said here several times in the last 10 days, the problem isn't
stopping spam but enabling batched interpersonal digital communications
where the recipients are happy to have some delivery costs shifted to them
because they have some confidence in the identity of the sender, the intent
of the relay, and the value (to the sender) of the reception.

don't think negative thoughts like "how can we save smtp from the spammers"
but rather positive ones like "what batched interpersonal digital
communications system can we create that fits the full spectrum of humanity"?
smtp was for the research community, where trust was implied by reachability.
in the full spectrum of humanity, that implication is, and has always been,
provably false.

smtp cannot be saved.  let's move on.
-- 
Paul Vixie

---

From: John Stracke <[email-masked]>
Date: Fri, 30 May 2003 15:17:30 -0400

Paul Vixie wrote:

>if we could make spamganging so illegal that they were eventually
I'm not sure why this follows.  Wiping out the fraudulent spammers would 
take a War On Spam, with massive levels of propaganda to stir up public 
opinion enough to justify the money it would cost.  In that kind of 
environment, even in the aftermath, no white hat would dare spam; their 
customers would immediately turn against them.

-- 
/============================================\
|John Stracke      [email-masked]     |
|Principal Engineer|http://www.centive.com   |
|Centive           |My opinions are my own.  |
|============================================|
|Never do card tricks for your poker buddies.|
\============================================/

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 30 May 2003 15:02:06 -0500

> customers would immediately turn against them.

The way to avoid that kind of blow-back against legitimate use is through
documented innocence (saving the explicit opt-in confirmation).

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Paul Vixie <[email-masked]>
Date: 31 May 2003 19:43:04 +0000

> >if we could make spamganging so illegal that they were eventually not
> >replaced, then their traffic would be replaced by bulk e-mail from every

i apologize for sounding too much like a futurist in the above-quoted text.
i did not mean to sound speculative.

i've met with these people.  not just a few, but with dozens.  i know what
they want to do and i know why they aren't doing it and i know how much money
they think they can make/save/both if they could only send unsolicited bulk
e-mail.  the mere fact that "if everybody did it then no e-mail would ever
again be read" does not sway them at all.  the mere fact that the pond was
once poisoned by evil-intending bad actors is completely irrelevant to them.

what i don't understand about the internet engineering mindset is that y'all
think you can build a universal "distributed system" for end to end 
communications and that somehow it won't be used to its full potential.  you
think other people will "leave money on the table".  they just won't, ever!

once thinking that smtp would be the right answer for a global population,
or now thinking that smtp can be saved if we can just make abuse expensive,
is indicative of designers who think that the only consumers are Just Like
Them And Their Friends.  they were but aren't, and they won't ever be again!

get a grip!  the full spectrum of humanity is impossibly broad and is not
subject to the kinds of politeness rules that you think are implicit in the
act of taking every breath or step in a life.  and an interpersonal batch
communication system addressing such broadness Won't look much like smtp.
-- 
Paul Vixie

---

From: Dave Crocker <[email-masked]>
Date: Sat, 31 May 2003 14:28:59 -0700

Paul,

PV> once thinking that smtp would be the right answer for a global population,
PV> or now thinking that smtp can be saved if we can just make abuse expensive,
PV> is indicative of designers who think that the only consumers are Just Like
PV> Them And Their Friends.  they were but aren't, and they won't ever be again!

Unfortunately, casting the issue in terms of saving or discarding SMTP
is also unhelpful. It makes sure that we debate lots of protocol
details, without having any basic agreement on the more import usage and
protection details that will drive the mundane technical choices.

On the other hand, a relevant point about current operations, versus
future operations, is how we move 100 million users.  But again, that is
best discussed in terms of the users and their usage, rather than the
protocol details.

So:  What is this better service supposed to look like?  How do we
transition a very large installed base to it.

Extra credit:  The transition question has more to do with user
motivation and choice than with technical engineering.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Paul Vixie <[email-masked]>
Date: Sat, 31 May 2003 22:05:37 +0000

(keith, this message is also meant to serve as an answer to your private mail)

> details, without having any basic agreement on the more import usage
> and protection details that will drive the mundane technical choices.

well, but, so, the debate about protocol details was raging before i
started saying "stop trying to save smtp" so these items are at best
unrelated.

> So:  What is this better service supposed to look like?

see further below, which was written in response to private mail from
keith before i saw dave's message (to which this is a direct reply.)

> How do we transition a very large installed base to it.

the whole installed base is in incredible pain right now and there will
be a lot of sites like mine who will switch over early, turning off
their tcp/25 service, as soon as we think there are reasonable
alternatives for reaching us.  (this contrasts with the ipv6 transition,
where the existing ipv4 installed base is not wholly in incredible pain;
only a few mega-networks are feeling incredible ipv4 pain, and the vast
majority of non-mega-networks is feeling no ipv4 pain at all.)

> motivation and choice than with technical engineering.

how well i know that!  like i said, the motivation is overwhelming.  an
alternative only needs to barely work before it will be seen as compelling.

that means i'll be cutting off tcp/25 while most of the world has only that
way, or fax and phones, to reach me.  thousands, no, tens of thousands of
other sites will do likewise.  the mega-exporters of batched interpersonal
communications traffic will be the last to change over, since their costs
are the highest, but they (hotmail, aol, etc) will be forced to endure these
costs when the "reachability base" dwindles.

in other words, by waiting until the barn is on fire and the horses are
dying, the quality and stability of our alternative, and the costs of getting
moved to it, have become almost irrelevant.  i guess we can all pat ourselves
on the back now, for letting things get this bad, before we acted.  (NOT.)

i am damned close to turning off tcp/25 even if the only alternative is
telco and fax and netnews and web.  THAT is how the pain:gain ratio is now.
SO, telling me that moving 100M users is hard, won't make me believe it.

--------

now as to the question keith and dave both asked: "why does smtp have to die?"

i guess it could be done on top of esmtp but we'll have to be able to
put sig(heloname,ipsrc,privatekey) in the HELO, sig(fromaddr,privatekey)
in the MAIL, and crypt(rcptaddr,publickey) in the RCPT.  the receiving
relay should be able to determine, at esmtp session time, whether the
owner of the heloname'd server knows what ip address it is using and
that it knows it is sending this mail and that there is transitive
trust from it to you.  similar restrictions on the MAIL fromaddr.
we'd want to sign the recipient name using its public key to indicate
that the sender has a way of learning/reaching this key.  and so on.

but at smtp session time, the smtp speaker must be able to determine a
shared value system between sender and recipient, which means the
recipient's values must be known to the smtp speaker.  things such as
"i only accept e-mail that is promised to be solicited or non-bulk"
have to be able to be matched up with promises of the form "i promise
that this mail is solicited" or "i promise that this mail is not bulk".
the combination of authentic promises and transitive trust is enough
to ensure that all communications is consensual, which is an explicit
goal now that the internet community no longer has this as an implicit
assumption (due to its originally small size and closed nature.)

so, whether we use esmtp as a bearer channel is indeed irrelevant, so
long as we'd be willing to stop trying to maintain a distinction between
an MTA and MUA, and perform recipient-specified functions at session time,
and so long as we can add attributes to all the existing protocol elements.
but those are HUGE elements of the (e)smtp model, and the resulting protocol
can in my opinion be called "new" even if it looks superficially like what
we do now.  there's absolutely no reason to use the same port number, for
example.  there's no fallback.  if newproto doesn't work because the firewall
or nat or whatever only allows tcp/25 sessions, then give up and use the
telephone or fax machine.

transitive trust and mailbox/server/relay certificates and chains of same
are the much harder issues, technically.  pgp and x.509 are both HARD for
the majority of mailbox owners (think of your parents of grandparents) to
use.  to make a new global-scale system, we have to make it marketable.

--------

now as to my own motives:

my reason for harping on the "stop trying to save smtp" issue in recent
days here is just my disbelief that baynesian or other filters, or lists
of who's been naughty and who's been nice, or anything like that, can be
bolted on "usefully".  that stuff is all crutches and bandaids while what's
needed is a whole new monster.

my reason for making "all communications must be consensual" as the key
design criteria for a new interpersonal batch communication system is that
it's what i've always believed, from the earliest days of the first RBL which
later became MAPS, and it's the single thing that's most failed in the (e)smtp
model as we've exploded the endpoint population to the hundreds of millions.

---

From: "Anthony Atkielski" <[email-masked]>
Date: Sun, 1 Jun 2003 00:09:07 +0200

I think it is totally impractical to talk about moving 100 million users to
anything.  These are not the pioneer days of computing; there are hundreds
of millions of people using computers today, and you aren't going to move
them all to anything.  It's rather like suggesting that North American
switch from 120 VAC to 480 VDC for home electrical power.  It's not going to
happen.  Look at QWERTY keyboards and NTSC if you believe otherwise.

---

From: Keith Moore <[email-masked]>
Date: Sat, 31 May 2003 18:09:44 -0400

> PV> once thinking that smtp would be the right answer for a global
> PV> population, or now thinking that smtp can be saved if we can just make
> PV> abuse expensive, is indicative of designers who think that the only
> PV> consumers are Just Like Them And Their Friends.  they were but aren't,
> PV> and they won't ever be again!
> 
> 
> is also unhelpful. 

as is casting the issue in terms of "those who are stuck in old-fashioned 
thinking" vs. those that aren't.

perhaps if Paul were to talk about the technical features that the system
needs to have, it would be obvious whether or not it's more feasible to
shoehorn this into SMTP or to transition to another system.   or at least,
we'd be able to evaluate his assertion.   but trying to discuss this in the
current mode is pointless.

---

From: "Christian de Larrinaga" <[email-masked]>
Date: Sat, 31 May 2003 23:16:49 +0100

Dave

> motivation and choice than with technical engineering.

I'm curious. Why not just have users send their mail directly if they wish?

Christian

---

From: Keith Moore <[email-masked]>
Date: Sat, 31 May 2003 18:43:35 -0400

> so, whether we use esmtp as a bearer channel is indeed irrelevant, 

okay, that's what I was thinking too.

but I also don't see any particular utility in authenticating either the
source IP address or the mail from address, and a fair amount of disadvantage
to both; I think we need a different token to authenticate the sender.
a lot of the problems with some of the existing proposals is that they try
to overload the existing from or mail from addresses too much.  even RFC 822
recognized that Sender is different than From is different than Return-Path.

> example.  there's no fallback.

I don't share your opinion that everyone will want to switch completely.
significant numbers of users will need fallback, at least for a transition 
period.  I would far prefer a solution that lets mail recievers turn a knob
that pessimizes handling of unauthenticated messages more and more (as more
and more legitimate senders start authenticating messages) than one which
forces receivers to maintain separate servers. (I'd also rather avoid the "how
do we design a new mail protocol" pandora's box - the amount of second-system
effect would be huge, and it would take years to sort it out)

even if we had to retro-fit something very different onto port 25,  it would
probably be more attractive to negotiate the new protocol within ESMTP than to
have a different protocol on a different port.

---

From: "Anthony Atkielski" <[email-masked]>
Date: Sun, 1 Jun 2003 00:53:11 +0200

Paul writes:

> the whole installed base is in incredible pain
> right now ...

Oooh ... let's not jump off the deep end here.  Spam is a nuisance for most
Internet users, not an "incredible pain."  It's very important to
distinguish between something that does real damage and something that
merely annoys.

> ... and there will be a lot of sites like mine who will
> switch over early, turning off their tcp/25 service, as
> soon as we think there are reasonable alternatives for
> reaching us.

Don't hold your breath.

> ... and the vast majority of non-mega-networks is feeling
> no ipv4 pain at all.

The same is true for spam.

> like i said, the motivation is overwhelming.

I don't think so.  Apart from a minority of power users, most people aren't
exactly fuming over spam.  It's a nuisance, like postal junk mail, but it's
not a preoccupation.

> i am damned close to turning off tcp/25 even if the
> only alternative is telco and fax and netnews and web.

Then perhaps you should do so.  It sounds like you're on the verge of a
heart attack right now.

> SO, telling me that moving 100M users is hard, won't

And not believing it won't make it any less true.  Think IMAP, encryption
and digital signatures (built in to OE for years now, never used by anyone),
PNG (around since forever, but still practically ignored), and so on.

> if newproto doesn't work because the firewall or nat
> or whatever only allows tcp/25 sessions, then give up
> and use the telephone or fax machine.

Wow!  You must have an unusual user community.  In some organizations, you'd
be fired for having an attitude like that.

> that stuff is all crutches and bandaids while what's

Nothing that depends on automation will work, because no automated system
can distinguish between spam and legitimate e-mail.  Any system will require
human intervention.  And human intervention is likely to be an obstacle to
implementation and compliance if human users see it as more trouble than it
is worth.  And human users who are not drowning in spam (and most aren't)
will be disinclined to favor any kind of human intervention at all.

> my reason for making "all communications must be
> consensual" as the key design criteria for a new
> interpersonal batch communication system is that
> it's what i've always believed ...

That's equivalent to having a screening device on your telephone that only
rings the phone if callers dial a certain number code.  This works for a
handful of telephone subscribers, but for most others, it is simply not
practical.  In most situations, you must be able to accepted unsolicited
communications, without requiring any kind of prior consent.

---

From: Paul Vixie <[email-masked]>
Date: Sun, 01 Jun 2003 02:14:07 +0000

> ... I also don't see any particular utility in authenticating either
> the source IP address or the mail from address, and a fair amount of
> disadvantage to both; I think we need a different token to
> authenticate the sender.

knowing the intent of the owner of the host is important.  we can't
tolerate any more hosts which become open proxies or open relays by
virtue of having swallowed an ms-outlook script-cookie pif-file.  if
a host hasn't made its intent to send e-mail known through a trusted
framework, then there's no reason for me to accept a tcp session from
it.  (credit goes to margie arbon of maps for proving this to me.)

> a lot of the problems with some of the existing proposals is that they
> try to overload the existing from or mail from addresses too much.
> even RFC 822 recognized that Sender is different than From is
> different than Return-Path.

the fact that any of the requirements which fall out of the new design
are incompatible with or inconvenient when viewed from an rfc821 or
rfc822 context should probably be viewed as neutral, or a feature.

using (e)smtp as a bearer channel for ibcs(*) might be expedient.  or not.

> I don't share your opinion that everyone will want to switch completely.

i know.  however, it doesn't matter.  you and a lot of people are free to
remain with the current system.  noone will stop you.  perhaps if you
become a small enough eyeball market, the spammers would leave you alone.

> significant numbers of users will need fallback, at least for a
> transition period.  I would far prefer a solution that lets mail
> recievers turn a knob that pessimizes handling of unauthenticated
> messages more and more (as more and more legitimate senders start
> authenticating messages) than one which forces receivers to maintain
> separate servers. (I'd also rather avoid the "how do we design a new
> mail protocol" pandora's box - the amount of second-system effect
> would be huge, and it would take years to sort it out)

i can imagine a fair number of mta authors deciding with you that they
ought to try ibcs first and then fall back to smtp if there is no _ibcs._tcp
SRV RR for the destination.  however, i expect that most remaining smtp
servers will be rejecting so damned much e-mail by that time that the
actual value of such fallback will be indistinguishable from nil.  (ymmv.)

> even if we had to retro-fit something very different onto port 25, it
> would probably be more attractive to negotiate the new protocol within
> ESMTP than to have a different protocol on a different port.

there's some small possibility of negotiating this on top of tcp/465,
but not tcp/25.  there's too much i need to know before i'll accept a
command, even a helo command, from an initiator.

---
(*) ibcs = interpersonnal batch communication system

--
Paul Vixie

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 31 May 2003 21:54:17 -0500

on 5/31/2003 5:43 PM Keith Moore wrote:

> but I also don't see any particular utility in authenticating either
> the source IP address or the mail from address,

One of the potential benefits of binding IP address to the host identity
would be enforcement. A spammer may be able to crank millions of
identities into thousands of domains, but they will only ever have a
handful of servers and that smaller subset would be significantly easier
to enforce against, at least until they move.

The principle benefit of binding sender identity is forgery prevention
while preserving mobility. Folks can argue that the optional mechanisms
for doing this today are adequate towards this, but as long as they are
optional they will be limited. There are also benefits for binding the
"return-path" identity at the transfer level, specifically for features
like header encryption, as well as for control messages (transfer headers
without message headers or a message body).

> disadvantage to both; I think we need a different token to authenticate
> the sender. a lot of the problems with some of the existing proposals
> is that they try to overload the existing from or mail from addresses
> too much.  even RFC 822 recognized that Sender is different than From
> is different than Return-Path.

Just as a point, the 822 separation between Sender and From has become
largely irrelevant in modern usage, and a modernized equivalent would
likely need to use a different metaphor (not that I want to argue over
what that metaphor should be right now, just pointing out that the 822
model isn't particular relevant anymore). I agree that there needs to be a
distinction between the ~entity who authored the message and the ~entity
who is responsible for the message being transferred. I also think they
need separate identitifiers when they are different.

> I don't share your opinion that everyone will want to switch
> completely. significant numbers of users will need fallback, at least
> for a transition period.

There will certainly be a need for SMTP for a long time to come. Lots of
sites won't be able to switch for some time to come, for whatever reasons,
and this will be especially true the further you get from the G8 world.
However, that doesn't mean that everybody will need to run SMTPng
gateways. There is also the ability to run SMTPng and SMTP as separate
transfer networks, with gateways in between them at critical points (such
as at ISPs).

For example, Hotmail and AOL exchange mail directly already, and they
could use SMTPng for these exchanges and immediately start to take
advantage of the available features, regardless of what they use to talk
to the rest of the world. Meanwhile, these same benefits would motivate
the telco ISPs to setup an SMTPng gateway that talked to those sites, and
these same features could motivate corporates and governments, and so
forth. As this spread outwards, the primary communication for the majority
of the site-to-site exchanges would move to the SMTPng network. Once the
smaller sites decided they wanted to participate in direct transfers, they
could just switch over with no loss in connectivity (except to places on
export-control lists, perhaps).

At some point it might even be possible to route legacy messages through
the SMTPng network somehow, for those remaining sites who haven't
upgraded. As a radical approach, ISPs could do this by making all DNS
lookups for MX RRs answer with their SMTP-SMTPng gateway address.

Anyway, my personal thinking is that we can probably reach a tipping point
with just a few thousand servers, if they are the right ones anyway, and
after that the upgrades can spread outwards independently. My guess is
that the 80/20 rule probably applies to email, in that 80% of the messages
are probably routed through 20% of the mail servers, and getting a sizable
portion of that 20% would likely be enough to capture most of the
messaging network.

BTW, this is the same reason why we don't need to worry about ~100 million
users. We only have to worry about the site-to-site external servers. We
don't have to worry about their internal servers.

> I would far prefer a solution that lets mail
> separate servers.

There are two reasons I have for not trying it. First of all, each change
request would be measured by the cost of that change in comparison to the
cost of modifying existing codebases. Several elements might be declared
as necessary but would still be rejected as unjustifiable. Some of the big
changes -- like splitting the transfer and message headers, which would
require new mailstore services for most systems -- would get tremendous
pushback, and might not even make the cut.

Secondarily, antyhing that did get through would be the proverbial square
peg, which would have to be encapsulated, converted, and bit-stuffed into
the proverbial round hole. All of this would make implementation much more
difficult, and would in turn make SMTP/821 more fragile.

My feeling is that the whole effort would be much more likely to succeed
if it were a from-scratch design. In fact, trying to do this inside SMTP
seems like a pretty good strategy for making sure that it either doesn't
happen at all, or that whatever gets through would be watered down and not
have been worth the effort.

> (I'd also rather avoid the "how do we design a new
> mail protocol" pandora's box - the amount of second-system effect would
> be huge, and it would take years to sort it out)

I competely agree with this sentiment, but that can be dealt with by
turning feature requests into architecture requests, and keeping the
features themselves out of the core specs. As an example, I want a
"recall" feature that lets me rescind an unread message from a recipient's
mailstore. That kind of feature has certain requirements (end-to-end
option encapsulation and verifiable identities), but that particular
feature should definitly not be in the spec. I can think of about 10-20
core architectural elements, but after that everything else probably
should be part of another spec.

---

From: [email-masked]
Date: Sat, 31 May 2003 23:42:32 -0400

> to enforce against, at least until they move.

We're starting to see the rise of zombie networks of trojaned machines to
send spam, similar to the DDoS networks.  Which makes Paul Vixie's comments
even more important: