Subject: Re: Last Call: draft-ietf-sasl-scram

From: Simon Josefsson <[email-masked]>
Date: Tue, 15 Sep 2009 10:55:32 +0200

I support publication of draft-ietf-sasl-scram.

I have implemented SCRAM in a beta version of GNU SASL [1], see [2], so
I have good confidence that the document is in sufficiently good
technical condition.

I have not yet managed to do interop testing with anyone else though.
Please contact me if you are implementing it and we haven't already
spoken about testing.

I have two remaining comments on the document:

1) This is an editorial issue.

The hashed password that needs to be stored with clients are either both
ClientKey and ServerKey OR only the SaltedPassword.  The SaltedPassword
can be used to compute ClientKey/ServerKey, so it may be preferable in
some situations where you only want to store one hash for the user.

OLD:
         client implementation MAY cache SaltedPassword/ClientKey for
NEW:
         client implementation MAY cache ClientKey/ServerKey (or just
         SaltedPassword) for

Also, clients may not need to have access to passwords if they have
access to the cached information.

OLD:
   To begin with, the SCRAM client is in possession of a username and
   password.
NEW:
   To begin with, the SCRAM client is in possession of a username and
   password (or a ClientKey/ServerKey or SaltedPassword).

2) This is a technical issue.

SCRAM does not say anything about the encoding used for the password
(e.g., Latin-1, UTF-8) or whether SASLprep should be applied to the
string or not.  I believe this violates this requirement in RFC 4422:

   In order to avoid interoperability problems due to differing
   normalizations, when a mechanism calls for character data (other than
   the authorization identity string) to be used as input to a
   cryptographic and/or comparison function, the specification MUST
   detail precisely how and where (client or server) the character data
   is to be prepared, including all normalizations, for input into the
   function to ensure proper operation.

   For simple user names and/or passwords in authentication credentials,
   SASLprep [RFC4013] (a profile of the StringPrep [RFC3454] preparation
   algorithm), SHOULD be specified as the preparation algorithm.

For highest interoperability the document could do MUST use UTF-8 and
MUST use SASLprep.  I understand that some people will not implement
SASLprep, and I agree with them that there are valid reasons for not
implementing SASLprep (such as it being based on obsolete Unicode
versions), so I would be equally happy with MUST use UTF-8 and SHOULD
use SASLprep.  I don't think the requirement on SASLprep can be relaxed
further without breaking the requirements in RFC 4422.  Personally, in
the long term I would prefer to deprecate SASLprep in favor of Net-UTF-8
(i.e., RFC 5198) for use in SASL applications.  I believe "SHOULD use
SASLprep" in SCRAM is a reasonable trade-off considering these factors.

For what its worth, my GNU SASL implementation supports SASLprep via GNU
Libidn [3] which implements SASLprep.  Libidn is free software, so
anyone is able to use it if they desire SASLprep support.

Thanks,
Simon

[1] http://www.gnu.org/software/gsasl/
[2] http://lists.gnu.org/archive/html/help-gsasl/2009-09/msg00003.html
[3] http://www.gnu.org/software/libidn/

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 15 Sep 2009 12:10:07 +0100

>I have two remaining comments on the document:
Hi Simon,
Thank you for the comments.

>   password (or a ClientKey/ServerKey or SaltedPassword).
I have no problems with these editorial changes.

>further without breaking the requirements in RFC 4422.
>
Yes.

>Personally, in
>(i.e., RFC 5198) for use in SASL applications.
>
Or we can revise SASLPrep.

>I believe "SHOULD use
Agreed.

>_______________________________________________
>sasl mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/sasl
>  
>

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 Sep 2009 09:41:47 -0400

--On Tuesday, September 15, 2009 10:55 +0200 Simon Josefsson
<[email-masked]> wrote:

>...
> SCRAM does not say anything about the encoding used for the
> password (e.g., Latin-1, UTF-8) or whether SASLprep should be
> applied to the string or not.  I believe this violates this
> requirement in RFC 4422:
> 
>    normalizations, when a mechanism calls for character data
> (other than    the authorization identity string) to be used
> as input to a    cryptographic and/or comparison function, the
> specification MUST    detail precisely how and where (client
> or server) the character data    is to be prepared, including
> all normalizations, for input into the    function to ensure
> proper operation.
> 
>    For simple user names and/or passwords in authentication
> credentials,    SASLprep [RFC4013] (a profile of the
> StringPrep [RFC3454] preparation    algorithm), SHOULD be
> specified as the preparation algorithm.
> 
> For highest interoperability the document could do MUST use
> UTF-8 and MUST use SASLprep.  I understand that some people
> will not implement SASLprep, and I agree with them that there
> are valid reasons for not implementing SASLprep (such as it
> being based on obsolete Unicode versions), so I would be
> equally happy with MUST use UTF-8 and SHOULD use SASLprep.  I
> don't think the requirement on SASLprep can be relaxed further
> without breaking the requirements in RFC 4422.  Personally, in
> the long term I would prefer to deprecate SASLprep in favor of
> Net-UTF-8 (i.e., RFC 5198) for use in SASL applications.  I
> believe "SHOULD use SASLprep" in SCRAM is a reasonable
> trade-off considering these factors.

For whatever it is worth, I agree with this analysis.  I'm not
sure that RFC 5198 is an adequate substitute for SASLprep, but
it is far better than unrestricted UTF-8 (which, IMO, we should
no longer be recommending in any protocol that requires
comparison of strings).  

Because of the unrestricted UTF-8 problem, and without taking a
position on deprecating SASLprep, my inclination would be to
strengthen Simon's proposed requirement a bit to "MUST use UTF-8
and SHOULD use SASLprep or at least Net-UTF-8" or its
equivalent.  

Alternately, I believe that any string that would successfully
come out of SASLprep would conform to Net-UTF-8, i.e., that the
set of valid SASLprep strings is a proper subset of the set of
valid Net-UTF-8 strings.   If that were true -- and someone with
significant Unicode normalization and SASLprep
knowledge/experience would need to verify it-- then "MUST use
Net-UTF-8 [RFC 5198] and SHOULD use SASLprep" would be an even
better formulation (perhaps with a note about that subset
relationship).

    john

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 15 Sep 2009 15:28:01 +0100

On Sep 15, 2009, at 2:41 PM, John C Klensin wrote:

> sure that RFC 5198 is an adequate substitute for SASLprep,

I am quite sure that RFC 5198 is not an adequate substitute for  
SASLprep as used in SCRAM to prepare usernames and passwords for  
(direct or indirect) comparison.  Net-UTF8 is not designed to support  
comparison of user names and passwords composed of Unicode characters,  
but for the transmission of text.

> but

I strongly oppose such an 'or' as SASLprep and Net-UTF-8 uses  
different Unicode normalization algorithms.  Such an or would lead to  
two classes of implementations (or modes of operation) that would not  
interoperate.

I believe the intent of the SASL WG was for SASLprep to be used here  
in accordance with RFC 4422.   If folks believe that wasn't the intent  
of SASL WG, or believe consensus of the SASL WG is unclear here, the I- 
D ought to be remanded to the WG for a consensus determination.

> better formulation (perhaps with a note about that subset

RFC 5198 says 'all character sequences SHOULD be normalized according  
to Unicode normalization form "NFC" (see Section 3).'
RFC 4013 says 'This profile specifies using Unicode normalization form  
KC, as described in Section 4 of [StringPrep].'

> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 15 Sep 2009 17:37:05 +0200

> no longer be recommending in any protocol that requires

Yes, agreed.

> and SHOULD use SASLprep or at least Net-UTF-8" or its

I share Kurt's concern that this opens up for several classes of
implementations that may not interoperate well.

> better formulation (perhaps with a note about that subset

Something along those lines may work, but I think there are serious
risks to get the details wrong with a quick fix like that here.  If
someone has time to study your question that would be useful input
nonetheless.

Pending something more baked than SASLprep and Net-UTF-8-in-SASL, I
continue to believe that "MUST UTF-8 and SHOULD SASLprep" is a
reasonable compromise.

Finally, I strongly believe that deferring publication of SCRAM
significantly to address this issue in a more thorough way does more
harm than good.  The state-of-the-art SASL mechanism today either
transfer passwords to servers without hashing them first, or they use
MD5 for the hash, and neither is acceptable.  If we don't publish SCRAM,
the SASL framework will be unusable for new protocols.

/Simon

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 15 Sep 2009 16:39:58 +0100

>John C Klensin <[email-masked]> writes:
 [...]

>implementations that may not interoperate well.
+1.

>continue to believe that "MUST UTF-8 and SHOULD SASLprep" is a
+1.

>harm than good.
I agree.

>The state-of-the-art SASL mechanism today either

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 15 Sep 2009 09:47:41 -0600

> the SASL framework will be unusable for new protocols.

I agree on all counts. We would dearly like to specify SCRAM as MTI in
draft-ietf-xmpp-3920bis (to replace DIGEST-MD5 from RFC 3920). And this
time we'd like truly interoperable implementations, which we have a hope
of with "MUST UTF-8 and SHOULD SASLprep" but not with "MUST Net-UTF-8
and SHOULD SASLprep".

Peter

- --
Peter Saint-Andre
https://stpeter.im/

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 15 Sep 2009 10:08:17 -0500

Nico
--

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 Sep 2009 12:16:44 -0400

Kurt,

Just for clarification...  I fear that parts of this note are
going to be a mini-tutorial on some Unicode subtleties, but one
can't understand this without them and I suspect some interested
readers don't have that level of understanding. 

--On Tuesday, September 15, 2009 15:28 +0100 Kurt Zeilenga
<[email-masked]> wrote:

> different Unicode normalization algorithms.

Well, not really.

>...
> RFC 5198 says 'all character sequences SHOULD be normalized
> according to Unicode normalization form "NFC" (see Section 3).'
> RFC 4013 says 'This profile specifies using Unicode
> normalization form KC, as described in Section 4 of
> [StringPrep].'

First, I know that you know this but to be sure no one is
confused by a slight terminology difference, "normalization form
KC" and "normalization form NFKC" are exactly the same thing.
The latter is a little redundant, but commonly used.

Now, NFKC processing is a proper superset of NFC processing.
NFC provides what is called, in Unicode-speak, "canonical
composition" -- turning different ways of expressing exactly the
same character into a single standard form.   For example,
applying toNFC to Latin Small Letter U (U+0075) followed by
Combining Diaeresis (U+0308) yields Latin Small Letter U with
Diaeresis (U+00FC), while applying it to U+00FC yields U+00FC
itself.

Without NFC (or NFD, but that is another topic), simple string
comparisons may fail depending on how a character is entered at
the keyboard.   That is generally a bad idea.  Unless one
permits embedded new line characters in one's "character
sequences", the main difference between "just UTF-8" (RFC 3629)
and RFC 5198 is that the latter requires NFC-compliant strings.
3629 doesn't require NFC, much less NFKC (see below).

NFKC is a more complex operation, combining canonical
composition with "compatibility composition" -- replacement of
characters that Unicode has identified as being part of the
standard for compatibility purposes with their base forms.
There are a wide variety of compatibility characters.  Some,
such as East Asian width variations, are as surely "the same
character" as the U-with-Diaeresis example above.  Others are
the same (or not) only in context.  For example, there are a
large number of "mathematical" letter characters that, if used
in non-mathematical running text are simply font variations
(consider the relationship between Mathematical Bold Script
Small A (U+1D4EA) and Latin Small Letter A (U+0061)) but, if
used in mathematical contexts are fundamentally different
characters, at least according to several mathematical societies
and publishers.  Apply toNFKC to U+1D4EA and you get U+0061, but
applying toNFC to the same character yields itself.   And still
others are much more different.

However, any string in NFKC form is, by definition, in NFC form.

Now, with the understanding that this is a comment about
SASLprep rather than about the current I-D, but that it may be
part of Simon's motivation and certainly is part of mine, it is
really unclear whether applying the NFKC transformation to
things like identifiers and passphrases in security contexts is
a good idea.  If I know that I'm going to be in environments in
which I know how to type U+1D4EA and know that it can be
processed appropriately, it is a nearly ideal element of a
string used as a security identifier or passphrase: it, and its
many relatives, vastly increase the repertoire of characters
available to me and hence the potential entropy in such a
string, an attacker doing shoulder-surfing may not be able to
identify it or figure out how to type it in, and so on.

But the bottom line is that there is a pretty strict hierarchy
in terms of the number of permitted characters and their
representational forms:

	  UTF-8/ RFC 3629  (any Unicode code point, often with
	the same character able to be represented in different
	ways)
	
	  Net-UTF-8 / RFC 5198 (NFC-compliant strings; different
	code sequences for exactly the same character are
	eliminated; otherwise the same as UTF-8
	
	  SASLprep / RFC 4013 (NFKC-compliant strings; all
	"compatibility characters" are eliminated by being
	mapped into their base forms; otherwise the same as
	Net-UTF-8.

I think that means that...

(1) If you want to maximize interoperability, possibly at the
expense of some implementations getting things wrong as I
understood Simon to be concerned about, the rule should be
MUST... SASLprep.  Period, no exceptions.

(2) If you want to have reasonable odds of implementations that
do not support/use SASLprep working, the best answer is 
   MUST... Net-UTF-8, SHOULD SASLprep
or, if you prefer
   MUST ... NFC, SHOULD SASLprep
which, in a SASL context, will be indistinguishable in practice. 

(3) If you think even that strong a constraint is hopeless and
want to say something, then what should be said is
   MUST ... UTF8, SHOULD SASLprep but, if not SASLprep, SHOULD
NFC

I really don't think (3) is a good idea, but an unqualified
   MUST ... UTF8, SHOULD SASLprep
strikes me as a terrible idea simply because the same character,
coded in different ways through no fault of the user, may not
compare equal.  The difference between (1) and (2) is less
significant in practice because, while there are many important
exceptions (with those East Asian width variants probably
heading the list), the vast majority of compatibility characters
are very hard to type in most environments. And that was really
the point I was trying to make.

     john

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Tue, 15 Sep 2009 17:33:46 -0400

> the point I was trying to make.

There's another important point to make here...

For something like a username, or passwords in PLAIN, the server has the 
original value, and can compensate for a client which under-normalizes. 
For such fields, an entirely appropriate policy would be "MUST UTF-8; 
server SHOULD SASLprep, or at least NFC; clients... MAY SASLprep, or NFC, 
or nothing".

But in SCRAM, passwords are not sent to the server.  They are used to 
derive cryptographic keying material by way of a one-way hash function. 
This means that the client and the server (or, whatever handled the 
password setting function for the server) MUST use _the same_ normalization 
operation, or they will fail to interoperate.  Further, the failure will be 
subtle -- some clients will fail to work with some servers, depending on 
what characters are in the user's password.

Do we really want to create a situation where, to debug an interoperability 
problem, a system administrator must understand Unicode normalization _and_ 
ask the user to reveal his password?

-- Jeff

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Tue, 15 Sep 2009 17:39:50 -0400

--On Tuesday, September 15, 2009 02:55:54 PM -0500 Nicolas Williams 
<[email-masked]> wrote:

> I think the right answer is to leave _query_ strings unnormalized and
> require that _storage_ strings be normalized (see my separate reply on
> that general topic, with a different Subject:, just now).

Or at least, leave query strings unnormalized until just before the query 
happens, and then normalize them in the same way as the storage string.

More generally, what you want is normalization-insensitive comparison, and 
normalization of storage strings when they are stored is just an 
optimization for that.

Except in cases like SCRAM, where strings are used to derive cryptographic 
keys or in other ways where it matters whether they're the same, but you 
don't get to compare them directly.  Then you need to insure that the same 
input string is always transformed in the same way, or things break. 
Unfortunately, for SCRAM passwords, it's the client that has to do that 
transformation on every transaction, so we must insure that all clients do 
so in the same way.

-- Jeff

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 Sep 2009 22:17:30 -0400

--On Tuesday, September 15, 2009 14:43 -0500 Nicolas Williams
<[email-masked]> wrote:

>...
> [OT for the draft-ietf-sasl-scram thread, but possibly of
> interest to the IETF list.]
> 
> NFSv4 left normalization form unspecified for filenames.  We
> ended up implementing normalization-insensitive and
> normalization-preserving behavior in ZFS in Solaris.  The
>...
> The lesson is, IMO, that in the general case I think we can
> get way with not specifying normalization forms for _query_
> strings, but not for _storage_ strings.

FWIW, I have gradually come to believe that
normalization-insensitive comparisons (i.e., strings compare
equal if their normalized forms are equal) are almost always a
better idea that remapping strings by normalization, especially
when the normalization method involved is NFKC or NFKD.   The
same comment, discussed at some length in The Unicode Standard,
applies to case comparisons, especially if people believe that
toCaseFold is necessary... reasonable comparison operation,
pretty bad mapping one.

As has been suggested in other notes, normalization of storage
strings is just an optimization of that comparison principle.

But, as has also been pointed out, that principle is useless for
Scram, where the hash must be computed client-side across the
same string the server is going to use (or has used) to compute
its hash.  It is equally useless for IDNA, at least until and
unless we "teach" the DNS about normalization-insensitive
comparisons of non-ASCII strings.

    john

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 15 Sep 2009 10:55:46 -0500

> On Sep 15, 2009, at 2:41 PM, John C Klensin wrote:

Strangely I don't have John's e-mail; I have only the quoted text to go
on.

> but for the transmission of text.

+1

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 15 Sep 2009 14:43:15 -0500

> no longer be recommending in any protocol that requires

[OT for the draft-ietf-sasl-scram thread, but possibly of interest to the
IETF list.]

NFSv4 left normalization form unspecified for filenames.  We ended up
implementing normalization-insensitive and normalization-preserving
behavior in ZFS in Solaris.  The normalization-insensitive part and
high-performance normalization code was the relatively easy part.  The
normalization-preserving part was non-trivial, or would be/have been for
filesystems that don't/didn't already hash directory contents (as ZFS
did).  (ZFS uses locally normalized file names as input to the hash
function, but stores application-/remote fs protocol-provided file names
unnormalized in the directory hash entries.)

The lesson is, IMO, that in the general case I think we can get way with
not specifying normalization forms for _query_ strings, but not for
_storage_ strings.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 15 Sep 2009 14:55:54 -0500

> > different Unicode normalization algorithms.

Really :)

> Now, NFKC processing is a proper superset of NFC processing.

An implementor that stores NFC strings will not interoperate with any
peer that sends query strings in NFKC.  That's because a peer could send
a query string that doesn't match any storage string without additional
normalization of the storage strings!

(Nodes that store strings have to have enough normalization code to
validate the normalization of query strings, if query strings are
required to be normalized.  Expecting implementors to normalize query
strings is not that big a deal.  Peers that send query strings will
typically need to be able to normalize too, for local reasons, but
there's no obvious reason why there must be such local reasons.)

Then the choice of normalization form for storage strings only affects
peers that read them back -- which is enough to justify requiring the
use of a normalization form for storage strings.  The choice of K or not
K then can be conceivably left to the implementor (provided peers are
required to support non-K when reading back storage strings).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 15 Sep 2009 16:56:01 -0500

> normalization of storage strings when they are stored is just an 

Indeed.

I think that normalization-on-create is more likely to result in better
interoperability as all implementors ensure that their displays can
handle a particular NF.  I say that as an advocate of normalization-
insensitive-but-preserving behavior in ZFS, but that was mostly a result
of realities on the ground (one NFS implementation normalizes to NFD on
create, most others don't normalize but their input methods tend to
generate NFC, and not all implementors properly display unnormalized
text nor text in all NFs).  For most apps, I think normalizing storage
strings is the better approach.

> transformation on every transaction, so we must insure that all clients do 

Right.  I didn't mean to imply that the client should not normalize
passwords; I should have mentioned that.

---

From: <[email-masked]>
Date: Tue, 22 Sep 2009 11:43:44 +0200

John C Klensin wrote:

> The difference between (1) and (2) is less significant in practice
> because, while there are many important exceptions (with those East
> Asian width variants probably heading the list), the vast majority
> of compatibility characters are very hard to type in most
> environments. And that was really the point I was trying to make.

Adding one data point here: While I have no idea how to type East
Asian width variants on my keyboard (normal Finnish/Swedish layout),
my keyboard does have three characters where NFC!=NFKC (so using any
of them in my password would be impossible if some SCRAM implementations 
use NFKC and some NFC):

Vulgar Fraction One Half (U+00BD)
Acute Accent (U+00B4)
Diaeresis (U+00A8)

Looking http://en.wikipedia.org/wiki/Keyboard_layout, it seems the
Finnish/Swedish layout is not special in any way, and many other
European keyboards would also have some small number of characters 
where NFC!=NFKC.

Best regards,
Pasi

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 Sep 2009 10:34:22 -0400

--On Tuesday, September 22, 2009 11:43 +0200
[email-masked] wrote:

> John C Klensin wrote:
> 
>> The difference between (1) and (2) is less significant in
>> practice because, while there are many important exceptions
>> (with those East Asian width variants probably heading the
>> list), the vast majority of compatibility characters are very
>> hard to type in most environments. And that was really the
>> point I was trying to make.
> 
> Adding one data point here: While I have no idea how to type
> East Asian width variants on my keyboard (normal
> Finnish/Swedish layout), my keyboard does have three
> characters where NFC!=NFKC (so using any of them in my
> password would be impossible if some SCRAM implementations 
> use NFKC and some NFC):
> 
> Looking http://en.wikipedia.org/wiki/Keyboard_layout, it seems
> the Finnish/Swedish layout is not special in any way, and many
> other European keyboards would also have some small number of
> characters  where NFC!=NFKC.

That is important data.  It seems to me that it implies:

	* if entropy in passwords and/or properly reflecting
	keyboards is more important than password
	interoperability (whatever that means), then we should
	be moving away from NFKC and, hence, from the current
	version of SASLprep.
	
	* if entropy in passwords is less important than
	interoperability with any Latin-based (or
	Latin-character-containing) keyboard one happens to
	encounter, then NFKC should be mandatory.  However, one
	needs to think about how far to carry that argument
	because, if it is taken very far, there is a strong case
	for restricting passwords to the basic, undecorated,
	Latin letters that appear on all such keyboards.  

I have no idea what this would mean for keyboards that don't
contain any Latin-based characters at all, which are the cases
I'm mostly been using when I try to think through these issues.

And, of course, it would be possible to decide that we are stuck
with the decisions made in SASLprep.  If so, it pretty strongly
suggests to me that we had better get a lot more careful and
conservative about whatever coding decisions we make in the
future.

best,
   john

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 22 Sep 2009 17:17:39 +0200

> 	be moving away from NFKC and, hence, from the current

I believe NFKC is sub-optimal for password processing.  It reduces
entropy for many non-ambigious characters.  For example NFKC('ª') = 'a'
which seems like a clear example of an unwanted conversion.

> 	Latin letters that appear on all such keyboards.  

I don't believe there is a good case for this.  Improving entropy in
passwords is important.  There shouldn't be any _technical_ reason in
authentication protocols why users cannot use 'ª' in a password to
provide more entropy to the system than using 'a'.

There are many environments where non-ascii characters are a bad idea
from technical or social reasons, but those environments should not
restrict less limited environments.  It is fine for a system to validate
passwords against [A-Za-z0-9] if that is required, and that system will
be able to use SCRAM too.

> conservative about whatever coding decisions we make in the

For SCRAM I believe we are stuck with SASLprep because there are no
drafts to provide a replacement that are close to being mature.

/Simon

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 Sep 2009 11:36:53 -0400

--On Tuesday, September 22, 2009 17:17 +0200 Simon Josefsson
<[email-masked]> wrote:

> I believe NFKC is sub-optimal for password processing.  It
> reduces entropy for many non-ambigious characters.  For
> example NFKC('ª') = 'a' which seems like a clear example of
> an unwanted conversion.

That (and your other comments about entropy in passwords) is
consistent with my personal belief as well, but I was trying to
present the choice is as balanced a way as possible.

>...
> For SCRAM I believe we are stuck with SASLprep because there
> are no drafts to provide a replacement that are close to being
> mature.

Here we disagree _very_ slightly.  Following part of the theme
of draft-iab-idn-encoding, I have become convinced that having
many Unicode profiles is a significant cost and involves
significant disadvantages.   Replacements for Stringprep
profiles, or updates to Stringprep itself, should be, IMO,
required to justify the costs and complexity they represent
relative to "just use NFC" or the nearly-equivalent "just use
RFC 5198".  In this case, if maturity of alternate
specifications were the issue, one could base SCRAM on NFC alone
(which is _very_ mature) if one wanted to start moving away from
per-protocol profiles.  I really don't know if that is the right
choice at this stage, but it certainly is a feasible one.

     john

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 22 Sep 2009 17:58:16 +0200

John C Klensin <[email-masked]> writes:

> significant disadvantages.

I agree.

> Replacements for Stringprep profiles, or updates to Stringprep itself,
> should be, IMO, required to justify the costs and complexity they
> represent relative to "just use NFC" or the nearly-equivalent "just
> use RFC 5198".  In this case, if maturity of alternate specifications
> were the issue, one could base SCRAM on NFC alone (which is _very_
> mature) if one wanted to start moving away from per-protocol profiles.
> I really don't know if that is the right choice at this stage, but it
> certainly is a feasible one.

I don't think using NFC is feasible at this point.  Noone (or?) have
evaluated whether NFC alone is a good chose for preparing passwords.  So
in that way, NFC is not at all mature.  RFC 4422 suggests use of
SASLprep.  SASLprep is mature in that it exists and have been evaluated
for appropriateness.  Just because SASLprep is not perfect doesn't mean
something else, NFC included, will not be worse.

Personally (speaking as one of few SASLprep implementers) I believe
using NFC alone would be better from many perspectives than SASLprep for
passwords.  But I can't point to any substantial document to support
that belief, and there are obvious disadvantages with the NFC-approach
(less stability because of versioning differences) that would need to be
addressed.  Given that SCRAM is in last call now, I'm not sure it is
feasible to develop a document that analyze NFC from this perspective
that we can have good confidence in and gain wide support for.

I'd be happy to help work on a document that analyzed the consequences
of replacing SASLprep with just-use-RFC5198 in SASL.  But I don't think
SCRAM should wait for something like it to materialize.

Finally a general observation.  I believe username and passwords are
different beasts when it comes to string preparation.  What makes sense
for usernames does not always make sense for passwords, and vice versa.
Usernames are typically transported in the clear, and thus it makes
little sense to enforce strong normalization like NFKC on it.  What may
be useful is to enforce weaker rules, like NFC, when comparing two
username strings for equivalence.  Passwords should not be transported
in the clear, and are often input to hash functions, and thus it is
motivated to require normalization.  I'm not convinced NFC is sufficient
here.  I think conflating username string preparation with password
string preparation is one problematic part of SASLprep.

/Simon

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 Sep 2009 12:07:17 -0400

--On Tuesday, September 22, 2009 17:58 +0200 Simon Josefsson
<[email-masked]> wrote:

>...
> Personally (speaking as one of few SASLprep implementers) I
> believe using NFC alone would be better from many perspectives
> than SASLprep for passwords.  But I can't point to any
> substantial document to support that belief, and there are
> obvious disadvantages with the NFC-approach (less stability
> because of versioning differences) that would need to be
> addressed.  Given that SCRAM is in last call now, I'm not sure
> it is feasible to develop a document that analyze NFC from
> this perspective that we can have good confidence in and gain
> wide support for.
> 
> I'd be happy to help work on a document that analyzed the
> consequences of replacing SASLprep with just-use-RFC5198 in
> SASL.  But I don't think SCRAM should wait for something like
> it to materialize.

Now that the issue has been raised and is presumably better
understood, I'm happy to leave that decision to the WG.

>...
> Finally a general observation.  I believe username and
> passwords are different beasts when it comes to string
> preparation.  What makes sense for usernames does not always
> make sense for passwords, and vice versa. Usernames are
> typically transported in the clear, and thus it makes little
> sense to enforce strong normalization like NFKC on it.  What
> may be useful is to enforce weaker rules, like NFC, when
> comparing two username strings for equivalence.  Passwords
> should not be transported in the clear, and are often input to
> hash functions, and thus it is motivated to require
> normalization.  I'm not convinced NFC is sufficient here.  I
> think conflating username string preparation with password

Agreed.

    john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 22 Sep 2009 19:29:58 +0200

IMO, this is a close relative of a different problem, one that's old and 
well-understood: Characters that shift to different keys when you cross 
a boundary.

I (now) live in Germany and come from Norway. Germany has Y and Z 
swapped. Shortly after I started travelling to Germany, I stopped using 
Y and Z in passwords. They were too much trouble. This is (at least 
among the people I know) the common solution.

> I may well be making a silly mistake, but my gut says that the 
> compatibility mappings will not have a serious enough impact on 
> password entropy that we must make an effort to migrate from 
> SASLprep.

I agree, because I think that if a character doesn't have a reliable, 
unchanging representation, then using that character in a password 
today is begging for trouble. Can't be typed on the wrong keyboard/OK, 
can't be transmitted through a program that happens to normalize the 
right/wrong way, etc.

Arnt

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 22 Sep 2009 11:42:43 -0500

> I'm mostly been using when I try to think through these issues.

My keyboard is a U.S.-type keyboard.  I regularly type Latin characters
with it by using compose key sequences.

The bigger concern is typically going to be: can the user cause a system
to understand compose sequences easily enough, particularly at a login
screen, and are those sequences portable enough.  But I don't think that
should stop us from moving away from US-ASCII :)

The compatibility mappings are few and far between.  The biggest
entropy-related impact of them is going to be on UIs for setting new
passwords.  How do you explain to a user password contents rules that
take Unicode compatibility mappings into account?

I may well be making a silly mistake, but my gut says that the
compatibility mappings will not have a serious enough impact on password
entropy that we must make an effort to migrate from SASLprep.

Incidentally, I did look at all the Latin character lists before forming
that gut feeling.

OTOH, for SCRAM, _now_ is the time to make this change if we'll make it
at all.  So if you have data[*] to prove my gut wrong, or if your guts
say mine is wrong, then let's look at this issue seriously now, not
later.

To be rigorous we'd put SCRAM on hold, gather the data we need to decide
whether SASLprep's use of NFKD has a severe enough impact on password
entropy, fix the problem if need be, then unblock SCRAM.  But I think
it's more important to progress SCRAM.

[*] E.g., frequency of characters that have compatibility mappings, and
    of the those mappings, in texts in various languages.  From that we
    might judge how likely-passwords are to be affected by compatibility
    mappings, and their statistical impact on password entropy.

Nico
--

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 23 Sep 2009 19:54:56 +0200

I have noticed an additional problem related to additional data in
SCRAM.  RFC 4422 section 5 item 2b says:

      b) An indication of whether the server is expected to provide
         additional data when indicating a successful outcome.  If so,
         if the server sends the additional data as a challenge, the
         specification MUST indicate that the response to this challenge
         is an empty response.

As far as I can tell, SCRAM does not specify that the response to a
server-final sent as a challenge MUST be an empty client response.  This
violates the requirements in RFC 4422 for new mechanisms.

Review section 3 of RFC 4422 before reading on.  SCRAM negotiations in
general will look like:

      C: Request authentication exchange
      S: Empty Challenge
      C: SCRAM client-first
      S: SCRAM server-first
      C: SCRAM client-final
      S: SCRAM server-final
      C: Empty Response
      S: Outcome of authentication exchange

(Four round-trips, ouch!)

If the application protocol supports additional data together with
outcome of authentication exchange, the negotiation will look like:

      C: Request authentication exchange
      S: Empty Challenge
      C: SCRAM client-first
      S: SCRAM server-first
      C: SCRAM client-final
      S: Outcome of authentication exchange with SCRAM server-final

If the application protocol supports optional initial responses, the
negotiation will look like:

      C: Request authentication exchange with SCRAM client-first
      S: SCRAM server-first
      C: SCRAM client-final
      S: SCRAM server-final
      C: Empty Response
      S: Outcome of authentication exchange

If the application protocol supports both additional data together with
outcome of authentication exchange AND optional initial response, the
negotiation will look like:

      C: Request authentication exchange with SCRAM client-first
      S: SCRAM server-first
      C: SCRAM client-final
      S: Outcome of authentication exchange with SCRAM server-final

I believe section 5 needs to be rewritten to take all these variants
into account.  Right now the wordings all assume the last situation.

OLD:
   First, the client sends the "client-first-message" containing:

NEW:
   If the application protocol does not support optional initial
   responses, the client will request authentication and the first
   server challenge MUST be empty.  The first non-empty client response
   is the "client-first-message" containing:

OLD:
   The server verifies the nonce and the proof, verifies that the
   authorization identity (if supplied by the client in the first
   message) is authorized to act as the authentication identity, and,
   finally, it responds with a "server-final-message", concluding the
   authentication exchange.

NEW:
   The server verifies the nonce and the proof and that the
   authorization identity (if supplied by the client in the first
   message) is authorized to act as the authentication identity.  If the
   application protocol supports sending outcome of the authentication
   exchange, it sends the outcome together with the
   "server-final-message", concluding the exchange.  Otherwise, the
   server sends the "server-final-message" as a request.

OLD:
   The client then authenticates the server by computing the
   ServerSignature and comparing it to the value sent by the server.  If
   the two are different, the client MUST consider the authentication
   exchange to be unsuccessful and it might have to drop the connection.

NEW:
   The client then authenticates the server by computing the
   ServerSignature and comparing it to the value sent by the server.  If
   the two are different, the client MUST consider the authentication
   exchange to be unsuccessful and it might have to drop the connection.
   If the application protocol does not support sending additional
   data together with the outcome of authentication, the client MUST
   respond to the server request with a empty response.

Note that this problem have consequences for my implementation: the
earlier SCRAM traces I posted are incorrect since they do not have a
final empty client response.

/Simon

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 23 Sep 2009 14:18:07 -0500

> > are no drafts to provide a replacement that are close to being

I know this horse is really dead by now, but...

> RFC 5198".  In this case, if maturity of alternate

Speaking from experience, stringprep profiles variations in
normalization form, prohibited output, and so on are easy to implement
as parameters of a stringprep implementation, provided one has
implementations of all normalization forms.  I wouldn't worry too much
about the cost of a new stringprep profile.  The _content_ of stringprep
profiles is what matters.

> choice at this stage, but it certainly is a feasible one.

I don't see your point: NFKC is as mature as NFC.

I agree that we could consider a change in stringprep profile for SCRAM,
but the process implications make that undesirable, and the cost of
sticking with SASLprep is almost certainly negligible.  No, I've not
done any statisitical analysis on the impact of compatibility mappings
on the entropy of likely passwords, and I'm not going to as I think we
can agree that their impact is very likely minimal.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 23 Sep 2009 14:26:08 -0500

> violates the requirements in RFC 4422 for new mechanisms.

I'm not sure that not saying this violates RFC4422: one could argue that
this is implied, and it'd be better RFC4422 had been written in such a
way that we needn't repeat this over and over in all mechanism
specifications.  But I don't mind new text to cover this.

> (Four round-trips, ouch!)

Blame SASL, or, rather, SASL application protocols for two of those :)

And, of course, you're missing the mechanism negotiation round-trip
before that:

	C: List server SASL mechanisms request
	S: Server SASL mechanism list response

>    is the "client-first-message" containing:

I don't think this is necessary.  Instead I think we can add text saying
that we're describing the full, uncompressed exchange, and that nothing
in SCRAM prevents either sending the client-first message with the
authentication request, nor sending the server-final message as
additional data of the outcome of authentication message + a redundant
re-statement of the rule from RFC4422 section 5, item 2(b).

For my money the result will be more readable this way.

Nico
--

---

From: <[email-masked]>
Date: Fri, 25 Sep 2009 13:24:55 +0200

> 	be moving away from NFKC and, hence, from the current

I don't know about the East Asian width variants, but for the ones in the
Finnish/Swedish layout, there is basically no entropy loss.  For some
of the characters, there's only one way to enter the NFKC form (so no
entropy is lost); and the number of characters affected is small, and
they're rarely used anyway (so the effect on entropy is extremely small).

So IMHO entropy is not a good reason to move away from NFKC.

There might be other reasons, but the complaint about SASLprep I've
heard most often (implementation complexity -- unless the platform
already has a normalize() call always available, many programmers will
"just use UTF-8") applies equally to NFC, too. So I'm not sure if
moving to NFC would really solve anything here...

But "just use UTF-8" probably won't lead to good interoperability
when the passwords are hashed (as opposed to sent and compared, like
usernames).
 
Best regards,
Pasi

---

From: <[email-masked]>
Date: Fri, 25 Sep 2009 13:32:47 +0200

> SCRAM should wait for something like it to materialize.

I agree that such work would take time, and we don't want to delay
SCRAM.

But as the discussion so far has shown, normalization is a very tricky
topic, and we can't really expect implementors to understand why "just
use UTF-8" is problematic. Perhaps we should add a note to the SCRAM
draft; something like
 
   Informative Note: Implementors are encouraged to create test cases
   that use both username passwords with non-ASCII characters. In
   particular, it's useful to test characters whose "normalization
   form C" and "normalization form KC" are different. Some examples of
   such characters include Vulgar Fraction One Half (U+00BD) and
   Acute Accent (U+00B4).

Do you think this would increase the likelihood of interoperability
with non-ASCII passwords?

Best regards,
Pasi

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 25 Sep 2009 13:45:57 +0200

+1.

> with non-ASCII passwords?

For implementers that decides to use SASLprep but just happens to get
things wrong, yes.  For those, I think test vectors would be even more
useful.

/Simon

---

From: <[email-masked]>
Date: Fri, 25 Sep 2009 13:54:54 +0200

Simon Josefsson wrote:

> >    form C" and "normalization form KC" are different. Some examples
> >    of such characters include Vulgar Fraction One Half (U+00BD) and
> >    Acute Accent (U+00B4).
> 
> +1.
> 

I was also hoping the tests would convince implementors to actually do
SASLprep :-) Otherwise, they might test their implementation with a
couple of non-ASCII passwords, and conclude that "just use UTF-8"
interoperates just fine (with someone else's code).

(If you just pick some non-ASCII password for your test case, it's
quite likely that its NFC and NFKC are equivalent, and the input
to your code is already NFC...)

Best regards,
Pasi

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 25 Sep 2009 14:00:58 +0200

> they're rarely used anyway (so the effect on entropy is extremely small).

Latin-1 0xAA ('ª') and 0xBD ('½') are easy to type on Finnish/Swedish
keyboards (AltGr-A and Shift-§) and the NFC and NFKC forms differs.

Agreed.

> when the passwords are hashed (as opposed to sent and compared, like

I'm hesitant to bring this up because it has so many other concerns, but
if you are looking for alternatives, another one is to flag the
normalization algorithm used in the protocol.  E.g., add a flag
'c=saslprep' or 'c=net-utf-8' or 'c=utf-8'.  This makes it possible to
apply a better heuristic on the server side.  Or treat normalization
like the hash algorithm, since it is also an continuously evolving and
apparently never-perfected technology, and make the mechanism name
SCRAM-SHA-1-SASLPREP or SCRAM-SHA-1-NET-UTF-8.  (You can figure out the
problems with this approach as good as I can, so I won't go into them..)

/Simon

---

From: Simon Josefsson <[email-masked]>
Date: Sat, 26 Sep 2009 10:39:25 +0200

Nicolas Williams <[email-masked]> writes:

> On Fri, Sep 25, 2009 at 02:00:58PM +0200, Simon Josefsson wrote:
> It doesn't really help because it'd have to be the server telling the
> client what the user's password's form is -- not the other way around.
> Chances are the password's been hashed already; recovering from use of a
> different NF (or just-utf-8) is not going to be feasible.

The server can store the password hashed in a couple of different forms,
and use the flag to determine which to use.  I realize that is possible
anyway (just iterate through all locally stored hashes), although
without some text in the document I don't think many servers will
implement that.

/Simon

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 25 Sep 2009 11:40:06 -0500

---

From: Nicolas Williams <[email-masked]>
Date: Sat, 26 Sep 2009 17:15:36 -0500

> without some text in the document I don't think many servers will

Yes, if we have a small number of stringprep profiles to use (which we
arguably do).  I'll think about whether this is worth it or troublesome.