Subject: Discussion of draft-hardie-advance-mechanics-00.txt

From: Ted Hardie <[email-masked]>
Date: Thu, 16 Sep 2010 16:25:58 -0700

Howdy,

The attached draft is part of the discussion Russ started up
with draft-housley-two-maturity-levels.  It is compatible with,
but does not require a reduction in maturity levels.  If you
would like to send me comments privately, I welcome them;
I suppose further public discussion would take place here.

regards,

Ted Hardie

---------- Forwarded message ----------
From:  <[email-masked]>
Date: Thu, Sep 16, 2010 at 2:00 PM
Subject: I-D Action:draft-hardie-advance-mechanics-00.txt
To: [email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts directories.

       Title           : Proposed mechanics for document advancement
       Author(s)       : T. Hardie
       Filename        : draft-hardie-advance-mechanics-00.txt
       Pages           : 6
       Date            : 2010-09-16

The IETF has maintained a three-stage standards track for some time,
with basic mechanics set out in RFC 2026 [RFC2026].  The mechanics as
described, however, are not currently used in practice and
advancement along the standards track is both unusual and more
difficult than many IETF participants find reasonable.  This document
proposes an update to those mechanics, in the interest of improving
the overall functioning of the IETF.  It is compatible with, but does
not require, a move to a two-stage standards track.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-hardie-advance-mechanics-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 16 Sep 2010 17:34:55 -0700

The current process involves a (weak) proof of interoperability to advance; interoperability is not even mentioned in this draft. Is that rather significant change intentional? Or did you want negative interoperability reports ("Vendor A is doing it wrong, so the spec must be unclear or have features that are unwanted") to considered objections?

--Paul Hoffman, Director
--VPN Consortium

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 17 Sep 2010 08:40:47 +0800

> The current process involves a (weak) proof of interoperability to
> advance; interoperability is not even mentioned in this draft. Is that
> rather significant change intentional? Or did you want negative
> interoperability reports ("Vendor A is doing it wrong, so the spec must
> be unclear or have features that are unwanted") to considered
> objections?

I had a similar question.  The proposal seems to suggest that there be no difference in the requirements or guidelines that a specification must meet at each stage.  Is this intentional?  Is it the intent to remove these more conditions?

If that is not the intent, then there still needs to be some definition of what is expected of a specification at a particular maturity such that objections can be assessed by the IESG.

--Martin

---

From: Ted Hardie <[email-masked]>
Date: Thu, 16 Sep 2010 17:53:43 -0700

> I had a similar question.  The proposal seems to suggest that there be no difference in the requirements or guidelines that a specification must meet at each stage.  Is this intentional?  Is it the…

Yes, this is intentional.  The current gates for proposed standard are
high.  If a doc passes them and no
one finds new issues in two years of use, it is probably done.  If
there are issues (filed errata, an ongoing
effort at a -bis, community reaction that it is not really in use), I
think two years will probably find them
well enough for a draft designation (and five for full).

Just my two cents,

Ted

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 17 Sep 2010 09:02:37 +0800

> The current gates for proposed standard are

Sounds reasonable to me.

I suspect that there should be at least the guidance that the protocol is in use.  This is a condition that a PS might not be able to meet, but one that Draft and Full should require.  Objections based at PS time are usually based on whether or not there are (interoperable) implementations, not whether there is interest in the protocol.  Presumably, said implementations don't disappear in two years, though interest may fade.  Objections on the grounds of low interest should be easy to dispense with though.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 16 Sep 2010 21:13:11 -0400

Two very different kinds of questions occur to me, reading this.

Firstly, the one really good thing about our current draft advancement 
process is that we go look carefully and figure out what pieces of the 
RFC have not been implemented.  Given that this takes work, it is 
unlikely to be raised in a verifiable fashion during a "call for 
objections." There is a big gap between "I didn't implement that part 
and I don't know anyone who did" and "no one implemented that part." 
While getting the removals right takes work, it is also a very good thing.

Secondly, I can not tell what constitutes a legitimate objection.  I 
presume "I don't like this protocol" is not a relevant or useful 
objection.  Even "I think the protocol should have made a different 
design choice," which is actionable, would seem to be inappropriate. 
You seem to imply that there are some things beyond errata that are 
appropriate.  But I think we need rather more clarity on what those 
might be.  Was it your intention to spell those out later, iff the 
community is interested in the basic idea?

Yours,
Joel

On 9/16/2010 8:53 PM, Ted Hardie wrote:
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Ted Hardie <[email-masked]>
Date: Thu, 16 Sep 2010 21:44:34 -0700

On Thu, Sep 16, 2010 at 6:13 PM, Joel M. Halpern <[email-masked]> wrote:
> process is that we go look carefully and figure out what pieces of the RFC
> have not been implemented.  Given that this takes work, it is unlikely to be
> raised in a verifiable fashion during a "call for objections." There is a
> big gap between "I didn't implement that part and I don't know anyone who
> did" and "no one implemented that part." While getting the removals right
> takes work, it is also a very good thing.
>
You're right that it takes work, and, sadly, it mostly doesn't happen
as a result.   It's a good theory, but the practice has diverged in
the common case.

With this procedure in place, any member of the community can
say "This has bits that didn't get implemented by anyone"; if someone
then steps forward to shepherd the document through the objection,
the issue may be resolved (by demonstrating that others did or by producing
a new document that doesn't contain those pieces).  If no one is willing
to work through the objections, the default assumption is that the first
speaker was either right or is working on something no one else cares about
much.  That holds the doc at its current level, which leaves us no worse
off than where we are now.

> Secondly, I can not tell what constitutes a legitimate objection.  I presume
> "I don't like this protocol" is not a relevant or useful objection.  Even "I
> think the protocol should have made a different design choice," which is
> actionable, would seem to be inappropriate. You seem to imply that there are
> some things beyond errata that are appropriate.  But I think we need rather
> more clarity on what those might be.  Was it your intention to spell those
> out later, iff the community is interested in the basic idea?
>

Actually, I was thinking in terms of the IESG's current "discuss criteria", but
it shouldn't ever be spelled out too far in my  opinion; it requires
some leeway.
But I thoroughly agree that "I don't like this protocol/design choice" is not
valid, and would be quickly dispatched by an IESG considering it.
Adapting the discuss criteria (or something like it) would be useful guidance,
and it this gets sufficient traction, it certainly could be done.

Thanks for reading the doc,

regards,

Ted

>>> I had a similar question.  The proposal seems to suggest that there be no
>>> difference in the requirements or guidelines that a specification must meet
>>> at each stage.  Is this intentional?  Is it the intent to remove these more
>>> conditions?
>>>
>>
>>> If that is not the intent, then there still needs to be some definition
>>> of what is expected of a specification at a particular maturity such that
>>> objections can be assessed by the IESG.
>>>
>>> --Martin
>>>

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 Sep 2010 10:51:58 -0400

I would strongly object to a change to our process that removed the requirement to demonstrate interoperability.

If we need additional incentives to advancement, perhaps we should require that proposed standards revert to informational or historic if no action is taken within three years.  (action being: recycle at proposed, advance to draft)

Keith

---

From: SM <[email-masked]>
Date: Fri, 17 Sep 2010 13:39:22 -0700

Hi Ted,
>but does not require a reduction in maturity levels.  If you

The -00 draft passes ID-nits.  There is a mistake in the XML template 
(see Requirements) used to generate the draft.  This problem is 
apparent in other drafts as well.

In Section 2:

   "In practice, IETF document processing has evolved to a model which
    can be described as "objection based processing".

I'll only consider WG submissions for simplicity and I have reduced 
the "processing" to two stages, the Working Group Last Call and the 
IETF Last Call.  The Working Group Last Call generally does not use 
the "lazy consensus" approach.  In other words, there should be 
support for a draft for its adoption as a Working Group item and for 
it to get through the working group stage.  It can be argued that the 
IETF Last Call, in general, tends to be "lazy consensus" as there 
aren't that many, if any, comments about WG drafts at that stage.  I 
don't see the model as one of "objection based processing" because to 
the first stage.

   "The IETF Last Call is, in practice, a way for the larger community
    to object to a document or elements within it."

The IETF Last Call is about getting cross-area review.  There is a 
subtle difference between that and "a way for the larger community to 
object".  The latter presumes that someone within the IETF community 
which is not in the working group has the time and energy to review the draft.

   "It is also relatively clear that the energy to object can always
    be found in the IETF, even when the energy to sponsor or shepherd
    a document is absent."

"Consensus by exhaustion" works well within the IETF.  It can be used 
to drive away other IETF participants.

I'll open a parenthesis to comment about shepherding.  One of the 
questions asked is:

  "Does the Document Shepherd have any concerns about the depth
   or breadth of the reviews that have been performed?"

If the shepherd is chosen by the author of the draft, it is highly 
unlikely that the answer will be a "yes".

In Section 3:

   "All documents which are published as Proposed Standard RFCs shall be
    entered in queue for advancement to Draft Standard, with automatic
    advancement to take place two years from the issue date of the RFC."

The requirement for an interoperability report is not mentioned 
here.  The proposal does not specify what happens if an 
interoperability report is not submitted.

I would have suggested the following:

   All documents which are published as Proposed Standard RFCs and which
   have not advanced to Draft Standard, within two years from the issue
   date of the RFC, will be moved to Historic status.

The onus is on the proponents of the RFC to submit an 
interoperability report.  There should be an option for the author to 
ask for Informational instead of Historic if the specification is still in use.

   "If no document shepherd comes forward, the objections are
    automatically sustained and the document remains at Proposed
    Standard."

That would keep some documents at Proposed Standard for a long 
time.  One question that comes to mind is whether it is the 
responsibility of the IETF to find a shepherd or if it is for the 
author to do so.  This is similar to the problem of how authors can 
get reviews for their draft.  I'll elaborate on that even though it 
is not relevant to the discussion.

A person writes a draft.  He or she does not know how to get people 
to review the draft.  As the person has not reviewed other drafts 
before, he or she might find it difficult to convince someone to 
write a review.  As the saying goes, I scratch your back and you 
scratch mine.  In plain English, if you review other people's drafts, 
it would be easier for you to get people to review your draft.  If 
you don't have the time or energy to do so and everybody does that, 
the mechanics won't work.

In Section 4:

   "Those who are entering errata for a published RFC should indicate
    whether they believe the issue raised is sufficient to block
    advancement."

That turns an errata into a DISCUSS.  I don't think that the errata 
mechanism should be used like that as it is going to create more 
problems than it will solve.  If errata becomes a discussion forum, 
it will be more work for the Area Directors.

   "Any individual who has been banned from the IETF main list may not
    post an objection, and repeated frivolous objections should be
    considered grounds for removal of posting rights."

If this draft moves forward, that sentence will probably be 
removed.  I am not going to comment on that point for now but I understand it.

In Section 8:

   "If the broader Internet community is judging protocol maturity based on
    standards level, there is some risk that changing the mechanism by
    which documents advance along the standards track may require that
    judgement to change."

If the broader Internet community is judging protocol maturity based 
on the RFC brand, there is nothing the IETF can do about it.  Some 
people view "Proposed Standard" as being a level above "Draft 
Standard".  Some authors do not even read BCP 78 even though they 
state compliance in their draft.

There is an presumption of mutually assured destruction in the 
current discussion model.  Those that have nothing to lose have 
nothing to fear.

Regards,
-sm

---

From: Ted Hardie <[email-masked]>
Date: Fri, 17 Sep 2010 16:10:35 -0700

Thanks for the comments, some replies inline.

On Fri, Sep 17, 2010 at 1:39 PM, SM <[email-masked]> wrote:
> The -00 draft passes ID-nits.  There is a mistake in the XML template (see
> Requirements) used to generate the draft.  This problem is apparent in other
> drafts as well.
>

Thanks for letting me know.

> I'll only consider WG submissions for simplicity and I have reduced the
> "processing" to two stages, the Working Group Last Call and the IETF Last
> Call.  The Working Group Last Call generally does not use the "lazy
> consensus" approach.  In other words, there should be support for a draft
> for its adoption as a Working Group item and for it to get through the
> working group stage.

This is a good point.  It is really late-stage processing that works mostly by
resolving issues (especially the IESG processing, but to a lesser extent the
Last Call processing).  I will update the description in the next version.

It can be argued that the IETF Last Call, in general,
> tends to be "lazy consensus" as there aren't that many, if any, comments
> about WG drafts at that stage.  I don't see the model as one of "objection
> based processing" because to the first stage.
>
> The IETF Last Call is about getting cross-area review.  There is a subtle
> difference between that and "a way for the larger community to object".  The
> latter presumes that someone within the IETF community which is not in the
> working group has the time and energy to review the draft.
>

Any review that produces useful input is generally accepted by the document
authors.  But only objections really change the processing from that
point on, but creating a requirement to resolve them before moving on.
In other words, the document processing state machine can only change
here if an objection is lodged.

> "Consensus by exhaustion" works well within the IETF.  It can be used to
> drive away other IETF participants.
>

Noted.

> I'll open a parenthesis to comment about shepherding.  One of the questions
> asked is:
>
> If the shepherd is chosen by the author of the draft, it is highly unlikely
> that the answer will be a "yes".
>

Agreed.  But the shepherding and initial document and shepherding
the resolution of an objection are a bit different.  Perhaps a different
term is needed to make this clear.  What I propose is that
after an objection is lodged to advancement, that the IESG will determine
whether it is a valid objection, based on input from the objector and
the work of the shepherd.

> The requirement for an interoperability report is not mentioned here.  The
> proposal does not specify what happens if an interoperability report is not
> submitted.
>

This proposal does not require an interoperablity report.

> The onus is on the proponents of the RFC to submit an interoperability
> report.  There should be an option for the author to ask for Informational
> instead of Historic if the specification is still in use.
>

I think this proposal is closer to Keith's thinking than mine.  My personal
belief is that neither the carrot nor the stick has been particularly effective
in eliciting interoperability reports and efforts to move to draft.  Given that,
we can rest on a single-stage system in the common case or use something
else to signal to the larger community how mature a standard is.
This proposal uses a time-and-consensus call approach.

If you believe that there is energy to use the interoperability-test based
 current system, then there is no need for the the change this proposes

> That would keep some documents at Proposed Standard for a long time.  One
> question that comes to mind is whether it is the responsibility of the IETF
> to find a shepherd or if it is for the author to do so.  This is similar to
> the problem of how authors can get reviews for their draft.  I'll elaborate
> on that even though it is not relevant to the discussion.
>

I think if no community member is interested in resolving an objection, then
the document is not needed.  I understand the issue that the author may
be an interested party and may offer to shepherd.  That could set up
an unfortunate author-vs-object dynamic.  The IESG still judges,
however, and can determine
whether the objection should be sustained.

> A person writes a draft.  He or she does not know how to get people to
> review the draft.  As the person has not reviewed other drafts before, he or
> she might find it difficult to convince someone to write a review.  As the
> saying goes, I scratch your back and you scratch mine.  In plain English, if
> you review other people's drafts, it would be easier for you to get people
> to review your draft.  If you don't have the time or energy to do so and
> everybody does that, the mechanics won't work.
>
> In Section 4:
>
> mechanism should be used like that as it is going to create more problems
> than it will solve.  If errata becomes a discussion forum, it will be more
> work for the Area Directors.

Do you think the presence of serious errata would currently prevent an
AD from taking an existing document through the advancement process,
even if implementation reports were received?  I do, but this may be
a question of interpreting "serious".

> If this draft moves forward, that sentence will probably be removed.  I am
> not going to comment on that point for now but I understand it.
>
> In Section 8:
>
> If the broader Internet community is judging protocol maturity based on the
> RFC brand, there is nothing the IETF can do about it.  Some people view
> "Proposed Standard" as being a level above "Draft Standard".  Some authors
> do not even read BCP 78 even though they state compliance in their draft.
>
> There is an presumption of mutually assured destruction in the current
> discussion model.  Those that have nothing to lose have nothing to fear.
>

Noted, but this still leaves the final decision in the hands of the IESG.

The upshot, though, is that any mechanism which shifts most drafts
up the ladder will increase the overall workload.  I've tried to lower
the friction for the "easy" cases and distribute it for all cases.  But
that doesn't create new energy.  If too much of the new energy
required comes from the IESG, this will obviously have other bad
effects.  Further input on how to better distribute the load is welcome.

regards,

Ted

---

From: Martin Rex <[email-masked]>
Date: Mon, 20 Sep 2010 17:51:59 +0200 (MEST)

Keith Moore wrote:
> 
> I would strongly object to a change to our process that removed
> the requirement to demonstrate interoperability.
> 
> If we need additional incentives to advancement, perhaps we should
> require that proposed standards revert to informational or historic
> if no action is taken within three years.
> (action being: recycle at proposed, advance to draft)

As I understand it, the original draft requirement (demonstrating interop)
was to improve the specification in several aspects:

  - ensure that there are implementations of the specification,
    because implementing a spec is a good method to uncover inconsistencies,
    ambiguities and contradictions

  - ensure that there are multiple independent implementations in order
    to find out whether independent implementers understand the spec
    in the same fashion.

  - find out which of the (interoperable) features of a spec are necessary
    and which one are more in the direction of bloat

Recycling a spec on proposed will primarily add new features, and include
errata and potential clarifications, but rarely drop features.
Therefore I'm no sure that "recycling" at proposed should be considered
a valid substitute for a demonstration of interoperability between
independent implementations.  At least, there should be a limit as to
how often recycling a spec at proposed should "exempt" a working group
from demonstrating interoperability.

Otherwise, there might grow a disconnect between the most recent spec
and what is most commonly used on the internet.  The TLS protocol is
a victim of the "recycle at proposed" rather than performing serious
interop testing.  With the result that the most recent spec is
TLS v1.2 (08/2008), whereas the protocol version that is most widely
used, and the _only_ TLS protocol version that can be safely used by
a client that does not implement an application-level reconnect
fallback is TLS v1.0 (01/1999).  If the forward extensibility options
in the TLS protocol had been more appropriately interop tested in
the years after TLSv1.0, this problem would likely be *much* smaller.

The PKIX Interet Certificate and CRL Profile has similar being
recycled at proposed serveral times (2459,3280,5280), and has
grown feature bloat rather than seen cleanup & feature reduction.

-Martin

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Sep 2010 14:34:24 -0400

> errata and potential clarifications, but rarely drop features.

I certainly recall instances where features were dropped from the Draft Standard version of a specification precisely because interoperability had not been demonstrated.

> from demonstrating interoperability.

Hmm.  I wouldn't consider recycling at proposed to be a substitute for demonstration of interoperability, but rather, a demonstration that there continues to be interest in working on the protocol.   Presumably that means that the protocol is useful and that there is a real need to continue investing in the specification.   (Of course it might have been better if the WG had done a better job with the original design, but it's hard to make a general statement about that.  Sometimes protocols get used very differently than was anticipated by the WG.)

At any rate, when a document gets recycled at proposed several times, I don't think it's due to avoidance of interoperability testing needed for Draft Standard. I suspect it's more likely to be because interoperability testing and/or experience has indicated that further work is needed.

Keith

---

From: [email-masked]
Date: Mon, 20 Sep 2010 13:51:46 -0700 (PDT)

> I certainly recall instances where features were dropped from the Draft
> Standard version of a specification precisely because interoperability had not
> been demonstrated.

As can I on quite a few occasions, but what I cannot provide is any evidence -
or even an anecdote - that in the present climate of high scrutiny before
getting to proposed, such changes demonstrably improved interoperability and
deployability in the field.

Now, it is entirely possible and even likely that this practice carries
benefits that far outweigh the costs. But we should not assume that's the case
jut because unused features offend our design sense.

				Ned

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Sep 2010 18:13:52 -0400

> deployability in the field.

I'm sure that there is a point of diminishing returns, past which more scrutiny doesn't result in improved interoperability or deployability, and may degrade the latter.

I also think that there is limited utility in having multiple review periods.  But I do believe that interoperability testing is valuable at helping debug a specification.  

Part of the problem with moving to Draft Standard in the current process may be that such testing, and resulting improvement of the specification, mostly benefits late adopters.  The early adopters who are the experts at the protocol already know what it takes to make it work, even if the specification is ambiguous in places.  Those are the very people who need to be involved in cleaning up the specification, but (depending on market conditions) they may see it as mostly benefiting their competitors.  

But I don't think that a prospective implementor cares (much) whether a specification is Proposed, Draft, or Full, or perhaps even Informational or Experimental.  I think they care about whether the specification reflects current widespread practice (or if the protocol is new, best known practice), whether it's sufficiently precise to permit implementations to interoperate, whether it's implementable with reasonable effort, whether its encumbered by patents or other constraints, and whether it is deployable.  

If I'm close to right about these things, maybe we would do well to rethink our standards process along these lines.   Rather than moving to Draft, then Full, maybe we should periodically make an assessment about how well a specification still reflects existing practice, how widely it is used, how precise it is, and so on.  

Keith

---

From: Ted Hardie <[email-masked]>
Date: Mon, 20 Sep 2010 16:01:23 -0700

On Mon, Sep 20, 2010 at 3:13 PM, Keith Moore <[email-masked]> wrote:

>Those are the very people who need to be involved in cleaning up the specification, but (depending on market conditions) they may see it as mostly benefiting their competitors.
>

For protocols where interoperability with others' implementations is
important for your customers'
view of your product, there is probably enough benefit to get the
involvement; if you look bad because
other people aren't able to read the spec, you have some incentive.
But that may no longer be the
common case.

> If I'm close to right about these things, maybe we would do well to rethink our standards process along these lines.   Rather than moving to Draft, then Full, maybe we should periodically make an a…
A periodic call for comments, say at  2 and 5 years out, with those
judged to be still useful moving up the ladder,
for example?

I understand that "objection based processing" was not the most polite
way to word this in my draft,
and I'm sufficiently chastened to change it in the next version.  But
I think any system we base on a periodic
assessment like the above has to have a default of "advance".  The
current Proposed standards
get a lot of scrutiny and generally are pretty good.  If we don't do
"default advance", we are adding
friction to the common case, where they should move forward.

We also have the much harder task of judging the consensus on
documents based on what is likely
to be small amounts of commentary.  That's tough, but getting more
than small amounts is
likely going to be substantial amounts of work.  For substantial
amounts of work, our current
criteria for draft are much cleaner: two interoperable implementations
from different code bases and license
grants.  For the record, I still think that it useful data to have,
but my personal read is that the community
as a whole doesn't see the value in it for things that work.  So the
common case stays at proposed.
Things get moved up when they're already being pruned of things that
didn't work or because some
external force requires them to.

Maybe a single stage with revisions at that stage is all we really
need.  But I suspect the opportunity to raise and
resolve issues would be valuable, especially if the "default advance"
attracted the attention.

Just my two cents,

Ted

---

From: John Leslie <[email-masked]>
Date: Mon, 20 Sep 2010 20:58:06 -0400

Ted Hardie <[email-masked]> wrote:
> 
> Title: Proposed mechanics for document advancement
> Author(s): T. Hardie
> Filename: draft-hardie-advance-mechanics-00.txt

   I seem to have been inordinately busy recently: sorry to take so long
getting around to this.

   Ted has correctly identified the problem as "mechanics for advancement,"
but IMHO has written a design for how to win the last war... :^(

   (He also got some details wrong, but those aren't so important.)

   We have a long-standing situation where three "maturity levels" have
been designed and documented in an IETF-consensus document (RFC 2026)
which has proven remarkably resistant to change.

   And we have a current effort at changing it which perpetuates the
frankly wrong-headed idea that the problem is that we got the number
of levels wrong. Thankfully, Ted has correctly noted that that's _not_
where the problem lies.

   I'm going to put on a hat I very seldom wear when I post to this
list -- the hat of a Narrative Scribe for IESG telechats.

   I've listened to the process of trying to advance maturity levels
of email-related documents. It's painful!

   No, it's _very_ painful!

   The IESG has a process for standard-track documents, which has evolved
to match the world-as-we-know-it. Basically, it formalizes LastCalls;
then proceeds to a shepherding process where one AD tries to get consensus
of the whole IESG that the document is "a reasonable basis on which to
build the salient part of the Internet infrastructure."

   Any other AD may call for discussion during a telechat. (Alas, the
agendas are so full that merely placing a DISCUSS doesn't mean that.)

   In general, ADs write either a DISCUSS or a COMMENT; and a DISCUSS
is (initially) blocking: the sponsoring AD must look for a way to
"resolve" the DISCUSS. Sometimes this is simple clarifying language;
sometimes it's not so simple...

   I have listened while agenda items related to advancing "maturity"
of email standards came up. It has been confusing, not only to me,
but to various IESG members as well.

   I've also been subscribed to mailing-lists such as <ietf-smtp>.
The folks on those lists are even more confused, and black helicopters
are frequently sighted. :^(

   Nobody seems to know quite what the IESG is supposed to do when it
considers a document for "advancement in maturity" (except John Klensin,
of course!) -- and IMHO that is the source of the confusion and pain.

   What we need, IMHO, is not a revision to RFC 2026, but a procedure
which eliminates the confusion and pain.

   And John Klensin was definitely on the right track when he wrote his
ISD draft (draft-ietf-newtrk-repurposing-isd) calling for "Internet
Standards Documents" to track maturity, etc. of standards-track RFCs.

   (I don't claim it was perfect -- indeed I co-authored a different
proposal intended to seriously minimize the IESG involvement in
writing such documents.)

   The essential point is that "advancement in maturity" should _not_
involve the IESG in a review de nova; but merely document the
implementation and deployment history.

   IMHO, John Klensin is much closer to the right solution to this
issue than Ted Hardie: Ted spends too much effort aiming for consensus
in things which shouldn't matter, while John sets out to document things
about which there's not a whole lot of room for disagreement.

   So, to recap, I give Ted Hardie high marks for identifying the
problem, but John Klensin higher marks for proposing a way forward.

--
John Leslie <[email-masked]>

---

From: Keith Moore <[email-masked]>
Date: Mon, 20 Sep 2010 23:29:01 -0400

On Sep 20, 2010, at 7:01 PM, Ted Hardie wrote:

> For protocols where interoperability with others' implementations is important for your customers'
> view of your product, there is probably enough benefit to get the involvement; if you look bad because
> other people aren't able to read the spec, you have some incentive.  But that may no longer be the
> common case.

I think it's going to be different from one protocol to the next. 

> A periodic call for comments, say at  2 and 5 years out, with those judged to be still useful moving up the ladder, for example?

I'm thinking more along the lines of every 5 years.  Maybe more frequently to start with, or more frequently for those protocols for which (in IESG's judgment) the status might change quickly.  But with no end to the periodic review, as long as the protocol is still considered a standard.  And I'm thinking that the review should address the specific points mentioned above.

I don't think that there should be an expectation of the document being revised and reissued for each review.  It would be a bit more like having a new applicability statement issued every few years, but even that should probably not be an RFC unless the applicability is complicated to describe. 

Nor do I see this as having the protocol advance in grade.  Instead I see a periodic re-evaluation of whether the protocol specification still should have IETF's endorsement.   And the endorsement is not just "yes, this is still a standard" or "no, it's not", but rather things like whether the design is still sound (especially where security is concerned), whether we have confidence that the specification is sufficiently precise, whether implementing the specification as written will produce an implementation that interoperates with other implementations written to the specification, whether the specification is appropriate for wide use or only in limited circumstances, whether the specification is actually used widely in practice, etc.

(For that reason, where the default is "to advance" or not is not an issue for me - because it's not about advancement.  But the periodic evaluations should have expiration dates.  If someone is looking at a long-expired evaluation, and there's not a more recent one, that tells them something about the currency of the protocol specification.)

As a separate concern, I think I'd still like to see a two-step process before IETF declares something a standard, and that it should still require interoperability testing.    But I don't think that a review process - where a WG labors mostly in isolation for 2-3 years and then the rest of the community looks at it only at Last Call - works very well.  In cases where a WG starts from scratch, I favor developing an Experimental prototype, having people implement it, and then developing a standard version of the protocol based on wide review of, and experience gained from, the prototype.  The goal should be to have initial prototype implementations no more than one year from WG formation.  There are two points to this:  One, start implementing earlier.   Two, get wide community review early on, before the WG is so exhausted and committed that it cannot change its thinking even for good reasons.

There could be multiple iterations of the prototype phase, or multiple prototypes.   The eventual standard should not break new ground as compared to the last prototype on which it was based.

In cases where there's already a well-understood protocol that forms the basis of the WG's work, the prototype stage could perhaps be skipped.  But the first stage in that case should be to accurately document existing practice.

Final last call and IESG review should not be a big step.  There should be a presumption that as long as the WG paid appropriate attention to review comments at the prototype stage, addressed those concerns appropriately, and the final standard doesn't deviate too much from the prototype in what it does (though it shouldn't have to be strictly compatible) - the document will be approved.  That's not to say that IESG can't deny approval if significant design flaws are found, but it would not be appropriate to hold up the specification for trivial reasons at that stage.  NIt picking about document text should only result in nits being fixed in the specification.  Those things are important - especially when clarity is concerned - but shouldn't hold up the specification's approval for very long.

Keith

---

From: SM <[email-masked]>
Date: Sun, 26 Sep 2010 11:28:59 -0700

Hi Ted,

[Bcc to the person quoted in this message]

>But that may no longer be the

Yes.

>and I'm sufficiently chastened to change it in the next version.  But

The term you used was clear and it got the idea through.

>assessment like the above has to have a default of "advance".  The

Setting a default of "advance" encourages objection instead of 
support.  This might encourage an adversarial model.  It may already 
be happening within the IETF as strong discontent could be viewed as such.

>friction to the common case, where they should move forward.

Implementation reports used the "default advance".  These documents 
do not get a lot of scrutiny.  I don't know the number of Proposed 
standards that fall through the cracks.

>likely going to be substantial amounts of work.  For substantial

Yes, on one hand the person determining consensus asks for commentary 
but he/she don't want too much commentary as it turns each document 
into too much work then.

>external force requires them to.

You can placate some vendors by throwing in some features or 
rewriting the base specification in such a way that the change 
becomes an intrinsic part of it.  That makes pruning a near-impossible task.

>resolve issues would be valuable, especially if the "default advance"

The current situation is a "single stage".  It is based on RFC 
branding.  It might work as follows:

  1. Coax or harass AD to get sponsorship for the I-D.

  2. Push proposal into a WG and let it simmer for some time, if applicable.

  3. Last Call the I-D.  If the author is unlucky, he/she might come across
     strong opposition to the proposal.

  4. Pray that a member of the IESG does not raise a DISCUSS.

  5. Make the changes to address imperial edict, if applicable.

  6. Declare victory when RFC is published.

There isn't any incentive to advance to Draft as:

  (a) It's not common practice.

  (b) The author does not even know that Draft exists.

  (c) The author does not need more grief.

There seems to be an assumption that there is widespread agreement on 
what the documents represent, i.e. consensus.  If that assumption 
holds, it is simply a matter of defining the 
mechanics.  draft-hardie-advance-mechanics-00 focuses on the 
mechanics for advancement of documents along the standards 
track.  Whether I agree with it or not is inconsequential as it has 
the merit of putting an idea into words to "fix" the statu quo.  If 
not changing the current state of affairs is the worse outcome, would 
people who do not support the proposal change their stance from 
"objecting" to neutral?

I may be taking the off-topic ramp or it may be read as an attempt to 
derail this proposal by getting into the mechanics of 
consensus.  Nevertheless, I'll quote a person that is participating 
in an IETF WG for the first time:

   "I leave to the Chairs of this WG the right to dismiss them [the 
comments made],
    and I'll accept it if that happens." [1]

Would the mechanics of consensus work better if a participant takes 
such a stance and send substantive comments?  Does a "+1" mean that a 
participant has actually read and understood the proposal?  In other 
words, would it be better if participants explained why they support 
a proposal?

I'll re-purpose some text from BCP 27:

   The "standards of evidence are materially beyond what can be reasonably
   accomplished".  A pragmatic approach is to leave it to the advocates to
   make a convincing argument that the specification fulfills the
   requirements.  The "specific way to make the argument is left to the
   advocate".

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/vwrap/current/msg00256.html