Subject: Survey design and interpretation (was: Re: Final reminder: Please fill out our survey on IETF authoring tools)

From: John C Klensin <[email-masked]>
Date: Wed, 04 Nov 2020 10:48:11 -0500

--On Wednesday, November 4, 2020 21:10 +1300 IETF Executive
Director <[email-masked]> wrote:

> Hi 
> 
> ** Final reminder **
> 
> The IETF is conducting a survey to help us understand what
> tools and formats you use for authoring IETF Internet Drafts
> (I-Ds), how you use them and what more you need from them.
> The data from this survey is vital for our future planning of
> tools investment.
>...
> Your email address has been selected because you have been
> listed as the author or submitter of an I-D in the last five
> years.  The survey is anonymous - we only record the data you
> provide and do not record your IP address or any details of
> your computer.

Jay,

I filled the survey out last night.  Partially prompted by other
recent events and discussions, I have several comments and
thoughts that I think deserve community, as well as LLC,
attention.  So, while I would normally send a note like this off
list (as I have done in the past), I think this one needs to be
public.  In no particular order:

(1) The experience would presumably be different for those for
whom the number of I-Ds or RFCs is small, but it took me close
to 45 minutes to make even the crudest of estimates for the
first page.  Just as some people (not a random sample of the
community) find my long analytical notes discouraging and won't
read to the end (or even start), surveys that require that sort
of time investment will discourage some people (not a random
sample of the community) from completing them and may even
discourage them from opening your surveys in the future.  That
means non-representative results and a survey that requires
great care to interpret especially in a way that does not just
reinforce prior biases or assumptions.

(2) Again probably different for those with small numbers of
I-Ds and, especially, RFCs but for those of us with large
numbers of the latter, the survey is not, as claimed above,
anonymous.  You don't need to collect IP addresses or data about
computers - the number of people who are authors or co-authors
of 51+ (if I remember correctly) RFCs is small enough that
examination of answers to a few questions after that would make
identifying us rather easy.

(3) Our rules (or lack thereof) about RFC and I-D "authorship"
sometimes don't predict well to actual construction of
documents.  There are exceptions, but the typical number of
people who actually interact with the tools and key or copy
words into their machine-usable representations is often one
independent of the number of listed authors.  When you include
someone in a survey like this who has not actually interacted
with a document or tools at a technical level, you get weird
results if they complete the survey and an artificially low
response rate if they do not.  At the other extreme, many of us
have helped relative newcomers by providing outline I-Ds in
xml2rfc form based on their notes but are not listed as authors
or even contributors -- efforts that don't count for survey
purposes (I would not be surprised to learn that similar
outlines have been provided in MS Word).  And, fwiw, the larger
amount of understanding required of a newcomer to flesh out an
outline in xml2rfc v3 as compared to v2 is one objection to the
new format that has not been discussed enough.

(4) The question and response structure of the survey instrument
is poorly designed (assuming you are after confident and maximum
interpretability).  I have many examples but, to keep this note
from getting longer, let me stop at three:

(4.a) While I imagine most of us could figure it out, putting a
"If you have any questions about this survey then please
contact..." comment after the survey is completed is fairly
pointless.  To be useful, such a statement should appear on the
first page and/or the cover letter suggesting that people take
the survey, not after it is done and it is too late to ask
questions about the survey itself.  Although I'm confident that
was just an oversight, it seems to me sufficiently obvious to
suggest that not enough thinking went into the survey design.

(4.b) A couple of decades ago (before xml2rfc existed), I used
to do most of my draft and RFC preparation in plain text, with
one or two in nroff, and one I recall going in and out of
WordPerfect or MS Word before being edited in plain text.  I
don't intend to go back there and other than giving me a bit of
extra perspective on those types of tools and methods, it is not
clear to me that history that far back is relevant to what you
are trying to learn.  However,except for that first question
about date ranges the survey doesn't say "within the past X"
years.  But your scales in that part of the survey are "Always/
Often / ... / Never".  So if, for example, I constructed an I-D
in 1998 using nroff obtained from the RFC Editor because I was
trying to revise or update an earlier document, I can't answer
"never" to a "used nroff" question even though I never used any
of the templates or specialized tools listed, nor can I answer
"always" to use of xml2rfc (even in the last decade) but pushing
me into "often" does not give you the information that "almost
always" might.  Either the choice of scale labels is unfortunate
or the questions need to be more refined.

(4.c) Then there are questions about how satisfied I am with
xml2rfc as an authoring tool, with a scale from "very satisfied"
to "very dissatisfied".  Reasonable question except that there
are some of us who are "very" (or closer than "nearly")
satisfied with v2 (perhaps for substantive reasons and perhaps
because we are just used to it) and very dissatisfied with v3.
I assume there are those who are "very" satisfied with v3 but
hate v2.   But the questions fail to allow the respondent to
differentiate so responses to any statement made about
satisfaction with xml2rfc (by implication, both versions) are
going to be so hard to interpret accurately as to be nearly
useless.

I'm not going to go on further except to say that, for those of
us who are trying to do technical work in the IETF, surveys like
this (and at least some of the "Consultations" you have posted)
take time away from that available to do that work.   That might
be worthwhile if the surveys/ consultations are infrequent and
designed well and carefully enough to inspire confidence in
their interpretation and use.  But, even then, I suggest that at
current levels they may be interfering with progress on
standards-related activities, which is one thing the LLC, at
least IMO, should absolutely not be doing.

thanks,
   john

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 17:46:36 +0100

John,

replying to one bit of special interest to me:

> because we are just used to it) and very dissatisfied with v3.

I would really like to understand what would make you be unhappy with v3
as opposed to v2. Could you please elaborate a bit (either here, or on a
more xml2rfc-specific list)?

Best regards, Julian

---

From: [email-masked]
Date: Wed, 04 Nov 2020 13:10:19 -0800 (PST)

John C Klensin <[email-masked]> wrote:

> I filled the survey out last night.

I filled it out the other day, although after completing it I wondered why
I bothered.

> Partially prompted by other

> first page.

I gave up on figuring out accurate values a little sooner than you did.

> Just as some people (not a random sample of the

It is necessarily the case that anyone who has authored/coauthored a fair
number of drafts and RFCs has arrived at a process that works for them - and as
you note, this includes always having someone else do the editing. What is less
clear is that knowledge of the process someone like you or I use - which owes a
lot more to to historical factors than to any rational selection process
conducted ab inito - is of any use in deciding the direction to take on
future tool development.

Which is why if I was doing this I would be far more interested in the views of
new and even prospective contributers.

> identifying us rather easy.

(It was at the point of trying to determine if I had crossed the 50 threshhold
that I decided to stop googling and start guessing.)

And perhaps more to the point, why ask the question at all? Does authoring more
stuff mean you're better judge of the tools to use? Or worse?

> new format that has not been discussed enough.

Having just embarked on a couple of new drafts, I've experienced this 
directly.

> least IMO, should absolutely not be doing.

+1

				Ned

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 6 Nov 2020 11:18:05 -0500

Like John, as a long-time draft and RFC author I had to wing some of the
questions since the answers have changed over time. For example, I've used
(at the very least) raw text, raw nroff, nroffedit (which I loved at the
time), the Word template, raw XML (v2), and Kramdown, which is my current
favorite. I tried to capture this in the open comment boxes.

Cheers,
Andy

---

From: IETF Executive Director <[email-masked]>
Date: Sat, 7 Nov 2020 07:39:09 +1300

Hi Ned

> On 5/11/2020, at 10:10 AM, Ned Freed <[email-masked]> wrote:
> 

We can see from Datatracker how many submissions there have been over the same period and bucket those in the same way as the survey.  Then we can compare the data from the survey to see what coverage we have.  i.e. if there are 500+ who submitted >20 times and we only captured data from 5 of them then that’s a concern.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: [email-masked]
Date: Mon, 09 Nov 2020 07:08:51 -0800 (PST)

> We can see from Datatracker how many submissions there have been over the same period and bucket those in the same way as the survey.  Then we can compare the data from the survey to see what cover…

Apparently you missed my main point, which was that the pepople who have
learned over time to operate the process in a fashion that lets them  produce
lots of documents may not be the best source of information about how to
improve the process for those who have not managed to do this.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 09 Nov 2020 10:57:05 -0500

--On Monday, November 9, 2020 07:08 -0800
[email-masked] wrote:

>...
>> We can see from Datatracker how many submissions there have
>> been over the same period and bucket those in the same way as
>> the survey.  Then we can compare the data from the survey to
>> see what coverage we have.  i.e. if there are 500+ who
>> submitted >20 times and we only captured data from 5 of them
>> then that's a concern.
> 
> Apparently you missed my main point, which was that the
> pepople who have learned over time to operate the process in a
> fashion that lets them  produce lots of documents may not be
> the best source of information about how to improve the
> process for those who have not managed to do this.

And, for the record, as the person who started this thread, I
agree with this.  I wouldn't wish my particular method of
producing documents on anyone and it probably wouldn't work for
many people other than me anyway.  Today's situation is almost
certainly better than the one we had a decade or two ago because
there are more choices.  However for a relative newcomer, having
enough knowledge to make the best choice for themselves of those
methods is, itself, a somewhat daunting task that creates a bit
of a chicken-and-egg problem.  At least in part, we have done as
well as we have with relative newcomers producing drafts because
volunteers from the community have quietly helped them structure
things, put mailing list comments into outline drafts, etc.,
_and_ because of extraordinary patience on the part of the
secretariat when drafts are rejected by the automated systems
and the would-be author tries to post manually.  

A different way to look at the issues with the survey instrument
is that it captures none of that from either the standpoint of
those first starting out or those providing the assistance.

    john

---

From: IETF Executive Director <[email-masked]>
Date: Tue, 10 Nov 2020 06:24:40 +1300

> On 10/11/2020, at 4:08 AM, Ned Freed <[email-masked]> wrote:
> 
>> Hi Ned
> 

And I think you’ve missed my point.  The fact that we are asking how many documents some has produced does not mean that we weight those higher.  I was aiming to detect skewing rather than correct for it.  

If you want to give higher weight to the answers from those that have produced a few documents then we need to know a) how many documents a respondent has produced; and b) if the responses overall are already weighted more towards one end of the spectrum of the other. 

Both use cases require that question about the number of documents.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]