Subject: Re: [109all] NOC update #2

From: John C Klensin <[email-masked]>
Date: Thu, 19 Nov 2020 03:17:50 -0500

Sean,

I think this is obvious but, just to check my understanding:
this implies that all of the attacks on Meetecho for
unreliability of their software during those incidents were
misguided or misdirected.  Correct?

thanks,
   john

--On Thursday, November 19, 2020 01:56 +0000 Sean Croghan
<[email-masked]> wrote:

> As previously reported, we tracked down the cause of the
> interruption of the iabopen session to an issue with an
> unexpected Azure network interface removal event on network
> interfaces provisioned with SR-IOV.  To prevent this happening
> again we intended to remove SR-IOV networking entirely.
> Unfortunately it now transpires that this change did not get
> applied to 2 of the 16 VMs including the application VM for
> the Plenary. So to add to the list of reasons to want 2020 to
> be over, towards the end of Plenary the same network interface
> removal event occurred and triggered an outage long enough to
> affect everyone.
> 
> I can confirm that the SR-IOV provisioning has now been
> removed from all VMs, which we believe eliminates the risk of
> the same thing happening again.  We continue to work with
> Azure Direct Support to determine the underlying cause of the
> removal events.
> 
> Please let me know if you have any questions.
> 
> Sean
> 
> 
> 
> On Nov 17, 2020, at 4:56 PM, Sean Croghan  wrote:
> 
> 
> 
> I have an update for those of you affected by the outage in
> yesterdays IABOPEN session. We have isolated this to a
> interrupt to the virtual machines network interface. We
> currently have no explanation for this outage. We have engaged
> the hardware and network team with Azure to determine the
> cause of this event but do not have an explanation at this
> time.
> 
> I will provide an update when we have received more
> information.
> 
> 
> For those interested in details:
> 
> At 07:56:36 UTC the network interface (eth0) went link down
> and the interface was removed from the VM At 08:00:28 UTC then
> a new interface was added to the VM At 08:00:29 UTC (eth1)
> went link up
> 
> Yes the VM added a new interface. The servers were provisioned
> with SR-IOV and we suspect that a migration event occurred
> that moved the VM to different hardware causing the NIC driver
> to be reloaded. We have found some evidence that would support
> our theory that a migration or unscheduled maintenance event
> occurred and are working to verify if that happened during
> this event. We have removed SR-IOV from the network interfaces
> on all servers.
> 
> I hope you are having a good and productive week
> 
> 
> — The IEFT NOC Team
> 
> --
> 109all mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/109all
>

---

From: Jay Daley <[email-masked]>
Date: Thu, 19 Nov 2020 22:57:20 +1300

Hi John

As this is quite a political question I will answer it for you.

If the criticisms are narrowly interpreted as being about the front end we are running then this information implies that they were likely based on an incorrect interpretation of the issues.  If however the criticisms are more broadly interpreted as about the whole approach of us taking responsibility for the systems integration rather than outsourcing to an integrated service provider, then this information does not change anything.

Jay

> On 19/11/2020, at 9:17 PM, John C Klensin <[email-masked]> wrote:
> 
> Sean,
> 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> [email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>
>> https://www.ietf.org/mailman/listinfo/109all <https://www.ietf.org/mailman/listinfo/109all>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Nov 2020 05:55:21 -0500

--On Thursday, November 19, 2020 22:57 +1300 Jay Daley
<[email-masked]> wrote:

> If the criticisms are narrowly interpreted as being about the
> front end we are running then this information implies that
> they were likely based on an incorrect interpretation of the
> issues.  If however the criticisms are more broadly
> interpreted as about the whole approach of us taking
> responsibility for the systems integration rather than
> outsourcing to an integrated service provider, then this
> information does not change anything.

Jay,

thanks.

I was seeing both of the criticisms (as well as what I think was
the implication that the network failures and their effect were
a result of the systems integration process [1]) and was
actually trying to find a way to separate them.

  john

[1] I may just be unlucky, but I've seen runs of failures with
symptoms similar to those we saw "with Meetecho" in recent
months with the likes of Zoom and GoToMeeting, so we should not
assume that outsourcing would completely eliminate the risk of
those problems.  Ignoring whether we could get needed
functionality, the only thing it would guarantee is having
someone else to blame.

---

From: [email-masked]
Date: Thu, 19 Nov 2020 04:52:06 -0800 (PST)

> If the criticisms are narrowly interpreted as being about the front end we
> are running then this information implies that they were likely based on an
> incorrect interpretation of the issues.

I see. So when I attempted to switch browsers in order to work around an audio
problem and was informed that I could not rejoin the meeting because emailcore
had run over its time slot, even though the meeting was still going on, this
was simply a "misinterpretation of the issues" on my part?

And no doubt the problem with the calext meeting this morning was also purely
PIBKEC.

				Ned

---

From: Meetecho IETF support <[email-masked]>
Date: Thu, 19 Nov 2020 15:04:41 +0100

Ned,

> was simply a "misinterpretation of the issues" on my part?

all sessions are configured to terminate 1 hour after the scheduled end 
time, and we close the sessions manually when the meeting is actually 
over, no matter how much it went overtime. The emailcore session closed 
at 08:36:42 UTC, as you can easily spot from the jabber logs:

	https://www.ietf.org/jabber/logs/emailcore/2020-11-17.html

The recording of that session is still in the uploading queue. Will show 
up on YouTube anytime soon and you may want to check that as well.

Also, FYI, there are examples of other sessions that went 15/30 minutes 
overtime.

> And no doubt the problem with the calext meeting this morning was also purely

Don't know what PIBKEC means, sorry. If you're referring to the room 
only opening at the very start time rather than 10 minutes in advance, 
that was because the IETF Jabber server was not functional and we were 
holding as the issue was being fixed. I think Jay explained that in an 
email.

Regards,
Alessandro

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 19 Nov 2020 14:09:56 +0000

Meetecho IETF support wrote on 19/11/2020 14:04:
> Don't know what PIBKEC means, sorry.

"Problem Is Between Keyboard And Chair".  I.e. user error.

Nick

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 19 Nov 2020 15:38:10 +0000

> On 19 Nov 2020, at 12:52, [email-masked] wrote:
> 
>> Hi John
> 

… and the loss of jabber this morning?

Stewart

---

From: [email-masked]
Date: Thu, 19 Nov 2020 06:18:50 -0800 (PST)

> 	https://www.ietf.org/jabber/logs/emailcore/2020-11-17.html

Well... guilty as charged, I guess.

It didn't occur to me that I needed to perform a forensic analysis of the
jabber logs before reporting a problem, and that not having done meant no
response was warranted.

Having now looked at those logs, I note that these logs show me leaving at
8:34:13 to switch browsers, which of course I tried to do as quickly as
possible, so as to be able to enter the conversation in a timely way.

FYI, I normally join meetings by clicking on the URL in the calendar entry, but
since the IETF doesn't include A/V links in their invites - no doubt due to
sound technical reasons beyond my ken - I use the alternate approach of having
the agenda up in a window. I click on the meetecho link and when I'm done I hit
back. In this case I modified the procedure slightly: I clicked back, copied
the link, brought up the other browwer, and pasted the link there.

Now, you say the meeting ended at 08:36:42. I don't actually see that in the
log - it looks to me like it happened at 08:37:20, but I'll defer to your
expertise in these matters. That means the switching process took ~2.5 minutes.
That seems... long, but I suppose it's possible that a previously undiagnosed
medical condition caused me to have a absence seizure.

Or maybe the process was pretty slow.

It also didn't occur to me that someone would (incorrectly, according to both
my understanding and what you wrote above) say that the meeting was about to be
be shut down and everyone would abruptly terminate the active discussion and
disconnect. In effect creating a miniature cascade failure for me.

Of course I would also have seen this had I looked at the logs.

Mea maxima culpa.

All that said, what occurs to me now is that I'm clearly not competent to use
the system you provide and that my expectations for it, based as they are on
multi-hour daily use of Zoom, are out of line. You may rest assured that I
have learned my lession, and I won't be particpating in future IETF meetings,
online or in person, again.

				Ned