From nfs4-wg-request@sunroof.eng.sun.com Mon Jan 8 14:28:07 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA14390 for ; Mon, 8 Jan 2001 14:28:06 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA29525; Mon, 8 Jan 2001 11:25:18 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA23744; Mon, 8 Jan 2001 11:24:52 -0800 (PST) Received: from jurassic.eng.sun.com (jurassic [129.146.89.31] (may be forged)) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f08JOc601924 for ; Mon, 8 Jan 2001 11:24:38 -0800 (PST) Received: from eng.sun.com (terra.Eng.Sun.COM [129.146.86.99]) by jurassic.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f08JObv478566 for ; Mon, 8 Jan 2001 11:24:37 -0800 (PST) Sender: Brent.Callaghan@eng.sun.com Message-ID: <3A5A13F7.C9A28B43@eng.sun.com> Date: Mon, 08 Jan 2001 11:24:39 -0800 From: Brent Organization: NFS Whackers X-Mailer: Mozilla 4.75 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: nfsv4-wg@sunroof.eng.sun.com Subject: NFSv4 WG slides from IETF-49 available Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit All slides presented at the last NFS v4 working group meeting in San Diego are now available via: http://playground.sun.com/pub/nfsv4/presentations/ietf49 All in PDF. Brent From nfs4-wg-request@sunroof.eng.sun.com Sat Jan 13 06:05:54 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA16329 for ; Sat, 13 Jan 2001 06:05:53 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id DAA01209; Sat, 13 Jan 2001 03:03:27 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id CAA23321; Sat, 13 Jan 2001 02:47:06 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0DAkj606391 for ; Sat, 13 Jan 2001 02:46:46 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id CAA23297 for ; Sat, 13 Jan 2001 02:46:29 -0800 (PST) From: CableDescrambler@rffr.fsnet.co.uk Received: from jun.thrunet.com ([210.117.63.42]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id DAA10407 for ; Sat, 13 Jan 2001 03:46:24 -0700 (MST) Received: from 145x ([63.27.103.40]) by jun.thrunet.com with Microsoft SMTPSVC(5.5.1875.185.18); Sat, 13 Jan 2001 16:55:50 +0900 Message-ID: <000070455937$0000261d$00004a00@x165> To: Subject: UNIVERSAL CABLE DESCRAMBLER Date: Fri, 12 Jan 2001 23:16:39 -0500 MIME-Version: 1.0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Priority: 1 X-MSMail-Priority: High Reply-To: nomail116@yahoo.com Content-Transfer-Encoding: quoted-printable UNIVERSAL CABLE DESCRAMBLER100

UNIVERS= AL CABLE DESCRAMBLER

100% CLEAR PICTURE!
WORKS ON ALL CABLE SYSTEMS!

CLICK HERE FOR MORE INFO
OR ORDER BY PHONE!
1-507-263-0500 code 45<= /strong>

Regular Price $449.99, but with this special offer
It's Only $349.00 Delivered ($325 + Shippin= g)!


Note: Normal price is $449.99, unless you give operator discount code!

= Our Coolbox is a complete unit. Comes complete with:
* Next generation AVENGER dec= oder board
* New upgraded 125 channel, s= uper clear Toshiba Digital Tuner
* US made, Digital Image Deco= der
* RCA outputs for stereo soun= d
* RCA and Coax cables include= d
* Easy to use Remote Control = with manual fine tuning
* Volume control and mute rig= ht on the remote
* Parental Controls
* Sleep timer
* STD, HRC, IRC and 1-Button = Video Inversion
* Channel Scan
* Last channel recall
* Favorite channel memory * Channel 3 or 4 output
* Bulletproof and non-address= able
* Inversion memory
* Complete easy to use Owners= manual with Quick Start setup guide
* 15 Day money back guarantee=
* 1 year Limited warranty

CLICK HERE FOR MORE INFO
OR ORDER BY PHONE!
1-507-263= -0500 code 45


_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
If you have received this message in error and would
like to be removed from future mailings, please reply
with the word remove in the subject.
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 17 08:37:22 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA17307 for ; Wed, 17 Jan 2001 08:37:22 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id FAA13300; Wed, 17 Jan 2001 05:35:46 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id FAA13845; Wed, 17 Jan 2001 05:35:29 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0HDZF614609; Wed, 17 Jan 2001 05:35:15 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id FAA13814; Wed, 17 Jan 2001 05:35:16 -0800 (PST) Received: from dns1.artoffice.co.jp ([210.251.116.130]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id FAA13050; Wed, 17 Jan 2001 05:35:13 -0800 (PST) Received: from host (cranston-ip-1-175.dynamic.ziplink.NET [209.206.4.175]) by dns1.artoffice.co.jp (2.5 Build 2640 (Berkeley 8.8.6)/8.8.4) with ESMTP id WAA09405; Wed, 17 Jan 2001 22:25:37 +0900 Message-Id: <200101171325.WAA09405@dns1.artoffice.co.jp> From: "Victor Barret" Subject: Candidate #231A To: join39d@dns1.artoffice.co.jp X-Mailer: Microsoft Outlook Express 4.72.1712.3 X-MimeOLE: Produced By Microsoft MimeOLE VÐßD.1712.3 Mime-Version: 1.0 Date: Wed, 17 Jan 2001 07:53:06 -0500 Content-Type: multipart/mixed; boundary="----=_NextPart_000_007F_01BDF6C7.FABAC1B0" This is a MIME Message ------=_NextPart_000_007F_01BDF6C7.FABAC1B0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0080_01BDF6C7.FABAC1B0" ------=_NextPart_001_0080_01BDF6C7.FABAC1B0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ***** This is an HTML Message ! ***** ------=_NextPart_001_0080_01BDF6C7.FABAC1B0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Executive Guild Membership ApplicationResponse-O-Matic Form

Dear Candidate,

You have been selected for a complimentary listing in the International Executive Guild Registry=2E Inclusion is limited to Executives,
Entrepreneurs, Business Owners and Professionals=2E

The International Executive Guild's 2001 - 2002 edition will be
published in two different formats; the searchable CD-ROM and Online
Registry=2E The CD-ROM is searchable by over twenty parameters, with
thousands of complete profiles of highly accomplished individuals from virtually every industry=2E

Since inclusion can be considered recognition of your career position, each candidate is evaluated in keeping with high standards of individual<= br> achievement=2E The International Executive Guild looks for experts in
= their respective field=2E

We look forward to your inclusion and appearance in the International
= Executive Guild's Registry=2E Best for your continued success=2E

International Executive Guild
Listing Department

If you wish to be removed from our list, please submit your request
at the bottom of this email=2E

International Executive Guild
Registration Form
(US and Canada Only)
Please fill out this form if you would like to be included on The International Executive Guild, For accuracy and publication purposes, please complete and send this form at the earliest opportunity=2E There is no charge or obligation to be listed on The International Executive Guild=2E
Your Name
Your Company
Title
Address
City
State or Province
Country
ZIP/Postal Code
Day Time Telephone
Home Phone
 (Not To Be Published)
Email

TO HELP US IN CONSIDERING YOUR APPLICATION, PLEASE TELL US A LITTLE ABOUT YOURSELF=2E=2E=2E
Your Business
(Financial Svcs, Banking, Computer Hardware, Software, Professional Svcs, Chemicals, Apparel, Aerospace, Food, Government, Utility, etc=2E)
Type of Organization
(M= fg, Dist/Wholesaler, Retailer, Law Firm,
Investment Bank, Commercial Bank, University,
Financial Consultants, Ad Agency, Contractor, Broker, etc=2E)
Your Business Expertise
(Corp=2EMgmt, Marketing, Civil Engineering,
Tax Law, Nuclear Physics, Database Development, Operations, Pathologist, Mortgage Banking, etc=2E)
Major Product Line
(Integrated Circuits, Commercial Aircraft, Adhesives, Cosmetics, Plastic Components, Snack Foods, etc=2E)


Note: Submitting this form= will be made by email, not by use of  www=2E  Confirmation of its de= livery is made by browsing your outgoing mail=2E

Thank you for filling in this form, we will contact you with more information=2E

List Removal
Click Here
------=_NextPart_001_0080_01BDF6C7.FABAC1B0-- ------=_NextPart_000_007F_01BDF6C7.FABAC1B0-- From nfs4-wg-request@sunroof.eng.sun.com Thu Jan 18 20:52:10 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA07969 for ; Thu, 18 Jan 2001 20:52:10 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA15210; Thu, 18 Jan 2001 17:50:24 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA12550; Thu, 18 Jan 2001 17:50:00 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0J0pSN01306 for ; Thu, 18 Jan 2001 16:51:29 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA11425 for ; Thu, 18 Jan 2001 15:05:25 -0800 (PST) Received: from eagle.webpros.com (eagle.professionals.com [206.127.192.10]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA16824 for ; Thu, 18 Jan 2001 15:05:26 -0800 (PST) Received: from frostback ([206.27.132.50]) by eagle.webpros.com (8.8.7/8.6.10) with SMTP id PAA23112 for ; Thu, 18 Jan 2001 15:05:01 -0800 (PST) Message-Id: <200101182305.PAA23112@eagle.webpros.com> Date: Thu, 18 Jan 2001 15:02:46 -0800 (PST) From: Mike Eisler Reply-To: Mike Eisler Subject: UNCHECKED mode and permission bits To: nfsv4-wg@sunroof.eng.sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: EX/OMOExYe/hbuswM35/Vw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc All RFCs for all three NFS versions say that when a file is created in non-exclusive, non-guarded mode (and in NFSv2, that's the only way to create a file) that attributes are applied to file if it exists. I find it hard to beleive that NFS servers actually modify permission bits or the group, or the ACL. Nor do I believe that's what is intended. Certainly, my simple experiments with the creat() system call against local files on UNIX platforms supports my disbelief. Looking at the Solaris NFS server code, or more correctly, the code for the create interface to the ufs filesystem, it does not appear that it implements to the NFSv2 and NFSv3 RFCs in this regard. But I think the RFCs are wrong, as is the NFSv4 RFC. If I understand things correctly, in practice, only the size attribute in a non-exclusive CREATE gets applied to the existing file. Since the NFSv4 RFC will be replaced, it might be good to clarify this, since I doubt NFSv4 servers will comply with current RFC. -mre From nfs4-wg-request@sunroof.eng.sun.com Thu Jan 18 21:58:37 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA09498 for ; Thu, 18 Jan 2001 21:58:36 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA02098; Thu, 18 Jan 2001 18:51:43 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA16034; Thu, 18 Jan 2001 18:51:20 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0J1qRN01879 for ; Thu, 18 Jan 2001 17:52:28 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA15974 for ; Thu, 18 Jan 2001 18:50:41 -0800 (PST) Received: from donkeykong.gpcc.itd.umich.edu (donkeykong.gpcc.itd.umich.edu [141.211.2.163]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id TAA25178 for ; Thu, 18 Jan 2001 19:50:40 -0700 (MST) Received: from galaxian.gpcc.itd.umich.edu (smtp@galaxian.gpcc.itd.umich.edu [141.211.2.146]) by donkeykong.gpcc.itd.umich.edu (8.8.8/4.3-mailhub) with ESMTP id VAA18815; Thu, 18 Jan 2001 21:50:39 -0500 (EST) Received: from localhost (kmsmith@localhost) by galaxian.gpcc.itd.umich.edu (8.8.8/5.1-client) with ESMTP id VAA07413; Thu, 18 Jan 2001 21:50:39 -0500 (EST) Precedence: first-class Date: Thu, 18 Jan 2001 21:50:38 -0500 (EST) From: "Kendrick M. Smith" X-Sender: kmsmith@galaxian.gpcc.itd.umich.edu To: Mike Eisler cc: nfsv4-wg@sunroof.eng.sun.com Subject: Re: UNCHECKED mode and permission bits In-Reply-To: <200101182305.PAA23112@eagle.webpros.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII > Looking at the Solaris NFS server code, or more correctly, the code for > the create interface to the ufs filesystem, it does not appear that it > implements to the NFSv2 and NFSv3 RFCs in this regard. But I think the > RFCs are wrong, as is the NFSv4 RFC. If I understand things correctly, > in practice, only the size attribute in a non-exclusive CREATE gets > applied to the existing file. Since the NFSv4 RFC will be > replaced, it might be good to clarify this, since I doubt NFSv4 servers > will comply with current RFC. doesn't the spec already say this? from p. 144 of the draft7 spec, "When an UNCHECKED create encounters an existing file, the attributes specified by createattrs is not used, except that when an object_size of zero is specified, the existing file is truncated." - k From nfs4-wg-request@sunroof.eng.sun.com Fri Jan 19 00:02:58 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA11850 for ; Fri, 19 Jan 2001 00:02:57 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id VAA26035; Thu, 18 Jan 2001 21:00:41 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id VAA05018; Thu, 18 Jan 2001 21:00:19 -0800 (PST) Received: from engmail1.Eng.Sun.COM (engmail1 [129.146.1.13]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0J41mN02768 for ; Thu, 18 Jan 2001 20:01:49 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id VAA04916 for ; Thu, 18 Jan 2001 21:00:01 -0800 (PST) Received: from emc.com (emcmail.lss.emc.com [168.159.48.78]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id VAA18195 for ; Thu, 18 Jan 2001 21:00:01 -0800 (PST) Received: from emc.com ([172.25.208.40]) by emc.com (8.10.1/8.10.1) with ESMTP id f0J4xM612995; Thu, 18 Jan 2001 23:59:22 -0500 (EST) Message-ID: <3A67C9B4.3EE55D64@emc.com> Date: Thu, 18 Jan 2001 23:59:32 -0500 From: Uresh Vahalia X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Kendrick M. Smith" CC: Mike Eisler , nfsv4-wg@sunroof.eng.sun.com Subject: Re: UNCHECKED mode and permission bits References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit That is different from Kendrick's statement and, I believe, from current practice in v2/v3. If a non-zero file size is specified in an unchecked create for an existing file, I believe many of today's servers would truncate the file to the specified size. YMMV. "Kendrick M. Smith" wrote: > > Looking at the Solaris NFS server code, or more correctly, the code for > > the create interface to the ufs filesystem, it does not appear that it > > implements to the NFSv2 and NFSv3 RFCs in this regard. But I think the > > RFCs are wrong, as is the NFSv4 RFC. If I understand things correctly, > > in practice, only the size attribute in a non-exclusive CREATE gets > > applied to the existing file. Since the NFSv4 RFC will be > > replaced, it might be good to clarify this, since I doubt NFSv4 servers > > will comply with current RFC. > > doesn't the spec already say this? from p. 144 of the draft7 spec, > > "When an UNCHECKED create encounters an existing file, the > attributes specified by createattrs is not used, except that when > an object_size of zero is specified, the existing file is > truncated." > > - k From nfs4-wg-request@sunroof.eng.sun.com Fri Jan 19 12:17:37 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA02770 for ; Fri, 19 Jan 2001 12:17:37 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA28836; Fri, 19 Jan 2001 09:15:09 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id JAA06959; Fri, 19 Jan 2001 09:06:39 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0JH6MN05716; Fri, 19 Jan 2001 09:06:23 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id JAA06915; Fri, 19 Jan 2001 09:06:21 -0800 (PST) From: emed11@libero.it Received: from cins.hanyang.ac.kr ([166.104.204.187]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA21523; Fri, 19 Jan 2001 09:06:12 -0800 (PST) Received: by cins.hanyang.ac.kr id BAA422693; Sat, 20 Jan 2001 01:40:52 +0900 (KST) Subject: Obtain Biotech IPOs! 138 Date: Fri, 19 Jan 2001 12:04:18 Message-Id: <306.381214.526372@libero.it> Reply-To: emed11@libero.it Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by cins.hanyang.ac.kr id BAA422693 To: undisclosed-recipients:; Content-Transfer-Encoding: quoted-printable

Help Beta Test Our Site and Be Eligible to Purchase Shares= of=20 Future IPOs In Which We Participate**

eMedsecurities has selected you as a possible participant = to help=20 test our online stock-trading engine for knowledge-based investing = in the=20 life sciences.  For your cooperation, you will be eligible = to purchase=20 shares of future IPOs in which we participate, for as long as y= ou=20 maintain your account with us.  This is limited to only 50 = qualified=20 testers!  Request Mor= e Information.**

eMedsecurities=85 The Cure for the Common Portfolio!

eMedsecurities provides you with a wealth of information, = all=20 compiled in a single, easy-to-use resource.  Learn about new r= esearch=20 and upcoming treatments for migraine headache, stroke, epilepsy, br= ain=20 tumor and much more.  Obtain critical investment information a= bout the=20 companies that are developing these treatments.  eMedsecuritie= s empowers=20 you to make more informed investment decisions.  Request More Information.**

Participation i= n eMedsecurities' Beta Test allows you:

  • Eligibility to purchase s= hares of IPOs in which=20 eMedsecurities participates for as long as you maintain your=20 account**

  • Valuable research of the = entire product=20 pipeline of companies, including stages of clinical development, = by=20 industry or specific disease

  • Useful information about = industry trends, recent=20 developments and upcoming IPOs

  • Commitment to customer se= rvice featuring=20 our Live Customer Service Online

  • Dedication to fast trade = executions at the best=20 possible price

The following guideline= s will=20 explain what we expect from an eMedsecurities Beta Tester:

  • Open a funded eMedsecurities=20 account

  • Visit our online trading site onc= e=20 a week

  • Execute trades through our web=20 site in accordance with your normal practice

  • Submit feedback to eMedsecurities= ' development team through a questionnaire sent via=20 email

  • Provide us with additional=20 feedback regarding the site as needed

The test is limited to = only 50 Beta=20 Testers so sign up now to be considered!  Request More Information.**=20

Please note:<= /B> All applications for the Beta Test must be s= ubmitted by January 24, 2001 to be considered.=20

Please=20 be advised that your information will stay in our proprietary datab= ase and=20 will not be sold, traded, given or otherwise provided to outside ve= ndors.  We respect your privacy.=20

=20

By=20 submitting your information, you implicitly state that this is some= thing=20 that interests you and that you
agree to receive periodic emails from=20 eMedsecurities.=20

Research indicated= that you=20 might benefit from our offer.  To be removed instantly and per= manently from=20 our database, simply click here= .  We respect=20 all removal requests.

** R= estrictions=20 Apply:  Beta test not open t= o residents of:  HI, IL, MI, MN, MS, NE, NH, TN, TX.  Initial Public= =20 Offerings are considered speculative investments and as such may no= t be=20 appropriate for every investor. If an investor chooses to participa= te in=20 IPOs, there are certain restrictions that apply:  Flipping- The first time an investo= r sells=20 his/her shares within the first 30 days the issue is trading in the= =20 secondary market, that investor will not be allocated shares for th= e next=20 90 days following the sale.  The second time that investor =93= flips=94,=20 they will not be allocated IPO shares for 180 days.  The third= time=20 that investor =93flips=94, they lose their IPO allocations permanen= tly.  Transferring shares- <= /B>If the=20 investor transfers IPO shares out of their account within the first= 30=20 days the issue is trading in the secondary market, they will perman= ently=20 lose their IPO allocations.  Beta investors will be chosen = from=20 all the applicants based on their income, net worth and investing=20 experience.  IPO shares will only be allocated from transa= ctions in=20 which eMedsecurities participates in the underwriting. A0007-1-A2=20
From nfs4-wg-request@sunroof.eng.sun.com Fri Jan 19 13:21:17 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA03714 for ; Fri, 19 Jan 2001 13:21:17 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA18441; Fri, 19 Jan 2001 10:16:58 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id KAA15503; Fri, 19 Jan 2001 10:16:22 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0JIG0N06122 for ; Fri, 19 Jan 2001 10:16:01 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id KAA15348 for ; Fri, 19 Jan 2001 10:15:37 -0800 (PST) Received: from eagle.webpros.com (eagle.professionals.com [206.127.192.10]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA11805 for ; Fri, 19 Jan 2001 10:15:38 -0800 (PST) Received: from frostback ([206.27.132.50]) by eagle.webpros.com (8.8.7/8.6.10) with SMTP id KAA21504; Fri, 19 Jan 2001 10:15:10 -0800 (PST) Message-Id: <200101191815.KAA21504@eagle.webpros.com> Date: Fri, 19 Jan 2001 10:12:54 -0800 (PST) From: Mike Eisler Reply-To: Mike Eisler Subject: Re: UNCHECKED mode and permission bits To: kmsmith@umich.edu, vahalia@emc.com Cc: nfsv4-wg@sunroof.eng.sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: qGH7aFA/fk03BLil6VJN/A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc >"Kendrick M. Smith" wrote: > >> > Looking at the Solaris NFS server code, or more correctly, the code for >> > the create interface to the ufs filesystem, it does not appear that it >> > implements to the NFSv2 and NFSv3 RFCs in this regard. But I think the >> > RFCs are wrong, as is the NFSv4 RFC. If I understand things correctly, >> > in practice, only the size attribute in a non-exclusive CREATE gets >> > applied to the existing file. Since the NFSv4 RFC will be >> > replaced, it might be good to clarify this, since I doubt NFSv4 servers >> > will comply with current RFC. >> >> doesn't the spec already say this? from p. 144 of the draft7 spec, >> >> "When an UNCHECKED create encounters an existing file, the >> attributes specified by createattrs is not used, except that when >> an object_size of zero is specified, the existing file is >> truncated." Indeed it does. Odd, I'd looked at that section in the RFC and missed it before I delivered my rant. Uresh Vahalia writes, in response to Kendrick M. Smith: >That is different from Kendrick's statement and, I believe, from current >practice in v2/v3. If a non-zero file size is specified in an unchecked >create for an existing file, I believe many of today's servers would truncate >the file to the specified size. YMMV. Not all file systems will support truncation to arbitrary sizes greather than zero. But I agreed that the NFSv4 language appears to be unnecessarily restrictive in this regard, especially since there is apaprently nothign special about an object size of zero when doing exclusive OPENs. Not that there are that many client implementations that create files with non-zero sizes. -mre From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 23 16:13:54 2001 Received: from mercury.Sun.COM ([192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA24743 for ; Tue, 23 Jan 2001 16:13:54 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA02550; Tue, 23 Jan 2001 13:12:43 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id NAA02891; Tue, 23 Jan 2001 13:10:43 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0NLAU213024 for ; Tue, 23 Jan 2001 13:10:30 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id NAA23251 for ; Tue, 23 Jan 2001 13:10:28 -0800 (PST) Received: from berzerk.gpcc.itd.umich.edu (berzerk.gpcc.itd.umich.edu [141.211.2.162]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA14678 for ; Tue, 23 Jan 2001 13:10:27 -0800 (PST) Received: from frogger.gpcc.itd.umich.edu (smtp@frogger.gpcc.itd.umich.edu [141.211.2.144]) by berzerk.gpcc.itd.umich.edu (8.8.8/4.3-mailhub) with ESMTP id QAA29618 for ; Tue, 23 Jan 2001 16:10:26 -0500 (EST) Received: from localhost (kmsmith@localhost) by frogger.gpcc.itd.umich.edu (8.8.8/5.1-client) with ESMTP id QAA24788 for ; Tue, 23 Jan 2001 16:10:25 -0500 (EST) Precedence: first-class Date: Tue, 23 Jan 2001 16:10:25 -0500 (EST) From: "Kendrick M. Smith" X-Sender: kmsmith@frogger.gpcc.itd.umich.edu To: nfsv4-wg@sunroof.eng.sun.com Subject: filehandle volatility, stateid, and "state leakage" Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Looking back at the post-bakeoff mailing list postings, it seems to me that we never reached consensus on whether stateids should be per lockowner or per (lockowner,file)-pair. The last mention I see of this is in Mike Eisler's message dated 11/03, detailing changes that should be made in the spec if it is decided that stateids should be per lockowner. Since those changes have not yet been made, is it safe to assume that stateids will remain per (lockowner,file)-pair? My reason for bringing this up is related to a problem that I have run into in our client implementation. Consider this scenario: 1. Client A opens a file. 2. Client A acquires locks on the file. 3. Client B renames the file. Because the filehandles of this server are volatile on rename, this changes the file's filehandle. 4. Client A tries to release its locks. In step 4, the client must issue a PUTFH for the file, since for a LOCKU operation, the spec requires the current filehandle to match the file being unlocked. Because the filehandle changed in step 3, the PUTFH will fail with error NFS4ERR_BADHANDLE. Futhermore, the client will not be able to recover the new filehandle, since it has no way of knowing the new filename specified by client B in step 3. Therefore it will never be able to release its locks on the file (or close it, since the CLOSE operation has the same filehandle restriction as the LOCKU operation). This "state leakage" will persist for the lifetime of the client (assuming that the client periodically emits RENEW operations)! One possible solution to this problem is to add the restriction that, while a file has been opened on the server, the server cannot change its filehandle. For our Linux server, this would be quite problematic; our filehandles have to be volatile on rename (in case anyone is curious why, I've summarized the reasons in a note at the end of this message). Another solution, which I think is the "right" way to handle this issue, is: 1. Agree that stateids are per (lockowner,file)-pair, rather than per lockowner. This means that, in a LOCKU or CLOSE operation, the stateid provided by the client is sufficient to specify the file; the current filehandle would be redundant. 2. Remove the restriction for the spec that, in a CLOSE or LOCKU operation, the current filehandle has to be set to the opened file. The stateid specified with the operation now identifies the file. What does everyone think of this? Is there a reason I'm unaware of for requiring the current filehandle to be set in a CLOSE or LOCKU operation? This proposal would give the NFSv4 stateid similar semantics to a UNIX file descriptor; in UNIX, a file descriptor can be used to refer to a file uniquely for as long as it remains open, even if the filename changes. This proposal would let the NFSv4 stateid refer to a file uniquely for as long as it remains open, even if the filehandle changes (although the analogy between stateids and file descriptors isn't perfect, since the stateid can change, whereas a file descriptor is constant). If the only reason that the filehandle is required to agree with the open file during a CLOSE or LOCKU is to give the server an error check, I personally think that the possible benefits of this additional check would be outweighed by the problems created by this "state leakage" issue. After all, the server can still do error-checking on the stateid, and determine if it is stale, bad, or old. Feedback? Kendrick p.s. In case anyone is wondering why our Linux server's filehandles have to remain volatile, here is an explanation. When a client sends a filehandle, we have to verify for security purposes that the file is contained in one of the directories approved for export on the server. If the file is a directory, this is easy; we just traverse parent directories until we find either an export, or the root of the filesystem. If the file is a non-directory, however, we have a problem; we can't uniquely determine the directory containing it, since it may be hardlinked into several directories on the server. At best, we could traverse a list of directories containing the file, to try to find one contained in an export (which would present a performance problem), but even this is impossible since the Linux low-level fs interfaces don't provide a way to obtain a list of all directories containing a given (non-directory) file. Therefore, our filehandles have to include, in addition to the inode number of the file, the inode number of its containing directory, so that we can locate the parent directory for export checking. When the file is renamed, its parent directory may also change, which means the filehandle can change. Our volatile filehandles are therefore a by-product of a server environment in which (1) we don't want to export the entire filesystem, only certain directories, and (2) because of hardlinks, it is not possible to determine the directory containing a given file. From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 23 17:08:26 2001 Received: from mercury.Sun.COM ([192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25381 for ; Tue, 23 Jan 2001 17:08:25 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id OAA24798; Tue, 23 Jan 2001 14:07:09 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id OAA15263; Tue, 23 Jan 2001 14:05:27 -0800 (PST) Received: from engmail1.Eng.Sun.COM (engmail1 [129.146.1.13]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0NM5F213096 for ; Tue, 23 Jan 2001 14:05:15 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id OAA15169 for ; Tue, 23 Jan 2001 14:05:14 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id OAA18614 for ; Tue, 23 Jan 2001 14:05:13 -0800 (PST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0NM5DL16565; Tue, 23 Jan 2001 14:05:13 -0800 (PST) Received: from ussvlexc06.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0NM5Do10911; Tue, 23 Jan 2001 14:05:13 -0800 (PST) Received: by ussvlexc06.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 23 Jan 2001 14:06:03 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E39@red.nane.netapp.com> From: "Noveck, Dave" To: "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com Subject: RE: filehandle volatility, stateid, and "state leakage" Date: Tue, 23 Jan 2001 14:05:09 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C08588.AC127A74" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C08588.AC127A74 Content-Type: text/plain; charset="iso-8859-1" Kendrick Smith wrote: > Looking back at the post-bakeoff mailing list postings, it seems to me > that we never reached consensus on whether stateids should be per > lockowner or per (lockowner,file)-pair. I don't think we have reached consensus on any of the major outstanding issues: stateid scope, are 64-bit stateid's big enough, and the NT server vs. open-downgrade imbroglio. There was a flurry of discussion and then nothing on the mailing list. I suspect (or hope) that it's basically a case of everybody being too busy coding nfs-v4 implementations to follow up. > The last mention I see of this is > in Mike Eisler's message dated 11/03, detailing changes that should be > made in the spec if it is decided that stateids should be per lockowner. > Since those changes have not yet been made, is it safe to assume that > stateids will remain per (lockowner,file)-pair? Since the current RFC embodies both interpretations (in different places), I don't think any assumption is safe right now, if you are talking about the ultimate resolution. OTOH, my server has (lockowner,file)-pair stateid's and it won't change for Connectathon unless a decision to go to per-lockowner stateid's happens within the next few weeks. I'm assuming that is true for a lot of implementations, so staying with with per (lockowner,file)-pair stateid's is probably safe for Connectathon, at least. > > My reason for bringing this up is related to a problem that I have run > into in our client implementation. Consider this scenario: > 1. Client A opens a file. > 2. Client A acquires locks on the file. > 3. Client B renames the file. Because the filehandles of this server > are volatile on rename, this changes the file's filehandle. > 4. Client A tries to release its locks. > In step 4, the client must issue a PUTFH for the file, since for a LOCKU > operation, the spec requires the current filehandle to match the file > being unlocked. Because the filehandle changed in step 3, the PUTFH will > fail with error NFS4ERR_BADHANDLE. Futhermore, the client will not be > able to recover the new filehandle, since it has no way of knowing the > new filename specified by client B in step 3. Therefore it will never be > able to release its locks on the file (or close it, since the CLOSE > operation has the same filehandle restriction as the LOCKU operation). > This "state leakage" will persist for the lifetime of the client (assuming > that the client periodically emits RENEW operations)! I feel a volatile filehandle tirade coming on, but I suppose I can choose not to commit it to writing. :-) > One possible solution to this problem is to add the restriction that, > while a file has been opened on the server, the server cannot change its > filehandle. For our Linux server, this would be quite problematic; our > filehandles have to be volatile on rename (in case anyone is curious > why, I've summarized the reasons in a note at the end of this message). > > Another solution, which I think is the "right" way to handle this issue, > is: > 1. Agree that stateids are per (lockowner,file)-pair, rather than per > lockowner. This means that, in a LOCKU or CLOSE operation, the stateid > provided by the client is sufficient to specify the file; the current > filehandle would be redundant. Only for LOCKU and CLOSE, I suppose. For READ and WRITE, this would not work, since you are allowed to specify the special stateid's which don't identify the file. > 2. Remove the restriction for the spec that, in a CLOSE or LOCKU > operation, the current filehandle has to be set to the opened file. The > stateid specified with the operation now identifies the file. That wouldn't be unprecedented. RENEW takes a stateid but no filehandle is mentioned in the spec (even the current filehandle). My server at least uses the stateid and doesn't look at the current filehandle for RENEW. > > What does everyone think of this? Is there a reason I'm unaware of for > requiring the current filehandle to be set in a CLOSE or LOCKU operation? My guess is that it just seemed natural to always pass the filehandle. In cases in which special stateid's may not be used, the server has to be able to determine what file you are talking about, so I don't see that requiring it explicitly is adding anything. > This proposal would give the NFSv4 stateid similar semantics to a UNIX > file descriptor; in UNIX, a file descriptor can be used to refer to a file > uniquely for as long as it remains open, even if the filename changes. > This proposal would let the NFSv4 stateid refer to a file uniquely for as > long as it remains open, even if the filehandle changes (although the > analogy between stateids and file descriptors isn't perfect, since the > stateid can change, whereas a file descriptor is constant). Also, you can't read and write with stateid's alone, unlike file descriptors. > > If the only reason that the filehandle is required to agree with the open > file during a CLOSE or LOCKU is to give the server an error check, I > personally think that the possible benefits of this additional check would > be outweighed by the problems created by this "state leakage" issue. > After all, the server can still do error-checking on the stateid, and > determine if it is stale, bad, or old. Feedback? > > Kendrick > > p.s. In case anyone is wondering why our Linux server's filehandles have > to remain volatile, here is an explanation. When a client sends a > filehandle, we have to verify for security purposes that the file is > contained in one of the directories approved for export on the server. > If the file is a directory, this is easy; we just traverse parent > directories until we find either an export, or the root of the filesystem. > If the file is a non-directory, however, we have a problem; we can't > uniquely determine the directory containing it, since it may be hardlinked > into several directories on the server. At best, we could traverse a list > of directories containing the file, to try to find one contained in an > export (which would present a performance problem), but even this is > impossible since the Linux low-level fs interfaces don't provide a way to > obtain a list of all directories containing a given (non-directory) file. > Therefore, our filehandles have to include, in addition to the inode > number of the file, the inode number of its containing directory, so that > we can locate the parent directory for export checking. When the file is > renamed, its parent directory may also change, which means the filehandle > can change. Our volatile filehandles are therefore a by-product of a > server environment in which (1) we don't want to export the entire > filesystem, only certain directories, and (2) because of hardlinks, it is > not possible to determine the directory containing a given file. > > > ------_=_NextPart_001_01C08588.AC127A74 Content-Type: text/html; charset="iso-8859-1" RE: filehandle volatility, stateid, and "state leakage"

Kendrick Smith wrote:
> Looking back at the post-bakeoff mailing list postings, it seems to me
> that we never reached consensus on whether stateids should be per
> lockowner or per (lockowner,file)-pair.

I don't think we have reached consensus on any of the major outstanding
issues: stateid scope, are 64-bit stateid's big enough, and the NT server
vs. open-downgrade imbroglio.  There was a flurry of discussion and then
nothing on the mailing list.  I suspect (or hope) that it's basically a
case of everybody being too busy coding nfs-v4 implementations to follow
up.  

> The last mention I see of this is
> in Mike Eisler's message dated 11/03, detailing changes that should be
> made in the spec if it is decided that stateids should be per lockowner. 
> Since those changes have not yet been made, is it safe to assume that
> stateids will remain per (lockowner,file)-pair?

Since the current RFC embodies both interpretations (in different places),
I don't think any assumption is safe right now, if you are talking about
the ultimate resolution.

OTOH, my server has (lockowner,file)-pair stateid's and it won't change for
Connectathon unless a decision to go to per-lockowner stateid's happens
within the next few weeks.  I'm assuming that is true for a lot of
implementations, so staying with with per (lockowner,file)-pair stateid's
is probably safe for Connectathon, at least.
 
>
> My reason for bringing this up is related to a problem that I have run
> into in our client implementation.  Consider this scenario:
>   1.  Client A opens a file.
>   2.  Client A acquires locks on the file.
>   3.  Client B renames the file.  Because the filehandles of this server
> are volatile on rename, this changes the file's filehandle.
>   4.  Client A tries to release its locks.
> In step 4, the client must issue a PUTFH for the file, since for a LOCKU
> operation, the spec requires the current filehandle to match the file
> being unlocked.  Because the filehandle changed in step 3, the PUTFH will
> fail with error NFS4ERR_BADHANDLE.  Futhermore, the client will not be
> able to recover the new filehandle, since it has no way of knowing the
> new filename specified by client B in step 3.  Therefore it will never be
> able to release its locks on the file (or close it, since the CLOSE
> operation has the same filehandle restriction as the LOCKU operation).
> This "state leakage" will persist for the lifetime of the client (assuming
> that the client periodically emits RENEW operations)!

I feel a volatile filehandle tirade coming on, but I suppose I can choose
not to commit it to writing. :-)

> One possible solution to this problem is to add the restriction that,
> while a file has been opened on the server, the server cannot change its
> filehandle.  For our Linux server, this would be quite problematic; our
> filehandles have to be volatile on rename (in case anyone is curious
> why, I've summarized the reasons in a note at the end of this message).
>
> Another solution, which I think is the "right" way to handle this issue,
> is:
>   1.  Agree that stateids are per (lockowner,file)-pair, rather than per
> lockowner.  This means that, in a LOCKU or CLOSE operation, the stateid
> provided by the client is sufficient to specify the file; the current
> filehandle would be redundant.

Only for LOCKU and CLOSE, I suppose.  For READ and WRITE, this would
not work, since you are allowed to specify the special stateid's
which don't identify the file.

>   2.  Remove the restriction for the spec that, in a CLOSE or LOCKU
> operation, the current filehandle has to be set to the opened file.  The
> stateid specified with the operation now identifies the file.

That wouldn't be unprecedented.  RENEW takes a stateid but no filehandle
is mentioned in the spec (even the current filehandle).  My server
at least uses the stateid and doesn't look at the current filehandle
for RENEW.
 
>
> What does everyone think of this?  Is there a reason I'm unaware of for
> requiring the current filehandle to be set in a CLOSE or LOCKU operation?

My guess is that it just seemed natural to always pass the filehandle.
In cases in which special stateid's may not be used, the server has
to be able to determine what file you are talking about, so I don't
see that requiring it explicitly is adding anything.
 
> This proposal would give the NFSv4 stateid similar semantics to a UNIX
> file descriptor; in UNIX, a file descriptor can be used to refer to a file
> uniquely for as long as it remains open, even if the filename changes.
> This proposal would let the NFSv4 stateid refer to a file uniquely for as
> long as it remains open, even if the filehandle changes (although the
> analogy between stateids and file descriptors isn't perfect, since the
> stateid can change, whereas a file descriptor is constant).

Also, you can't read and write with stateid's alone, unlike file
descriptors.
 
>
> If the only reason that the filehandle is required to agree with the open
> file during a CLOSE or LOCKU is to give the server an error check, I
> personally think that the possible benefits of this additional check would
> be outweighed by the problems created by this "state leakage" issue. 
> After all, the server can still do error-checking on the stateid, and
> determine if it is stale, bad, or old.  Feedback?
>
> Kendrick
>
> p.s.  In case anyone is wondering why our Linux server's filehandles have
> to remain volatile, here is an explanation.  When a client sends a
> filehandle, we have to verify for security purposes that the file is
> contained in one of the directories approved for export on the server. 
> If the file is a directory, this is easy; we just traverse parent
> directories until we find either an export, or the root of the filesystem.
> If the file is a non-directory, however, we have a problem; we can't
> uniquely determine the directory containing it, since it may be hardlinked
> into several directories on the server.  At best, we could traverse a list
> of directories containing the file, to try to find one contained in an
> export (which would present a performance problem), but even this is
> impossible since the Linux low-level fs interfaces don't provide a way to
> obtain a list of all directories containing a given (non-directory) file.
> Therefore, our filehandles have to include, in addition to the inode
> number of the file, the inode number of its containing directory, so that
> we can locate the parent directory for export checking.  When the file is
> renamed, its parent directory may also change, which means the filehandle
> can change.  Our volatile filehandles are therefore a by-product of a
> server environment in which (1) we don't want to export the entire
> filesystem, only certain directories, and (2) because of hardlinks, it is
> not possible to determine the directory containing a given file.
>
>
>

------_=_NextPart_001_01C08588.AC127A74-- From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 23 18:05:37 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA26101 for ; Tue, 23 Jan 2001 18:05:37 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA27880; Tue, 23 Jan 2001 15:04:43 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA28659; Tue, 23 Jan 2001 15:03:06 -0800 (PST) Received: from jurassic.eng.sun.com (jurassic [129.146.81.144] (may be forged)) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0NN2p213285 for ; Tue, 23 Jan 2001 15:02:51 -0800 (PST) Received: from pil (pil.Eng.Sun.COM [129.146.86.124]) by jurassic.eng.sun.com (8.11.2+Sun/8.11.2) with SMTP id f0NN2o9506843 for ; Tue, 23 Jan 2001 15:02:51 -0800 (PST) Message-Id: <200101232302.f0NN2o9506843@jurassic.eng.sun.com> Date: Tue, 23 Jan 2001 15:03:03 -0800 (PST) From: Salit Gazit Reply-To: Salit Gazit Subject: Re: filehandle volatility, stateid, and "state leakage" To: nfsv4-wg@sunroof.eng.sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: NaS1j1vHlukK4hh31fFh9g== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Kendrick, >Looking back at the post-bakeoff mailing list postings, it seems to me >that we never reached consensus on whether stateids should be per >lockowner or per (lockowner,file)-pair. The last mention I see of this is >in Mike Eisler's message dated 11/03, detailing changes that should be >made in the spec if it is decided that stateids should be per lockowner. >Since those changes have not yet been made, is it safe to assume that >stateids will remain per (lockowner,file)-pair? > >My reason for bringing this up is related to a problem that I have run >into in our client implementation. Consider this scenario: > 1. Client A opens a file. > 2. Client A acquires locks on the file. > 3. Client B renames the file. Because the filehandles of this server >are volatile on rename, this changes the file's filehandle. > 4. Client A tries to release its locks. >In step 4, the client must issue a PUTFH for the file, since for a LOCKU >operation, the spec requires the current filehandle to match the file >being unlocked. Because the filehandle changed in step 3, the PUTFH will >fail with error NFS4ERR_BADHANDLE. The error should be NFS4ERR_FHEXPIRED to indicate that the filehandle expired so the client can try to look it up (but I assume that the server probably does not know that it expired when it gets the old filehandle?). Obviously if the path to it was changed by a different client, the lookup will still fail. >Futhermore, the client will not be >able to recover the new filehandle, since it has no way of knowing the >new filename specified by client B in step 3. Therefore it will never be >able to release its locks on the file (or close it, since the CLOSE >operation has the same filehandle restriction as the LOCKU operation). >This "state leakage" will persist for the lifetime of the client (assuming >that the client periodically emits RENEW operations)! > >One possible solution to this problem is to add the restriction that, >while a file has been opened on the server, the server cannot change its >filehandle. For our Linux server, this would be quite problematic; our >filehandles have to be volatile on rename (in case anyone is curious >why, I've summarized the reasons in a note at the end of this message). Would it work to have the server set the attribute fh4_expire_type to (FH4_NOEXPIRE_WITH_OPEN | FH4_VOL_RENAME), and reject rename requests while the file is open, unless the directory is the same (perhaps with an exception when the renaming client is the one holding the lock)? Salit > >Another solution, which I think is the "right" way to handle this issue, >is: > 1. Agree that stateids are per (lockowner,file)-pair, rather than per >lockowner. This means that, in a LOCKU or CLOSE operation, the stateid >provided by the client is sufficient to specify the file; the current >filehandle would be redundant. > 2. Remove the restriction for the spec that, in a CLOSE or LOCKU >operation, the current filehandle has to be set to the opened file. The >stateid specified with the operation now identifies the file. > >What does everyone think of this? Is there a reason I'm unaware of for >requiring the current filehandle to be set in a CLOSE or LOCKU operation? >This proposal would give the NFSv4 stateid similar semantics to a UNIX >file descriptor; in UNIX, a file descriptor can be used to refer to a file >uniquely for as long as it remains open, even if the filename changes. >This proposal would let the NFSv4 stateid refer to a file uniquely for as >long as it remains open, even if the filehandle changes (although the >analogy between stateids and file descriptors isn't perfect, since the >stateid can change, whereas a file descriptor is constant). > >If the only reason that the filehandle is required to agree with the open >file during a CLOSE or LOCKU is to give the server an error check, I >personally think that the possible benefits of this additional check would >be outweighed by the problems created by this "state leakage" issue. >After all, the server can still do error-checking on the stateid, and >determine if it is stale, bad, or old. Feedback? > >Kendrick > >p.s. In case anyone is wondering why our Linux server's filehandles have >to remain volatile, here is an explanation. When a client sends a >filehandle, we have to verify for security purposes that the file is >contained in one of the directories approved for export on the server. >If the file is a directory, this is easy; we just traverse parent >directories until we find either an export, or the root of the filesystem. >If the file is a non-directory, however, we have a problem; we can't >uniquely determine the directory containing it, since it may be hardlinked >into several directories on the server. At best, we could traverse a list >of directories containing the file, to try to find one contained in an >export (which would present a performance problem), but even this is >impossible since the Linux low-level fs interfaces don't provide a way to >obtain a list of all directories containing a given (non-directory) file. >Therefore, our filehandles have to include, in addition to the inode >number of the file, the inode number of its containing directory, so that >we can locate the parent directory for export checking. When the file is >renamed, its parent directory may also change, which means the filehandle >can change. Our volatile filehandles are therefore a by-product of a >server environment in which (1) we don't want to export the entire >filesystem, only certain directories, and (2) because of hardlinks, it is >not possible to determine the directory containing a given file. > > > From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 23 18:56:32 2001 Received: from mercury.Sun.COM ([192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA26606 for ; Tue, 23 Jan 2001 18:56:32 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA00199; Tue, 23 Jan 2001 15:44:51 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA09035; Tue, 23 Jan 2001 15:44:32 -0800 (PST) Received: from jurassic.eng.sun.com (jurassic [129.146.85.31] (may be forged)) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0NNiJ213311 for ; Tue, 23 Jan 2001 15:44:19 -0800 (PST) Received: from modestmouse (modestmouse.Eng.Sun.COM [129.146.85.83]) by jurassic.eng.sun.com (8.11.2+Sun/8.11.2) with SMTP id f0NNiH9518558; Tue, 23 Jan 2001 15:44:17 -0800 (PST) Message-Id: <200101232344.f0NNiH9518558@jurassic.eng.sun.com> Date: Tue, 23 Jan 2001 15:41:06 -0800 (PST) From: Eric Kustarz Reply-To: Eric Kustarz Subject: RE: filehandle volatility, stateid, and "state leakage" To: kmsmith@umich.edu, nfsv4-wg@sunroof.eng.sun.com, Dave.Noveck@netapp.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: Ljst6vpCHYtVEbqWfyvuBg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5_17 SunOS 5.9 sun4u sparc >From: "Noveck, Dave" >To: "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com >Subject: RE: filehandle volatility, stateid, and "state leakage" >Date: Tue, 23 Jan 2001 14:05:09 -0800 >MIME-Version: 1.0 > >OTOH, my server has (lockowner,file)-pair stateid's and it won't change for >Connectathon unless a decision to go to per-lockowner stateid's happens >within the next few weeks. I'm assuming that is true for a lot of >implementations, so staying with with per (lockowner,file)-pair stateid's >is probably safe for Connectathon, at least. > We (SUN) too will be going with stateid as a (lockowner, file) pair. eric From nfs4-wg-request@sunroof.eng.sun.com Thu Jan 25 22:05:40 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA14835 for ; Thu, 25 Jan 2001 22:05:39 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id TAA24523; Thu, 25 Jan 2001 19:05:06 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id TAA19822; Thu, 25 Jan 2001 19:04:35 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0Q349217460; Thu, 25 Jan 2001 19:04:09 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id TAA19777; Thu, 25 Jan 2001 19:04:09 -0800 (PST) Received: from dns1.i-s-c.co.jp (dns1.i-s-c.co.jp [210.196.157.162]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id TAA29487; Thu, 25 Jan 2001 19:04:07 -0800 (PST) Message-Id: <200101260304.TAA29487@venus.Sun.COM> Received: from dsisd898 (03-127.044.popsite.net [64.24.247.127]) by dns1.i-s-c.co.jp with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.1960.3) id DQQYKTD8; Fri, 26 Jan 2001 12:03:49 +0900 From: "Melvin Dupont" Subject: Only a Few More Openings #6A30 To: today78u@venus.Sun.COM X-Mailer: Microsoft Outlook Express 4..72.1712.3 X-MimeOLE: Produced By Microsoft MimeOLE V(null).1712.3 Mime-Version: 1.0 Date: Thu, 25 Jan 2001 19:44:20 -0500 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ietf.org id WAA14835 *Earn $2000 - $5000 weekly-starting within 1-4 weeks *78% Profit Paid Daily *No Selling *No Risk Guarantee *Work from home, No overhead, or employees. *High Tech Training & Support *Not MLM, 100x more profitable *Multibillion Dollar Travel Industry The most incredible part of our business is that ALL MY CLIENTS CALL ME! DO YOU QUALIFY FOR OUR MENTOR PROGRAM? ACCEPTING ONLY 12 NEW ASSOCIATES This is not a hobby! Serious Inquires Only!! Please reply with the following information NAME: EMAIL ADDRESS: PHONE: (Required) BEST TIME TO CALL: TO: mailto:ytzz@cmpmail.com?subject=tell_me_more //////////////////////////////////////////////////// Please remove at: mailto:theater@building.com?subject=remove //////////////////////////////////////////////////// From nfs4-wg-request@sunroof.eng.sun.com Fri Jan 26 18:04:15 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA24234 for ; Fri, 26 Jan 2001 18:04:14 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA28743; Fri, 26 Jan 2001 15:03:24 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA27694; Fri, 26 Jan 2001 15:02:51 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0QN2X218876 for ; Fri, 26 Jan 2001 15:02:33 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA17745 for ; Fri, 26 Jan 2001 15:02:29 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA17258 for ; Fri, 26 Jan 2001 15:02:28 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0QN2T901645; Fri, 26 Jan 2001 18:02:29 -0500 (EST) From: "Sergey Klyushin" To: Cc: Subject: Stateid and seqid Date: Fri, 26 Jan 2001 18:02:44 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Content-Transfer-Encoding: 7bit Q1. Can anybody explain why do we need both stateid and seqid? (consider that stateid is shorthand to lockowner+file, and stateid includes some incrementing sequence inside itself) Note, that there is no operation, that takes seqid only without stateid (except OPEN, which is the only operation to create stateid). Q2. Some time ago here was a discussion about the order of checking stateid, seqid and lease. As far as I remember conclusion was : 1. check stateid. OK means that stateid_seq_number = stored_stateid_seq_number. OLD_STATEID means stateid_seq_number = stored_stateid_seq_number - 1 BAD_STATEID some other errors. 2. check seqid. OK means seqid = stored_seqid + 1 replay means seqid = stored_seqid and NFS Server should return stored reply error - everything else. 3. check lease Question: how is it possible to receive request with the previous seqid (seqid = stored_seqid) and not the same stateid. Is it real retransmission? Thanks, Sergey From nfs4-wg-request@sunroof.eng.sun.com Sat Jan 27 13:39:38 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA17361 for ; Sat, 27 Jan 2001 13:39:37 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA19019; Sat, 27 Jan 2001 10:38:13 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id KAA14928; Sat, 27 Jan 2001 10:37:20 -0800 (PST) Received: from phys-ha10nwka.ebay.sun.com (root@phys-ha10nwka.EBay.Sun.COM [129.150.151.210] (may be forged)) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0RIb8219566 for ; Sat, 27 Jan 2001 10:37:08 -0800 (PST) Received: from ebay.sun.com (dsl197-15.Eng.Sun.COM [129.146.197.15]) by phys-ha10nwka.ebay.sun.com (8.8.8+Sun/8.8.8/ENSMAIL,v2.0) with ESMTP id KAA08036; Sat, 27 Jan 2001 10:37:06 -0800 (PST) Message-ID: <3A731505.3988A46@ebay.sun.com> Date: Sat, 27 Jan 2001 10:35:49 -0800 From: David Robinson X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Sergey Klyushin CC: nfsv4-wg@sunroof.eng.sun.com, Dave.Noveck@netapp.com Subject: Re: Stateid and seqid References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sergey Klyushin wrote: > Q1. Can anybody explain why do we need both stateid and seqid? > (consider that stateid is shorthand to lockowner+file, and stateid includes > some > incrementing sequence inside itself) > Note, that there is no operation, that takes seqid only without stateid > (except OPEN, which is the > only operation to create stateid). The original logic was that the client controls the seqid space while the server controls the stateid space. Stateid's are not required to be a monotonically increasing integer while seqid's are. Given a stateid the client cannot tell what the next stateid may be. In theory a server may not change the stateid after a transaction where the seqid is required, so they were kept seperate. > Q2. > Some time ago here was a discussion about the order of checking stateid, > seqid and lease. > As far as I remember conclusion was : > 1. check stateid. OK means that stateid_seq_number = > stored_stateid_seq_number. > OLD_STATEID means stateid_seq_number = stored_stateid_seq_number - 1 > BAD_STATEID some other errors. > 2. check seqid. OK means seqid = stored_seqid + 1 > replay means seqid = stored_seqid and NFS Server should return stored reply > error - everything else. > 3. check lease > > Question: how is it possible to receive request with the previous seqid > (seqid = stored_seqid) > and not the same stateid. Is it real retransmission? In theory, a client upgrading a lock from read to write (shared to exclusive) does not require that a stateid be changed. All outstanding I/O requests are still valid because the client is given more permissions. But the seqid must change to prevent a subsequent request such as unlock to be ordered correctly. -David From nfs4-wg-request@sunroof.eng.sun.com Sat Jan 27 19:49:51 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA19104 for ; Sat, 27 Jan 2001 19:49:51 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id QAA22320; Sat, 27 Jan 2001 16:48:41 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id QAA00084; Sat, 27 Jan 2001 16:48:22 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0S0m8219668 for ; Sat, 27 Jan 2001 16:48:08 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id QAA00865 for ; Sat, 27 Jan 2001 16:48:08 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA23687; Sat, 27 Jan 2001 16:48:08 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0S0m6614527; Sat, 27 Jan 2001 19:48:06 -0500 (EST) From: "Sergey Klyushin" To: "David Robinson" Cc: Subject: RE: Stateid and seqid Date: Sat, 27 Jan 2001 19:48:25 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3A731505.3988A46@ebay.sun.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Content-Transfer-Encoding: 7bit > -----Original Message----- > From: David Robinson [mailto:David.Robinson@ebay.sun.com] > Sent: Saturday, January 27, 2001 1:36 PM > To: Sergey Klyushin > Cc: nfsv4-wg@sunroof.eng.sun.com; Dave.Noveck@netapp.com > Subject: Re: Stateid and seqid > > > Sergey Klyushin wrote: > > Q1. Can anybody explain why do we need both stateid and seqid? > > (consider that stateid is shorthand to lockowner+file, and > stateid includes > > some > > incrementing sequence inside itself) > > Note, that there is no operation, that takes seqid only > without stateid > > (except OPEN, which is the > > only operation to create stateid). > > The original logic was that the client controls the seqid space > while the server controls the stateid space. Stateid's are > not required to be a monotonically increasing integer while > seqid's are. Given a stateid the client cannot tell what the > next stateid may be. In theory a server may not change the stateid > after a transaction where the seqid is required, so they were > kept seperate. > > > Q2. > > Some time ago here was a discussion about the order of > checking stateid, > > seqid and lease. > > As far as I remember conclusion was : > > 1. check stateid. OK means that stateid_seq_number = > > stored_stateid_seq_number. > > OLD_STATEID means stateid_seq_number = stored_stateid_seq_number - 1 > > BAD_STATEID some other errors. > > 2. check seqid. OK means seqid = stored_seqid + 1 > > replay means seqid = stored_seqid and NFS Server should > return stored reply > > error - everything else. > > 3. check lease > > > > Question: how is it possible to receive request with the > previous seqid > > (seqid = stored_seqid) > > and not the same stateid. Is it real retransmission? > > In theory, a client upgrading a lock from read to write (shared to > exclusive) does not require that a stateid be changed. All > outstanding > I/O requests are still valid because the client is given more > permissions. But the seqid must change to prevent a subsequent > request such as unlock to be ordered correctly. > > -David > If implementation of NFS Server changes stateid on reply (otherwise I don't see why to return stateid, if it's not changed), I can't see correct processing of replay scenario. I'll try to explain with more details. Replay request has seqid = stored_seqid, but request's stateid != stored_stateid. So, to follow above scenario (first check stateid, then check seqid) error BAD_STATEID or OLD_STATEID will be returned, before checking seqid. BTW, seqid = stored_seqid doesn't mean real replay, because full request must be the same, and NFS Server should keep previous request to compare it with current one. In some implementation it's done on RPC layer. From my point of view, including index inside stateid is enough to monitor replay without seqid at all. (which makes live more easy) Sergey From nfs4-wg-request@sunroof.eng.sun.com Sun Jan 28 13:27:56 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA09741 for ; Sun, 28 Jan 2001 13:27:56 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA26915; Sun, 28 Jan 2001 10:27:09 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id KAA15910; Sun, 28 Jan 2001 10:26:50 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0SIQZ220062 for ; Sun, 28 Jan 2001 10:26:35 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id KAA12800 for ; Sun, 28 Jan 2001 10:26:34 -0800 (PST) Received: from berzerk.gpcc.itd.umich.edu (berzerk.gpcc.itd.umich.edu [141.211.2.162]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA01870 for ; Sun, 28 Jan 2001 11:26:33 -0700 (MST) Received: from galaxian.gpcc.itd.umich.edu (smtp@galaxian.gpcc.itd.umich.edu [141.211.2.146]) by berzerk.gpcc.itd.umich.edu (8.8.8/4.3-mailhub) with ESMTP id NAA07170 for ; Sun, 28 Jan 2001 13:26:33 -0500 (EST) Received: from localhost (kmsmith@localhost) by galaxian.gpcc.itd.umich.edu (8.8.8/5.1-client) with ESMTP id NAA28924 for ; Sun, 28 Jan 2001 13:26:32 -0500 (EST) Precedence: first-class Date: Sun, 28 Jan 2001 13:26:32 -0500 (EST) From: "Kendrick M. Smith" X-Sender: kmsmith@galaxian.gpcc.itd.umich.edu To: nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Sat, 27 Jan 2001, Sergey Klyushin wrote: > If implementation of NFS Server changes stateid on reply > (otherwise I don't see why to return stateid, if it's not changed), > I can't see correct processing of replay scenario. > I'll try to explain with more details. > Replay request has seqid = stored_seqid, but request's stateid != > stored_stateid. > So, to follow above scenario (first check stateid, then check seqid) error > BAD_STATEID or > OLD_STATEID will be returned, before checking seqid. I suppose this means that, to guarantee correct replay protection, a server implementation must retain the last expired stateid (per lockowner) in its stateid table. This way, if a replay is received, the server can recognize the replayed stateid, compare seqids, and replay its cached response instead of erroneously returning NFS4ERR_OLD_STATEID. This is a good point which, to my knowledge, hasn't come up before. Should something like this be written into the spec (or at least the implementor's guidelines)? > BTW, seqid = stored_seqid doesn't mean real replay, because full request > must be the same, and > NFS Server should keep previous request to compare it with current one. In > some implementation > it's done on RPC layer. > >From my point of view, including index inside stateid is enough to monitor > replay without seqid at all. > (which makes live more easy) It sounds to me like you are suggesting removing seqids from the protocol entirely, and relying on stateids to provide replay protection for non-idempotent operations. Perusing the protocol, it seems that every non-idempotent operation does change a stateid, so it should be possible to use stateids for replay protection and still have the correct replay semantics. So I think that this is really a performance issue, not a correctness issue. It is the same issue as, what "granularity" should the server use when storing responses in its replay cache? Under the current spec, the server will cache the most recent outstanding request on a per-lockowner basis. If replay protection were done using stateids instead of seqids as you suggest, the server would cache the most recent outstanding request on a per-(lockowner, file) basis, so the replay cache would consume more server memory. The flip side of this is that, currently, non-idempotent operations have to be serialized on a per-lockowner basis, i.e. a client can't have two concurrent outstanding requests with the same lockowner (or seqids might get out of sync on the client & server). If replay protection were done using stateids, non-idempotent operations would be serialized on a per-(lockowner, file) basis. A client could have two concurrent requests with the same lockowner, as long as they were for different files. So the issue of seqids vs. stateids for replay protection is really a performance tradeoff between increased server memory and better concurrency properties. My personal intuition is that it is best to continue using seqids for replay protection, since the memory cost of caching on a per-(lockowner, file) basis is probably quite significant, but this is something that can only really be settled by measurement and profiling. Does anyone know of performance studies that might shed some light on this issue? - k From nfs4-wg-request@sunroof.eng.sun.com Sun Jan 28 23:32:19 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA14034 for ; Sun, 28 Jan 2001 23:32:18 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA01173; Sun, 28 Jan 2001 20:31:03 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id UAA16594; Sun, 28 Jan 2001 20:30:45 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0T4UV220297 for ; Sun, 28 Jan 2001 20:30:32 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id UAA16564 for ; Sun, 28 Jan 2001 20:30:30 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id UAA04887 for ; Sun, 28 Jan 2001 20:30:21 -0800 (PST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0T4UK218210; Sun, 28 Jan 2001 20:30:20 -0800 (PST) Received: from ussvlexc06.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0T4UKL21817; Sun, 28 Jan 2001 20:30:20 -0800 (PST) Received: by ussvlexc06.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Sun, 28 Jan 2001 20:30:58 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E53@red.nane.netapp.com> From: "Noveck, Dave" To: "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid Date: Sun, 28 Jan 2001 20:30:15 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Kendrick Smith wrote: > > On Sat, 27 Jan 2001, Sergey Klyushin wrote: > > > If implementation of NFS Server changes stateid on reply > > (otherwise I don't see why to return stateid, if it's not changed), > > I can't see correct processing of replay scenario. > > I'll try to explain with more details. > > Replay request has seqid = stored_seqid, but request's stateid != > > stored_stateid. > > So, to follow above scenario (first check stateid, then check seqid) error > > BAD_STATEID or > > OLD_STATEID will be returned, before checking seqid. > > I suppose this means that, to guarantee correct replay protection, a > server implementation must retain the last expired stateid (per lockowner) > in its stateid table. This way, if a replay is received, the server can > recognize the replayed stateid, compare seqids, and replay its cached > response instead of erroneously returning NFS4ERR_OLD_STATEID. This is a > good point which, to my knowledge, hasn't come up before. Should > something like this be written into the spec (or at least the > implementor's guidelines)? I don't think you have to separately save the last superseded (I think that's what you mean here; expired has some other connotations) stateid. What I do is first use the stateid to map to an entry in my table of state structures. At this point, I may generate STALE_STATEID and BAD_STATEID errors, but not OLD_STATEID. This requires that I know the range of valid values of the stateid subfield which is incremented on each change, but I need that anyway to properly distinguish between BAD_STATEID and OLD_STATEID. Once I have the address of a state structure, I can use this to find the corresponding owner structure which has the current seqid and replay information. So at this point, I can notice a replay and respond. If it is not a replay (and we don't have a BAD_SEQID error), then I can check the stateid against the current stateid in the state structure and generated aN OLD_STATEID error if appropriate. I thought this was discussed in the session Andy organized regarding order-of-processing issues at the last bake-off. I thought this was on the flow-charts that we came up with. Then I again I was pretty tired so maybe my memmory is faulty. In any case I think this should be documented somewhere (implementation RFC?). > > > BTW, seqid = stored_seqid doesn't mean real replay, because full request > > must be the same, and > > NFS Server should keep previous request to compare it with current one. In > > some implementation > > it's done on RPC layer. > > >From my point of view, including index inside stateid is enough to monitor > > replay without seqid at all. > > (which makes live more easy) > > It sounds to me like you are suggesting removing seqids from the protocol > entirely, and relying on stateids to provide replay protection for > non-idempotent operations. Perusing the protocol, it seems that every > non-idempotent operation does change a stateid, so it should be possible > to use stateids for replay protection and still have the correct replay > semantics. I don't think so. In the case of open, I don't see how you can do without a seqid. The fact that it generates a stateid (updated or not) doesn't give you any replay protection. I send an open for read-write, deny=read-write. It succeeds and I send back a stateid. The reply is lost. A replay of the OPEN is sent and it fails because open for read or write is denied, which is definitely not what you want. You might fall back on various kind of ad hoc replay protection mechanisms specific to OPEN but that doesn't give me a very good feeling. > So I think that this is really a performance issue, not a > correctness issue. It is the same issue as, what "granularity" should the > server use when storing responses in its replay cache? Under the current > spec, the server will cache the most recent outstanding request on a > per-lockowner basis. If replay protection were done using stateids > instead of seqids as you suggest, the server would cache the most recent > outstanding request on a per-(lockowner, file) basis, so the replay cache > would consume more server memory. The flip side of this is that, > currently, non-idempotent operations have to be serialized on a > per-lockowner basis, i.e. a client can't have two concurrent outstanding > requests with the same lockowner (or seqids might get out of sync on the > client & server). If replay protection were done using stateids, > non-idempotent operations would be serialized on a per-(lockowner, file) > basis. A client could have two concurrent requests with the same > lockowner, as long as they were for different files. So the issue of > seqids vs. stateids for replay protection is really a performance tradeoff > between increased server memory and better concurrency properties. My > personal intuition is that it is best to continue using seqids for replay > protection, since the memory cost of caching on a per-(lockowner, file) > basis is probably quite significant, but this is something that can only > really be settled by measurement and profiling. Does anyone know of > performance studies that might shed some light on this issue? - k I think it would be possible to serialize all opens and closes for a given lockowner on the on hand, and then use a separate mechanism stateid-only or stateid-plus-file-specific seqid to handle sequencing of locks and unlocks on per-(lockowner+file) basis. My guess is that the performance benefit of this approach would be limited to very few applications. Most multi-threaded apps that would want to have multiple locks or unlocks in flight would probably be working most of the time on a single file so would get no benefit from a scheme that allowed multiple requests to be in-flight only when they were to different files. From nfs4-wg-request@sunroof.eng.sun.com Mon Jan 29 11:05:46 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA04627 for ; Mon, 29 Jan 2001 11:05:46 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA00653; Mon, 29 Jan 2001 08:04:04 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA10197; Mon, 29 Jan 2001 08:03:50 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0TG3U220571 for ; Mon, 29 Jan 2001 08:03:31 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA10113 for ; Mon, 29 Jan 2001 08:03:31 -0800 (PST) Received: from donkeykong.gpcc.itd.umich.edu (donkeykong.gpcc.itd.umich.edu [141.211.2.163]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA05774 for ; Mon, 29 Jan 2001 08:03:31 -0800 (PST) Received: from skywalker.citi.umich.edu (skywalker.citi.umich.edu [141.211.92.240]) by donkeykong.gpcc.itd.umich.edu (8.8.8/4.3-mailhub) with ESMTP id LAA23988; Mon, 29 Jan 2001 11:03:29 -0500 (EST) Received: (from andros@localhost) by skywalker.citi.umich.edu (8.8.8/5.1-client) id LAA17278; Mon, 29 Jan 2001 11:03:29 -0500 (EST) Message-Id: <200101291603.LAA17278@skywalker.citi.umich.edu> Precedence: first-class X-Mailer: exmh version 2.0.2 2/24/98 To: "Noveck, Dave" Cc: "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com Subject: Re: Stateid and seqid In-reply-to: Dave.Noveck's message of Sun, 28 Jan 2001 20:30:15 -0800. Mime-Version: 1.0 Content-Type: multipart/mixed ; boundary="==_Exmh_-15980722880" Date: Mon, 29 Jan 2001 11:03:29 -0500 From: "William A.(Andy) Adamson" This is a multipart MIME message. --==_Exmh_-15980722880 Content-Type: text/plain; charset=us-ascii here is (finnally!) what i have for the flow diagram of stateid, seq num, and lease processing that was agreed upon at the last bakeoff, and discussed in the flurry of post bake-off email. i think it includes the processing that dave described concerning stateid and sequence number. i would appreciate it if the list would look it over - it would be great to agree on this issue before connectathon.. -->Andy --==_Exmh_-15980722880 Content-Type: text/plain ; name="stateid_seq_num_and_lease"; charset=us-ascii Content-Description: stateid_seq_num_and_lease Content-Disposition: attachment; filename="stateid_seq_num_and_lease" Operations that have a stateid and a sequence number ---------------------------------------------------- CLOSE, LOCK, LOCKU, OPEN_DOWNGRADE, OPEN_CONFIRM (stateid in verifier otherwise server can't locate lockowner) NOTE: the server cannot deallocate the information associated with a CLOSE'd state until another request with the next sequence number arrives for the same lockowner, or the client reboots, or there is no longer any locking state for that owner and the server receives nothing from that lockowner for a while and forgets the associated lockowner information, yada, yada, yada. check stateid / | \ \ / | \ \ / \ \ \ / \ \ \_______ / | \ \ stale good old bad / | \ | | | | server can't bump seq num | | | client undo seq num increment | | | don't renew lease | | | NFSERR_BAD_STATEID seq knowlege is | \ reset on client | \ and server | check sequence number NFS4ERR_STALE_STATEID | / \ | bad or good replay (should hit this) | | send from replay cache | | | good: server increment seq num | good: client leaves seq num alone | NFSERR_OLD_STATEID | check sequence number / | \ / | \ / | \ bad replay good / | \ don't bump seqnum | \ don't renew lease? | \ NFS4ERR_BAD_SEQ_NUM | \ | \ don't bump seq num \ return replay \ | | check lease | | | bad good | / \ check lease don't renew lease / | renew / | lease bad good / | / | NFS4ERR_EXPIRED | seq num bumped | | | renew lease perform operation Operations that have a sequence number, and no stateid ------------------------------------------------------ OPEN check clientid / \ bad good / | don't bump seq num | don't renew lease | NFS4ERR_STALE_CLIENTID | | check sequence number / | \ / | \ / | \ bad replay good / | \ don't bump seqnum | \ don't renew lease | \ NFS4ERR_BAD_SEQ_NUM | \ | \ don't bump seq num \ return replay \ | | check lease | | | bad good | / \ check lease don't renew lease / | renew / | lease bad good / | / | NFS4ERR_EXPIRED | (seq num bumped) | | | renew lease perform operation Operations that have a stateid and no sequence number ------------------------------------------------------ READ, WRITE, RENEW, SETATTR for TRUNCATE This case is clear, the lease is renewed only if the stateid is valid. check stateid / \ bad good / | don't renew lease | NFS4ERR_XXX_STATEID | | | check lease / | bad good / | / | NFS4ERR_EXPIRED | | | renew lease perform operation --==_Exmh_-15980722880-- From nfs4-wg-request@sunroof.eng.sun.com Mon Jan 29 11:33:38 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA05324 for ; Mon, 29 Jan 2001 11:33:37 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA23773; Mon, 29 Jan 2001 08:32:37 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA13269; Mon, 29 Jan 2001 08:32:20 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0TGW3220584; Mon, 29 Jan 2001 08:32:03 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA13207; Mon, 29 Jan 2001 08:32:03 -0800 (PST) From: magmktg@hotmail.com Received: from blight.micron.net (blight.micron.net [204.229.122.191]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA18130; Mon, 29 Jan 2001 08:32:02 -0800 (PST) Received: from 10.224.0.191 ([10.224.0.191]) by blight.micron.net (Netscape Messaging Server 4.1) with SMTP id G7XKMY00.F7A; Mon, 29 Jan 2001 08:35:22 -0700 Received: from mailboss1.mymail.com (1Cust18.tnt6.syracuse.ny.da.uu.net [63.21.205.18]) by with SMTP (MailShield v1.5); Mon, 29 Jan 2001 08:35:21 -0700 Subject: How about this one... Date: Mon, 29 Jan 2001 07:01:03 Message-Id: <761.432079.644625@mailboss1.mymail.com> X-SMTP-HELO: mailboss1.mymail.com X-SMTP-MAIL-FROM: magmktg@hotmail.com X-SMTP-PEER-INFO: 1Cust18.tnt6.syracuse.ny.da.uu.net [63.21.205.18] To: undisclosed-recipients:; WOULD YOU LIKE TO EARN AN EXTRA $500 TO $3000 PER WEEK just by mailing our business circulars from your home? HELP SOLVE YOUR MONEY PROBLEMS. No More worries over inflation, recession, bills, rising gasoline and other costs. This is easy extra income that can help to relieve financial pressures! HERE IS YOUR CHANCE to earn extra money working at home by becoming an active participant of our successful mailing association. You receive cash weekly for mailing our circulars. There is no limit, You may mail as many as you wish. NO EXPERIENCE OR SPECIAL SKILLS REQURED. Our HOMEMAILER'S PROGRAM is the easiest work anyone can ever do - all from the comfort of your own home! We even provide you with easy, step-by-step instructions! You may work in the comfort of your own home, choose your own hours, and set your own pace. No need to leave your present job. The possibilities are unlimited! THIS IS POSSIBLE BECAUSE We Publish, sell and distribute information booklets, guides, reports, manuals and computer software all across The United States and Canada. We do the majority of our business by mail and we need to expand our business. But if we hire more employees, we would have to supervise them, rent more office space, pay more taxes and insurance, all involving more cost and paperwork. THIS IS WHY WE NEED YOU. It is much easier to set it up so that independent homeworkers like yourself can earn money doing the work yourselves. This program is designed to help people like you cash in with us. You receive good commissions to mail the circulars for us. We can not reach all the potential customers by ourselves therefore you would be sending out our circulars in response to customer inquiries. When you mail our circulars, you'll be greatly helping us by getting our offers out to more customers. ALL BUSINESS CAN BE DONE BY MAIL and from your hometown area, and, we give you complete assistance at every step to insure your success. You can START THE SAME DAY you receive the instructions and begin RECEIVING MONEY WITHIN TWO WEEKS, and every week from then on, as long as you desire. You will be supplied with the envelopes, which will already be stamped and addressed by your customers. GUARANTEE: We welcome you to this program and extend to you our unconditional guarantee that you will be delighted with the money you make. We are positive you will be 100% satisfied with the Home Mailer Program. This is your opportunity to become an Independent Commission Mailer for our company! The Home Mailers Program will give you complete easy, step-by-step instructions! IN ORDER TO GET YOU STARTED IMMEDIATELY, we must require a one time processing fee of only $30.00. This covers our expense in showing you what to do and guarantees you can work with us as long as desired. You will not be required or asked to pay for any additional information, manuals, or instructions. In as much as we would like to send you our program without the small charge, we must protect ourselves from those who are not serious and have no intention other than to satisfy their own curiosity. Naturally, no business can afford to send out costly material to everyone who writes in asking for it. This small charge assures us that you are serious about working from home with our company! DON'T DELAY - START IMMEDIATELY - TIME IS MONEY, you can begin now by putting your time to the best of it's use. **THE ONLY PEOPLE WHO NEVER ACHIEVE ARE THE ONES WHO NEVER TRY** ----------------------------------------------------------------- HOME MAILERS PROGRAM ORDER-FORM ----------------------------------------------------------------- Please RUSH me my Package of the HOME MAILERS PROGRAM right away!!! I am enclosing $30.00 in U.S Funds. (Includes Postage & Handling) **** Use a Postal Money order and get free priority shipping*** MAIL TO: MAGNUM MARKETING 10125 Carousel Center Syracuse, NY USA 13290-0121 NAME:______________________________________ ADDRESS:___________________________________ CITY:_________________STATE/PROVINCE:______ ZIP/POSTAL CODE:___________________________ E-MAIL ADDRESS:____________________________ (Orders payable by check please allow 4-6 weeks for delivery. Returned check fee $25) 01/21/01 ---------------------------------------------------------------- To receive no further offers from our company regarding this matter or any other matter, please reply to this e-mail with the word 'Remove' in the subject line. From nfs4-wg-request@sunroof.eng.sun.com Mon Jan 29 19:58:05 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA13040 for ; Mon, 29 Jan 2001 19:58:04 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id QAA18717; Mon, 29 Jan 2001 16:57:17 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id QAA08348; Mon, 29 Jan 2001 16:56:43 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0U0uK220912 for ; Mon, 29 Jan 2001 16:56:20 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id QAA07292 for ; Mon, 29 Jan 2001 16:56:20 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA02781 for ; Mon, 29 Jan 2001 16:56:19 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0U0uB625794; Mon, 29 Jan 2001 19:56:11 -0500 (EST) From: "Sergey Klyushin" To: "Noveck, Dave" , Subject: RE: Stateid and seqid Date: Mon, 29 Jan 2001 19:56:30 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <8C610D86AF6CD4119C9800B0D0499E331A6E53@red.nane.netapp.com> Content-Transfer-Encoding: 7bit > > Kendrick Smith wrote: > > > > On Sat, 27 Jan 2001, Sergey Klyushin wrote: > > > > > If implementation of NFS Server changes stateid on reply > > > (otherwise I don't see why to return stateid, if it's not changed), > > > I can't see correct processing of replay scenario. > > > I'll try to explain with more details. > > > Replay request has seqid = stored_seqid, but request's stateid != > > > stored_stateid. > > > So, to follow above scenario (first check stateid, then check seqid) error > > > BAD_STATEID or > > > OLD_STATEID will be returned, before checking seqid. > > > > I suppose this means that, to guarantee correct replay protection, a > > server implementation must retain the last expired stateid (per lockowner) > > in its stateid table. This way, if a replay is received, the server can > > recognize the replayed stateid, compare seqids, and replay its cached > > response instead of erroneously returning NFS4ERR_OLD_STATEID. This is a > > good point which, to my knowledge, hasn't come up before. Should > > something like this be written into the spec (or at least the > > implementor's guidelines)? > > I don't think you have to separately save the last superseded (I think > that's what you mean here; expired has some other > connotations) stateid. > > What I do is first use the stateid to map to an entry in my table of > state structures. At this point, I may generate STALE_STATEID and > BAD_STATEID errors, but not OLD_STATEID. This requires that I know > the range of valid values of the stateid subfield which is incremented > on each change, but I need that anyway to properly distinguish between > BAD_STATEID and OLD_STATEID. > How "old" could be stateid? Probably, any operation, that takes both stateid and seqid, makes range of valid values of the stateid subfield equal 1? BTW, how useful ERROR_OLD_STATEID for client? What client suppose to do after receiving this error? > Once I have the address of a state structure, I can use this > to find the > corresponding owner structure which has the current seqid and replay > information. So at this point, I can notice a replay and respond. > So, you don't store stateid (or even full request) for reply message, but you are based only on seqid = stored_seqid? How you could be sure, that it's real replay, but not logical error? I want to be sure at this point, that I understand replay correctly: for me it's real retransmission of exactly the same package. Is it true? BTW, do you check lease time before reply? > If it is not a replay (and we don't have a BAD_SEQID error), then I > can check the stateid against the current stateid in the > state structure and generated aN OLD_STATEID error if appropriate. > > I thought this was discussed in the session Andy organized regarding > order-of-processing issues at the last bake-off. I thought this > was on the flow-charts that we came up with. Then I again I was > pretty tired so maybe my memmory is faulty. In any case I think this > should be documented somewhere (implementation RFC?). > > > > > > BTW, seqid = stored_seqid doesn't mean real replay, because full request > > > must be the same, and > > > NFS Server should keep previous request to compare it with current one. In > > > some implementation it's done on RPC layer. > > > From my point of view, including index inside stateid is enough to monitor > > > replay without seqid at all. (which makes live more easy) > > > > It sounds to me like you are suggesting removing seqids from the protocol > > entirely, and relying on stateids to provide replay protection for > > non-idempotent operations. Perusing the protocol, it seems that every > > non-idempotent operation does change a stateid, so it should be possible > > to use stateids for replay protection and still have the correct replay > > semantics. > > I don't think so. In the case of open, I don't see how you can do without > a seqid. The fact that it generates a stateid (updated or not) doesn't > give you any replay protection. > > I send an open for read-write, deny=read-write. It succeeds and I send > back a stateid. The reply is lost. A replay of the OPEN is sent and > it fails because open for read or write is denied, which is definitely > not what you want. You might fall back on various kind of ad hoc replay > protection mechanisms specific to OPEN but that doesn't give me a very good > feeling. I don't see any place when you need both seqid and stateid in one request. OPEN may take seqid and generate stateid, and stateid will be used later without seqid But, probably, changing seqid to stateid is really too late now (even if it's needed only for OPEN). > > > So I think that this is really a performance issue, not a > > correctness issue. It is the same issue as, what "granularity" should the > > server use when storing responses in its replay cache? Under the current > > spec, the server will cache the most recent outstanding request on a > > per-lockowner basis. Indeed, Server should keep cache per lockowner, but current agreement of stateid is lockowner+file. That's why I don't believe, that checking seqid without stateid is enough for replay indication. > > If replay protection were done using stateids > > instead of seqids as you suggest, the server would cache the most recent > > outstanding request on a per-(lockowner, file) basis, so the replay cache > > would consume more server memory. The flip side of this is that, > > currently, non-idempotent operations have to be serialized on a > > per-lockowner basis, i.e. a client can't have two concurrent outstanding > > requests with the same lockowner (or seqids might get out of sync on the > > client & server). If replay protection were done using stateids, > > non-idempotent operations would be serialized on a per-(lockowner, file) > > basis. A client could have two concurrent requests with the same > > lockowner, as long as they were for different files. So the issue of > > seqids vs. stateids for replay protection is really a performance tradeoff > > between increased server memory and better concurrency properties. My > > personal intuition is that it is best to continue using seqids for replay > > protection, since the memory cost of caching on a per-(lockowner, file) > > basis is probably quite significant, but this is something that can only > > really be settled by measurement and profiling. Does anyone know of > > performance studies that might shed some light on this issue? - k > > I think it would be possible to serialize all opens and closes for a > given lockowner on the on hand, and then use a separate mechanism > stateid-only or stateid-plus-file-specific seqid to handle sequencing > of locks and unlocks on per-(lockowner+file) basis. My guess is that > the performance benefit of this approach would be limited to very > few applications. Most multi-threaded apps that would want to have > multiple locks or unlocks in flight would probably be working most > of the time on a single file so would get no benefit from a scheme > that allowed multiple requests to be in-flight only when they were > to different files. > From nfs4-wg-request@sunroof.eng.sun.com Mon Jan 29 20:05:27 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA13116 for ; Mon, 29 Jan 2001 20:05:26 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id RAA19483; Mon, 29 Jan 2001 17:04:31 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id RAA09594; Mon, 29 Jan 2001 17:04:00 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0U13g220920; Mon, 29 Jan 2001 17:03:42 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id RAA01231; Mon, 29 Jan 2001 17:03:42 -0800 (PST) Received: from dfw-smtpout1.email.verio.net (dfw-smtpout1.email.verio.net [129.250.36.41]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA22355; Mon, 29 Jan 2001 17:03:41 -0800 (PST) Received: from [129.250.38.61] (helo=dfw-mmp1.email.verio.net) by dfw-smtpout1.email.verio.net with esmtp id 14NP9P-00048B-00; Tue, 30 Jan 2001 01:00:03 +0000 Received: from [206.133.83.14] (helo=localhost) by dfw-mmp1.email.verio.net with esmtp id 14NP6e-0003XP-00; Tue, 30 Jan 2001 00:57:12 +0000 X-Sender: misterprivacy@hushmail.com From: Dave To: "customer" Date: Mon, 29 Jan 2001 16:13:40 -0500 Subject: I could go to JAIL for selling this CD! MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_001__17122755_58420.84" Message-Id: This is a Multipart MIME message. ------=_NextPart_000_001__17122755_58420.84 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit ------=_NextPart_000_001__17122755_58420.84 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: base64 PCFkb2N0eXBlIGh0bWwgcHVibGljICItLy93M2MvL2R0ZCBodG1sIDQuMCB0cmFuc2l0aW9u YWwvL2VuIj4NCjxodG1sPg0KPGhlYWQ+DQogICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50 LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD1pc28tODg1OS0xIj4NCiAgIDxt ZXRhIG5hbWU9IkdFTkVSQVRPUiIgY29udGVudD0iTW96aWxsYS80LjYxIFtlbl0gKFdpbjk4 OyBJKSBbTmV0c2NhcGVdIj4NCiAgIDx0aXRsZT5UaGUgQkFOTkVEIENEITwvdGl0bGU+DQo8 L2hlYWQ+DQo8Ym9keT4NCjxmb250IHNpemU9KzE+SGksPC9mb250Pg0KPHA+PGZvbnQgc2l6 ZT0rMT5JIGhhdmUgYmVlbiByZWNpZXZpbmcgZW1haWxzIHNheWluZyB0aGF0IEknbSBjb250 cmlidXRpbmcNCnRvPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+dGhlICJtb3JhbCBkZWNh eSBvZiBzb2NpZXR5IiBieSBzZWxsaW5nIHRoZSBCYW5uZWQgQ0QuJm5ic3A7DQpUaGF0PC9m b250Pg0KPGJyPjxmb250IHNpemU9KzE+bWF5IGJlLCBidXQgSSBmZWVsIHN0cm9uZ2x5IHRo YXQgeW91IGhhdmUgYSByaWdodCB0bw0KYmVuZWZpdCBmcm9tPC9mb250Pg0KPGJyPjxmb250 IHNpemU9KzE+dGhpcyBoYXJkLXRvLWZpbmQgaW5mb3JtYXRpb24uPC9mb250Pg0KPHA+PGZv bnQgY29sb3I9IiNDQzAwMDAiPjxmb250IHNpemU9KzE+U28gSSBhbSBnaXZpbmcgeW91IE9O RSBMQVNUIENIQU5DRQ0KdG8gb3JkZXI8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9y PSIjQ0MwMDAwIj48Zm9udCBzaXplPSsxPnRoZSBCYW5uZWQgQ0QhPC9mb250PjwvZm9udD4N Cjxicj4mbmJzcDsNCjx0YWJsZSBCT1JERVI9MCBDT0xTPTIgV0lEVEg9Ijc1JSIgPg0KPHRy Pg0KPHRkIFdJRFRIPSIxJSI+PGltZyBTUkM9Imh0dHA6Ly9jb21wdXpvbmV1c2EuY29tL2lt YWdlcy9zcHkuanBnIiBoZWlnaHQ9MTI4IHdpZHRoPTg2PjwvdGQ+DQoNCjx0ZD48Zm9udCBz aXplPSsxPldpdGggdGhpcyBwb3dlcmZ1bCBDRCwgeW91IHdpbGwgYmUgYWJsZSB0byBpbnZl c3RpZ2F0ZQ0KeW91ciBmcmllbmRzLCBlbmVtaWVzIGFuZCBsb3ZlcnMgaW4ganVzdCBtaW51 dGVzIHVzaW5nIHRoZSBJbnRlcm5ldC4mbmJzcDsNCllvdSBjYW4gdHJhY2sgZG93biBvbGQg ZmxhbWVzIGZyb20gY29sbGVnZSwgb3IgeW91IGNhbiBkaWcgdXAgc29tZSBkaXJ0DQpvbiB5 b3VyIGJvc3MgdG8gbWFrZSBzdXJlIHlvdSBnZXQgdGhhdCBuZXh0IHByb21vdGlvbiE8L2Zv bnQ+PC90ZD4NCjwvdHI+DQo8L3RhYmxlPg0KDQo8cD48Zm9udCBzaXplPSsxPk9yIG1heWJl IHlvdSB3YW50IGEgZmFrZSBkaXBsb21hIHRvIGhhbmcgb24geW91ciBiZWRyb29tPC9mb250 Pg0KPGJyPjxmb250IHNpemU9KzE+d2FsbC4mbmJzcDsgWW91J2xsIGZpbmQgYWRkcmVzc2Vz IGZvciBjb21wYW5pZXMgdGhhdA0KbWFrZSB0aGVzZTwvZm9udD4NCjxicj48Zm9udCBzaXpl PSsxPmRpcGxvbWFzIG9uIHRoZSBCYW5uZWQgQ0QuPC9mb250Pg0KPHA+PGZvbnQgc2l6ZT0r MT5OZWVkIHRvIGRpc2FwcGVhciBmYXN0IGFuZCBuZXZlciBsb29rIGJhY2s/Jm5ic3A7IE5v IHByb2JsZW0hPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+VXNpbmcgdGhlIEJhbm5lZCBD RCwgeW91IHdpbGwgbGVhcm4gaG93IHRvIGJ1aWxkIGEgY29tcGxldGVseTwvZm9udD4NCjxi cj48Zm9udCBzaXplPSsxPm5ldyBpZGVudGl0eS48L2ZvbnQ+DQo8cD48Zm9udCBzaXplPSsx Pk9idmlvdXNseSwgdGhlIFBvd2VycyBUaGF0IEJlIGRvbid0IHdhbnQgeW91IHRvIGhhdmUg dGhlPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+QmFubmVkIENELiZuYnNwOyBUaGV5IGhh dmUgdGhyZWF0ZW5lZCBtZSB3aXRoIGxhd3N1aXRzLA0KZmluZXMsPC9mb250Pg0KPGJyPjxm b250IHNpemU9KzE+YW5kIGV2ZW4gaW1wcmlzb25tZW50IHVubGVzcyBJIHN0b3Agc2VsbGlu ZyBpdCBpbW1lZGlhdGVseS48L2ZvbnQ+DQo8YnI+PGZvbnQgc2l6ZT0rMT5CdXQgSSBmZWVs IHRoYXQgWU9VIGhhdmUgYSBDb25zdGl0dXRpb25hbCByaWdodCB0byBhY2Nlc3M8L2ZvbnQ+ DQo8YnI+PGZvbnQgc2l6ZT0rMT50aGlzIHR5cGUgb2YgaW5mb3JtYXRpb24sIGFuZCBJIGNh bid0IGJlIGludGltaWRhdGVkLjwvZm9udD4NCjxicj4mbmJzcDsNCjx0YWJsZSBCT1JERVI9 MCBDT0xTPTIgV0lEVEg9IjUwJSIgPg0KPHRyPg0KPHRkIFdJRFRIPSIxMDAlIj48aT48Zm9u dCBjb2xvcj0iIzMzMzNGRiI+PGZvbnQgc2l6ZT0rMT5VbmNsZSBTYW0gYW5kIHlvdXINCmNy ZWRpdG9ycyBhcmUgaG9ycmlmaWVkIHRoYXQgSSBhbSBzdGlsbCBzZWxsaW5nIHRoaXMgcHJv ZHVjdCEmbmJzcDsgVGhlcmUNCm11c3QgYmUgYSBwcmljZSBvbiBteSBoZWFkITwvZm9udD48 L2ZvbnQ+PC9pPjwvdGQ+DQoNCjx0ZCBXSURUSD0iMSUiPjxpbWcgU1JDPSJodHRwOi8vY29t cHV6b25ldXNhLmNvbS9pbWFnZXMvb3V0bGF3LmpwZyIgaGVpZ2h0PTEyOCB3aWR0aD05Mz48 L3RkPg0KPC90cj4NCjwvdGFibGU+DQoNCjxwPjxmb250IHNpemU9KzE+V2h5IGFyZSB0aGV5 IHNvIHVwc2V0PyBCZWNhdXNlIHRoaXMgQ0QgZ2l2ZXMgeW91IGZyZWVkb20uPC9mb250Pg0K PGJyPjxmb250IHNpemU9KzE+QW5kIHlvdSBjYW4ndCBidXkgZnJlZWRvbSBhdCB5b3VyIGxv Y2FsIFdhbG1hcnQuJm5ic3A7DQpZb3Ugd2lsbDwvZm9udD4NCjxicj48Zm9udCBzaXplPSsx PmhhdmUgdGhlIGZyZWVkb20gdG8gYXZvaWQgY3JlZGl0b3JzLCBqdWRnbWVudHMsIGxhd3N1 aXRzLA0KSVJTPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+dGF4IGNvbGxlY3RvcnMsIGNy aW1pbmFsIGluZGljdG1lbnRzLCB5b3VyIGdyZWVkeSBleC13aWZlDQpvcjwvZm9udD4NCjxi cj48Zm9udCBzaXplPSsxPmV4LWh1c2JhbmQsIGFuZCBNVUNIIG1vcmUhPC9mb250Pg0KPHA+ PGZvbnQgc2l6ZT0rMT5KdXN0IGxvb2sgYXQgc29tZSBvZiB0aGUgdGhpbmdzIHlvdSBjYW4g ZG8gd2l0aCB0aGlzIENEDQouLi48L2ZvbnQ+DQo8dWw+DQo8bGk+DQo8Yj48Zm9udCBjb2xv cj0iIzAwMDA5OSI+U2F2ZSBodW5kcmVkcyBvciBldmVuIHRob3VzYW5kcyBvZiBkb2xsYXJz IG9uDQp5b3VyIHRheGVzIGJ5IHVzaW5nIHRoZSBzZWNyZXQgdGF4IHRpcHMgY29udGFpbmVk IGluIHRoaXMgcGFja2FnZS4mbmJzcDsNClRoZSBJUlMgZG9lcyBOT1Qgd2FudCB5b3UgdG8g a25vdyB0aGVzZSBsb29waG9sZXMhPC9mb250PjwvYj48L2xpPg0KDQo8bGk+DQo8Yj48Zm9u dCBjb2xvcj0iIzAwMDA5OSI+VHJhY2sgZG93biBmcmllbmRzIGFuZCBvbGQgZmxhbWVzIGZy b20gdGhlIGNvbWZvcnQNCm9mIHlvdXIgb3duIGhvbWUgd2l0aCBvdXIgSW50ZXJuZXQgaW52 ZXN0aWdhdGlvbiByZXNvdXJjZXMhPC9mb250PjwvYj48L2xpPg0KDQo8bGk+DQo8Yj48Zm9u dCBjb2xvcj0iIzAwMDA5OSI+RmluZCBvdXQgd2hlcmUgeW91ciBFWCBpcyBoaWRpbmcgdGhl aXIgbW9uZXkgdG8NCmF2b2lkIGFsaW1vbnksIGNoaWxkIHN1cHBvcnQsIG9yIGRpdm9yY2Ug c2V0dGxlbWVudHMuIFRoZW4gZ2V0IHlvdXIgaGFuZHMNCm9uIHRoYXQgbW9uZXkuJm5ic3A7 IEJsZWVkICdlbSBkcnkhJm5ic3A7IFByaXZhdGUgaW52ZXN0aWdhdG9ycyBjaGFyZ2UNCnRo b3VzYW5kcyBvZiBkb2xsYXJzIGZvciB0aGlzIHNlcnZpY2UsIGJ1dCB5b3UgY2FuIGRvIGl0 IHdpdGggYSBjb21wdXRlcg0KYW5kIGFuIGludGVybmV0IGNvbmVjdGlvbiB3aGVuIHlvdSBi dXkgdGhlIEJhbm5lZCBDRCE8L2ZvbnQ+PC9iPjwvbGk+DQoNCjxsaT4NCjxiPjxmb250IGNv bG9yPSIjMDAwMDk5Ij5MZWFybiBob3cgdG8gdXNlIG9mZnNob3JlIG1vbmV5IGhhdmVucyBh bmQgYXNzZXQNCnByb3RlY3Rpb24gdHJ1c3RzIHRvIGF2b2lkIGxhd3N1aXRzLCBqdWRnbWVu dHMsIGFuZCBmb29sIHRoZSBtb3N0IGFnZ3Jlc3NpdmUNCnRheCBjb2xsZWN0b3IhJm5ic3A7 IElmIHlvdXIgbW9uZXkgaXMgc2l0dGluZyBpbiBhIFVTIGJhbmsgYWNjb3VudCwgeW91cg0K ZnVuZHMgY2FuIGJlIGxldmllZCBvciBmcm96ZW4gY29tcGxldGVseSBhdCBhbnkgdGltZS4m bmJzcDsgVGhlIFVTIGdvdmVybm1lbnQNCnVzZXMgImNpdmlsIGFzc2V0IGZvcmZlaXR1cmUi IHRvIHN0ZWFsIGJpbGxpb25zIG9mIGRvbGxhcnMgZWFjaCB5ZWFyIGZyb20NCmhhcmQtd29y a2luZywgbGF3IGFiaWRpbmcgY2l0aXplbnMuJm5ic3A7IEdldCB5b3VyIG1vbmV5IG91dCBv ZiB0aGUgY291bnRyeQ0KYmVmb3JlIFVuY2xlIFNhbSBzbmF0Y2hlcyBpdCBhbGwuPC9mb250 PjwvYj48L2xpPg0KDQo8bGk+DQo8Yj48Zm9udCBjb2xvcj0iIzAwMDA5OSI+RmluZCBvdXQg d2hlcmUgdG8gYnV5ICJmb3JiaWRkZW4gcHJvZHVjdHMiIG9uDQp0aGUgSW50ZXJuZXQuIEZ1 cnRoZXIgZWxhYm9yYXRpb24gc2hvdWxkIG5vdCBiZSBuZWNlc3Nhcnk7IGp1c3QgdXNlIHlv dXINCmltYWdpbmF0aW9uLjwvZm9udD48L2I+PC9saT4NCg0KPGxpPg0KPGI+PGZvbnQgY29s b3I9IiMwMDAwOTkiPkdldCBhIGJldHRlciBqb2IgYnkgcHVyY2hhc2luZyBhIGNvbGxlZ2Ug ZGVncmVlDQooaW5jbHVkaW5nIGEgUGhkISkgZm9yIGEgdmVyeSBsb3cgZmVlLiBObyBzdHVk eSByZXF1aXJlZCE8L2ZvbnQ+PC9iPjwvbGk+DQoNCjxsaT4NCjxiPjxmb250IGNvbG9yPSIj MDAwMDk5Ij5JZiB5b3VyIEVYIGlzIGNvbWluZyBhZnRlciB5b3VyIG1vbmV5LCBzdGF5IG9u ZQ0Kc3RlcCBhaGVhZCBvZiB0aGUgbGF3eWVycyBhbmQga2VlcCB0aGVpciBncmVlZHkgaGFu ZHMgb2ZmIHlvdXIgbG9vdC4gRmluZA0Kb3V0IGhvdyB0byBoaWRlIHlvdXIgbW9uZXkgd2hl cmUgaXQgd2lsbCBuZXZlciBiZSBmb3VuZC48L2ZvbnQ+PC9iPjwvbGk+DQoNCjxsaT4NCjxi Pjxmb250IGNvbG9yPSIjMDAwMDk5Ij5Bdm9pZCBsZWdhbCBwcm9ibGVtcywganVkZ21lbnRz LCBjb252aWN0aW9ucywNCmFuZCBldmVuIHByaXNvbiBzZW50ZW5jZXMgYnkgb2J0YWluaW5n IGEgY29tcGxldGVseSBuZXcgaWRlbnRpdHkgYW5kIGRpc2FwcGVhcmluZw0Kd2l0aG91dCBh IHRyYWNlITwvZm9udD48L2I+PC9saT4NCg0KPGxpPg0KPGI+PGZvbnQgY29sb3I9IiMwMDAw OTkiPkVyYXNlIHlvdXIgY3JpbWluYWwgcmVjb3JkIGFuZCBvYnRhaW4gYSBjb3B5IG9mDQp5 b3VyIEZCSSBmaWxlIHVzaW5nIHRoZSBGcmVlZG9tIG9mIEluZm9ybWF0aW9uIEFjdC4mbmJz cDsgRmluZCBvdXQgaWYgdGhlDQpnb3Zlcm5tZW50IGlzIGludmVzdGlnYXRpbmcgeW91IE5P VyBiZWZvcmUgaXQncyB0b28gTEFURSE8L2ZvbnQ+PC9iPjwvbGk+DQo8L3VsPg0KPGZvbnQg c2l6ZT0rMT5JZiB5b3UgaGF2ZSBiZWVuIHB1dHRpbmcgb2ZmIGJ1eWluZyBUaGUgQmFubmVk IENELCB3aGF0IGFyZQ0KeW91PC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+d2FpdGluZyBm b3I/Jm5ic3A7IEJpZyBCcm90aGVyIGlzIGFscmVhZHkgYnJlYXRoaW5nIGRvd24NCm15IG5l Y2ssIHNvPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+SSBjYW4ndCBrZWVwIHNlbGxpbmcg aXQgZm9yZXZlci4mbmJzcDsgSWYgeW91IGtlZXAgd2FpdGluZw0KdG8gcGxhY2U8L2ZvbnQ+ DQo8YnI+PGZvbnQgc2l6ZT0rMT55b3VyIG9yZGVyLCB5b3UgbWF5IG5ldmVyIGZpbmQgVGhl IEJhbm5lZCBDRCBhZ2Fpbi4mbmJzcDsNCkZvciBvbmx5PC9mb250Pg0KPGJyPjxmb250IHNp emU9KzE+JDE5Ljk5LCB5b3Ugd2lsbCBnYWluIHZhbHVhYmxlIHBlYWNlLW9mLW1pbmQga25v d2luZw0KaG93IHRvPC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+c2FmZWd1YXJkIHlvdXJz ZWxmLCB5b3VyIGZhbWlseSwgYW5kIHlvdXIgbW9uZXkuJm5ic3A7DQpXaG8gY2FuIHB1dCBh PC9mb250Pg0KPGJyPjxmb250IHNpemU9KzE+cHJpY2Ugb24gZnJlZWRvbT8hPC9mb250Pg0K PHA+PGk+PHU+PGZvbnQgc2l6ZT0rMT5IZXJlIGlzIGFub3RoZXIgbG9vayBhdCB0aGUgY29u dGVudHMgb2YgdGhlIEJBTk5FRA0KQ0Q6PC9mb250PjwvdT48L2k+DQo8cD48aW1nIFNSQz0i aHR0cDovL2NvbXB1em9uZXVzYS5jb20vaW1hZ2VzL2J1dHRvbi5naWYiIGhlaWdodD0xOSB3 aWR0aD0xOT48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT4NCkZpbmQgY29u ZmlkZW50aWFsIGluZm8gb24gYW55b25lIGluIDMwIG1pbnV0ZXMgb3I8L2ZvbnQ+PC9mb250 Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPmxlc3Mgb24gdGhl IEludGVybmV0LiBZb3UnbGwgYmUNCmFibGUgdG8gdHJhY2sgZG93biB5b3VyPC9mb250Pjwv Zm9udD4NCjxicj48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5vbGQgZmxh bWUsIGZpbmQgb3V0IGhvdyBtdWNoIG1vbmV5DQp5b3VyIGV4IGlzIGhpZGluZzwvZm9udD48 L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+aW4gdGhl aXIgYmFuayBhY2NvdW50LCBvciBydW4gYQ0KYmFja2dyb3VuZCBjaGVjayBvbjwvZm9udD48 L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+cHJvZXNw ZWN0aXZlIGNsaWVudCBvciBlbXBsb3llZS4NCkV2ZW4gZ292ZXJubWVudDwvZm9udD48L2Zv bnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+YWdlbmNpZXMg aGF2ZSB0cm91YmxlIG9idGFpbmluZw0KbXVjaCBvZiB0aGlzIGluZm9ybWF0aW9uLjwvZm9u dD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+WW91 IHdpbGwgaGF2ZSBhbGwgdGhlIHJlc291cmNlcw0Kb2YgYW55IHByb2Zlc3Npb25hbDwvZm9u dD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+aW52 ZXN0aWdhdG9yIHJpZ2h0IGF0IHlvdXIgZmluZ2VydGlwcw0KLSBvbiB5b3VyIGhvbWU8L2Zv bnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPmNv bXB1dGVyITwvZm9udD48L2ZvbnQ+DQo8cD48aW1nIFNSQz0iaHR0cDovL2NvbXB1em9uZXVz YS5jb20vaW1hZ2VzL2J1dHRvbi5naWYiIGhlaWdodD0xOSB3aWR0aD0xOT48Zm9udCBjb2xv cj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT4NCkxpc3Qgb2YgY29tcGFuaWVzIHdobyB3aWxs IGlzc3VlIHlvdSBhIGNvbGxlZ2U8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIj NjY2NjY2Ij48Zm9udCBzaXplPSsxPmRlZ3JlZSAoaW5jbHVkaW5nIGEgUGhkLikgZm9yIGEN CmZlZS4gTm8gc3R1ZHkgcmVxdWlyZWQhPC9mb250PjwvZm9udD4NCjxwPjxpbWcgU1JDPSJo dHRwOi8vY29tcHV6b25ldXNhLmNvbS9pbWFnZXMvYnV0dG9uLmdpZiIgaGVpZ2h0PTE5IHdp ZHRoPTE5Pjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPg0KTGVhcm4gaG93 IHRvIGdldCBGUkVFIGNhYmxlIGFuZCBEU1MgY2hhbm5lbHMsPC9mb250PjwvZm9udD4NCjxi cj48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5pbmNsdWRpbmcgdGhlIGFk dWx0IHN0YXRpb25zIGFuZA0KUGF5LVBlci1WaWV3ITwvZm9udD48L2ZvbnQ+DQo8cD48aW1n IFNSQz0iaHR0cDovL2NvbXB1em9uZXVzYS5jb20vaW1hZ2VzL2J1dHRvbi5naWYiIGhlaWdo dD0xOSB3aWR0aD0xOT48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT4NCkhv dyB0byBPYnRhaW4gTWljcm9zb2Z0IFByb2R1Y3RzIGFic29sdXRlbHk8L2ZvbnQ+PC9mb250 Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPkZSRUUsIHRoZSBz YWZlIGFuZCBMRUdBTCB3YXkhPC9mb250PjwvZm9udD4NCjxwPjxpbWcgU1JDPSJodHRwOi8v Y29tcHV6b25ldXNhLmNvbS9pbWFnZXMvYnV0dG9uLmdpZiIgaGVpZ2h0PTE5IHdpZHRoPTE5 Pjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPg0KTGlzdCBvZiBzdXBwbGll cnMgb2YgInF1ZXN0aW9uYWJsZSIgaXRlbXMsIHN1Y2g8L2ZvbnQ+PC9mb250Pg0KPGJyPjxm b250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPmFzIGZjYyBiYW5uZWQgY29tbXVu aWNhdGlvbnMgZGV2aWNlcywNCnNjYW5uZXJzLDwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQg Y29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+bmlnaHQgdmlzaW9uIGdvZ2dsZXMsIHBh c3Nwb3J0cywNCmZha2UgaWRlbnRpZmljYXRpb24sPC9mb250PjwvZm9udD4NCjxicj48Zm9u dCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5hbmQgZXZlcnl0aGluZyBlbHNlIHRo YXQgeW91IHdvbid0DQpmaW5kIGluIHRoZSBiYWNrPC9mb250PjwvZm9udD4NCjxicj48Zm9u dCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5vZiBTb2xkaWVyIG9mIEZvcnR1bmUg bWFnYXppbmUhPC9mb250PjwvZm9udD4NCjxwPjxpbWcgU1JDPSJodHRwOi8vY29tcHV6b25l dXNhLmNvbS9pbWFnZXMvYnV0dG9uLmdpZiIgaGVpZ2h0PTE5IHdpZHRoPTE5Pjxmb250IGNv bG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPg0KRmluZCBwcm9kdWN0cyB0byByZXNlbGwg b24gdGhlIEludGVybmV0IHdpdGggb3VyPC9mb250PjwvZm9udD4NCjxicj48Zm9udCBjb2xv cj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5XaG9sZXNhbGUgRGlyZWN0b3J5IGNvbnRhaW5p bmcNCm92ZXIgMSBtaWxsaW9uPC9mb250PjwvZm9udD4NCjxicj48Zm9udCBjb2xvcj0iIzY2 NjY2NiI+PGZvbnQgc2l6ZT0rMT4odGhhdCdzIHJpZ2h0LCBvbmUgbWlsbGlvbiEpIHdob2xl c2FsZQ0Kc291cmNlcyBmb3I8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2 NjY2Ij48Zm9udCBzaXplPSsxPmNvbXB1dGVycywgZWxlY3Ryb25pY3MsIGJlYW5pZXMsDQp0 cmVuZCBpdGVtcyw8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48 Zm9udCBzaXplPSsxPmpld2VscnksIGNhcnMsIGNvbGxlY3RpYmxlcywgYW5kDQpldmVyeXRo aW5nIGVsc2UhPC9mb250PjwvZm9udD4NCjxwPjxpbWcgU1JDPSJodHRwOi8vY29tcHV6b25l dXNhLmNvbS9pbWFnZXMvYnV0dG9uLmdpZiIgaGVpZ2h0PTE5IHdpZHRoPTE5Pjxmb250IGNv bG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPg0KT3ZlciAyNSBtaWxsaW9uIGVtYWlsIGFk ZHJlc3NlcywgZnJlc2ggYW5kPC9mb250PjwvZm9udD4NCjxicj48Zm9udCBjb2xvcj0iIzY2 NjY2NiI+PGZvbnQgc2l6ZT0rMT50YXJnZXRlZCEgWW91IGNhbiB1c2UgdGhlc2UgdG8NCmNv bnRhY3QgdGhlIHBlb3BsZTwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2 NjYiPjxmb250IHNpemU9KzE+YW5kIHNlbmQgdGhlbSB5b3VyIGFkdmVydGlzZW1lbnRzLjwv Zm9udD48L2ZvbnQ+DQo8cD48aW1nIFNSQz0iaHR0cDovL2NvbXB1em9uZXVzYS5jb20vaW1h Z2VzL2J1dHRvbi5naWYiIGhlaWdodD0xOSB3aWR0aD0xOT48Zm9udCBjb2xvcj0iIzY2NjY2 NiI+PGZvbnQgc2l6ZT0rMT4NCkZpbmQgb3V0IGhvdyB0byBnZXQgYSBjb21wbGV0ZWx5IG5l dyBpZGVudGl0eS48L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48 Zm9udCBzaXplPSsxPkRpc2FwcGVhciB3aXRob3V0IGEgdHJhY2UhPC9mb250PjwvZm9udD4N CjxwPjxpbWcgU1JDPSJodHRwOi8vY29tcHV6b25ldXNhLmNvbS9pbWFnZXMvYnV0dG9uLmdp ZiIgaGVpZ2h0PTE5IHdpZHRoPTE5Pjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXpl PSsxPg0KRmluZCBvdXQgaG93IHRvIEVSQVNFIGJhZCBjcmVkaXQgYW5kIGV2ZW48L2ZvbnQ+ PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPmNyZWF0 ZSBhIHdob2xlIG5ldyBmaWxlIGluIHRoZQ0KY3JlZGl0IGJ1cmVhdSBjb21wdXRlcnMuPC9m b250PjwvZm9udD4NCjxwPjxpbWcgU1JDPSJodHRwOi8vY29tcHV6b25ldXNhLmNvbS9pbWFn ZXMvYnV0dG9uLmdpZiIgaGVpZ2h0PTE5IHdpZHRoPTE5Pjxmb250IGNvbG9yPSIjNjY2NjY2 Ij48Zm9udCBzaXplPSsxPg0KTGVhcm4gaG93IHRvIGJlYXQgdGhlIElSUy4gVGF4IHRpcHMg Zm9yIHRoZTwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250 IHNpemU9KzE+cmVzdCBvZiB1cy48L2ZvbnQ+PC9mb250Pg0KPHA+PGltZyBTUkM9Imh0dHA6 Ly9jb21wdXpvbmV1c2EuY29tL2ltYWdlcy9idXR0b24uZ2lmIiBoZWlnaHQ9MTkgd2lkdGg9 MTk+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+DQpDb21wbGV0ZSBndWlk ZSBvbiBob3cgdG8gY2xhaW0gcHVibGljIGxhbmQ8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250 IGNvbG9yPSIjNjY2NjY2Ij48Zm9udCBzaXplPSsxPm9mZmVyZWQgYnkgdGhlIEdvdmVybm1l bnQuIFlvdQ0KY2FuIGZpbmQgeW91cjwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9 IiM2NjY2NjYiPjxmb250IHNpemU9KzE+ZHJlYW0gaG9tZSBmb3IgYSByaWRpY3Vsb3VzbHkg bG93DQpwcmljZSBvciBidXk8L2ZvbnQ+PC9mb250Pg0KPGJyPjxmb250IGNvbG9yPSIjNjY2 NjY2Ij48Zm9udCBzaXplPSsxPnByb3BlcnRpZXMgdG8gcmVzZWxsIGZvciBhIGh1Z2UNCnBy b2ZpdCE8L2ZvbnQ+PC9mb250Pg0KPHA+PGltZyBTUkM9Imh0dHA6Ly9jb21wdXpvbmV1c2Eu Y29tL2ltYWdlcy9idXR0b24uZ2lmIiBoZWlnaHQ9MTkgd2lkdGg9MTk+PGZvbnQgY29sb3I9 IiM2NjY2NjYiPjxmb250IHNpemU9KzE+DQpBY2NlcHQgY2hlY2tzIGZyb20geW91ciBjdXN0 b21lcnMgYnkgZmF4LDwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYi Pjxmb250IHNpemU9KzE+cGhvbmUgYW5kIGVtYWlsIHdpdGggdGhlIGZ1bGwgd29ya2luZw0K dmVyc2lvbiBvZjwvZm9udD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxm b250IHNpemU9KzE+Q2hlY2tlciBTb2Z0d2FyZSBpbmNsdWRlZCE8L2ZvbnQ+PC9mb250Pg0K PHA+PGltZyBTUkM9Imh0dHA6Ly9jb21wdXpvbmV1c2EuY29tL2ltYWdlcy9idXR0b24uZ2lm IiBoZWlnaHQ9MTkgd2lkdGg9MTk+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9 KzE+DQpCdXNpbmVzcyBzb2Z0d2FyZSB0byBoZWxwIHlvdSBydW4geW91ciBzbWFsbDwvZm9u dD48L2ZvbnQ+DQo8YnI+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+YnVz aW5lc3MsIGluY2x1ZGluZyBsYWJlbCBtYWtlcg0Kc29mdHdhcmUsIG1vbmV5PC9mb250Pjwv Zm9udD4NCjxicj48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5tYW5hZ2Vt ZW50IHNvZnR3YXJlLCBJUlMgZm9yZ2l2ZW5lc3MNCnByb2dyYW0sPC9mb250PjwvZm9udD4N Cjxicj48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT5kYXRhYmFzZSBzb2Z0 d2FyZSwgYW5kIG11Y2ggbW9yZS48L2ZvbnQ+PC9mb250Pg0KPHA+PGltZyBTUkM9Imh0dHA6 Ly9jb21wdXpvbmV1c2EuY29tL2ltYWdlcy9idXR0b24uZ2lmIiBoZWlnaHQ9MTkgd2lkdGg9 MTk+PGZvbnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+DQpBIGh1Z2UgY29sbGVj dGlvbiBvZiBzY3JlZW5zYXZlcnMsIGdyYXBoaWNzLDwvZm9udD48L2ZvbnQ+DQo8YnI+PGZv bnQgY29sb3I9IiM2NjY2NjYiPjxmb250IHNpemU9KzE+Y2xpcGFydCwgYW5kIGRlc2t0b3Ag dGhlbWVzIHRvDQpzcGljZSB1cCB5b3VyIGNvbXB1dGVyLjwvZm9udD48L2ZvbnQ+DQo8cD48 aW1nIFNSQz0iaHR0cDovL2NvbXB1em9uZXVzYS5jb20vaW1hZ2VzL2J1dHRvbi5naWYiIGhl aWdodD0xOSB3aWR0aD0xOT48Zm9udCBjb2xvcj0iIzY2NjY2NiI+PGZvbnQgc2l6ZT0rMT4N Ck1VQ0gsIE1VQ0ggTU9SRSB0aGF0IHdlIGRvbid0IGhhdmUgcm9vbSB0byBsaXN0ITwvZm9u dD48L2ZvbnQ+DQo8cD48Zm9udCBzaXplPSsxPldlIGFyZSBzbyBjb25maWRlbnQgdGhhdCB5 b3Ugd2lsbCBsb3ZlIHRoaXMgQ0QgdGhhdCB3ZTwvZm9udD4NCjxicj48Zm9udCBzaXplPSsx Pm9mZmVyIGEgZnVsbCAzMC1kYXksIG5vIHF1ZXN0aW9ucyBhc2tlZCBtb25leS1iYWNrPC9m b250Pg0KPGJyPjxmb250IHNpemU9KzE+Z3VhcmFudGVlLiBUbyBvcmRlciB0aGUgIkJhbm5l ZCBDRCIgcmlnaHQgbm93IGZvciB0aGU8L2ZvbnQ+DQo8YnI+PGZvbnQgc2l6ZT0rMT5zdXBl ciBsb3cgcHJpY2Ugb2Ygb25seSAkMTkuOTksIGp1c3QgY2xpY2sgb24gdGhlIGxpbms8L2Zv bnQ+DQo8YnI+PGZvbnQgc2l6ZT0rMT5iZWxvdyB0byBwYXkgd2l0aCBhIFZpc2Egb3IgTWFz dGVyY2FyZDo8L2ZvbnQ+DQo8YnI+Jm5ic3A7DQo8dGFibGUgQk9SREVSPTAgQ09MUz0yIFdJ RFRIPSI1MCUiID4NCjx0cj4NCjx0ZD48Zm9udCBzaXplPSsyPjxhIGhyZWY9Imh0dHA6Ly93 d3cuY29tcHV6b25ldXNhLmNvbS9jY3BheW1lbnQvYmFubmVkY2Q3Mi5odG0iPk9yZGVyDQpO b3chPC9hPjwvZm9udD48L3RkPg0KDQo8dGQ+DQo8Y2VudGVyPjxpbWcgU1JDPSJodHRwOi8v Y29tcHV6b25ldXNhLmNvbS9pbWFnZXMvZ3VhcmFudGVlLmdpZiIgaGVpZ2h0PTk2IHdpZHRo PTk1PjwvY2VudGVyPg0KPC90ZD4NCjwvdHI+DQo8L3RhYmxlPg0KDQo8cD5PciB5b3UgbWF5 IHBheSB3aXRoIGNhc2gsIGNoZWNrIG9yIG1vbmV5IG9yZGVyIGJ5IHByaW50aW5nIHRoZSBv cmRlcg0KPGJyPmZvcm0gYmVsb3cgYW5kIHNlbmRpbmcgaXQgd2l0aCB5b3VyIHBheW1lbnQu DQo8cD48Yj4tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tIENVVCBIRVJFIC0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tPC9iPg0KPHA+UHJvZHVjdDogIlRoZSBCYW5uZWQgQ0QiDQo8YnI+UHJp Y2U6ICQxOS45OSArICQ0LjAxIHNoaXBwaW5nL2hhbmRsaW5nDQo8cD5IT1cgVE8gT1JERVIg QlkgTUFJTDogUHJpbnQgb3V0IHRoaXMgb3JkZXIgZm9ybSBhbmQgc2VuZCBjYXNoLA0KPGJy PnBlcnNvbmFsIGNoZWNrLCBtb25leSBvcmRlciBvciBjYXNoaWVyJ3MgY2hlY2sgdG8gdGhl IGFkZHJlc3MgbGlzdGVkDQpiZWxvdzoNCjxwPlF1aWtzaWx2ZXIgRW50ZXJwcmlzZXMgSW5j Lg0KPGJyPjM3OTIgQnJvYWR3YXkgIzM5OA0KPGJyPk5ldyBZb3JrLCBOWSAxMDAzMg0KPHA+ WW91ciBTaGlwcGluZyBJbmZvcm1hdGlvbjoNCjxwPllvdXIgTmFtZV9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPGJyPllvdXIgQWRkcmVzc19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPGJyPllvdXIgQ2l0eV9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPGJyPlN0YXRl IC8gWmlwX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPGJy PlBob25lICM6IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Xw0KPGJyPihGb3IgcHJvYmxlbXMgd2l0aCB5b3VyIG9yZGVyIG9ubHkuIE5vIHNhbGVzbWVu IHdpbGwgY2FsbC4pDQo8cD5FbWFpbCBBZGRyZXNzX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXw0KPHA+UGxlYXNlIG5vdGUgdGhhdCBtYWlsLWluIG9yZGVycyBt YXkgYmUgZGVsYXllZCAyLTQgd2Vla3MuDQo8cD5bIF0gSSBhbSBlbmNsb3NpbmcgYSBjaGVj ayBvciBtb25leSBvcmRlciBmb3IgJDI0LjAwDQo8cD5bIF0gSSBhbSBtYWlsaW5nIG15IGNy ZWRpdCBjYXJkIG51bWJlci4gKE5vdGUgeW91ciBjYXJkIHdpbGwgYmUgY2hhcmdlZA0KZm9y ICQyNC4wMCkNCjxwPklmIHBheWluZyBieSBjcmVkaXQgY2FyZCwgcGxlYXNlIGZpbGwgaW4g dGhlIGluZm9ybWF0aW9uIGJlbG93Og0KPHA+Q3JlZGl0IENhcmQgTnVtYmVyOl9fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fDQo8YnI+RXhwaXJhdGlvbiBEYXRlOl9fX19fX19f X19fX19fX19fX19fX19fX19fXw0KPGJyPlNpZ25hdHVyZTpfX19fX19fX19fX19fX19fX19f X19fX19fDQo8YnI+RGF0ZTpfX19fX19fX19fX19fX19fX19fXw0KPHA+Tk9URTogVGhpcyBD RCBpcyBmb3IgaW5mb3JtYXRpb25hbCwgZWR1Y2F0aW9uYWwsIG9yDQo8YnI+ZW50ZXJ0YWlu bWVudCBwdXJwb3NlcyBvbmx5LiZuYnNwOyBTb21lIG9mIHRoZSBhY3Rpdml0aWVzDQo8YnI+ ZGVzY3JpYmVkIG9uIHRoaXMgQ0QgbWF5IGJlIGlsbGVnYWwgaWYgY2FycmllZCBvdXQuDQo8 YnI+VGhpcyBDRCBjb250YWlucyBsaW5rcyBhbmQgYWRkcmVzc2VzIG9mIGNvbXBhbmllcw0K PGJyPmFuZCBpbmRpdmlkdWFscyB3aGljaCBwcm9kdWNlIHZhcmlvdXMgcHJvZHVjdHMsIG9y DQo8YnI+b2ZmZXIgdmFyaW91cyBzZXJ2aWNlcywgd2hpY2ggbWF5IGJlIGlsbGVnYWwgaW4g eW91cg0KPGJyPmNvdW50cnksIHN0YXRlLCBvciBjaXR5LiZuYnNwOyBIb3dldmVyLCBpdCBp cyBwZXJmZWN0bHkNCjxicj5MRUdBTCB0byBwdXJjaGFzZSBhbmQgcG9zc2VzcyB0aGUgQmFu bmVkIENEIGluIHRoZQ0KPGJyPlVuaXRlZCBTdGF0ZXMgb2YgQW1lcmljYS4mbmJzcDsgQ29u dGFpbnMgYXBwcm94aW1hdGVseQ0KPGJyPjEwMCBtZWdzIG9mIGluZm9ybWF0aW9uLg0KPHA+ U2hpcHBpbmcgb3V0c2lkZSBVU0EgYWRkICQxMC4wMA0KPHA+KioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCjxicj5Zb3UgYXJl IHJlY2lldmluZyBhbiB1bnNvbGljaXRlZCBjb21tZXJjaWFsIGVtYWlsIGZyb20NCjxicj5R dWlrc2lsdmVyIEVudGVycHJpc2VzIEluYy4mbmJzcDsgV2UgcmVjb2duaXplIHRoYXQgeW91 IG1heQ0KPGJyPm5vdCB3aXNoIHRvIHJlY2lldmUgc3VjaCBlbWFpbHMgaW4gdGhlIGZ1dHVy ZSwgYW5kIHdlDQo8YnI+aGF2ZSBwcm92aWRlZCBhIGxpbmsgdG8gYXV0b21hdGljYWxseSBh bmQgaW5zdGFudGx5DQo8YnI+cmVtb3ZlIHlvdXJzZWxmOiA8YSBocmVmPSJodHRwOi8vd3d3 LmNvbXB1em9uZXVzYS5jb20vcmVtb3ZlLmh0bSI+UkVNT1ZFPC9hPg0KPGJyPlRoaXMgbWVz c2FnZSBpcyBiZWluZyBzZW50IHRvIHlvdSBpbiBjb21wbGlhbmNlIHdpdGgNCjxicj5mZWRl cmFsIGd1aWRlbGluZXMgZ292ZXJuaW5nIHRoZSB0cmFuc21pc3Npb24gb2YNCjxicj51bnNv bGljaXRlZCBjb21tZXJjaWFsIGVtYWlsLCBpbmNsdWRpbmcgdGhlIHByb3Zpc2lvbiBvZg0K PGJyPmNvbnRhY3QgaW5mb3JtYXRpb24gZm9yIG91ciBjb21wYW55IHdpdGhpbiB0aGlzIGVt YWlsLCBhDQo8YnI+dmFsaWQgcmV0dXJuIGVtYWlsIGFkZHJlc3MsIGFuZCBhIHdheSBmb3Ig Y3VzdG9tZXJzDQo8YnI+dG8gcmVtb3ZlIHRoZW1zZWx2ZXMuJm5ic3A7IFBsZWFzZSBub3Rl LCBob3dldmVyLCB0aGF0IG91cg0KPGJyPmVtYWlsIGFkZHJlc3Mgd2FzIHZhbGlkIGF0IHRo ZSB0aW1lIG9mIHNlbmRpbmcsIGJ1dCBtYXkNCjxicj5iZSBjYW5jZWxsZWQgYnkgdGhlIElT UCBzaG9ydGx5IHRoZXJlYWZ0ZXIgZHVlIHRvIG91cg0KPGJyPnVzZSBvZiB1bnNvbGljaXRl ZCBlbWFpbC4mbmJzcDsgWW91IGNhbiByZWFkIGFib3V0IHRoZSB2YXJpb3VzDQo8YnI+bGF3 cyBnb3Zlcm5pbmcgdW5zb2xpY2l0ZWQgZW1haWw6IGh0dHA6Ly9zcGFtbGF3cy5jb20NCjxi cj4iVW5zb2xpY2l0ZWQgY29tbWVyY2lhbCBlbGVjdHJvbmljIG1haWwgY2FuIGJlIGFuIGlt cG9ydGFudA0KPGJyPm1lY2hhbmlzbSB0aHJvdWdoIHdoaWNoIGJ1c2luZXNzZXMgYWR2ZXJ0 aXNlIGFuZCBhdHRyYWN0DQo8YnI+Y3VzdG9tZXJzIGluIHRoZSBvbmxpbmUgZW52aXJvbm1l bnQuIiZuYnNwOyAtLSBVLlMuIENvbmdyZXNzLA0KPGJyPkguUi4gMTA2dGggQ09OR1JFU1Mg MmQgU2Vzc2lvbiBILiBSLiAzMTEzIFNlYy4gMiAoYSkzIEp1bHkNCjxicj4xOSwgMjAwMCBo dHRwOi8vd3d3LnNwYW1sYXdzLmNvbS9mZWRlcmFsL2hyMzExMy5odG1sDQo8YnI+KioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioN Cjxicj4mbmJzcDsNCjxicj4mbmJzcDsNCjwvYm9keT4NCjwvaHRtbD4NCg== ------=_NextPart_000_001__17122755_58420.84-- From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 11:13:17 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA11573 for ; Tue, 30 Jan 2001 11:13:16 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA26564; Tue, 30 Jan 2001 08:12:36 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA09726; Tue, 30 Jan 2001 08:11:04 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UGAm221377 for ; Tue, 30 Jan 2001 08:10:48 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id IAA09632 for ; Tue, 30 Jan 2001 08:10:48 -0800 (PST) Received: from donkeykong.gpcc.itd.umich.edu (donkeykong.gpcc.itd.umich.edu [141.211.2.163]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA25249 for ; Tue, 30 Jan 2001 08:10:47 -0800 (PST) Received: from breakout.gpcc.itd.umich.edu (smtp@breakout.gpcc.itd.umich.edu [141.211.2.141]) by donkeykong.gpcc.itd.umich.edu (8.8.8/4.3-mailhub) with ESMTP id LAA13821 for ; Tue, 30 Jan 2001 11:10:47 -0500 (EST) Received: from localhost (kmsmith@localhost) by breakout.gpcc.itd.umich.edu (8.8.8/5.1-client) with ESMTP id LAA02007 for ; Tue, 30 Jan 2001 11:10:46 -0500 (EST) Precedence: first-class Date: Tue, 30 Jan 2001 11:10:46 -0500 (EST) From: "Kendrick M. Smith" X-Sender: kmsmith@breakout.gpcc.itd.umich.edu To: nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Mon, 29 Jan 2001, Sergey Klyushin wrote: > > I don't think you have to separately save the last superseded (I > > think that's what you mean here; expired has some other > > connotations) stateid. > > > > What I do is first use the stateid to map to an entry in my table of > > state structures. At this point, I may generate STALE_STATEID and > > BAD_STATEID errors, but not OLD_STATEID. This requires that I know > > the range of valid values of the stateid subfield which is incremented > > on each change, but I need that anyway to properly distinguish between > > BAD_STATEID and OLD_STATEID. > > > > Once I have the address of a state structure, I can use this to > > find the corresponding owner structure which has the current seqid > > and replay information. So at this point, I can notice a replay > > and respond. > > In our server implementation, we can't determine the lockowner corresponding to an old stateid without retaining an entry in our stateid table. (We found doing this to be impractical using 64-bit stateids.) So it seems that retaining the last superseded stateid may or may not be an issue, depending on the server implementation. I do think that this would be a useful remark for the implementor's guidelines (but probably not needed in the spec). > BTW, how useful ERROR_OLD_STATEID for client? What client suppose to do > after receiving this error? I was wondering this too.. On our client, we couldn't really make use of the distinction between NFS4ERR_OLD_STATEID and NFS4ERR_BAD_STATEID (although NFS4ERR_STALE_STATEID is certainly useful, as an entry point to the server-reboot recovery code). Offhand, does anyone know the rationale for making seperate errors NFS4ERR_OLD_STATEID, NFS4ERR_BAD_STATEID? > So, you don't store stateid (or even full request) for reply message, > but you are based only on seqid = stored_seqid? How you could be sure, > that it's real replay, but not logical error? I want to be sure at > this point, that I understand replay correctly: for me it's real > retransmission of exactly the same package. Is it true? BTW, do you > check lease time before reply? We only retransmit the response if the old request and new request are byte-by-byte identical, including rpc headers. The client's lease is checked and renewed only if the seqid is correct. Is this what everyone else is doing? > > I don't think so. In the case of open, I don't see how you can do > without > > a seqid. The fact that it generates a stateid (updated or not) > doesn't > > give you any replay protection. > > > > I send an open for read-write, deny=read-write. It succeeds and I > send > > back a stateid. The reply is lost. A replay of the OPEN is sent and > > it fails because open for read or write is denied, which is definitely > > not what you want. You might fall back on various kind of ad hoc > replay > > protection mechanisms specific to OPEN but that doesn't give me a very > good > > feeling. > I don't see any place when you need both seqid and stateid in one request. > OPEN may take seqid and generate stateid, and stateid will be used later > without seqid > But, probably, changing seqid to stateid is really too late now > (even if it's needed only for OPEN). It does seem that, if per-lockowner seqids were used only for OPEN, and per-(lockowner, file) stateids were used everywhere else, replay protection would still be ensured. But this would have the side effect of requiring the server to cache replays on a per-(lockowner, file) basis; my feeling is that this would negatively impact performance. Kendrick From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 12:24:02 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA13345 for ; Tue, 30 Jan 2001 12:24:02 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA12383; Tue, 30 Jan 2001 09:23:20 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id JAA04768; Tue, 30 Jan 2001 09:14:24 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UHEB221400 for ; Tue, 30 Jan 2001 09:14:12 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id JAA14713 for ; Tue, 30 Jan 2001 09:14:11 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA11466 for ; Tue, 30 Jan 2001 09:14:11 -0800 (PST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0UHEB210962; Tue, 30 Jan 2001 09:14:11 -0800 (PST) Received: from ussvlexc06.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0UHEAG18395; Tue, 30 Jan 2001 09:14:10 -0800 (PST) Received: by ussvlexc06.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 30 Jan 2001 09:14:44 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E57@red.nane.netapp.com> From: "Noveck, Dave" To: "'Sergey Klyushin'" , "Noveck, Dave" , nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid Date: Tue, 30 Jan 2001 09:14:07 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sergei Klyushin wrote: > > > > Kendrick Smith wrote: > > > > > > On Sat, 27 Jan 2001, Sergey Klyushin wrote: > > > > > > > If implementation of NFS Server changes stateid on reply > > > > (otherwise I don't see why to return stateid, if it's not changed), > > > > I can't see correct processing of replay scenario. > > > > I'll try to explain with more details. > > > > Replay request has seqid = stored_seqid, but request's stateid != > > > > stored_stateid. > > > > So, to follow above scenario (first check stateid, then check seqid) error > > > > BAD_STATEID or > > > > OLD_STATEID will be returned, before checking seqid. > > > > > > I suppose this means that, to guarantee correct replay protection, a > > > server implementation must retain the last expired stateid (per lockowner) > > > in its stateid table. This way, if a replay is received, the server can > > > recognize the replayed stateid, compare seqids, and replay its cached > > > response instead of erroneously returning NFS4ERR_OLD_STATEID. This is a > > > good point which, to my knowledge, hasn't come up before. Should > > > something like this be written into the spec (or at least the > > > implementor's guidelines)? > > > > I don't think you have to separately save the last superseded (I think > > that's what you mean here; expired has some other > > connotations) stateid. > > > > What I do is first use the stateid to map to an entry in my table of > > state structures. At this point, I may generate STALE_STATEID and > > BAD_STATEID errors, but not OLD_STATEID. This requires that I know > > the range of valid values of the stateid subfield which is incremented > > on each change, but I need that anyway to properly distinguish between > > BAD_STATEID and OLD_STATEID. > > > How "old" could be stateid? Probably, any operation, that takes both > stateid and seqid, > makes range of valid values of the stateid subfield equal 1? > BTW, how useful ERROR_OLD_STATEID for client? What client suppose to do > after receiving this error? According to the spec, OLD_STATEID should be returned for any stateid that is not the current correct one but did designate (earlier) that same file-lockowner pair. So if an open assigns stateid X and you then LOCK/UNLOCK a bunch of times and the current stateid is X+n, then any stateid between X and X+n-1 (inclusive) results in OLD_STATEID. In the case of requests that have both a stateid and a seqid, there isn't much value in the distinction between between BAD_STATEID and OLD_STATEID. Getting either indicates that the client or server is broken somehow (or a router is stomping on your packets, etc.). The point of OLD_STATEID is for requests that don't have seqid's (e.g. READ and WRITE). In these cases BAD_STATEID indicates someone is broken, but OLD_STATEID doesn't. On OLD_STATEID, the client will probably retry to deal with a synchronization problem. Suppose you have IO requests in-flight when a subsequent request bumps the stateid. Depending on request ordering, the IO requests may get an OLD_STATEID error, upon receiving which, the client can retry with the new stateid. The client could wait for all such in-flight requests to return before issuing the new request but, sometimes he may not know at the time the request is issued(some cases of OPEN), such synchronization could be annoying given that IO requests typically are done (essentially) asynchronously, and in most cases ordering is good enough that the optimistic approach of just issuing the new request will work OK. OLD_STATEID allows the client a clean way of dealing with the case in which the stateid-bumping request and the IO request are received by the server in the "wrong" order. > > Once I have the address of a state structure, I can use this > > to find the > > corresponding owner structure which has the current seqid and replay > > information. So at this point, I can notice a replay and respond. > > > So, you don't store stateid (or even full request) for reply message, > but you are based > only on seqid = stored_seqid? How you could be sure, that it's real > replay, but not logical error? > I want to be sure at this point, that I understand replay correctly: > for me it's real > retransmission of exactly the same package. Is it true? Right now, I only check the seqid and operation. I might save more information if I find that there are broken clients that are sending a seqid that matches a previous request with a different request. Saving the whole operation seems excessive to me, particurlay given OPEN which can be fairly large. If this really turns out to be a problem (and I kind of doubt it will), then maybe saving a 64-checksum of the operation seems reasonable. The issues are similar to the normal reply cache. I don't know what other systems do, but there we match on transaction id and then check a few other items. We certainly don't save and check the entire request. I doubt many systems do. > BTW, do you check lease time before reply? Right now my lease expiration code isn't there yet. My inclination is to treat replay as merely retransmitting what already happened, so I wouldn't check it. > > If it is not a replay (and we don't have a BAD_SEQID error), then I > > can check the stateid against the current stateid in the > > state structure and generated aN OLD_STATEID error if appropriate. > > > > I thought this was discussed in the session Andy organized regarding > > order-of-processing issues at the last bake-off. I thought this > > was on the flow-charts that we came up with. Then I again I was > > pretty tired so maybe my memmory is faulty. In any case I think this > > should be documented somewhere (implementation RFC?). > > > > > > > > > BTW, seqid = stored_seqid doesn't mean real replay, because full request > > > > must be the same, and > > > > NFS Server should keep previous request to compare it with current one. In > > > > some implementation it's done on RPC layer. > > > > From my point of view, including index inside stateid is enough to monitor > > > > replay without seqid at all. (which makes live more easy) > > > > > > It sounds to me like you are suggesting removing seqids from the protocol > > > entirely, and relying on stateids to provide replay protection for > > > non-idempotent operations. Perusing the protocol, it seems that every > > > non-idempotent operation does change a stateid, so it should be possible > > > to use stateids for replay protection and still have the correct replay > > > semantics. > > > > I don't think so. In the case of open, I don't see how you can do without > > a seqid. The fact that it generates a stateid (updated or not) doesn't > > give you any replay protection. > > > > I send an open for read-write, deny=read-write. It succeeds and I send > > back a stateid. The reply is lost. A replay of the OPEN is sent and > > it fails because open for read or write is denied, which is definitely > > not what you want. You might fall back on various kind of ad hoc replay > > protection mechanisms specific to OPEN but that doesn't give me a very good > > feeling. > I don't see any place when you need both seqid and stateid in one request. > OPEN may take seqid and generate stateid, and stateid will be used later > without seqid It is possible that you might be able to come up with something along these lines. But I don't think it is easy. Unless you are prepared to devote 32 bits of the stateid to actually being the seqid (e.g. for sequencing of OPEN's and CLOSE's in the same sequence space one of which has a seqid and the other a stateid), you have a lot of edge conditions to deal with. Devoting 32 bits of stateid to a sequence isn't possible if stateid is kept at 64 bits. What might be posssible is making stqeid 96 bits, 32 of which are defined as a sequence number and the rest designating a (file,lockowner)-pair which doesn't change on each state update. > But, probably, changing seqid to stateid is really too late now > (even if it's needed only for OPEN). There may be some official definition of the proper standards right now, but I'd say that such a significant change shouldn't be made, unless: The protocol just won't work as it stands. There is major class of servers or clients that just can't implement the protocol as it stands. There is a *major* performance problem with the protocol as it stands in important environments. If what we have is implementable, let's implement it. - From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 12:39:29 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA13768 for ; Tue, 30 Jan 2001 12:39:29 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA19286; Tue, 30 Jan 2001 09:38:54 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id JAA09010; Tue, 30 Jan 2001 09:37:12 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UHb0221408 for ; Tue, 30 Jan 2001 09:37:00 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id JAA27304 for ; Tue, 30 Jan 2001 09:37:00 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA09795 for ; Tue, 30 Jan 2001 09:36:59 -0800 (PST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0UHaw212579; Tue, 30 Jan 2001 09:36:58 -0800 (PST) Received: from ussvlexc06.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0UHawE24513; Tue, 30 Jan 2001 09:36:58 -0800 (PST) Received: by ussvlexc06.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 30 Jan 2001 09:37:32 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E58@red.nane.netapp.com> From: "Noveck, Dave" To: "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid Date: Tue, 30 Jan 2001 09:36:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Kendrick Smith wrote: > On Mon, 29 Jan 2001, Sergey Klyushin wrote: > > > > I don't think you have to separately save the last superseded (I > > > think that's what you mean here; expired has some other > > > connotations) stateid. > > > > > > What I do is first use the stateid to map to an entry in my table of > > > state structures. At this point, I may generate STALE_STATEID and > > > BAD_STATEID errors, but not OLD_STATEID. This requires that I know > > > the range of valid values of the stateid subfield which is incremented > > > on each change, but I need that anyway to properly distinguish between > > > BAD_STATEID and OLD_STATEID. > > > > > > Once I have the address of a state structure, I can use this to > > > find the corresponding owner structure which has the current seqid > > > and replay information. So at this point, I can notice a replay > > > and respond. > > > > > In our server implementation, we can't determine the lockowner > corresponding to an old stateid without retaining an entry in our stateid > table. (We found doing this to be impractical using 64-bit stateids.) So > it seems that retaining the last superseded stateid may or may not be an > issue, depending on the server implementation. I do think that this would > be a useful remark for the implementor's guidelines (but probably not > needed in the spec). > > > BTW, how useful ERROR_OLD_STATEID for client? What client suppose to do > > after receiving this error? > > I was wondering this too.. On our client, we couldn't really make use of > the distinction between NFS4ERR_OLD_STATEID and NFS4ERR_BAD_STATEID > (although NFS4ERR_STALE_STATEID is certainly useful, as an entry point to > the server-reboot recovery code). Offhand, does anyone know the rationale > for making seperate errors NFS4ERR_OLD_STATEID, NFS4ERR_BAD_STATEID? See my answer to Sergei. > > > So, you don't store stateid (or even full request) for reply message, > > but you are based only on seqid = stored_seqid? How you could be sure, > > that it's real replay, but not logical error? I want to be sure at > > this point, that I understand replay correctly: for me it's real > > retransmission of exactly the same package. Is it true? BTW, do you > > check lease time before reply? > > We only retransmit the response if the old request and new request are > byte-by-byte identical, including rpc headers. The client's lease is > checked and renewed only if the seqid is correct. Is this what everyone > else is doing? The entire request??? If I do LOCK followed by WRITE of 64K, are you going to save the 64K I write so that on a subsequent replay you can compare the entire request including all that data? Seems like a lot of work. I'm not doing that. I implement the seqid logic at the operation level. If the seqid matches, the saved response information becomes the response information for that operation. Preceding and subsequent operations within the same request are handled normally (although they are subject to the ordinary replay cache so if the entire request hits there is no real execution of any operation in the request; the old reply is just resent). Your mention of the rpc headers prompts me to think that I might reach across levels here and save the transaction id in the per-lockowner replay information. Checking that when the seqid matches would not add significant overhead. > > > > I don't think so. In the case of open, I don't see how you can do > > without > > > a seqid. The fact that it generates a stateid (updated or not) > > doesn't > > > give you any replay protection. > > > > > > I send an open for read-write, deny=read-write. It succeeds and I > > send > > > back a stateid. The reply is lost. A replay of the OPEN is sent and > > > it fails because open for read or write is denied, which is definitely > > > not what you want. You might fall back on various kind of ad hoc > > replay > > > protection mechanisms specific to OPEN but that doesn't give me a very > > good > > > feeling. > > I don't see any place when you need both seqid and stateid in one request. > > OPEN may take seqid and generate stateid, and stateid will be used later > > without seqid > > But, probably, changing seqid to stateid is really too late now > > (even if it's needed only for OPEN). > > It does seem that, if per-lockowner seqids were used only for OPEN, and > per-(lockowner, file) stateids were used everywhere else, replay > protection would still be ensured. But this would have the side effect of > requiring the server to cache replays on a per-(lockowner, file) basis; my > feeling is that this would negatively impact performance. Depends on what information you are saving for the cache replays. From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 16:17:35 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA17723 for ; Tue, 30 Jan 2001 16:17:34 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA21006; Tue, 30 Jan 2001 13:16:22 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA04528; Tue, 30 Jan 2001 12:59:56 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UKw7221595 for ; Tue, 30 Jan 2001 12:58:07 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA12364 for ; Tue, 30 Jan 2001 12:58:06 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA13399 for ; Tue, 30 Jan 2001 13:58:01 -0700 (MST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0UKvq611924; Tue, 30 Jan 2001 15:57:52 -0500 (EST) From: "Sergey Klyushin" To: "Noveck, Dave" , "'Kendrick M. Smith'" , Subject: RE: Stateid and seqid Date: Tue, 30 Jan 2001 15:58:10 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-reply-to: <8C610D86AF6CD4119C9800B0D0499E331A6E58@red.nane.netapp.com> Content-Transfer-Encoding: 7bit Kendrick Smith wrote: > > In our server implementation, we can't determine the lockowner > corresponding to an old stateid without retaining an entry in > our stateid table. (We found doing this to be impractical using 64-bit > stateids.) So it seems that retaining the last superseded stateid may or > may not be an issue, depending on the server implementation. I do think > that this would be a useful remark for the implementor's guidelines (but probably not > needed in the spec). > According to spec, if you can't locate nfs_lockowner by stateid, it means BAD_STATEID or STALE_STATEID, but not OLD_STATEID > > > > So, you don't store stateid (or even full request) for reply message, > > > but you are based only on seqid = stored_seqid? How you could be sure, > > > that it's real replay, but not logical error? I want to be sure at > > > this point, that I understand replay correctly: for me it's real > > > retransmission of exactly the same package. Is it true? > > Kendrick Smith wrote: > > We only retransmit the response if the old request and new request are > > byte-by-byte identical, including rpc headers. > Noveck Dave wrote: >The entire request??? > >If I do LOCK followed by WRITE of 64K, are you going to save the 64K >I write so that on a subsequent replay you can compare the entire >request including all that data? Seems like a lot of work. > ..... According to spec., NFS Server must store and replay last response to any request with seqid only, but not for all requests. Looks like processing of replay is implementation depended. It must be based on seqid, but spec. says nothing more. Probably, I would compare full request also (at least for the first v.4 implementation) to avoid logical bugs. > > > BTW, do you check lease time before reply? > Kendrick Smith wrote: > The client's lease is checked and renewed only if the seqid is correct. Is this > what everyone else is doing? > Noveck Dave wrote: >Right now my lease expiration code isn't there yet. My inclination >is to treat replay as merely retransmitting what already happened, >so I wouldn't check it. I would not check and renew lease on replay also. It would be consistent with RPC layer cashing. Not renewing lease on replay could cause some problems: if NFS client finally receives reply, the next request may get ERR_EXPIRED, but using RENEW could help, and my feeling, that it's better, than keeping file opened (locked,...) forever, plus it will be consistent with RPC layer cashing. Another point to check lease at OPEN, if clientid OK -> nfs_lockowner found -> seqid OK. I would not check lease here also, but I would renew one. OPEN is the main point where lease starts and lease is one for all nfs_lockowners from the same client. If lease has expired before, but NFS Server still has nfs_lockowner in his table, checking lease at this point will never allow to OPEN a file. In case clientid OK -> nfs_lockowner NOT found, I would NOT check lease also, bump seqid, renew lease and ask for OPEN_CONFIRM. One more note: good stateid and replay_seqid should never happened. It's logical error. I think BAD_SEQ_NUM should be returned. As far as I remember, there was agreement not to bump seqid, if any error returned (including ERR_EXPIRED). What about lease? I would NOT renew lease on ANY error also. > > I don't see any place when you need both seqid and stateid in one request. > > OPEN may take seqid and generate stateid, and stateid will be used later > > without seqid But, probably, changing seqid to stateid is really too late now > > (even if it's needed only for OPEN). > > It does seem that, if per-lockowner seqids were used only for OPEN, and > per-(lockowner, file) stateids were used everywhere else, replay > protection would still be ensured. But this would have the > side effect of requiring the server to cache replays on a per-(lockowner, > file) basis; my feeling is that this would negatively impact performance. > No, replies are stored per seqid (which is per lockowner), but it doesn't look like we are going to change spec. now anyway. I would look at this with more details for the first minor version. Sergey From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 18:36:21 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA19241 for ; Tue, 30 Jan 2001 18:36:21 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA14782; Tue, 30 Jan 2001 15:35:14 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA28994; Tue, 30 Jan 2001 15:34:33 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UNYH221765 for ; Tue, 30 Jan 2001 15:34:18 -0800 (PST) Received: from venus.Sun.COM (venus.EBay.Sun.COM [129.150.69.5]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA19668 for ; Tue, 30 Jan 2001 15:34:18 -0800 (PST) Received: from note.orchestra.cse.unsw.EDU.AU (note.orchestra.cse.unsw.EDU.AU [129.94.242.29]) by venus.Sun.COM (8.9.3+Sun/8.9.3) with SMTP id PAA02193 for ; Tue, 30 Jan 2001 15:34:15 -0800 (PST) Received: From notabene.cse.unsw.edu.au ([129.94.211.34] == elephant.orchestra.cse.unsw.EDU.AU) (for ) (for ) By note With Smtp ; Wed, 31 Jan 2001 10:34:00 +1100 Received: from neilb by notabene.cse.unsw.edu.au with local (Exim 3.12 #1 (Debian)) id 14NkHf-0004te-00; Wed, 31 Jan 2001 10:33:59 +1100 From: Neil Brown To: "Kendrick M. Smith" Date: Wed, 31 Jan 2001 10:33:59 +1100 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14967.20327.28551.244965@notabene.cse.unsw.edu.au> Cc: nfsv4-wg@sunroof.eng.sun.com Subject: Re: filehandle volatility, stateid, and "state leakage" In-Reply-To: message from Kendrick M. Smith on Tuesday January 23 References: X-Mailer: VM 6.72 under Emacs 20.7.2 X-face: [Gw_3E*Gng}4rRrKRYotwlE?.2|**#s9D > Looking back at the post-bakeoff mailing list postings, it seems to me > that we never reached consensus on whether stateids should be per > lockowner or per (lockowner,file)-pair. The last mention I see of this is > in Mike Eisler's message dated 11/03, detailing changes that should be > made in the spec if it is decided that stateids should be per lockowner. > Since those changes have not yet been made, is it safe to assume that > stateids will remain per (lockowner,file)-pair? > > My reason for bringing this up is related to a problem that I have run > into in our client implementation. Consider this scenario: > 1. Client A opens a file. > 2. Client A acquires locks on the file. > 3. Client B renames the file. Because the filehandles of this server > are volatile on rename, this changes the file's filehandle. > 4. Client A tries to release its locks. > In step 4, the client must issue a PUTFH for the file, since for a LOCKU > operation, the spec requires the current filehandle to match the file > being unlocked. Because the filehandle changed in step 3, the PUTFH will > fail with error NFS4ERR_BADHANDLE. The server does not have to give an error here. Just because the filehandle-that-would-be-generated has changed doesn't mean filehandles that were generated earlier need no longer be recognised. It is quite valid for a server to recognise multiple different filehandles as referring to the same file. There is a "unique_handles" attribute which allows the client to discover if this might happen. In your implementation, there should be no problem locating the file for an old (pre-rename) filehandle, especially if it is still in the dcache, which I suspect it would be if there were a current lock on the file. So when Client A presents an "old" filehandle, there should be no problem recognising it and doing what A expects. Things might be bit harder if the server was restarted, but they aren't impossible. This is where the FH4_NOEXPIRE_WITH_OPEN flag is particularly relevant. If a server provides neither FH4_PERSISTENT nor FH4_NOEXPIRE_WITH_OPEN, then a client cannot really trust the server enough to do locking reliably. To provide FH4_NOEXPIRE_WITH_OPEN semantics with your implementation you would need to do a bit of extra work: Whenever a file which is open is renamed to a different directory, you would need to log to stable storage enough information to reliably recognise old filehandles. I suspect that the minimum that you could get away with is a record which says: "File X on filesystem Y is now in directory Z". where "X" is probably an inum+generation or whatever is meaningful to the filehandle. Given such a log, then when an old filehandle is presented during the state-reclaiming process, you can find out which directory (or directories) the file is now in, and you and do the necessary permission checks. I suspect that in real life, renaming of files which are open does not happen very often, so this log would not impose an excessive performance hit. > Futhermore, the client will not be > able to recover the new filehandle, since it has no way of knowing the > new filename specified by client B in step 3. Therefore it will never be > able to release its locks on the file (or close it, since the CLOSE > operation has the same filehandle restriction as the LOCKU operation). > This "state leakage" will persist for the lifetime of the client (assuming > that the client periodically emits RENEW operations)! > > One possible solution to this problem is to add the restriction that, > while a file has been opened on the server, the server cannot change its > filehandle. For our Linux server, this would be quite problematic; our > filehandles have to be volatile on rename (in case anyone is curious > why, I've summarized the reasons in a note at the end of this message). > > Another solution, which I think is the "right" way to handle this issue, > is: > 1. Agree that stateids are per (lockowner,file)-pair, rather than per > lockowner. This means that, in a LOCKU or CLOSE operation, the stateid > provided by the client is sufficient to specify the file; the current > filehandle would be redundant. > 2. Remove the restriction for the spec that, in a CLOSE or LOCKU > operation, the current filehandle has to be set to the opened file. The > stateid specified with the operation now identifies the file. > > What does everyone think of this? I'm actually in favour of not ever requiring a filehandle if you have a stateid, but that has be discussed before - search for "statehandle" in the list archives and you should find the right thread. NeilBrown From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 18:46:22 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA19343 for ; Tue, 30 Jan 2001 18:46:21 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA18810; Tue, 30 Jan 2001 15:45:59 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA00858; Tue, 30 Jan 2001 15:44:26 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0UNiG221777 for ; Tue, 30 Jan 2001 15:44:16 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id PAA09805 for ; Tue, 30 Jan 2001 15:44:16 -0800 (PST) Received: from note.orchestra.cse.unsw.EDU.AU (note.orchestra.cse.unsw.EDU.AU [129.94.242.29]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with SMTP id QAA08552 for ; Tue, 30 Jan 2001 16:44:14 -0700 (MST) Received: From notabene.cse.unsw.edu.au ([129.94.211.34] == elephant.orchestra.cse.unsw.EDU.AU) (for ) (for ) (for ) By note With Smtp ; Wed, 31 Jan 2001 10:44:05 +1100 Received: from neilb by notabene.cse.unsw.edu.au with local (Exim 3.12 #1 (Debian)) id 14NkRQ-0004u1-00; Wed, 31 Jan 2001 10:44:04 +1100 From: Neil Brown To: "Sergey Klyushin" Date: Wed, 31 Jan 2001 10:44:04 +1100 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14967.20932.254178.16459@notabene.cse.unsw.edu.au> Cc: , Subject: Re: Stateid and seqid In-Reply-To: message from Sergey Klyushin on Friday January 26 References: X-Mailer: VM 6.72 under Emacs 20.7.2 X-face: [Gw_3E*Gng}4rRrKRYotwlE?.2|**#s9D Q1. Can anybody explain why do we need both stateid and seqid? > (consider that stateid is shorthand to lockowner+file, and stateid includes > some > incrementing sequence inside itself) > Note, that there is no operation, that takes seqid only without stateid > (except OPEN, which is the > only operation to create stateid). I spent a while trying to understand these issues properly a while back and the simplest answer that I have to your question is: Every request that might change state contains a stateid and a seqid. If the operation is successful and state changes, then the stateid changes. If the operation fails and the state doesn't change, then the seqid changes. This doesn't describe the full picture, as the seqid changes if the operation succeeds, but I think it does draw out the central purpose of the seqid - that being to record failed attempts to change state. I actually think, and I believe that the recuring discussions support the idea, that the current model for state management is much too complex. It is a great prototype as it deal with (I believe) all the important issues and presents real solutions to them. However it seems to be much too complex to be understood by implementors (and possibly even by the designers, I'm not sure). And like any good prototype, I think that it needs to be redesigned from scratch. I think that the result could easily be very similar in a lot of ways to the current scheme, but could still be a lot clearer. I have a very clear idea in my head as to how it could be done. I tried to present in on the list a while ago (over a year a think) but I didn't get much (or any) response. I'm happy to try again if anyone is interested. NeilBrown From nfs4-wg-request@sunroof.eng.sun.com Tue Jan 30 20:15:00 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA20017 for ; Tue, 30 Jan 2001 20:15:00 -0500 (EST) Received: from engmail1.Eng.Sun.COM ([129.146.1.13]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id RAA14956; Tue, 30 Jan 2001 17:14:24 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id RAA22046; Tue, 30 Jan 2001 17:14:01 -0800 (PST) Received: from jurassic.eng.sun.com (jurassic [129.146.89.31] (may be forged)) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0V1Dk221818 for ; Tue, 30 Jan 2001 17:13:47 -0800 (PST) Received: from eng.sun.com (terra.Eng.Sun.COM [129.146.86.99]) by jurassic.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0V1Dk9755986 for ; Tue, 30 Jan 2001 17:13:47 -0800 (PST) Sender: Brent.Callaghan@eng.sun.com Message-ID: <3A7766D0.7FAA41F0@eng.sun.com> Date: Tue, 30 Jan 2001 17:13:52 -0800 From: Brent Organization: NFS Whackers X-Mailer: Mozilla 4.75 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 CC: nfsv4-wg@sunroof.eng.sun.com Subject: Re: Stateid and seqid References: <14967.20932.254178.16459@notabene.cse.unsw.edu.au> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Neil Brown wrote: > I actually think, and I believe that the recuring discussions support the > idea, that the current model for state management is much too complex. > It is a great prototype as it deal with (I believe) all the important > issues and presents real solutions to them. However it seems to be > much too complex to be understood by implementors (and possibly even > by the designers, I'm not sure). Count me as one who is not thrilled with the level of complexity of the state management. It's certainly raised a lot of questions as we've gotten our fingers dirty attempting to implement it. I worry too that it will cause interoperability problems, not for normal operation, but during those edge conditions where clients are retransmitting, servers crashing and networks dropping packets. On the other hand, now that we're building this protocol into running code, we're reluctant to change the protocol without a good reason. Any protocol change means code redesign, rewrite, retest and another spin on the RFC. > And like any good prototype, I think that it needs to be redesigned > from scratch. I think that the result could easily be very similar in > a lot of ways to the current scheme, but could still be a lot clearer. > > I have a very clear idea in my head as to how it could be done. I > tried to present in on the list a while ago (over a year a think) but > I didn't get much (or any) response. I'm happy to try again if anyone > is interested. Go ahead and run it by us again. With some implementation experience behind us, we may view your proposal differently. Though I recommend you preface your proposal with the reasons why the current state model is flawed. Brent From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 13:30:07 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA22722 for ; Wed, 31 Jan 2001 13:30:07 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id KAA13673; Wed, 31 Jan 2001 10:28:55 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id KAA25248; Wed, 31 Jan 2001 10:28:25 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VISA222434 for ; Wed, 31 Jan 2001 10:28:10 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id KAA25185 for ; Wed, 31 Jan 2001 10:28:09 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA08458 for ; Wed, 31 Jan 2001 10:28:09 -0800 (PST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0VIS4215470; Wed, 31 Jan 2001 10:28:04 -0800 (PST) Received: from ussvlexc01.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0VIS3H27730; Wed, 31 Jan 2001 10:28:03 -0800 (PST) Received: by ussvlexc01.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 31 Jan 2001 10:28:00 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E60@red.nane.netapp.com> From: "Noveck, Dave" To: "'Sergey Klyushin'" , "Noveck, Dave" , "'Kendrick M. Smith'" , nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid Date: Wed, 31 Jan 2001 10:28:00 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sergei Klyushin wrote: > Kendrick Smith wrote: > > > > In our server implementation, we can't determine the lockowner > > corresponding to an old stateid without retaining an entry in > > our stateid table. (We found doing this to be impractical using 64-bit > > stateids.) So it seems that retaining the last superseded stateid may or > > may not be an issue, depending on the server implementation. I do think > > that this would be a useful remark for the implementor's guidelines (but > > probably not > > needed in the spec). > > > > According to spec, if you can't locate nfs_lockowner by stateid, > it means BAD_STATEID or STALE_STATEID, but not OLD_STATEID I don't see the problem recognizing OLD stateid's as OLD (and not BAD). The spec says you are supposed to do it and I don't see why it would be difficult. > > > > > > > So, you don't store stateid (or even full request) for reply message, > > > > but you are based only on seqid = stored_seqid? How you could be sure, > > > > that it's real replay, but not logical error? I want to be sure at > > > > this point, that I understand replay correctly: for me it's real > > > > retransmission of exactly the same package. Is it true? > > > > Kendrick Smith wrote: > > > We only retransmit the response if the old request and new request are > > > byte-by-byte identical, including rpc headers. > > > Noveck Dave wrote: > >The entire request??? > > > >If I do LOCK followed by WRITE of 64K, are you going to save the 64K > >I write so that on a subsequent replay you can compare the entire > >request including all that data? Seems like a lot of work. > > ..... > > According to spec., NFS Server must store and replay last response to any > request with seqid only, but not for all requests. Looks like processing > of replay is implementation depended. It must be based on seqid, but spec. > says nothing more. Probably, I would compare full request also (at least > for the first v.4 implementation) to avoid logical bugs. There is a real problem with the way the spec describes this, which has led to two divergent interpretations. I think we need to resolve this. For example the spec says (section 8.1.5): If a request with a previous sequence number (r < L) is received, it is rejected with the return of error NFS4ERR_BAD_SEQID. Given a properly-functioning client, the response to (r) must have been received before the last request (L) was sent. If a duplicate of last request (r == L) is received, the stored response is returned. The problem is such phrases as "request with a previous sequence number". Requests don't really have sequence numbers, LOCK/UNLOCK, etc. operations do. I suspect that this text was written before COMPOUND became the all-encompassing way to do things and was never rewritten to reflect the fact that operations and requests are no longer one-to-one. So what do you implement? I've implemented my seqid stuff to be operation oriented; I save the response for the operation and replay that. The other interpretation is to take "request" literally and try to save the entire response. For such requests as READ+UNLOCK, this can take a lot of space. It would be quite easy to drive a server to its knees, saving the contents of many long READ's for many different lockowner's. This could happen as a denial-of-service attack but might even happen with no malicious clients. I think there are other problems with the whole-request approach. Suppose you issue a request that has an operation with seqid X and the entire request successfully executed on the server. Now suppose the reply is lost and the client resends the request. Now the spec says "If a duplicate of the last request (r == L) is received, the stored response is returned." The problem with this is the "is received". Is the server supposed to check this when the request is recieved (i.e. before it executes any of the operations)? I would expect that it would only notice r=L when it gets to the operation that has the seqid. So if it starts executing the initial sequence of operations and encounters an error (transient IO error, expired fh,, whatever), what then? The server will think that the last seqid accted on was X while the client will think X-1. The client will do something to recover but when operation that includes seqid X is issued, it will be in the context of a new rqeuest and it won't match. If you strictly embrace whole-request interpretation you can say this is a logical error, but how is the client to re-issue the operation (e.g. UNLOCK) that may or may not have been executed on the server if the original request is completed (in this case with an error). On the operation-only interpretation, the client can re-issue the UNLOCK as part of a new request with the same seqid and it will be honored. Also note that the paragraph (partially) quoted above lists the cases in terms of of the relationships between r and L and then just assumes that that completely characterizes the relationships between the two requests (or operations). So when it says "If a duplicate of last request (r == L) is received", it ignores the case of r == L when it is duplicate of the last operation but it not part of a duplicate request or when it is r == L but is not a duplicate of the previous operation. Are these errors? If so, what errors? > > > > > > BTW, do you check lease time before reply? > > > Kendrick Smith wrote: > > The client's lease is checked and renewed only if the seqid is correct. > Is this > > what everyone else is doing? > > > Noveck Dave wrote: > >Right now my lease expiration code isn't there yet. My inclination > >is to treat replay as merely retransmitting what already happened, > >so I wouldn't check it. > > I would not check and renew lease on replay also. > It would be consistent with RPC layer cashing. > Not renewing lease on replay could cause some problems: if NFS client > finally > receives reply, the next request may get ERR_EXPIRED, but using RENEW could > help, > and my feeling, that it's better, than keeping file opened (locked,...) > forever, plus > it will be consistent with RPC layer cashing. I agree. When a client first issues a request at time X and receives a response at time Y, all it can assume is that the server renewed the lease at some time between X and Y, and he should schedule his RENEWAL logic accordingly. (The case of relativistic server-client relative motion is left as an exercise for the (not-yet-but-soon-to-be) truly deranged :-). > > Another point to check lease at OPEN, if clientid OK -> nfs_lockowner > found -> seqid OK. > I would not check lease here also, but I would renew one. > OPEN is the main point where lease starts and lease is one for all > nfs_lockowners > from the same client. If lease has expired before, but NFS Server still has > nfs_lockowner in his table, checking lease at this point will never allow to > OPEN a file. Right. You check a lease if you are using a lock that it covers. For OPEN, you are creating a new lease (and renewing any that still exist). Section 8.4 talks about this. It doesn't specifically exclude the case of replay, but it does speak of "events" that trigger renewal of which OPEN is one, but it is unclear whether the "event" is execution of an OPEN or simply responding to one. > > In case clientid OK -> nfs_lockowner NOT found, I would NOT check lease > also, bump seqid, > renew lease and ask for OPEN_CONFIRM. Agree. > > One more note: good stateid and replay_seqid should never happened. It's > logical error. > I think BAD_SEQ_NUM should be returned. I think the spec needs to be more explicit about error situations like that. > > As far as I remember, there was agreement not to bump seqid, if any error > returned > (including ERR_EXPIRED). Yes, that was what we all agreed at the bake-off. > What about lease? I would NOT renew lease on ANY error also. That seems troublesome to me. The purpose of the renewal mechanism is so that the server can know the client is still up and responsive. There may be situations in which many requests will get errors because of server conditions. It seems wrong to me to say that in such a situation, the server must give up the ability to keep track of the leases of clients. Of course that means that we have to define what errors will renew and those that won't. Sigh. From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 14:24:43 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA24116 for ; Wed, 31 Jan 2001 14:24:42 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA10579; Wed, 31 Jan 2001 11:23:51 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id LAA20510; Wed, 31 Jan 2001 11:19:31 -0800 (PST) Received: from engmail1.Eng.Sun.COM (engmail1 [129.146.1.13]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VJJI222450 for ; Wed, 31 Jan 2001 11:19:19 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail1.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id LAA02841 for ; Wed, 31 Jan 2001 11:19:18 -0800 (PST) Received: from mx01-a.netapp.com ([198.95.226.53]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id MAA27181 for ; Wed, 31 Jan 2001 12:19:17 -0700 (MST) Received: from frejya.corp.netapp.com (frejya.corp.netapp.com [10.10.20.91]) by mx01-a.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0VJJG219569; Wed, 31 Jan 2001 11:19:16 -0800 (PST) Received: from ussvlexc01.corp.netapp.com (localhost [127.0.0.1]) by frejya.corp.netapp.com (8.11.1/8.11.1/NTAP-1.2) with ESMTP id f0VJJFX12590; Wed, 31 Jan 2001 11:19:15 -0800 (PST) Received: by ussvlexc01.corp.netapp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 31 Jan 2001 11:19:12 -0800 Message-ID: <8C610D86AF6CD4119C9800B0D0499E331A6E62@red.nane.netapp.com> From: "Noveck, Dave" To: "'Neil Brown'" , Sergey Klyushin Cc: nfsv4-wg@sunroof.eng.sun.com, "Noveck, Dave" Subject: RE: Stateid and seqid Date: Wed, 31 Jan 2001 11:19:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Neil Brown wrote: > On Friday January 26, sergey@hcl.com wrote: > > Q1. Can anybody explain why do we need both stateid and seqid? > > (consider that stateid is shorthand to lockowner+file, and stateid includes > > some > > incrementing sequence inside itself) > > Note, that there is no operation, that takes seqid only without stateid > > (except OPEN, which is the > > only operation to create stateid). > I spent a while trying to understand these issues properly a while > back and the simplest answer that I have to your question is: > > Every request that might change state contains a stateid and a > seqid. Except for OPEN. > If the operation is successful and state changes, then the stateid > changes. According to David Robinson, the server may in some cases return the same stateid (ie. not change it). > If the operation fails and the state doesn't change, then the > seqid changes. The current approach is that seqid does not change if the operation fails. That's what people are implementing to, but there has been disagreement about whether the spec says that (e.g. Mike Eisler believes it does) > This doesn't describe the full picture, as the seqid changes if the > operation succeeds, but I think it does draw out the central purpose of > the seqid - that being to record failed attempts to change state. The consensus was that there was no need to record that. If an operation fails, nothing *really* happened so there is no need for a corrective to just letting people retry and get at-least-one-attempt- but-exactly-one-acted-upon-operation semantics. > I actually think, and I believe that the recuring discussions support the > idea, that the current model for state management is much too complex. > It is a great prototype as it deal with (I believe) all the important > issues and presents real solutions to them. However it seems to be > much too complex to be understood by implementors (and possibly even > by the designers, I'm not sure). I certainly think that the current design could be cleaned up some, but I think a fair amount of the complexity is inherent in what we are trying to do. We want to support stateful operations but have relatively simple recovery from server reboots while working over the unreliable transports, all the while satisfying additional goals for operation in high-latency environments (i.e the complexity introduced by COMPOUND). My feeling about the recurring discussion is that they don't show problems with the current protocol but reflect major problems in the way it has been described. The spec treats seqid, stateid, leases, more or less separately, and without a lot of implementation detal (out of a desire I think not to constrain the implementation). When you get to coding the implementation, you find that lots and lots of questions aren't answered by the spec, because then you need to decide what order the various checks, each of which hqas been described in a different section of the spec, have to be done. > And like any good prototype, I think that it needs to be redesigned > from scratch. I think that the result could easily be very similar in > a lot of ways to the current scheme, but could still be a lot clearer. > I have a very clear idea in my head as to how it could be done. I > tried to present in on the list a while ago (over a year a think) but > I didn't get much (or any) response. I'm happy to try again if anyone > is interested. I'm certainly curious to see if we could have something simpler (but not too simple). Even if people don't think it enough better to change v4 at this point (and that is going to be a high hurdle), it would be valuable as input to the next minor version. From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 14:43:05 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA24449 for ; Wed, 31 Jan 2001 14:43:04 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA21939; Wed, 31 Jan 2001 11:40:19 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id LAA07281; Wed, 31 Jan 2001 11:37:08 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VJau222455 for ; Wed, 31 Jan 2001 11:36:56 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id LAA07193 for ; Wed, 31 Jan 2001 11:36:56 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA09991 for ; Wed, 31 Jan 2001 11:36:55 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0VJak602799; Wed, 31 Jan 2001 14:36:46 -0500 (EST) From: "Sergey Klyushin" To: "Noveck, Dave" , Subject: RE: Stateid and seqid Date: Wed, 31 Jan 2001 14:37:03 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <8C610D86AF6CD4119C9800B0D0499E331A6E60@red.nane.netapp.com> Content-Transfer-Encoding: 7bit > > > > According to spec., NFS Server must store and replay last response to any > > request with seqid only, but not for all requests. Looks like processing > > of replay is implementation depended. It must be based on seqid, but spec. > > says nothing more. Probably, I would compare full request also (at least > > for the first v.4 implementation) to avoid logical bugs. > > There is a real problem with the way the spec describes this, which > has led to two divergent interpretations. I think we need to resolve > this. For example the spec says (section 8.1.5): > > If a request with a previous sequence number (r < L) is received, it > is rejected with the return of error NFS4ERR_BAD_SEQID. Given a > properly-functioning client, the response to (r) must have been > received before the last request (L) was sent. If a duplicate of > last request (r == L) is received, the stored response is returned. > > The problem is such phrases as "request with a previous sequence number". > Requests don't really have sequence numbers, LOCK/UNLOCK, etc. operations > do. I suspect that this text was written before COMPOUND became the > all-encompassing way to do things and was never rewritten to reflect > the fact that operations and requests are no longer one-to-one. > > So what do you implement? I've implemented my seqid stuff to be > operation oriented; I save the response for the operation and > replay that. Sorry for misunderstanding. I've very ease used "request". My implementation is operation oriented also. As I understand idea of seqid is to guarantee sequence of OPEN,LOCK,UNLOCK,CLOSE, in another words OPERATIONS that take seqid, and reply must be stored for such OPERATIONS, but not for the whole request > > > What about lease? I would NOT renew lease on ANY error also. > > That seems troublesome to me. The purpose of the renewal mechanism > is so that the server can know the client is still up and responsive. > There may be situations in which many requests will get errors > because of server conditions. It seems wrong to me to say that > in such a situation, the server must give up the ability to keep > track of the leases of clients. Of course that means that we > have to define what errors will renew and those that won't. Sigh. > Agree. We need to look more close on what error renew lease. Right now I am trying to review Andy's diagrams, and will post my changes shortly. (Adny, please send SETCLIENTID and CONFIRM_CLIENTID diagrams) From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 15:12:27 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA24992 for ; Wed, 31 Jan 2001 15:12:27 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id MAA06139; Wed, 31 Jan 2001 12:10:51 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA02726; Wed, 31 Jan 2001 12:10:35 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VKAN222465 for ; Wed, 31 Jan 2001 12:10:23 -0800 (PST) Received: from lukla.Sun.COM (lukla.Central.Sun.COM [129.147.5.31]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA14780 for ; Wed, 31 Jan 2001 12:10:22 -0800 (PST) Received: from eagle.webpros.com (eagle.professionals.com [206.127.192.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA04118 for ; Wed, 31 Jan 2001 13:10:21 -0700 (MST) Received: from frostback ([206.27.132.50]) by eagle.webpros.com (8.8.7/8.6.10) with SMTP id MAA16492; Wed, 31 Jan 2001 12:09:47 -0800 (PST) Message-Id: <200101312009.MAA16492@eagle.webpros.com> Date: Wed, 31 Jan 2001 12:07:29 -0800 (PST) From: Mike Eisler Reply-To: Mike Eisler Subject: Re: Stateid and seqid To: "William A.(Andy) Adamson" Cc: nfsv4-wg@sunroof.eng.sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: UVMkdfxPshGkrXuy/2iPeQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc > here is (finnally!) what i have for the flow diagram of stateid, seq > num, and > lease processing that was agreed upon at the last bakeoff, and discussed > in > the flurry of post bake-off email. i think it includes the processing > that > dave described concerning stateid and sequence number. > > i would appreciate it if the list would look it over - it would be great > to > agree on this issue before connectathon.. check stateid / | \ \ / | \ \ / \ \ \ / \ \ \_______ / | \ \ stale good old bad / | \ | | | | server can't bump seq num | | | client undo seq num increment | | | don't renew lease | | | NFSERR_BAD_STATEID seq knowlege is | \ reset on client | \ and server | check sequence number NFS4ERR_STALE_STATEID | / \ | bad or good replay (should hit this) | | send from replay cache | | | good: server increment seq num | good: client leaves seq num alone ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ mre>If the sequence number is good, then why does client leave its seq num alone? Perhaps you are assuming the client increments seq# when the request is sent, rather than when the response is received. If so, you might make this clearer, not that I agree with that implementation practice. | NFSERR_OLD_STATEID ^^^^^^^^^^^^^^^^^^^^ mre>What happens to the sequence number if NFSERR_OLD_STATEID is returned in this branch? | check sequence number / | \ / | \ / | \ bad replay good / | \ don't bump seqnum | \ don't renew lease? | \ NFS4ERR_BAD_SEQ_NUM | \ | \ don't bump seq num \ return replay \ | | check lease | | | bad good | / \ check lease don't renew lease / | renew / | lease bad good mre> Should a replay a renew a lease? This leaves the server open to attacks. Keep in midn that the lease is on all the open and locked objects sharing the client id. it's odd that this replay branch renews the lease but the old stateid --> replay seq. num. branch does not. / | / | NFS4ERR_EXPIRED | seq num bumped | ^^^^^^^^^^^^^^ mre>Why increment sequence number on an error? -mre From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 15:27:17 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA25182 for ; Wed, 31 Jan 2001 15:27:16 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id MAA20300; Wed, 31 Jan 2001 12:26:08 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA17209; Wed, 31 Jan 2001 12:25:40 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VKPT222470 for ; Wed, 31 Jan 2001 12:25:29 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id MAA17149 for ; Wed, 31 Jan 2001 12:25:28 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id MAA19730 for ; Wed, 31 Jan 2001 12:25:28 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.13.46]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0VKP1606700; Wed, 31 Jan 2001 15:25:02 -0500 (EST) From: "Sergey Klyushin" To: "Noveck, Dave" , "'Neil Brown'" Cc: Subject: RE: Stateid and seqid Date: Wed, 31 Jan 2001 15:25:19 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <8C610D86AF6CD4119C9800B0D0499E331A6E62@red.nane.netapp.com> Content-Transfer-Encoding: 7bit > > > If the operation is successful and state changes, then the stateid > > changes. > > According to David Robinson, the server may in some cases return the > same stateid (ie. not change it). > Probably, it's possible, if stateid based on lockowner, but not lockowner+file pair. If locking(open, close,..) state of the file was changed, new stateid should be returned. > > If the operation fails and the state doesn't change, then the > > seqid changes. > > The current approach is that seqid does not change if the operation > fails. That's what people are implementing to, but there has been > disagreement about whether the spec says that (e.g. Mike > Eisler believes it does) > > > This doesn't describe the full picture, as the seqid changes if the > > operation succeeds, but I think it does draw out the central purpose of > > the seqid - that being to record failed attempts to change state. > > The consensus was that there was no need to record that. If an > operation fails, nothing *really* happened so there is no need for > a corrective to just letting people retry and get > at-least-one-attempt- > but-exactly-one-acted-upon-operation semantics. > Looking more close to this problem, I would increment seqid, if operation itself failed, (e.g. LOCK), but would not increment seqid, if some precondition checking failed (e.g. checking stateid or seqid). Here are some more details. 1. seqid not changed if operation failed. Server stores seqid X, client sends LOCK with seqid X+1. stateid OK, seqid OK Server tries to LOCK file, but operation fails (e.g. some other lockowner locked file) Server doesn't change seqid X and doesn't store error_reply (it may store error_reply, but it would be useless) Client decrement seqid to X to be ready for next request Client resend (not network replay) LOCK request with seqid X+1. Everything starts from the very beginning. Server doesn't know if it's replay or new request. 2. seqid changed if operation failed. Server stores seqid X, client sends LOCK with seqid X+1. stateid OK, seqid OK Server tries to LOCK file, but operation fails (e.g. some other lockowner locked file) Server does change seqid to X+1 and does store error_reply Client doesn't decrement seqid. Client resend (not network replay) LOCK request with seqid X+2. Now Server does know if it's replay or new request and because error reply stored Server may resend it from the cache. I would prefer second scenario, but once again I would not change seqid if some precondition checking failed. > > I actually think, and I believe that the recuring discussions support the > > idea, that the current model for state management is much too complex. > > It is a great prototype as it deal with (I believe) all the important > > issues and presents real solutions to them. However it seems to be > > much too complex to be understood by implementors (and possibly even > > by the designers, I'm not sure). > > I certainly think that the current design could be cleaned up some, > but I think a fair amount of the complexity is inherent in what we > are trying to do. We want to support stateful operations but have > relatively simple recovery from server reboots while working over > the unreliable transports, all the while satisfying additional goals > for operation in high-latency environments (i.e the complexity > introduced by COMPOUND). > > My feeling about the recurring discussion is that they don't show > problems with the current protocol but reflect major problems in the > way it has been described. The spec treats seqid, stateid, leases, > more or less separately, and without a lot of implementation detal > (out of a desire I think not to constrain the implementation). When > you get to coding the implementation, you find that lots and lots > of questions aren't answered by the spec, because then you need to > decide what order the various checks, each of which hqas been > described in a different section of the spec, have to be done. > > > And like any good prototype, I think that it needs to be redesigned > > from scratch. I think that the result could easily be very > similar in > > a lot of ways to the current scheme, but could still be a > lot clearer. > > > I have a very clear idea in my head as to how it could be done. I > > tried to present in on the list a while ago (over a year a think) but > > I didn't get much (or any) response. I'm happy to try again if anyone > > is interested. > > I'm certainly curious to see if we could have something simpler (but > not too simple). Even if people don't think it enough better to > change v4 at this point (and that is going to be a high hurdle), > it would be valuable as input to the next minor version. > My personal opinion is to make first version of protocol much clear, even if it delays RFC approval. It will make our future much easy to clean it up right now then to discover bugs in different implementations just because misunderstanding of the protocol. From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 17:42:11 2001 Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA27138 for ; Wed, 31 Jan 2001 17:42:10 -0500 (EST) Received: from engmail2.Eng.Sun.COM ([129.146.1.25]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id OAA08404; Wed, 31 Jan 2001 14:41:14 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id OAA29344; Wed, 31 Jan 2001 14:25:51 -0800 (PST) Received: from engmail2.Eng.Sun.COM (engmail2 [129.146.1.25]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f0VMPa222539 for ; Wed, 31 Jan 2001 14:25:36 -0800 (PST) Received: from saturn.sun.com (saturn.EBay.Sun.COM [129.150.69.2]) by engmail2.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id OAA29301 for ; Wed, 31 Jan 2001 14:25:37 -0800 (PST) Received: from mhoutside.hcl.com (mx.hcl.com [205.211.178.70]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id OAA11482 for ; Wed, 31 Jan 2001 14:25:36 -0800 (PST) Received: from sergeylt (sergeylt.hcl.com [10.1.14.29]) by mhoutside.hcl.com (8.11.2/8.11.2) with SMTP id f0VMPR616243; Wed, 31 Jan 2001 17:25:27 -0500 (EST) From: "Sergey Klyushin" To: "Noveck, Dave" Cc: Subject: RE: Stateid and seqid Date: Wed, 31 Jan 2001 17:25:46 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <8C610D86AF6CD4119C9800B0D0499E331A6E62@red.nane.netapp.com> Content-Transfer-Encoding: 7bit Paragraph 14.2.18. Operation 20: OPEN_CONFIRM - Confirm Open says: The server must hold unconfirmed OPEN state until one of three events occur. First, the client sends an OPEN_CONFIRM request with the appropriate sequence id and confirmation verifier within the lease period. In this case, the OPEN state on the server goes to confirmed, and the nfs_lockowner on the server is fully established. Second, the client sends another OPEN request with a sequence id that is incorrect for the nfs_lockowner (out of sequence). In ^^^^^^^ ^^^^^^^^^^^^^^^ this case, the server assumes the second OPEN request is valid and the first one is a replay. The server cancels the OPEN state of the first OPEN request, establishes an unconfirmed OPEN state for the second OPEN request, and responds to the second OPEN request with an indication that an OPEN_CONFIRM is needed. The process then repeats itself. What Server should do if it receives second OPEN request with correct (X+1) seqid? Here are two possible answers: 1. Consider that sequence is confirmed, even without OPEN_CONFIRM. Server sets confirmed state. Personally, I don't like it. Server could receive OPEN_CONFIRM later with seqid from first OPEN. 2. Consider this case like bad seqid also. My feeling, that it would work better. Sergey From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 22:47:45 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id WAA04385 for ; Wed, 31 Jan 2001 22:47:44 -0500 (EST) Received: from engmail3.Eng.Sun.COM ([129.144.170.5]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id TAA26701; Wed, 31 Jan 2001 19:47:05 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id TAA14137; Wed, 31 Jan 2001 19:46:34 -0800 (PST) Received: from engmail4.Eng.Sun.COM (engmail4 [129.144.134.6]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f113kJ222740; Wed, 31 Jan 2001 19:46:19 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id TAA03777; Wed, 31 Jan 2001 19:46:19 -0800 (PST) From: jewelers@building.com Received: from mail.sunlin.ac.kr ([211.223.206.3]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id TAA26394; Wed, 31 Jan 2001 19:46:16 -0800 (PST) Received: from 211.223.206.3 (router [211.223.206.126]) by mail.sunlin.ac.kr (8.8.8+Sun/8.8.8) with SMTP id JAA06048; Thu, 1 Feb 2001 09:54:22 +0900 (KST) Message-Id: <200102010054.JAA06048@mail.sunlin.ac.kr> Date: Wed, 31 Jan 01 21:08:01 EST To: g411hjd8h@idnkjkwo0.com Subject: Worth a Shot The Program ------------------------------------------------------------------------------------------------------- AS SEEN ON NATIONAL TV --- INCREDIBLE $0 to $50,000 in 90 days!!! ------------------------------------------------------------------------------------------------------- You can earn $50,000 or more in next the 90 days sending e-mail. Seem impossible? Read on for details. This is the letter you've been reading about in the news lately. Due to the popularity of this letter on the Internet, a major nightly news program recently devoted an entire show to the investigation of the program described below to see if it really can make people money. The show also investigated whether or not the program was legal. Their findings proved once and for all that there are absolutely no laws prohibiting the participation in the program. This has helped to show people that this is a simple, harmless and fun way to make some extra money at home. The results of this show have been truly remarkable. So many people are participating that those involved are doing much better than ever before. Since everyone makes more as more people try it out, its been very exciting to be a part of lately. You will understand once you experience it. HERE IT IS BELOW: *** Print This Now For Future Reference *** The following income opportunity is one you may be interested in taking a look at. It can be started with VERY LITTLE investment and the income return is TREMENDOUS!!! $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ If you would like to make at least $50,000 in less than 90 days ! Please read the enclosed program...THEN READ IT AGAIN!!! $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ THIS IS A LEGITIMATE, LEGAL, MONEY MAKING OPPORTUNITY.It does not require you to come into contact with people, do any hard work, and best of all, you never have to leave the house except to get the mail. If you believe that someday you'll get that big break that you 'vebeen waiting for, THIS IS IT! Simply follow the instructions, andyour dreams will come true. This multi-level e-mail order marketingprogram works perfectly...100% EVERY TIME. E-mail is the sales tool of the future. Take advantage of this non-commercialized method of advertising NOW!!! The longer you wait, the more people will be doing business using e-mail. Get your piece of this action!!! MULTI-LEVEL MARKETING (MLM) has finally gained respectability. It is being taught in the Harvard Business School, and both Stanford Research and the Wall Street Journal have stated that between 50% and 65% of all goods and services will be sold through multi-level methods by the mid to late 1990's. This is a Multi-Billion Dollar industry and of the 500,000 millionaires in the U.S., 20% (100,000) made their fortune in the last several years in MLM. Moreover, statistics show 45 people become millionaires everyday through Multi-Level Marketing. You may have heard this story before, but over the summer Donald Trump made an appearance on the David Letterman show. Dave asked him what he would do if he lost everything and had to start over from scratch. Without hesitating, Trump said he would find a good network marketing company and get to work. The audience started to hoot and boo him. He looked out at the audience and dead-panned his response: "That's why I'm sitting up here and you are all sitting out there!" The enclosed information is something I almost let slip through my fingers. Fortunately, sometime later I re-read everything and gave somethought and study to it. My name is Johnathon Rourke. Two years ago, the corporation I worked at for the past twelve years down-sized and my position was eliminated. After unproductive job interviews, I decided to open my own business. Over the past year, I incurred many unforeseen financial problems. I owed my family, friends and creditors over $35,000. The economy was taking a toll on my business and I just couldn't seem to make ends meet. I had to refinance and borrow against my home to support my family and struggling business. AT THAT MOMENT something significant happened in my life and I am writing to share the experience in hopes that this will change your life FOREVER FINANCIALLY!!! In mid December, I received this program via e-mail. Six month's prior to receiving this program I had been sending away for information on various business opportunities. All of the programs I received, in my opinion, were not cost effective. They were either too difficult for me to comprehend or the initial investment was too much for me to risk to see if they would work or not. One claimed that I would make a million dollars in one year...it didn't tell me I'd have to write a book to make it! But like I was saying, in December of 1997 I received this program. I didn't send for it, or ask for it, they just got my name off a mailing list.THANK GOODNESS FOR THAT!!! After reading it several times, to make sure I was reading it correctly, I couldn't believe my eyes. Here was a MONEY MAKING PHENOMENON. I could invest as much as I wanted to start, without putting me further into debt. After I got a pencil and paper and figured it out, I would at least get my money back. But like most of you I was still a little skeptical and a little worried about the legal aspects of it all. So I checked it out with the U.S. Post Office (1-800-725-2161 24-hrs) and they confirmed that it is indeed legal! After determining the program was LEGAL and NOT A CHAIN LETTER, I decided "WHY NOT." Initially I sent out 10,000 e-mails. It cost me about $15 for my time on-line. The great thing about e-mail is that I don't need any money for printing to send out the program, and because all of my orders are fulfilled via e-mail, my only expense is my time. I am telling you like it is I hope it doesn't turn you off, but I promised myself that I would not "rip-off" anyone, no matter how much money it made me. In less than one week, I was starting to receive orders for REPORT #1 By January 13, I had received 26 orders for REPORT #1. Your goal is to "RECEIVE at least 20 ORDERS FOR REPORT #1 WITHIN 2 WEEKS. IF YOU DON'T, SEND OUT MORE PROGRAMS UNTIL YOU DO!" My first step in making $50,000 in 90 days was done. By January 30, I had received 196 orders for REPORT #2. Your goal is to "RECEIVE AT LEAST 100+ ORDERS FOR REPORT #2 WITHIN 2 WEEKS. IF NOT, SEND OUT MORE PROGRAMS UNTIL YOU DO. ONCE YOU HAVE 100 ORDERS, THE REST IS EASY, RELAX, YOU WILL MAKE YOUR $50,000 GOAL." Well, I had 196 orders for REPORT #2, 96 more than I needed. So I sat back and relaxed. By March 1, of my e-mailing of 10,000, I received $58,000 with more coming in every day. I paid off ALL my debts and bought a much needed new car. Please take time to read the attached program, IT WILL CHANGE YOUR LIFE FOREVER!! ! Remember, it won't work if you don't try it. This program does work , but you must follow it EXACTLY! Especially the rules of not trying to place your name in a different place. It won't work and you'll lose out on a lot of money! In order for this program to work, you must meet your goal of 20+ orders for REPORT #1, and 100+ orders for REPORT #2 and you will make $50,000 or more in 90 days. I AM LIVING PROOF THAT IT WORKS!!! If you choose not to participate in this program, I am sorry. It really is a great opportunity with little cost or risk to you. If you choose to participate, follow the program and you will be on your way to financial security. If you are a fellow business owner and are in financial trouble like I was, or you want to start your own business, consider this a sign. I DID! Sincerely, Johnathon Rourke A PERSONAL NOTE FROM THE ORIGINATOR OF THIS PROGRAM: By the time you have read the enclosed program and reports, you should have concluded that such a program, and one that is legal, could not have been created by an amateur. Let me tell you a little about myself. I had a profitable business for 10 years. Then in 1979 my business began falling off. I was doing the same things that were previously successful for me, but it wasn't working. Finally, I figured it out. It wasn't me, it was the economy. Inflation and recession had replaced the stable economy that had been with us since 1945.I don't have to tell you what happened to the unemployment rate... because many of you know from first hand experience. There were more failures and bankruptcies than ever before. The middle class was vanishing. Those who knew what they were doing invested wisely and moved up. Those who did not, including those who never had anything to save or invest, were moving down into the ranks of the poor. As the saying goes, "THE RICH GET RICHER AND THE POOR GET POORER." The traditional methods of making money will never allow you to "move up" or "get rich", inflation will see to that. You have just received information that can give you financial freedom for the rest of your life, with "NO RISK" and "JUST A LITTLE BIT OF EFFORT." You can make more money in the next few months than you have ever imagined. I should also point out that I will not see a penny of this money, nor anyone else who has provided a testimonial for this program. I have already made over 4 MILLION DOLLARS!I have retired from the program after sending thousands and thousands of programs. Follow the program EXACTLY AS INSTRUCTED. Do not change it in any way . It works exceedingly well as it is now. Remember to e-mail a copy of this exciting report to everyone you can think of. One of the people you send this to may send out 50,000...and your name will be on everyone of them! Remember though, the more you send out the more potential customers you will reach. So my friend, I have given you the ideas, information, materials and opportunity to become financially independent. IT IS UP TO YOU NOW! "THINK ABOUT IT" Before you delete this program from your mailbox, as I almost did, take a little time to read it and REALLY THINK ABOUT IT. Get a pencil and figure out what could happen when YOU participate. Figure out the worst possible response and no matter how you calculate it, you will still make a lot of money! You will definitely get back what you invested. Any doubts you have will vanish when your first orders come in. IT WORKS! Jody Jacobs, Richmond, VA HERE'S HOW THIS AMAZING PROGRAM WILL MAKE YOU THOUSANDS OF DOLLAR$ INSTRUCTIONS: This method of raising capital REALLY WORKS 100% EVERY TIME. I am sure that you could use up to $50,000 or more in the next 90 days. Before you say "BULL... ", please read this program carefully. This is not a chain letter, but a perfectly legal money making opportunity. Basically, this is what you do: As with all multi-level businesses, we build our business by recruiting new partners and selling our products. Every state in the USA allows you to recruit new multi-level business partners, and we offer a product for EVERY dollar sent. YOUR ORDERS COME BY MAIL AND ARE FILLED BY E-MAIL, so you are not involved in personal selling. You do it privately in your own home, store or office. This is the GREATEST Multi-Level Mail Order Marketing anywhere. This is what you MUST do: 1. Order all 4 reports shown on the list below (you can't sell them if youdon't order them). -- For each report, send $5.00 CASH, the NAME & NUMBER OF THE REPORT YOU ARE ORDERING, YOUR E-MAIL ADDRESS, and YOUR NAME & RETURN ADDRESS (in case of a problem) to the person whose name appears on the list next to the report. MAKE SURE YOUR RETURN ADDRESS IS ON YOUR ENVELOPE IN CASE OF ANY MAIL PROBLEMS! -- When you place your order, make sure you order each of the four reports. You will need all four reports so that you can save them on your computer and resell them. -- Within a few days you will receive, via e-mail, each of the four reports. Save them on your computer so they will be accessible for you to send to the 1,000's of people who will order them from you. 2. IMPORTANT DO NOT alter the names of the people who are listed next to each report, or their sequence on the list, in any way other than is instructed below in steps "a" through "f" or you will lose out on the majority of your profits. Once you understand the way this works, you'll also see how it doesn't work if you change it. Remember, this method has been tested,and if you alter it, it will not work. a. Look below for the listing of available reports. b. After you've ordered the four reports, take this advertisement and remove the name and address under REPORT #4. This person has made it through the cycle and is no doubt counting their $50,000! c. Move the name and address under REPORT #3 down to REPORT #4. d. Move the name and address under REPORT #2 down to REPORT #3. e. Move the name and address under REPORT #1 down to REPORT #2. f. Insert your name/address in the REPORT #1 position. Please make sure you COPY ALL INFORMATION, every name and address, ACCURATELY! 3. Take this entire letter, including the modified list of names, and save it to your computer. Make NO changes to the instruction portion of this letter. Your cost to participate in this is practically nothing (surely you can afford $20). You obviously already have an Internet connection and e-mail is FREE! There are two primary methods of building your downline: METHOD #1: SENDING BULK E-MAIL Let's say that you decide to start small, just to see how it goes, and we'll assume you and all those involved send out only 2,000 programs each. Let's also assume that the mailing receives a 0.5% response. Using a good list the response could be much better. Also, many people will send out hundreds of thousands of programs instead of 2,000. But continuing with this example, you send out only 2,000 programs. With a 0.5% response, that is only 10 orders for REPORT #1. Those 10 people respond by sending out 2,000 programs each for a total of 20,000. Out of those 0.5%, 100 people respond and order REPORT #2. Those 100 mail out 2,000 programs each for a total of 200,000. The 0.5% response to that is 1,000 orders for REPORT #3. Those 1,000 send out 2,000 programs each for a 2,000,000 total. The 0.5% response to that is 10,000 orders for REPORT #4. That's 10,000 $5 bills for you. CASH!!! Your total income in this example is $50 + $500 + $5,000 + $50,000 for a total of $55,550!!! REMEMBER FRIEND, THIS IS ASSUMING 1,990 OUT OF THE 2,000 PEOPLE YOU MAIL TO WILL DO ABSOLUTELY NOTHING AND TRASH THIS PROGRAM! DARE TO THINK FOR A MOMENT WHAT WOULD HAPPEN IF EVERYONE, OR HALF SENT OUT 100,000 PROGRAMS INSTEAD OF 2,000. Believe me, many people will do justthat, and more! By the way, your cost to participate in this is practically nothing. You obviously already have an Internet connection and e-mail is FREE!!! REPORT #2 will show you the best methods for bulk e-mailing, tell you where to obtain free bulk e-mail software and where to obtain e-mail lists. METHOD #2 - PLACING FREE ADS ON THE INTERNET Advertising on the internet is very, very inexpensive, and there are HUNDREDS of FREE places to advertise. Let's say you decide to start small just to see how well it works. Assume your goal is to get ONLY 10 people to participate on your first level. (Placing a lot of FREE ads on the Internet will EASILY get a larger response.) Also assume that everyone else in YOUR ORGANIZATION gets ONLY 10 downline members. Follow this example to achieve the STAGGERING results below: 1st level--your 10 members with $5.......................................$50 2nd level--10 members from those 10 ($5 x 100)..................$500 3rd level--10 members from those 100 ($5 x 1,000)...........$5,000 4th level--10 members from those 1,000 ($5 x 10,000).....$50,000 THIS TOTALS ---------->$55,550 Remember friends, this assumes that the people who participate only recruit 10 people each. Think for a moment what would happen if they got 20 people to participate! Most people get 100's of participants! THINK ABOUT IT! For every $5.00 you receive, all you must do is e-mail them the report they ordered. THAT'S IT! ALWAYS PROVIDE SAME-DAY SERVICE ON ALL ORDERS! This will guarantee that the e-mail THEY send out with YOUR name and address on it will be prompt because they can't advertise until they receive the report! AVAILABLE REPORTS *** Order Each REPORT by NUMBER and NAME *** Notes: -- ALWAYS SEND $5 CASH (U.S. CURRENCY) FOR EACH REPORT. CHECKS NOT ACCEPTED. -- ALWAYS SEND YOUR ORDER VIA FIRST CLASS MAIL. -- Make sure the cash is concealed by wrapping it in at least two sheets of paper. On one of those sheets of paper, include: (a) the number & name of the report you are ordering, (b) your e-mail address, and (c) your name & postal address. PLACE YOUR ORDER FOR THESE REPORTS NOW: REPORT #1 "The Insider's Guide to Advertising for Free on the Internet" ORDER REPORT #1 FROM : B.Lapole P.O. Box 642 Spring, Texas 77383-0642 REPORT #2 "The Insider's Guide to Sending Bulk E-mail on the Internet" ORDER REPORT #2 FROM: A. Williams 10606 Twin Circle Montgomery, Texas 77356 REPORT #3 "The Secrets to Multi-level Marketing on the Internet" ORDER REPORT #3 FROM: R. Ehrgott 10130 Mossycup Conroe, Texas 77304 REPORT #4 "How to Become a Millionaire Utilizing the Power of Multi-level Marketing and the Internet" ORDER REPORT #4 FROM: D. Vasicak 479 Bennett Street Luzerne, PA 18709 About 50,000 new people get online every month! ******* TIPS FOR SUCCESS ******* -- TREAT THIS AS YOUR BUSINESS! Be prompt, professional, and follow the directions accurately. -- Send for the four reports IMMEDIATELY so you will have them when the orders start coming in because: When you receive a $5 order, you MUST send out the requested product/report. -- ALWAYS PROVIDE SAME-DAY SERVICE ON THE ORDERS YOU RECEIVE. -- Be patient and persistent with this program. If you follow the instructions exactly, your results WILL BE SUCCESSFUL! -- ABOVE ALL, HAVE FAITH IN YOURSELF AND KNOW YOU WILL SUCCEED! ******* YOUR SUCCESS GUIDELINES ******* Follow these guidelines to guarantee your success: If you don't receive 20 orders for REPORT #1 within two weeks, continue advertising or sending e-mails until you do. Then, a couple of weeks later you should receive at least 100 orders for REPORT#2. If you don 't, continue advertising or sending e-mails until you do. Once you have received 100 or more orders for REPORT #2, YOU CAN RELAX, because the system is already working for you, and the cash will continue to roll in! THIS IS IMPORTANT TO REMEMBER: Every time your name is moved down on the list, you are placed in front of a DIFFERENT report. You can KEEP TRACK of your PROGRESS by watching which report people are ordering from you. If you want to generate more income, send another batch of e-mails or continue placing ads and start the whole process again! There is no limit to the income you will generate from this business! Before you make your decision as to whether or not you participate in this program. Please answer one question. DO YOU WANT TO CHANGE YOUR LIFE? If the answer is yes, please look at the following facts about this program: 1. You are selling a product which does not Cost anything to PRODUCE, SHIP OR ADVERTISE. 2. All of your customers pay you in CASH! 3. E-mail is without question the most powerful method of distributing information on earth. This program combines the distribution power of e-mail together with the revenue generating power of multi-level marketing. 4. Your only expense--other than your initial $20 investment--is your time! 5. Virtually all of the income you generate from this program is PURE PROFIT! 6. This program will change your LIFE FOREVER. ACT NOW!Take your first step toward achieving financial independence. Orderthe reports and follow the program outlined above--SUCCESSwill be yourreward. Thank you for your time and consideration. PLEASE NOTE: If you need help with starting a business, registering a business name, learning how income tax is handled, etc., contact your localoffice of the Small Business Administration (a Federal Agency) 1-800-827-5722 for free help and answers to questions. Also, the InternalRevenue Service offers free help via telephone and free seminars aboutbusiness tax requirements. Your earnings are highly dependant on youractivities and advertising. The information contained on this site and in the report constitutes no guarantees stated nor implied. In the event that it is determined that this site or report constitutes a guarantee of any kind, that guarantee is now void. The earnings amounts listed on this site and in the report are estimates only. If you have any questions of the legality of this program, contact the Office of Associate Director for Marketing Practices, Federal Trade Commission, Bureau of Consumer Protection in Washington, DC. ///////////////////////////////////////////////////////////////// Remove at boardgames@goplay.com ///////////////////////////////////////////////////////////////// From nfs4-wg-request@sunroof.eng.sun.com Wed Jan 31 23:36:16 2001 Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA05418 for ; Wed, 31 Jan 2001 23:36:16 -0500 (EST) Received: from engmail4.Eng.Sun.COM ([129.144.134.6]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id UAA12539; Wed, 31 Jan 2001 20:35:26 -0800 (PST) Received: from sunroof.eng.sun.com (sunroof.Eng.Sun.COM [129.146.168.88]) by engmail4.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id UAA08121; Wed, 31 Jan 2001 20:35:10 -0800 (PST) Received: from engmail3.Eng.Sun.COM (engmail3 [129.144.170.5]) by sunroof.eng.sun.com (8.11.2+Sun/8.11.2) with ESMTP id f114Yv222751 for ; Wed, 31 Jan 2001 20:34:57 -0800 (PST) Received: from patan.sun.com (patan.Central.Sun.COM [129.147.5.43]) by engmail3.Eng.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id UAA20286 for ; Wed, 31 Jan 2001 20:34:57 -0800 (PST) Received: from note.orchestra.cse.unsw.EDU.AU (note.orchestra.cse.unsw.EDU.AU [129.94.242.29]) by patan.sun.com (8.9.3+Sun/8.9.3) with SMTP id UAA12304 for ; Wed, 31 Jan 2001 20:34:54 -0800 (PST) Received: From notabene.cse.unsw.edu.au ([129.94.242.2] == beethoven.orchestra.cse.unsw.EDU.AU) (for ) (for ) (for ) By note With Smtp ; Thu, 1 Feb 2001 15:34:36 +1100 Received: from neilb by notabene.cse.unsw.edu.au with local (Exim 3.12 #1 (Debian)) id 14OBNq-0005eR-00; Thu, 01 Feb 2001 15:30:10 +1100 From: Neil Brown To: "Noveck, Dave" Date: Thu, 1 Feb 2001 15:30:08 +1100 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <14968.58960.594822.614268@notabene.cse.unsw.edu.au> Cc: Sergey Klyushin , nfsv4-wg@sunroof.eng.sun.com Subject: RE: Stateid and seqid In-Reply-To: message from Noveck, Dave on Wednesday January 31 References: <8C610D86AF6CD4119C9800B0D0499E331A6E62@red.nane.netapp.com> X-Mailer: VM 6.72 under Emacs 20.7.2 X-face: [Gw_3E*Gng}4rRrKRYotwlE?.2|**#s9D > If the operation is successful and state changes, then the stateid > > changes. > > According to David Robinson, the server may in some cases return the > same stateid (ie. not change it). I can see, from a operational perspective, that you could argue this - all operations that effect state maintenance also have a seqid to ensure sequencing and - the operations that only have a stateid (read/write) will not be adversely affected by certain state changes - e.g. gaining a lock. however I think that if you try to understand a protocol like NFSv4 from a purely operation perspective, you will get bad confused very quickly (Well, maye you won't but I suspect that many would, myself included). Complexity needs to managed by abstraction (that is what computer science is all about) and so we need to understand and reason about the protocol at an abstract level. When you are reasoning like this (abstractly), having a "stateid" that "uniquely defines the locking state granted by the server for a specific lock owner for a specific file" and then not changing that stateid when the locking state changes is a recipe for disaster. > > > If the operation fails and the state doesn't change, then the > > seqid changes. > > The current approach is that seqid does not change if the operation > fails. That's what people are implementing to, but there has been > disagreement about whether the spec says that (e.g. Mike Eisler believes > it does) > ButButButButButButButButButButButButButBut Numbers in [brackets] are seqids. As our story opens, the "last sequence number (L) received" is [L]. Client say [L+1] "lock this bit of that file". Server says "No, there is a conflicting lock" but reply is delayed in transit. As this is an error return (NFS4ERR_DENIED), the server doesn't touch the seqid (you seem to say), it leave it at [L]. Client is impatient and again says [L+1] "lock this bit of that file". Client gets first reply and says "oh well, such it life". SomeotherClient unlocks "this bit of that file". Server gets resend. The rpc reply cache has been busy and it doesn't catch this one. The seqid looks fine ([L+1] just like the last time), and the server notices that there is no conflicting lock now, so it returns with success. It updates the "last sequence number received" to [L+1] and stores "the response that was returned. Client doesn't notice this reply because it doesn't really care any more. Maybe it was lost in transit anyway - it happens. Client tries to [L+1] "lock that other bit of that file". Now this has the same seqid as the last successful lock request for this lockowner, so "the stored response is returned" - success. Net result: client thinks that it locked "that other bit" and server thinks that it has "this bit" locked. Not a good situation. Am I missing something? (all quotes from rfc3010 section 8.1.5). > > I'm certainly curious to see if we could have something simpler (but > not too simple). Even if people don't think it enough better to > change v4 at this point (and that is going to be a high hurdle), > it would be valuable as input to the next minor version. > I'm working on it, but it wont be today. NeilBrown