From peter.barada@logicpd.com Mon Jan 25 20:39:53 2010 Received: from mail.logicpd.com ([66.162.60.3] helo=smtp.logicpd.com) by stoneboat.aleph1.co.uk with esmtp (Exim 4.69) (envelope-from ) id 1NZVis-0005Cw-DD for yaffs@lists.aleph1.co.uk; Mon, 25 Jan 2010 20:39:53 +0000 Received: from 10.0.5.29 ([10.0.5.29]) by lpdsrv13.logicpd.com ([10.0.4.21]) with Microsoft Exchange Server HTTP-DAV ; Mon, 25 Jan 2010 20:39:39 +0000 Received: from thunk by mail.logicpd.com; 25 Jan 2010 15:39:46 -0500 From: Peter Barada To: YAFFS Content-Type: multipart/mixed; boundary="=-u4AvvBPuJPyuIVMloOz4" Organization: Logic Product Development, Inc. Date: Mon, 25 Jan 2010 15:39:46 -0500 Message-ID: <1264451986.13568.100.camel@thunk> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 X-SA-Exim-Connect-IP: 66.162.60.3 X-SA-Exim-Mail-From: peter.barada@logicpd.com X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on stoneboat.aleph1.co.uk X-Spam-Level: X-Spam-Status: No, score=-2.5 required=4.5 tests=AWL,BAYES_00,HTML_MESSAGE autolearn=no version=3.2.5 X-SA-Exim-Version: 4.2.1 (built Wed, 25 Jun 2008 17:14:11 +0000) X-SA-Exim-Scanned: Yes (on stoneboat.aleph1.co.uk) Subject: [Yaffs] Weirndess testing YAFFS2 with large files - md5sums don't match when copied. X-BeenThere: yaffs@lists.aleph1.co.uk X-Mailman-Version: 2.1.11 Precedence: list List-Id: Discussion of YAFFS NAND flash filesystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2010 20:39:53 -0000 --=-u4AvvBPuJPyuIVMloOz4 Content-Type: multipart/alternative; boundary="=-D8Vmb0jvcLyrsgtOqpxT" --=-D8Vmb0jvcLyrsgtOqpxT Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I've run into a problem using the latest YAFFS code on linux-2.6.28-rc8 using today's YAFFS CVS code. First off I had to modify the code in yaffs_write_begin to only call grab_cache_pages_write_begin if the kernel version is newer than 2.6.28 since 2.6.28-rc8 does *not* have __grab_cache_page whereas 2.6.28 proper does: #if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 28) pg =3D grab_cache_page_write_begin(mapping, index, flags); #else pg =3D __grab_cache_page(mapping, index); #endif Second I added "dev->nPageWrites++;" to nandmtd2_ReadChunkWithTagsFromNAND and "dev->nPageReads++;" to nandmtd2_WriteChunkWithTagsToNAND to track the number of page read/writes. With this code, I'm seeing 30MB files that are created have mismatching checksums while running the attached test script. The output from the test looks like: OMAP-35x# . /media/mmcblk0p1/x = =20 Create 30M file and get md5sum =20 30720+0 records in =20 30720+0 records out =20 5b04790304a4221f1016a8c310da4746 somefile.1 =20 **>> Block 710 needs retiring =20 **>> yaffs write required 2 attempts =20 **>> Block 710 retired =20 Block 710 is in state 9 after gc, should be erased =20 Calculate md5sums for copied files =20 5b04790304a4221f1016a8c310da4746 somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.2 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 8c8d5a7974d0b9da747bc59edd1991f6 somefile.4 =20 execute sync and recalculate md5sums =20 save exit: isCheckpointed 1 =20 5b04790304a4221f1016a8c310da4746 somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.2 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 5b04790304a4221f1016a8c310da4746 somefile.4 =20 Delete one of the files =20 5b04790304a4221f1016a8c310da4746 somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 3cb7668eb7760202d96970a6a9a3361f somefile.4 =20 recopy the deleted file =20 f6dba6d5af7a7a89481da1849035a417 somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 5b04790304a4221f1016a8c310da4746 somefile.4 =20 f6dba6d5af7a7a89481da1849035a417 somefile.7 =20 Creating test folder and some junk files in that folder =20 1+0 records in =20 1+0 records out =20 ae1028b8d6aef86d020c9edfae29ca3d junk.1 =20 md5sums of all files in test folder =20 ae1028b8d6aef86d020c9edfae29ca3d junk.1 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.2 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.3 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.4 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.5 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.6 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.7 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.8 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.9 =20 execute sync and recalculate md5sums =20 save exit: isCheckpointed 1 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.1 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.2 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.3 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.4 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.5 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.6 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.7 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.8 =20 ae1028b8d6aef86d020c9edfae29ca3d junk.9 =20 Remove some files and recreate them =20 Calculate md5sums for 30M files again =20 f6dba6d5af7a7a89481da1849035a417 somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 5b04790304a4221f1016a8c310da4746 somefile.4 =20 1fee3f481bfa5cf3403efe9e481a0374 somefile.7 =20 execute sync and recalculate md5sums =20 save exit: isCheckpointed 1 =20 a7b6ccfa31115aa75a0fdca07073293d somefile.1 =20 5b04790304a4221f1016a8c310da4746 somefile.3 =20 1abb3d578e2d129341df26916090b869 somefile.4 =20 f6dba6d5af7a7a89481da1849035a417 somefile.7 =20 OMAP-35x#=20 In the output, note that the md5sum of "somefile.*" should all match. Anyone seen anything like this before? Test attached. --=-D8Vmb0jvcLyrsgtOqpxT Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable I've run into a problem using the latest YAFFS code on linux-2.6.28-rc8 usi= ng today's YAFFS CVS code.

First off I had to modify the code in  yaffs_write_begin to only call = grab_cache_pages_write_begin if the kernel version is newer than 2.6.28 sin= ce 2.6.28-rc8 does *not* have __grab_cache_page whereas 2.6.28 proper does:=

#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 28)
pg =3D grab_cache_page_write_begin(mapping, index, flags);
#else
pg =3D __grab_cache_page(mapping, index);
#endif

Second I added "dev->nPageWrites++;" to nandmtd2_ReadChunkWith= TagsFromNAND and "dev->nPageReads++;" to nandmtd2_WriteChunkWi= thTagsToNAND to track the number of page read/writes.

With this code, I'm seeing 30MB files that are created have mismatching che= cksums while running the attached test script.  The output from the te= st looks like:

OMAP-35x# . /media/mmcblk0p1/x       &nb= sp;            =             &nb= sp;            =     
Create 30M file and get md5sum       &nb= sp;            =             &nb= sp;            =     
30720+0 records in         &nb= sp;            =             &nb= sp;            =             &nb= sp; 
30720+0 records out         &n= bsp;            = ;            &n= bsp;            = ;            &n= bsp;
5b04790304a4221f1016a8c310da4746  somefile.1    &n= bsp;            = ;            &n= bsp;     
**>> Block 710 needs retiring      &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;     
**>> yaffs write required 2 attempts     &nb= sp;            =             &nb= sp;            =
**>> Block 710 retired        = ;            &n= bsp;            = ;            &n= bsp;           
Block 710 is in state 9 after gc, should be erased    &= nbsp;           &nbs= p;             =
Calculate md5sums for copied files       = ;            &n= bsp;            = ;            &n= bsp;
5b04790304a4221f1016a8c310da4746  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.2    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
8c8d5a7974d0b9da747bc59edd1991f6  somefile.4    &n= bsp;            = ;            &n= bsp;     
execute sync and recalculate md5sums      &nb= sp;            =             &nb= sp;           
save exit: isCheckpointed 1        =             &nb= sp;            =             &nb= sp;      
5b04790304a4221f1016a8c310da4746  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.2    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.4    &n= bsp;            = ;            &n= bsp;     
Delete one of the files        &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;         
5b04790304a4221f1016a8c310da4746  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
3cb7668eb7760202d96970a6a9a3361f  somefile.4    &n= bsp;            = ;            &n= bsp;     
recopy the deleted file        &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;         
f6dba6d5af7a7a89481da1849035a417  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.4    &n= bsp;            = ;            &n= bsp;     
f6dba6d5af7a7a89481da1849035a417  somefile.7    &n= bsp;            = ;            &n= bsp;     
Creating test folder and some junk files in that folder   &n= bsp;            = ;        
1+0 records in          &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;    
1+0 records out          =             &nb= sp;            =             &nb= sp;            =     
ae1028b8d6aef86d020c9edfae29ca3d  junk.1     =             &nb= sp;            =          
md5sums of all files in test folder      &nbs= p;            &= nbsp;           &nbs= p;             =
ae1028b8d6aef86d020c9edfae29ca3d  junk.1     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.2     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.3     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.4     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.5     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.6     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.7     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.8     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.9     =             &nb= sp;            =          
execute sync and recalculate md5sums      &nb= sp;            =             &nb= sp;           
save exit: isCheckpointed 1        =             &nb= sp;            =             &nb= sp;      
ae1028b8d6aef86d020c9edfae29ca3d  junk.1     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.2     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.3     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.4     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.5     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.6     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.7     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.8     =             &nb= sp;            =          
ae1028b8d6aef86d020c9edfae29ca3d  junk.9     =             &nb= sp;            =          
Remove some files and recreate them      &nbs= p;            &= nbsp;           &nbs= p;             =
Calculate md5sums for 30M files again      &n= bsp;            = ;            &n= bsp;          
f6dba6d5af7a7a89481da1849035a417  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.4    &n= bsp;            = ;            &n= bsp;     
1fee3f481bfa5cf3403efe9e481a0374  somefile.7    &n= bsp;            = ;            &n= bsp;     
execute sync and recalculate md5sums      &nb= sp;            =             &nb= sp;           
save exit: isCheckpointed 1        =             &nb= sp;            =             &nb= sp;      
a7b6ccfa31115aa75a0fdca07073293d  somefile.1    &n= bsp;            = ;            &n= bsp;     
5b04790304a4221f1016a8c310da4746  somefile.3    &n= bsp;            = ;            &n= bsp;     
1abb3d578e2d129341df26916090b869  somefile.4    &n= bsp;            = ;            &n= bsp;     
f6dba6d5af7a7a89481da1849035a417  somefile.7    &n= bsp;            = ;            &n= bsp;     
OMAP-35x#

In the output, note that the md5sum of "somefile.*" should all ma= tch.

Anyone seen anything like this before?  Test attached.


--=-D8Vmb0jvcLyrsgtOqpxT-- --=-u4AvvBPuJPyuIVMloOz4 Content-Type: application/x-shellscript; name="yaffs-30M-test" Content-Disposition: attachment; filename="yaffs-30M-test" Content-Transfer-Encoding: base64 IyEvYmluL2Jhc2gKCiMgY3JlYXRlIGEgMzAgTSBmaWxlCmVjaG8gIkNyZWF0ZSAzME0gZmlsZSBh bmQgZ2V0IG1kNXN1bSIKZGQgaWY9L2Rldi91cmFuZG9tIG9mPXNvbWVmaWxlLjEgYnM9MWsgY291 bnQ9MzA3MjAKCm1kNXN1bSBzb21lZmlsZS4xCgojIGNyZWF0ZSBjb3BpZXMgb2YgZmlsZQpmb3Ig ZiBpbiAyIDMgNCA7CmRvCiAgIGNwIHNvbWVmaWxlLjEgc29tZWZpbGUuJGYKZG9uZQoKZWNobyAi Q2FsY3VsYXRlIG1kNXN1bXMgZm9yIGNvcGllZCBmaWxlcyIKbWQ1c3VtIHNvbWVmaWxlLioKZWNo byAiZXhlY3V0ZSBzeW5jIGFuZCByZWNhbGN1bGF0ZSBtZDVzdW1zIgpzeW5jCm1kNXN1bSBzb21l ZmlsZS4qCgplY2hvICJEZWxldGUgb25lIG9mIHRoZSBmaWxlcyIKcm0gc29tZWZpbGUuMgptZDVz dW0gc29tZWZpbGUuKgplY2hvICJyZWNvcHkgdGhlIGRlbGV0ZWQgZmlsZSIKY3Agc29tZWZpbGUu MSBzb21lZmlsZS43Cm1kNXN1bSBzb21lZmlsZS4qCgplY2hvICJDcmVhdGluZyB0ZXN0IGZvbGRl ciBhbmQgc29tZSBqdW5rIGZpbGVzIGluIHRoYXQgZm9sZGVyIgpta2RpciAtcCB0ZXN0CgpjZCB0 ZXN0CgpkZCBpZj0vZGV2L3VyYW5kb20gb2Y9anVuay4xIGJzPTFLIGNvdW50PTEKCm1kNXN1bSBq dW5rLjEKCmZvciBmIGluIDIgMyA0IDUgNiA3IDggOTsKZG8KICAgY3AganVuay4xIGp1bmsuJGYK ZG9uZQoKZWNobyAibWQ1c3VtcyBvZiBhbGwgZmlsZXMgaW4gdGVzdCBmb2xkZXIiCm1kNXN1bSBq dW5rLioKCmVjaG8gImV4ZWN1dGUgc3luYyBhbmQgcmVjYWxjdWxhdGUgbWQ1c3VtcyIKc3luYwoK bWQ1c3VtIGp1bmsuKgoKZWNobyAiUmVtb3ZlIHNvbWUgZmlsZXMgYW5kIHJlY3JlYXRlIHRoZW0i CmZvciBmIGluICAzIDUgODsKZG8KICAgcm0ganVuay4kZgpkb25lCgpmb3IgZiBpbiAgOCAzIDU7 CmRvCiAgIGNwIGp1bmsuMSBqdW5rLiRmCmRvbmUKCmNkIC4uCgplY2hvICJDYWxjdWxhdGUgbWQ1 c3VtcyBmb3IgMzBNIGZpbGVzIGFnYWluIgptZDVzdW0gc29tZWZpbGUuKgplY2hvICJleGVjdXRl IHN5bmMgYW5kIHJlY2FsY3VsYXRlIG1kNXN1bXMiCnN5bmMKbWQ1c3VtIHNvbWVmaWxlLioKCg== --=-u4AvvBPuJPyuIVMloOz4--