[Yaffs] data loss under heavy pressure on Yaffs2 GC

[Charles Manning]

Hi Artis

I had a bit more of a look at this.

On Saturday 30 July 2005 02:16, Artis Kugevics wrote:
> Hello,
>
> It looks like I have found a bug in Garbage Collection.
>
> If yaffs2 is under heavy pressure on free space, sometimes it corrupts some
> files. File corruption is such, that part of file is lost (all bytes are 0
> there). When reading such file, it gives
> 	Chunk -1 not found zero instead
> messages.
> That corruption can be detected only after file cache is flushed (I'm
> unmounting and mounting back that nand).

By this do you mean that the file looks fine until you unmount and remount?

If so, then I think  it is a scanning problem and not a gc problem. If it was 
a gc problem then the data would be lost immediately after the faulty gc and 
not after the remount.

> Here is trace output, when that file is written:
>
> 	yaffs_mknod: making oject for xxx, mode 81a4 dev 0
> 	Writing chunk 12538 tags 439 0
> 	Writing chunk 12539 tags 439 1
> 	Writing chunk 12540 tags 439 2
> 	Writing chunk 12541 tags 439 3
> 	Writing chunk 12542 tags 439 4
> 	Writing chunk 12543 tags 439 5

The above were written to block (12538/64) = 195

> 	Writing chunk 14784 tags 439 6
>
> 	GC Selected block 66 with 8 free
> 	yaffs: GC erasedBlocks 4 aggressive 1
> 	Collecting block 66, in use 56, shrink 0,
> 	Writing chunk 14785 tags 887 22
> 	line 2521 delete of chunk 4224
> 	...<GC stuff here>
> 	Writing chunk 14840 tags 1185 16
> 	line 2521 delete of chunk 4287
>
> 	Writing chunk 14841 tags 439 7
> 	Writing chunk 14842 tags 439 8
> 	Writing chunk 14843 tags 439 9
> 	Writing chunk 14844 tags 439 10
> 	Writing chunk 14845 tags 439 11
> 	Writing chunk 14846 tags 439 12
> 	Writing chunk 14847 tags 439 13

The above were written to block 231.

> 	Writing chunk 20160 tags 439 14
>
> 	GC Selected block 52 with 4 free
> 	yaffs: GC erasedBlocks 4 aggressive 1
> 	Collecting block 52, in use 60, shrink 0,
> 	Writing chunk 20161 tags 1126 30
> 	line 2521 delete of chunk 3328
> 	...<GC stuff here>
> 	Writing chunk 20220 tags 887 2
> 	line 2521 delete of chunk 3391
>
> 	Writing chunk 20221 tags 439 15
> 	Writing chunk 20222 tags 439 16
> 	Writing chunk 20223 tags 439 17

The above were written to block 315

> 	Writing chunk 37824 tags 439 18
>
> 	GC Selected block 240 with 4 free
> 	yaffs: GC erasedBlocks 4 aggressive 1
> 	Collecting block 240, in use 60, shrink 0,
> 	Writing chunk 37825 tags 1448 76
> 	line 2521 delete of chunk 15361
> 	...<GC stuff here>
> 	Writing chunk 37884 tags 1729 6
> 	line 2521 delete of chunk 15420
>
> 	Writing chunk 37885 tags 439 19
> 	Writing chunk 37886 tags 439 20
> 	Writing chunk 37887 tags 439 21

The above were written to block 591.

> 	Writing chunk 45056 tags 439 22

This was written to block 704.

As I mentioned before, gc is done on a block basis and not a file basis.

If you look at the blocks used, then you will see that various file parts are 
on various blocks.

Now some garbage has been detected on block 195 (it says 196, but because you 
are using block zero this is really 195). So what will happen now is we copy 
over the chunks 0..5 because they are useful data on that block.

> 	GC Selected block 196 with 3 free
> 	yaffs: GC erasedBlocks 4 aggressive 1
> 	Collecting block 196, in use 61, shrink 0,
> 	Writing chunk 45057 tags 932 243
> 	line 2521 delete of chunk 12544
> 	...<GC stuff here>
> 	Writing chunk 45109 tags 932 244
> 	line 2521 delete of chunk 12596
> 	Writing chunk 45110 tags 932 245
> 	line 2521 delete of chunk 12597
> 	Writing chunk 45111 tags 932 0
> 	line 2521 delete of chunk 12601
>
> <here comes the interesting part - GC collects file 'xxx', which is being
> written right now>


>
> 	Writing chunk 45112 tags 439 0
> 	line 2521 delete of chunk 12602
> 	Writing chunk 45113 tags 439 1
> 	line 2521 delete of chunk 12603
> 	Writing chunk 45114 tags 439 2
> 	line 2521 delete of chunk 12604
> 	Writing chunk 45115 tags 439 3
> 	line 2521 delete of chunk 12605
> 	Writing chunk 45116 tags 439 4
> 	line 2521 delete of chunk 12606
> 	Writing chunk 45117 tags 439 5
> 	line 2521 delete of chunk 12607
>
> <here GC ends its collection. Note, that blocks 0..5 of 'xxx' are rewritten
> by GC. It looks like here it looses those blocks 6..22 >

Chunks 6..22 were on blocks that were not collected so they should not be 
collected and should not be being lost.

>
> 	Writing chunk 45118 tags 439 23
> 	Writing chunk 45119 tags 439 24
> 	Writing chunk 3264 tags 439 25

In summary I don't see anything too problematic with the gc here.

-- Charles