[Yaffs] Re: mv problem -->yaffs/mtd interface

[Charles Manning]

Pardon the top post.

Ian's synopsis is great. The yaffs/mtd interface is by far the softest part of 
using YAFFS on mtd. People using yaffs in an RTOS/WinCE/whatever environment 
typically write their own nand access functions and don't have these 
problems. From what I have heard, few Linux folk have written their own 
stripped down drivers to support yaffs more cleanly.

As Ian says, YAFFS' requirements are very simple, but mtd tries to do too much 
and has a nursemaid approach to mtd users making it difficult to access mtd 
at the level required.

Things are further complicated by mtd changes not being tested against yaffs. 
As a result, changes in mtd can break yaffs.  For example, when AUTOPLACE was 
added, it was tested against JFFS2 (which does not use OOB) but not against 
YAFFS. It broke YAFFS because YAFFS was using oob and the oob was being 
corrupted.

For yaffs1 (or yaffs2 in yaffs1 compatability mode), yaffs uses the following 
basic access functions:

* Read data and oob
* Read oob only to get tags (eg. during scanning)
* Write data + oob (write_ecc)
* Write oob only (write_oob) (used to mark bad blocks and deleted chunks).

To further complicate this, you can use either yaffs or mtd's ECC in the 
yaffs1 codebase. If you use yaffs' ECC then the mtd nursemaid gives warnings 
about writing without using mtd's ECC. 
read_oob/write_oob have an issue in that they don't provide oobinfo fields - 
just raw oob access. Thus we have a disconnect in that the read/write_ecc and 
read/write_oob can become inconsistent.

If you have nand verify on, then mtd bleats if you do valid nand writes that 
break the verification.  In NAND, a write of 0xFF is essentially a no-op, so 
to write the page deletion marker yaffs writes:
0xff...0xff 0x00, 0xff..0xff.  This is anded with what is already there. 
However, if NAND verification is on, then the verification will fail.

YAFFS2 uses a more abstract interface to be able to cater for a wider variety 
of geometries. This provides the ability to query a block state and to mark 
blocks bad etc. It also uses the autoplace to allow better integration with 
For the actual reading/writing. yaffs2 uses the following functions:
* Read data + oob using autoplace.
* Read only oob (should use autoplace, but that parameter is missing).
* Write data + oob using autoplace.

This interface is far cleaner, except that the read_oob does not currently 
support passing an oob_info parameter. I have been promised that this will 
happen, but I have been waiting a while and have not heard back.

I think it would be a Good Idea to mode YAFFS1 to this interface too since it 
is far cleaner. However, for that to happen, the mtd stuff needs to become 
available first.

Now I know that the mtd code is all just there and I can hack it and try push 
it back into the mtd group, but quite frankly:
1) I don't have the time to do this. I still have to kiss the wife and say 
hello to the kids occasionally.
2)YAFFS is an unpaid after hours activity for me. I don't actually use yaffs 
linux in any project that I work on. 
3) My time is gest spent on the core stuff.
4) I don't even have a Linux device that uses yaffs. I do all the testing I 
have done on mtd emulations in RAM.

If someone feels like taking up this area and working with mtd to work it to 
completion I am more than happy and will give that effort all the support I 
can.

-- Charles



On Tuesday 27 September 2005 06:28, ian at brightstareng.com wrote:
> On Wednesday 21 September 2005 17:55, Sergey Kubushyn wrote:
> > Can you simply tell something like "When using its internal
> > ECC YAFFS uses such and such MTD functions and expects that
> > they would return the raw NAND data?" It would be very easy to
> > fix those functions instead of trying to understand what YAFFS
> > is trying to do and what MTD think of that?
>
> This would be very helpful. I have invested more than a few days
> myself working out the wrinkles with the yaffs/mtd interface.
> And while doing this I had wished precisely this. A brief
> statement on the functional requirements of MTD by yaffs. I now
> know there are at least three use cases: original yaffs, yaffs2
> code running in yaffs1 mode, and yaffs2 running yaffs2.
>
> > I simply can't
> > believe that it requires two years to fix a couple of
> > functions in MTD. Name those functions and I'll fix them
> > myself.
>
> There are issues with all the functions that yaffs has to use to
> talk to MTD, there are issues with read, read_oob, write,
> write_oob, and block_markbad, isbad. Please take up your gripes
> with the MTD crew and help make this a priority.
>
> You have been beating-up on Yaffs which is unfortunate, because
> Yaffs actually has a very simple and clear interface to the
> underlying flash primitives. The MTD interfaces that are
> exported are in some ways too high-level for yaffs needs and
> unlike JFFS, which is maintained by the same crew that make
> decisions about MTD -- Yaffs is an innocent third-party just
> trying to get along.
>
> Now, even if we had a concise functional description of Yaffs
> requirements of MTD (and I do think that would be an excellent
> step forward) you would still have to reverse-engineer the
> functionality of a particular blend of MTD. So a concise
> description of the MTD API functions would be good too. And I
> don't mean some out-of-sync HOWTO or README. I would suggest
> including it in the appropriate source files.
>
> > What is "some mtds?" Are there many of them? I do only know
> > one, that in the Linux kernel.
>
> MTD does not have 'official' releases -- you may have a
> particular snapshot in your kernel source, there are many other
> snapshots. Most of us have a blend of MTD code retrieved from
> CVS at various times, with patches and local mods applied.
>
> The MTD crew no longer supports the 2.4 kernel base, so there's
> a major disconnect there.
>
> > Generic MTD utilities (nanddump, nandwrite etc.) do NOT have
> > any problems getting anything they need from MTD. That means
> > they DO know how to ask MTD layer for what they want.
>
> These utilities have minimal functionality and are part of the
> MTD code based - keeping them in-sync with the MTD driver code
> should be a no-brainer. They operate at the ioctl syscall level
> and have a relatively stable interface definition. They not
> really comparable to an MTD API client like Yaffs.
>
> > Then, when using a bare YAFFS, without NAND ECC and
> > tools--just erase the flash, mount it and copy files to the
> > mounted filesystem--you mean MTD somehow changes data it
> > receives from yaffs when writing files and send back to yaffs
> > when reading them?
>
> Yes, it does 'change data' when handling the critical oob/spare
> data. This region of each flash page is used for two distinct
> purposes: (a) to hold the ECC data associated with the real user
> data, and (b) to hold filesystem metadata, i.e. Yaffs tags etc.
> These two sources of data have to be packed and unpacked from
> the oob/spare area. With MTD's current API design the
> packing/unpacking has to be done by MTD _if_ MTD is responsible
> for ECC operations. This is the most unstable/buggy issue with
> Yaffs/MTD interworking.
>
> > I AM trying to be constructive.
>
> I (now) think you are.
>
> -imcd
>
> _______________________________________________
> yaffs mailing list
> yaffs at stoneboat.aleph1.co.uk
> http://stoneboat.aleph1.co.uk/cgi-bin/mailman/listinfo/yaffs