factory report about first 1000 Ben NanoNotes
wolfgang at sharism.cc
Sun Jan 31 17:26:43 EST 2010
the first 1000 Ben NanoNotes were reflashed in our factory in Shenzhen two
weeks ago with the 20100113 image, here is how it went:
--- day 1
Reflashed 280 devices, 257 passed, 23 were singled out:
2: battery connector too short & battery loose
2: microSD connector has UV glue residue & card got stuck
3: battery empty when reflashing
4: didn't boot from microSD (don't know which card)
1: didn't boot from 16 GB class 6 microSD
6: hung when booting from NAND
2: LCD color with red cast
1: neither power-off button nor 'halt' works
1: 'halt' works but power-off button does not work
1: right-most keys don't work
For documentation of the reflashing steps we used see
We often (1/3rd of devices) got 'erase counter' warnings from ubiformat:
"ubiformat: warning!: only 1153 of 4000 eraseblocks have valid erase counter
erase counter 0 will be used for all eraseblocks
continue? yes/no: yes"
Eventually we added a "-e 0" parameter to ubiformat to silence the warnings.
Also we had a number of problems flashing until we added an
"nerase 0 4096 0 0" command before writing u-boot and the kernel.
Both of this should probably be thought through better... :-)
--- day 2
Reflashed 320 devices, 306 passed, 14 singled out:
1: device had a relatively loose hinge, reworked
2: USB connection problem ("cannot enumerate USB device")
1: cannot flash kernel (u-boot flashed OK)
1: cannot boot from NAND
2: cannot turn off with power button
1: cannot turn on
1: kernel hang when booting
2: doesn't boot from microSD
3: cannot be turned off with 'halt' or power-button
Much smoother than on first day, still some issues with bad blocks.
--- day 3
Reflashed 387 devices, 379 passed, 8 singled out:
1: doesn't boot from SD card (kept the card)
1: cannot flash kernel (u-boot OK)
1: ubiformat fails with i/o error reading from microSD card
1: hinge too lose (reworked)
1: can't turn on (turned out to be battery connector, reworked)
1: doesn't boot from SD card
2: root@(none) after I/O error when flashing rootfs
--- bottom line
Overall I am SUPER HAPPY about the hardware quality we have seen in the 1000
devices, and also with the quality of our kernel and reflashing software at
Thanks so much to all the people who helped, from Mirko Vogt and Lindner to
Lars, Xiangfu, Yi, Javi and many others I forgot.
We will start to collect quality notes, pass/fail criteria etc. in the wiki.
Check it out there are some cool pics and background infos up there already:
When you work with 1000 devices, you run into many strange one-off problems
and it takes some time and patience to find patterns and root causes.
We spent a few more days to work through the trouble-makers we had singled
out in the morning-to-midnight reflashing days...
Two devices stubbornly refused to connect to several notebooks running
Linux ("cannot enumerate USB device"), but worked fine when connected to
Windows machines. Very strange stuff. We could not find an electrical problem
on the boards, so for those 2 just decided to give up and not use those boards.
We still are not very good dealing with NAND and bad blocks in general. From
simple things like erase counter warnings, to bigger problems like a bug in
ubiformat when running into bad blocks, see
"ubiformat runs into I/O error at EOF after encountering bad block"
I also think the following 3 bugs, which can only be reproduced on a particular
device each, may be NAND or bad block related:
(the proprietary dictionary software works just fine on these devices)
"Linux kernel hangs on this particular device"
"this particular device cannot boot from microSD"
We found that 6 out of 1000 devices had 1 bad block in the first 40 MB of
NAND (out of 2GB total), so we expect 30% of devices will have 1 bad block
somewhere in the whole 2 GB.
We singled out those 6 devices and will get them in front of developers so
that hopefully our bad block handling will get more robust soon.
All in all - very nice time in the factory, lots of good work and good
progress happening. Need to do follow-up bug hunting now, so hopefully by
the time we are doing the next run everything will go even smoother.
Everybody keep up the faith and keep up the good work! :-)
And thanks again to the wonderful help we are getting from everywhere.
Without your help none of this would be possible!
More information about the discussion