Pushing toybox infrastructure upstream.

[Rob Landley]

On Sunday 14 March 2010 13:22:52 Rob Landley wrote:
> I'll detail the actual steps in doing it next message.

The central idea is having all the code for each command in a single file, with 
the other files generated from that file.

According to http://busybox.net/FAQ.html#adding the current things that are 
scattered around the tree instead of auto-generated from a central location 
are:

  Configuration entries (Config.in)
  Makefile entries (Kbuild)
  The lookup table (include/applets.h)
  Help text (include/usage.h)

In addition toybox handles command line options and per-command globals 
differently than busybox does.  You shouldn't have to manually call the 
getopt(), or #define FLAG_x or #define thingy G.thingy, or have an INIT_G().  It 
should all happen behind the scenes for you.

The first step would be creating a new makefile snippet (possibly an included 
makefile) that handles the "generated" directory.  All of the files generated 
from data in *.c files should live in a single place that you can clean with 
"rm -rf generated".  (It's nice to avoid mixing volatile and repository data 
where possible.)

Moving the Config.in entries is easy and could even be done incrementally, just 
create a new generated/Config.in file and have converted apps add their blocks 
to it.  The directory layout could be used to create menus.  Then during the 
transition period, just include the new Config.in from the old one.

The easiest way to do this is to convert one subdirectory at a time.  That way 
you don't have two menus for the same subdirectory, and don't have to worry 
about renaming config symbols to avoid conflicts from two menus covering the 
same symbol (and thus making people change their .configs to add the new 
symbol; yes kbuild should automatically enable the menu guard when a symbol in 
the menu is set, but it doesn't yet).

Makefile entries can be converted similarly, include generated/Makefile from 
another makefile.  Those are even easier to do because there aren't menu sumbol 
conflicts.

The lookup table is an issue: that has to be converted all at once due to the 
alphebetization.

None of those should have any size impact, it's all just refactoring.

Converting the globals probably comes next, and may actualy be a size win.  
(We'd have to measure.  It would certainly result in _cleaner_ code, with the 
"#define thingy G.thingy" stacks and the INIT_G() blocks going away.)

From this point, we hit places where toybox itself is unfinished.

Converting the help is its own post, and involved rephrasing the help text and 
moving it into the Config.in entries.  The toybox help parser isn't quite 
finished, it's supposed to assemble sub-options' "usage:" lines into a single 
coherent usage: line but doesn't yet.  (Kinda hard to do in bash, and I don't 
want to introduce python or something as a build dependency.  I should 
probably make a C program for it.)  Also, it doesn't have configurable different 
levels of help text.  And toybox handles help text very differently than 
busybox does;  there's a "help" command a bit like bash help, but it's not 
shell-specific.  I've pondered aliasing man to it and having fallback behavior 
to look in the filesystem, but haven't gone there yet.

We'd have to work out what we want to do for help text, and presumably hold off 
doing it until the low hanging fruit was out of the way first.

Converting the option parsing logic is the biggest win in terms of code 
cleanup, but it's also the biggest design change, and still has some room for 
improvement.

This is tied into the lookup table generation, and probably involves swapping 
the old getopt32() for the new one I wrote that doesn't depend on the libc 
getopt() at all.

The main missing bit in toybox option parsing is that toybox is not currently 
autogenerating the "#define FLAG_x (1<<0)" macros.  There should be a big 
#include file that has #ifdef blocks for each applet which define the flags for 
you, generated by parsing the option string.  (So if they move in the option 
string, you don't have to change the C code.)  Unfortunately, I haven't worked 
out the details of implementing that yet...

The globals handling is _almost_ right.  I need to make the #define TT toybox 
currently has go away, probably some kind of:

  #define THIS wc
  #include "busybox.h"

And use that #define internally to do behind the scenes magic.  (This might 
also be able to make the autogenerated FLAG stuff work.  I need to study the 
c99 preprocessor spec to see what I've got to work with, but that can come 
later...)

Anyway, at least some of it can be done in stages.

Oh, and along the way I need to do some serious #ifdefectomy on this code...

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds