RFD: Rework/extending functionality of mdev

[Harald Becker]

Hi Laurent !

 >> 1) Starting up a system with mdev usually involves the same steps to
>> mount proc, sys and a tmpfs on dev, then adding some subdirectories
>> to /dev, setting owner, group and permissions, symlinking some
>> entries and possibly mounting more virtual file systems. This work
>> need to be done by the controlling startup script, so we reinvent the
>> wheel in every system setup.
>
>   The thing is, this work doesn't always need to be done, and when it
> does, it does whether you use udevd, mdev, or anything else. /dev
> preparation is one of the possible building blocks of an init routine,
> and it's quite separate from mounting /proc and /sys, and it's also
> separate from populating /dev with mdev -s.

You missed the fact, I told everything stays under control of the admin 
or system setup. Nobody shall be forced to do things in a specific way. 
I want just give some extended functionality and flexibility to do this 
setup in an very easy and IMO straight looking way.

If you like/need to do those mounts in a specific way, just put them in 
your script and leave out the mount lines from mdev.conf, otherwise if 
you need to setup those virtual file systems with specific options you 
can specify them in the same line of configuration.


> ... I dislike the idea of integrating early init functions into mdev, because those
> functions are geographically (as in the place where they're performed) close to mdev,
 > but they have nothing to do with mdev logically.

Sorry, I don't agree. mdev's purpose is to setup the device file system 
infrastructure, or at least to help with setting up this. Ok, at first 
leave out proc and sysfs, there you may be right, but what about devpts 
on /dev/pts, or /dev/mqueue, etc., and finally the tmpfs for /dev 
itself. Do you really say they do not belong to the device file system 
infrastructure? Or all those symlinks like /dev/fd, /dev/stdin, 
/dev/stdout, etc., do you consider them not to belong to mdev? Not 
talking about setting owner informations and permissions for those 
entries in the device file system.

Sure, all that can be done with some shell scripts, scattering around 
all that information at different places, or the need to setup and 
read/process different configuration files. My intention is to get this 
information which depend on system setup in a single, relatively simple 
to use or modifiable location. The startup script itself just invokes 
mdev, which gets the information from /etc/mdev.conf, which calls the 
necessary commands to do the operation. That is, it frees the distro 
manager to write script code to parse the system specific information 
into the startup script.

I consider Busybox to be a set of tools, anybody may use to setup a 
system to there own wishes, not to force anybody to do things in the way 
any person may feel being it's the best way to do things, but I want to 
enable functionality for those who like to collect this information and 
put them in a central place. Information usually scattered around and 
hidden deep in the scripts controlling the startup.


>> MOUNTPOINT  UID:GID  PERMISSIONS  %FSTYPE [=DEVICE] [OPTIONS]
>
>   I can't help but think this somehow duplicates the work of mount with
> /etc/fstab. If you really want to integrate the functionality in some
> binary instead of scripting it, I think the right way would be to
> patch the mount applet so it accepts any file (instead of hardcoding
> /etc/fstab), so you could have a "early mount points" file, distinct
> from your /etc/fstab, that you would call mount on before running
> mdev -s.

Laurent you didn't look at the examples. I do not want to hard code any 
*functionality* in mdev. I want add extra functionality to the current 
mdev.conf syntax to allow to do some more stuff, which is usually done 
"geographically close to calling mdev", and done in so many systems in 
similar matter.

Look at the usual usage of /etc/fstab, on how many systems do you find 
there information about your virtual file systems? Usually fstab is used 
for the disk devices. In addition what about creating the mount points, 
setting owner, group and permissions? This is not done by mount and not 
specified in fstab. So changing anything there would mean to modify 
fstab syntax, possibility breaking other programs and scripts, reading 
and modifying fstab.

Neither do I want to code any special functionality in a binary, nor do 
I try to duplicate the operation of mount. I just want to extend the 
mdev.conf syntax to add simple configuration information for those 
"close to mdev" operations at a central place, parse this information 
and call the usual commands, e.g. mount with the right options, as shell 
scripts do.

And what else are a few lines, in mdev.conf, describing those mounts, 
other then placing them in a separate "early mount points file"? ... ok, 
lets go one step further on this. Let us add an "include" option to 
mdev.conf, which allow to split the mdev configuration in different 
files, and/or place mdev.conf lines in separate files in a /etc/mdev.d/ 
directory. You like to have an "early mount points file"? Put those 
mount informations in a file and add an include for that file in 
mdev.conf ... ?

... and what is wrong about calling mount from mdev? Setting up the 
device files system includes mount operations, so mdev should be able to 
call mount, when and how it is required.

Using this functionality for proc, sysfs, etc. is just picking up the 
idea of an "early mount file systems file". Use it, if you like it, as 
other things, or leave it out, if you dislike. Neither do I want to code 
any specific *functionality* in a *binary*, nor is my intention to 
massively blow up the code size of mdev.


>> 2) I like to use netlink to obtain hotplug information and avoid
>> massive respawning of mdev as hotplug helper when several events
>> arrive quickly. That is, I want to auto fork a daemon which just open
>> the netlink socket.
>
>   So far, what you're talking about already exists:
>   http://skarnet.org/software/s6-linux-utils/s6-devd.html

Thanks for pointing me to this. I do not want to reinvent the wheel, 
just add that operation to BB, or at least ready usage of BB with some 
(small) external tool. I will have a look on this and on nldev 
(mentioned by Nathanael), and see what and how we can adapt this. As I 
told, I'm in the early stage of brain storming about that project, not 
in the phase of hacking on the code. Adding netlink operation to mdev 
has already been discussed, but dropped due it's complexity and amount 
of work to be done on this. I'm now willing to spend the time doing this 
rework of mdev, were this shall not produce extreme changes for system 
maintainers (or better not breaking things, when possible). In addition 
I want extend the mdev.conf syntax to easy some work (for those who like).


>> When events arrive it forks again, creating a
>> pipe. The new instance read mdev.conf, build a table of rules in
>> memory, then read hotplug operations from the pipe (send by the first
>> instance).
>
>   ... but here it's different. s6-devd spawns a helper program for
> every event, like /proc/sys/kernel/hotplug would do, but it reads the
> events from the netlink so they're serialized.

Spawning a process for every event produce massive slowdown on system 
startup. My intention is to fork only one process, parse the conf table 
once and then read sanitized events from stdin, scanning the in memory 
conf table and invoking the right operations.


>   In your design, the long-lived process forks a unique helper that
> reads mdev.conf... so it's not meant to be used with any program
> compatible with /proc/sys/kernel/hotplug - it's only meant to be
> used with mdev. So, why fork at all ? Your temporary instance is
> unnecessary - just have a daemon that parses mdev.conf, then
> listens to the netlink, and handles the events itself.

No, it is not unnecessary. The long lived instance tries to stay at low 
memory usage, then forks and second instance read mdev.conf into the 
memory table. When the event system goes idle, as it is most of the 
time, the second instance dies and memory is freed to the system.

The compatibility to hotplug helper is no problem, we can add the 
function to parse the environment into variables then read mdev.conf and 
do the operation, sharing most of the code ... but I got some additional 
ideas from your and Nathanaels message. We can create a modular system 
which allows either sysfs class scanning, hotplug helper or netlink 
usage, still avoiding process spawning/conf table parsing for every 
single event. I will pick that up later.


>   ... and you just reinvented udevd. Congratulations ! ;)

Laurent this is not very friendly. As mdev's operation is similar to 
udev, every work/extension of mdev allowing to use netlink will usually 
be some kind of udev reinvention. This does not mean to reinvent the 
complex rule set management of udev or the *coding of functionality into 
a binary*. I like to stay at the KISS principle, but allow flexibility 
for those who like, especially the embedded device maintainers world, 
who need to stay at the functionality provided by BB and can't install 
every upstream package they would like.


>   Seriously, udev is a hard problem. udevd is bloated and its configuration
> is too complex; mdev is close to the point where parsing a config file for
> every invocation will become too expensive - I believe that it has not
> reached that point yet, but if you add more stuff to the conf language, it
> will.

Right, adding more to the conf language adds complexity there, but 
removes complexity of the surrounding scripts and collect system 
specific information at a central place (or places), usually hidden deep 
in the startup scripts or some config files they use.

Beside this, I do not want to change existing mdev.conf syntax, so it 
shall not break existing things (at least if they do not use unusual 
syntax) ... and I never would like to add the udev complexity, which I 
never managed (due to no interest). ... and we can add an include option 
to mdev.conf, then you can split of that information into separate 
files, e.g. early mount file systems. Nevertheless collecting system 
depended information at some central places gives you a definitive 
starting point for looking at that kind of configuration (/etc/mdev.conf).


>   What we need is a configuration language for udev that's easy to
> understand
> (for a human) and fast to parse (for a machine). More thought needs to be
> poured into it - and it's in my plans, but not for the short term. What I'm
> sure of, though, is that the "fork a helper for every event" vs. "handle
> events in a unique long-lived program" debate is the wrong one - it's an
> implementation detail, that can be solved *after* proper udev language
> design.

IMO you are going off topic. Now you are talking about udev, and 
modifying / using udev. We are most likely looking at systems using mdev 
and are based on BB, not the upstream tools. If you want to use udev, 
use udev. My changes shall not block you on that, nor make your live 
more difficult. Beside that the syntax of mdev.conf is not so bad for 
the most kinds of operation, and may be we can easy some more 
operations. I'm open to ideas on this.

What kind of other function is required to let mdev setup the device 
file system? What operations are complicated and need complex helper 
scripts to get them up? What else should/could be setup in mdev.conf? 
Let us collect, then sort out those of interest.


Ok, from Nathaneals message and yours I got some hints and ideas, let me 
collect them. Looks like we can make things easier when splitting some 
things, that allows modular usage. With some (inline) wrapper logic we 
should be able to achieve compatibility to current usage.

Later more in a separate message ...

--
Harald