Pseudo trip report for the Linux power management summit, Linux kernel summit, and Ottawa Linux Symposium.

Power Management Summit

This year, before the annual Linux kernel summit, we held a small "Power Management Summit" (to quote Pat, "All the other cool kids are having a summit, why not us too?") This turned to out work very well, as every year for the past 4 years, most of us have gotten together at OLS for a few hours and talked over the pressing power management issues. Then, after we go our separate ways, we quickly forget what we agreed upon, and almost nothing gets done. This year we were determined to not duplicate this, and so, we sat in a room for 8 hours and hashed everything out. The important thing is a few of us took notes, which got merged together by Pat Mochel into a summary of what happened, and got posted to lwn.net. Now that we all agree on what needs to be done with power management, we somehow need to figure out how to get the time to do the work. Hopefully some of the affected companies will be able to help us out here.

Also, it looks like we intend to have another one of these meetings, in about 6 months, and let some vendor specific people attend. I know Nokia and Palm want to attend, as they were at OLS and talked to us about the issues they are facing with regards to getting Linux working on small devices, with minimal power consumption.

Kernel Summit

The lwn.net writeups of what happened here are the best place for someone to get a good overview I gave a quick "5 minute" talk in which I asked two different things:

"Does anyone in this room, other than Andrew, have any objection to me removing devfs from the kernel tree in the near future."

No one objected (Andrew's worry is that people are relying on it, but aren't speaking up. Hopefully the 2.6.13 release will flush these people out of the woodwork, as that kernel just disables the config option for devfs, but the code is still present.)

Earlier in the day, Kay gave me a shirt that looked like this:

Only in a crowd like this could I wear it and constantly generate laughs when people read it. I'm hanging it up on the wall in my office to keep.

My other question to the crowd was,

"Does anyone have any remaining issues with the current Linux driver model code?" I got 4 general comments:

• Subclasses, or some type of thing like this. I talked a lot with the frame buffer and DRM developers, and they really need something like this in order to represent their interaction with each other and their devices properly. Also, the input subsystem needs this, and in talking with Marcel Holtmann, bluetooth could use it also. So, I'll be working on solving this in the next month or so.
• Documentation. Yes, the documentation in the kernel tree is very out of date, I'll be fixing that up, and hopefully moving a lot of the content that is in LDD3 into there, to help out with this.
• Debugging ability. It turns out that sysfs and the driver core are implemented in a lot of kernel oopses, as they are the last thing that is called usually by broken drivers or driver subsystems when they are buggy. This usually isn't a bug in the driver core or sysfs itself, just in how it is being used, but it would be very nice to help debug some of these issues better. There are two kernel config options that I always use to help with this, CONFIG_DEBUG_KOBJECT and CONFIG_DEBUG_DRIVER. Enable those and you will get a lot of output to the syslog, which will usually show you where the true problem is.
• Flexibility in the api. Right now, it's too easy to do things wrong in the driver core. We need to make it harder to get wrong, and easier to get right. That's just a matter of slowly refactoring the interfaces over time, nothing earth-shattering here.

So, only 4 objections, and nothing major at all. Afterward, a kernel developer said to me

"Man, they are all such liars! And chicken! When you confront them and ask for comments they just shut up. But to hear them complain on IRC and on LKML, you would think that the driver model was the worst thing in the kernel and should be instantly ripped out."

This was my intention of getting up in front of everyone, now no one can ever complain about the driver core and sysfs again.

The hardware vendors panel was quite good this year, not at all what I expected. One thing that I noted, that was not in the lwn.net write up, was both Emulex and Qlogic complaining about having to maintain at least 3 different versions of their drivers, one for SLES, one for RHEL, and one for mainline. They really only want to maintain one version, mainline, if they can. This seems in direct opposite of what they tell us (SuSE) about not changing the SLES api at all to prevent breaking their driver. I wonder how long it will take for them to realize they are the ones creating the issue...

The format of the Kernel summit seems to be moving toward more of the "the subsystems have a meeting and report back to everyone". Networking and Power Management did this and this trend will probably continue, as trying to have a good discussion with 100 people all at once, is a pretty lousy way to reach any kind of conclusion.

This was proved in the very last of the talks, "The kernel development process." It started late, and ran out of time almost instantly, yet it was a very important topic for everyone in the room. Next year I hope it will happen on the first day, and that there is a moderator for the topic, as people seem to get off-topic and drift very easily. Nothing really groundbreaking came out of this topic, other than we might be only using git to send patches before -rc1, and after that only patches in email. And that all subsystem maintainers need to do their big merges within a few days of a main kernel release, before -rc1, so that -rc1 really is a "Release Candidate" kernel. Hopefully this will work out...

Ottawa Linux Symposium

As always, general coverage of the conference can be found at lwn.net and other news sites (newsforge had a good series of articles.) I'll just touch on stuff that pertains to me.

Once again, it is very good to meet all of the different kernel and other developers in person. Lots of good development and conversations happen in the hallways and on the couches in the general area.

During the opening presentation from Jon Corbet, it was nice to see that people are caring about the -stable kernel releases. I know that Chris and I don't really get much feedback from them, except when we take a patch that someone doesn't agree with...

My "Write a Real, Working Linux driver" tutorial was much more popular than I expected. I had over 80 people reserve a spot by email, and lots more attended in person. I gave the tutorial twice in the same day, in only 2 hours (I thought I had 3 hours.) It was very rushed, but in the end, I think almost everyone left with a working driver and a device. I really want to thank all of the other kernel people who helped me out with the presentation, taking the time to individually help out different people when they got stuck: Kay Sievers, Pat Mochel, Chris Wright, Matt Mackall, and probably lots of others that I can't remember at the moment. I've posted the slides for the presentation here and the example code and handouts I gave out in the talk here.

In talking with others afterward, everyone was of the impression that this was the first time something like this had ever happened, a tutorial that created something real, for the Linux kernel. If this is true, it's about damm time. The demand was definitly there, with more people wanting to attend than I had room. Odds are I can probably take this on the road and give it to every conference out there with no problem. No reason others couldn't do the same thing, if they do so, it would greatly increase the number of kernel developers in the world, which is a good thing.

One word of warning for others who want to give the same talk/tutorial multiple times in the same day, make some kind of list to follow, otherwise you will think you have already said something in the second presentation, when really you only said it the first time around. Makes for some confusing moments... Good thing was, the second group got the benefit of the first group debugging the driver for them. The number of kernel oopses in the second group was much less than the first one.

The "Persistant(sic) Device Naming" BOF went very well, with about 100 people attending. It was basically a presentation from Kay Sievers showing udev and HAL working in real-time handing a very nasty chain of USB devices containing a bunch of usb-storage and card reader devices on a USB hub. The speed at which all of the devices were discovered, recognized by the kernel, and then properly named in a persistent way was amazing. The card reader also handled removal and insertion of media from it, destroying and creating the proper device nodes properly (thanks to HAL which creates a thread for every removable device, just like other operating systems do to handle devices that can't detect media changes.)

The work Kay and David have been doing on udev and HAL in the past few months is absolutely wonderful. Everyone at the presentation were told that this was going to be the persistent naming solution for Red Hat, SuSE, and Gentoo. The other distros are free to pick it up if they wish, but we are not going to be messing around with any nasty spec from any other organizations, as that would just be pointless. Later some Debian developers said they would also add it to their tree, so that covers about 90% of the Linux distros in the market. What else is there left to say then :)

For examples of how this is all going to work, look at the Gentoo unstable tree in the next few months, and the next Red Hat and SuSE releases. All of this will be implemented there, and I'll go into details later, if it's not obvious.

I will not say much about the Linux Security Modules BOF, other than I think a few people misunderstood one of the comments I made there. Chris Wright was trying to get across the point that we need to get more "real" users of the LSM interface into the kernel, and soon. Otherwise, we have no real justification of the interface. He also stated that we need to have real users of the current hooks in the kernel tree, otherwise they will be removed. My comment was to back up this last statement, that the hooks will be going away unless they are used. It was in no way meant to say that the whole LSM interface will be going away soon. However, if we don't get any real users in the main kernel tree soon, this will be a harder and harder stance to justify to the other kernel developers. It has been over 2 years now since LSM is in the mainline, what's taking people so long...

The final keynote from Dave Jones was quite good, if not a real downer. From his viewpoint, we aren't handling the number of bugs that are getting added to the kernel. Due to the increased speed at which we are adding new code, this is probably quite true. And his main point that we need better tools to handle all of this is very valid.

What I did take great exception to is the number of people who, in the question period, stated, "just write a regression test for the drivers." I've heard this numerous times on the linux-kernel mailing list too. It's laughable at why people, who have no idea of how drivers or the kernel really work, feel they can give this answer up as a potential solution for the problem. I personally know one of the people who said that during the question period, who really should know better, as I tried to educate him as to the true nature of drivers during my period at IBM. {sigh} Oh well, some people never learn.

I guess, as I'm going to be giving the keynote next year at OLS, I should just talk about "Why all drivers suck" or something, to make a nice continuum from this year's presentation.

Actually, I'm looking forward to the keynote in some perverse way. Every year, for the past 3 years or so, I've submitted at least two different things to OLS, one "safe" talk, and one "fun" one, that I really want to give instead. Every year the "fun" one has been turned down, and the "safe" one accepted instead. Come on, you don't really think that my 2004 talk about struct kref was my first choice do you? Now, there's no one to tell me "no" for whatever topic I pick...

posted Fri, 29 Jul 2005 in [/linux]