linux kernel monkey log

Position Statement on Linux Kernel Modules

Mon, 23 Jun 2008 04:59:00 GMT

As part of the Linux Foundation Technical board, we confront the issue of closed source Linux kernel modules all the time, and we wanted to do something that could be seen as a general "public statement" about them that is easy to understand and point to when people have questions.

So, after working on this for a while, and asking some of the other major contributors and maintainers of the kernel, what we have is below.

There is also a site that contains a link to a statement from the Linux Foundation about this topic, as well as some more descriptions and background information, and a copy of the full statement as well.

I've also put a pretty pdf version here in case people want to print it out.

If there are any kernel developers who want to add their names to this statement, please let me know by private email and I will be glad to add it.

Linux Plumbers Conference 2008

Thu, 19 Jun 2008 00:29:00 GMT

The Linux Plumbers Conference has announced that registration is now open, and the call for papers has also gone out.

This conference was created by a bunch of Linux people living in Portland, Oregon with the goal of having a technical conference in the US that deals with the low-level "plumbing" issues relating to the whole Linux system. This includes the kernel, udev, HAL, dbus, xorg, pulse audio, and other related things.

It's a non-profit conference, with all of the money raised for it from registration fees and sponserships going directly into the conference itself to try to provide a good experience.

I'm running one of the "microconference" tracks dealing with the fun around the Linux kernel/userspace interface issues. If you are interested in presenting a talk about this issue, be sure to let me know.

Linux kernel development talk

Wed, 11 Jun 2008 23:31:00 GMT

Ever since my talk at OLS last year about the Linux kernel development community and the companies involved, I've been traveling around, giving the talk in one form or another to lots of different companies and community groups.

Last week I gave the talk at Google, and they kindly recorded it and put it up for everyone to see.

So, if you're curious about the current state of the Linux kernel when it comes to how fast it is going, who is doing the work, who is sponsoring the work, and why that matters to your company, sit back and enjoy the talk.

Oh, the slides are right here if you really want to see them. Without the context of the talk they really don't mean that much, but people seem to always want to see them.

I also did an interview for linux-magazin.de a month or so ago, and that is also online now as well.

Maybe now I will no longer have to travel around so much...

linux-staging kernel tree

Tue, 10 Jun 2008 20:16:00 GMT

Yes, yet-another-linux-kernel-patch-set.

This one is for code that is good enough to build and run, but not good enough to get merged into the main kernel.org tree just yet.

See the announcement for more details if you are interested in adding patches to this tree, or in finding new kernel projects to work on.

what I am doing, RIGHT NOW

Wed, 14 May 2008 21:14:00 GMT

I've been watching twitter for a while now, amused at the ability for it to keep people appraised of what you are doing at the moment, if they really care. I didn't think it was really worth it.

Until I read this post last night which was linked off of some site that I forgot (probably reddit but I did think it was from the every wonderful Arachaia, which if you are a programmer, you should be paying attention to.)

I just couldn't resist...

So, if you want to see what I am doing, RIGHT NOW (well what I just did, it waits for the command to complete before sending it off to twitter), you can follow along right here.

I'm only enabling it on a few of my terminal windows for now, watching me constantly run mutt and offlineimap would get a bit boring.

I wonder how long it's going to be before I type in my password accidentally to this thing. Or until twitter bans me. Any odds on which is going to happen first?

I pity anyone who subscribes to this twit feed, they are going to start hating me very quickly, like the Portland, Oregon local feed already has...

Linux Driver Project Status Report as of April 2008

Mon, 07 Apr 2008 21:52:00 GMT

This is a status report for the Linux Driver Project as of April 2008, describing what has happened in the past year of work. It was originally posted on the developer mailing list.

Who wrote Linux update

Tue, 01 Apr 2008 23:23:00 GMT

Back in July last year, I wrote a paper for the Linux Symposium about who was doing the actual work in the Linux kernel. Today the Linux Foundation has released a new version of it that contains new data, as well as a lot better writing thanks to Jon Corbet of lwn.net and Amanda McPherson of the Linux Foundation.

The paper was released in both html form, and a PDF version for those who like the pretty graphs to be a bit bigger.

It seems that the trade press has picked this up already, with reviews by the 451 group, internetnews.com, cnet, and infoworld, as well as lwn.net.

Funny note, companies are now emailing me complaining that we aren't counting their contributions properly. Hey, the numbers don't lie, take a look at the tools and logs I used to create all of this if you don't believe them.

To be fair to one company, Google, we were incorrectly counting their representation, keeping Andrew Morton in the "Linux Foundation" bucket instead of the "Google" bucket. That will change the list of top companies placing Google somewhere between 10 and 13, I haven't re-run the numbers yet to get the exact placement.

The poster travels on...

Sun, 20 Jan 2008 06:21:00 GMT

I've been carting the "big wall of kernel developers" poster around the world with me for the past 6 months, getting it signed by as many kernel developers as I could find.

First off was the Ottawa Linux Symposium, where the poster was unleashed apon an unsuspecting crowd of people. They mostly just laughed at it, and I had to update a few names by hand for some late patches that slipped in:

At that conference, which had around 600 attendees, I collected 101 different signatures. That ment that 1 out of every 6 people attending that conference, had gotten a patch into the last kernel version that had just come out (2.6.22 at the time.) That's quite a high concentration all in one spot.

I then displayed it at 2007 OSCON. While a fun conference, it is not very kernel oriented at all, and so, I picked up a few more signatures, mostly from Ubuntu people who happened to be attending due to a recent conference.

After that, it was a jump over the pond to the 2007 Linux kernel summit in Cambridge. I collected a few more signatures there, after having to clear off one whole wall of the conference room, much to the hotel's dismay.

Then, it was a few more plane rides, having it get lost by the airline between London and Hamburg, a train ride to Nuremburg, and then a bus ride to the Czech countryside for a SuSE Labs conference. Once again, the hotel staff looked at me strange, but they eventually found a way to set it up, and a few more signatures were aquired.

In the end, I collected 165 signatures, and the poster traveled with me to five different countries.

But today, I finally said goodbye to it. I sent it off in the mail to linux.conf.au, where it is to be raffled off for charity, after collecting a few more signatures that it is missing.

If you see it, say hello, for after lugging it on more airlines that I want to recall, and having to explain it to more airport security people than should have been necessary, it feels strange to not have to tote it after me anymore.

Maybe I'll go create a new poster for all of the work that happened in 2007 in the kernel, that one should only run about 50 meters long and give me something new to lug around the world again...

Devices Lacking Linux Support Needed

Tue, 23 Oct 2007 03:54:00 GMT

There was a lot of very good press coverage over my last announcement of the restart of the Linux Driver Project and my involvement in it now full time. It's been a few weeks since that announcement, and we now have over 300 different developers signed up to help create, and maintain Linux drivers!

I've also posted a short status report about the current projects, and what is going on with them. Since then, one more project has started, and there are a handful still in the planning stage.

What we need now is more companies participating in the project, we have the developers, but not enough work to keep them busy.

So how do we change this? I'm thinking that possibly, there really isn't a large number of different devices out there that need Linux support written for them.

As proof of this, I give you the Linux Foundation's Vendor Advisory Board. This group of companies publish a list of priorities that they feel need to be worked on in order to help Linux succeed.

Coming in at number 3 is "Device Driver Support". So, I approached this group and asked them specifically what devices did they see in common use that are not supported by Linux (the obvious 2 video cards being a known exception.) Despite this being such a high priority for this group, they had no examples to provide.

And neither do I. I don't currently know of any common piece of hardware in use today that is not supported on Linux. And since these vendors do not know, and I don't, I'm asking the world to help out.

So, please, let me know what specific type of device you know of that is not properly supported on Linux. If you want, please mark up the wiki page at:

http://linuxdriverproject.org/twiki/bin/view/Main/DriversNeeded

Or just email me with your recommendations. If patterns emerge, I'll approach the companies and ask them if they will work with us.

Hopefully with your help, we can find some work for these 310 developers to do :)

Linux Driver Project

Thu, 27 Sep 2007 22:51:00 GMT

Way back in January, I announced a program to write Linux drivers for companies for free. When I did that, I never expected the response to be as large as it was.

It turns out that there were two large groups of people who responded to the announcement, companies wanting drivers, and developers wanting to help out.

I never imagined that so many different people would offer to help out. There is a real need for a place where developers can find a "real" project to work on in the Linux kernel. The Kernel Janitors project is a great place to start out, but what to do from there? It turns out that over 100 different developers offered up their services. Clearly this was a huge untapped group of talented people who wanted to help out.

Also, the number of companies expressing interest in this has exceeded all of my wildest expectations. Already this announcement has caused a number of drivers to end up in the main Linux kernel source tree, with more in the pipeline.

But unfortunately, I was not able to handle all of these different developers and company requests on my own. I had a full-time job, and a full part-time hobby doing Linux kernel development. These requests ended up going unanswered, and I sincerely apologize for that.

Now this has all changed.

My employer, Novell, has modified my position to now allow me to work full time on this project. Namely getting more new Linux kernel drivers written, for free, for any company that so desires. And to help manage all of the developers and project managers who want to help out.

We have kicked off the project again with this announcement on the mailing list set up for the developers wishing to help out.

If you want to join this group of people, or are a company wanting a free Linux driver written for them, please see our new web page at:

http://www.linuxdriverproject.org/

and follow the instructions there on how to join in.

I can't thank the people who have helped make this possible at Novell enough. They really care about helping make Linux support as many devices as possible, with fully opensource drivers.

Now let's get busy writing code...

Linux Symposium 2007 wrap up

Thu, 12 Jul 2007 05:47:00 GMT

Just a few random things I wanted to get out about this year's Linux Symposium.

My big "wall of kernel developers" turned out very well, with lots of different people taking pictures of it. I wanted to get everyone who had a patch to sign the thing, and I counted 105 signatures at the end of the week. So, since the attendance was somewhere around 700 people or so, 1/7 of the attendees had gotten a kernel patch into the Linux kernel within the past 2 months.

That is a pretty concentrated number of kernel developers for any type of conference.

I'll be dragging the poster around to different places through the year (OSCON is next), and eventually take it, or send it to linux.conf.au where it will be raffled off for charity.

If you are somewhere you think a lot of kernel developers are going to be, and want me to send you the poster so you can get their signatures, just let me know.

My talk went really well, despite being misquoted about the fact the size of the Linux kernel is growing both with number of individual developers as well as the overall code size. My paper explaining all of this can be found here, and the slides I used for the talk can be found here for people who want to see the real numbers and information provided.

The scripts used to create this information can be found in the paper, and I'll be updating them later in the week, along with the scripts I used to create the big posters too.

Papers you should read if you have not already:

Len Brown, ACPI in Linux -- Myths vs. Reality
Zachary A. Brown, Asynchronous system calls
Venkatesh Pallipadi and Adam Belay, cpuidle - Do nothing, efficiently...
Inaky Perez-Gonzalez, My bandwidth is wider than yours: Ultrawideband, Wireless USB and WiNET in Linux
Arnaldo Carvalho de Melo, The 7 dwarves: debugging information beyond gdb
Steven Rostedt, Internals of the RT Patch

That last one about RT makes me not so afraid of the whole -rt kernel patchset and it even looks kind of fun to me now...

Heck, go read them all, they are all great papers and you will learn something by just browsing through them.

Overall, the conference was great. I had a blast and James's final keynote was great, talking about evolutionary theory, obsolete computer architectures, and how to embrace the myrad of forks we all generate everyday within the Linux communities. What more could anyone want out of a week?

Quote of the party

Mon, 02 Jul 2007 03:47:00 GMT

Spoken at the final party at this years Linux Symposium by a kernel developer working for a Linux distribution that will not be named:

So the problems started happening when we added the 7th wireless stack to the kernel...

40 feet of kernel developers

Tue, 26 Jun 2007 06:16:00 GMT

So Pete thinks I just do "big announcements" here, so I'll try to stop that and do more frequent little things.

I just got the printed out version of the 2.6.22 kernel developer chart from the printer before flying out to the Linux Symposium and it's amazingly large in person:

Over 40 feet long, hopefully I can find a wall that long for people to be able to look at the thing...

I also uploaded the PDF files if others want to try their hand at printing this monstrosity out.

Inkscape Rocks

Sat, 23 Jun 2007 09:27:00 GMT

I've been working on the graphics for my Linux Symposium talk and it turns out that the graph of Linux kernel developers and their relationship is a bit larger and more complex than I originally guessed.

I wanted to create a single big poster that showed the whole Signed-off-by: path for the past year or so, but when I finally got it rendered it turned out to be over 165 feet long by 3 feet tall...

Needless to say, that's a bit impractical, so I'm having to resort to graphing each individual release, but they are still turning out quite large, over 40 feet long for some of these.

I've put up some sample renderings if people want to take a look at them:

Be forwarned that they are very large and might crash your browser.

Also ran into the problem that the local Kinkos (an American copy/printing company) can not handle images this big. The PDF version of the file crashes Acrobat Reader, even the professional version, and Illustrator can not handle single images this large. Somehow I need to figure out how to get these banners printed out in time for my plane ride to Ottawa next week.

After spending all week using Inkscape to create these images all I have to say is "it rocks!". I used to use Illustrator a lot a long time ago (the result of a number of college classes in graphic design) and Inkscape does all that, and handles images this large in size with absolutely no problems.

Future of Enterprise Linux kernels

Tue, 19 Jun 2007 23:51:00 GMT

First off, I am a Novell employee, but do not speak for the company at all, the following is my personal opinion, but one that is held by a number of different people at different companies...

I've been hearing from a lot of different companies and users about how the current "enterprise" Linux distros manage their kernels and the current problems they are having with them. With all products that bundle up an every-moving target of an OpenSource projects, there are trade-offs and for the past five years or so the two big Linux distros, SUSE and Red Hat have been trying to walk the line between stability and new features and doing so quite well.

But now that we have been living with these models for a few years, a number of problems have come to light, with the way things currently work. This document describes how the current kernels are managed in these two distros, the problems companies and users are having with them, and three proposed solutions.

Current Model of kernels

An "enterprise" Linux kernel as packaged by Red Hat and SUSE is made up of a kernel.org kernel taken from a specific point in time, and then it is tested, tuned, stabilized and then packaged up and supported for usually a long period of time (5-9 years, depending on the product.) After the product is first released, a series of "service packs" or "maintenance updates" (from now on called "major update") are released, about one every 8-18 months depending on the vendor and the length of time the product has been released.

During the time between these major updates, the distro works to ensure that any bugfix or security update that happens will not break any of the current kernel API or internally visible ABI. It does this so that any third-party kernel modules will not need to change or be "recertified" because of them.

Currently both distros provide a way to have third-party module creators register with them and be notified when an internal ABI change is going to happen in order for them to be able to respin their pre-built modules. They also provide ways for third-party modules to be easily created and build against these kernels so as to ease the build issues that can be caused when attempting to build and package against a distro kernel.

When a major update is released, it usually consists of the original kernel version for the product, with a wide range of new features, drivers, and other fixes backported from the currently released kernel.org kernel to the originally released kernel. This enables new hardware to be supported and new features that are requested by the distro's users and partners to be rolled into the product. This release almost always has ABI and API changes due to the new features and drivers.

At the time of this newly released update, all third party modules must be rebuilt and sometimes reworked due to the new features and backports that were applied to the kernel. Hopefully all third-party module vendors work with the distro to be aware of the proposed changes, but sometimes there is a lag before the new modules are available for the customers to use.

An example of how this works can be seen in the latest Novell SLES10 Service Pack 1 release. Originally the SLES10 kernel was based on the 2.6.16 kernel release with a number of bugfixes added to it. At the time of the Service Pack 1 release, it was still based on the 2.6.16 kernel version, but the SCSI core, libata core, and all SATA drivers were backported from the 2.6.20 kernel.org kernel release to be included in this 2.6.16 based kernel package. This changed a number of ABI issues for any external SCSI or storage driver that they would need to be aware of when producing an updated version of their driver for the Service Pack 1 release.

Problems caused by this model

Here are some of the problems that I have heard with customers and partners who have been working with this kind of release model for a number of years:

Partners have to get their changes, features, and new drivers into the upstream kernel.org kernel in order for the distros to be willing to accept them. After that happens, they must then backport the feature/driver to the older vendor specific kernel, test it to verify everything works properly, and then ask the vendor to accept the patch for their next service pack, by the deadline imposed by the vendor. This causes a lot of extra work by the partners, having to track at least two vendor kernels, as well as the upstream kernel.org tree.
If the partner misses the release date for the next service pack, their hardware will not be supported within the whole product until 12-18 months later, when the next service pack is released.
Partners hate working with externally available drivers, through a driver-disk or some other process. Reasons cited for their dislike of this range from confusion of users for how to get access to these drivers, to security issues, to issues surrounding drivers for boot devices or network devices when doing network installs.
Due to the changing API between each service pack, third-party vendors need to create a different module build for every major product release (RHEL3, RHEL4, etc) as well as for every maintenance update. Because some customers do not upgrade to the latest maintenance update for various reasons, they are forced to support their driver on all maintenance releases, a very large combination.
It imposes the old Unix slow release cycle on to Linux, cutting off one of the main reasons people switch to Linux in the first place.
For machines that must work with new hardware all the time (laptops and some desktops), the 12-18 month cycle before adding new device support makes them pretty much impossible to use at times. (i.e. people want you to support the latest toy they just bought from the store.) This makes things like "enterprise" kernels that are directed toward desktops quite uncomfortable to use after even a single year has passed.

Potential solutions and pros and cons of them

Keep doing the same as before. This model has evolved over the years to the current state based on input from partners, users, developers and the companies.
- Pro: People know the model and are used to it.
- Con: See the above listed reasons why partners and third-party vendors dislike it.
Change the major update to only include kernel bugfixes and security fixes. No new features or drivers will be added to the kernel, helping to ensure that the ABI does not change. If an API has to change due to a more intrusive bugfix, this might still happen at this time, but the change is limited to only a specific area, involving a minor number of symbols and structures, instead of large sweeping change like currently happens.
- Pro: Partners will not have to worry about the release cycle of the distro kernels, and only focus on getting their drivers working with a set kernel version, not multiple versions over the lifetime of the product.
- Con: Many new features will not be able to be added in this manner, as they touch core portions of the kernel and can not be updated with external modules (ACPI and the large PCI quirk table for misbehaving hardware are two such examples.)
On every major update, the kernel is updated to the latest kernel.org release, much like the consumer products are (Fedora, openSUSE, Ubuntu, Mandriva, etc.) This will ensure that any upstream update for drivers and new features will be automatically included.
- Pro: All of the latest kernel drivers and features will be automatically supported and included by the distro, enabling the Partners to focus on upstream kernel.org development and not worry about backporting things to older kernel versions. All bugfixes and security updates that the vendor has not included in their minor updates are also pulled in at this time (and there are a lot of them.)
- Con: Partners whose code is not present in kernel.org releases for whatever reason (do not want it, incompatible licenses, etc.) will have to do a bit more work in tracking the new releases, although this should be only be slightly more than the current amount of development and testing that they currently do.

So, what to do? Currently this discussion is happening in the major distros and their partners. Let me know what you think the model should be for enterprise Linux distros in the future.

Do you like one of the currently proposed models above, or do you have some other model you feel would work better?

Thanks to the many different people who read earlier versions of this document and helped to form the ideas here. This includes the whole Novell/SuSE kernel team who while did not all agree with the ideas here, helped out immensely. Also a number of people at different companies helped out, but probably do not wish to be named here, I want to thank them anyway.