This is a status report for the Linux Driver Project as of June 2009, describing what has happened in the past year of work. It was originally posted on the Linux Driver Project developer mailing list.
posted Thu, 04 Jun 2009 in [/linux]
The 2.6.29 kernel release is now out, and if you look closely, there are a number of drivers for the Android platform merged in, starting with this patch.
Yes, there's still a lot to go, like cleaning up the user/kernel interfaces, and prodding the core Android developers to start working upstream more, which they have already started to do. It's a great start, and one that I hope will succeed overall, as it really is a nice system to develop for, and one the phone manufacturers have been asking for from the Linux community, for years.
And besides, how can you not like such a cute robot:
posted Tue, 24 Mar 2009 in [/linux]
It's been many months since the Linux Kernel developers conference, where the linux-staging tree was discussed and role changed. It turns out that people are still a bit confused as to what the staging tree is for, and how it works.
So here's a short summary, I'm not going into the history or background here, that's a much longer writeup that I'd be glad to do if people are interested.
The Linux Staging Tree, what it is and is not.
What the Linux Staging tree is
The Linux Staging tree (or just "staging" from now on) is used to hold stand-alone drivers and filesystems that are not ready to be merged into the main portion of the Linux kernel tree at this point in time for various technical reasons. It is contained within the main Linux kernel tree so that users can get access to the drivers much easier than before, and to provide a common place for the development to happen, resolving the "hundreds of different download sites" problem that most out-of-tree drivers have had in the past.
What the Linux Staging tree is not
The staging tree is not a place to dump code and run away, hoping that someone else will to the cleanup work for you. While there are developers available and willing to do this kind of work, you need to get them to agree to "babysit" the code in order for it to be accepted.
Location and Development
The staging tree is now contained within the main Linux kernel source tree at the location drivers/staging/. All development happens within the main kernel source tree, like any other subsystem within the kernel. This means:
- The linux-next tree contains the latest version of the staging tree, with bugfixes that are about to be merged into Linus's tree, as well as the patches that are to be merged into the next major kernel release.
- If you wish to do work on the staging tree, checkout the linux-next tree and send patches based on that.
Runtime
When code from the staging tree is loaded in the kernel, a warning message will be printed to the kernel log saying:
MODULE_NAME: module is from the staging directory, the quality is unknown, you have been warned.
and the kernel will be tainted with the TAINT_CRAP flag. This flag shows up in any kernel oops that might be produced after the driver has been loaded.
Note, most kernel developers have expressed the warning that they will not work on bugs for when this taint flag has happened, so if you run into a kernel problem after loading such a module, please work to reproduce the issue without a staging module loaded in order to be able to get help from the community.
If anyone has any questions that this summary doesn't answer, please let me know.
posted Wed, 18 Mar 2009 in [/linux]
A few years ago, when one of my kids was asked what their dad did for work, they replied, "Sit in the basement and write email."
It's not that far from the truth.
I looked back at my mail server logs for 2008 to see just how much email I did write. For 2008, it turned out that I wrote 19057 unique emails.
Yes, that's an average of about 52 emails a day, and no, that is not a number of individual recipients (one email sent to three people was counted as one email, not three.)
I do send out a lot of patches for review (for -stable trees, and for when patches get sent to Linus for inclusion in the main kernel tree), and I also send out a bunch of "Now that you got a patch into the Linux kernel, we were wondering who you work for", type emails to help lwn.net with their statistic gathering.
But that doesn't justify the numbers overall, I just think I have a bad habit.
Here's a breakdown of emails sent by the time of day for the whole year:
You can pretty clearly see when I go eat dinner and then switch over to using my laptop in the evening (time zone is local time of the day.)
When I tried to graph per day, you can kind of see lower numbers on the weekends if you squint:
I guess it's no wonder that Google thinks my email address is a spam-bot and refuses to let me sign up for any google groups with it. I think it is trying to tell me to cut down on my output or something.
Clearly I need help...
posted Fri, 16 Jan 2009 in [/linux]
A while ago I talked about piping all of my bash commands to twitter.com. I've kind of stopped doing that now, after I maxed out at over 14000 updates in about 2 weeks, but it was fun while it lasted.
But in order to do this kind of thing nicely, I ended up writing a command line program to make it easier. Some people have noticed it at times, by poking around in my kernel.org home directory, so I might as well announce the thing publicly.
So, consider this an announcement of the tool, bti. It allows you to send tweets to twitter.com or identi.ca directly from the command line from any Linux machine. It probably works on other systems as well, but you will have to tweak the Makefile yourself.
The latest version can always be found here.
The development for the tool is done in git, and the tree can be found on the ever-awesome github.com in this repository.
posted Wed, 22 Oct 2008 in [/linux]
The video for my keynote has been published on Google Video, hopefully the other talks get posted there soon as well.
posted Tue, 23 Sep 2008 in [/linux]
It was nice to see the large response from my Linux Plumbers Conference talk, and there seems to be a few common themes of questions about the talk that I figured I'd clear up here.
I have seen the video of the talk, and the video team from the Linux Plumbers Conference is working to put it up online somewhere, hopefully soon. I'll link to it when it is available.
First off, my numbers for the binutils development was completely WRONG. Kees and I sat down and tried to figure out exactly why I didn't count his valid contribution, and it turned out that binutils puts a ChangeLog into each subdirectory, the top-level one is not the summary of all of the individual parts of the project. So I apologize about that one, Canonical really did have one binutils patch in the past 3 years. Not that this really affects any of the main points of my talk though...
All of the other numbers for the other projects are still correct, from what I can tell. If anyone thinks I got them wrong, please let me know and I will be glad to review them. Feel free to review the changelog and svn and git trees of the different projects if want to verify.
One main question that I saw a lot, and was even asked about during my talk, was "what about Canonical's work on the desktop/Gnome/KDE"? I really don't know if they have contributed a lot of effort back upstream on these projects, that wasn't my point here.
Remember, this was given at the Linux Plumbers Conference a gathering of developers of the low-level plumbing of Linux. This wasn't a group of desktop developers, so remember the audience that this was addressed to please.
If Canonical has contributed a lot to Gnome/KDE, that's great, I'm sure someone will post the numbers soon to verify this. Either way, please remember that this was not the audience that I was addressing.
I sat down with Matt the day after my talk, as he described, and hopefully the Canonical kernel developers will work to become more of a valid part of the community, which is what I am sincerely hopeing will happen here.
Oh, and Amanda, I have given this very same kind of talk to Amazon, a number of months ago, as well as many other companies over the past 1 1/2 years, so it's not like I am ignoring them at all.
And this response brings me back to my main point of my talk, which most people seem to have missed as they were upset at me pointing out Canonical's lack of upstream contributions. And that point was, and still is:
The market right now is just too good for individual developers who have experience in writing open source software for Linux, especially the low-level plumbing of Linux, to waste their time working for companies who do not allow them to contribute back, if they want to.
This was a developer conference. I am a developer, talking as myself only, and not as a representative of any company (note the total lack of any corporate branding on my slides), to other developers who I totally respect and want to see be as happy in their day-job as I am in mine.
I would like to point out that lwn.net's summary of the talk did get this correct, which was great to see.
Hopefully this helps clear things up, if not, let me know and I'll be glad to address it.
posted Mon, 22 Sep 2008 in [/linux]
I was honored to give the opening keynote of the first Linux Plumbers Conference this year in Portland, Oregon. Here's the slides and text of my talk (well, the text is what I intended to say, the actual words that came out probably sounded a bit different.)
I'll comment later on a few things that I've noticed people bringing up, but I figured it would be good to get the text and slides out for everyone to be able to see first.
The talk was recorded, and I'll provide a link to it when it is available so you can compare it to what I have below.
If you want to link directly to this talk, please use this link.
Update: I've responded to some of the response about this talk here.
Update 2: The video has now been published on Google Video.
posted Wed, 17 Sep 2008 in [/linux]
As part of the Linux Foundation Technical board, we confront the issue of closed source Linux kernel modules all the time, and we wanted to do something that could be seen as a general "public statement" about them that is easy to understand and point to when people have questions.
So, after working on this for a while, and asking some of the other major contributors and maintainers of the kernel, what we have is below.
There is also a site that contains a link to a statement from the Linux Foundation about this topic, as well as some more descriptions and background information, and a copy of the full statement as well.
I've also put a pretty pdf version here in case people want to print it out.
If there are any kernel developers who want to add their names to this statement, please let me know by private email and I will be glad to add it.
posted Sun, 22 Jun 2008 in [/linux]
Ever since my talk at OLS last year about the Linux kernel development community and the companies involved, I've been traveling around, giving the talk in one form or another to lots of different companies and community groups.
Last week I gave the talk at Google, and they kindly recorded it and put it up for everyone to see.
So, if you're curious about the current state of the Linux kernel when it comes to how fast it is going, who is doing the work, who is sponsoring the work, and why that matters to your company, sit back and enjoy the talk.
Oh, the slides are right here if you really want to see them. Without the context of the talk they really don't mean that much, but people seem to always want to see them.
I also did an interview for linux-magazin.de a month or so ago, and that is also online now as well.
Maybe now I will no longer have to travel around so much...
posted Wed, 11 Jun 2008 in [/linux]
Yes, yet-another-linux-kernel-patch-set.
This one is for code that is good enough to build and run, but not good enough to get merged into the main kernel.org tree just yet.
See the announcement for more details if you are interested in adding patches to this tree, or in finding new kernel projects to work on.
posted Tue, 10 Jun 2008 in [/linux]
This is a status report for the Linux Driver Project as of April 2008, describing what has happened in the past year of work. It was originally posted on the developer mailing list.
posted Mon, 07 Apr 2008 in [/linux]
There was a lot of very good press coverage over my last announcement of the restart of the Linux Driver Project and my involvement in it now full time. It's been a few weeks since that announcement, and we now have over 300 different developers signed up to help create, and maintain Linux drivers!
I've also posted a short status report about the current projects, and what is going on with them. Since then, one more project has started, and there are a handful still in the planning stage.
What we need now is more companies participating in the project, we have the developers, but not enough work to keep them busy.
So how do we change this? I'm thinking that possibly, there really isn't a large number of different devices out there that need Linux support written for them.
As proof of this, I give you the Linux Foundation's Vendor Advisory Board. This group of companies publish a list of priorities that they feel need to be worked on in order to help Linux succeed.
Coming in at number 3 is "Device Driver Support". So, I approached this group and asked them specifically what devices did they see in common use that are not supported by Linux (the obvious 2 video cards being a known exception.) Despite this being such a high priority for this group, they had no examples to provide.
And neither do I. I don't currently know of any common piece of hardware in use today that is not supported on Linux. And since these vendors do not know, and I don't, I'm asking the world to help out.
So, please, let me know what specific type of device you know of that is not properly supported on Linux. If you want, please mark up the wiki page at:
Or just email me with your recommendations. If patterns emerge, I'll approach the companies and ask them if they will work with us.
Hopefully with your help, we can find some work for these 310 developers to do :)
posted Mon, 22 Oct 2007 in [/linux]
Way back in January, I announced a program to write Linux drivers for companies for free. When I did that, I never expected the response to be as large as it was.
It turns out that there were two large groups of people who responded to the announcement, companies wanting drivers, and developers wanting to help out.
I never imagined that so many different people would offer to help out. There is a real need for a place where developers can find a "real" project to work on in the Linux kernel. The Kernel Janitors project is a great place to start out, but what to do from there? It turns out that over 100 different developers offered up their services. Clearly this was a huge untapped group of talented people who wanted to help out.
Also, the number of companies expressing interest in this has exceeded all of my wildest expectations. Already this announcement has caused a number of drivers to end up in the main Linux kernel source tree, with more in the pipeline.
But unfortunately, I was not able to handle all of these different developers and company requests on my own. I had a full-time job, and a full part-time hobby doing Linux kernel development. These requests ended up going unanswered, and I sincerely apologize for that.
Now this has all changed.
My employer, Novell, has modified my position to now allow me to work full time on this project. Namely getting more new Linux kernel drivers written, for free, for any company that so desires. And to help manage all of the developers and project managers who want to help out.
We have kicked off the project again with this announcement on the mailing list set up for the developers wishing to help out.
If you want to join this group of people, or are a company wanting a free Linux driver written for them, please see our new web page at:
and follow the instructions there on how to join in.
I can't thank the people who have helped make this possible at Novell enough. They really care about helping make Linux support as many devices as possible, with fully opensource drivers.
Now let's get busy writing code...
posted Thu, 27 Sep 2007 in [/linux]
First off, I am a Novell employee, but do not speak for the company at all, the following is my personal opinion, but one that is held by a number of different people at different companies...
I've been hearing from a lot of different companies and users about how the current "enterprise" Linux distros manage their kernels and the current problems they are having with them. With all products that bundle up an every-moving target of an OpenSource projects, there are trade-offs and for the past five years or so the two big Linux distros, SUSE and Red Hat have been trying to walk the line between stability and new features and doing so quite well.
But now that we have been living with these models for a few years, a number of problems have come to light, with the way things currently work. This document describes how the current kernels are managed in these two distros, the problems companies and users are having with them, and three proposed solutions.
Current Model of kernels
An "enterprise" Linux kernel as packaged by Red Hat and SUSE is made up of a kernel.org kernel taken from a specific point in time, and then it is tested, tuned, stabilized and then packaged up and supported for usually a long period of time (5-9 years, depending on the product.) After the product is first released, a series of "service packs" or "maintenance updates" (from now on called "major update") are released, about one every 8-18 months depending on the vendor and the length of time the product has been released.
During the time between these major updates, the distro works to ensure that any bugfix or security update that happens will not break any of the current kernel API or internally visible ABI. It does this so that any third-party kernel modules will not need to change or be "recertified" because of them.
Currently both distros provide a way to have third-party module creators register with them and be notified when an internal ABI change is going to happen in order for them to be able to respin their pre-built modules. They also provide ways for third-party modules to be easily created and build against these kernels so as to ease the build issues that can be caused when attempting to build and package against a distro kernel.
When a major update is released, it usually consists of the original kernel version for the product, with a wide range of new features, drivers, and other fixes backported from the currently released kernel.org kernel to the originally released kernel. This enables new hardware to be supported and new features that are requested by the distro's users and partners to be rolled into the product. This release almost always has ABI and API changes due to the new features and drivers.
At the time of this newly released update, all third party modules must be rebuilt and sometimes reworked due to the new features and backports that were applied to the kernel. Hopefully all third-party module vendors work with the distro to be aware of the proposed changes, but sometimes there is a lag before the new modules are available for the customers to use.
An example of how this works can be seen in the latest Novell SLES10 Service Pack 1 release. Originally the SLES10 kernel was based on the 2.6.16 kernel release with a number of bugfixes added to it. At the time of the Service Pack 1 release, it was still based on the 2.6.16 kernel version, but the SCSI core, libata core, and all SATA drivers were backported from the 2.6.20 kernel.org kernel release to be included in this 2.6.16 based kernel package. This changed a number of ABI issues for any external SCSI or storage driver that they would need to be aware of when producing an updated version of their driver for the Service Pack 1 release.
Problems caused by this model
Here are some of the problems that I have heard with customers and partners who have been working with this kind of release model for a number of years:
- Partners have to get their changes, features, and new drivers into the upstream kernel.org kernel in order for the distros to be willing to accept them. After that happens, they must then backport the feature/driver to the older vendor specific kernel, test it to verify everything works properly, and then ask the vendor to accept the patch for their next service pack, by the deadline imposed by the vendor. This causes a lot of extra work by the partners, having to track at least two vendor kernels, as well as the upstream kernel.org tree.
- If the partner misses the release date for the next service pack, their hardware will not be supported within the whole product until 12-18 months later, when the next service pack is released.
- Partners hate working with externally available drivers, through a driver-disk or some other process. Reasons cited for their dislike of this range from confusion of users for how to get access to these drivers, to security issues, to issues surrounding drivers for boot devices or network devices when doing network installs.
- Due to the changing API between each service pack, third-party vendors need to create a different module build for every major product release (RHEL3, RHEL4, etc) as well as for every maintenance update. Because some customers do not upgrade to the latest maintenance update for various reasons, they are forced to support their driver on all maintenance releases, a very large combination.
- It imposes the old Unix slow release cycle on to Linux, cutting off one of the main reasons people switch to Linux in the first place.
- For machines that must work with new hardware all the time (laptops and some desktops), the 12-18 month cycle before adding new device support makes them pretty much impossible to use at times. (i.e. people want you to support the latest toy they just bought from the store.) This makes things like "enterprise" kernels that are directed toward desktops quite uncomfortable to use after even a single year has passed.
Potential solutions and pros and cons of them
Keep doing the same as before. This model has evolved over the years to the current state based on input from partners, users, developers and the companies.
- Pro: People know the model and are used to it.
- Con: See the above listed reasons why partners and third-party vendors dislike it.
Change the major update to only include kernel bugfixes and security fixes. No new features or drivers will be added to the kernel, helping to ensure that the ABI does not change. If an API has to change due to a more intrusive bugfix, this might still happen at this time, but the change is limited to only a specific area, involving a minor number of symbols and structures, instead of large sweeping change like currently happens.
- Pro: Partners will not have to worry about the release cycle of the distro kernels, and only focus on getting their drivers working with a set kernel version, not multiple versions over the lifetime of the product.
- Con: Many new features will not be able to be added in this manner, as they touch core portions of the kernel and can not be updated with external modules (ACPI and the large PCI quirk table for misbehaving hardware are two such examples.)
On every major update, the kernel is updated to the latest kernel.org release, much like the consumer products are (Fedora, openSUSE, Ubuntu, Mandriva, etc.) This will ensure that any upstream update for drivers and new features will be automatically included.
- Pro: All of the latest kernel drivers and features will be automatically supported and included by the distro, enabling the Partners to focus on upstream kernel.org development and not worry about backporting things to older kernel versions. All bugfixes and security updates that the vendor has not included in their minor updates are also pulled in at this time (and there are a lot of them.)
- Con: Partners whose code is not present in kernel.org releases for whatever reason (do not want it, incompatible licenses, etc.) will have to do a bit more work in tracking the new releases, although this should be only be slightly more than the current amount of development and testing that they currently do.
So, what to do? Currently this discussion is happening in the major distros and their partners. Let me know what you think the model should be for enterprise Linux distros in the future.
Do you like one of the currently proposed models above, or do you have some other model you feel would work better?
Thanks to the many different people who read earlier versions of this document and helped to form the ideas here. This includes the whole Novell/SuSE kernel team who while did not all agree with the ideas here, helped out immensely. Also a number of people at different companies helped out, but probably do not wish to be named here, I want to thank them anyway.
posted Tue, 19 Jun 2007 in [/linux]
My Linux Stuff
RSS