As part of the Linux Foundation Technical board, we confront the issue of closed source Linux kernel modules all the time, and we wanted to do something that could be seen as a general "public statement" about them that is easy to understand and point to when people have questions.
So, after working on this for a while, and asking some of the other major contributors and maintainers of the kernel, what we have is below.
There is also a site that contains a link to a statement from the Linux Foundation about this topic, as well as some more descriptions and background information, and a copy of the full statement as well.
I've also put a pretty pdf version here in case people want to print it out.
If there are any kernel developers who want to add their names to this statement, please let me know by private email and I will be glad to add it.
posted Sun, 22 Jun 2008 in [/linux]
Ever since my talk at OLS last year about the Linux kernel development community and the companies involved, I've been traveling around, giving the talk in one form or another to lots of different companies and community groups.
Last week I gave the talk at Google, and they kindly recorded it and put it up for everyone to see.
So, if you're curious about the current state of the Linux kernel when it comes to how fast it is going, who is doing the work, who is sponsoring the work, and why that matters to your company, sit back and enjoy the talk.
Oh, the slides are right here if you really want to see them. Without the context of the talk they really don't mean that much, but people seem to always want to see them.
I also did an interview for linux-magazin.de a month or so ago, and that is also online now as well.
Maybe now I will no longer have to travel around so much...
posted Wed, 11 Jun 2008 in [/linux]
Yes, yet-another-linux-kernel-patch-set.
This one is for code that is good enough to build and run, but not good enough to get merged into the main kernel.org tree just yet.
See the announcement for more details if you are interested in adding patches to this tree, or in finding new kernel projects to work on.
posted Tue, 10 Jun 2008 in [/linux]
This is a status report for the Linux Driver Project as of April 2008, describing what has happened in the past year of work. It was originally posted on the developer mailing list.
posted Mon, 07 Apr 2008 in [/linux]
There was a lot of very good press coverage over my last announcement of the restart of the Linux Driver Project and my involvement in it now full time. It's been a few weeks since that announcement, and we now have over 300 different developers signed up to help create, and maintain Linux drivers!
I've also posted a short status report about the current projects, and what is going on with them. Since then, one more project has started, and there are a handful still in the planning stage.
What we need now is more companies participating in the project, we have the developers, but not enough work to keep them busy.
So how do we change this? I'm thinking that possibly, there really isn't a large number of different devices out there that need Linux support written for them.
As proof of this, I give you the Linux Foundation's Vendor Advisory Board. This group of companies publish a list of priorities that they feel need to be worked on in order to help Linux succeed.
Coming in at number 3 is "Device Driver Support". So, I approached this group and asked them specifically what devices did they see in common use that are not supported by Linux (the obvious 2 video cards being a known exception.) Despite this being such a high priority for this group, they had no examples to provide.
And neither do I. I don't currently know of any common piece of hardware in use today that is not supported on Linux. And since these vendors do not know, and I don't, I'm asking the world to help out.
So, please, let me know what specific type of device you know of that is not properly supported on Linux. If you want, please mark up the wiki page at:
Or just email me with your recommendations. If patterns emerge, I'll approach the companies and ask them if they will work with us.
Hopefully with your help, we can find some work for these 310 developers to do :)
posted Mon, 22 Oct 2007 in [/linux]
Way back in January, I announced a program to write Linux drivers for companies for free. When I did that, I never expected the response to be as large as it was.
It turns out that there were two large groups of people who responded to the announcement, companies wanting drivers, and developers wanting to help out.
I never imagined that so many different people would offer to help out. There is a real need for a place where developers can find a "real" project to work on in the Linux kernel. The Kernel Janitors project is a great place to start out, but what to do from there? It turns out that over 100 different developers offered up their services. Clearly this was a huge untapped group of talented people who wanted to help out.
Also, the number of companies expressing interest in this has exceeded all of my wildest expectations. Already this announcement has caused a number of drivers to end up in the main Linux kernel source tree, with more in the pipeline.
But unfortunately, I was not able to handle all of these different developers and company requests on my own. I had a full-time job, and a full part-time hobby doing Linux kernel development. These requests ended up going unanswered, and I sincerely apologize for that.
Now this has all changed.
My employer, Novell, has modified my position to now allow me to work full time on this project. Namely getting more new Linux kernel drivers written, for free, for any company that so desires. And to help manage all of the developers and project managers who want to help out.
We have kicked off the project again with this announcement on the mailing list set up for the developers wishing to help out.
If you want to join this group of people, or are a company wanting a free Linux driver written for them, please see our new web page at:
and follow the instructions there on how to join in.
I can't thank the people who have helped make this possible at Novell enough. They really care about helping make Linux support as many devices as possible, with fully opensource drivers.
Now let's get busy writing code...
posted Thu, 27 Sep 2007 in [/linux]
First off, I am a Novell employee, but do not speak for the company at all, the following is my personal opinion, but one that is held by a number of different people at different companies...
I've been hearing from a lot of different companies and users about how the current "enterprise" Linux distros manage their kernels and the current problems they are having with them. With all products that bundle up an every-moving target of an OpenSource projects, there are trade-offs and for the past five years or so the two big Linux distros, SUSE and Red Hat have been trying to walk the line between stability and new features and doing so quite well.
But now that we have been living with these models for a few years, a number of problems have come to light, with the way things currently work. This document describes how the current kernels are managed in these two distros, the problems companies and users are having with them, and three proposed solutions.
Current Model of kernels
An "enterprise" Linux kernel as packaged by Red Hat and SUSE is made up of a kernel.org kernel taken from a specific point in time, and then it is tested, tuned, stabilized and then packaged up and supported for usually a long period of time (5-9 years, depending on the product.) After the product is first released, a series of "service packs" or "maintenance updates" (from now on called "major update") are released, about one every 8-18 months depending on the vendor and the length of time the product has been released.
During the time between these major updates, the distro works to ensure that any bugfix or security update that happens will not break any of the current kernel API or internally visible ABI. It does this so that any third-party kernel modules will not need to change or be "recertified" because of them.
Currently both distros provide a way to have third-party module creators register with them and be notified when an internal ABI change is going to happen in order for them to be able to respin their pre-built modules. They also provide ways for third-party modules to be easily created and build against these kernels so as to ease the build issues that can be caused when attempting to build and package against a distro kernel.
When a major update is released, it usually consists of the original kernel version for the product, with a wide range of new features, drivers, and other fixes backported from the currently released kernel.org kernel to the originally released kernel. This enables new hardware to be supported and new features that are requested by the distro's users and partners to be rolled into the product. This release almost always has ABI and API changes due to the new features and drivers.
At the time of this newly released update, all third party modules must be rebuilt and sometimes reworked due to the new features and backports that were applied to the kernel. Hopefully all third-party module vendors work with the distro to be aware of the proposed changes, but sometimes there is a lag before the new modules are available for the customers to use.
An example of how this works can be seen in the latest Novell SLES10 Service Pack 1 release. Originally the SLES10 kernel was based on the 2.6.16 kernel release with a number of bugfixes added to it. At the time of the Service Pack 1 release, it was still based on the 2.6.16 kernel version, but the SCSI core, libata core, and all SATA drivers were backported from the 2.6.20 kernel.org kernel release to be included in this 2.6.16 based kernel package. This changed a number of ABI issues for any external SCSI or storage driver that they would need to be aware of when producing an updated version of their driver for the Service Pack 1 release.
Problems caused by this model
Here are some of the problems that I have heard with customers and partners who have been working with this kind of release model for a number of years:
- Partners have to get their changes, features, and new drivers into the upstream kernel.org kernel in order for the distros to be willing to accept them. After that happens, they must then backport the feature/driver to the older vendor specific kernel, test it to verify everything works properly, and then ask the vendor to accept the patch for their next service pack, by the deadline imposed by the vendor. This causes a lot of extra work by the partners, having to track at least two vendor kernels, as well as the upstream kernel.org tree.
- If the partner misses the release date for the next service pack, their hardware will not be supported within the whole product until 12-18 months later, when the next service pack is released.
- Partners hate working with externally available drivers, through a driver-disk or some other process. Reasons cited for their dislike of this range from confusion of users for how to get access to these drivers, to security issues, to issues surrounding drivers for boot devices or network devices when doing network installs.
- Due to the changing API between each service pack, third-party vendors need to create a different module build for every major product release (RHEL3, RHEL4, etc) as well as for every maintenance update. Because some customers do not upgrade to the latest maintenance update for various reasons, they are forced to support their driver on all maintenance releases, a very large combination.
- It imposes the old Unix slow release cycle on to Linux, cutting off one of the main reasons people switch to Linux in the first place.
- For machines that must work with new hardware all the time (laptops and some desktops), the 12-18 month cycle before adding new device support makes them pretty much impossible to use at times. (i.e. people want you to support the latest toy they just bought from the store.) This makes things like "enterprise" kernels that are directed toward desktops quite uncomfortable to use after even a single year has passed.
Potential solutions and pros and cons of them
Keep doing the same as before. This model has evolved over the years to the current state based on input from partners, users, developers and the companies.
- Pro: People know the model and are used to it.
- Con: See the above listed reasons why partners and third-party vendors dislike it.
Change the major update to only include kernel bugfixes and security fixes. No new features or drivers will be added to the kernel, helping to ensure that the ABI does not change. If an API has to change due to a more intrusive bugfix, this might still happen at this time, but the change is limited to only a specific area, involving a minor number of symbols and structures, instead of large sweeping change like currently happens.
- Pro: Partners will not have to worry about the release cycle of the distro kernels, and only focus on getting their drivers working with a set kernel version, not multiple versions over the lifetime of the product.
- Con: Many new features will not be able to be added in this manner, as they touch core portions of the kernel and can not be updated with external modules (ACPI and the large PCI quirk table for misbehaving hardware are two such examples.)
On every major update, the kernel is updated to the latest kernel.org release, much like the consumer products are (Fedora, openSUSE, Ubuntu, Mandriva, etc.) This will ensure that any upstream update for drivers and new features will be automatically included.
- Pro: All of the latest kernel drivers and features will be automatically supported and included by the distro, enabling the Partners to focus on upstream kernel.org development and not worry about backporting things to older kernel versions. All bugfixes and security updates that the vendor has not included in their minor updates are also pulled in at this time (and there are a lot of them.)
- Con: Partners whose code is not present in kernel.org releases for whatever reason (do not want it, incompatible licenses, etc.) will have to do a bit more work in tracking the new releases, although this should be only be slightly more than the current amount of development and testing that they currently do.
So, what to do? Currently this discussion is happening in the major distros and their partners. Let me know what you think the model should be for enterprise Linux distros in the future.
Do you like one of the currently proposed models above, or do you have some other model you feel would work better?
Thanks to the many different people who read earlier versions of this document and helped to form the ideas here. This includes the whole Novell/SuSE kernel team who while did not all agree with the ideas here, helped out immensely. Also a number of people at different companies helped out, but probably do not wish to be named here, I want to thank them anyway.
posted Tue, 19 Jun 2007 in [/linux]
In talking with a lot of different companies recently, I've come to the realization that we really need to do something about companies that violate the kernel's GPLv2 license. It has been a common criticism that "Well, our company abides by the GPL by releasing the code properly for our kernel modules, but what about all of those other companies that do not?" The companies that are good members of the community are getting a lot of pressure by people internal to them to stop releasing the code. This is justified by pointing to the companies that do not release their code as they are not having any "penalties" by doing this.
Previously the kernel community has ignored almost all violators (with the obvious exception of the gpl-violations.org project). The kernel changes so rapidly that it is quite expensive for a company to try to keep up without releasing their changes and becoming part of the community. This has been done by the constant API changes and other rapid evolution that dictates Linux kernel development. However, if a company throws a bunch of money and developers at the problem, it is pretty trivial to keep up with this development model.
Another big problem is the fact that the kernel is created and copyrighted by thousands of individual developers and companies. Companies are usually not eager to sue other companies as they most likely already have business agreements with them. So that leaves the individual developers, but it's hard for an individual to do anything on their own against companies who violate their code.
So, what can the community do to handle these problems? Here's a few ideas that I've come up with:
- continue as before, ignoring violators. But if we do this, more companies will see this as an implicit acceptance of the closed source modules and start to do it more and more.
- Go after companies with legal action. This is good, but it's pretty "mean" and doesn't do much to try to help convert companies who could learn to become good members of the community. Remember, we need their help, and antagonizing them isn't the best way to accomplish this.
- Ask companies who are friendly toward Linux to apply business pressure against these other companies to get them to change their ways.
- Quietly take legal action against companies who violate the license. This has been the way the FSF has done things in the past, and for some instances this does work. But it does not give any publicity for the other offending companies to provide a disincentive.
- Do big public notices of the offending companies. This will work in some cases, but others, not at all.
The best thing to do is probably a mixture of the above, but I really don't have any solid answers as to how to achieve some of them very well. Taking legal action against another company takes time and resources that individual developers do not usually have. Perhaps a combination of these things is probably the best thing to do.
Anyone have any other ideas?
Or how about incentives for companies who do abide by the license and are good community members. What can we do for them that will help make other companies realize the benefits of doing the same? It's usually easier to convince companies of a tangible benefit, rather than just the risks of incurring legal action.
Anyone have any good ideas about how we can help these companies more?
We need to do something, as I don't think that the current status is workable over the long term.
Thanks to everyone who read and provided feedback to early versions of this document. This includes the whole SuSE kernel development team (not all of whom agree with this document, but they provided their input anyway), and members of other companies that probably do not want to be listed here...
posted Wed, 28 Mar 2007 in [/linux]
Due to all of the responses I've gotten for the Linux driver announcement I have gotten a lot of questions asked to me about this. Here is a list of some of the most common ones, and their answers.
Again, if anyone has questions about this program, or wishes to take advantage of it, please feel free to contact me.
Q: You forgot to mention the most obvious benefit to the company, "They might sell more of their devices!" How can they turn down an offer like that?
A: Good point :)
Q: How are you going to write a GPL driver by signing an NDA? Is it going to require a binary blob or some other way of obfuscating the code?
A: No, not at all. I have written many drivers after signing NDAs with companies. They are usually signed either to keep information about the device private until it is announced at a specific date, or to just keep the actual specification documents from being released to the public directly. All code created by this NDA program is to be released under the GPL for inclusion in the main kernel tree, nothing will be obfuscated at all.
Q: This is a lame publicity stunt, Linux development has always been done this way.
A: Well, the NDA program that we have set up with The Linux Foundation is new. But yes, other than that, this is exactly how Linux kernel development has been done. But it is good to point out exactly how it all works for those who are not familiar with how it works.
Q: You are putting the people who do this kind of development for a living out of a job, stop that!
A: This is just not true at all. In fact a number of people who are consultants doing this kind of development have contacted me thanking me for bringing the issue more publicity. They know that some companies really want to pay people to do development and support in order to achieve contractual issues and have some one on the hook for providing support in ways that the community can not guarantee.
Q: Are companies really going to do this?
A: Yes, already we have received a number of serious queries from companies about producing Linux drivers for their devices. More information will be available later when details are firmed up.
Q: Can you write a driver for my [insert device name here] to get it to work? It isn't made anymore and no one has the specs for it.
A: Sorry, but this project is for devices in which we have the specification and hopefully the manufacturer's support. We don't have the time or effort that is needed to reverse engineer the device on our own, sorry.
Q: Do you need developer help for this project?
A: Yes, just drop me an email and I'll add you to the list of people who have volunteered to help out. We are always glad to accept help.
Q: Are companies actually taking you up on this offer?
A: Yes, the initial response to this was amazing, a measurable number of new Linux drivers will be created thanks to this program.
Q: What about helping to get out-of-the-tree GPL drivers into the main kernel tree?
A: A number of people have offered to help out with this in the past, and all code is welcome to Linux. But we aren't going to be pulling in code to the main kernel tree without the permission of the authors.
Q: What about the BSDs?
A: What about them? They are free to do whatever they wish, I have no input into their development at all, sorry.
posted Fri, 09 Feb 2007 in [/linux]
Yes, that's right, the Linux kernel community is offering all companies free Linux driver development. No longer do you have to suffer through all of the different examples in the Linux Device Driver Kit, or pick through the thousands of example drivers in the Linux kernel source tree trying to determine which one is the closest to what you need to do.
All that is needed is some kind of specification that describes how your device works, or the email address of an engineer that is willing to answer questions every once in a while. A few sample devices might be good to have so that debugging doesn't have to be done by email, but if necessary, that can be done.
In return, you will receive a complete and working Linux driver that is added to the main Linux kernel source tree. The driver will be written by some of the members of the Linux kernel developer community (over 1500 strong and growing). This driver will then be automatically included in all Linux distributions, including the "enterprise" ones. It will be automatically kept up to date and working through all Linux kernel API changes. This driver will work with all of the different CPU types supported by Linux, the largest number of CPU types supported by any operating system ever before in the history of computing.
As for support, the driver will be supported through email by the original developers, when they can help out, and by the "enterprise" Linux distributors as part of their service agreements with their customers.
If your company is worried about NDA issues surrounding your device's specifications, we have arranged a program with OSDL/TLF's Tech Board to provide the legal framework where a company can interact with a member of the kernel community in order to properly assure that all needed NDA requirements are fulfilled.
Now your developers will have more time to work on drivers for all of the other operating systems out there, and you can add "supported on Linux" to your product's marketing material.
This offer is in effect for all different types of devices, from USB toys to PCI video devices to high-speed networking cards. If you manufacture it, we can get Linux drivers working for it.
For any questions about this program, please feel free to respond to this email, or contact me directly at greg@kroah.com. I will also be available at FreedomHEC 2007 held adjacent to WinHEC, if anyone wants to bring devices and work face-to-face.
posted Mon, 29 Jan 2007 in [/linux]
The book I was working on for most of 2006 is now finally showing up in bookstores so it's time to announce it here.
Linux Kernel in a Nutshell is now availble online, for free, from the kernel.org world-wide mirror systems (so that I really have no idea how many people download it, and that's fine with me.)
To quote the book's official site:
If you want to know how to build, configure, and install a custom Linux kernel on your machine, buy this book. It is written by someone who spends every day building, configuring, and installing custom kernels as part of the development process of this fun, collaborative project called Linux.
I'm especially proud of the chapter on how to figure out how to configure a custom kernel based on the hardware running on your machine. This is an essential task for anyone wanting to wring out the best possible speed and control of your hardware.
Not satisified with just offering up electonic PDF versions that mirror the book as it was exactly printed, I have availble the DocBook source that is heavily marked up by some tools that was used to generate the PDF files from.
Also, as a first in the history of publishing books (that I know of), the entire revision history of the book is availble as a git archive so you can see how slowly the process went, how bad my writing really is before the excellent editors at O'Reilly get ahold of it, and how much the Technical Reviewers for the book really helped me out in making this something that everyone will want to have on their bookshelf.
It weighs in at a slim 175 pages, which is amazing as both the editor and myself thought it would be about 350 pages before the layout people attacked it with a bunch of small pointy fonts, saving a few forests in their glee to pack as much information as possible on every printed page.
And yes, to answer all of the questions that people have asked when reading the introduction, the book did start out at over 1,000 pages, consisting of nothing other than a description of every kernel configuration option possible (see the git tree if you don't believe me). Luckily, cooler heads prevailed, and a whole bunch of useful content was written by me to make this something that would acually sell.
What are you waiting for? Go buy a copy or if you are cheap, just download the thing and start building kernels!
What next? No more books for a while, that's for sure. But before quiting the writing gig, I was lucky to write a chapter for a special compilation project from O'Reilly that has turned into an amazing book that I feel honored to have been able to contribute to. More on that in a few months when it gets closer to publication date.
posted Tue, 09 Jan 2007 in [/linux]
A large number of people have expressed interest recently in the userspace i/o driver core which allows userspace drivers to be written to handle some types of hardware.
Right now the UIO core is working and in the [-mm kernel releases][mm]. It's been rewritten from the last time patches were posted to lkml and is much simpler. It also includes full documentation and two example drivers and two example userspace programs that test those drivers.
This core allows for a very tiny kernel driver to be written to handle the interrupt generated by the hardware. Everything else can be done in userspace (direct memory access, interrupt processing, controller logic, etc.) In some instances, this framework has shown a noticable improvement over an all-in-kernel driver.
But in order to get this core into the kernel tree, we need to have some "real" drivers written that use it. So, for anyone that wants to see this go into the tree, now is the time to step forward and post your patches for hardware that this kind of driver interface is needed.
If no such drivers appear, then there is a very slim chance that this interface will be accepted into the tree.
The patches can be found in the -mm releases or at:
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio.patch - UIO core
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-documentation.patch - UIO documentation
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-dummy.patch http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/uio-irq.patch - two example kernel modules and userspace programs showing how to use the UIO interface.
posted Wed, 13 Dec 2006 in [/linux]
The whole Linux ecosystem is quite strange for people who are used to the way "traditional" companies operate. Here's a short primer for those who are need a reminder.
In a "traditional" technology-based company, you have the executives making the deep strategic decisions, the managers carrying out those decisions and telling the engineers what to develop (yeah, there's marketing and sales in there too, but for this essay, let's just ignore them for now.) Normally the executives and managers and engineers are focused on creating the "next best thing" in their industry, and are directly competing with other companies in the same industry.
And this works well, everyone is used to it, and whole major portions of our world's economy depend on this model.
Then there's Linux.
And by Linux, I'm going to focus on the kernel here, but you can extrapolate the idea to any part of the Linux distribution ecosystem too, it all pretty much works the same, in one way or the other.
In Linux, the rules are different, as well as the way the whole system works.
For Linux, the "product" is created by engineers working in the community. These community members get together and work for a common project, making it the best that they know how. These engineers are a very large group, coming from a huge number of different companies and backgrounds.
Companies have realized that Linux is a good thing to use, and have tried to figure out how to make money bundling it up into a semi-stable release and supporting it.
These companies have executives, managers, and engineers much like a traditional technology company has. But here's where things start to work differently. These executives still make deep strategic decisions, the managers work to carry out those decisions and tell the engineers what to develop, but the engineers have to work with the community in order to achieve those goals. These engineers rely on the community to be able to create something that the whole company can then bundle up and make money off of.
Because of this reliance on the community, companies tend to fall into two broad categories (yes, there are more categories, but for now, let's keep things simple.):
- ignore the community. These companies merely bundle up whatever the community creates and sells that. As such, they do not have much influence on the product and become minor players in the market.
- embrace the community. These companies hire members of the community or allow their employees to join the community, as they realize that it is the only way to drive the direction of Linux in ways that they feel it should go. Because of this, the company is able to provide things to their customers a little bit ahead of when the companies in the first category can, and they can support their customers much better as the community members are able to directly help them.
Companies in the first category, can pretty much act however they like, their executives can make deals with whomever, and they can try to be a direct competitor with other companies, and no one in the community will really care, as they are bit players and don't really matter.
Companies in the second category, must pay attention to the community. Their executives can't go around doing things that are not in the community's best interest, their managers can't try to directly compete with their competitors, and their engineers can't ignore the community's wishes or opinions. If any of these things happen, the company will soon be ignored by the community, forced to implement all of the great things the executives and managers dream up on their own, and slowly end up becoming a member of the first category of companies, relegated to being a bit player, as the community, and the companies that are real members of the community, pass them by and go off on their own way.
History is already littered with the remnants of companies who originally started out embracing the community, but then make the mistake that they think for some reason they are a "traditional" technology company.
It will be interesting to see how well the current companies in the Linux ecosystem really understand how different their business environment really is.
Note, all ideas that you might have about what real-life company falls into which category, or why this essay was ever written in the first place, are all in your own mind...
posted Fri, 24 Nov 2006 in [/linux]
A number of Linux kernel developers got together and made a unscientific survey of what they felt about the current version of the GPLv3. This information was sent to the different companies that are on the Committee B of the GPLv3 review process. The results were posted today on the linux kernel mailing list for everyone else who might be interested in seeing them.
Also, a smaller number of us got together and wrote a position paper on why we feel the currently drafted version of the GPLv3 is such a bad thing right now for both the Linux kernel, and all current Linux distros. That paper was also posted here, if anyone is interested in seeing it.
posted Fri, 22 Sep 2006 in [/linux]
The organizers of OLS were kind enough to allow me the opportunity to speak this year as the closing keynote speaker. Here's the slides and text of my talk (well, the text is what I intended to say, the actual words that came out probably sounded a bit different.)
If you want to link directly to this talk, please use this link.
posted Sun, 23 Jul 2006 in [/linux]
My Linux Stuff
RSS