linux kernel monkey log

Android and the Linux kernel community

Wed, 03 Feb 2010 00:18:00 GMT

As the Android kernel code is now gone from the Linux kernel, as of the 2.6.33 kernel release, I'm starting to get a lot of questions about what happened, and what to do next with regards to Android. So here's my opinion on the whole matter...

Stable kernel tree status, January 18, 2010

Mon, 18 Jan 2010 19:11:00 GMT

Here's the state of the -stable kernel trees, as of January 18, 2010.

2.6.27-stable

The 2.6.27-stable kernel tree is still living on, as a "long-term" stable release. But, I do have to warn users of this tree, the older it gets, the less viable it becomes. Not all bugfixes are being backported to this kernel version due to massive code changes in the over 2 years since this kernel has been released. I am doing my best to backport fixes that I become aware of, and I encourage anyone who does fix any types of bugs in the main kernel tree to let me know if the change should be applied to this older kernel version.

I'll probably keep maintaining it for at least 6-8 more months, but after that, I can not guarantee it's viability. Note, one other developer has volunteered to pick up the tree after I am finished with it, but I can not speak for him at this time.

2.6.31-stable

Today the last 2.6.31-stable kernel was released, all users of this kernel series are strongly encouraged to switch to the 2.6.32 kernel series, as there will not be any more updates for this branch in the future.

2.6.32-stable

I'd like to announce that the 2.6.32-stable tree is also going to be maintained as a "long-term" stable release, living for 2-3 years, like the 2.6.27 kernel is. This is because a number (i.e. more than 2) Linux distributions are basing their "enterprise" releases on this kernel version, and it will make their lives easier if I keep it alive.

Note, the viability of me keeping this tree alive for such a length of time relies on the developers working for those distros to keep me informed of patches that need to be backported and applied to it. Without their help, I will have no problem in stopping the maintenance of the tree.

Submitting patches for stable trees

Again, the easiest way to get your patch into a -stable tree is to merely add the line:

Cc: stable <stable@kernel.org>

to the Signed-off-by: area of your patch. When the patch goes into Linus's tree, it will be automatically sent to the stable address, and I will know to apply it to the trees. If I have any problem applying it at that time, I will email the author and reviewers of the patch about it.

If you forgot to add this line to the patch, or you have found a patch written by someone else that you wish to have applied to the stable trees, email the git commit id of the patch as it shows up in Linus's tree to the stable@kernel.org email address. Any stable correspondence sent to my personal accounts has the chance of being lost in the shuffle, so please try to not do that.

If a patch needs to be backported to one of the stable trees because it does not apply directly, please send the backported patch, along with the git commit id of the original patch, to the stable@kernel.org address, with a description of which kernel tree it should be applied to.

If anyone has any other questions about stable releases, please let me know.

Twitter experiment over

Wed, 06 Jan 2010 00:52:00 GMT

Way back in 2008 I started to use twitter as a dump of my command line. Well, 23 thousand tweets later, most of them being something as boring as q ref, I've stopped the experiment. Turns out just seeing one side of the conversation (what I type) and not the output, is a pretty boring thing.

That being said, 155 people did find it amusing enough to follow, and for that, I am totally amazed. True, 150 of them are probably bots, I sure hope their scripts enjoyed the show. Twitter also started to rate limit me after the first few months, in a silent manner. The server would say it accepted the message, yet it would never show up anywhere.

The good thing out of this, was the tool, bti, which lots of people seem to be using as it provides an easy way to pipe output to twitter-like services. It has also grown way beyond my initial expectations, now providing the ability to read messages, and lots of other things (readline? Yeah, readline, the GPL trojen horse, is now supported, I need to work on making it so that readline doesn't force a GPLv3 change to the codebase.)

So, it's back to normal for twitter and identi.ca for me now, just kernel release announcements and complaints about companies not working well with the kernel community.

Staging tree status for the .33 kernel merge

Wed, 23 Dec 2009 01:01:00 GMT

This was originally sent to the linux-kernel and driver-devel mailing lists. Might as well post it here to get a wider audience as the last report was received well.

Here's a summary of the state of the drivers/staging/ tree, basically what will be coming in the 2.6.33 merge, and what the status of the different drivers are so far.

Sorry it's late (after the merge), but hey, better late than never.

Again, drivers/staging/ is NOT a dumping ground for dead code. If no one steps up to maintain and work to get the code merged into the main portion of the kernel, the drivers will be removed.

Also, drivers can now be merged from mainling into the staging directory, providing a path out of the kernel for some obsolete and/or broken drivers.

So, here's some drivers that will be removed in the 2.6.33 kernel:

android drivers. Google and no one else stepped up to maintain them, so they will be dropped. So sad...
dst The developer isn't working on this anymore and recommended that it be removed as no one is using it.

Here is some new drivers that will show up in .33:

arlan, netwave, strip, wavelan - wireless drivers that are on their way out of the kernel. If anyone is actually using this old, obsolete hardware, speak up soon, otherwise they will be removed in a few kernel releases.
ramzswap - a compressed ram driver
rtl8192u - yet another wireless driver
samsung-laptop - laptop for the N128 Samsung laptop
batman-adv - a network protocol
dt3155 - a frame grabber driver
sm7xx - another frame buffer driver

Here's the list of drivers that have had work done on them that will show up in the .33 release:

comedi - lots of development effort happened here, mostly all cleanups, but there are some logic changes. More is needed, and it's moving along nicely.
line6 - lots of work happening, very nice to see
rt* - loads of cleanups and other merges. Will be obsolete soon due to a "real" wireless driver being worked on, but it's still nice to have these be a working alternative until then.
rtl* - more wireless driver work, horrible code, but it seems to work for the users. Hopefully more development time can be spent here in the new year.
dream - here's the platform specific code for the Android G1 platform. This might be the way the android code sneaks back into the kernel, as there is developers trying to get this to work. Of course, it's all happening without Google's help. {sigh}
et131x - loads of cleanups, more left to do. Good solid progress happening here.
iio - a new driver added to this subsystem, along with other fixes and cleanups. Looking nice.
poch - still some work happening, nice to see it pick back up.
panel - minor cleanups.
vme - cleanups and minor tweaks, still alive and kicking
vt66* - more wireless drivers, will be obsoleted by a "real" driver again.
wlags49 - more cleanup work as well.

Here's some drivers not listed so far, that have had work done recently, after the 2.6.33-rc1 merge happened, so it has to wait for the .34 kernel release:

wlan-ng
slicoss
mimo
asus_oled
udlfb
w35und

Hm, so, what's up with all of the other staging drivers, and why have they not had any development? What is the status of them? They are now on the short list to be deleted in 2 kernel releases, unless some real development happens on them.

This means, unless someone steps up and starts doing real work (not trivial spelling fixes) on the following drivers, they will be removed in the future kernel releases.

arlan, netwave, strip, wavelan - wireless drivers mentioned above that are on the way out. Slated for removal in 2.6.35
hv - Microsoft Hyper V drivers. The developers again seem to have disappeared, this is getting old. Slated for removal in 2.6.35
p9auth - this will be removed in .34 unless someone steps up.
frontier - slated for removal in .35. Will be easy for someone to pick up if they want to (hint, hint, hint)
altpciechdma - this will be removed in .34 unless someone steps up.
b3dfg - this will be removed in .34 unless someone steps up.
pohmelfs - filesystem under development out of tree, would be nice to get the patches merged back into here
quatechusb2 and serqtusb2 - usb to serial drivers that need to get merged into mainline.
rar and sep - Intel needs to step up here and get this code cleaned up properly, or it too will be removed in .34

Again, if someone is looking for some kernel development work to do, picking any of the above drivers up to get them merged into mainline, would be a great thing to do.

And, on a final, and sadder note, I'd like to announce the failure of the driver project to complete a driver for a company that requested it. This was totally my fault, and I would publically like to apologize about my lack of getting a SCSI driver written in time for a company that had asked for it. So, while the Linux driver project is doing great work for a large number of companies, every once in a while we do fail, we are only human.

How I apply patches to the stable tree

Tue, 15 Dec 2009 19:08:00 GMT

I decided to do a screencast of how I apply patches to the Linux stable tree to give people an idea of my patch workflow.

The video is here:

HOWTO - apply a Linux kernel patch to the stable tree.

If anyone has any questions about it, let me know. If there is interest in how I do a stable kernel release from these patches, or anything else, let me know.

How Software is Built Interview

Thu, 26 Nov 2009 04:44:00 GMT

As others have noticed, I was interviewed by the team at How Software is Built a few weeks ago, and the interview is now posted here.

It was a fun interview, and a long one as well, and I think it turned out very good. If you are interested in how Linux is developed, or the history of the Linux Driver Project, check it out.

And while you are there, take some time to read the other excellent interviews that they have done in the past. I personally liked Dirk Hohndel's discussion about Moblin and Linux and Alexey Rusakov talk about Russia and Free Software as well as ALT-Linux, but there are many, many, more there. Highly recommended.

LinuxCon 2009 tutorial

Fri, 11 Sep 2009 20:48:00 GMT

Somehow I got convinced to give a tutorial at LinuxCon this year, and it was originally scheduled to be my normal "Write a Real, Working, Linux Driver" tutorial I've been giving for the past 4 years or so (which happens to be online here, if you are bored and need something to fall asleep to.)

But that's old-hat, as people on 4 major continents have seen it before. So, to try to break up the boredom, I'm please to announce a change:

Write and Submit Your First Kernel Patch

This tutorial will cover the steps necessary to properly compose, describe, and submit a Linux kernel patch. It will cover the basic usage of git, and how that works with the Linux kernel development cycle. As part of the tutorial, every attendee will compose and submit a patch to the Linux kernel that will be included in the main kernel tree.

Every attendee should have a solid grasp of the C language, and know how to build and install, a Linux kernel from scratch (if not, reading the book, Linux Kernel in a Nutshell, free online, ahead of time would be a very good idea.) The latest source tree, from the git repository, of the Linux kernel should be installed on every attendees laptop before they arrive.

Sign up on the tutorial web site if you are going to attend so I get a clue how many people to expect. Right now I have unique material for 100 people to write new patches for, but can come up with more if needed.

See you all at LinuxCon, should be a fun time. I'm also giving a few other talks there as well, so come and heckle.

Staging tree status for the .32 kernel merge

Wed, 09 Sep 2009 21:48:00 GMT

This was originally sent to the linux-kernel and driver-devel mailing lists. Based on some of the feedback I got, I figured I should post it here as well.

Here's a summary of the state of the drivers/staging/ tree, basically what will be coming in the 2.6.32 merge, and what the status of the different drivers are so far.

First off, drivers/staging/ is NOT a dumping ground for dead code. If no one steps up to maintain and work to get the code merged into the main portion of the kernel, the drivers will be removed.

As proof of that, the epl (Ethernet Power Link) driver will be removed in the .32 kernel, as no one is working on it, the upstream developers never respond to my emails, and no one seems to care about it.

The pata_rdc driver is also going to be removed, as there is a "better" one being merged through the libata tree for this hardware.

So, taking the drivers in chunks, here's some that have had active development on for the .32 release:

rt* wireless drivers. Bart has done amazing work merging all of these together into something much better than they originally were. And even better, they still work! Great job Bart!
rtl* wireless drivers. Again, Bart has been doing great work here.
wlan-ng driver: a bit of work here, but this seems to be dropping off, with the loss of a test platform for the driver. Hopefully someone has a device around and can help out here.
comedi drivers had only a bit of work done, lots more is needed here, let's not loose the fact that this is getting closer to a mergable shape.
android drivers have had a bit of work done, but upstream seems to not care at all about what is going on here, as they are working to forward port their code to the 2.6.29! kernel. {sigh}. If this keeps up, the drivers will be dropped in the 2.6.32 kernel release. Note, Pavel has been adding some of the Dream hardware drivers, which are separate from the core Android drivers. I have no objection to those, but they should work to get merged to their "correct" places in the tree in another release or so.
w35und driver. It's slowly being worked on.
echo driver. This one is now in good enough shape to merge into the main kernel tree. I'll send out review patches soon for this.
eth131x driver. Alan Cox is working on fixing up the issues in this driver. Hopefully it will get into mergable shape soon.

New drivers that will show up in the .32 kernel release:

vt66* wireless drivers. These VIA drivers are being actively worked on to get into a much better shape. Nice job.
new rt3090, and rtl8192e wireless drivers have been added and worked on cleaning up issues involved in them.
hv (Microsoft Hyper-V) drivers. Over 200 patches make up the massive cleanup effort needed to just get this code into a semi-sane kernel coding style (someone owes me a bit bottle of rum for that work!) Unfortunately the Microsoft developers seem to have disappeared, and no one is answering my emails. If they do not show back up to claim this driver soon, it will be removed in the 2.6.33 release. So sad...
quatech_usb2 driver. I don't know if it quite works, but someone is developing it, so I'm not complaining :)
VME bus drivers. Yeah! They are progressing nicely.
SEP and RAR drivers. Alan Cox has been working on cleaning these up a lot.
IIO (Industrial I/O), these are new drivers that are being actively worked on.
pohmelfs and dst. It seems that DST is dead, so I think I will remove it in .33. pohmelfs is under active development outside of the tree, and hopefully patches start moving in here to help out with keeping it up to date.
cowloop. Yes, another COW driver! :) Seriously, this does things that DM can't do, so it might be useful. The upstream developer is interested in getting this merged properly, and is working on cleaning it up.

Drivers not being actively worked on:

otus This is sitting here until a "real" wireless driver will be merged through the wireless tree. Hopefully that happens soon.
agnx wireless driver. No one seems to care about it. If no one steps up soon, it will be removed in .33.
altpciechdma Upstream developers seem to have disappeared. Again, if no one steps up, it will be removed in .33
asus_oled This only needs minor cleanups to get merged properly into the main tree. If someone wants an easy project, this would be it.
at76_usb wireless driver. Again, no one working on it, it will be dropped in .33.
b3dfg I really do not think anyone cares about this. again, will be dropped if this is true in .33.
cpc-usb After the initial flurry of development, everyone seems to have run away. Was it the fact that I hadn't showered in a few days? Again, will be removed if no one comes back. And I am wearing deodorant now...
frontier A nice driver, again, should not be hard to get merged into the main tree, if someone wants an easy project...
go7007 Ugh. Unless someone steps up now to take this over, it's going to be removed in .33. There is no hardware made with this anymore, and no specs around that I know of, and the code isn't the nicest in the world.
heci A wonderful example of a company throwing code over the wall, watching it get rejected, and then running away as fast as possible, all the while yelling over their shoulder, "it's required on all new systems, you will love it!" We don't, it sucks, either fix it up, or I am removing it.
line6 Another nice driver that should be simple to get merged. Please, if you are looking for something to do, this is it.
me4000 and meilhaus They work on the same hardware, and they duplicate the existing COMEDI drivers. Someone thinks that custom userspace interfaces are fun and required. Turns out that being special and unique is not what to do here, use the COMEDI drivers instead. These will be removed. Heck, I'll go remove them for .32, there is no reason these should still be around, except to watch the RT guys squirm as they try to figure out the byzantine locking and build logic here (which certainly does count for something, cheap entertainment is always good.)
mimio Another driver that should be simple to get merged. Someone just step up to do this please, there are users of this hardware out there.
p9auth While it seemed like a good idea, I don't think that anyone actually uses this. It will be removed in .33 unless someone comes forward.
panel Another one that should be simple to merge. Anyone?
phison What? I thought I asked for this to be merged a while ago, sorry about that, no reason it should still be in staging anymore, it's just so small it slipped through the cracks...
poch A long-suffering company is enduring the slowest developers in the world here. Hopefully the code will be replaced with a UIO driver, but testing the userspace side seems to be difficult and slow. I have to give Redrapids major credit for being patient here, they are being amazing.
rspiusb A weird, very expensive camera, from a company that does not want to release the specs, and wants custom userspace interfaces. The code hasn't built since the 2.6.20 days. I'll go delete it now from .32, it doesn't deserve to live as no one cares about it, least of all, the original authors of the code :(
slicoss and sxg These are being developed by a consulting company for the main producer of the chips. Yet they seem to have disappeared half-way through the job. Odd. Hopefully they come back soon.
stlc45xx Another wireless driver that no one seems to care about. So sad. I guess no one will miss it when it goes away in .33.
udlfb Video over USB, it doesn't get anymore whacked than that. This is still being developed but the developer doesn't like to do incremental updates for some odd reason. Hopefully he pops up again with an update. But for now, it is quite workable for a number of developers.
usbip USB over IP. I guess if you ran video over IP to your USB device, that would be more whacked than just video over USB. This did get one big update during the .32 development cycle, hopefully the developer can come back again when they get some free time to continue working on it. Rumor has it that some major distros are starting to rely on this code, so it would be nice to get their help to get it working better...

That should cover all of the 600+ patches in the staging tree for the .32 kernel merge, and the existing drivers/staging/ tree. If I missed anything, please let me know.

VME bus support for Linux

Mon, 03 Aug 2009 21:25:00 GMT

Today another nice thing for the Linux kernel happened, we got working VME bus drivers and infrastructure submitted to the kernel tree. Now, I don't expect it to generate as much press as the Microsoft kernel driver thing did, but it should, as I feel it's more important in a way.

The VME bus code has lived outside of the kernel for many years, and there was at least three different implementations at the same time floating around. Martyn Welch from GE Faunc took the time, merged all of them together, and rounded up the different copyright holders of the code and got legal approval from all of them to properly release the code under the GPLv2.

Now as someone who has tried to do this kind of thing, I know how thankless and how difficult it can be. So here's a great big thank you to Martyn and his employer for getting this work done, and taking the time to work toward getting it merged into the main kernel tree.

The patches are here, here, here, here, and a good readme for the api is here.

It's getting cold in here

Tue, 21 Jul 2009 03:17:00 GMT

As announced on the linux-kernel and driver project mailing lists here, Microsoft has released their Linux Hyper-V drivers under the GPL version 2, and have submitted them for inclusion in the main Linux kernel source tree.

I've been working with the developers at Microsoft for a while now trying to make this happen, through the Linux Driver Project.

Now, on one hand this is no different from any other company that I have worked with through the driver project. We are averaging about 2 new companies a month right now, working with them to get their code cleaned up and merged into the Linux kernel tree. Stuff like this happens all the time with new companies becoming part of the Linux kernel community every day. This is backed up with the statistics published for every kernel release on the LWN.net kernel pages.

But, on the other hand, this is Microsoft, so it is a big deal. There are two major aspects of what they did here:

They released the code under the GPLv2 and publicly stated that this is a valid license for companies to release code under. They will be continuing to contribute under this license, as they work to clean up the code, and add new features and fix bugs as time goes on. This is a huge step forward for Microsoft from what they have previously stated in the past.
They publicly stated that the proper license to release a Linux kernel driver is under the GPLv2, To quote from the notes they sent out to a number of press members:

Q: Why release the code?

A: Because we have utilized Linux code, Microsoft has an obligation to open source the device drivers. This is the process outlined by the Linux community.

Q: Why open source the code?

A: Because this is a requirement of the community, and critical in ensuring that as the Linux Kernel evolves, and as Hyper-V evolves, that the Hyper-V Linux Device Drivers evolve as well.

As one person involved in this whole process said to me, "It looks like hell just froze over."

Update:
Steve gives a little more of the backstory of what caused me to start talking to Microsoft in the first place.

Linux Driver Project Status Report as of June 2009

Thu, 04 Jun 2009 20:14:00 GMT

This is a status report for the Linux Driver Project as of June 2009, describing what has happened in the past year of work. It was originally posted on the Linux Driver Project developer mailing list.

Loving the 'droid

Tue, 24 Mar 2009 23:57:00 GMT

The 2.6.29 kernel release is now out, and if you look closely, there are a number of drivers for the Android platform merged in, starting with this patch.

Yes, there's still a lot to go, like cleaning up the user/kernel interfaces, and prodding the core Android developers to start working upstream more, which they have already started to do. It's a great start, and one that I hope will succeed overall, as it really is a nice system to develop for, and one the phone manufacturers have been asking for from the Linux community, for years.

And besides, how can you not like such a cute robot:

The Linux Staging Tree, what it is and is not

Wed, 18 Mar 2009 18:42:00 GMT

It's been many months since the Linux Kernel developers conference, where the linux-staging tree was discussed and role changed. It turns out that people are still a bit confused as to what the staging tree is for, and how it works.

So here's a short summary, I'm not going into the history or background here, that's a much longer writeup that I'd be glad to do if people are interested.

The Linux Staging Tree, what it is and is not.

What the Linux Staging tree is

The Linux Staging tree (or just "staging" from now on) is used to hold stand-alone drivers and filesystems that are not ready to be merged into the main portion of the Linux kernel tree at this point in time for various technical reasons. It is contained within the main Linux kernel tree so that users can get access to the drivers much easier than before, and to provide a common place for the development to happen, resolving the "hundreds of different download sites" problem that most out-of-tree drivers have had in the past.

What the Linux Staging tree is not

The staging tree is not a place to dump code and run away, hoping that someone else will to the cleanup work for you. While there are developers available and willing to do this kind of work, you need to get them to agree to "babysit" the code in order for it to be accepted.

Location and Development

The staging tree is now contained within the main Linux kernel source tree at the location drivers/staging/. All development happens within the main kernel source tree, like any other subsystem within the kernel. This means:

The linux-next tree contains the latest version of the staging tree, with bugfixes that are about to be merged into Linus's tree, as well as the patches that are to be merged into the next major kernel release.
If you wish to do work on the staging tree, checkout the linux-next tree and send patches based on that.

Runtime

When code from the staging tree is loaded in the kernel, a warning message will be printed to the kernel log saying:

MODULE_NAME: module is from the staging directory, the quality is unknown, you have been warned.

and the kernel will be tainted with the TAINT_CRAP flag. This flag shows up in any kernel oops that might be produced after the driver has been loaded.

Note, most kernel developers have expressed the warning that they will not work on bugs for when this taint flag has happened, so if you run into a kernel problem after loading such a module, please work to reproduce the issue without a staging module loaded in order to be able to get help from the community.

If anyone has any questions that this summary doesn't answer, please let me know.

I write too much email

Sat, 17 Jan 2009 07:55:00 GMT

A few years ago, when one of my kids was asked what their dad did for work, they replied, "Sit in the basement and write email."

It's not that far from the truth.

I looked back at my mail server logs for 2008 to see just how much email I did write. For 2008, it turned out that I wrote 19057 unique emails.

Yes, that's an average of about 52 emails a day, and no, that is not a number of individual recipients (one email sent to three people was counted as one email, not three.)

I do send out a lot of patches for review (for -stable trees, and for when patches get sent to Linus for inclusion in the main kernel tree), and I also send out a bunch of "Now that you got a patch into the Linux kernel, we were wondering who you work for", type emails to help lwn.net with their statistic gathering.

But that doesn't justify the numbers overall, I just think I have a bad habit.

Here's a breakdown of emails sent by the time of day for the whole year:

You can pretty clearly see when I go eat dinner and then switch over to using my laptop in the evening (time zone is local time of the day.)

When I tried to graph per day, you can kind of see lower numbers on the weekends if you squint:

I guess it's no wonder that Google thinks my email address is a spam-bot and refuses to let me sign up for any google groups with it. I think it is trying to tell me to cut down on my output or something.

Clearly I need help...

bti: tweets from the command line

Wed, 22 Oct 2008 18:45:00 GMT

A while ago I talked about piping all of my bash commands to twitter.com. I've kind of stopped doing that now, after I maxed out at over 14000 updates in about 2 weeks, but it was fun while it lasted.

But in order to do this kind of thing nicely, I ended up writing a command line program to make it easier. Some people have noticed it at times, by poking around in my kernel.org home directory, so I might as well announce the thing publicly.

So, consider this an announcement of the tool, bti. It allows you to send tweets to twitter.com or identi.ca directly from the command line from any Linux machine. It probably works on other systems as well, but you will have to tweak the Makefile yourself.

The latest version can always be found here.

The development for the tool is done in git, and the tree can be found on the ever-awesome github.com in this repository.