So, one comment in my recent response to the Sun kernel developer stated:

Linux most often runs on hardware where such errors can not be detected, as we run on a zillion different platforms (although not as many as NetBSD).

Well, it turns out that this is not true. The illustrious Rick Moen pointed me to the Linux User Group HOWTO which he maintains. It lists all of the current architectures and platforms that Linux currently runs on.

Comparing that list to the list of NetBSD ports it is now evident that Linux has been ported to more platforms than NetBSD.

So, not only is it safe to state that Linux supports more devices "out of the box" than any other operating system, you can also say that it supports more processor types than any other operating system in the world.

Which just goes to show how flexible Linux is...

posted Wed, 29 Sep 2004 in [/linux]

I was glad to see that Eric took the time to address my previous rebuttal to his previous comments. I welcome good technical discussions like this, in the open, without rude flames by anyone. It's fun, and lots of people get to understand things a bit better about the topic

That being said, I'd first like to address his closing comment, which was regarding my comment about Linux not going anywhere:

For some reason, all Linux advocates have an "us or them" philosophy. In the end, we have exactly what I said at the beginning of my first post. Solaris and Linux have different goals and different philosophies. Solaris is better at many things. Linux is better at many things.

I agree completely. I wasn't trying to put up any "us vs. them" type attitude, I was merely trying to explain in my message the reasons why the Linux kernel has or does not have those different features that Eric was discussing. My comment at the end was a bit glib, I agree, I was merely trying to state that Linux isn't going anywhere, and will welcome all Sun users and developers if they decide that Linux will work for them.

Ok, on to the technical stuff:

First off, thanks for giving specifics about your points of reliability, serviceability, observability, and resource management. Let's address these points.

• Reliability - Of course reliability is more than "better than Windows." Geesh, what a low bar to shoot for these days. Linux had better be able to handle hardware failures where ever possible, when ever it can be detected. Ah, that last part is the biggest issue. Linux most often runs on hardware where such errors can not be detected, as we run on a zillion different platforms (although not as many as NetBSD). For systems that we can detect these kinds of errors, we do (like PCI error reporting on the PPC64 platforms for example.) The hardware that Solaris usually runs on also has that kind of error reporting capabilities, and so the OS takes advantage of it. So Linux and Solaris are pretty equal here. As for the claims that the ZFS people are stating, I think that Linux filesystems like Lustre and SSD do pretty much the same thing (automatic error correction for large collection of disks all without the application needing to fix it up.)
• Serviceability - Sure, things go wrong all the time. That's why enterprise distros add the crash dump, kernel debuggers, and dprobes code to their kernels in order to be able to help service their customers. Nothing different from Solaris there (although you mentioning the ability to have a firmware dump of hardware errors is pretty cool, but again, that's a hardware feature, not an OS one.)
• Observability - DTrace does sound like the all-singing, all-dancing solution to everything that a kernel could possibly report to a user. And if so, I commend you all for creating such a wonderful tool. As for Linux, if you want much the same functionality, use the LTT code, or the dprobes code. Again, many enterprise related Linux distros ship their kernels with these features added to them, as their customers ask for it.
• Resource Management - That sounds pretty much exactly what the CKRM project does for the Linux kernel. Again, enterprise distros ship it, so their customers can have it. And this feature is getting fixed up to be acceptable for the mainline kernel, and will probably get merged into it within a year or so (but again, if you want that option, it's available to you.)

As for the comment about Solaris having these features "more polished" than Linux's, I will not disagree. But they are getting better over time, as companies realize they want these features in Linux, and address any shortcomings that these features may have.

Binary compatibility. You state:

We have customers paying tens of millions of dollars precisely because we claim backwards compatibility.

You have customers paying that much money for driver compatibility? Geesh, I didn't know drivers were that expensive... Seriously, for Linux, this argument doesn't even register. A customer does not have to worry about the huge investment they just made in hardware, and the fact that the driver they have for it better work for any future version, as they have the source to it (it's in the kernel tree.) They know that it will work with all future versions of Linux, and if something goes wrong, they can either fix it themselves, or hire someone else to fix it for them, for far less than "millions of dollars."

I really need to write a article/essay about why Linux does not have driver api stability. I touched on it in my previous post, but in reading your response, and the responses by others, you all seem to miss the main points. It's not that we don't know how to create a binary api with padding structures out, and offering up new functions, it's the fact that because we have the source to all of our drivers, we do not have to.

One minor comment though, the fact that we have the source to everything changes all of the old rules that operating systems had to live by. Backwards compatibility is no longer necessary, enabling us to move faster, and be more flexible than ever. As proof of that, look at the huge range of machines that Linux runs very well on. Everything from this tiny little gumstick, up to a 512 way ia64 box. And our drivers work on all of those platforms, with no changes needed.

So, let me bring up a few other points about where Linux is better than Solaris:

• Device support - As I mentioned before, and as you mentioned, Linux's device support is unmatched anywhere else. We support, out-of-the-box, more devices than any other operating system. We also support them on every platform that Linux is supported on (somewhere over 21 unique architectures with lots of minor arches last I looked). That's power and flexibility that enables customers to start out with a small Linux box (running on a cheap i386 clone) and if needed, they can go out and buy a ia64, or amd64 box, and use the same investment in devices (disk controllers, etc.) and with no changes to their applications (other than a recompile) and "just work". This also enables customers to not have to rely on any one specific hardware vendor, preventing any "lock in". They can pit HP vs. IBM vs. Dell vs. the whitebox dealer down the street, or just build machines themselves (like a lot of places do, Google for example), in order to get the best service and price and support for their specific situation. That's incredibly powerful for the customer/user, and very scary for the big companies that previously had relied on the cost that a customer had spent already to tie them to that company for a long time.
• Speed - Last I looked, Linux beat the pants off of any other operating system on the same exact hardware configuration. Both in microbenchmarks like lmbench, and in macrobenchmarks like specweb and TPC numbers. I've seen loads of unpublished results at a lot of different companies for these macrobenchmarks backing this up (unpublished usually due to the cost of publishing them, and for political reasons because the company also sold other operating systems on the same hardware, and didn't want to loose those customers in large numbers.) And, I'm sure the fact that we don't have to keep around old, broken apis, or have padded structures to keep binary compatibility might have a little to do with this speed advantage :)
• Flexibility - I touched on this up above in the device support point, but Linux scales from very tiny to very huge systems, all based on the exact same kernel code base. We support mmuless embedded devices, and NUMA systems. No other operating system can claim that kind of flexibility that I know of (ok, maybe NetBSD, but it doesn't support the range of devices that we do...) But aside of the system flexibility, Linux's main flexibility is that if it doesn't do something you want it to do, then you can change it to do it yourself. If you are a customer, and really like Solaris, but happen to want to use it to put it into your barcode scanner, you can either try to convince Sun to do this for you, and pay them a whole lot of money, or you can find a os that can do this. Sure, I know you will say that Solaris isn't in that market, and that they are only going for the enterprise, which is a completely valid response. It's just that Linux meets the needs of everyone, not just a single market segment. It's this point that will ensure that any feature that is missing from Linux, will allow it to be easily added.

So, that's the three main points that I think Linux beats every other operating system out there on. Combine those points, and you have a very powerful system, for a huge range of people.

I welcome any responses to this post.

posted Sun, 26 Sep 2004 in [/diary]

Ah, so I read the comments that Eric Schrock had about why Sun doesn't want to help with Linux kernel development, and as a Linux kernel developer I thought I would attempt to address his major points, and explain as to why Linux developers feel this way. It's still early in the morning for me, so this might seem a little disorganized, but what the hell:

Ok, his first statement:

The main reason we can't just jump into Linux is because Linux doesn't align with our engineering principles, and no amount of patches will ever change that. In the Solaris kernel group, we have strong beliefs in reliability, observability, serviceability, resource management, and binary compatibility. Linus has shown time and time again that these just aren't part of his core principles, and in the end he is in sole control of Linux's future

To state that the Linux kernel developers don't believe in those good, sound engineering values, is pretty disingenuous. Hey, one of the first thing that people think about when Linux is mentioned is "reliability" (hm, I used to have a link showing that, can't find it anymore...) These is pretty much a baseless argument just wanting to happen, and as he doesn't point out anything specific, I'll ignore it for now.

He continues on with specific issues:

Projects such as crash dumps, kernel debuggers, and tracing frameworks have been repeatedly rejected by Linus, often because they are perceived as vendor added features.

Ah, ok, specifics. Let's get into these:
Crash dumps. The main reason this option has been rejected is the lack of a real, working implementation. But this is being fixed. Look at the kexec based crashdump patch that is now in the latest -mm kernel tree. That is the way to go with regards to crash dumps, and is showing real promise. Eventually that feature will make it into the main kernel tree. But to state that Linux kernel developers just reject this feature is to ignore the fact that it really wasn't ready to be merged into the tree.

Kernel debuggers. Ah, a fun one. I'll leave this one alone only to state that I have never needed to use one, in all the years of my kernel development. But I know other developers who swear by them. So, to each their own. For hardware bringup, they are essential. But for the rest of the kernel community, they would be extra baggage. I know Andrew Morton has been keeping the kernel debugger patches in his tree, for those people who want this feature. And since Linux is about choice, and this choice is offered to anyone who wants to use it, to say it is not available, is again, not a valid argument.

Tracing frameworks. Hm, then what do you call the kprobes code that is in the mainline kernel tree right now? :) This suffered the same issue that the crash dump code suffered, it wasn't in a good enough state to merge, so it took a long time to get there. But it's there now, so no one can argue it again.

Which brings me to the very valid point about how Linux kernel development differs from any other OS development. We (kernel developers) do not have to accept any feature that we deem is not properly implemented, just because some customer or manager tells us we have to have it. In order to get your stuff into the kernel, you must first tell the community why it is necessary, and so many people often forget this. Tell us why we really need to add this new feature to the kernel, and ensure us that you will stick around to maintain it over time. Then, if the implementation seems sane, and it works, will it be added. A lot of these "enterprise" kernel patches didn't have sane implementations, and were not successfully explained to the kernel community as to why they are necessary (and no, the argument, "but we did it in INSERT_YOUR_DYING_OS_NAME_HERE, so Linux needs it" is not a valid excuse.)

Then Eric continues:

Not to mention the complete lack of commitment to binary compatibility (outside of the system call interface). Kernel developers make it nearly impossible to maintain a driver outside the Linux source tree (nVidia being the rare exception), whereas the same apps (and drivers) that you wrote for Solaris 2.5.1 will continue to run on Solaris 10.

First off, this is an argument that no user cares about. Applications written for Linux 1.0 work just fine on the 2.6 kernel. Applications written for older versions of Linux probably work just fine too, I don't know if anyone tried it in a long time. The Linux kernel developers care a lot about userspace binary compatibility. If that breaks, let us know and we will fix it. So to say that old apps written for old versions of Solaris will work on the latest Solaris, is not a valid argument at all.

But the issue of driver compatibility. For all of the people that seem to get upset about this, I really don't see anyone understand why Linux works this way. Here's why the Linux kernel does not have binary driver compatibility, and why it never will:

• We want to fix the bugs that we find. If we find a bug in a kernel interface, we fix it, fix up all drivers that use that api call, and everyone is happy.
• We learn over time how to write better interfaces. Take a look at the USB driver interface in Windows (as an example). They have rewritten the USB interface in Windows at least 3 times, and changed the driver interface a bit every time. But every time they still have to support that first interface, as there are drivers out there somewhere that use it. So they can never drop old driver apis, no matter how buggy or broken they are. So that code remains living inside that kernel forever. In Linux we have had at least 3 major changes in the USB driver interface (and it looks like we are due for another one...) Each time this happened, we fixed up all of the drivers in the kernel tree, and the api, and got rid of the old one. Now we don't have to support an old, broken api, and the kernel is smaller and cleaner. This saves time and memory for everyone in the long run.
• compiler versions and kernel options. If you select something as simple as CONFIG_SMP, that means that core kernel structures will be different sizes, and locks will either be enabled, or compiled away into nothing. So, if you wanted to ship a binary driver, you would have to build your driver for that option enabled, and disabled. Now combine that with the zillion different kernel options that are around that change the way structures are sized and built, and you have a huge number of binary drivers that you need to ship. Combine that with the different versions of gcc which align things differently (and turn on some kernel options themselves, based on different features available in the compiler) and there's no way you can successfully ship a binary kernel driver that will work for all users. It's just an impossible dream of people who do not understand the technology.
• Drivers outside the kernel tree and binary drivers take away from Linux, they give nothing back. This was one of the main statements from Andrew Morton's 2004 OLS keynote, and I agree. Out of the box, Linux supports more hardware devices than any other operating system. That is very important, and is something that we could not have done without the drivers being in our tree.
• If a kernel api is not being used by anyone in the tree, we delete it. We have no way of knowing if there is some user of this api in a driver living outside on some sf.net site somewhere. I have been yelled at for removing apis like this, when there was no physical way I could have possibly known that someone was using this interface. To prevent this, get your driver into the kernel tree. By doing this, it will force you to write good kernel code (up to the standard of the rest of the tree), and it will let other people fix your bugs easier, and if any kernel api changes, the person who changes it will fix your driver for you. This is the power of opensoruce development, why try to subvert it?

Ok, the final point I want to address:

Large projects like Zones, DTrace, and Predictive Self Healing could never be integrated into Linux simply because they are too large and touch too many parts of the code. Kernel maintainers have rejected patches simply because of the amount of change (SMF, for example, modified over 1,000 files).

This is just wrong. Look at the LSM kernel change that I helped get into the kernel tree? It touched pretty much every part of the kernel. It touched almost every core kernel structure. And yet, the LSM developers worked with the kernel community, addressed their concerns, and got the changes accepted into the tree. And Linux now benefits from having an extremely flexible security model now because of this effort. So to say it is impossible to get such a change into Linux is false.

It's ok that Sun doesn't want to use Linux for their systems, no one is forcing them to, and Solaris is quite nice in places. And when Sun realizes the error of their ways, we'll still be here making Linux into one of the most stable, feature rich, and widely used operating systems in the world.

posted Thu, 23 Sep 2004 in [/diary]

Yeah, finally got a new verision of the hotplug package out. Along with that, I deleleted the hotplug startup script (well, pretty much so, it doesn't do anything now) from the Gentoo package, and created a new package, coldplug, for those users who really like to load their modules at boot time. I also disabled the ability to unload the whole USB subsystem at shutdown time. That just caused too many lockups, problems, confusion, and even a few kernel oopses.

No wonder none of the others distos unloded all of USB at shutdown time :)

posted Mon, 20 Sep 2004 in [/diary]

Ugh. Trying to work out how to write "static" functions for sysfs attributes. You know, you just want to say, "Export this specific field in this structure. It's a u16 and I don't want to be notified when someone reads it."

Today you have to write a bit of code for every single thing that you want to export in sysfs. For structures with a lot of different fields, you end up doing some fun with #defines that would make any sane programmer cringe if they noticed it. For example, look at this fun stuff I wrote in the PCI core code in the kernel:

/* show configuration fields */
#define pci_config_attr(field, format_string)                           \
static ssize_t                                                          \
field##_show(struct device *dev, char *buf)                             \
{                                                                       \
        struct pci_dev *pdev;                                           \
                                                                        \
        pdev = to_pci_dev (dev);                                        \
        return sprintf (buf, format_string, pdev->field);               \
}

pci_config_attr(vendor, "0x%04x\n");
pci_config_attr(device, "0x%04x\n");
pci_config_attr(subsystem_vendor, "0x%04x\n");
pci_config_attr(subsystem_device, "0x%04x\n");
pci_config_attr(class, "0x%06x\n");
pci_config_attr(irq, "%u\n");

See what I mean? That's about the cleanest way I've seen this done, and even then, we're burning a function per attribute.

But hey, it's way better than anything you ever did with /proc, but still, it could be made easier. So that's why I'm trying to make it easier...

posted Thu, 16 Sep 2004 in [/diary]

Fun with __iomem marking fixups. Finished up the USB host controller drivers.

Applied a few patches from others fixing up minor things.

posted Wed, 15 Sep 2004 in [/diary]

More patch queue flushing.

Then attacked the __iomem marking. Got the PCI core and the PCI hotplug drivers fixed up. The USB host controller drivers are pretty much seriously messed up in this regard, as they treat a __u32 as a iomem address, which I don't think is really portable, but for some reason is working. I'll let those driver authors figure it out on their own...

Tried to add some type safety to the kobject event code, but gcc doesn't seem to check enumerated type values at compile time (or at run time for that matter.) Doesn't even matter if I make it a typedef. Oh well, maybe I'll dig into sparse one of these days to see if I can add that kind of support to it.

Despite the lack of checking I released a patch that made things a bit safer (we now specify a type of event, instead of a raw string.) This should help us keep any major typos from accidentally becoming the kernel<->userspace interface like a lot of people are worried about.

Need to get a new version of the hotplug scripts out the door, they are really starting to bother me at times (and I've been ignoring the gentoo bug reports for them, unfortunately)

posted Tue, 14 Sep 2004 in [/diary]

Not much coding due to a Linux Journal deadline. Finished the article and dealt with a lot of merge issues because 2.6.9-rc2 was released in the early morning hours. Think I got them all worked out, and noticed that a lot of __iomem marks need to be added to a lot of drivers.

So I fixed up the I2C bus drivers, but many more remain. Hm, feel like a kernel janitor a lot these days...

Put out a new udev release (032) due to a bug in udev callouts that was causing people with cdroms to not have them named properly. For once, our regression scripts didn't catch that one, need to go add another test to them to fix that.

posted Mon, 13 Sep 2004 in [/diary]

Finally got a new version of udev out the door. Killed a lot of time trying to update the in-tree version of klibc to the latest version, but gave up trying to track down odd segfaults. Oh well, maybe next time...

Also applied a few odd USB patches, and am working on the kevent patch as I type...

posted Fri, 10 Sep 2004 in [/diary]

Had fun cleaning up lots of s/usb_unlink_urb/usb_kill_urb/ fixes to the usb-serial drivers. Also cleaned up more instances of pci_find_device() although there are still hundreds left to do...

posted Thu, 09 Sep 2004 in [/diary]

Ah, pretty much caught up on patches sent to me from people while I was at SUCON.

posted Wed, 08 Sep 2004 in [/diary]

Yeah, finally ripped out usbdevfs from the kernel!

posted Sun, 05 Sep 2004 in [/diary]

Fixed up a few minor errors with the fact that we now export the SEQNUM from userspace to the kernel through a sysfs file.

posted Sat, 04 Sep 2004 in [/diary]

Applied a bunch of USB stuff from lots of different people.

posted Fri, 03 Sep 2004 in [/diary]