As described previously, the Linux kernel security team does not identify or mark or announce any sort of security fixes that are made to the Linux kernel tree. So how, if the Linux kernel were to become a CVE Numbering Authority (CNA) and responsible for issuing CVEs, would the identification of security fixes happen in a way that can be done by a volunteer staff? This post goes into the process of how kernel fixes are currently automatically assigned to CVEs, and also the other “out of band” ways a CVE can be issued for the Linux kernel project.
This is a post in the series about the Linux kernel CVE release process:
- Linux kernel versions, how the Linux kernel releases are numbered.
- Tracking kernel commits across branches, how to keep track of Linux kernel commits as they move from the main release branch into the different stable releases in an automated way.
- Linux kernel security work how the Linux kernel security team works to fix reported security bugs.
- Linux CVE assignment process (this post), how the Linux CVE team processes bugs and classifies them as CVEs.
- Linux CVE tools, how a kernel commit is turned into a CVE entry automatically and kept up to date over time (future post).
A few months after Linux became a CNA, Lee Jones, one of the members of the kernel.org CNA team wrote an article on lwn.net as to how CVEs were being assigned, along with some other background information. That process has pretty much remained the same since then, but given the number of questions we have had over the past few years, it’s best to go into a bit more detail as to how all of this works, and why some bug fixes are marked as a CVE and others are not.
How to find a bugfix in a sea of changes
As is often mentioned, the rate of development of the Linux kernel is very high, averaging around 9 changes a hour for the past decade or so. Luckily, the kernel developers and maintainers have been manually marking specific commits as “bug fixes” for decades, allowing them to be backported to the various stable kernel branches. This is to allow users to be able to use the last kernel release, or a “long term” kernel as a base on which to rely on until they upgrade to the newer version. This process was described earlier here,
Because there is a “feed of known bugfixes” out of the 9 changes an hour, averaging about 30 changes per day, the kernel CNA team determined that they can focus their review efforts on that feed, and ignore the other changes, as obviously those changes do not fix a problem. And if the missed changes do fix a problem, odds are someone will eventually ask for those changes to be backported to a stable kernel, so they can be certain that this is a good set of changes to be reviewing for potential vulnerability fixes.
What is a vulnerability
While many security people love to argue what is, or is not, a vulnerability, while dealing with CVEs, a CNA must follow the definition that cve.org gives us which is:
“An instance of one or more weaknesses in a Product that can be exploited, causing a negative impact to confidentiality, integrity, or availability; a set of conditions or behaviors that allows the violation of an explicit or implicit security policy.”
So with that definition in mind, the kernel CNA team members look at every bugfix that is added to the stable kernel releases and reviews it to determine if it meets this criteria.
What is a kernel vulnerability
At the level that the Linux kernel runs, almost any type of bug that can affect a running system can be classified as a vulnerability. Specifically this means that the following bugfixes are classified as fixing a vulnerability:
- Any change that resolves a user-triggered crash of the system
- Any change that resolves a memory leak, overflow, or use-after-free.
- Any change that resolves secrets leaking from the kernel to userspace (with the exception of some address randomization stuff, which any local user can easily determine and is not deemed a secret, unless it can be determined remotely.)
- Any change that fixes incorrect boundary checks of buffers
- Any change that fixes a denial-of-service problem (i.e. the machine locking up, stopping talking on some communication channels, rebooting, etc.)
- Any change that resolves logic errors that could cause any sort of security bypass or elevation.
- Lots of other types of changes along the above lines (i.e. “we know it when we see it.”)
Why is triggering WARN() a vulnerability?
Many bugfixes in the kernel resolve a problem where a WARN() or WARN_ON() type of assertion is hit in the kernel. Normally Linux just prints out to the kernel log a message about what went wrong, and a stack dump and memory information, and continues on its merry way, as it is generally considered bad form to crash just because a issue was hit that the kernel can properly detect and recover from. However, there is a kernel option called panic_on_warn that if enabled, Linux will instantly panic and stop if a warning message is ever hit. This is a defensive setting that many users like to enable to protect any possible bug-like condition from being allowed to continue, as they would rather just reboot the machine entirely at that point in time. This option is set for a few billion embedded systems running Linux, as well as many many cloud/server systems as they can handle fall-over easily.
Because of this setting, any kernel fix that resolves a user-reachable WARN_ON triggering, must be assigned a CVE entry. If this option were not present, the total number of CVEs assigned by Linux would greatly drop.
What is NOT a kernel vulnerability
When the kernel CNA team announced that they were going to start assigning CVEs to kernel fixes, the initial worry was that all kernel changes could be thought of as fixing a vulnerability. Luckily it turns out that most of the changes that go into the stable kernel releases that fix bugs do not meet the definition of a “vulnerability fix”, so they do not get assigned a CVE. Changes that are common in stable updates that do not get a CVE are:
- Performance fixes
- Device tree fixes
- Bugfixes for things that are not externally triggered by a user.
There are some bugfixes that many people would think would be classified as a vulnerability, but do not meet the cve.org definition, so we are not allowed to assign a CVE for them. Those include:
- Data loss
- Data corruption
- Filesystem corruption
- Hardware bug fixes / workarounds
Personally, I think that filesystem corruption or data loss bug fixes should be considered a CVE fix, but until cve.org changes their policy, we can not assign CVEs for those types of fixes. That is why it is essential for users to always take all stable kernel updates, as odds are, users do like their filesystems to stay uncorrupted with no data loss.
For some filesystem bugs a CVE will not be assigned when a maliciously crafted or accidentally corrupted filesystem image is mounted by the kernel, and the kernel crashes or has other problems with it, unless the filesytem-specific userspace fsck program previously ran on the image before mounting, and it did not find any problems with it. In simpler terms, always run fsck on any untrusted filesystem image before mounting it.
For many CPU bug fixes, the hardware vendor is responsible for assigning a CVE, which is why the huge category of the ongoing Spectre cpu bugs do not get CVEs assigned by the kernel CNA, but rather the hardware company should be the ones doing that assignment. Many times they do not, but there is nothing that us as a CNA can really do about that..
Assigning a CVE
There are 3 core members of the Linux kernel CNA team. The three of us take a list of every commit that is applied to each stable kernel release, and review them to determine if they feel the commit fixes a problem that would resolve a vulnerability based on the above definition and descriptions. We all do this work in different ways. I review each patch in mutt (after dumping them to a mbox file) and saving off the ones that I think meet the definition. Another member uses a custom tool that queries some LLM oracles and reviews the output of them for validity, while another uses a mix of regular expressions combined with manual review using a custom tool.
We all check into the vulns.git tree our lists of commits we feel should be assigned a CVE id for. Then another tool runs on those lists, and if at least 2 members vote for a commit, it is marked to be assigned a CVE id.
We also accept reviews from other community members. At least two different developers regularly provide reviews which they email to us in patch form to be applied to the vulns.git repository. Examples of that can be seen here and here.
These external reviews are taken into consideration after the main voting round happens. If an external person voted for a commit to be marked as a vulnerability fix, and a single core member also did, then the commit is reviewed again manually to determine what the core member missed, and often times that commit will then be assigned a CVE id.
All of this work happens in public, in the vulns.git repository. CVE ids are usually assigned one to two weeks after the commits have been in a released kernel version, to allow users time to update their systems. Which is one key reason why users should always update to newer kernels (more on how to keep a system up to date with the increased numbers of CVE ids being allocated in a later article in this series.)
Of course sometimes the CNA team miss commits that should be assigned a CVE, or a developer or company wishes to have a CVE id assigned before they get through the review process. If that is the case, a simple email to the kernel CNA team at the public email address is all that is needed, and then the id will be automatically assigned if the team agrees that the commit meets the requirements of a vulnerability fix.
This method is used by many developers who know they fixed a vulnerability and wish to get a CVE assigned for a presentation or paper due to print deadlines, and by many companies who do reviews based on reports sent to them by users. Requests are usually handled within a few days, depending on the travel schedule of the CNA team members.
Disputing a CVE
Along with sometimes missing commits that should be assigned a CVE, the CNA team also many times gets reviews incorrect. If anyone thinks that a CVE is assigned to a commit incorrectly, because that commit really does not fix a vulnerability, please email the CNA team and they will review it and often times reject the CVE based on that review. This has happened many times when a kernel subsystem maintainer patiently explains to the CNA team why a specific commit really is not a vulnerability due to their knowledge of exactly how that portion of the kernel works.
We welcome such emails from all kernel developers and maintainers, for without this type of checks, many commits often are missed, and incorrectly assigned. Over the past few years the CNA members have gotten better at reviews and the rate of rejection has gotten lower as both the community and the CNA members adjust their expectations.
Changing a CVE
As the information for what is contained in a CVE record comes directly from the git commit that fixes the vulnerability, sometimes that information can be incorrect, or needs to be clarified better. Examples of this is where the range of a vulnerability can not be correctly determined from the commit itself (i.e. when the vulnerability showed up, or was backported to), or where references to external reports for the vulnerability should be added. The kernel CNA team has ways to handle all of this by adding the needed range bounds, or changing the text, or adding references when requested. Easiest way to do this is to just send a patch for the vulns.git repository, adding the needed information. I’ll go into how this all works in more detail into the next article in this series, but if people are interested now, take a look at the “schema” file that we use, and some commits in the vulns.git repository by developers from different organizations.
CVEs for everyone!
Currently, the kernel CNA is issuing about 60 CVEs a week, making it the second most prolific CNA currently in the cve.org ecosystem. Trying to keep on top of this type of feed has been “difficult” for traditional security teams that are wishing to attempt to manually review and triage every CVE report. As part of the requirement for us to become a CNA, we are required to produce CVE records that can be automatically parsed and handled by tools, without the need for human interaction.
As part of that, each CVE we create specifically documents the files affected in the kernel source tree for this specific issue, and the exact git ranges, or version ranges if you want that, that the vulnerability originally showed up, and where it was fixed. With that information, every Linux user should be able to instantly determine if a CVE is applicable for them to review or not by filtering on that data.
Most CVEs are not for you!
Based on some reviews, on average, only 10 CVEs are even applicable to a Linux system for a normal week, reducing the amount of manual review down to a very small amount (reading 10 reports a week should be “trivial” for most system integrators.) This quantity is much lower overall than the other “major” operating system vendors produce in a week, if counting CVE numbers as a reference between projects actually means anything to you.
Remember that software version numbers and git ids are part of a graph, and are not something that you can do a simple “less or greater than” type of comparison. We have seen many companies get this wrong many times. ALWAYS properly walk the git repository tree to determine range checks.
And don’t get us started about the uselessness of CPE values, while we do populate those values in the CVE record, they are useless for any real range checking, so please never use them.
And yes, PURL will help solve a bit of this (it’s better than CPE, a low bar), but PURL does not work for self-hosted projects at the moment, like Linux, or GNOME, or KDE, or glibc, or gcc, or perl, or python, or curl, well, you get the idea. The PURL developers know this, and we are working with them to solve this issue in the standard to hopefully resolve it in the future.
Some CVEs are for you!
Despite a low number of kernel fixes being applicable to your system every week, many of them do fix real issues that the developers and community know about, so you should upgrade your systems. Ignoring this reported feed will directly affect the security of your system, so do so at your own risk.
We can not assign severity
As stated previously, kernel developers do not know how Linux is used, as open source can not dictate use. Because of that, we have no way of judging the severity of any CVE fix for a user, only the integrator of Linux into their system can properly determine that. So any group that attempts to give a “severity score” to a Linux CVE is lying to you, UNLESS they know exactly your use case.
ALWAYS ignore any attempt that groups such as NIST/NVD that purport to assign things like CVSS scores to a vulnerability. Those numbers are false and give companies a “fake sense of security”. We have asked NIST to stop assigning these scores, as it goes against the recently published CISA guidelines and we know it is keeping users from updating when they should be, as well as causing users to update when they don’t need to be (why would your remote server with no physical access to it need to be updated due to a USB gadget driver vulnerability?)
There have been some “working groups” of organizations that use Linux in the same way (i.e. cloud hosting companies) that have attempted to score CVE entries in the past. That is the only real way that any attempt to give a score should ever be used, and we welcome groups doing this. However, it seems that all of these groups have stopped doing this work, and instead, have found it simpler to just update to the latest stable kernel releases on a more regular cadence.
cve.org is looking into having third-party “enrichments” of existing CVE ids, based on the use case of that third party, be added directly to the cve.org json record as additional fields. If you are interested in doing this type of thing, please contact cve.org, but note that we do not think this is going to be scalable at all, but are more than willing to be proven wrong.
CVEs are assigned after-the-fact
As mentioned above, CVE ids are usually assigned on a one to two week delay from when the fix has landed in a released stable kernel version. This allows users who are regularly taking the Linux stable releases to ensure that their systems are secure before CVEs are announced to the world.
Kernel CVEs only reference a specific fix in the kernel tree, and do not reference any previous changes that might be needed in the tree before that specific change. They are also not tested in an independent way, but only as a whole with all other changes in a specific release by the kernel developer community. Because of this, any attempt to only “cherry pick” specific CVE fixes can be difficult and is not recommended to be done at all.
The Linux kernel developer community provides, for free, a constant feed of tested bugfixes in stable kernel releases for all to use. Attempting to pick and choose individual commits from that feed will cause systems to miss needed fixes, as well as create a system that no one has ever seen or tested and can not be supported by anyone else.
The bonus of just applying all stable releases is that you also get the data corruption and performance improvements for your system as well as the security updates. A result that many ecosystems have already realized is much simpler overall to do.
Turning a commit into a CVE
After a kernel commit has been declared to be fixing a vulnerability, and assigned a CVE, publishing that data to the world in an automated way while being able to keep the records up to date easily over time at a high volume of assigned CVEs was a goal of the kernel CNA team. Over the past 2 years, they have written a number of custom tools and a database to accomplish this, evolving from a single bash script to many thousands of lines of multiple programs written in Rust and C. More about how all of those tools work, and the difficulties in doing this all on a fast moving codebase like Linux, will be explained in the next article in this series.