A Framework for Evaluating SBOM Tools

FOSSA Editorial Team -

Today, it’s very difficult to run a successful SBOM (software bill of materials) program without the right SBOM tools.

Although manual (or partially automated) ways of generating, managing, and consuming SBOMs may have worked in earlier generations of software development, today’s realities demand a more sophisticated approach. Not only are modern applications very complex and updated frequently, but there’s a growing need for SBOMs to be created in specific machine-readable formats. Additionally, SBOMs have an increasing number of important use cases, including regulatory compliance, customer requests, open source vulnerability management, and open source license compliance, and tooling should be versatile enough to support all of them.

But with so many SBOM tools on the market today, how can you find the right one for your organization? Different businesses will have different needs, but these eight capabilities can play particularly important roles in a comprehensive and successful SBOM program.

(For context and clarity: FOSSA’s open source management platform includes a popular SBOM tool, which we’ve developed with these principles in mind. Please feel free to get in touch with any questions or if you’d like more information.)

1. Supports SPDX and CycloneDX — and Enables Compliance with the Cybersecurity Executive Order

The U.S. government’s 2021 Executive Order on Improving America’s Cybersecurity had a significant impact on the modern SBOM landscape. The executive order requires certain organizations to produce an SBOM to accompany each product. While this mandate is relatively narrow in scope — it directly applies to only federal agencies and companies that do business with them — there’s been a noticeable trickle-down effect into the private sector.

The executive order also did more than just require organizations to produce SBOMs. It dictated how those SBOMs should be created. Specifically, in July 2021, the federal government published minimum required elements for an EO-complaint bill of materials. These minimum elements include:

  • Data fields: Your SBOM should include supplier name, component name, component version, unique identifiers, dependency relationship, author of SBOM data, and timestamp
  • Automation support: Your SBOM should be generated in one of three human- and machine-readable formats — SPDX, CycloneDX, or SWID Tag. (In practice, we see that most organizations prefer SPDX or CycloneDX, which are more comprehensive specifications than SWID Tag.)

Regardless of whether your organization currently does business with the federal government, it’s wise to prioritize an SBOM tool that checks these boxes. In addition to fulfilling customer requests, comprehensive, machine-readable SBOMs enable a greater breadth of software supply chain security and transparency initiatives.

It’s also worth mentioning that the CycloneDX and SPDX specifications support multiple file formats, including .JSON and .XML. SBOM tools that give users the ability to pick their preferred SPDX/CycloneDX format provide additional flexibility (and can help satisfy a broader range of customer requests).

2. Has Comprehensive Programming Language and Ecosystem Support

Another part of producing accurate SBOMs is using a tool with comprehensive programming language and ecosystem support.

Like we mentioned earlier, you want to avoid tools that require you to create SBOMs in one way for a particular ecosystem or language and a different way for another language or ecosystem — but not only because this creates a poor user experience. It can also lead to inconsistent and possibly inaccurate results caused by each tool’s unique interpretation of SBOM fields. For example, a tool may interpret the package supplier as the author of the package, the author’s parent organization, the package manager distributing the package, the registry hosting the package, or a combination of the above. Multiply these inconsistencies by the number of fields in any SBOM, and you can see how significant issues may arise.

An important related capability is how well the SBOM tool (and/or the vendor behind it) can handle the sorts of edge cases that will inevitably pop up, especially for larger and more complex engineering organizations. For example, just because a tool claims to support Yarn doesn’t mean it will necessarily be able to navigate the nuances of Yarn 1 vs. Yarn 2. Or Go projects before and after the introduction of Go modules.

We encourage organizations to have the language coverage conversation with prospective vendors early in the tooling evaluation process.

3. Is Customizable

We mentioned earlier that it’s important for SBOM tools to support multiple export formats and file types. It’s also critical for users to be able to customize the metadata that gets included in a specific bill of materials. This is important because SBOMs have multiple use cases — customer requests, due diligence, security, regulatory compliance, open source license compliance and more — and what makes sense to include in one SBOM might make sense to exclude in another.

And, while different organizations will have different preferences when it comes to customizability, a good general rule is that you should be able to pick and choose:

  • Components included in your report: open source licenses, first-party licenses, direct dependencies, transitive dependencies, copyrights, license headers
  • Dependency metadata included in your report: package, author, description, declared license, discovered license, license header, package URL, full license text, file path, dependency path
  • Format: SPDX, CycloneDX, HTML, plain text, CSV, Markdown

4. Is Easy to Use

The logical place to start when evaluating an SBOM tool’s usability is its user interface. Does the tool require a lot of training to use effectively? Can you go from the homepage to an up-to-date SBOM in a handful of steps? Is it easy to customize which elements to include (and exclude) from your bill of materials?

There are also several technical specifications that can end up having a significant impact on usability. They are:

  • Configuration: You’ll have a superior experience if you don't have to customize the SBOM tool for every language ecosystem
  • Static analysis: Look for SBOM tools that don’t require a runtime integration or tool to use

Another part of usability is how your tool keeps SBOMs up to date. Does the tool automatically generate a new SBOM when there’s a new version of your software? Or does it require a more manual, time-consuming process?

5. Supports Third-Party SBOM Import and Analysis

Although generating SBOMs is a top tooling use case, it’s not the only important one. The ability to import and analyze third-party SBOMs is also critical. As a starting point, look for SBOM tools that can ingest popular machine-readable formats like CycloneDX. And, consider tools that surface third-party SBOM data in a way that allows you to leverage it.

The goal is to be able to analyze an imported SBOM like you would your own code. For example: Let’s say there’s a new zero-day vulnerability impacting a certain open source library. You should be able to analyze the third-party SBOM to quickly determine whether the application relies on the open source component (and, if so, whether it uses a vulnerable version).

6. Provides Actionable, Structured Insights

An effective SBOM program has multiple objectives, including satisfying customer requests, complying with regulatory requirements, and reducing software supply chain risk. It's critical to prioritize tools that can operationalize SBOM data to achieve these objectives.

Given that modern applications can have hundreds of open source dependencies, it’s critical to quickly detect issues and take action to address the areas of most significant risk.

SBOM tools can support risk-management initiatives by providing:

  • At-a-glance visibility into vulnerabilities (with the ability to filter by severity) that may impact your project
  • Remediation guidance so you can quickly upgrade affected components to safe versions
  • Automated policy implementation (which can fail builds if your application conflicts with the policies you create to govern open source license compliance and security)
  • Audit-grade reporting allows users to annotate security and licensing concerns to isolate mitigated risk from active risk

7. Has Breadth and Depth

An SBOM is only as useful as it is accurate. And, two of the biggest factors that contribute to SBOM accuracy are the breadth and depth of your tool’s coverage.

Breadth: Can the tool scan all of your projects with a base level of SBOM coverage? You don’t want a scenario where it’s difficult or even impossible to integrate your tool with certain repositories.

Depth: Can the tool detect not only your direct dependencies but also transitive dependencies? Are there limits to how many layers of transitive dependencies?

Often, SBOM tools that offer breadth and depth have multiple integration points to analyze component usage during every phase of the software development lifecycle. This includes:

  • VCS integration during code generation to quickly ensure all projects receive a base level of coverage
  • CI integration during the build phase (pre-application deployment) to assess risk in components included at build time
  • CD integration including container scanning to assess risk during application delivery

8. Supports Third- and First-Party Elements

The vast majority of the code in modern applications is open source; some estimates peg this number around 90%. So, it goes without saying that SBOMs tools will need to be able to detect and report on your open source components.

But an SBOM won’t be accurate unless the tool used to generate it supports all software components in a given application — including those built in-house and acquired from commercial suppliers. Specifically, look for SBOM tools that:

  • Provide complete dependency graph coverage; even without access to a component, your SBOM tool should still be aware of the component(s), allowing you to annotate missing information when desired.
  • Analyze these components as binaries to find license information
  • Tell you what other projects are using your in-house components to assess the impact (how pervasive the issue is throughout your organization) of any determined risks

Evaluating SBOM Tools: The Bottom Line

There’s no one-size-fits-all approach to picking the right SBOM tools, but there are certain capabilities that, generally speaking, play important roles in a successful SBOM program. And, we hope this list serves as a useful starting point as you begin to evaluate the numerous SBOM tools on the market today.

FOSSA’s suite of open source license compliance and security products includes a widely used SBOM tool; in fact, leading analyst firm Forrester awarded our platform the highest score possible for SBOM support.

If you’d like more information on using FOSSA to generate, import, and/or manage SBOMs, please feel free to get in touch with our team by filling out the form on this page.