Seeking short form video ideas

[popey]

Hullo!

As part of my DevRel activities, I’m keen to craft some short-form videos around the topics of SBOM management and vulnerability search using our friends Syft and Grype!

My goal is to make short, bite-size videos - maybe around three to ten minutes long, maximum. I have some ideas already, which start at the absolute basics, but am keen to hear suggestions from everyone, about what we might find interesting to share.

These could be published here on the site, embedded in our docs, or more widely spread on social platforms. Step one is ideation, then prioritisation, scripting, and recording. I’ll take care of everything after step one. But I need some ideas!

Here’s my starting 10, to kick things off. Nothing set in stone, titles and content subject to change. Some of these may already exist, but need updating / refreshing. Would be good to get some more complex ones in the mix.

Title	Content
What is Syft	What is it, what does it do, where do I get it
What is Grype	What is it, what does it do, where do I get it
Using Syft to scan containerised Python Applications	Practical example demo: Using Syft on developer workstation
What is an SBOM - Why do I need them	Define SBOM succinctly in video form
Automated Vulnerability Scanning with Grype & GitHub	Using our GitHub actions to automate SBOM & Vuln scan
How do I install Syft	Demo showing where to get and install Syft
How do I install Grype	Demo showing how to get and install Grype
How do I use Syft effectively	Tips and tricks for using Syft
How do I update Syft	Encouraging users to keep up to date - how to do that
Contributing to Syft	Summarised video version of Keith’s upcoming blog about binary identification

[joshbressers]

You could also create videos about how to use Syft and Grype to detect some recent vulnerability (there are many filed every day). Any video in the series would be very similar, but I think that’s OK because anyone looking for that info won’t be searching for general topics

[spiffcs]

Quick ideas for videos that might be helpful

Title	Content
What are the pros of the SPDX format	What is it, how do I use syft to generate it, why/when is the format used
What are the pros of the CycloneDX format	What is it, how do I use syft to generate it, why/when is the format used
How do I read grype vulnerability results?	What’s the json format? How are they linked back to packages found by syft?
What is a version constraint?	How do these constraints show up in grype? Why does it show that my software is vulnerable?
Where does grype get its data?	Show the open process behind the grype data
What is <VULN_FEED>?	How does grype normalize <VULN_FEED> into a single data source that other feeds share?

For the last one take any of the current vuln sources and each one could be its own video on going through the schema, data available, how matches work, and how it might eventually end up in a scan done by grype from a syft SBOM

[popey]

@joshbressers

You could also create videos about how to use Syft and Grype to detect some recent vulnerability (there are many filed every day). Any video in the series would be very similar, but I think that’s OK because anyone looking for that info won’t be searching for general topics

Ooh! That’s a natty idea.

Now, wait for the next XZ or find something else relatively prominent? Or just pick whichever one is out on the day I write the script and record?

[popey]

@joshbressers How about CVE-2024-1086?

Federal agency warns critical Linux vulnerability being actively exploited

[spiffcs]

I think a good list to target is the Kev Catalog (Known Exploited Vulnerabilities)

Here is the list of Vendors and their count included since April of 2024. Any of these could potentially be a good demo for syft and grype if they relate to a package that was installed. There is some research that will have to be done here since some of these relate to hardware that would be tough to do a demo against.

jq -r '.vulnerabilities[] | select(.dateAdded | startswith("2024")) | [.vendorProject] | @tsv' < kev.json | sort | uniq -c | sort -nr
  11 Microsoft
   7 Google
   5 Ivanti
   5 D-Link
   5 Apple
   3 Cisco
   3 Android
   2 Fortinet
   2 Citrix
   2 Apache
   2 Adobe
   1 VMware
   1 Sunhillo
   1 Spreadsheet::ParseExcel
   1 Roundcube
   1 Palo Alto Networks
   1 Nice
   1 NextGen Healthcare
   1 Linux
   1 Laravel
   1 Justice AV Solutions
   1 Joomla!
   1 JetBrains
   1 GitLab
   1 CrushFTP
   1 ConnectWise
   1 Check Point
   1 Atlassian

[spiffcs]

Alright! I did some more cooking on this one with @willmurphy

To find some recent ones we thought would make a good demo we used this command against the kev.json. Basically there is a shortlist from the vendors we found earlier that we wanted to investigate on a hunch we could detect things easily

jq -r '.vulnerabilities[] | select(.dateAdded | startswith("2024")) | [.cveID, .dateAdded, .vendorProject, .vulnerabilityName] | @tsv' < kev.json | rg -i -e apache -e gitlab -e vmware -e linux -e laravel
CVE-2023-27524	2024-01-08	Apache	Apache Superset Insecure Default Initialization of Resource Vulnerability
CVE-2018-15133	2024-01-16	Laravel	Laravel Deserialization of Untrusted Data Vulnerability
CVE-2023-34048	2024-01-22	VMware	VMware vCenter Server Out-of-Bounds Write Vulnerability
CVE-2023-7028	2024-05-01	GitLab	GitLab Community and Enterprise Editions Improper Access Control Vulnerability
CVE-2020-17519	2024-05-23	Apache	Apache Flink Improper Access Control Vulnerability
CVE-2024-1086	2024-05-30	Linux	Linux Kernel Use-After-Free Vulnerability

Using this process we found this CVE in the kev list CVE-2023-27524

CVE-2023-27524 - GitHub Advisory Database

We generated an SBOM for the vulnerable image on dockerhub, but the table output was gross so we filtered down the matches to ones that showed up in kev only

The SBOM generated was from this image:
https://hub.docker.com/r/apache/superset

syft -o json apache/superset:v2.0.0 > superset-2.0.0.json

cat superset-2.0.0.json | grype --by-cve | ./key-bootstrap.py
Pillow                          9.1.0                 10.0.1                        python  CVE-2023-4863        High
apache-superset                 2.0.0                 2.1.0                         python  CVE-2023-27524       Critical
libc-bin                        2.31-13+deb11u2       2.31-13+deb11u7               deb     CVE-2023-4911        High
libc-dev-bin                    2.31-13+deb11u2       2.31-13+deb11u7               deb     CVE-2023-4911        High
libc6                           2.31-13+deb11u2       2.31-13+deb11u7               deb     CVE-2023-4911        High
libc6-dev                       2.31-13+deb11u2       2.31-13+deb11u7               deb     CVE-2023-4911        High
libnghttp2-14                   1.43.0-1              1.43.0-1+deb11u1              deb     CVE-2023-44487       High
libwebp-dev                     0.6.1-2.1             0.6.1-2.1+deb11u2             deb     CVE-2023-4863        Negligible
libwebp6                        0.6.1-2.1             0.6.1-2.1+deb11u2             deb     CVE-2023-4863        Negligible
libwebpdemux2                   0.6.1-2.1             0.6.1-2.1+deb11u2             deb     CVE-2023-4863        Negligible
libwebpmux3                     0.6.1-2.1             0.6.1-2.1+deb11u2             deb     CVE-2023-4863        Negligible

Here is the python script that does this - you need a local version of the kev.json for it to work

#!/usr/bin/env python
import sys
import json
import re

def load_kev_data(file_path):
    with open(file_path) as kev_file:
        return json.load(kev_file)

def main():
    # Load kev.json
    kev_data = load_kev_data("/some/local/kev.json")

    # Read each line from standard input
    for line in sys.stdin:
        # Extract CVE ID from the line
        cve_match = re.search(r'\bCVE-\d{4}-\d{4,}\b', line)
        if cve_match:
            cve_id = cve_match.group(0)
            # Check if CVE ID exists in kev_data
            if any(cve_id == vulnerability["cveID"] for vulnerability in kev_data["vulnerabilities"]):
                print(line.strip())

if __name__ == "__main__":
    main()

Feel free to recycle this process against any other good candidate CVE that you find in that kev list.

[spiffcs]

Vulnerability demo research

Hello!

I’ve done a bit more work on some commands that could be useful for researching vulnerabilities that might be good demo targets.

Local Requirements

The first thing you might want to look at is the known exploited vulnerabilities catalog from cisa.

curl -o kev.json https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

You also want jq installed and available on path so you can explore the kev json document.

Because we’re testing syft/grype also be sure to have those tools installed:

Install syft
Install grype

Reducing the surface area

Now that we have our data and tools let’s take a look at which entries might be of interest to syft and grype.

What vendors are we looking at?

jq '.vulnerabilities[].vendorProject' < kev.json | sort | uniq -c | sort -nr | awk '$1 > 9'

The above command takes all entries in the kev.json and sorts for vendors with > 9 entries. This is a little arbitrary, but we can do analysis on this side first and then focus on the more horizontal set of single vendor entries as a separate exercise.

 286 "Microsoft"
  75 "Apple"
  71 "Cisco"
  67 "Adobe"
  58 "Google"
  33 "Oracle"
  32 "Apache"
  19 "VMware"
  19 "D-Link"
  17 "Ivanti"
  16 "Citrix"
  13 "Linux"
  13 "Fortinet"
  12 "Atlassian"
  11 "Samsung"
  11 "QNAP"
  11 "Mozilla"
  10 "Trend Micro"
  10 "SAP"
  10 "Android"

Let’s take a look at the Linux, Atlassian, and Apache entries. We know syft and grype do well when scanning containers so this gives us a good spread of potential vendor/os software packages to investigate,

What known exploited vulnerabilities exist for my chosen subset?

jq -r '.vulnerabilities[] | [.cveID, .dateAdded, .vendorProject, .vulnerabilityName] | @tsv' < kev.json | rg -i -e apache -e linux -e atlassian | sort

The above command will give a pretty long list. Let’s try and build some windows to explore the data. We can grow or shrink the dataAdded filter as needed.

Note that dateAdded field in this next command is not when the CVE was issued. It’s when the CVE was added to the known exploited vulnerabilities list.

jq -r '.vulnerabilities[] | select(.dateAdded >= "2022-05-30" and .dateAdded <= "2024-05-30") | [.cveID, .dateAdded, .vendorProject, .vulnerabilityName] | @tsv' < kev.json | rg -i -e apache -e linux -e atlassian
CVE-2022-26134	2022-06-02	Atlassian	Atlassian Confluence Server and Data Center Remote Code Execution Vulnerability
CVE-2022-26138	2022-07-29	Atlassian	Atlassian Questions For Confluence App Hard-coded Credentials Vulnerability
CVE-2022-24706	2022-08-25	Apache	Apache CouchDB Insecure Default Initialization of Resource Vulnerability
CVE-2022-24112	2022-08-25	Apache	Apache APISIX Authentication Bypass Vulnerability
CVE-2013-6282	2022-09-15	Linux	Linux Kernel Improper Input Validation Vulnerability
CVE-2013-2596	2022-09-15	Linux	Linux Kernel Integer Overflow Vulnerability
CVE-2013-2094	2022-09-15	Linux	Linux Kernel Privilege Escalation Vulnerability
CVE-2022-36804	2022-09-30	Atlassian	Atlassian Bitbucket Server and Data Center Command Injection Vulnerability
CVE-2021-3493	2022-10-20	Linux	Linux Kernel Privilege Escalation Vulnerability
CVE-2022-33891	2023-03-07	Apache	Apache Spark Command Injection Vulnerability
CVE-2023-0266	2023-03-30	Linux	Linux Kernel Use-After-Free Vulnerability
CVE-2021-45046	2023-05-01	Apache	Apache Log4j2 Deserialization of Untrusted Data Vulnerability
CVE-2014-0196	2023-05-12	Linux	Linux Kernel Race Condition Vulnerability
CVE-2010-3904	2023-05-12	Linux	Linux Kernel Improper Input Validation Vulnerability
CVE-2016-8735	2023-05-12	Apache	Apache Tomcat Remote Code Execution Vulnerability
CVE-2023-33246	2023-09-06	Apache	Apache RocketMQ Command Execution Vulnerability
CVE-2023-22515	2023-10-05	Atlassian	Atlassian Confluence Data Center and Server Broken Access Control Vulnerability
CVE-2023-46604	2023-11-02	Apache	Apache ActiveMQ Deserialization of Untrusted Data Vulnerability
CVE-2023-22518	2023-11-07	Atlassian	Atlassian Confluence Data Center and Server Improper Authorization Vulnerability
CVE-2023-27524	2024-01-08	Apache	Apache Superset Insecure Default Initialization of Resource Vulnerability
CVE-2023-22527	2024-01-24	Atlassian	Atlassian Confluence Data Center and Server Template Injection Vulnerability
CVE-2020-17519	2024-05-23	Apache	Apache Flink Improper Access Control Vulnerability
CVE-2024-1086	2024-05-30	Linux	Linux Kernel Use-After-Free Vulnerability

Another useful command here is filtering for negative matches. We know there are some limitations currently on how syft works when scanning windows targets so what does our list look like if we filter that out?

Here is a command that shows everything added to the known exploited vulnerabilities list since March 2024 that is not labeled as a Microsoft vendored product

jq -r '.vulnerabilities[] | select(.dateAdded >= "2024-03-01") | [.cveID, .dateAdded, .vendorProject, .vulnerabilityName] | @tsv' < kev.json | rg -i -v microsoft

You can even sort by CVE ID rather than date added to kev by piping sort at the end of this

jq -r '.vulnerabilities[] | select(.dateAdded >= "2024-03-01") | [.cveID, .dateAdded, .vendorProject, .vulnerabilityName] | @tsv' < kev.json | rg -i -v microsoft | sort

To explore a single vulnerability and it’s details we can use this jq command:

jq -r '.vulnerabilities[] | select(.cveID == "CVE-2023-27524") | to_entries | .[] | "\(.key): \(.value)"' kev.json

cveID: CVE-2023-27524
vendorProject: Apache
product: Superset
vulnerabilityName: Apache Superset Insecure Default Initialization of Resource Vulnerability
dateAdded: 2024-01-08
shortDescription: Apache Superset contains an insecure default initialization of a resource vulnerability that allows an attacker to authenticate and access unauthorized resources on installations that have not altered the default configured SECRET_KEY according to installation instructions.
requiredAction: Apply mitigations per vendor instructions or discontinue use of the product if mitigations are unavailable.
dueDate: 2024-01-29
knownRansomwareCampaignUse: Unknown
notes: https://lists.apache.org/thread/n0ftx60sllf527j7g11kmt24wvof8xyk

Leveraging the docker API for fast demo results

This python script will allow you to generate an SBOM for the top n images by pull count on dockerhub (n is 20 here). It saves them under a folder called (popular_sbom). There might be some errors produced if the image manifest doesn’t have an image pushed for your local computer cpu architecture:

#!/usr/bin/env python3
# usage: ./popular-docker.py -n 100
# generates an sbom for the top 100 images in docker hub and places them at popular_sbom/
import os
import requests
import subprocess
import json
import argparse

def get_latest_tag(repository):
    try:
        response = requests.get(f"https://hub.docker.com/v2/repositories/library/{repository}/tags/?page_size=1")
        if response.status_code == 200:
            data = response.json()
            tags = data.get("results", [])
            if tags:
                return tags[0]["name"]
        print(f"Failed to retrieve tags for repository '{repository}'. Using 'latest' tag.")
        return "latest"
    except Exception as e:
        print(f"Error retrieving tags for repository '{repository}': {e}")
        return "latest"

def generate_sbom(repository, tag):
    try:
        image_with_tag = f"{repository}:{tag}"
        # Run syft command to generate SBOM
        output = subprocess.check_output(["syft", "-o", "json", image_with_tag], stderr=subprocess.STDOUT, text=True)
        sbom_data = json.loads(output)
        return sbom_data
    except subprocess.CalledProcessError as e:
        print(f"Error generating SBOM for image {repository}:{tag}: {e.output}")
        return None

def save_sbom(repository, tag, sbom_data):
    if sbom_data:
        folder_name = "popular_sbom"
        if not os.path.exists(folder_name):
            os.makedirs(folder_name)
        file_name = os.path.join(folder_name, f"{repository}_{tag}_sbom.json")
        with open(file_name, "w") as f:
            json.dump(sbom_data, f, indent=2)
        print(f"SBOM saved to {file_name}.")
    else:
        print(f"SBOM data is invalid. Cannot save to file.")

def list_top_images(n=10):
    url = "https://hub.docker.com/v2/repositories/library/?page_size=100"
    response = requests.get(url)
    if response.status_code != 200:
        print("Failed to fetch data from Docker Hub.")
        return
    
    data = response.json()
    results = data.get("results", [])
    
    # Extract image names and pull counts
    image_pull_counts = {}
    for result in results:
        image_name = result.get("name")
        pull_count = result.get("pull_count", 0)
        if image_name:
            image_pull_counts[image_name] = pull_count
    
    # Sort images by pull counts and list top n images
    top_images = sorted(image_pull_counts.items(), key=lambda x: x[1], reverse=True)[:n]
    for i, (repository, _) in enumerate(top_images, 1):
        print(f"Finding latest tag for repository {repository}...")
        latest_tag = get_latest_tag(repository)
        print(f"Generating SBOM for image {repository}:{latest_tag}...")
        sbom_data = generate_sbom(repository, latest_tag)
        save_sbom(repository, latest_tag, sbom_data)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Generate SBOMs for top Docker images")
    parser.add_argument("-n", "--number", type=int, default=10, help="Number of top Docker images to process")
    args = parser.parse_args()

    list_top_images(args.number)

After using the above python script you can use the following bash script to check each sbom from the top 20 images

#!/bin/bash
# parse_popular_sbom.sh
folder_path=popular_sbom

# Check if the folder exists
if [ -d "$folder_path" ]; then
    # Loop through each file in the folder
    for file in $folder_path/*.json; do
        echo "Processing SBOM file: $file"
        cat "$file" | grype --by-cve | ./kev-bootstrap.py kev.json
    done
else
    echo "Folder '$folder_path' not found."
fi

Note: To use the above script you need the kev-bootstrap.py locally which filters the results. It needs a local path to the kev.json downloaded earlier in this post.

#!/usr/bin/env python
# kev-bootstrap.py
import sys
import json
import re
import argparse

def load_kev_data(file_path):
    with open(file_path) as kev_file:
        return json.load(kev_file)

def main(kev_file):
    # Load kev.json
    kev_data = load_kev_data(kev_file)

    # Read each line from standard input
    for line in sys.stdin:
        # Extract CVE ID from the line
        cve_match = re.search(r'\bCVE-\d{4}-\d{4,}\b', line)
        if cve_match:
            cve_id = cve_match.group(0)
            # Check if CVE ID exists in kev_data
            if any(cve_id == vulnerability["cveID"] for vulnerability in kev_data["vulnerabilities"]):
                print(line.strip())

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Filter lines containing CVE IDs based on KEV data")
    parser.add_argument("kev_file", help="Path to the KEV file")
    args = parser.parse_args()

    main(args.kev_file)

The final result should look like this:

./results.sh
Processing SBOM file: popular_sbom/alpine_latest_sbom.json
Processing SBOM file: popular_sbom/busybox_uclibc_sbom.json
Processing SBOM file: popular_sbom/debian_unstable-slim_sbom.json
Processing SBOM file: popular_sbom/golang_alpine3.20_sbom.json
Processing SBOM file: popular_sbom/httpd_alpine3.20_sbom.json
Processing SBOM file: popular_sbom/mariadb_noble_sbom.json
stdlib            go1.18.2                              go-module  CVE-2023-44487  High
Processing SBOM file: popular_sbom/memcached_alpine3.20_sbom.json
Processing SBOM file: popular_sbom/mongo_8.0.0-rc6-jammy_sbom.json
stdlib               go1.18.2                                   go-module  CVE-2023-44487  High
Processing SBOM file: popular_sbom/mysql_oraclelinux9_sbom.json
stdlib        go1.18.2                                 go-module  CVE-2023-44487  High
Processing SBOM file: popular_sbom/nginx_stable-alpine3.19-perl_sbom.json
Processing SBOM file: popular_sbom/node_lts-alpine3.20_sbom.json
Processing SBOM file: popular_sbom/postgres_alpine3.20_sbom.json
Processing SBOM file: popular_sbom/python_3.9.19-alpine3.20_sbom.json
Processing SBOM file: popular_sbom/rabbitmq_3.12.14-management-alpine_sbom.json
Processing SBOM file: popular_sbom/redis_7.0.15-alpine3.20_sbom.json
stdlib         go1.18.2              go-module  CVE-2023-44487  High
Processing SBOM file: popular_sbom/ubuntu_rolling_sbom.json
Processing SBOM file: popular_sbom/wordpress_beta-php8.3-fpm-alpine_sbom.json

To view more information about why the SBOM for that image reported a vulnerability found in the kev data you can do this command:

cat popular_sbom/redis_7.0.15-alpine3.20_sbom.json | grype -o json --by-cve | grype explain --id CVE-2023-44487
CVE-2023-44487 from nvd:cpe (High)
The HTTP/2 protocol allows a denial of service (server resource consumption) because request cancellation can reset many streams quickly, as exploited in the wild in August through October 2023.
Matched packages:
    - Package: stdlib, version: go1.18.2
      PURL: pkg:golang/stdlib@1.18.2
      Match explanation(s):
          - nvd:cpe:CVE-2023-44487 CPE match on `cpe:2.3:a:golang:go:1.18.2:-:*:*:*:*:*:*`.
      Locations:
          - /usr/local/bin/gosu
URLs:
    - https://nvd.nist.gov/vuln/detail/CVE-2023-44487

In this case grype found the binary gosu in the image which was compiled by a version of go subject to that HTTP/2 protocol vulnerability.

It would be up to the user of this image if this vulnerability was applicable or not.

This method can be repeated against different collections of images for different results. In this case we only found binary compiled with a vulnerable version of go. The actual exploitability of these artifacts is unknown, but on first glance seems very low.

Doing analysis for popular-docker.py -n 50

After expanding the net a bit more to pull in more images I came up with a short list from the top 50 images that might be good for demo material. The format for the SBOM file is __sbom.json

So for the first SBOM the docker command to grab the image would be:

docker pull cassandra:4.0.13-jammy

Here are the results compared with the kev.json findings:

Processing SBOM file: popular_sbom/cassandra_4.0.13-jammy_sbom.json
stdlib                 go1.18.2                                   go-module     CVE-2023-44487  High

Processing SBOM file: popular_sbom/couchbase_latest_sbom.json
golang.org/x/net                                               v0.10.0                             0.17.0            go-module     CVE-2023-44487       High
google.golang.org/grpc                                         v1.55.0                             1.56.3            go-module     CVE-2023-44487       High

Processing SBOM file: popular_sbom/jenkins_2.60.3_sbom.json
jenkins-core              2.60.3                      2.138.4                            java-archive    CVE-2018-1000861     Critical
libfreetype6              2.6.3-3.2                   2.6.3-3.2+deb9u2                   deb             CVE-2020-15999       Medium
script-security           1.13                        1.54                               jenkins-plugin  CVE-2019-1003029     Critical
spring-beans              2.5.6.SEC03                 5.2.20.RELEASE                     java-archive    CVE-2022-22965       Critical
spring-webmvc             2.5.6.SEC03                 5.2.20.RELEASE                     java-archive    CVE-2022-22965       Critical

Processing SBOM file: popular_sbom/joomla_php8.3-fpm_sbom.json
redis                      6.0.2                                      php-pecl  CVE-2022-0543     Critical

Processing SBOM file: popular_sbom/mariadb_noble_sbom.json
stdlib            go1.18.2                              go-module  CVE-2023-44487  High

Processing SBOM file: popular_sbom/mongo_8.0.0-rc6-jammy_sbom.json
stdlib               go1.18.2                                   go-module  CVE-2023-44487  High

Processing SBOM file: popular_sbom/mysql_oraclelinux9_sbom.json
stdlib        go1.18.2                                 go-module  CVE-2023-44487  High

Processing SBOM file: popular_sbom/percona_psmdb-6.0.6_sbom.json
stdlib               go1.19.1                                                      go-module  CVE-2023-44487  High


Processing SBOM file: popular_sbom/redis_7.0.15-alpine3.20_sbom.json
stdlib         go1.18.2              go-module  CVE-2023-44487  High

Processing SBOM file: popular_sbom/sentry_onbuild_sbom.json
Pillow               4.2.1                       10.0.1                             python  CVE-2023-4863        High

What makes this list interesting is every vulnerability found for the above sbom are from the known exploited vulnerabilities list.

The jenkins image is still in the 50 most popular images pulled but has this warning on their landing page

DEPRECATED; use "jenkins/jenkins:lts" instead

It’s easy to see why they recommend another image given the number of actively exploited vulnerabilities in this image scan, but that still doesn’t stop people from pulling the older image.

The above images, when filtered against the kev list give good insight into how anchore tools can be used to spot potential vulnerabilities being actively exploited in common images used by the general developer population

What about EPSS data?

EPSS or Exploit Prediction Scoring System is a data-driven effort for estimating the likelihood (probability) that a software vulnerability will be exploited in the wild.

You can find their data here along with the research, User Guide and information about their tooling.

If you have a version of their data locally here is a python script that can filter a grype table result to filter CVE based on their EPSS percentile. The percentile of a given score is based on the proportion of all scored vulnerabilities with the same or a lower EPSS score.

#!/usr/bin/env python
# example usage: epss-bootstrap.py
# cat popular_sbom/sentry_onbuild_sbom.json | grype --by-cve | ./epss-bootstrap.py epss_scores-2024-06-03.csv --percentile 0.98
import sys
import csv
import re
import argparse

def load_epss_data(file_path):
    with open(file_path) as epss_file:
        return {row['cve']: {'epss': float(row['epss']), 'percentile': float(row['percentile'])} for row in csv.DictReader(epss_file)}

def main(epss_file, percentile_threshold):
    # Load epss_scores-2024-06-03.csv
    epss_data = load_epss_data(epss_file)

    # Read each line from standard input
    for line in sys.stdin:
        # Extract CVE ID from the line
        cve_match = re.search(r'\bCVE-\d{4}-\d{4,}\b', line)
        if cve_match:
            cve_id = cve_match.group(0)
            # Retrieve entry from epss_data
            entry = epss_data.get(cve_id)
            if entry is not None:
                if entry['percentile'] >= percentile_threshold:
                    # Print the line along with the EPSS score
                    print(f"{line.strip()} EPSS Score: {entry['epss']}; EPSS Percentile: {entry['percentile']}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Filter lines containing CVE IDs based on EPSS data")
    parser.add_argument("epss_file", help="Path to the EPSS file")
    parser.add_argument("--percentile", type=float, default=0.98, help="Percentile threshold (default: 0.98)")
    args = parser.parse_args()

    main(args.epss_file, args.percentile)

We can use this script in combination with our popular_sbom folder to check for very severe vulnerabilities in the top 100 images in docker:

# generate our sbom for the top 100 images on dockerhub by pulls
./popular-docker.py -n 100

#!/bin/bash

folder_path="popular_sbom"

# Check if the folder exists
if [ -d "$folder_path" ]; then
    # Loop through each file in the folder
    for file in "$folder_path"/*.json; do
        echo "Processing SBOM file: $file"
        cat "$file" | grype --by-cve | ./epss-bootstrap.py epss_scores-2024-06-03.csv --percentile 0.995
    done
else
    echo "Folder '$folder_path' not found."
fi

This should process each sbom, generate a vulnerability report, and only print lines where the CVE-ID has an EPSS percentile in the 0.995th. Users can tweak that number to expand or contract the results based on the risk they want to simulate a given environment would tolerate for images in production.

Here is a snapshot of CVE for the top 100 images on dockerhub that meet that 0.995th percentile marker for EPSS data downloaded on 2024-06-03

./popular-docker-results-epss.fish
Processing SBOM file: popular_sbom/centos_centos7.9.2009_sbom.json
nss                          3.53.1-3.el7_9                                   rpm   CVE-2014-3566     High EPSS Score: 0.97488; EPSS Percentile: 0.99974
nss-sysinit                  3.53.1-3.el7_9                                   rpm   CVE-2014-3566     High EPSS Score: 0.97488; EPSS Percentile: 0.99974
nss-tools                    3.53.1-3.el7_9                                   rpm   CVE-2014-3566     High EPSS Score: 0.97488; EPSS Percentile: 0.99974

Processing SBOM file: popular_sbom/django_onbuild_sbom.json
libmysqlclient-dev            5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
libmysqlclient18              5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
libpq-dev                     9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
libpq5                        9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
mysql-client                  5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
mysql-client-5.5              5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
mysql-common                  5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
postgresql-client-9.4         9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975

Processing SBOM file: popular_sbom/glassfish_latest_sbom.json
libnss3                  2:3.17.2-1.1+deb8u2         2:3.26-1+debu8u1                  deb           CVE-2015-4000     Medium EPSS Score: 0.97405; EPSS Percentile: 0.99927

Processing SBOM file: popular_sbom/hipache_0.3.1_sbom.json
redis-server          2:2.8.4-2                         (won't fix)                                 deb   CVE-2022-0543        Medium EPSS Score: 0.97114; EPSS Percentile: 0.99791
redis-tools           2:2.8.4-2                         (won't fix)                                 deb   CVE-2022-0543        Medium EPSS Score: 0.97114; EPSS Percentile: 0.99791
sudo                  1.8.9p5-1ubuntu1.2                (won't fix)                                 deb   CVE-2021-3156        High EPSS Score: 0.96952; EPSS Percentile: 0.99727
supervisor            3.0b2-1                           3.0b2-1ubuntu0.1                            deb   CVE-2017-11610       Medium EPSS Score: 0.9745; EPSS Percentile: 0.99951

Processing SBOM file: popular_sbom/httpd_alpine3.20_sbom.json
httpd          2.4.59                binary  CVE-2007-0450   Medium EPSS Score: 0.97307; EPSS Percentile: 0.99872

Processing SBOM file: popular_sbom/jenkins_2.60.3_sbom.json
commons-beanutils         1.8.3                       1.9.2                              java-archive    CVE-2014-0114        High EPSS Score: 0.97314; EPSS Percentile: 0.99876
jenkins-core              2.60.3                      2.138.4                            java-archive    CVE-2018-1000861     Critical EPSS Score: 0.9732; EPSS Percentile: 0.99878
spring-beans              2.5.6.SEC03                 5.2.20.RELEASE                     java-archive    CVE-2022-22965       Critical EPSS Score: 0.97483; EPSS Percentile: 0.99972
spring-webmvc             2.5.6.SEC03                 5.2.20.RELEASE                     java-archive    CVE-2022-22965       Critical EPSS Score: 0.97483; EPSS Percentile: 0.99972

Processing SBOM file: popular_sbom/joomla_php8.3-fpm_sbom.json
redis                      6.0.2                                      php-pecl  CVE-2022-0543     Critical EPSS Score: 0.97114; EPSS Percentile: 0.99791

Processing SBOM file: popular_sbom/nuxeo_latest_sbom.json
log4j-core                             2.11.1                        2.12.2                        java-archive  CVE-2021-45046       Critical EPSS Score: 0.97363; EPSS Percentile: 0.99902
log4j-core                             2.11.1                        2.12.2                        java-archive  CVE-2021-44228       Critical EPSS Score: 0.97559; EPSS Percentile: 0.99998
log4j-core                             2.11.1                        2.12.3                        java-archive  CVE-2021-45105       Medium EPSS Score: 0.96625; EPSS Percentile: 0.99629
xstream                                1.4.10                        1.4.18                        java-archive  CVE-2021-39144       High EPSS Score: 0.97191; EPSS Percentile: 0.99821
xstream                                1.4.10                        1.4.14-jdk7                   java-archive  CVE-2020-26217       High EPSS Score: 0.97384; EPSS Percentile: 0.99912

Processing SBOM file: popular_sbom/owncloud_9-fpm_sbom.json
libpq-dev                9.6.10-0+deb9u1                                                deb           CVE-2019-9193        Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
libpq5                   9.6.10-0+deb9u1                                                deb           CVE-2019-9193        Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
pear/archive_tar         1.4.1                       1.4.11                             php-composer  CVE-2020-28949       High EPSS Score: 0.96283; EPSS Percentile: 0.99538
redis                    3.1.6                                                          php-pecl      CVE-2022-0543        Critical EPSS Score: 0.97114; EPSS Percentile: 0.99791

Processing SBOM file: popular_sbom/piwik_latest_sbom.json
apache2                  2.4.25-3+deb9u5             2.4.25-3+deb9u11                   deb           CVE-2021-40438       Critical EPSS Score: 0.97435; EPSS Percentile: 0.99943
apache2                  2.4.25-3+deb9u5             2.4.25-3+deb9u7                    deb           CVE-2019-0211        High EPSS Score: 0.97417; EPSS Percentile: 0.99933
apache2-bin              2.4.25-3+deb9u5             2.4.25-3+deb9u11                   deb           CVE-2021-40438       Critical EPSS Score: 0.97435; EPSS Percentile: 0.99943
apache2-bin              2.4.25-3+deb9u5             2.4.25-3+deb9u7                    deb           CVE-2019-0211        High EPSS Score: 0.97417; EPSS Percentile: 0.99933
apache2-data             2.4.25-3+deb9u5             2.4.25-3+deb9u11                   deb           CVE-2021-40438       Critical EPSS Score: 0.97435; EPSS Percentile: 0.99943
apache2-data             2.4.25-3+deb9u5             2.4.25-3+deb9u7                    deb           CVE-2019-0211        High EPSS Score: 0.97417; EPSS Percentile: 0.99933
apache2-utils            2.4.25-3+deb9u5             2.4.25-3+deb9u11                   deb           CVE-2021-40438       Critical EPSS Score: 0.97435; EPSS Percentile: 0.99943
apache2-utils            2.4.25-3+deb9u5             2.4.25-3+deb9u7                    deb           CVE-2019-0211        High EPSS Score: 0.97417; EPSS Percentile: 0.99933
libphp                   7.1.22                                                         binary        CVE-2019-11043       Critical EPSS Score: 0.97447; EPSS Percentile: 0.9995
libphp                   7.1.22                                                         binary        CVE-2018-19518       High EPSS Score: 0.9687; EPSS Percentile: 0.99704
pear/archive_tar         1.4.3                       1.4.11                             php-composer  CVE-2020-28949       High EPSS Score: 0.96283; EPSS Percentile: 0.99538
php-cli                  7.1.22                                                         binary        CVE-2019-11043       Critical EPSS Score: 0.97447; EPSS Percentile: 0.9995
php-cli                  7.1.22                                                         binary        CVE-2018-19518       High EPSS Score: 0.9687; EPSS Percentile: 0.99704
redis                    3.1.6                                                          php-pecl      CVE-2022-0543        Critical EPSS Score: 0.97114; EPSS Percentile: 0.99791

Processing SBOM file: popular_sbom/rails_onbuild_sbom.json
libmysqlclient-dev            5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
libmysqlclient18              5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
libpq-dev                     9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
libpq5                        9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
mysql-client                  5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
mysql-client-5.5              5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
mysql-common                  5.5.53-0+deb8u1                                               deb     CVE-2012-5613     Negligible EPSS Score: 0.96437; EPSS Percentile: 0.9958
postgresql-client-9.4         9.4.9-0+deb8u1                                                deb     CVE-2019-9193     Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975

Processing SBOM file: popular_sbom/sentry_onbuild_sbom.json
libpq-dev            9.6.15-0+deb9u1                                                deb     CVE-2019-9193        Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975
libpq5               9.6.15-0+deb9u1                                                deb     CVE-2019-9193        Negligible EPSS Score: 0.9749; EPSS Percentile: 0.99975

[popey]

@spiffcs - wonderful research! Thank you!

I’m gonna pull together these suggestions and rework my video suggestion list and get working on a couple to see if I’m on the right track!