Local testing GitHub Actions

[popey]

Last week, somewhere I recall someone (maybe @kzantow ) suggesting that there might be use in being able to iterate on, and debug GitHub actions locally, rather than wait for the GitHub runners. I suspect this was in the context of a bug, but this week’s brain has swapped out last weeks context, sorry.

Anyway, I just stumbled on act which may be of use. It’s a tool written in Go which does pretty much what you expect. It leverages your local Docker setup to spin up GHA-like containers in which to run the actions.

As a test I grabbed sbom-action to give it a shake. I’m using an M1 mac, so it recommended I add the --container-architecture linux/amd64 and pointed it at the test.yml action, thus:

act -W ./.github/workflows/test.yml --container-architecture linux/amd64

Any of you lot tried this before? It’s super quick (on my M3 Pro), feels pretty handy, and could speed up some iteration, assuming the tests work locally in docker.

Here’s the top of the output:

INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock'
[build-test/build                 ] 🚀  Start image=catthehacker/ubuntu:act-latest
[build-test/test-as-action        ] 🚀  Start image=catthehacker/ubuntu:act-latest
[build-test/test                  ] 🚀  Start image=catthehacker/ubuntu:act-latest
[build-test/test-on-fixture-dirs-2] 🚧  Skipping unsupported platform -- Try running with `-P ubuntu-latest=...`
[build-test/test-on-fixture-dirs-1] 🚀  Start image=catthehacker/ubuntu:act-latest
[build-test/build                 ]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform=linux/amd64 username= forcePull=true
[build-test/test-as-action        ]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform=linux/amd64 username= forcePull=true
[build-test/test-on-fixture-dirs-1]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform=linux/amd64 username= forcePull=true
[build-test/test                  ]   🐳  docker pull image=registry:2 platform=linux/amd64 username= forcePull=true
[build-test/test                  ]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform=linux/amd64 username= forcePull=true
[build-test/build                 ]   🐳  docker create image=catthehacker/ubuntu:act-latest platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host"

Here’s the bottom of the output:

|  PASS  tests/integration/GitHubSnapshot.test.ts
|   GitHub Snapshot
|     ✓ runs with default inputs (272 ms)
|
----------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------
| File                  | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                            
| ----------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------
| All files             |   88.11 |       75 |   91.42 |   88.33 |                                                                                              
|  src                  |     100 |      100 |     100 |     100 |                                                                                              
|   SyftVersion.ts      |     100 |      100 |     100 |     100 |                                                                                              
|  src/github           |      86 |    72.32 |   89.47 |    86.3 |                                                                                              
|   Executor.ts         |     100 |      100 |     100 |     100 |                                                                                              
|   GithubClient.ts     |   82.03 |    54.16 |   89.28 |   82.25 | 107-111,157,168-169,305,384,388-389,419-420,447,454-457,477-490                              
|   SyftGithubAction.ts |   87.89 |    77.27 |      92 |   88.18 | 65-66,124,134,148,193-194,205-210,220,246-247,289-290,420,429,433-436,509,535,545-546,563,590-593
|   Util.ts             |   66.66 |      100 |       0 |   66.66 | 4                                                                                            
|  tests                |    96.8 |       90 |   93.75 |   96.73 |                                                                                              
|   mocks.ts            |    96.8 |       90 |   93.75 |   96.73 | 90,155,208                                                                                   
| ----------------------|---------|----------|---------|---------|---------------------------------------------------------------------------------------------------
| Test Suites: 1 failed, 3 passed, 4 total
| Tests:       12 failed, 37 passed, 49 total
| Snapshots:   9 passed, 9 total
| Time:        18.588 s
| Ran all test suites.
[build-test/test                  ]   ❌  Failure - Main npm test
[build-test/test                  ]   ⚙  ::set-env:: ANCHORE_SBOM_ACTION_PRIOR_ARTIFACT=test-repo-a_job.cyclonedx.json
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ]   ⚙  ::add-path:: /tmp/actions/cache/syft/1.9.0/x64
[build-test/test                  ] exitcode '1': failure
[build-test/test                  ] 🏁  Job failed

A useful tool for the armoury? Anyone else tried this?

(I did have to patch the workflow I used above, because port 5000 is in use by some macOS thing):

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 89aae24..48e53a9 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -62,14 +62,14 @@ jobs:
       registry:
         image: registry:2
         ports:
-          - 5000:5000
+          - 5001:5000
     steps:
       - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
       - name: Build images
         run: |
           for distro in alpine centos debian; do
-            docker build -t localhost:5000/match-coverage/$distro ./tests/fixtures/image-$distro-match-coverage
-            docker push localhost:5000/match-coverage/${distro}:latest
+            docker build -t localhost:5001/match-coverage/$distro ./tests/fixtures/image-$distro-match-coverage
+            docker push localhost:5001/match-coverage/${distro}:latest
           done
       - run: npm ci
       - run: npm test

[willmurphy]

I had previously tried to set up act, and it seems to have remembered what images I was using, so I had to make a couple of changes to get this working. Specifically, I had to explicitly pass -P ubuntu-latest=catthehacker/ubuntu:act-latest or it was using to node:16-buster-slim which is both too old a Node and lacks the git cli.

I also had it hang on git diff and had to change the command to git --no-pager diff to prevent it from waiting for someone to page down in less I think.

Regarding port 5000, I think you might find that it’s actually a previous test run that is using port 5000, which is something we probably want to fix. @popey if you docker ps, is there a registry:2 image listening on 5000?

When running this locally, I usually see a successful test run followed by a cryptic failure. For example, Error: Job 'build' failed. (I’ve seen other job failures as well, such as Error: Job 'test-as-action' failed. I’m not sure what to make of these failures being printed later. Is that others’ experience with this test setup? It would be nice if it ran the whole workflow and exited zero

I was also trying to make local and GHA testing of sbom-action more similar, and had opened a small PR, but GHA runs the local registry as a service, rather than as an imperative part of the tests, which might be preferable for a reason I’m not thinking of.

[popey]

Would you believe, it’s AirPlay Receiver that’s taking port 5000!

vs