Is there a way to limit the number of threads Syft will use? We are running Syft in a container, and sometimes we get the error:
failed to create new os thread (have 24 already; errno=11)
may need to increase max user processes (ulimit -u)
We’re running in a Kubernetes cluster we don’t fully control, so probably will not be able to up the ulimit in our containers.
Thanks!
You should be able to use the --parallelism <number> flag to limit the number of parallel goroutines, which should correlate to the number of threads that go spawns to support them.
Ah, thanks. I did see that option, but didn’t know if that correlated to the total number of threads. Will that limit the total number of threads, or just the number of additional threads spawned from the main worker? In other words, if my limit (as it appears) is 24 threads, should I do --parallelism 20 to leave room for the main thread and any other “control” threads?
The --parallellism flag will limit the total concurrent goroutines that we control. (There may be some libraries doing something outside of our control.) I suppose there could be at least 2 additional “control threads” – one for the main process and the GUI.
The number of system threads ends up entirely dependent on how the go runtime decides to allocate them to support the goroutine load. The good news is you can use the environment variable GOMAXPROCS if you really need to limit the number of system threads. I think we should set this if you specify --parallelism, but we aren’t doing that today.
The current behavior is to allow runtime.NumCPU() * 4 concurrent goroutines by default. This was chosen somewhat arbitrarily by testing on my local system, I’d be interested to know if there was some better default behavior!