Problem :
Launching a JVM I have the message : "Cannot create GC thread. Out of system resources"
- Enough memory
- Enough swap
- Enough ulimit
- Enough
threads-max Enough CPU
Event extend the PID limit...
Important (at the end) : debian version = 10.11
Solution :
After a hours of googling, I found :
- https://stackoverflow.com/questions/18078859/java-run-out-of-memory-issue
- Fix the number of CPU for GC (in case of many CPU available ... greater than 8)
- https://serverfault.com/questions/662992/java-on-linux-insufficient-memory-even-though-there-is-plenty-of-available-memor
- Check ulimit -u
- Check ulimit -Hu
- Check open files (https://shaarli.hoab.fr/?8yFkBw)
- https://www.cyberciti.biz/tips/howto-linux-increase-pid-limits.html
Extend PID max : cat /proc/sys/kernel/pid_max
But none of these solutions works and none was matching the number I had :
- number of open files < ulimit -n
- maximum process/tasks < ulimit -u
But in a thread, I found something that was working : UserTasksMax.
I'm running SystemD, I have around 10805 task running for my user.
And from : https://manpages.debian.org/stretch/systemd/logind.conf.5.en.html
UserTasksMax=
Sets the maximum number of OS tasks each user may run concurrently. This controls the TasksMax= setting of the per-user slice unit, see systemd.resource-control(5) for details. If assigned the special value "infinity", no tasks limit is applied. Defaults to 33%, which equals 10813 with the kernel's defaults on the host, but might be smaller in OS containers.
For my suspect PID (a lot of files) :
- cat /proc/21890/status | grep Thread => 1 thread
- ls /proc/21890/task | wc
- confirmed by the usual command : ps -eLf | grep calrisk | wc
I have around 10805 threads running for a given JVM very close to the limit.
Complete guide :
https://www.journaldufreenaute.fr/nombre-maximal-de-threads-par-processus-sous-linux/
Parameters not present in all man page, it could grown up to 12288 on latest version.
To be check !