Hi Michael,
For some general reading on cluster tuning, this blog post may be of interest.
Im not an expert on wildfly configuration, however I shall talk to some Tomcat concepts which may resonate. On tomcat, there are two distinct thread pools. One is for foreground requests, eg REST API calls or tasklist/cockpit initiated calls. The second pool is the job executor thread pool for background job execution or asynchronous continuations. Hence your foreground threadpool should be set large enough for the number of concurrent client requests you expect. 3000 request per minute is only 50 per second. Given you have 4 nodes, I would expect 25 foreground threads per server to be adequate.
Now both thread pools will use a common database connection pool. Hence the number of connections in the connection pool should be as small as possible, but large enough for both thread pools. Hence I would size it to match the sum of max threads in each pool.
To set the size of the job executor thread pool depends. If your background tasks are cpu intensive, keep it low - eg twice number of cpu cores. If you are making lots of network API calls and your threads are thus I/O bound, then increase the size of the job executor pool to be say 8 times CPU cores.
Next you need to tune the job acquisition strategy. In particular, if you tend to have a backlog of async jobs, then get more jobs per acquisition cycle. Perhaps use a number say half the number of max job executor threads.
These kinds of number would be my starting numbers, tuning can take a lot of experimentation, but as a rule of thumb, less is typically more.
You also need to ensure the DB tier is tuned for your usage scenario, but thats another forum…
regards
Rob