Hi,
with regard to your question…
One question I can’t figure out is, do each of the core Job Executor threads actively “poll” the database for a task?
No (At least not in the Tomcat case and I believe its the same pattern for other app servers). Each job executor has a single job acquisition thread. The logic is count number of ready jobs → lock and acquire a subset → dispatch jobs to the job executor thread pool → repeat.
Thus the analogy I use is the job executor is a lot like a steam train. The executor thread pool is like the boiler, the jobs to execute table is like the coal supply, the job acquisition thread is like the engineer shoveling coal.
The (Tomcat) detail is as follows. Provided there are jobs ready to run, the job executor will keep locking a subset (jobs per acquisition cycle) and dispatch to the executor thread pool. When a job is dispatched to the thread pool, the next free thread will execute the job. If there are no free threads, the job will be queued (in memory). If max jobs are queued, the acquisition thread will back off. Thus the size of jobs acquired each acquisition cycle is a critical factor - too large and you get blocked by the thread pool, too small and you throttle throughput via repeated round trips to the DB. Note in a clustered environment job acquisition threads may compete with each other. Thus if the job acquisition thread cannot lock all the jobs it requests, it assumes it was competing against another node and thus backs off to avoid getting in lockstep with another node.
As an aside, if DB tier is saturated and you cant vertically scale, there are large production instances using a sharded architecture. For example Zalando use a sharded architecture with something like eight shards…
regards
Rob