I also deleted all tokens that were sitting on http-connector service tasks but the other jobs did not get picked up by the executor.
Okayā¦
So get it working, but unclear what was causing the problem.
I manually deleted all process instances that were running across all definitions except for a few instances that were waiting at the blank start event.
The executor still did not pick up the jobs.
So i restarted the container and upon restart camunda picked-up the jobs and executed them without issues.
Running curl or wget from the camunda container to the endpoints that http-connector was connecting to, results in zero issues; everything connects quickly and as expected.
Most of the tokens that were waiting were in Receive Task and Event Gateways. But this may not be relevant given that the executor did not resolve the problem until server restart. Meaning that the http-connector service tasks could have been causing the problem but there was no indicator of resolution until server restart.
@thorben if jobs are in flight, and the process instance is paused/suspended, does the active job get cancelled or would it remain in queue until completion?
@thorben Additional question:
When you delete an active process instance that has a job that is currently executing; Does the Job get cancelled mid execution?
Edit: Did some tests. Looks like if a Job is being executed and the process instance is deleted the job will not be cancelled. I tested this by creating a load of jobs and the HTTP-connector endless response issue occurred (i think). And i deleted some potentially problematic process instances, but the job executor did not pick up the changes. Once i restarted the container, the job executor went back to normal
Edit: Have narrowed the problem down to be related to http-connector requests. Further testing tomorrow related to outside network issues (likely).
@thorben can the httpclient configuration settings be accessed within the connector variable of http-connector? For example using a listener or a script in one of the inputs to modify the timeouts? I looked through the docs, but I could not clearly see where the access could be.
Hey Stephen,
It runs until completion and then fails with an OptimisticLockingException since the job and execution tree was updated in the meantime.
No, same optimistic locking behavior as above.
I donāt think so, but havenāt really looked into this. The config I linked to above is used when the client is built. This only happens once, so you canāt configure the timeout on per-request basis this way. But maybe thereās another way with Apache Http client to do this.
@thorben okay great. We have narrowed it down then to specific use case network connectivity (only re-creatable under certain load conditions cause by camunda.
Any ideas on where to look for potentially modifying the per-request timeouts?
Im still puzzled, shouldnt there be a lock visible in the job table?
R
Few items that came about this as changes / needs for the engine
-
Doing REST API calls using GET /job or GET /job/:id / when listing jobs, should have a āLockedā property so you can see which jobs are currently locked by the engine / are currently being processed. (https://app.camunda.com/jira/browse/CAM-8192)
-
Would be really nice to have a input parameter for HTTP-Connector to set a timeout. By Default it should be set to something like 60 seconds, and you can use the input parameter to override the timeout to a custom value. (https://app.camunda.com/jira/browse/CAM-8191)
-
@camunda is there any specific reasons for allowing jobs to ārun foreverā ? and not having a forced default time limit on a jobās execution?
-
We used a third party API-gateway to force the timeout rather than make changes to HTTP-Connector. But making changes to HTTP Connector appears to be the better longer term option.
-
@camunda is there documentation about a Job being deleted, but it remains in the executor until restart? (aka deleting a in-flight job does not cancel it).
-
Adding a
response timereturn variable into http-connector would be very helpful. This would provide the response time it took for the request.
Feel free to raise a feature request. Instead of a boolean property, this should expose the current lock expiration time.
Feel free to raise a feature request.
Itās not possible to safely interrupt a thread in Java that is stuck in an infinite loop, waiting forever, etc.
To rephrase it: you would like to immediately cancel a job when it gets deleted (correct me if Iām wrong). Then the answer is the same as for question 3. Not doable, especially not in cluster setups.
Not sure if this is generally useful. What would you do with that variable in your example?
The response times on http-connector was more of a side feature that is useful for generating related incidents / indicators of issues. (If response times start to get too long it can be easily logged ro later intervention/review).
Would this scenario not warrant a wrapper around a job so that infinite loops, waiting forever, etc cannot occur? So that even if a loop does occur as a job, the higher level abstraction will timeout.
Feature requests raised
(https://app.camunda.com/jira/browse/CAM-8192)
(https://app.camunda.com/jira/browse/CAM-8191)
If you have a concrete idea, please provide some code. I donāt know how to build such a thing in a good way.
Note that the jobās lock expiration time is already such a timeout. This ensures that the job will be triggered again, but it does not cancel the ongoing execution. So if you have an infinite loop in a JavaDelegate or similar, all the job executorās threads will be busy after some time.
Thanks @thorben. may have some time in the fall to have someone look at this.
@thorben 4 Year bug! @StephenOTT put in the time to debug and even submit features requests for solutions which were ignored.
Might I ask why it has been ignored for 4 years when many people have reported this as a problem?
As Stephen says in this thread, EVERY TASK stops working and the only solution I have found is a complete wipe and reload.
I personally have had to completely wipe and reload Camunda about 4 times in the past year we have tried to evaluate Camunda as a BPM solution, however Iām sure you would agree that would be hard decision for anyone to move forward with a solution having that bad of a track record.
For a platform that is supposed to be a rock-solid at enforcing business processes and rules that requires reloading ever few months falls quite short of a reliable platform.
Below are just the first few post of a search for āJob Executor Stuckā forum search.
@GChester1 let us calm down a bit hereā¦ the issue appears related to the http connector, and clearly the connector and connectors in general have been given a very low priory for maintenance or new development by Camunda (the company), but nothing has stopped others from sourcing and fixing the problem: no one in the community has fixed it either.
There are multiple solutions to the problem at the community level, and there is always paid support you can get from Camunda or other providers.
Lots of options to solve your problem.
No need to be singling out developers.
5 Likes
@thorben was clearly the person who was working on this before and Iām simply asking why it has been allowed to go this long unsolved.
I thought I handled it quite well, after wasting an entire year of my life evaluating this product.
When you have wasted the amount of time and resources for a bug that has been ignored for 4 years, yes there is a bit to get upset about.
If he is not the person who works on the Camunda team, then my apologies.
On the other hand, if he is on the Camunda team, he seems like the best place to start on the mystery of why this issue was dropped.
Hello @GChester1,
As of the time of writing, the platform has 929 open feature requests and 867 open bug reports (2504 and 2646 have been implemented and fixed over the years). We have to carefully prioritize our work and choose those tickets that matter most to us. https://jira.camunda.com/browse/CAM-8191 hasnāt been part of that so far.
With this experience and that you were not able to resolve this problem in another way (using external tasks, using a custom JavaDelegate, using another HTTP client library like JSoup as Stephen suggests, contributing a bug fix because Camunda is open-source), I honestly think that Camunda is not the right technology for you.
The job executor is not a closed system and can encounter various problem, sometimes related to Camunda components, sometimes to the code/systems our users integrate. Users need to understand its workings a bit and also how to diagnose problems. For that purpose, I have published an extensive blogpost in the past: The Job Executor: What Is Going on in My Process Engine? | Camunda. Maybe it will help you in the future.
I hope none of this comes across as aggressive, itās just my honest plain view on your comments.
Cheers,
Thorben
3 Likes
Iām not sure you did handle this particularly wellā¦ Thereās no need to take out your frustration on someone else, especially someone who really canāt be blamed for you own decision making.
An issue for sure, is that you apparently spent a year evaluating Camunda and somehow never came across the fact that we donāt suggest using connectors in a variety of scenarios. That could be your fault for not being able to google for our docs or ours for not being clearer about getting the message out there. Connectors are very basic and there are much better options out there - depending on what youāre looking to do. So consider a different solution.
1 Like
I could not agree more that bugs need to be prioritized.
I would have thought an issue that is causing quite a few people problems which is verifiable by a simple search of the forums would justify as at least a bit of priority. A simple search produced 227 results
This bug Shuts down the entire Executor, in many cases never to start again and people complain about it over and over in the forums.
Possibly could you guys have stayed teamed up with the original Activiti group if you are so far behind?
Awww, yes, Iām quite familiar with that document. Here is even a thread I posted almost 4 months ago trying to work through how to enable the debugger.
I might point out this thread is still unanswered to this day.
Iām so familiar with that Support Document after reading it at least a dozen times that I could probably give a Public talk on it.
It has never really helped us solve the issue, so if you guys are under the impression it is a work-around solution for everyone, I not sure it is.
Actually @Niall Your assumption could not be farther from the truth. As I just stated above, Iāve spent months trying to debug this problem with Camunda and hour upon hours searching Support Documents, Forums Posts, StackOverflow, Github, and about everything else I could think of. As mentioned, many of my posts about the subject have gone unanswered.
I have never heard of another option other than the JSoup. I originally did not want to use JSoup since it was an external resource; not internal to Camunda.
Adding more complexity to use External Resources only complicates platforms even more.
A simple search on any of the reviews sites will tell you users think Camunda is Overly Complex already so I wanted to keep things simple and internal.
I assume just about every single person who uses Camunda needs to connect to an external APIs, I wanted to use an internal feature of Camunda thinking it would be more stable. So I must have been wrong there.
@Niall @StephenOTT We are simply going to have to agree to disagree. I have even reread my post and I think it made perfect sense to go back to the source and deal with the person who originally helped diagnose the problem. I did not blame anyone, I simply ask questions of why this LARGE BUG has been ignored, when all my other posts were practically ignored, kinda like the original bug.
On the other hand, you guys seem quick to blame me and not take responsibility for ignoring a HUGE BUG that has cost so many of your loyal followers hours, days, and years of their lives.
@thorben did not seem offended in his response and no offense was meant, though I was trying to be direct.
Obviously, it worked and got your attention.
Please address the gaping wound, stop the bleeding of your loyal users and fix the Job Executor Bug!
It seems so simple, @StephenOTT is quite talented and has done the hard work.
Hi @GChester1,
Thanks for your extensive feedback. I think most points have been made in this discussion. I disagree with some statements of your last post, however I wonāt argue with you as I have the impression that this will lead us nowhere.
Our decision to not prioritize https://jira.camunda.com/browse/CAM-8191 remains unchanged for now, but we will keep your points in mind for future planning. If youād like to contribute a fix, we are happy to help.
Itās correct that I am not offended by your posts. That said, your use of language comes across as aggressive and does not help your concern.
I wish you all the best.
Cheers,
Thorben
1 Like
It got you guys to finally address the issue instead of ignoring like my posts over the last half-year; so I got what I wanted.
But of course, there will never be any admission of non-response or fault from the other side.
However, Iāll point out there was still no action taking, just a bunch of chest-puffing and posturing.
Iām not a developer anymore and just run the company and if that were not enough, I just do not feel yo u guys have provided enough details to dive into the Job Executor to fix this bug.
On the other hand dealing with Camunda the past year has really let us see the future of Camunda, but we already know the history quite well as I have done my homework.
Iāve seen project after project for over 30 years do just as the Activiti project did. Piwik just to name one, Redmine and many others. It is usually over greed and/or arrogance.