Hi @hieudvm, welcome to the forums! What you are running into is likely due to the exporter. There are several discussions in the forums around this (here’s one comprehensive thread). Essentially, when you query the API, you querying cold storage (by default Elasticsearch), which requires data to be exported to it first.
For a better, scalable solution, I would recommend one of two approaches:
- Use execution listeners to push the state of the process to your application. There are two downsides to this approach: first, you have to set the execution listener on every task you need to monitor; and second, they don’t support user tasks yet (support for user task listeners is coming in 8.8). However, while you’re currently using user tasks, you might be able to use different task types depending on your needs.
- Use a custom exporter that triggers events in your application based on the exported data.