Broker pod will not start and logs contain Hazelcast Exporter errors

andyh · June 1, 2020, 4:44pm

Hi

I have the following Zeebe cluster running in our development environment:
Cluster size: 1
Partition Factor: 1
Replication Factor: 1

Zeebe version: 0.22.1

The cluster has both the ElasticSearch and Hazelcast exporters with Simple Monitor configured.

The Zeebe Cluster has been running fine for about 30 days but this morning the Zeebe Broker pod started failing. The logs report that the Hazelcast exporter cannot recover the partitions followed by a bunch of exception. Although the data isn’t that important the pod still hasn’t started in over 5 hours.

Any ideas what is causing there errors?

Thanks for your help,
Andy

2020-06-01 11:48:41.585 [] [hz._hzInstance_1_dev.cached.thread-7] INFO  com.hazelcast.internal.cluster.impl.MembershipManager - [10.244.2.75]:5701 [dev] [3.12.5] Mastership is claimed with: MembersView{version=19, members=[MemberInfo{address=[10.244.2.75]:5701, uuid=c6572eb2-dde5-44fb-aaa7-6703c9df98c5, liteMember=false, memberListJoinVersion=18}]}
2020-06-01 11:49:13.389 [] [hz._hzInstance_1_dev.cached.thread-6] INFO  com.hazelcast.transaction.TransactionManagerService - [10.244.2.75]:5701 [dev] [3.12.5] Committing/rolling-back live transactions of [10.244.2.64]:5701, UUID: 2e1080d9-d8b6-4a88-a04d-9c348eb58d1d
2020-06-01 11:49:33.807 [] [hz._hzInstance_1_dev.migration] INFO  com.hazelcast.internal.partition.InternalPartitionService - [10.244.2.75]:5701 [dev] [3.12.5] Fetching most recent partition table! my version: 7449
2020-06-01 11:49:47.964 [] [hz._hzInstance_1_dev.migration] INFO  com.hazelcast.internal.partition.InternalPartitionService - [10.244.2.75]:5701 [dev] [3.12.5] Most recent partition table version: 7449
2020-06-01 11:50:53.121 [] [hz._hzInstance_1_dev.migration] WARN  com.hazelcast.internal.partition.impl.MigrationManager - [10.244.2.75]:5701 [dev] [3.12.5] partitionId=0 is completely lost!
2020-06-01 11:51:19.640 [] [hz._hzInstance_1_dev.migration] WARN  com.hazelcast.internal.partition.impl.MigrationManager - [10.244.2.75]:5701 [dev] [3.12.5] partitionId=1 is completely lost!
2020-06-01 11:51:23.058 [] [hz._hzInstance_1_dev.migration] WARN  com.hazelcast.internal.partition.impl.MigrationManager - [10.244.2.75]:5701 [dev] [3.12.5] partitionId=2 is completely lost!

....

2020-06-01 12:06:38.803 [] [hz._hzInstance_1_dev.migration] WARN  com.hazelcast.internal.partition.impl.MigrationManager - [10.244.2.75]:5701 [dev] [3.12.5] partitionId=270 is completely lost!
2020-06-01 12:06:56.167 [] [hz._hzInstance_1_dev.IO.thread-in-2] INFO  com.hazelcast.nio.tcp.TcpIpConnection - [10.244.2.75]:5701 [dev] [3.12.5] Initialized new cluster connection between /10.244.2.75:36609 and /10.244.2.76:5701
2020-06-01 12:07:02.672 [] [hz._hzInstance_1_dev.cached.thread-9] INFO  com.hazelcast.internal.partition.InternalPartitionService - [10.244.2.75]:5701 [dev] [3.12.5] Remaining migration tasks in queue => 1. (lastRepartitionTime=Thu Jan 01 00:00:00 UTC 1970, completedMigrations=271, totalCompletedMigrations=271, elapsedMigrationTime=0ms, totalElapsedMigrationTime=0ms)
2020-06-01

java.util.concurrent.ExecutionException: com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> [10.244.2.76]:5701 over Connection[id=40, /10.244.2.75:34599->/10.244.2.76:5701, qualifier=null, endpoint=[10.244.2.76]:5701, alive=false, type=MEMBER]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:92) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:179) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.AbstractJoiner.sendSplitBrainJoinMessage(AbstractJoiner.java:264) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.AbstractJoiner.sendSplitBrainJoinMessageAndCheckResponse(AbstractJoiner.java:229) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:129) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227) [zeebe-hazelcast-exporter.jar:0.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80) [zeebe-hazelcast-exporter.jar:0.7.0]
Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> [10.244.2.76]:5701 over Connection[id=40, /10.244.2.75:34599->/10.244.2.76:5701, qualifier=null, endpoint=[10.244.2.76]:5701, alive=false, type=MEMBER]
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeRemote(Invocation.java:657) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:631) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:592) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:256) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.AbstractJoiner.sendSplitBrainJoinMessage(AbstractJoiner.java:262) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.AbstractJoiner.sendSplitBrainJoinMessageAndCheckResponse(AbstractJoiner.java:229) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:129) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at ------ submitted from ------.(Unknown Source) ~[?:?]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:126) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        ... 13 more
2020-06-01 14:48:06.204 [] [hz._hzInstance_1_dev.cached.thread-15] INFO  com.hazelcast.internal.cluster.impl.MembershipManager - [10.244.2.75]:5701 [dev] [3.12.5] Removing Member [10.244.2.76]:5701 - 5181e962-c86e-40db-a8bc-d65ad274437b
2020-06-01 14:48:06.355 [] [hz._hzInstance_1_dev.cached.thread-11] INFO  com.hazelcast.internal.partition.InternalPartitionService - [10.244.2.75]:5701 [dev] [3.12.5] Remaining migration tasks in queue => 2. (lastRepartitionTime=Mon Jun 01 12:51:55 UTC 2020, completedMigrations=23, totalCompletedMigrations=294, elapsedMigrationTi

0-06-01 15:04:24.130 [] [hz._hzInstance_1_dev.priority-generic-operation.thread-0] INFO  com.hazelcast.internal.cluster.impl.ClusterJoinManager - [10.244.2.75]:5701 [dev] [3.12.5] [10.244.2.76]:5701 should merge to us , both have the same data member count: 1
2020-06-01 15:04:29.527 [] [hz._hzInstance_1_dev.generic-operation.thread-0] INFO  com.hazelcast.internal.cluster.impl.ClusterJoinManager - [10.244.2.75]:5701 [dev] [3.12.5] [10.244.2.76]:5701 should merge to us , both have the same data member count: 1
2020-06-01 15:04:30.841 [] [hz._hzInstance_1_dev.cached.thread-9] INFO  com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager - [10.244.2.75]:5701 [dev] [3.12.5] System clock apparently jumped from 2020-06-01 15:00:45.301 to 2020-06-01 15:03:51.673 since last heartbeat (+181372 ms)
2020-06-01 15:04:20.607 [] [hz._hzInstance_1_dev.cached.thread-13] INFO  com.hazelcast.internal.partition.InternalPartitionService - [10.244.2.75]:5701 [dev] [3.12.5] Remaining migration tasks in queue => 2. (lastRepartitionTime=Mon Jun 01 12:51:55 UTC 2020, completedMigrations=23, totalCompletedMigrations=294, elapsedMigrationTime=1865294ms, totalElapsedMigrationTime=1865294ms)
2020-06-01 15:04:36.022 [] [hz._hzInstance_1_dev.cached.thread-9] WARN  com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager - [10.244.2.75]:5701 [dev] [3.12.5] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 181372 ms, Heartbeat-Timeout: 60000 ms
2020-06-01 15:03:38.565 [] [hz._hzInstance_1_dev.migration] ERROR com.hazelcast.internal.partition.impl.MigrationManager - [10.244.2.75]:5701 [dev] [3.12.5] Promotion commit to [10.244.2.75]:5701 - c6572eb2-dde5-44fb-aaa7-6703c9df98c5 failed for 22 migrations
java.util.concurrent.ExecutionException: PromotionCommitOperation got rejected before execution due to not starting within the operation-call-timeout of: 60000 ms. Current time: 2020-06-01 15:01:43.338. Start time: 2020-06-01 14:52:39.675. Total elapsed time: 575442 ms. Invocation{op=com.hazelcast.internal.partition.operation.PromotionCommitOperation{serviceName='hz:core:partitionService', identityHash=2052331847, partitionId=-1, replicaIndex=0, callId=-708, invocationTime=1591020079378 (2020-06-01 14:01:19.378), waitTimeout=-1, callTimeout=60000}, tryCount=2147483647, tryPauseMillis=500, invokeCount=2, callTimeoutMillis=60000, firstInvocationTimeMs=1591023159675, firstInvocationTime='2020-06-01 14:52:39.675', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[10.244.2.75]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=1, connection=null}
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:163) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:104) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:163) ~[zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionsToDestination(MigrationManager.java:1476) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionsToDestination(MigrationManager.java:1486) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionsToDestination(MigrationManager.java:1486) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionsToDestination(MigrationManager.java:1486) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionsToDestination(MigrationManager.java:1486) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.commitPromotionMigrations(MigrationManager.java:1360) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.promoteBackupsForMissingOwners(MigrationManager.java:1346) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationManager$RepairPartitionTableTask.run(MigrationManager.java:1286) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationThread.processTask(MigrationThread.java:122) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationThread.doRun(MigrationThread.java:98) [zeebe-hazelcast-exporter.jar:0.7.0]
        at com.hazelcast.internal.partition.impl.MigrationThread.run(MigrationThread.java:67) [zeebe-hazelcast-exporter.jar:0.7.0]
Caused by: com.hazelcast.core.OperationTimeoutException: PromotionCommitOperation got rejected before execution due to not starting within the operation-call-timeout of: 60000 ms. Current time: 2020-06-01 15:01:43.338. Start time: 2020-06-01 14:52:39.675. Total elapsed time: 575442 ms. Invocation{op=com.hazelcast.internal.partition.operation.PromotionCommitOperation{serviceName='hz:core:partitionService', identityHash=2052331847, partitionId=-1, replicaIndex=0, callId=-708, invocationTime=1591020079378 (2020-06-01 14:01:19.378), waitTimeout=-1, callTimeout=60000}, tryCount=2147483647, tryPauseMillis=500, invokeCount=2, callTimeoutMillis=60000, firstInvocationTimeMs=1591023159675, firstInvocationTime='2020-06-01 14:52:39.675', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[10.244.2.75]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=1, connection=null}

Philipp_Ossler · June 5, 2020, 4:55am

Hi @andyh,

thank you for sharing!

I’m not sure why Hazelcast has problems with its internal partitions. If you’re using just a single broker with a single partition then there is also just one Hazelcast instance in the cluster.

Can you reproduce the problem somehow?

If you’re running your Zeebe cluster for a longer time then you can also think of using an external Hazelcast cluster. The advantage is that it is independent and can be restarted or clustered differently.

Best regards,
Philipp

andyh · June 12, 2020, 6:39pm

Hi @philipp.ossler

Thanks for getting back to me.

After restarting the pod a few times things just started to work and we haven’t been able to reproduce the issue.

Would you have any documentation on how to configure an external hazelcast cluster?

Thanks
Andy

Philipp_Ossler · June 15, 2020, 4:08am

This section contains an example how to configure an external Hazelcast cluster: https://github.com/zeebe-io/zeebe-hazelcast-exporter#configuration

andyh · June 18, 2020, 10:54am

Hi @philipp.ossler

Thanks for the link.

After a few days of things works fine, today I have noticed in the logs a bunch of errors (see below).

I have also noticed that the workflow definitions that were deployed to Zeebe and visible in Simple Monitor are no longer visible and when I attempt to upload a new workflow definition Simple Monitor accepts the definition but after a refresh of the page the workflow does not appear. Although I cannot see the definitions I can see the workflow instances:

Attempt to terminate a workflow instance:

Attempt to view a workflow:

Any ideas why?

Thanks
Andy

Logs:

Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:415) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2020-06-18 10:27:06.799  INFO 1 --- [nio-8080-exec-7] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:415) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2020-06-18 10:27:18.244 ERROR 1 --- [nio-8080-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.samskivert.mustache.MustacheException$Context: No method or field with name 'resource' on line 10] with root cause

com.samskivert.mustache.MustacheException$Context: No method or field with name 'resource' on line 10
        at com.samskivert.mustache.Template.checkForMissing(Template.java:344) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValue(Template.java:247) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValueOrDefault(Template.java:292) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$VariableSegment.execute(Mustache.java:872) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:831) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.execute(Template.java:137) ~[jmustache-1.15.jar!/:na]
        at org.springframework.boot.web.servlet.view.MustacheView.renderMergedTemplateModel(MustacheView.java:80) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1373) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1118) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1057) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2020-06-18 10:27:18.248 ERROR 1 --- [nio-8080-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

com.samskivert.mustache.MustacheException$Context: No method or field with name 'trace' on line 10
        at com.samskivert.mustache.Template.checkForMissing(Template.java:344) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValue(Template.java:247) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValueOrDefault(Template.java:292) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$VariableSegment.execute(Mustache.java:872) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.execute(Template.java:137) ~[jmustache-1.15.jar!/:na]
        at org.springframework.boot.web.servlet.view.MustacheView.renderMergedTemplateModel(MustacheView.java:80) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1373) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1118) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1057) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2020-06-18 10:27:18.248 ERROR 1 --- [nio-8080-exec-4] o.a.c.c.C.[Tomcat].[localhost]           : Exception Processing ErrorPage[errorCode=0, location=/error]

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is com.samskivert.mustache.MustacheException$Context: No method or field with name 'trace' on line 10
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:394) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:253) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:348) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: com.samskivert.mustache.MustacheException$Context: No method or field with name 'trace' on line 10
        at com.samskivert.mustache.Template.checkForMissing(Template.java:344) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValue(Template.java:247) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValueOrDefault(Template.java:292) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$VariableSegment.execute(Mustache.java:872) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.execute(Template.java:137) ~[jmustache-1.15.jar!/:na]
        at org.springframework.boot.web.servlet.view.MustacheView.renderMergedTemplateModel(MustacheView.java:80) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1373) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1118) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1057) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        ... 27 common frames omitted

2020-06-18 10:27:21.844 ERROR 1 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.samskivert.mustache.MustacheException$Context: No method or field with name 'resource' on line 10] with root cause

com.samskivert.mustache.MustacheException$Context: No method or field with name 'resource' on line 10
        at com.samskivert.mustache.Template.checkForMissing(Template.java:344) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValue(Template.java:247) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.getValueOrDefault(Template.java:292) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$VariableSegment.execute(Mustache.java:872) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:831) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.executeSegs(Template.java:170) ~[jmustache-1.15.jar!/:na]
        at com.samskivert.mustache.Template.execute(Template.java:137) ~[jmustache-1.15.jar!/:na]
        at org.springframework.boot.web.servlet.view.MustacheView.renderMergedTemplateModel(MustacheView.java:80) ~[spring-boot-2.2.2.RELEASE.jar!/:2.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1373) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1118) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1057) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2020-06-18 10:27:21.849 ERROR 1 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

com.samskivert.mustache.MustacheException$Context: No method or field with name 'trace' on line 10
        at com.samskiver

Philipp_Ossler · June 19, 2020, 4:46am

Hi @andyh,

thank you for the update.

Some of the errors seem to be related to the failures in the UI. For example, when it tries to show a BPMN but the workflow is not imported.

It’s hard to say that is the problem here. Maybe, there is something in the logs that can give us a hint.

Can you reproduce the issues?
Please share your setup (i.e. docker-compose file, configuration, etc.).

If a workflow is deployed but not visible in the monitor then it seems that there is a problem during the export from the broker or the import into the monitor.

If a workflow disappears in the monitor then you should check if you’re using a persistent database. This should not happen because the data in the monitor is not deleted.

andyh · June 24, 2020, 3:42pm

Hi @philipp.ossler

The configuration for our dev environment:

Zeebe cluster deployed to Azure Kubernetes Service:
clusterSize: 1
partitionCount: 1
replicationFactor: 1

Exporters:
ElasticSearch
Hazelcast (Simple Monitor)

32Gi Persistent Volume (for ES and Broker)

Version: 0.22.1

We have configured our helm chart to deploy the hazelcast jar file with the Zeebe Helm Chart. There is no external database for the hazelcast exporter, I assumed this means hazelcast exporter writes to the provisioned disk in k8s?

After deleting the pod we are no longer seeing the errors, but we are also not seeing the workflow definitions. After redeploying the definition they do appear in Simple Monitor, but the version has been incremented suggesting the original definitions still exit.

Thanks
Andy

system · January 31, 2024, 10:11am