Operate get "failed to parse date field" in elasticsearch post request

Camunda Platform 7 Topics Camunda Platform 7 Process Engine
1

I use the demo Operate instance as a docker service with Zeebe.

It works fine, except for a recurrent error rapidly filling up the logs (1200/minute!):

failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]

It seems related to Elasticsearch date-time format:

date_time or strict_date_time

A formatter that combines a full date and time, separated by a T : yyyy-MM-dd'T'HH:mm:ss.SSSZZ .

The complete log error:

2020-07-20 08:54:26.363 ERROR 6 --- [       Thread-1] o.c.o.w.z.o.OperationExecutor            : Something went wrong, while executing operations batch. Will be retried. Underlying exception: 
2020-07-20 08:54:26.364 ERROR 6 --- [       Thread-1] o.c.o.w.z.o.OperationExecutor            : Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.8.7.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2030) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
	at org.camunda.operate.webapp.es.reader.OperationReader.acquireOperations(OperationReader.java:90) ~[classes!/:?]
	at org.camunda.operate.webapp.es.writer.BatchOperationWriter.lockBatch(BatchOperationWriter.java:104) ~[classes!/:?]
	at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.executeOneBatch(OperationExecutor.java:95) ~[classes!/:?]
	at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.run(OperationExecutor.java:71) [classes!/:?]
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/operate-operation-0.23.0_/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"operate-operation-0.23.0_","node":"CyOZxJJERUWNREpmaVX08g","reason":{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2020-07-20T08:54:26.351+0000' could not be parsed at index 23"}}}}]},"status":400}
		at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
		at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
		at org.camunda.operate.webapp.es.reader.OperationReader.acquireOperations(OperationReader.java:90) ~[classes!/:?]
		at org.camunda.operate.webapp.es.writer.BatchOperationWriter.lockBatch(BatchOperationWriter.java:104) ~[classes!/:?]
		at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.executeOneBatch(OperationExecutor.java:95) ~[classes!/:?]
		at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.run(OperationExecutor.java:71) [classes!/:?]
	Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/operate-operation-0.23.0_/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"operate-operation-0.23.0_","node":"CyOZxJJERUWNREpmaVX08g","reason":{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2020-07-20T08:54:26.351+0000' could not be parsed at index 23"}}}}]},"status":400}
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
		at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
		at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
		at java.lang.Thread.run(Unknown Source) ~[?:?]
2

@vtexier, Currently only the ISO Date Format yyyy-MM-dd'T'HH:mm:ss is supported. Example value: 2013-01-23T13:42:42

https://docs.camunda.org/manual/latest/reference/embedded-forms/controls/date-inputs/#date-format

3

Thanks for your help.

But the date do not come from a form. It is increasing as time goes by…

For your information, I use the demo account of Operate to test it.
I have never created any forms.

So I do not know where the wrongly formatted datetime comes from…

4

@vtexier, it’s due to the date fields stored in elastic search. Elasticsearch stores date fields as like below:

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#built-in-date-formats

date_optional_time or strict_date_optional_time

A generic ISO datetime parser, where the date must include the year at a minimum, and the time (separated by T ), is optional. Examples: yyyy-MM-dd'T'HH:mm:ss.SSSZ or yyyy-MM-dd.

But camunda optimize supports date format as like below:

5

I do not have Optimize installed, as I said in the post, I installed the “cloud” platform with Zeebe and Operate only.

As suggested in the Operate guide, I will go to the Zeebe forum to ask about Operate.

But the problem seems as you say, a date format incompatibility.

Thanks for your quick help, I have a better picture of the Camunda/Zeebe ecosystem now.

6

@vtexier, actually i misread it. zeebe related queries can be asked here.

The issue is with elastic date format. Maybe you can convert to the operate supported date format.

https://docs.zeebe.io/bpmn-workflows/timer-events/timer-events.html?highlight=date#time-date

1 Like