Optimize not working after updating to 8.5

okaeiz · May 29, 2024, 12:55pm

I have just updated my Camunda to 8.5 using docker compose and after the successful installation, I see that the Optimize container is unhealthy.
The logs say

Caused by: org.camunda.optimize.service.exceptions.OptimizeRuntimeException: The database Optimize schema version [3.11.5] doesn't match the current Optimize version [3.13.1]. Please make sure to run the Upgrade first.
	at org.camunda.optimize.service.db.schema.DatabaseMetadataService.lambda$validateMetadata$0(DatabaseMetadataService.java:56)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)

I did not have the problem before the upgrade. What is the issue?

okaeiz · May 30, 2024, 9:15am

It seems like Optimize is not compatible with Elasticsearch. What should I do?

priyaprabhu · July 23, 2024, 7:57am

I too am facing the same issue

okaeiz · July 28, 2024, 10:59am

Do you still have this problem? Mine is still problematic.

priyaprabhu · August 25, 2024, 11:35am

It worked for me after deleting the Volume recreating all the volumes

okaeiz · September 1, 2024, 9:43am

Yeah it was resolved after deleting the volumes but after one week I had the same problem again! It’s annoying.

priyaprabhu · September 2, 2024, 3:42am

Just check if you have upgraded the zeebe/optimize/operate/tasklist/elasticsearch without migration/upgrade of the elastricsearch data. In recent troubleshoot i noticed that this may be the reason. When we delete the volume the data is also getting deleted hence we are able to bring up the server without trouble

Jesse_Simpson · September 7, 2024, 1:47am

To upgrade optimize, you have to go through the migration steps here:

so exec into the optimize container

docker exec -it optimize sh

and run the upgrade script.

./upgrade/upgrade.sh

Deleting the volumes also works.

What’s happening is that optimize expects optimize indices in Elasticsearch to be in a certain format, and between versions of optimize, the format changes. The upgrade script will reformat the data for the new optimize version.

okaeiz · September 7, 2024, 5:49am

Hey there! @Jesse_Simpson
Thanks for your response. I have tried the upgrade script multiple times. It makes the container healthy and the error logs disappear. But still, Optimize is not accessible and it takes forever to load.
After upgrading, you can see the login page and once you click on the login button, it crashes and is never accessible. Here are the logs after restarting the whole Camunda platform stack in Docker:

Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=no_shard_available_action_exception, reason=No shard available for [get [optimize-metadata][1]: routing [null]]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:986)
	at org.camunda.optimize.service.db.es.OptimizeElasticsearchClient.get(OptimizeElasticsearchClient.java:290)
	at org.camunda.optimize.service.db.es.schema.ElasticSearchMetadataService.readMetadata(ElasticSearchMetadataService.java:92)
	... 109 common frames omitted
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://elasticsearch:9200], URI [/optimize-metadata/_doc/1], status line [HTTP/1.1 503 Service Unavailable]
error={"root_cause":[{"type":"no_shard_available_action_exception","reason":"No shard available for [get [optimize-metadata][1]: routing [null]]"}],"type":"no_shard_available_action_exception","reason":"No shard available for [get [optimize-metadata][1]: routing [null]]"} status=503
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
		... 114 common frames omitted

Jesse_Simpson · September 7, 2024, 8:48pm

Happy to help. the issue about

no_shard_available_action_exception :  No shard available for [get [optimize-metadata][1] ...

means that elasticsearch is looking for a 2nd elasticsearch node to allocate storage on. This often happens when the application is configured to have read replicas.

In the camunda-platform repository, I added a commit as part of this PR

which adds the extra optimize configuration into the subdirectory with the alpha5 images.

There’s a hidden folder called .optimize with an environment-config.yaml with this snippet:

es:
  settings:
    index:
      number_of_replicas: 0

Can you try upgrading to the latest version of the camunda-platform repository so that you have this file?

However, if you installed a version of camunda-platform’s compose file before my patch and try to upgrade, your indices will still be set as though they want a read replica since upgrades won’t change how the existing indices are configured.

You can check if this is the case by querying elasticsearch

curl 'localhost:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state'

the unassigned reason would be either REPLICA_ADDED or INDEX_CREATED

index                                                         shard prirep state      node         unassigned.reason
optimize-external-process-variable_v2-000001                  0     r      UNASSIGNED              INDEX_CREATED
optimize-single-process-report_v11                            0     r      UNASSIGNED              INDEX_CREATED

and the 3rd column would report r which means read replica.

To fix this for an index, you can change the setting for that index like so:

curl -H 'Content-type: application/json' -X PUT localhost:9200/optimize-external-process-variable_v2-000001/_settings -d '{"settings": {"number_of_replicas": 0}}'
{"acknowledged":true}

However, that is pretty cumbersome to do manually for each index. so I would try to put it in a script like this

#!/bin/bash
for index in $(curl 'localhost:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state' | grep UNASSIGNED | awk '{print $1}'); do
	curl -H 'Content-type: application/json' -X PUT localhost:9200/$index/_settings -d '{"settings": {"number_of_replicas": 0}}'
done

Hope this helps.

Jesse_Simpson · September 7, 2024, 9:34pm

I’m confused now. Are you on 8.5 or alpha? these problems should not exist in 8.5.

okaeiz · September 8, 2024, 6:19am

I am using the latest version of camunda-platform which I believe is released last month (8.5) and I suppose I have encountered that hidden .optimize folder and the configuration file. I didn’t touch it though.
I ran the curl command; none of the indices had r as their prirep. They were all p. And the unassigned.reason value was empty for all of them.