Compose Makes Elasticsearch Major Version Upgrades Easier

Published

Compose for Elasticsearch users now have the ability to upgrade to a new major version with a click of a button right from the Compose UI.

Previously, users had to create an on-demand backup, then restore that backup from the Compose UI and select a new Elasticsearch version. This is now easier. If you're upgrading from version 5.x to 6.x, you can now simply select the major version you want to upgrade to through the Compose UI Settings panel, and click a button to start the upgrade process.

Now, there is no need to create a new deployment and change your database's connection strings. This is an in-place major upgrade, so all you need to do is click a button, wait a bit until the upgrade is complete, and you'll have the latest version of Elasticsearch on Compose.

Things to Note Before Upgrading

Before upgrading from Elasticsearch 5.x to 6.x, make sure your indices in 5.x are compatible with 6.x. This means that any indices created in previous major versions before 5.x will not be compatible with 6.x. There's an entire description of breaking changes released by Elasticsearch here and you'll want to read through them.

In general, new indices created in 6.x won't support multiple mapping types per index like version 5.x. Therefore, if you're creating new indices in version 6.x, make sure you only have a single mapping type per index. When you upgrade from version 5.x to 6.x, however, multiple mapping types will continue to be supported, but you'll want to make sure to update your mappings eventually.

Also, indices created, for instance, in version 2.x that exist in your version 5.x deployment, will not be supported in version 6.x. The upgrade path is to reindex your version 5.x deployment, then upgrade to version 6.x. If you attempt to upgrade from version 5.x to 6.x without reindexing anything that might have been created in version 2.x, you'll get an error. We talk about that below.

In order to view which indices were created in version 2.x, you could run the following cURL command:

curl -u user:password 'https://<host>:<port>/_all/_settings?human'  

The result will look something like:

{
  "shakespeare": {
    "settings": {
      "index": {
        "creation_date_string": "2020-01-07T18:53:12.539Z",
        "number_of_shards": "3",
        "provided_name": "shakespeare",
        "creation_date": "1578423192539",
        "store": {
          "type": "niofs"
        },
        "unassigned": {
          "node_left": {
            "delayed_timeout": "5m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "fJLe2g2KQPib4Z3KmS2OWg",
        "version": {
          "created_string": "5.6.16",
          "created": "5061699"
        }
      }
    }
  }
}

The important part here is "created_string": "5.6.16" which indicates that this index shakespeare was created with version 5.6.16. Therefore, we're fine to upgrade from version 5.x to 6.x.

Upgrading to 6.x

First, create a backup of your database. You can do that through the Compose UI by clicking on the Backups button on the left menu. The click on the green button on the right that says Back up now, which will start the backup process. Also, you'll want to pause as much indexing as necessary to make sure that you upgrade runs smoothly. You should anticipate a short downtime while your deployment is upgrading in the background - similar to the minor upgrade process.

To upgrade, go to the Compose UI, click on the Settings button from the left menu, then in the Settings view you will see the Change Version panel that looks like the following:

You will see your current version of Compose for Elasticsearch, and the new major version that is available to upgrade to. To upgrade, click on the Change Version button and your deployment will start upgrading. After a little bit, you'll have the latest version of Elasticsearch and you didn't have to change your connection strings to a new deployment.

Have Indices from Version 2.x?

If you looked at your deployment's indices and you found an index with "created_string": "2.4.6" like:

{
  "shakespeare": {
    "settings": {
      "index": {
        "creation_date_string": "2020-01-07T21:23:52.179Z",
        "number_of_shards": "3",
        "creation_date": "1578432232179",
        "store": {
          "type": "niofs"
        },
        "unassigned": {
          "node_left": {
            "delayed_timeout": "5m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "6PVujjPzSrSnTnisqY3ohA",
        "version": {
          "upgraded": "5061699",
          "created_string": "2.4.6",
          "upgraded_string": "5.6.16",
          "created": "2040699"
        }
      }
    }
  }
}

This means that you will not be able to upgrade from version 5.x to 6.x without reindexing your version 5.x deployment first.

In order to reindex your version 5.x deployment, use Elasticsearch's Reindex API. In this example, the index that was created in 2.x is called shakespeare. This index needs to be reindexed. Essentially, we'll use the cURL command to send a POST request to the _reindex endpoint with the following data:

{
  "source": {
    "index": "shakespeare"
  },
  "dest": {
    "index": "new_shakespeare"
  }
}

The new_shakespeare index will be the one that's reindexed and it will be created for you with a mapping that's appropriate for version 5.x.

Looking again at your Elasticsearch indices, you will see the following:

{
  "shakespeare": {
    "settings": {
      "index": {
        "creation_date_string": "2020-01-07T21:23:52.179Z",
        "number_of_shards": "3",
        "creation_date": "1578432232179",
        "store": {
          "type": "niofs"
        },
        "unassigned": {
          "node_left": {
            "delayed_timeout": "5m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "6PVujjPzSrSnTnisqY3ohA",
        "version": {
          "upgraded": "5061699",
          "created_string": "2.4.6",
          "upgraded_string": "5.6.16",
          "created": "2040699"
        }
      }
    }
  },
  "new_shakespeare": {
    "settings": {
      "index": {
        "creation_date_string": "2020-01-07T22:37:33.018Z",
        "number_of_shards": "3",
        "provided_name": "new_shakespeare",
        "creation_date": "1578436653018",
        "store": {
          "type": "niofs"
        },
        "unassigned": {
          "node_left": {
            "delayed_timeout": "5m"
          }
        },
        "number_of_replicas": "1",
        "uuid": "MNiZrm5MQxWI2JvtX3xv_w",
        "version": {
          "created_string": "5.6.16",
          "created": "5061699"
        }
      }
    }
  }

Note how your index contains "created_string": "5.6.16", which will be perfect for upgrading now to version 6.x. Make sure, however, to test whether all of your documents have been reindexed into the new index. After that, you'll need to delete the old index before upgrading.

Need to upgrade from version 2.x to 5.x?

If you want to move up to version 6.x and you're still on version 2.x, we also have you covered. What you'll have to do is the following. From the Compose UI, you'll need to go to the Backups view by clicking on the Backups button on the left menu.

From there, that will take you to the Backups view. Now, create a backup by clicking on the green button that says Back up now. That will kickoff the backup process. Once you've made a backup you will see the backup appear in the view like:

Click on the blue, circular button next to the backup time. You will be taken to another window where you can restore your backup into a 5.x deployment.

Here, you'll want to give your deployment a name, select the location, and keep the latest upgrade as it is. In this case it's 5.6.16. Next, allocate the same amount of resources as your previous deployment to your new one. After that, click Create Deployment. This will restore your new backup into a new deployment running Compose for Elasticsearch 5.6.16.

Once your new deployment is ready, you'll need to reindex your indices so that they are on version 5.6.16. The process for that is described above. Then you can start the upgrade process to 6.x, which is also above.

Summary

We walked you through Compose for Elasticsearch's new feature that allows you to upgrade your Elasticsearch deployment with the click of a button. You no longer need to run a backup and restore into a new deployment, while also having to make changes to your application to point to the new database. Now, everything is done seamlessly without having the need to create a new deployment - we've made it simpler for you.

If you have any questions, let support know and we'll have you covered.

attribution Colin Carter

Abdullah Alger
Abdullah Alger is a former University lecturer who likes to dig into code, show people how to use and abuse technology, talk about GIS, and fish when the conditions are right. Coffee is in his DNA. Love this article? Head over to Abdullah Alger’s author page to keep reading.

Conquer the Data Layer

Spend your time developing apps, not managing databases.