Setting up an Elasticsearch testing cluster

Setting up an Elasticsearch testing cluster

Elasticsearch is an open source search and analytics service and is one of the most popular technologies these days for full text search and text analytics. What would you need that for? Well, a full text search engine like Elasticsearch is used in apps that want to give users a way to search for data that may be fast moving and always changing. A shopping website with millions of different products that are constantly being added and sold is just one example. Also, dashboards that integrate with active scrapers is another. Or a log manager. It also makes it extremely easy for users to submit a query, say, “socks”, and for the search results to return “brown socks”, “holiday socks”, “sock puppets”, and “rock-em-sock-em robots”. Basically, Elasticsearch is way more advanced, faster, and easier to get search results from than a standard SQL or NoSQL database, which are primarily used for storage.

I won’t go too much into the other features of Elasticsearch here, and I’m going to assume you already know why you want to use this technology and how to use it if you are. If you want a more in depth overview of it, I suggest reading what the creators have to say about it.

This post will be about how to set up a testing cluster on your machine so you can run integration tests without running into conflict with your development environment. In general, you never want to run 2 elasticsearch clusters on a single machine unless you have a TON of RAM (like 64+). But since this is just for development and testing, you should be OK!

To run a testing cluster, we’re essentially going to do 2 things.

1) Create a new configuration file for elasticsearch

2) Learn the command to start a new cluster from this configuration file.

That’s all it takes! The steps you take from there will depend on your application setup, but a common theme will be to make sure that the testing server is running on the correct port of your machine while the tests are running, and then making sure your app’s tests are pointing to the correct port.

Creating a new config file

The command you use to start an elasticsearch cluster is normally /bin/elasticsearch or maybe just elasticsearch. This command uses a config file located at $home/elasticsearch-X.X.X/config/elasticsearch.yml (or wherever your elasticsearch was installed to). If you’re using mac or linux, you can find this file using the find command like so: find / -name elasticsearch.yml.

Navigate to the elasticsearch directory and copy the entire config folder into a new folder called config.test.

cp config config.test

Next, open ./config.test/elasticsearch.yml in a text editor and let’s make some adjustments. We need to:

1) Change the cluster name.

2) Change the data and logs paths so we don’t conflict with our main cluster.

3) Change the port the cluster runs on.

The config file is divided into sections and is entirely commented, so this is a matter of finding the right line, uncommenting it, and then changing the value.

Change the cluster name:

# ---------------------------------- Cluster -----------------------------------
# Use a descriptive name for your cluster:
# es-test

Change the paths:

# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
# /path/to/elasticsearch-X.X.X/data.test
# Path to log files:
path.logs: /path/to/elasticsearch-X.X.X/logs.test

Change the port:

# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
# Set a custom port for HTTP:
http.port: 9201

Now save the changes and exit.

Run the command to start the test cluster

The original command /bin/elasticsearch or elasticsearch stays the same, but we need to change the config path that it utilizes. This is set using the Environmental variable ES_PATH_CONF which should have been set during installation.

But to change it from the command line for a single instance of the command, you can type:

ES_PATH_CONF=/path/to/elasticsearch-X.X.X/config.test /bin/elasticsearch

And that will start your test server. I highly recommend that you save this command to an Alias in your ~/.bashrc or ~/.zshrc file or wherever you have your aliases stored.

Now, when you visit port 9201, a cluster of elasticsearch will be running that is completely separate from the cluster you normally run on port 9200.

Happy testing!

Also see: the elasticsearch config docs


Because every coding blog needs a comments section.

Please keep comments respectful! Harassment and general arrogance will not be tolerated.