How to Install OpenSearch on Debian 11

OpenSearch is a community-driven project by Amazon and a fork of Elasticsearch and Kibana. it's a fully open-source search engine and analytics suite with rich features and innovative functionality. The OpenSearch project's main component is OpenSearch (a fork of Elasticsearch) and the OpenSearch Dashboards (a fork of Kibana). Both components provide features such as enterprise security, alerting, machine learning, SQL, index state management, and more.

OpenSearch is 100% open-source and licensed under Apache 2.0-licensed. It enables you to easily ingest, secure, search, aggregate, view, and analyze data for a number of use cases such as log analytics, application search, enterprise search, and more.

In this tutorial, you'll deploy OpenSearch - an open-source search, analytics, and visualization suite - to the Debian 11 server. This process includes downloading the OpenSearch package and installing it manually on your Debian system. You'll also set up secure SSL/TLS certificates for OpenSearch and secure the deployment with authentication and authorization.

In addition to that, you'll also deploy and install the OpenSearch Dashboards - an open-source visualization tool - and configure it with OpenSearch. In the end, you'll have data analytics and visualization suite installed on your Debian server and you need to send your data via tools such as fluentd, Logstash, filebeat, and many more.

Prerequistes

To complete this guide, you must have the following requirements:

• A server with Debian 11 installation and at least 8GB of RAM. This example uses a Debian system with hostname 'node1' and local IP address '192.168.5.50'.
• A non-root user with sudo/root administrator privileges.

If these requirements are ready, you can start the OpenSearch installation now.

Setup System

In the first step, you'll set up and optimize your Debian server for OpenSearch deployment. You'll set up the system hostname and fqdn, disable memory paging and swap, then you'll increase the max memory maps number.

Disabling the memory paging and swapping on the OpenSearch host will improve performance. For the max memory maps, you'll need to set up a number of at least '262144' for production.

Run the below 'hostnamectl' command to set up the system hostname. Then, add the fqdn configuration to the '/etc/hosts' file. In this example, the hostname should be 'node1' with fqdn 'node1.hwdomain.lan'.

sudo hostnamectl set-hostname node1
echo "192.168.5.50  node1.hwdomain.lan  node1" >> /etc/hosts

Now verify the fqdn of your server via the below hostname command. In the example, the fqdn of the server should be 'node1.hwdomain.lan'.

hostname -f

Output:

Next, you will need to disable the swap on your system. Whether you're using swap via partition or file, the swap configuration is stored at the '/etc/fstab' file.

Run the below command to disable the swap on your system. The 'sed' command here will disable swap permanently via the '/etc/fstab' file by adding commend '#' to the beginning of line swap settings. The 'swapoff' command will disable swap in the current session.

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a

Verify the swap status via the below command. If disabled, you should get an output '0' in the swap section.

free -m

Output:

Lastly, you'll add the configuration to the '/etc/sysctl.conf' file to increase the max memory maps on your system.

Run the below command to add the parameter 'vm.max_map_count=262144' to the '/etc/sysctl.conf' file. Then, apply the changes via the 'sysctl -p' command.

sudo echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sudo sysctl -p

Verify the max memory maps by running the below command. If successful, your max memory map should be '262144'.

cat /proc/sys/vm/max_map_count

Output:

With your system configured and optimized, you're ready to install OpenSearch on top of it.

Downloading OpenSearch

OpenSearch can be installed in many ways. In this example, you'll install OpenSearch via Tarball. In this step, you'll create a new dedicated system user 'opensearch', download the OpenSearch tarball package, then set up the OpenSearch installation directory with correct permissions and ownership.

First, add the new system user 'opensearch' using the below command.

sudo adduser --system --shell /bin/bash -U 10001 --no-create-home opensearch

Add a new group 'opensearch' using the groupadd command below. Then, add your system user 'opensearch' to the group 'opensearch' via the usermod command.

sudo groupadd opensearch
sudo usermod -aG opensearch opensearch

Now create a new home directory '/home/opensearch' and change the ownership of the home directory '/home/opensearch' to the 'opensearch' user.

mkdir -p /home/opensearch
sudo chown -R opensearch /home/opensearch

Output:

Next, download the OpenSource package via the wget command. Once the download is finished, extract the file via the tar command below. In this example, you'll install OpenSearch 2.4.1. Check the latest version of OpenSearch on the official download page.

wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.4.1/opensearch-2.4.1-linux-x64.tar.gz
tar xf opensearch-2.4.1-linux-x64.tar.gz

After the OpenSearch package is extracted, move the extracted directory to '/opt/opensearch'. This will be the main installation directory for OpenSearch. Then, run the chown command to change the ownership of the directory '/opt/opesearch' to the user 'opensearch'.

mv opensearch-2.4.1 /opt/opensearch
sudo chown -R opensearch /opt/opensearch

Output:

Now that you've downloaded the OpenSearch package, configured the target installation directory to '/opt/opensearch'. Next, you'll set up and configure your OpenSearch installation.

Configuring OpenSearch

In this step, you'll set up OpenSearch to run on a specific IP address, run on single-node, enable OpenSearch security plugins, and set up max heap memory for the OpenSearch process. All of this can be done by editing the OpenSearch config file '/opt/opensearch/config/opensearch.yml' and the OpenSearch JVM options file 'config/jvm.options'.

Move the working directory to '/opt/opensearch' via the cd command.

cd /opt/opensearch

Open the OpenSearch config file 'config/opensearch.yml' using the below nano editor command.

sudo nano config/opensearch.yml

Add the following lines to the file.

# Bind OpenSearch to interface or IP address
network.host: 192.168.5.50

# OpenSearch deployment type
discovery.type: single-node

# Re-enable security plugins 
plugins.security.disabled: false

Save and exit the file 'config/opensearch.yml' when finished.

Details parameters:

• The 'network.host' parameter is used to bind OpenSearch to a specific IP address. In this example, the OpenSearch will be running on the internal IP address '192.168.5.50'.
• The parameter 'discovery.type: single-node' is used when you want to deploy OpenSearch in a single node.
• The parameter 'plugins.security.disabled' is set to 'false', which means the security plugin for OpenSearch will be enabled.

Next, open the file 'config/jvm.options' using the below nano editor command.

sudo nano config/jvm.options

Adjust the max heap memory for the OpenSearch process. This setting is depending on your server's available memory or RAM. This example will allocate 2GB of memory for OpenSearch.

-Xms2g
-Xmx2g

Save and exit the file 'config/jvm.options' when finished.

Lastly, run the below command to set up an environment variable 'OPENSEARCH_JAVA_HOME' on your current session. The OpenSearch package included the required java and JDK packages which are available in the '/opt/opensearch/jdk' directory.

export OPENSEARCH_JAVA_HOME=/opt/opensearch/jdk
echo $OPENSEARCH_JAVA_HOME

With the basic OpenSearch configurations finished. next, you'll generate TLS certificates and secure OpenSearch deployment.

Generating TLS Certificates

In this step, you'll be generating multiple certificates that will be used to secure OpenSearch deployment. You'll secure node-to-node communications with TLS certificates and secure REST-layer traffics between client-server communications via TLS.

Below is the list of certificates that will be generated:

• Root CA certificates: These certificates will be used to sign other certificates.
• Admin certificates: These certificates will be used to get administrative rights to perform all tasks related security plugin.
• Node and Client Certificates: These certificates will be used by nodes and clients within the OpenSearch cluster.

Before you get started, run the below command to create a new directory '/opt/opensearch/config/certs', and move your working directory into it. This directory will be used to store TLS certificates.

mkdir -p /opt/opensearch/config/certs; cd /opt/opensearch/config/certs

Generating Root CA Certificates

Generate a private key for the root CA certificates using the below.

openssl genrsa -out root-ca-key.pem 2048

Now generate a self-signed root CA certificate via the below command. You can also change values within the '-subj' parameter with your information.

openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=CA/ST=ONTARIO/L=TORONTO/O=ORG/OU=UNIT/CN=ROOT" -out root-ca.pem -days 730

With this, you should get the root CA private key 'root-ca-key.pem' and the root CA certificate 'root-ca.pem'.

Output:

Generating Admin Certificates

Generate the new admin certificate private key 'admin-key-temp.pem' using the below command.

openssl genrsa -out admin-key-temp.pem 2048

Convert the default admin private key to PKCS#8 format. For the Java application, you need to convert the default private key to PKCS#12-compatible algorithm (3DES). With this, your admin private key should be 'admin-key.pem'.

openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem

Next, run the below command to generate the admin CSR (Certificate Signing Request) from the 'admin-key.pem' private key. Your generated CSR should now be 'admin.csr' file.

Because this certificate is used for authenticating elevated access and is not tied to any hosts, you can use anything in the 'CN' configuration.

openssl req -new -key admin-key.pem -subj "/C=CA/ST=ONTARIO/L=TORONTO/O=ORG/OU=UNIT/CN=A" -out admin.csr

Lastly, run the below command to sign the admin CSR with the root CA certificate and private key. The output of the admin certificate is the 'admin.pem' file.

openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730

Your admin certificate should now be 'admin.pem' file which is signed with root CA certificates. And the admin private key is 'admin-key.pem', which is converted to PKCS#8 format.

Output:

Generating Node Certificates

The process of generating node certificates is similar to admin certificates. But, you can specify the CN value with the hostname or IP address of your node.

Generate the node private key using the below command.

openssl genrsa -out node1-key-temp.pem 2048

Convert the node private key to PKCS#8 format. Your node private key should now be 'node1-key.pem'.

openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem

Next, create a new CSR for the node certificate. Be sure to change the 'CN' value with the hostname of your node. This certificate is tied to hosts, and you must specify the CN value with the hostname or IP address of your OpenSearch node.

openssl req -new -key node1-key.pem -subj "/C=CA/ST=ONTARIO/L=TORONTO/O=ORG/OU=UNIT/CN=node1.hwdomain.lan" -out node1.csr

Before signing the node certificate, run the below command to create a SAN extension file 'node1.ext'. This will contain the node hostname or FQDN or IP address

echo 'subjectAltName=DNS:node1.hwdomain.lan' > node1.ext

Lastly, sign the node certificate CSR file with root CA certificate and private using the below command.

openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node1.pem -days 730 -extfile node1.ext

With this, your node certificate is a 'node1.pem' file and the private key is 'node1-key.pem'.

Output:

Setting up Certificates

Run the below command to remove the temporary certificate, CSR, and SAN extension file.

rm *temp.pem *csr *ext
ls

Convert the root CA certificate to .crt format.

openssl x509 -outform der -in root-ca.pem -out root-ca.crt

Add the root CA certificate to your Debian system using the below command. Copy the root-ca.crt file to the '/usr/local/share/ca-certificates/' directory and load the new root CA certificate to your system.

sudo cp root-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

The output '1 added' confirms that the new root CA certificates are added to your system.

Lastly, run the below command to set up the proper permission and ownership of your certificates. The ownership of the directory '/opt/opensearch/config/certs' should be the user 'opensearch' with permission 0700. And for all certificate files, the permission should be 0600.

sudo chown -R opensearch /opt/opensearch/config/certs
sudo chmod 0700 /opt/opensearch/config/certs

sudo chmod 0600 /opt/opensearch/config/certs/*.pem
sudo chmod 0600 /opt/opensearch/config/certs/*.crt

Adding TLS Certificates to OpenSearch

With TLS certificates generated, the root CA, admin certificates, and node certificates are. You'll next add certificates to the OpenSearch config file '/opt/opensearch/config/opensearch.yml'. In this example, you'll create a new bash script that will add certificates and TLS security plugin settings to OpenSearch.

Create