Product docs and API reference are now on Akamai TechDocs.
Search product docs.
Search for “” in product docs.
Search API reference.
Search for “” in API reference.
Search Results
 results matching 
 results
No Results
Filters
Deploy the Elastic Stack on Kubernetes
Traducciones al EspañolEstamos traduciendo nuestros guías y tutoriales al Español. Es posible que usted esté viendo una traducción generada automáticamente. Estamos trabajando con traductores profesionales para verificar las traducciones de nuestro sitio web. Este proyecto es un trabajo en curso.
What is the Elastic Stack?
The Elastic Stack is a collection of open source projects from Elastic that help collect and visualize a wide variety of data sources. Elasticsearch can store and aggregate data such as log files, container metrics, and more. The products in the stack include: Elasticsearch, Logstash, Kibana, and now Beats.
This guide provides instructions to:
- Configure and deploy a number of Helm charts in a Kubernetes cluster in order to set up components of the Elastic Stack.
- Configure and run Kibana in the web browser.
- Install Metricbeat and deploy dashboards to Kibana to explore Kubernetes cluster data.
You can further use the Elastic Stack deployed and configured for application logs or monitoring Kubernetes.
This guide’s example instructions creates the following billable resources on the Linode account: four (4) Linodes and three (3) Block Storage volumes. If you do not want to keep using the example cluster that you create, be sure to delete the cluster Linodes and volumes after you have finished the guide.
If you remove the resources afterward, you are billed only for the hour(s) that the resources were present on the account. For more information, see Billing and Payments guide about how hourly billing works and for a table of plan pricing.
Before You Begin
Install the Kubernetes CLI (
kubectl
) on the local computer.Follow the instructions in Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial to create and connect to an LKE cluster.
Note Ensure that the LKE cluster that you create has three nodes and one master node with 4GB Linode instances. Also ensure that theKUBECONFIG
context is persistentYou should also make sure that Kubernetes CLI is using the right cluster context. Run the
get-contexts
subcommand to check:kubectl config get-contexts
Set up Helm in the Kubernetes cluster by following the Install Helm section in the How to Install Apps on Kubernetes with Helm 3 guide.
Configure Helm
You should now have a Kubernetes cluster with Helm installed and configured.
Add the
elastic
chart repository to the local installation of Helm:helm repo add elastic https://helm.elastic.co
Fetch the updated list of charts from all configured chart repositories:
helm repo update
Search for the official
elasticsearch
chart to confirm Helm has been configured correctly. Note that this chart released by Elastic differs from the chart bundled with the default installation of Helm.helm search hub elasticsearch
This command returns all the charts available for
elasticsearch
in the hub. Select the one listed below. The exact version numbers may be different; at the time of writing this guide the version is 7.8.0.NAME CHART VERSION APP VERSION DESCRIPTION https://hub.helm.sh/charts/elastic/elasticsearch 7.8.0 7.8.0 Official Elastic helm chart for Elasticsearch
The Helm environment is now ready to install official Elasticsearch charts to the Kubernetes cluster.
Install Charts
Install Elasticsearch
Before installing the chart, ensure that resources are set appropriately. By default, the elasticsearch
chart allocates 1GB of memory to the JVM heap and sets Kubernetes resource requests and limits to 2GB. Using a Linode 4GB instance is compatible with these defaults, but if you are using a different instance type, you need to provide different values to the chart at install time in order to ensure that running Pods are within the resource constraints of the node sizes you have chosen.
Install the
elasticsearch
chart:helm install elasticsearch elastic/elasticsearch
An output similar to the following appears:
LAST DEPLOYED: Tue Jul 7 14:46:52 2020 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: 1. Watch all cluster members come up. $ kubectl get pods --namespace=default -l app=elasticsearch-master -w 2. Test cluster health using Helm test. $ helm test elasticsearch --cleanup
A three-node Elasticsearch cluster is now configured and available locally to the Kubernetes cluster. To confirm this, first port-forward a local port to the Elasticsearch service. Leave this command running in a terminal window or tab in the background for the remainder of this tutorial.
kubectl port-forward svc/elasticsearch-master 9200:9200
Note This command times out after 5 minutes, if you find that and want to have the port forward for longer, consider using the following command to keep it open:while true; do kubectl port-forward svc/elasticsearch-master 9200:9200; done
In another terminal window, send a request to this port:
curl http://localhost:9200/
An output similar to the following appears:
{ "name" : "elasticsearch-master-1", "cluster_name" : "elasticsearch", "cluster_uuid" : "2eKh30v2Q1ybT9HTPqQw9w", "version" : { "number" : "7.8.0", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65", "build_date" : "2020-06-14T19:35:50.234439Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }
Note The specific version numbers and dates may be different in this JSON response. Elasticsearch is operational, but not receiving or serving any data.
Install Filebeat
In order to start processing data, deploy the filebeat
chart to the Kubernetes cluster. This collects all Pod logs and stores them in Elasticsearch, after which they can be searched and used in visualizations within Kibana.
Deploy the
filebeat
chart. No customvalues.yaml
file should be necessary:helm install filebeat elastic/filebeat
An output similar to the following appears:
NAME: filebeat LAST DEPLOYED: Tue Jul 7 15:33:52 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. Watch all containers come up. $ kubectl get pods --namespace=default -l app=filebeat-filebeat -w
Confirm that Filebeat has started to index documents into Elasticsearch by sending a request to the locally-forwarded Elasticsearch service port in a different terminal:
curl http://localhost:9200/_cat/indices
At least one
filebeat
index should be present, and output should be similar to the following:green open filebeat-7.8.0-2020.07.07-000001 6CYTk-UWQSeG7Y5-XjbQww 1 1 16975 0 10mb 5.8mb
Install Kibana
Kibana provides a frontend to Elasticsearch and the data collected by Filebeat.
Deploy the
kibana
chart:helm install kibana elastic/kibana
An output similar to the following appears:
NAME: kibana LAST DEPLOYED: Tue Jul 7 15:40:21 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None
Port-forward the
kibana-kibana
service in order to access Kibana locally. Leave this command running in the background as well for the remainder of this tutorial.kubectl port-forward svc/kibana-kibana 5601:5601
Note This command times out after 5 minutes, if you find that and want to have the port forward for longer, consider using the following command to keep it open:while true; do kubectl port-forward svc/kibana-kibana 5601:5601; done
Configure Kibana
Before visualizing Pod logs, Kibana must be configured with an index pattern for Filebeat’s indices.
With the previous
port-forward
command running in another terminal window, open a browser and navigate to http://localhost:5601A welcome page similar to the following appears in the browser. Click the Explore on my own button.
Open the menu, then go to Stack Management > Kibana > Index Patterns to create a new index pattern. The Index patterns page appears.
Click the Create index pattern button to begin.
In the Define index pattern window, type
filebeat-*
in the Index pattern text box and click the Next step button.In the Configure settings window, select
@timestamp
from the Time Filter field name dropdown menu and click the Create index pattern button.A page with the index pattern details appears. Open the menu, then go to Kibana > Discover to view incoming logs.
The Discover page provides a realtime view of logs as they are ingested by Elasticsearch from the Kubernetes cluster. The histogram provides a view of log volume over time, which by default, spans the last 15 minutes. The sidebar on the left side of the user interface displays various fields parsed from JSON fields sent by Filebeat to Elasticsearch.
Use the Filters box to search only for logs arriving from Kibana Pods by filtering for
kubernetes.container.name : "kibana"
. Click the Update button to apply the search filter.Note When searching in the filters box, field names and values are auto-populated.In order to expand a log event, click the arrow next to an event in the user interface.
Scroll down to view the entire log document in Kibana. Observe the fields provided by Filebeat, including the
message
field, which contains standard out and standard error messages from the container, as well as the Kubernetes node and Pod name in fields prefixed withkubernetes
.Look closely at the
message
field in the log representation and note that the text field is formatted as JSON. While the terms in this field can be searched with free text search terms in Kibana, parsing the field generally yields better results.
Metricbeat
In addition to collecting logs with Filebeat, Metricbeat can collect Pod and node metrics in order to visualize information such as resource utilization.
Install Metricbeat
Deploy the
metricbeat
chart.helm install metricbeat elastic/metricbeat
An output similar to the following appears:
NAME: metricbeat LAST DEPLOYED: Tue Jul 7 18:43:58 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. Watch all containers come up. $ kubectl get pods --namespace=default -l app=metricbeat-metricbeat -w
Confirm that Metricbeat has started to index documents into Elasticsearch by sending a request to the locally-forwarded Elasticsearch service port:
curl http://localhost:9200/_cat/indices
At least one
metricbeat
index should be present, similar to the following:green open metricbeat-7.8.0-2020.07.07-000001 wAWu5op1SJqlbaXKOj_tSg 1 1 1214 0 3.5mb 1.7mb
Load Dashboards
Metricbeat can install default Dashboards into Kibana to provide out-of-the-box visualizations for data collected by Kubernetes.
Before following these steps, ensure that the port-forward
command to expose Kibana over port 5601
locally is still running.
Run the following commands on the local machine. This communicates with Kibana over 127.0.0.1:5601
to import default Dashboards that is populated with data from Metricbeat.
The commands should use the same version of Metricbeat deployed to the Kubernetes cluster. You can find this version by issuing the following command:
helm get values --all metricbeat | grep imageTag
For Linux
Get the Metricbeat package.
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.8.0-linux-x86_64.tar.gz
Unzip the package.
tar xvzf metricbeat-7.8.0-linux-x86_64.tar.gz
Navigate to the directory.
cd metricbeat-7.8.0-linux-x86_64
Setup the dashboards.
./metricbeat setup --dashboards
For MacOS
Get the Metricbeat package.
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.8.0-darwin-x86_64.tar.gz
Unzip the package.
tar xzvf metricbeat-7.8.0-darwin-x86_64.tar.gz
Navigate to the directory.
cd metricbeat-7.8.0-darwin-x86_64
Setup the dashboards.
./metricbeat setup --dashboards
Explore Dashboards
Open a browser window to http://localhost:5601 and click the Dashboards in the left sidebar.
In the search box, enter “kubernetes” and press
Enter
. Select the [Metricbeat Kubernetes] Overview ECS dashboard.The following dashboard displays several types of metrics about the Kubernetes cluster.
You can explore the various visualizations on this page in order to view metrics about Pods, nodes, and the overall health of the Kubernetes cluster.
Next Steps
From this point onward, any additional workloads started in Kubernetes is processed by Filebeat and Metricbeat in order to collect logs and metrics for later introspection within Kibana. As Kubernetes nodes are added or removed, the Filebeat and Metricbeat DaemonSets
automatically scale out Pods to monitor nodes as they join the Kubernetes cluster.
More Information
You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.
This page was originally published on