ELK is a cloud native app used for observability in software. ![[Pasted image 20250215212726.png]] - *ELK* has support for logging, metrics and tracing. - *ELK* can do [anomaly detection](https://www.elastic.co/what-is/anomaly-detection). - *ELK* has a cloud version called *Elastic Cloud* ### ElasticSearch Is a software mainly used for any kind of search. *ElasticSearch* is the main component of ELK stack. It is the *only* stateful software in this stack. > [!note] > *OpenSearch* is an ElasticSearch fork by Amazon that removes ELK license issues. #### Node types 1. `master`: main nodes for the cluster 2. `data_[content/hot/warm/cold/frozen]`: nodes for specific data type. A hotter data means it has more reads. 3. `ingest`: controls the input traffic 4. `ml`: machine learning 5. unset: a [coordinating node](https://www.elastic.co/guide/en/elasticsearch/reference/current/node-roles-overview.html#coordinating-only-node-role) handles the connection between other nodes. #### Data Type 1. **Index**: is the relative synonym of a *table* in SQL-like databases. 2. **Document**: is the JSON object that is stored in the Index. ### Sharding vs. Replication *Sharding* is when a data is **partitioned** into different nodes. *Replication* is when a data is **copied** into different nodes. In Replication, the *same data* is present in different nodes. ### Kibana Kibana is the web-based visualisation software of ELK stack. Although Kibana can also change the configuration of ELK stack, it is stateless itself. ### Beats & Agents Beats are different data collectors for converting data into a schema suitable for ELK. ![[Pasted image 20250222130121.png]] --- Elastic Agent is a single binary designed to provide the [same functionality](https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html#additional-capabilities-beats-and-agent) that the various Beats provide today. Agents are a relatively new concept in ELK and can potentially override Beats in the long run. #### Fleet Fleet provides a web-based UI in Kibana for centrally managing Elastic Agents and their policies. ![[Pasted image 20250222131224.png]] ### ELK in scale #### Variable Income Rates When the number of incoming data to the stacks increases, Logstash or ElasticSearch can't keep up with the data. Especially when the incoming rate of data is not fixed, a **message queue** can fix this problem. This also helps not to miss any data. Here beats write the data to one of these message queues, then Logstash when ready, reads the incoming data upon free time. ![[Pasted image 20250305165459.png]] #### Different Elastic Nodes When the scale increases, the Elasticsearch node can be the bottleneck. Here we can multiply it wisely. Additional nodes can be created with different [[ELK#Node types|types]] to meed the needs of a certain data. ### Fluentd Sometimes, *Fluentd* is used as an alternative for Logstash. (known as EFK instead of ELK) Fluentd also can sometimes replace some *beats* and buffering methods. Also Fluentd offers a better compatibility layer. ![[Pasted image 20250305171151.png]] #### Fluentbit is a Fluentd alternative. It's a more lightweight program which also does forwarding and used where resource efficiency is paramount. Fluentbit is also used in embedded devices as its lightweight nature. ![[Pasted image 20250305171709.png]] Also they can be used alongside with each other: ![[Pasted image 20250305172951.png]]