Grafana is a visualization program for observability. Also, Grafana offers many related software in observability fields like logs ingestion, on-calling, notifier and etc. Grafana also offers organizations, groups and accessibility features for teams. Authorization for organizations and users are done in RBAC. > [!info] > Grafana reuses the components, development schemas and deployment methods across its software. When adapted to its ecosystem, adding/changing a new Grafana software is easy. ### State Grafana has an internal SQLite database that keeps its configurations, dashboards, data sources users, organizations and etc. However, you can substitute it with an external MySQL or PostgreSQL database. This way you can make Grafana stateless and have high-availablity. ### Configuration Here are some of variables in setting up Grafana that can have different values. Some of them should turned into a `volume`: - `config` path - `data` path - `provisioning` path - `logs` path - `plugins` path - grafana `home directory` A Grafana *plugin* can be both installed using `commands` and `plugins`. Grafana's default user-pass are `admin`. After the first login, a new password is requested to be set. ### Provisioning One can set of Grafana panels, data-sources and plugins before it runs and make them reproducible. This is called provisioning. ### Alerting Grafana offers comprehensive alerting tools. It also integrates with alert-manager. One of the benefits of doing alerts in Grafana is that since we have both our logs, metrics, and tracing already in Grafana, writing a homogenous alert on them is much easier. >[!info] > Grafana itself **fires** and **notifies** alerts. However when using alertmanager, prometheus fires the alerts and alertmanager notifies. Alert Rule is a rule that will be fired based on the define alert. ![[Pasted image 20250317191953.png]] --- Alerts in Grafana has three states: 1. Normal 2. Pending 3. Firing --- Both *alerting* and *scheduling* themselves can be clustered for high-availablility. ## Tempo *Tempo* is a tracing backend for Grafana and can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry. ## Loki ![[Pasted image 20250317195021.png]] Loki is logging software by Grafana introduced in 2018. It markets itself as "prometheus for logs". Loki is very lightweight and fast. Compared to [[ELK|EFK]], it has less features but it is very robust and easy-to-setup. --- Combined with `promtail`, Loki can read the logs from various files, docker containers and etc. Combined with an **objection storage**, Loki can store its files on a object storage instead of itself. #### Loki Deployment Like other modern Grafana softwares, each of these Loki components [can be individually scaled and clustered.](https://grafana.com/docs/loki/latest/get-started/deployment-modes/) 1. All its components will be set up as a monolith together. ![[monolithic-mode.png]] 2. Read and Write components are separate. ![[microservices-mode.png]] 3. Based on each components' usage, they set up and scaled separately. ![[scalable-monolithic-mode.png]]