# Enterprise Elastic Stack Deployment: Architecture, Security and Operations

Deploy a secure Elastic Stack on Ubuntu: TLS, Fleet, log collection, data streams, ILM, clustering, snapshots and operational troubleshooting.

## Start with the missing evidence, not the installer

A Windows account locks out at 09:12, an application returns 502, and a firewall records denied connections. Without a shared timeline these remain three separate tickets. In this hypothetical deployment, the first acceptance test is to find all three events with the correct host, original timestamp and source. A running Kibana page alone does not meet that test.

The reproducible baseline is Elastic Stack and Elastic Agent 9.5.6 on Ubuntu Server 24.04 LTS amd64. The live Debian installation guide identifies 9.5.6; the support matrix includes Ubuntu 24.04. Pin Elasticsearch, Kibana, Logstash and Agent consistently, then assess later security fixes before an actual rollout. Integration packages have their own versions and minimum Kibana requirements: record those separately. These examples were reviewed against official documentation; no live cluster execution or production benchmark is claimed.

[Elastic: Debian package installation and version pinning](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package)

[Elastic support matrix: operating systems, JVM and agents](https://www.elastic.co/support/matrix)

[Ubuntu 24.04 LTS release notes and support period](https://documentation.ubuntu.com/release-notes/24.04/)

## Assign responsibilities before assigning machines

![Elastic data plane: agents or relays to ingestion and Elasticsearch; Fleet manages agent policy](/assets/img/articles/content/elk-stack-enterprise-installation-guide/ingestion.svg)

Elasticsearch stores, indexes and searches events. Kibana supplies exploration, visualisation, management and rule interfaces. An Ingest Pipeline runs processors inside Elasticsearch. Logstash is a separate transformation and routing service, useful for complex parsing, multiple outputs and a persistent queue. It is optional: adding a Logstash hop to every source without a requirement creates another queue to operate.

Elastic Agent runs integrations on hosts or collectors. Fleet in Kibana manages policies and enrollment; Fleet Server distributes that management state. With an Elasticsearch output, agents send event payloads directly to Elasticsearch, not through Fleet Server. Beats remain an alternative for specialised or existing deployments; Filebeat ships files and Winlogbeat ships Windows events. Do not run Agent and Beats against the same files unless duplication is intentional.

| Model | Topology and availability | Starting allocation; measure before buying |
| --- | --- | --- |
| Lab | One VM, all services, single Elasticsearch node; no HA | 4–8 vCPU, 16 GB RAM, SSD; sample data only |
| Small production | Three mixed-role ES nodes across failure domains; separate Kibana and Fleet collector | Per ES node: 8 vCPU, 32 GB RAM as a pilot budget; storage from measured volume |
| Enterprise production | Three dedicated masters, at least two hot/content data nodes, separate ingest as needed; two Kibana/Fleet instances | Master and data resources sized separately; warm nodes only with a measured retention need |

Three virtual machines on one ESXi host or one shared storage failure domain are not three independent copies of the service. Spread placement and account for losing one data node during maintenance. A small organisation that accepts an outage can run one production node with snapshots, but it must record that availability exception; increasing RAM does not create HA.

## Prepare names, disks and network paths

The concrete small-production example uses es01.logs.example.net / 10.20.30.11, es02 / .12 and es03 / .13, plus kibana.logs.example.net / .21 and fleet.logs.example.net / .22. Replace every example address and certificate SAN before execution. DNS must resolve consistently from servers and collectors; NTP must agree before comparing event time with ingestion time. Ubuntu, RHEL and other platforms have different support rows; this procedure covers Ubuntu 24.04, not every Linux distribution.

| Flow | Port | Restriction |
| --- | --- | --- |
| Agent / Kibana / Logstash → Elasticsearch | TCP 9200 | Approved sources only; HTTPS |
| Elasticsearch ↔ Elasticsearch | TCP 9300 | Cluster nodes only; transport TLS |
| Browser → Kibana | TCP 5601 | Management network or authenticated HTTPS reverse proxy |
| Agent → Fleet Server | TCP 8220 | Managed endpoints; HTTPS |
| Device → collector | UDP/TCP 514 or configured port | Per-source ACL; UDP is lossy and unencrypted |
| Collector → vCenter / SQL / registry | TCP 443 / 1433 / HTTPS | 1433 is a common SQL default, not every named-instance port; allow only required services |

```bash
hostnamectl
getent hosts es01.logs.example.net es02.logs.example.net es03.logs.example.net
timedatectl status
lsblk -f
df -hT
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg unzip sysstat
printf 'vm.max_map_count=1048576
' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl --system
sysctl vm.max_map_count
iostat -xz 1 5
```

Expect synchronised time, the intended SSD mount and vm.max_map_count at least 1048576. iostat is an observation tool, not a synthetic benchmark: sustained latency and queueing under a representative ingestion/search load matter more than the disk model name. Keep data outside the root filesystem where possible and mount it before service start. Use the bundled JDK; installing an arbitrary system Java is unnecessary. Leave automatic heap sizing initially; a manually assigned heap must leave memory for the filesystem cache and normally remain at or below half the available RAM.

[Elastic: current vm.max_map_count guidance](https://www.elastic.co/docs/deploy-manage/deploy/self-managed/vm-max-map-count)

[Elastic: JVM and heap settings](https://www.elastic.co/docs/reference/elasticsearch/jvm-settings)

## Install the same release on every role

Run repository setup on each Ubuntu host. Install only the package matching its role: Elasticsearch on es01–03, Kibana on .21, and Logstash only on an optional transformation host. apt verifies repository signatures; inspect the signing-key fingerprint against the vendor documentation before trusting a newly downloaded key. Stop if the exact pinned release is unavailable rather than silently installing another release.

```bash
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch -o /tmp/elastic-signing-key.asc
gpg --show-keys --with-fingerprint /tmp/elastic-signing-key.asc
sudo gpg --dearmor --output /usr/share/keyrings/elasticsearch-keyring.gpg /tmp/elastic-signing-key.asc
echo 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main' | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt-get update
apt-cache policy elasticsearch kibana logstash
# Run the appropriate line on its designated host, not all three everywhere.
sudo apt-get install elasticsearch=9.5.6
sudo apt-mark hold elasticsearch
# On the Kibana host:
sudo apt-get install kibana=9.5.6
sudo apt-mark hold kibana
# On an optional Logstash host:
LOGSTASH_DEB_VERSION=$(apt-cache madison logstash | awk '$3 ~ /(^|:)9[.]5[.]6(-|$)/ {print $3; exit}')
test -n "$LOGSTASH_DEB_VERSION" || { echo 'Pinned Logstash release is absent'; exit 1; }
sudo apt-get install "logstash=$LOGSTASH_DEB_VERSION"
sudo apt-mark hold logstash
```

Elasticsearch and Kibana use 9.5.6 package versions. The Logstash command selects only upstream 9.5.6 while retaining the actual Debian epoch/revision from repository metadata; it stops if that release is absent. Review the selected value before confirming apt. Hold prevents an unattended cross-component upgrade; it also means security patching becomes an explicit operational responsibility. Record dpkg-query output in the deployment inventory.

```bash
dpkg-query -W -f='${Package} ${Version}
' elasticsearch
# Run on the corresponding hosts:
dpkg-query -W -f='${Package} ${Version}
' kibana
dpkg-query -W -f='${Package} ${Version}
' logstash
```

## Issue certificates before starting the cluster

This is a manual-TLS installation from the first startup; do not merge its configuration with auto-generated security blocks from a previously started node. The example uses a restricted certificate-issuing workstation with the same Elasticsearch tools. An enterprise PKI can issue the equivalent certificates instead. Each node needs its own private key and DNS SAN. The issuer private key stays on the issuing system, never on data nodes or agents. Keep an encrypted offline backup of the CA and establish renewal ownership.

```bash
sudo -i
sudo install -d -m 0700 /root/elastic-pki
cd /root/elastic-pki
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem --out ca.zip
sudo unzip ca.zip
sudo chmod 0600 ca/ca.key
sudo tee instances.yml >/dev/null <<'YAML'
instances:
  - name: es01
    dns: [es01.logs.example.net]
    ip: [10.20.30.11]
  - name: es02
    dns: [es02.logs.example.net]
    ip: [10.20.30.12]
  - name: es03
    dns: [es03.logs.example.net]
    ip: [10.20.30.13]
  - name: kibana
    dns: [kibana.logs.example.net]
    ip: [10.20.30.21]
  - name: fleet
    dns: [fleet.logs.example.net]
    ip: [10.20.30.22]
YAML
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem \
  --in instances.yml --ca-cert ca/ca.crt --ca-key ca/ca.key --out instances.zip
sudo unzip instances.zip
exit
```

Transfer only ca.crt and the destination host’s certificate/key over an authenticated channel. On each Elasticsearch node place them as /etc/elasticsearch/certs/ca.crt, node.crt and node.key, owned by root:elasticsearch, directory 0750 and files 0640. On Kibana use root:kibana and /etc/kibana/certs/. On Fleet use root:root, directory 0700 and private key 0600. This sample uses an unencrypted service key with restrictive permissions; an organisation requiring encrypted keys must configure the appropriate secure passphrase setting rather than leaving an interactive boot dependency.

[Elastic certutil: per-instance SANs and PEM certificates](https://www.elastic.co/docs/reference/elasticsearch/command-line-tools/certutil)

## Bootstrap a three-node small-production cluster

On a fresh es01 write the following /etc/elasticsearch/elasticsearch.yml. On es02/es03 change only node.name and network.host to their own values and install their own keys. Keep cluster.initial_master_nodes identical on these three new master-eligible nodes for this first bootstrap only. It is not a list for future restarts. A pre-existing cluster must never be bootstrapped again.

```yaml
cluster.name: meetaj-logs-prod
node.name: es01
network.host: 10.20.30.11
http.port: 9200
transport.port: 9300
node.roles: [master, data, ingest, transform]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
discovery.seed_hosts: [es01.logs.example.net, es02.logs.example.net, es03.logs.example.net]
cluster.initial_master_nodes: [es01, es02, es03]
xpack.security.enabled: true
xpack.security.autoconfiguration.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.certificate_authorities: [certs/ca.crt]
xpack.security.transport.ssl.certificate: certs/node.crt
xpack.security.transport.ssl.key: certs/node.key
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate_authorities: [certs/ca.crt]
xpack.security.http.ssl.certificate: certs/node.crt
xpack.security.http.ssl.key: certs/node.key
```

```bash
sudo systemctl edit elasticsearch
# Enter the following drop-in, save, then exit:
# [Service]
# LimitNOFILE=65535
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch --no-pager
sudo journalctl -u elasticsearch -n 80 --no-pager
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i
systemctl show elasticsearch -p LimitNOFILE
```

Start all three nodes, then run the password reset once on a healthy member. Supply a generated administrator password interactively and store it in the approved vault. The command uses the file realm internally; do not disable that realm before bootstrap. In a restricted administrator shell, use curl -u elastic so curl prompts instead of putting the password in history. Copy only the public ca.crt to that workstation.

```bash
curl --cacert ./ca.crt -u elastic https://es01.logs.example.net:9200/
curl --cacert ./ca.crt -u elastic 'https://es01.logs.example.net:9200/_cat/nodes?v'
curl --cacert ./ca.crt -u elastic 'https://es01.logs.example.net:9200/_cluster/health?pretty'
```

Expected evidence is one shared cluster_uuid, three nodes, one elected master marker in _cat/nodes and no missing primary shard. Read the actual status rather than copying sample output: a new empty cluster may be green while an integration index with an unallocated replica can make it yellow. After formation remove cluster.initial_master_nodes from all three files. Never restore that setting during a restart, node replacement or disaster recovery. Keep discovery.seed_hosts.

### Lab exception

For an isolated learning VM keep TLS, use one node and discovery.type: single-node, and omit both discovery.seed_hosts and cluster.initial_master_nodes. Set replicas to zero only in the lab’s own templates, accepting that no second shard copy exists. Do not convert a populated single-node lab to production merely by copying its YAML; build and validate the target cluster and migrate through a supported data or snapshot path.

[Elastic: first cluster bootstrap and removal of initial master settings](https://www.elastic.co/docs/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster)

## Bring Kibana online over HTTPS

On one Elasticsearch member set the kibana_system password interactively. This service user is for Kibana’s backend connection; it is not a browser login. On the Kibana host use the same CA and the kibana certificate issued above. Write /etc/kibana/kibana.yml as follows; do not use the elastic superuser in this file.

```bash
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system -i
```

```yaml
server.name: meetaj-kibana
server.host: 10.20.30.21
server.publicBaseUrl: https://kibana.logs.example.net:5601
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key
elasticsearch.hosts:
  - https://es01.logs.example.net:9200
  - https://es02.logs.example.net:9200
  - https://es03.logs.example.net:9200
elasticsearch.username: kibana_system
elasticsearch.ssl.certificateAuthorities: [/etc/kibana/certs/ca.crt]
elasticsearch.ssl.verificationMode: full
xpack.security.secureCookies: true
```

```bash
sudo env KBN_PATH_CONF=/etc/kibana /usr/share/kibana/bin/kibana-keystore create
sudo env KBN_PATH_CONF=/etc/kibana /usr/share/kibana/bin/kibana-keystore add elasticsearch.password
sudo -u kibana /usr/share/kibana/bin/kibana-encryption-keys generate
# Add each generated value interactively; do not paste real keys into a ticket.
sudo env KBN_PATH_CONF=/etc/kibana /usr/share/kibana/bin/kibana-keystore add xpack.security.encryptionKey
sudo env KBN_PATH_CONF=/etc/kibana /usr/share/kibana/bin/kibana-keystore add xpack.encryptedSavedObjects.encryptionKey
sudo env KBN_PATH_CONF=/etc/kibana /usr/share/kibana/bin/kibana-keystore add xpack.reporting.encryptionKey
sudo chown root:kibana /etc/kibana/kibana.keystore
sudo chmod 0640 /etc/kibana/kibana.keystore
sudo systemctl enable --now kibana
sudo journalctl -u kibana -n 80 --no-pager
curl --cacert ./ca.crt -I https://kibana.logs.example.net:5601/login
```

Use the generated encryption values in the corresponding prompts and retain them securely for recovery. A 200 or login redirect over a verified certificate confirms the web path, not ingest health. Log in first with the bootstrap administrator, create named operator accounts and least-privilege roles, then reserve elastic for recovery. Restrict read roles to required logs-* datasets and spaces. Identity-provider SSO, detailed audit logging and advanced security features must be checked against the subscription matrix; do not budget them as universally Basic.

[Elastic: secure settings and keystores](https://www.elastic.co/docs/deploy-manage/security/secure-settings)

[Elastic self-managed subscription feature matrix](https://www.elastic.co/subscriptions)

## Separate agent enrollment from log delivery

In Kibana open Fleet → Settings. Set Fleet Server hosts to https://fleet.logs.example.net:8220. Configure the default Elasticsearch output with all three HTTPS endpoints and the PEM CA under advanced SSL settings: ssl.certificate_authorities: ["/etc/elastic-agent/certs/ca.crt"]. That path must exist on every agent; alternatively embed the public PEM CA in the output YAML. The certificate-authorities install flag trusts Fleet Server; it does not automatically solve trust for the Elasticsearch data output.

Create a Fleet Server policy and a service token through Add Fleet Server using the production/advanced flow. Record the actual policy ID. Download the 9.5.6 Linux x86_64 Agent archive and its published SHA-512 from Elastic’s official download page; verify the checksum before extracting. On the Fleet host put fleet.crt, fleet.key and ca.crt in /etc/elastic-agent/certs/ with the permissions described above. Set the paths below to the verified archive already obtained.

```bash
sha512sum elastic-agent-9.5.6-linux-x86_64.tar.gz
tar -xzf elastic-agent-9.5.6-linux-x86_64.tar.gz
cd elastic-agent-9.5.6-linux-x86_64
read -r -p 'Fleet policy ID: ' FLEET_POLICY_ID
read -r -s -p 'Fleet service token: ' FLEET_SERVICE_TOKEN
printf '
'
sudo ./elastic-agent install \
  --url=https://fleet.logs.example.net:8220 \
  --fleet-server-es=https://es01.logs.example.net:9200 \
  --fleet-server-service-token="$FLEET_SERVICE_TOKEN" \
  --fleet-server-policy="$FLEET_POLICY_ID" \
  --fleet-server-es-ca=/etc/elastic-agent/certs/ca.crt \
  --certificate-authorities=/etc/elastic-agent/certs/ca.crt \
  --fleet-server-cert=/etc/elastic-agent/certs/fleet.crt \
  --fleet-server-cert-key=/etc/elastic-agent/certs/fleet.key \
  --fleet-server-port=8220
unset FLEET_SERVICE_TOKEN
sudo elastic-agent status
```

The token is not hardcoded or retained in shell history, but expanded CLI arguments may be observable by privileged local processes. Run enrollment in a restricted administrative session; for automation use the release-supported token-file option or a protected secret-delivery mechanism after checking the command reference. Never put tokens into a shared script repository. Do not add --insecure to hide a CA or SAN error.

### Enroll an Ubuntu endpoint

Create a separate infrastructure agent policy, add System integration, and obtain an enrollment token for that policy. Install the public CA at the same /etc/elastic-agent/certs/ca.crt path. Extract the verified matching archive and run the following from its directory. Fleet must show Healthy and the endpoint must also send events; Healthy without new documents is insufficient.

```bash
read -r -s -p 'Agent enrollment token: ' ENROLLMENT_TOKEN
printf '
'
sudo ./elastic-agent install \
  --url=https://fleet.logs.example.net:8220 \
  --enrollment-token="$ENROLLMENT_TOKEN" \
  --certificate-authorities=/etc/elastic-agent/certs/ca.crt
unset ENROLLMENT_TOKEN
sudo elastic-agent status
logger -t meetaj-acceptance 'elastic collection acceptance marker'
```

[Official Elastic Agent packages and checksums](https://www.elastic.co/downloads/elastic-agent)

[Fleet: certificates and TLS for management and data paths](https://www.elastic.co/docs/reference/fleet/secure-connections)

[Elastic Agent install/enroll command reference](https://www.elastic.co/docs/reference/fleet/agent-command-reference)

## Collect each source through its supported interface

### Ubuntu and Windows Server

For Ubuntu enable System auth/syslog datasets and confirm /var/log/auth.log and /var/log/syslog actually exist. Minimal images may use journald without those files: choose the supported journald input or configure rsyslog, rather than monitoring a nonexistent path. Avoid granting blanket read access to all /var/log files. The logger marker must appear with the endpoint host.name and recent @timestamp.

On Windows Server use the verified 9.5.6 Windows x86_64 Agent package from an elevated PowerShell session. Enroll with the command generated by Fleet, replacing its CA path with C:\ProgramData\Elastic\certs\ca.crt and setting the output CA path appropriately for Windows or using embedded PEM. Enable System Application/System/Security event channels or the Windows integration datasets actually required; Security collection requires suitable service privileges and audit policy. Event 4625 appears only when the relevant Windows auditing is enabled.

```powershell
Get-Service 'Elastic Agent'
Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4625} -MaxEvents 5
& 'C:\Program Files\Elastic\Agent\elastic-agent.exe' status
```

### Nginx and Docker

Add Nginx integration to the web-server policy, set access/error paths, and match its supported log format. Start with /var/log/nginx/access.log and error.log only if that server uses those paths. Send one controlled request returning a known status and find the matching event; multiline custom formats need explicit parsing. The stub_status endpoint supplies optional metrics and should remain internal.

For Docker, enable the Docker container_logs dataset and grant access to the log files selected by the active logging driver. The default json-file path is /var/lib/docker/containers/*/*-json.log; journald or remote drivers require a different collection path. Docker discovery/metrics may require the Unix socket; even a read-only mount of docker.sock exposes a powerful API, so do not treat it as harmless file access. Keep multiline exception handling bounded and verify container.id survives parsing.

[Elastic Nginx integration: paths, formats and datasets](https://www.elastic.co/docs/reference/integrations/nginx)

[Elastic Docker integration: files and socket requirements](https://www.elastic.co/docs/reference/integrations/docker)

### Cisco, MikroTik and ESXi: collect on a relay

Install Agent on a dedicated Linux collector, not on an IOS switch or ESXi host. Give Cisco IOS integration its own listener, for example TCP 9002, with ACLs limited to approved device IPs. On supported IOS/IOS XE configure service timestamps log datetime msec localtime show-timezone and logging host 10.20.30.22 transport tcp port 9002; choose a stable logging source-interface and severity appropriate to the incident requirements. The exact interface name and transport support depend on the switch release. Verify the actual syslog with a controlled interface event during an approved test.

For MikroTik use its remote logging action into a dedicated Custom UDP Logs listener, for example port 9514, or a local syslog relay. This guide does not assume an official MikroTik-specific parser. On a RouterOS version supporting these fields the example below sends info and warning topics. Avoid enabling debug globally; retain raw messages and add a reviewed parser for topics, device identity and timestamps. UDP has neither delivery acknowledgement nor TLS: keep it on a management segment or inside a protected tunnel and monitor relay loss.

```routeros
/system logging action add name=elasticremote target=remote remote=10.20.30.22 remote-port=9514 remote-log-format=syslog
/system logging add topics=info action=elasticremote
/system logging add topics=warning action=elasticremote
/system logging print
/system logging action print detail
```

The VMware vSphere integration collects metrics through vCenter/ESXi API and logs through a syslog receiver; these are distinct paths. Enable its log listener on the collector and configure ESXi remote syslog for that listener, open the ESXi syslog firewall ruleset, reload syslog and send a test message using the release-supported ESXCLI procedure. Preserve vmkernel/hostd log identity. API credentials for metrics should be read-only; API polling alone does not ship all ESXi log files. TLS support and trusted CA configuration must be checked for the exact ESXi release.

[Elastic Cisco IOS integration and supported inputs](https://www.elastic.co/docs/reference/integrations/cisco_ios)

[MikroTik RouterOS logging configuration](https://help.mikrotik.com/docs/spaces/ROS/pages/328094/Log)

[RouterOS current logging action reference: remote-log-format and transport](https://manual.mikrotik.com/docs/cli-reference/system/logging/action/)

[Elastic vSphere: separate log and metrics paths](https://www.elastic.co/docs/reference/integrations/vsphere)

For a collector with the vSphere log listener explicitly configured on UDP 514, the following ESXCLI sequence demonstrates a management-segment test. It is unencrypted and lossy, so use a protected management path; where evidence policy requires encrypted delivery, configure an approved TLS syslog relay and ESXi certificate trust instead. The syslog mark must appear at the collector with the ESXi host identity before considering collection complete.

```bash
esxcli system syslog config get
esxcli system syslog config set --loghost='udp://10.20.30.22:514'
esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli network firewall refresh
esxcli system syslog reload
esxcli system syslog mark --message='meetaj-esxi-acceptance'
esxcli system syslog config get
```

[Broadcom: ESXi remote syslog, firewall and acceptance marker](https://knowledge.broadcom.com/external/article/318939/configuring-syslog-on-esxi.html)

### SQL Server audit is a configured source

Use Microsoft SQL Server integration for its documented audit/error-log/metrics datasets; do not confuse transaction_log metrics with a database transaction-log backup. Its audit dataset reads a configured Windows event channel, so first create and enable SQL Server Audit and specifications with the appropriate event-log target and permissions. A .sqlaudit binary file is not automatically parsed by monitoring its directory as text. Error-log collection follows that package’s supported file settings; the 2.18.0 package documents filestream handling for UTF-16LE-BOM logs. Pin the package version in the deployment record and verify Windows channel/event fields using one controlled failed database login.

[Microsoft SQL Server integration: audit targets, logs and permissions](https://www.elastic.co/docs/reference/integrations/microsoft_sqlserver)

[Microsoft: SQL Server Audit architecture and targets](https://learn.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine)

## Keep raw evidence, then normalise a small sample

ECS is a field convention: host.name identifies a host, source.ip is an IP field, event.outcome describes success/failure and @timestamp is the event time. It is not a parser that understands every vendor line. Grok extracts known text patterns; date parsing must handle timezone explicitly. A mapping decides field types and cannot be casually changed after incompatible data has been indexed. Test a parser with successful, failed and malformed events before increasing throughput.

The following example is a custom application contract, not a claim about a default Cisco or Nginx format: 2026-11-07T09:12:00Z WARN auth-gateway source=192.0.2.15 outcome=failure. Run these REST examples in Kibana Dev Tools → Console with a temporary provisioning role. Managed integration data streams already own their templates and pipelines; do not overwrite logs-*-* or edit the managed pipeline in place.

```http
PUT _ingest/pipeline/meetaj-app-v1
{
  "processors": [
    {"set": {"field": "event.original", "copy_from": "message", "override": false}},
    {"grok": {"field": "message", "patterns": ["%{TIMESTAMP_ISO8601:tmp_time} %{LOGLEVEL:log.level} %{NOTSPACE:service.name} source=%{IP:source.ip} outcome=%{WORD:event.outcome}"]}},
    {"date": {"field": "tmp_time", "formats": ["ISO8601"], "target_field": "@timestamp"}},
    {"remove": {"field": "tmp_time"}}
  ],
  "on_failure": [{"set": {"field": "error.message", "value": "{{ _ingest.on_failure_message }}"}}]
}
POST _ingest/pipeline/meetaj-app-v1/_simulate
{"docs":[{"_source":{"message":"2026-11-07T09:12:00Z WARN auth-gateway source=192.0.2.15 outcome=failure"}}]}
```

Expect a parsed source.ip, event.outcome=failure and the original UTC timestamp; a malformed line should retain event.original and gain error.message. This example keeps failures in the stream for inspection; a high-volume deployment should route or alert on failures so a parser regression is visible. A literal date in the acceptance sample is deliberate: Discover must use that date range.

[Elastic ingest pipelines and simulation](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines)

## Retention is a policy, not an empty disk promise

![Hot writable data rolls over, moves to warm storage when available, then expires; snapshots are independent](/assets/img/articles/content/elk-stack-enterprise-installation-guide/lifecycle.svg)

A data stream is an append-oriented name backed by indices and requires @timestamp. Its index template supplies mappings, shard settings and an ILM policy. A primary shard is a unit of indexing and search; a replica is another copy on a different node. More shards add overhead and do not automatically make a small dataset faster. Use rollover based on primary shard size and age, then measure actual shard size and query latency.

This standalone custom namespace uses 30 days retention and one primary/one replica. It does not apply to integration-owned streams. The warm phase is omitted because the small-production topology has no dedicated warm tier. Add a warm phase only after provisioning nodes with the corresponding roles. ILM age after rollover is measured from rollover time, so a 30-day delete phase is not an exact per-document legal expiry mechanism.

```http
PUT _ilm/policy/meetaj-logs-30d
{
  "policy": {"phases": {
    "hot": {"actions": {"rollover": {"max_primary_shard_size": "30gb", "max_age": "1d"}}},
    "delete": {"min_age": "30d", "actions": {"delete": {}}}
  }}
}
PUT _index_template/meetaj-app-template
{
  "index_patterns": ["logs-meetaj.app-*"],
  "priority": 501,
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "meetaj-logs-30d",
      "index.default_pipeline": "meetaj-app-v1"
    },
    "mappings": {"properties": {
      "@timestamp": {"type": "date"},
      "source": {"properties": {"ip": {"type": "ip"}}},
      "event": {"properties": {"outcome": {"type": "keyword"}}},
      "service": {"properties": {"name": {"type": "keyword"}}}
    }}
  }
}
POST logs-meetaj.app-prod/_doc?refresh=wait_for
{"@timestamp":"2026-11-07T09:12:00Z","message":"2026-11-07T09:12:00Z WARN auth-gateway source=192.0.2.15 outcome=failure"}
GET logs-meetaj.app-prod/_ilm/explain
GET _cat/shards/logs-meetaj.app-prod?v
```

### A 100 GB/day storage estimate

Assume 100 decimal GB/day before parsing, a measured primary-index expansion factor of 1.2, 30 days searchable, one replica and a target 75% disk utilisation. Searchable capacity is 100 × 1.2 × 30 × 2 ÷ 0.75 = 9,600 GB, about 9.6 TB decimal. The 1.2 factor is a hypothetical pilot result, not an Elastic guarantee; some mappings expand more and some datasets compress better. Translogs, merges, delayed rollover, system indices and temporary recovery consume additional space. This is cluster-wide usable data capacity, not 9.6 TB on every node.

At 120 GB of primary data per day, a 30 GB target implies roughly four primary shards per day across the dataset; rollover cadence and source partitioning affect the actual count. A one-primary stream rolls over several times daily under that load. For three equally loaded data nodes, surviving one failure while retaining the same 75% target needs approximately 14.4 TB aggregate pre-failure usable capacity, because only two-thirds remain. Add the snapshot repository separately; snapshots are incremental, so size it from changed segments and retention rather than multiplying the full dataset by the number of snapshots.

[Elastic: shard sizing and rollover guidance](https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards)

[Elastic: index lifecycle management](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management)

## Add Logstash where buffering or custom routing pays for itself

A local relay may already write normalised application lines to /var/log/meetaj/app.log. The optional pipeline below reads only that file, uses the custom ingest pipeline created above and writes to the custom data stream. The relay-to-file permissions and rotation policy must be arranged first. This is not the path for Fleet-managed integration streams, whose datasets and ingest pipelines must be preserved. Set queue.type: persisted and a disk budget such as queue.max_bytes: 8gb in /etc/logstash/logstash.yml; a persistent queue buffers bounded outages but does not replace source durability or HA.

```ruby
input {
  file {
    path => ["/var/log/meetaj/app.log"]
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/meetaj-app.sincedb"
  }
}
output {
  elasticsearch {
    hosts => ["https://es01.logs.example.net:9200", "https://es02.logs.example.net:9200"]
    ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
    api_key => "${ES_API_KEY}"
    ecs_compatibility => "v8"
    data_stream => true
    data_stream_type => "logs"
    data_stream_dataset => "meetaj.app"
    data_stream_namespace => "prod"
    pipeline => "meetaj-app-v1"
    manage_template => false
    ilm_enabled => false
  }
}
```

Save it as /etc/logstash/conf.d/meetaj-app.conf. Provision the template/pipeline separately with an administrator role. Give Logstash an API key scoped to create_doc for logs-meetaj.app-* and the auto_configure permission only if it must create the stream; grant only the cluster privileges required by the selected output-plugin checks. Store the id:api_key value interactively in the Logstash keystore, not in the .conf file. Runtime Logstash should not have manage_security or privileges to read unrelated indices.

```bash
sudo /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
sudo /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_API_KEY
sudo chown logstash:root /etc/logstash/logstash.keystore
sudo chmod 0600 /etc/logstash/logstash.keystore
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.test_and_exit
sudo systemctl enable --now logstash
sudo journalctl -u logstash -n 80 --no-pager
```

The validation command should report a valid configuration; that checks syntax, not end-to-end access or parsing. Append one approved sample line, find its document, restart Logstash and confirm the file checkpoint prevents a duplicate replay. start_position applies only when there is no existing sincedb checkpoint. Monitor queue growth and oldest-event delay; a queue that always grows is a capacity problem.

[Logstash Elasticsearch output: TLS, API keys and data streams](https://www.elastic.co/docs/reference/logstash/plugins/plugins-outputs-elasticsearch)

[Logstash persistent queues and durability limits](https://www.elastic.co/docs/reference/logstash/persistent-queues)

## Build a dashboard that can expose missing data

Create a logs-* data view in Discover with @timestamp as its time field, then narrow it by dataset. For the fixed example above use the 7 November 2026 UTC time range. For live sources use the recent interval and compare @timestamp with event.ingested when the integration provides it. A graph with zero errors cannot distinguish a healthy application from a stopped collector. Include events per host, last-seen time and parse failures beside service failures.

```kql
data_stream.dataset : "system.auth" and event.outcome : "failure"
data_stream.dataset : "nginx.access" and http.response.status_code >= 500
data_stream.dataset : "meetaj.app" and error.message : *
host.name : "web01" and @timestamp >= now-15m
```

These are separate KQL filters, not a single multiline query. Validate actual ECS fields in a received event before using the first filter; integrations do not always categorise every failure identically. In Lens chart Nginx status-code counts over five minutes, add the top host.name table and save it in an operations dashboard. A practical rule is more than 20 HTTP 5xx events for one host over five minutes, evaluated every minute with a look-back longer than the expected ingestion delay. Calibrate the number using traffic and maintenance windows.

Create a separate missing-data rule for expected active hosts; group by host.name and define what absence means for quiet servers. Save a controlled alert test and its recovery notification. Connector availability and rule types depend on the subscription; use the licensed connector actually available rather than promising every email, webhook or ML rule on Basic. Log management does not itself supply SOC triage ownership, detection coverage or incident response.

[Keep metric dashboards connected to the log investigation workflow](/articles/grafana-installation-zabbix-integration)

[Zabbix deployment: host monitoring alongside event collection](/articles/zabbix-server-linux-windows-agents-backup)

## Scale roles after measuring contention

![Enterprise roles: three masters, hot/content data, optional warm/ingest pools, redundant Kibana and Fleet, external snapshots](/assets/img/articles/content/elk-stack-enterprise-installation-guide/production.svg)

For a larger cluster move master duties onto three dedicated nodes with node.roles: [master]. Data nodes need data_hot and data_content, or the broader data role; add data_warm only for a real warm pool. If all data nodes lose ingest and master nodes are master-only, at least one separate ingest node is needed for pipelines; deploy two for availability. Never send routine client searches to the dedicated masters. The three master-eligible nodes require a majority of two for elections and cluster-state publication; two masters do not provide the same single-failure resilience.

Put two Kibana instances behind a health-checked HTTPS load balancer, keep required encryption keys identical and server identities unique, and distribute Elasticsearch endpoints. Deploy two Fleet Servers across failure domains and advertise their supported host list or a health-checked endpoint. Existing agents generally continue collecting with their last policy during a Fleet management outage, but cannot receive policy changes or enroll new hosts. Test both control-plane and data-plane failure separately.

Do not stretch one cluster casually across a high-latency WAN. For branch outages place durable relays or appropriately sized collectors near sources, estimate outage buffering as peak bytes/sec × tolerated outage duration, and monitor the oldest buffered event. Separate regional clusters with snapshots and a reviewed remote-search design may fit better than one geographically fragile quorum. Cross-cluster replication and searchable snapshot capabilities have subscription conditions; a diagram does not grant them.

[Elastic node roles and mandatory data roles](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/node-settings)

Keep transform capacity in the enterprise role plan: Fleet and relevant Elastic Security workflows require the transform role. The mixed-role installation already includes it; when splitting roles, retain it on suitable data nodes or provide a dedicated transform pool. Required ml or remote_cluster_client roles likewise follow the enabled feature set rather than a blanket removal of defaults.

## Prove a restore before relying on replicas

A replica serves availability; deletion, a bad mapping operation or compromised credentials can affect both copies. Elasticsearch snapshots are the supported data backup mechanism; copying a live data directory or taking unrelated VM snapshots is not a supported cluster backup. Choose an external object repository or a correctly implemented shared filesystem with independent failure domains and restricted credentials.

For the following filesystem example mount the same /mnt/elastic-snapshots path on every master/data node, provide consistent Elasticsearch UID/GID access and filesystem locking semantics, then add path.repo: ["/mnt/elastic-snapshots"] to their YAML and perform a controlled rolling restart. This is a real shared mount, not five unrelated local directories. Only one cluster writes the repository; register it read-only on a restore cluster.

```http
PUT _snapshot/meetaj_backup
{"type":"fs","settings":{"location":"/mnt/elastic-snapshots","compress":true}}
POST _snapshot/meetaj_backup/_verify
PUT _slm/policy/meetaj-nightly
{
  "schedule":"0 30 1 * * ?",
  "name":"<meetaj-{now/d}>",
  "repository":"meetaj_backup",
  "config":{"indices":["logs-*"],"include_global_state":true},
  "retention":{"expire_after":"30d","min_count":3,"max_count":40}
}
POST _slm/policy/meetaj-nightly/_execute
GET _slm/stats
GET _snapshot/meetaj_backup/_all
```

The cron is 01:30 UTC, not the browser’s local time. Verification must list reachable repository nodes and the completed snapshot must show SUCCESS with no failed shards. Review feature-state coverage for the required system data; include_global_state does not back up node YAML, keystores or external PKI. Store service configuration, integration versions, enrollment recovery procedures and encryption keys separately. Set snapshot-age and failure alerts.

```http
POST _snapshot/meetaj_backup/REPLACE_WITH_COMPLETED_SNAPSHOT/_restore
{
  "indices":"logs-meetaj.app-prod",
  "include_global_state":false,
  "rename_pattern":"(.+)",
  "rename_replacement":"restored-$1"
}
GET _cat/recovery/restored-*?v
GET restored-logs-meetaj.app-prod/_count
```

Run the restore only in a compatible isolated recovery cluster, with required matching templates for restored data streams and enough storage. Replace the snapshot-name placeholder. Wait for recovery, search the known marker and compare document counts and time coverage with the acceptance record. A green status alone does not prove the right data was restored. Record measured RPO/RTO; nightly snapshots alone can imply almost a day of data loss.

[Elastic: supported snapshot/restore and feature-state coverage](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore)

[Elastic: shared filesystem repository requirements](https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore/shared-file-system-repository)

[Design HA, backups and disaster recovery as separate controls](/articles/high-availability-backup-disaster-recovery-enterprise-guide)

## Troubleshoot from evidence to a bounded corrective action

The commands below describe expected evidence, not captured outputs. REST requests belong in Kibana Console unless a shell command is shown. Keep the time range, affected source and last known good event with each incident. Avoid deleting indices, forcing stale primaries or removing TLS verification merely to turn a status green.

### Service fails to start

```bash
sudo systemctl status elasticsearch --no-pager
sudo journalctl -u elasticsearch -b -n 120 --no-pager
sudo tail -n 100 /var/log/elasticsearch/meetaj-logs-prod.log
systemctl show elasticsearch -p LimitNOFILE
sysctl vm.max_map_count
```

A failed unit and its first configuration/bootstrap exception should identify a bad YAML key, unreadable certificate, low limit or occupied mount. Correct that specific error, restore file ownership or the mount, then restart and verify an active unit plus the same cluster UUID. journalctl may contain only startup context; the application log is essential.

### Yellow cluster

```http
GET _cluster/health
GET _cat/shards?v
GET _cluster/allocation/explain
```

Yellow means primaries are assigned but at least one replica is not. Expect UNASSIGNED on replica rows; allocation explain distinguishes insufficient nodes, tier filters and disk limits. Restore capacity or correct the relevant placement policy, then require assigned replicas. Setting production replicas to zero conceals the availability problem.

### Red cluster

```http
GET _cluster/health
GET _cat/shards?v
POST _cluster/allocation/explain
{"index":"REPLACE_WITH_AFFECTED_INDEX","shard":0,"primary":true}
```

Red indicates an unassigned primary and incomplete data access. Expect an allocation decision for the actual affected shard; replace both index and shard number. Reconnect the original node/storage when safe, or restore a known snapshot. If no valid shard copy remains, treat it as data loss requiring an explicit recovery decision; a stale-primary force allocation can lose acknowledged writes. Verify data time coverage after recovery.

### Unassigned shards

```http
GET _cat/allocation?v
GET _cluster/settings?include_defaults=true&flat_settings=true
GET _cluster/allocation/explain
```

Expect a named decider and NO/THROTTLE reason. A warm-only filter with no warm node, awareness constraint or recovery throttle has a different remedy from disk exhaustion. Correct the specific cause and wait for recovery; do not clear all routing settings. Confirm STARTED state and intended failure-domain placement.

### Disk watermark exceeded

```http
GET _cat/allocation?v
GET _cluster/settings?include_defaults=true&flat_settings=true
GET logs-*/_settings/index.blocks.read_only_allow_delete
```

Expect high disk.percent or a flood-stage read-only block. Confirm filesystem consumption on the affected host; expand storage or remove data through the approved retention process after backup/retention checks. Do not raise watermarks to postpone a full disk. Once usage is below the applicable high watermark the flood-stage block is normally released automatically; verify a new approved test event is indexed.

### Kibana cannot connect

```bash
sudo journalctl -u kibana -n 120 --no-pager
getent hosts es01.logs.example.net
curl --cacert /etc/kibana/certs/ca.crt -I https://es01.logs.example.net:9200/
```

A 401 from an unauthenticated curl proves HTTP/TLS reachability; it does not prove Kibana credentials. Timeout suggests route/firewall; TLS failure suggests trust/SAN; Kibana log 401 suggests its service password or keystore. Correct the matching layer, restart Kibana if its secure settings changed, and verify login plus a Discover query.

### Logs never arrive

```text
sudo elastic-agent status
sudo elastic-agent diagnostics
ss -lntup
GET logs-*/_search
{"size":1,"sort":[{"@timestamp":"desc"}],"query":{"term":{"host.name":"REPLACE_WITH_HOST"}}}
```

Run the first three as shell commands and the GET/body in Console. Expect healthy integration units, the configured listener on a collector and a current document. A healthy agent with no document may have a wrong file path, unreadable rotated file, missing audit policy, parser rejection or output/API-key failure. Correct one path and send a unique marker; verify it end to end. Diagnostics archives can contain secrets and must stay restricted.

### TLS trust or certificate errors

```bash
openssl s_client -connect es01.logs.example.net:9200 -servername es01.logs.example.net -CAfile ./ca.crt -verify_return_error </dev/null
openssl x509 -in ./node.crt -noout -dates -ext subjectAltName
```

Expect successful chain verification and a SAN matching the requested DNS name. Unknown authority means the correct issuing chain is absent; expired dates or name mismatch require renewal/reissuance, not verification_mode: none. Distribute the corrected public chain, rotate service certificates using the supported reload/restart path and repeat the same verified connection.

### Authentication and authorisation failures

```http
GET _security/_authenticate
GET _security/user/_privileges
```

Expect the intended principal and only its required privileges. HTTP 401 indicates invalid/expired credentials or the wrong realm; 403 indicates an authenticated principal lacking an action privilege. Rotate or reissue the credential with a scoped role, update its secure store, and verify the exact failed ingest/search action. Do not grant superuser as a permanent fix.

### Sustained JVM pressure

```http
GET _nodes/stats/jvm,breaker
GET _nodes/hot_threads
GET _cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m
```

Short saw-tooth heap use is normal; sustained pressure with frequent old-generation GC, circuit-breaker trips or queue growth is not. Correlate expensive aggregations, shard count and ingestion bursts with node resources. Reduce query breadth or field cardinality, rebalance shards and add capacity based on evidence. Raising heap can remove filesystem cache and worsen latency. Verify sustained pressure falls under the same representative load.

### Slow searches

```http
GET _nodes/stats/thread_pool,indices
GET _tasks?detailed=true&actions=*search
GET _cat/shards?v&s=store:desc
```

Expect evidence of search queue/rejections, a long-running task or uneven shard sizes; none is automatically the root cause. Narrow the time range, inspect the actual query and use profiling on a controlled sample because profiling adds overhead. Correct wildcard-heavy filters, excessive shards or saturated storage as indicated. Compare the same query’s latency and event completeness before/after, including concurrent ingest.

[Elastic: red/yellow cluster diagnosis and allocation](https://www.elastic.co/docs/troubleshoot/elasticsearch/red-yellow-cluster-status)

[Elastic: disk watermarks and read-only blocks](https://www.elastic.co/docs/troubleshoot/elasticsearch/fix-watermark-errors)

## What must be true before handing over operations?

| Control | Acceptance evidence |
| --- | --- |
| Version and configuration inventory | Exact package/integration versions, reviewed config, support matrix and named owners |
| Security | Verified TLS on every hop, SAN/expiry monitoring, restricted ports, least-privilege users/API keys and secret rotation |
| Collection and parsing | A traceable marker from each source; correct timestamp, host and fields; known malformed-line behaviour |
| Availability | Loss of one eligible node and one collector tested in a controlled window with capacity left for recovery |
| Retention and capacity | Actual primary/replica growth, ILM explain, rollover and legal retention approved; storage alert threshold set |
| Recovery | Completed independent snapshot, isolated restore/search test, measured RPO/RTO and secured config/keys |
| Monitoring and handover | Last-seen, ingestion delay, parse failures, JVM, disk, cluster and snapshot alerts with an owner and escalation path |

Before upgrades take and validate a compatible snapshot, back up configuration and secure settings, read release-specific upgrade prerequisites and test integrations/searches on a representative staging copy. Unhold packages only inside the planned change. Elasticsearch does not support an in-place downgrade of a data directory; rollback means restoring a compatible pre-upgrade snapshot into a separate compatible cluster and redirecting clients, with an explicit plan for writes accepted after that snapshot. Keep old client endpoints and buffering plans until acceptance completes.

[Elastic: supported upgrades and rollback limitations](https://www.elastic.co/docs/deploy-manage/upgrade/deployment-or-cluster/elasticsearch)

[Compare Elastic and Splunk operating models before committing to a platform](/articles/elk-stack-vs-splunk-enterprise)

[Splunk Enterprise deployment: the alternative collection and operations model](/articles/splunk-enterprise-installation-guide)

## Questions that affect the design

### Is Logstash mandatory?

No. Elastic Agent integrations can send directly to Elasticsearch and use ingest pipelines. Add Logstash for a justified transformation, routing or buffering requirement.

### Can replicas replace backups?

No. Replicas improve availability but reproduce logical deletion and corruption. Use an independent snapshot repository and prove a restore.

### Does this installation create a complete SIEM?

It creates a secured log platform. Elastic Security adds security workflows, but source coverage, tuned detections, licensed capabilities and an incident response process still require deliberate implementation.

[Persian Markdown edition](/downloads/elk-stack-enterprise-installation-guide/article.fa.md)

[English Markdown edition](/downloads/elk-stack-enterprise-installation-guide/article.en.md)