The mtail project focuses on extracting metrics from application logs to be ingested into a timeseries database like Prometheus. mtail does this by using a set of user-defined extraction programs reading from specified log files, named pipes or UNIX sockets. It’s intended to run one mtail per machine serving multiple applications.
Kubernetes has a documentation page describing it’s logging
architecture. All system components (kubelet, containerd) which aren’t
running as containers write their logs if systemd is present to journald else in a log file under /var/log. The
documentation mentions two configurations to apply a logging-agent like mtail.
As a side-car to the application, where logs are being shared through a emptyDir volume.
As a node-level logging agent using a DaemonSet and access to the nodes log directories using a hostPath volume.
Kubernetes exposes container logs at /var/log/containers these logs might be symlinks to a different directory like
/var/lib/docker/containers. Therefore, the symlinked directory must be made available through a volume mount to the
logging-agent as well.
An example of mtail being configured as a logging-agent using a DaemonSet can be found on this GitHub Gist.