Simple Splunk Log Ingestion from Docker Containers

I self-host as many services as I practically can. It's liberating on a number of fronts (which I will write about in a future post) while also providing a great way to learn new technologies. As a brief introduction, I host everything from one server using Traefik as a reverse-proxy and docker-compose to manage the services. One advantage of this setup is that Traefik automatically routes data to the correct containers using a handful of labels in the docker-compose.yaml file (at least most of the time!).

However, while doing some work on Via I wanted to view logs in a quick and easy way and monitor them during development on it. There are a lot of recommendations for Prometheus, all of which were inapplicable to me as I was looking for actual log messages and not just visualizing metrics. In work, we use Splunk for managing our logs, and there are many examples using docker-compose to self-host a Splunk instance. However, I only use the free license as I don't see myself using Splunk long-term as a logs viewer for my personal work, so I didn't want to invest a large amount of time building up tooling around getting all my services' logs into it. I wanted a quick and easy, just enough to work type of solution.

Most docker-compose.yaml files use the volumes tag to specify filepaths that are shared between the host machine and the Docker container. These are very useful as they allow your containers to be quickly and easily updated/moved/recreated and make backups easy. They also allow me to share my entire stack configuration without exposing anything sensitive! Using these volumes, you can get your logs from any container on the host into Splunk in no time. First, add the filepath of the logs as a volume similar to this:

version: '3.3'

services:
  whoami-service:
    image: "containuous/whoami:latest"
    
    volumes:
      - /var/log/instance_logs/whoami:/var/log

This tells Docker that the whoami-service container should use the host machine's /var/log/instance_logs/whoami whenever it wants to put anything in /var/log. Now in your Splunk docker-compose.yaml, add that /var/log/instance_logs path as a volume also. This allows the separate Splunk container to read these logs easily:

version: '3.6'

services:
  splunk:
    image: splunk/splunk:latest
    
    volumes:
      - /var/log/instance_logs:/var/log/external_instance_logs

Finally, you can just set up Splunk to read from this directory and set the host to the service it is coming from. Go to Data inputs -> Files & Directories -> New Local File & Directory and configure the following:

File or Directory: /var/log/external_instance_logs
Host: Segment in path (Segment number: 4)
Index: external_instance_logs

And you're done! Log events should be arriving in real time and have the host set to the container you have specified. Any suggestions of better tools for the job, cleaner ways of implementing this functionality, or suggestions of a self-hosted, FOSS alternative to Splunk please get in touch and let me know.