I just tried to run Fluentd instances on Mesos + Docker + Marathon in Vagrant boxes. Here is the sample repository.
The overview diagram:
How?
See the github README.md.
What?
Fluentd
Fluentd is one of log collector middleware. Most web companies in Japan use this for log collecting. td-agent is a package version of fluentd including ruby interpreter.
Some uses want to spin up multiple fluentd workers across many physical servers because fluentd only use one cpu core. Mesos is the right software to manage such situation.
Mesos
Computer resource management software. You can combine many servers as a one “virtual computer” with Mesos. Mesos itself is just allocating resources, so you have to use frameworks to manage these resources.
Marathon
Marathon is a framework on Mesos, which is like “upstart for virtual computer”. Marathon makes sure the number of running tasks, so if one of Mesos slave goes down, Marathon starts new instances on other slaves.
Marathon also manages port of instances. You can get the list of host:port of your applications just asking Marathon API; this is kind of service discovery.
Docker
LXC manager. You don’t have to take care of server environment any more. Just create a docker container and commit it. Then, you can run the application anywhere you want. This is very convenient for computer cluster like Mesos.
Mesosphere provides a simple Mesos executor for docker. It runs docker container and bind ports, so you can connect containers from outside of the slave.
HAProxy
Fluentd has http input interface, so I connect them with HAProxy running on Docker. First getting the list of host:port from Marathon, then generating config file of HAProxy. If the list is updated, just restart the HAProxy container.
Elasticsearch, Kibana
- Open Source Distributed Real Time Search & Analytics | Elasticsearch
- Kibana | Overview | Elasticsearch
Elasticsearch and Kibana is one of the best combination of storing logs and visualizing them.
Fluentd has pluggable input/output and there is Elasticsearch output plugin. I installed it in the fluentd docker image. Giving ES host:port to the container as environment variables, fluentd can connect the ES. I run ES and Kibana on Docker, too.
Conclusion
It was very interesting exercise for me. I’d like to keep practicing Mesos, Docker, Marathon, Vagrant, whatever. I hope this sample helps you, too.
BTW, there were so much trouble especially Vagrant integration. /etc/hosts file is very tricky. I’d like to know more reasonable way to manage Vagrant hosts information…