<![CDATA[As a SW/Ops/DB Engineer]]>2014-12-26T01:21:17+09:00http://tech.riywo.com/Octopress<![CDATA[Terraform module for Mesos + Ceph cluster and Packer template]]>2014-12-25T23:13:16+09:00http://tech.riywo.com/blog/2014/12/25/terraform-module-for-mesos-plus-ceph-cluster-and-packer-templateriywo/mesos-ceph
I’ve just tried to use Terraform and Packer to create a Mesos + Ceph cluster in AWS. Yes, I know Mesosphere applications supporting deployment of Mesos cluster in some IaaS (see Getting Started), but I’d like to understand what’s going on there. So, I did it by Terraform and Packer. I’m gonna explain a little bit more about this.
Through this work, I’ve learned a lot of things. I will write something below too.
What is this?
This repository includes Terraform module and Packer template. Terraform module will manage a VPC subnet, igw, sg, etc. and instances below:
admin
ssh gateway
run ceph-deploy
master1, master2, master3
mesos master
marathon
ceph mon
ceph mds
mount cephfs
slaves (default 3)
mesos slave
ceph osd with EBS
mount cephfs
Why not only Mesos, did I manage Ceph cluster? In Mesos, I’d like to use a shared file system because:
Share executer files, etc.
Control cluster software on Mesos
Save uploaded files to Mesos tasks (ex. image files to blog apps)
Or just for fun :)
How to use?
Very simple, make your own terraform config like below:
After spinning up all instances, this resource runs provisioners to initialize Ceph cluster like above. Once created this virtual resource, this procedure won’t be executed any more.
Ceph initializing and provisioning
Considering initializing process above, there are two types of resource creation; Initializing and provisioning.
Initializing means the completely initial timing. At this time, each instance doesn’t need to be provisioned because the cluster initialization process will do everything.
Provisioning, on the other hand, means after the cluster is initialized. For example, when a master/slave/admin instance is terminated, Terraform tries to re-create the instance and this is provisioning. It is a little bit different process than initializing because the cluster already exists.
See terraform/scripts/init_master.sh. if ceph_initialized; block is for this problem. So, all instances can be terminated anytime even after a Ceph cluster is initialized. Try terminate a instance and terraform apply!
SSH gateway and provisioners
Terraform provisioners assume the instance can be connected by ssh directly. In this module, I don’t open ssh port for cluster instances except admin instance, so it is a SSH gateway.
If you configure connection block to the gateway instead of the actual instance, you can use provisioners, but they run on the gateway.
So, I copy AWS key file into the admin instance to be able to login to other instances (there is not a way to use agent forward so far).
For each instance provisioning, I upload script files prefixed by the instance id to prevent overwriting race condition. See terraform/master1.tf. script_path option for connection is undocumented so far, btw.
Master IP addresses
I want to fix master IP addresses because they are hardcoded anywhere. I tried Terraform variable as list of IP addresses, but so far it was impossible.
Because of that, I fixed the number of masters and created each aws_instance resource.
When the list variable feature is implemented, I will refactor these configuration.
Concatenate provisioner scripts
I use bash script for provisioners and there are a lot of common functions, so I wrote a shared script terraform/scripts/header.sh.
To run each provisioner correctly, this file must be concatenated. Also, an entry point main must be concatenated. So I did like this:
12345
provisioner "remote-exec" {
inline = [
"echo main foo bar | cat header.sh init_foo.sh - | bash"
]
}
I think there are much better ways, though…
Isolation between Terraform and Packer
There is a philosophy:
Packer
Install file resources from the Internet
Terraform
Install only runtime information, like IP address
Do not fetch file resources from the Internet
So, any apt-get install are done by Packer nor Terraform. This philosophy is like The Twelve-Factor App.
BTW, awk is great
I wanted a way to use the instance id built by Packer in Terraform automatically. I found that Packer has -machine-readable output option and it is CSV format. So, I started to write a processor script:
Print the machine readable output as well as normal Packer output
Write ami.tf file using the instance id built just now
See packer/process. This is my first awk executable script. awk was very nice in this case and I could find a lot of awk tips.
To write complex shell script for Packer/Terraform, I read the bashstyle article above. I don’t understand all of them, but there are tons of best practice.
Conclusion
It was fun!
Hey, wait a moment… At the beginning, I just wanted to run many applications on Mesos using Docker, for example WordPress, MySQL Galera Cluster, etc. To implement them, I needed a shared file system, so I started to learn about Ceph which I had an eye on before. To deploy the cluster into AWS, I needed some orchestration software, then started to see Terraform, Packer to create AMI… Too long yak shaving it was ;(
Now, let’s start learning about Mesos/Marathon/Docker and many applications!
]]><![CDATA[Fluentd on Mesos + Docker + Marathon]]>2013-12-20T17:13:00+09:00http://tech.riywo.com/blog/2013/12/20/fluentd-on-mesos-plus-docker-plus-marathonI just tried to run Fluentd instances on Mesos + Docker + Marathon in Vagrant boxes. Here is the sample repository.
Fluentd is one of log collector middleware. Most web companies in Japan use this for log collecting. td-agent is a package version of fluentd including ruby interpreter.
Some uses want to spin up multiple fluentd workers across many physical servers because fluentd only use one cpu core. Mesos is the right software to manage such situation.
Computer resource management software. You can combine many servers as a one “virtual computer” with Mesos. Mesos itself is just allocating resources, so you have to use frameworks to manage these resources.
Marathon is a framework on Mesos, which is like “upstart for virtual computer”. Marathon makes sure the number of running tasks, so if one of Mesos slave goes down, Marathon starts new instances on other slaves.
Marathon also manages port of instances. You can get the list of host:port of your applications just asking Marathon API; this is kind of service discovery.
LXC manager. You don’t have to take care of server environment any more. Just create a docker container and commit it. Then, you can run the application anywhere you want. This is very convenient for computer cluster like Mesos.
Mesosphere provides a simple Mesos executor for docker. It runs docker container and bind ports, so you can connect containers from outside of the slave.
Fluentd has http input interface, so I connect them with HAProxy running on Docker. First getting the list of host:port from Marathon, then generating config file of HAProxy. If the list is updated, just restart the HAProxy container.
Elasticsearch and Kibana is one of the best combination of storing logs and visualizing them.
Fluentd has pluggable input/output and there is Elasticsearch output plugin. I installed it in the fluentd docker image. Giving ES host:port to the container as environment variables, fluentd can connect the ES. I run ES and Kibana on Docker, too.
Conclusion
It was very interesting exercise for me. I’d like to keep practicing Mesos, Docker, Marathon, Vagrant, whatever. I hope this sample helps you, too.
BTW, there were so much trouble especially Vagrant integration. /etc/hosts file is very tricky. I’d like to know more reasonable way to manage Vagrant hosts information…
]]><![CDATA[chown with user's default group]]>2013-10-11T12:55:00+09:00http://tech.riywo.com/blog/2013/10/11/chown-with-users-default-groupYou may have used chown command to change the ownership of files or directories like below.
… If a colon but no group name follows the user name, that user is made the owner of the files and the group of the files is changed to that user's login group. …
]]><![CDATA[mesos tutorial 3]]>2013-09-29T10:24:00+09:00http://tech.riywo.com/blog/2013/09/29/mesos-tutorial-3To understand development of Mesos Framework, I developed a simple useless framework example called msh.
msh launches a msh-executor using one of resource offers
msh-executor starts the given command as a subprocess
msh reads its stdin and send the input to the slave as FrameworkMessage
msh-executor receives messages and write them to the subprocess.stdin
msh-executor reads the subprocess.stdout and sends the output to msh as FrameworkMessage
Each other sends a message ‘EOF’ as a signal of end of the pipe
msh prints received message
TODO
work with no stdin
streaming pipe
pipe multiple tasks like UNIX shell
interactive shell
error handling
Conclusion
Development of Mesos framework in Python is very easy. I hope this toy project is useful for someone who is interested in Mesos.
]]><![CDATA[mesos tutorial 2]]>2013-09-28T00:02:00+09:00http://tech.riywo.com/blog/2013/09/28/mesos-tutorial-2In the pervious post, I tried “cpu = 0.1” for Mesos tasks. But it didn’t work precisely. If you have only one cpu on each slave server, you can run only one task on each slave at the same time even if you specify “cpu = 0.1” for the tasks.
It only happens on the slave which has just one cpu. If a slave has more than one cpus, it can run tasks more than the number of cpus. I use m1.small and it has only one cpu, unfortunately…
So, I changed Mesosphere’s runner script a little to configure --resources flag of mesos-slave.
Now you can see 2 cpus in your Mesos cluster, then you can run more than 2 tasks on it at the same time.
]]><![CDATA[mesos introduction 1]]>2013-09-27T00:53:00+09:00http://tech.riywo.com/blog/2013/09/27/mesos-introduction-1Today, Mesosphere released an important milestone. I really desired this integration.
I would like to learn this new generation “OS”, so I started to play with Mesos. Mesosphere is a great company and they built packages of Mesos and Marathon. Building Mesos and tools are most painful part of this ecosystem, but now we can super easily start to use Mesos and Marathon. Thanks, Mesosphere:)
I use AWS for Mesos cluster. At first, I created a m1.small instance for a master of Mesos, Zookeeper and Marathon. I use EC2-Classic.
]]><![CDATA[My Github Trends 2013-09-04]]>2013-09-04T22:19:00+09:00http://tech.riywo.com/blog/2013/09/04/my-github-trends-2013-09-04I love github. I introduce some github repositories from Today’s trends or my recent starred. I will continuously post Github Trends every day hopefully…
I read only README, so my impression must be useless, but I hope it will be a chance for readers to know awesome github repositories.
Most people upload their dotfiles into github. They must use this tool. You can manage your dotfiles and home directory using separate git repositories. Awesome.
Excellent… The best local git repository management application. I usually use tig, but I will try this super awesome tool.
See ya!
]]><![CDATA[omniauth-openid with pow and nginx SSL termination]]>2013-08-24T16:14:00+09:00http://tech.riywo.com/blog/2013/08/24/omniauth-openid-with-pow-and-nginx-ssl-terminationI tried to use omniauth-openid with my new rails application. It worked when I developed using HTTP. After I started to develop with HTTPS, it failed. Finally I got the coolest workaround: OmniAuth.config.full_host and X-Forwarded-Host/Port.
My Environment
I use Pow for a development server. To use HTTPS with Pow, I installed nginx and configured like below.
class Users::OmniauthCallbacksController < Devise::OmniauthCallbacksController
skip_before_action :verify_authenticity_token
def google_apps
…
They worked if I access to Pow directly using HTTP.
What was the problem 1?
I guess all people using omniauth-oauth2 behind a reverse proxy encounter “Redirect uri mismatch” error. Because redirect_uri query string parameter is like https://myapp.127.0.0.1.xip.io:80/users/auth/github. :80 ?????
And omniauth-google-apps didn’t work neither. My browser tried to redirect to https://myapp.127.0.0.1.xip.io:80/users/auth/google_apps/callback…
def port
if port = host_with_port.split(/:/)[1]
port.to_i
elsif port = @env['HTTP_X_FORWARDED_PORT']
port.to_i
elsif @env.has_key?("HTTP_X_FORWARDED_HOST")
DEFAULT_PORTS[scheme]
else
@env["SERVER_PORT"].to_i
end
end
YES!!!!!!!!!!!!!!!
Solution
I added X-Forwarded-Host on my nginx.conf, then it worked! Because I had already configured X-Forwarded-Proto.
proxy_set_header X-Forwarded-Host $host;
Or X-Forwarded-Port also worked.
proxy_set_header X-Forwarded-Port 443;
Conclusion
If you use SSL termination(nginx, Apache, LB, etc.) with your application:
You should use SSL termination for your development environment.
Otherwise, you will have some trouble between dev and prod…
You should pass much information using X-Forwarded-* headers.
Many libraries handle these headers better than you:)
]]><![CDATA[Using columns of a intermediate table with ActiveRecord has_many association]]>2013-06-22T09:51:00+09:00http://tech.riywo.com/blog/2013/06/22/using-columns-of-a-intermediate-table-with-activerecord-has-many-associationI have n:m relation tables and want to use columns of the intermediate table.
activerecord (4.0.0.rc2)
class User < ActiveRecord::Base
has_many :user_bookmarks
has_many :bookmarks, through: :user_bookmarks
end
class UserBookmark < ActiveRecord::Base
belongs_to :user
belongs_to :bookmark
# has column "star"
end
class Bookmark < ActiveRecord::Base
has_many :user_bookmarks
has_many :users, through: :user_bookmarks
end
> User.first.bookmarks.first.star
NoMethodError: undefined method `star' for #<Bookmark:0x007ff40e45a7b8>
I used a scope block for has_many.
class User < ActiveRecord::Base
has_many :user_bookmarks
has_many :bookmarks, -> { select("bookmarks.*, user_bookmarks.star") }, through: :user_bookmarks
end
> User.first.bookmarks.first.star
10
I’m not sure whether this is the best way for the purpose, but it works.
]]><![CDATA[ruby-build --enable-load-relative]]>2013-06-22T09:27:00+09:00http://tech.riywo.com/blog/2013/06/22/ruby-build-enable-load-relativeCRuby builds a ruby binary and scripts with the absolute path hard corded.
So, if you move the installed directory location, the ruby and scripts don’t work. To avoid this kind of problems, you can use --enable-load-relative configure option.
]]><![CDATA[git rollback]]>2013-06-18T17:20:00+09:00http://tech.riywo.com/blog/2013/06/18/git-rollbackOne day, I wanted to rollback git repository to a previous commit with a new “rollback” commit because the wrong commits had already pushed to the master.
MODIFIED: The best way is git revert -n INITHASH..HEAD!
Thanks, @miyagawa!
$ git init
$ echo important > important
$ touch important_empty
$ git add .
$ git commit -m 'init'
$ ls
important important_empty
$ git rm important important_empty
$ echo wrong > wrong
$ touch wrong_empty
$ git add .
$ git commit -m 'wrong' ## WRONG COMMIT!
$ ls
wrong wrong_empty
$ git revert -n INITHASH..HEAD
# On branch master
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# new file: important
# renamed: wrong_empty -> important_empty
# deleted: wrong
#
$ ls
important important_empty
old posts
Generally, git revert -n HASH works, however, if some files are added or deleted, it doesn’t work well. Here is an example.
git revert is not the way to revert “to HASH”, but to revert “HASH”.
$ git revert -n INITHASH ## want to rollback to INITHASH
$ git status
# On branch master
nothing to commit, working directory clean
$ ls
wrong wrong_empty
So, I tried some ways. First, git checkout INITHASN .. It worked only for deleted files.
$ git checkout INITHASH .
$ git status
# On branch master
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# new file: important
# new file: important_empty
#
$ ls
important important_empty wrong wrong_empty
Second, git checkout -b rollback INITHASH and git merge. Unfortunately, INITHASH had already merged, so it happened nothing.
$ git checkout -b rollback INITHASH
$ ls
important important_empty
$ git checkout master
$ git merge rollback
Already up-to-date.
Third, git diff and patch. It affected only non-empty files because git diff output for empty files was not good for patch command.
$ git diff HEAD..INITHASH | patch -p1
patching file important
patching file wrong
$ git status
# On branch master
# Changes not staged for commit:
# (use "git add/rm <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# deleted: wrong
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# important
no changes added to commit (use "git add" and/or "git commit -a")
$ ls
important wrong_empty
Finally, I found the best way, that is git apply.
$ git diff HEAD..INITHASH > /tmp/patch
$ git apply /tmp/patch
$ git status
# On branch master
# Changes not staged for commit:
# (use "git add/rm <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# deleted: wrong
# deleted: wrong_empty
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# important
# important_empty
no changes added to commit (use "git add" and/or "git commit -a")
$ ls
important important_empty
Hey, git experts, is this the best way to create a new “rollback” commit?
]]><![CDATA[anyenv]]>2013-06-11T00:23:00+09:00http://tech.riywo.com/blog/2013/06/11/anyenvI developed a experimental simple wrapper for rbenv-style version managers.
anyenv recognizes $ANYENV_ROOT/envs/* as rbenv-style version managers, so anyenv init - outputs export **ENV_ROOT=$ANYENV_ROOT/envs/**env, **env init - and adds $ANYENV_ROOT/envs/**env/bin to $PATH.
]]><![CDATA[Sonic Notify - a real broadcasting]]>2013-05-07T21:57:00+09:00http://tech.riywo.com/blog/2013/05/07/sonic-notify-a-real-broadcasting前置きというか能書き
]]><![CDATA[mruby on MIPS in PQI Air Pen]]>2013-05-07T07:40:00+09:00http://tech.riywo.com/blog/2013/05/07/mruby-on-mips-in-pqi-air-penmoyashi さんの記事を読んで以来、ずっと欲しかったPQI Airシリーズなんですが、USのAmazonでは売ってなかったり無駄に高かったりして二の足を踏んでました。ある日、郵便局を使うと数百円で荷物が送れることが分かったので、日本のAmazonで買って実家に発送して送ってもらうことでついにゲットしました。
mrubyのことは実はさっぱり分かってないのですが、色々楽しめそうです。興味ある方はぜひ遊んでみて下さい or 遊び方教えて下さい。
]]><![CDATA[webtail + ncat = simple log monitoring!]]>2013-04-23T18:58:00+09:00http://tech.riywo.com/blog/2013/04/23/webtail-plus-ncat-equals-simple-log-monitoring-slashwebtail is a super simple log monitoring tool. You can monitor a log streaming on a server via your web browser.
I wanted to monitor multi servers log, so I tried ncat in mmap. ncat supports multi sessions.
You can monitor all web servers logs with a single web page.
Enjoy!
]]><![CDATA[python mox example]]>2013-04-19T19:55:00+09:00http://tech.riywo.com/blog/2013/04/19/python-mox-examplePythonの小さいスクリプトを書いたんですが、せっかくなんでテストも書いてみようと思ってあがいたメモ。やりたかったのは、スクリプトの中で行ってるIO(ファイルとIPC)をテスト中は上書きして、テストデータを返す、みたいなこと。
