Flavio Castelli

Debugging my life.

Orchestrating Docker Containers on openSUSE

| Comments

A couple of weeks ago the 11th edition of SUSE’s hackweek took place. This year I decided to spend this time to look into the different orchestration and service discovery tools build around Docker.

In the beginning I looked into the kubernetes project. I found it really promising but AFAIK not yet ready to be used. It’s still in its early days and it’s in constant evolution. I will surely keep looking into it.

I also looked into other projects like consul and geard but then I focused on using etcd and fleet, two of the tools part of CoreOS.

I ended up creating a small testing environment that is capable of running a simple guestbook web application talking with a MongoDB database. Both the web application and the database are shipped as Docker images running on a small cluster.

The whole environment is created by Vagrant. That project proved to be also a nice excuse to play with this tool. I found Vagrant to be really useful.

You can find all the files and instructions required to reproduce my experiments inside of this repository on GitHub.

Happy hacking!

Building Docker Images With KIWI

| Comments

I’m pleased to announce Marcus Schäfer has just made possible to build docker images with KIWI.

For those who never heard about it, KIWI is a tool which creates Linux systems for both physical and virtual machines. It can create openSUSE, SUSE and other types of Linux distributions.

Update: I changed the required version of kiwi and the openSUSE 13.1 template. Kiwi just received some improvements which do no longer force the image to include the lxc package.

Why is this important?

As you might know Docker has already its build system which provides a really easy way to create new images. However these images must be based on existing ones, which leads to the problem of creating the 1st parent image. That’s where KIWI comes to the rescue.

Indeed Kiwi can be used to build the openSUSE/SUSE/whatever docker images that are going to act as the foundation blocks of other ones.

Requirements

Docker support has been added to KIWI 5.06.87. You can find this package inside of the Virtualization:Appliances project on OBS.

Install the kiwi and the kiwi-doc packages on your system. Then go to the /usr/share/doc/packages/kiwi/examples/ directory where you will find a simple openSUSE 13.1 template.

Building the system

Just copy the whole /usr/share/doc/packages/kiwi/examples/suse-13.1/suse-docker-container directory to another location and make your changes.

The heart of the whole image is the config.xml file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="6.1" name="suse-13.1-docker-guest">
  <description type="system">
    <author>Flavio Castelli</author>
    <contact>fcastelli@suse.com</contact>
    <specification>openSUSE 13.1 docker image</specification>
  </description>
  <preferences>
    <type image="docker" container="os131">
      <machine>
        <vmdisk/>
        <vmnic interface="eth0" mode="veth"/>
      </machine>
    </type>
    <version>1.0.0</version>
    <packagemanager>zypper</packagemanager>
    <rpm-check-signatures>false</rpm-check-signatures>
    <rpm-force>true</rpm-force>
    <locale>en_US</locale>
    <keytable>us.map.gz</keytable>
    <hwclock>utc</hwclock>
    <timezone>US/Eastern</timezone>
  </preferences>
  <users group="root">
    <user password="$1$wYJUgpM5$RXMMeASDc035eX.NbYWFl0" home="/root" name="root"/>
  </users>
  <repository type="yast2">
    <source path="opensuse://13.1/repo/oss/"/>
  </repository>
  <packages type="image">
    <package name="coreutils"/>
    <package name="iputils"/>
  </packages>
  <packages type="bootstrap">
    <package name="filesystem"/>
    <package name="glibc-locale"/>
    <package name="module-init-tools"/>
  </packages>
</image>

This is a really minimal image which contains just a bunch of packages.

The first step is the creation of the image’s root system:

1
2
kiwi -p /usr/share/doc/packages/kiwi/examples/suse-13.1/suse-docker-container \
     --root /tmp/myimage

The next step compresses the file system of the image into a single tarball:

1
kiwi --create /tmp/myimage --type docker -d /tmp/myimage-result

The tarball can be found under /tmp/myimage-result. This can be imported into docker using the following command:

1
docker import - myImage < /path/to/myimage.tbz

The image named myImage is now ready to be used.

What’s next

In the next days I’ll make another blog post explaining how to build docker images using KIWI and the Open Build Service. This is a powerful combination which allows to achieve continuous delivery.

Stay tuned and have fun!

Docker and openSUSE Getting Closer

| Comments

I have some good news about Docker and openSUSE.

First of all the Docker package has been moved from my personal OBS project to the more official Virtualization one. The next step is to get the Docker package into Factory :)

I’m going to drop the docker package from home:flavio_castelli:docker, so make sure to subscribe to the Virtualization repository to get latest versions of Docker.

I have also submitted some openSUSE related documentation to the official Docker project. If you visit the “Getting started” page you will notice the familiar geeko logo. Click it to be redirected to the openSUSE’s installation instructions.

Better Docker Experience on openSUSE

| Comments

I don’t know if you are aware of that, but Docker 0.7.0 has been released a couple of days ago.

You can read the full announcement here, but let me talk about the biggest change introduced by this release: storage drivers!

Docker has always used AUFS, a “unionfs-like” file system, to power its containers. Unfortunately AUFS is neither part of the official kernel nor of the openSUSE/SLE one.

In the past I had to build a custom patched kernel to run Docker on openSUSE. That proved to be a real pain both for me and for the end users.

Now with storage drivers Docker can still use AUFS, but can also opt for something different. In our case Docker is going to use thin provisioning, a consolidated technology which is part of the mainstream kernel since quite some time.

Moreover Docker’s community is working on experimental drivers for BTRFS, ZFS, Gluster and Ceph.

What changes now for openSUSE?

Running Docker is incredibly simple now: just use the 1 click install and download it from the ‘home:flavio_castelli:docker’ project.

As I said earlier: no custom kernel is required. You are going to keep the one shipped by the official openSUSE repositories.

Just keep in mind that Docker does some initialization tasks on its very first execution (it configures thin provisioning). So just wait a little before hitting its API with the Docker cli tool (you will just get an error because docker.socket is not found).

The road ahead

Support SLE

Right now Docker works fine on openSUSE 12.3 and 13.1 but not on SLE 11 SP3. During the next days I’m going to look into this issue. I want to have a stable and working package for SLE.

Make it more official

Once the package is proved to be stable enough I’ll submit it for inclusion inside of the Virtualization project on OBS.

So please, checkout Docker package and provide me your feedback!

Docker and openSUSE: A Container Full of Geekos

| Comments

SUSE’s Hackweek #9 is over. It has been an awesome week during which I worked hard to make docker a first class citizen on openSUSE. I also spent some time working on an openSUSE container that could be used by docker’s users.

The project has been tracked on this page of hackweek’s wiki, this is a detailed report of what I achieved.

Installing docker on openSUSE 12.3

Docker has been packaged inside of this OBS project.

So installing it requires just two commands:

1
2
sudo zypper ar http://download.opensuse.org/repositories/home:/flavio_castelli:/docker/openSUSE_12.3 docker
sudo zypper in docker

There’s also a 1 Click Install for the lazy ones :)

Zypper will install docker and its dependencies which are:

  • lxc: docker’s “magic” is built on top of LXC.
  • bridge-utils: is used to setup the bridge interface used by docker’s containers.
  • dnsmasq: is used to start the dhcp server used by the containers.
  • iptables: is used to get containers’ networking work.
  • bsdtar: is used by docker to compress/extract the containers.
  • aufs3 kernel module: is used by docker to track the changes made to the containers.

The aufs3 kernel module is not part of the official kernel and is not available on the official repositories. Hence adding docker will trigger the installation of a new kernel package on your machine.

Both the kernel and the aufs3 packages are going to be installed from the home:flavio_castelli:docker repository but they are in fact links to the packages created by Michal Hrusecky inside of his aufs project on OBS.

Note well: docker works only on 64bit hosts. That’s why there are no 32bit packages.

Docker appliance

If you don’t want to install docker on your system or you are just curious and want to jump straight into action there’s a SUSE Studio appliance ready for you. You can find it here.

If you are not familiar with SUSE Gallery let me tell you two things about it:

  • you can download the appliance on your computer and play with it or…
  • you can clone the appliance on SUSE Studio and customize it even further or…
  • you can test the appliance from your browser using SUSE Studio’s Testdrive feature (no SUSE Studio account required!).

The latter option is really cool, because it will allow you to play with docker immediately. There’s just one thing to keep in mind about Testdrive: outgoing connections are disabled, so you won’t be able to install new stuff (or download new docker images). Fortunately this appliance comes with the busybox container bundled, so you will be able to play a bit with docker.

Running docker

The docker daemon must be running in order to use your containers. The openSUSE package comes with a init script which can be used to manage it.

The script is /etc/init.d/docker, but there’s also the usual symbolic link called /usr/sbin/rcdocker.

To start the docker daemon just type:

1
sudo /usr/sbin/rcdocker start

This will trigger the following actions:

  1. The docker0 bridge interface is created. This interface is bridged with eth0.
  2. A dnsmasq instance listening on the docker0 interface is started.
  3. IP forwarding and IP masquerading are enabled.
  4. Docker daemon is started.

All the containers will get an IP on the 10.0.3.0/24 network.

Playing with docker

Now is time to play with docker.

First of all you need to download an image: docker pull base

This will fetch the official Ubuntu-based image created by the dotCloud guys.

You will be able to run the Ubuntu container on your openSUSE host without any problem, that’s LXC’s “magic” ;)

If you want to use only “green” products just pull the openSUSE 12.3 container I created for you:

1
docker pull flavio/opensuse-12-3

Please experiment a lot with this image and give me your feedback. The dotCloud guys proposed me to promote it to top-level base image, but I want to be sure everything works fine before doing that.

Now you can go through the examples reported on the official docker’s documentation.

Create your own openSUSE images with SUSE Studio

I think it would be extremely cool to create docker’s images using SUSE Studio. As you might know I’m part of the SUSE Studio team, so I looked a bit into how to add support to this new format.

– personal opinion –

There are some technical challenges to solve, but I don’t think it would be hard to address them.

– personal opinion –

If you are interested in adding the docker format to SUSE Studio please create a new feature request on openFATE and vote it!

In the meantime there’s another way to create your custom docker images, just keep reading.

Create your own openSUSE images with KIWI

KIWI is the amazing tool at the heart of SUSE Studio and can be used to create LXC containers.

As said earlier docker runs LXC containers, so we are going to follow these instructions.

First of all install KIWI from the Virtualization:Appliances project on OBS:

1
2
sudo zypper ar http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUSE_12.3 virtualization:appliances
sudo zypper in kiwi kiwi-doc

We are going to use the configuration files of a simple LXC container shipped the kiwi-doc package:

1
cp -r /usr/share/doc/packages/kiwi/examples/suse-11.3/suse-lxc-guest ~/openSUSE_12_3_docker

The openSUSE_12_3_docker directory contains two configuration files used by KIWI (config.sh and config.xml) plus the root directory.

The contents of this directory are going to be added to the resulting container. It’s really important to create the /etc/resolv.conf file inside of the final image since docker is going to mount the resol.conf file of the host system inside of the running guest. If the file is not found docker won’t be able to start our container.

An empty file is enough:

1
touch ~/openSUSE_12_3_docker/root/etc/resolv.conf

Now we can create the rootfs of the container using KIWI:

1
sudo /usr/sbin/kiwi --prepare ~/openSUSE_12_3_docker --root /tmp/openSUSE_12_3_docker_rootfs

We can skip the next step reported on KIWI’s documentation, that’s not needed with docker because it will produce an invalid container. Just execute the following command:

1
sudo tar cvjpf openSUSE_12_3_docker.tar.bz2 -C /tmp/openSUSE_12_3_docker_rootfs/ .

This will produce a tarball containing the rootfs of your container.

Now you can import it inside of docker, there are two ways to achieve that:

  • from a web server.
  • from a local file.

Importing the image from a web server is really convenient if you ran KIWI on a different machine.

Just move the tarball to a directory which is exposed by the web server. If you don’t have one installed just move to the directory containing the tarball and type the following command:

1
python -m SimpleHTTPServer 8080

This will start a simple http server listening on port 8080 of your machine.

On the machine running docker just type:

1
docker import http://mywebserver/openSUSE_12_3_docker.tar.bz2 my_openSUSE_image latest

If the tarball is already on the machine running docker you just need to type:

1
cat ~/openSUSE_12_3_docker.tar.bz2 | docker import - my_openSUSE_image latest

Docker will download (just in the 1st case) and import the tarball. The resulting image will be named ‘my_openSUSE_image’ and it will have a tag named ‘latest’.

The name of the tag is really important since docker tries to run the image with the ‘latest’ tag unless you explicitly specify a different value.

Conclusion

Hackweek #9 has been both productive and fun for me. I hope you will have fun too using docker on openSUSE.

As usual, feedback is welcome.

Qjson 0.8.1 Released

| Comments

Just a quick information, QJson 0.8.1 has been released. This release ensure API and ABI compatibility with version 0.7.1.

The previous 0.8.0 release broke ABI compatibility without changing the SOVERSION.

Toward QJson 1.0.0

I’m not entirely happy with some parts of QJson’s API. I addressed these issues inside of the 1_0_0 branch.

I would appreciate to hear your opinion before merging this branch into master and releasing QJson 1.0.0.

QJson 0.8.0 Released

| Comments

Almost three years passed since latest release of QJson. A lot of stuff happened in my life and QJson definitely paid for that. I have to admit I’m a bit ashamed.

So here we go, QJson 0.8.0 is out!

What changed

A lot of bugs has been smashed during this time, this new release will fix issues like this one and this in a nicer way.

QJson’s API is still backward compatible, while the ABI changed.

Symbian support

Some work has also been done to get QJson work on the Symbian platform. The development happened a long time before Symbian was declared dead.

Currently I do not offer any kind of support for the Symbian platform because IMHO Symbian development is a mess and given the current situation in the mobile world I don’t see any point in investing more efforts on that.

Obviously Symbian patches and documentation are still accepted, as long as they don’t cause issues to the other target platforms.

QMake support

QJson always used cmake as build system but since some Windows developers had problems with it I decided to add some .pro files. That proved to be a bad choice for me since I had to support two build systems. I prefer to invest my time fixing bugs in the code and adding interesting features rather then triaging qmake issues on Windows. Hence I decided to remove them from git.

If you are a nostalgic you can still grab these files from git. They have been removed with commit 66d10c44dd3b21.

Relocating to Github

I decided to move QJson’s code from Gitorious to Github. Github’s issue sysyem is going to replace Sourceforge’s bug tracking system.

I currently use Github a lot, both for personal projects and for work, and I simply love it. I think it offers the best tools in the market and that’s really important to me.

QJson’s website and mailing lists are still going to be hosted on Sourceforge.



I think that’s all from now. If you want more details about the changes introduced take a look at the changelog or checkout QJson’s website.

Introducing Dister, a Heroku Like Solution for SUSE Studio

| Comments

SUSE Studio is  an awesome tool, with a couple of clicks you can create an openSUSE/SUSE based system and deploy to your hard drive, an usb flash,  a live dvd, a VMware/VirtualBox/Xen server and even Amazon EC2 cloud.

Suppose you want to create a tailored SUSE Studio appliance to run a Ruby on Rails app, this is a list of things you have to take care of:

  • install all the gems required by the app (this can be a long list).
  • install and configure the database used by the app.
  • install and configure a webserver.
  • ensure all the required services are started at boot time. You can save some time by cloning this appliance shared on SUSE Gallery, but this is still going to be boring.

Dister to the rescue!

Dister is a command line tool similar to the one used by Heroku (one of the coolest ways to run your Rails app into the cloud). Within a few steps you will be able to create a SUSE Studio appliance running your rails application, download it and deploy wherever you want.

Dister is named after SUSE Studio robot. It has been created by  Dominik Mayer and me during the latest hackweek.

How it works

We are going to create a SUSE Studio appliance running a rails application called “SUSE history”. The app uses bundler to handle its dependencies. This is its Gemfile file:

source 'http://rubygems.org'
gem 'rails', '3.0.5'
gem 'pg'
gem "flutie", "~> 1.1"

As you can see the app uses rails3, the flutie gem and PostgreSQL as database.

Appliance creation

Move into the suse_history directory and execute the following command:

dister create suse_history

As you can see dister has already done a couple of things for you:

  • created an appliance using latest version of openSUSE supported by SUSE Studio (you can use a different base system of course)
  • added the devel:language:ruby:extensions repository to the appliance: this repo contains tons of ruby packages (like mod_passenger)
  • installed a couple of things:
    • devel_C_C++ pattern: this will allow you to build native gems.
    • devel_ruby pattern: provides ruby, rubygems and a couple of development packages needed to build native gems.
    • rubygem-bundler: bundler is required by dister in order to handle the dependencies of your rails application.
    • rubygem-passenger-apache2: dister uses passenger and apache2 to deploy your rails application.
    • postgresql-server: dister noticed suse_history uses PostgreSQL as database, hence it automatically installs it.
    • rubygem-pg: dister noticed suse_history uses PostgreSQL as database, hence it automatically installs the ruby’s library forPostgreSQL.
  • uploaded a custom build script which ensures:
    • mod_passenger module is loaded by Apache
    • both Apache and PostgreSQL are always started at boot time.
    • all dependencies are installed: this is done only during the first boot using bundler.
    • the database user required by your rails app is created. This is done only during the first boot using some SQL code.
    • the database used by the appliance is properly initialized (aka rails db:create db:migrate). This is done only during the first boot.

Upload your code

It’s time to upload suse_history code. This is done using the following command:

dister push

As you can see dister packaged the application source code and all its dependencies into a single archive. Then uploaded the archive to SUSE Studio as an overlay file. Dister uploaded also the configuration files required by Apache and by PostgreSQL setup.

Build your appliance

It’s build time!

dister build

The appliance has automatically being built using the raw disk. You can use different formats of course.

Testdrive

Testdrive is one of the coolest features of SUSE Studio. Unfortunately dister doesn’t support it yet. Just visit your appliance page and start testdrive from your browser. Just enable testdrive networking and connect to your appliance:

Download

Your appliance is working flawlessly. Download it and deploy it wherever you want.

dister download

Current status

As you can see dister handles pretty fine a simple Rails application, but there’s still room for improvements.

Here’s a small list of the things on my TODO list:

  • The dependency management should install gems using rpm packages. This would make the installation of native gems easier, right now the user has to manually add all the development libraries required by the gem. Moreover it would reduce the size of the overlay files uploaded to SUSE Studio.
  • SUSE Studio Testdrive should be supported.
  • It should be possible to deploy the SUSE Studio directly to EC2.
  • Fix bugs!

Bugs and enhancements are going to be tracked here.

Contribute

Dister code can be found here on github, fork it and start contributing.

If you are a student you can work on dister during the next Google Summer of code, apply now!

Jump: A Bookmarking System for the Bash Shell

| Comments

Let me introduce a small project I’ve been working on with a friend of mine, Giuseppe Capizzi. The project is called jump and allows you to quickly change directories in the bash shell using bookmarks.

Thanks to Jump, you won’t have to type those long paths anymore.

You can find jump’s source code, detailed documentation and installation instructions here.

SUSE packages can be found here.

Fast User Switch Plasmoid Improvements

| Comments

Just a quick note, I released a new version of the fastuserswitch plasmoid. This new release implements all the improvements suggested by the users plus some minor fixes.

Code can be downloaded from here. openSUSE packages are already available on the build service.

These are some screenshots illustrating fastuserswitch’s new features.