Introduce QEMU support #2952

jjlakis · 2016-07-20T11:56:38Z

Extending current makefile mechanism to build more than one image for single flavor (with differences in rootfs) and adding QEMU as a hypervisor.
PR contains:

Refactor of stage1/stage1.mk, more specific: STAGE1_ACI_RULE to accept two parameters (flavor and list of images), instead of just one (flavor). List of images are parsed directly in stage1.mk, common and hv-specific files are done still in usr_from_flavor makefiles. STAGE1_ACI_RULE is now responsible to copy files from common directory and build images (added new rule).
stage1/usr_from_kvm/qemu.mk - responsible for cloning sources, building qemu and including its binaries in stage1-kvm.aci.
stage1/init/kvm/hypervisor/qemu_driver.go - similar to lkvm_driver.go in the form created in KVM: Hypervisor support for KVM flavor #2684
Refactor of kvm init part. All functions for starting hypervisors are included, init is checking what binary is contained in aci image, and calls proper command line. Due to this, none_driver.go is removed.

lkvm is still default hypervisor

To build rkt with qemu support, type:
./configure --with-stage1-kvm-hypervisors=qemu

To built rkt with both qemu and lkvm support, type:
./configure --with-stage1-kvm-hypervisors=lkvm,qemu

And you will get two images:
stage1-kvm-lkvm.aci
stage1-kvm-qemu.aci

(if only one hypervisor is specified, its name will remain as stage1-kvm.aci)

I followed up most of @krnowak comments for qemu.mk (Had a problem with simplification of copying files: Introduce QEMU support #2952 (comment))
Nothing is changed for other flavors, and nothing is changed for default configure-make path.
Only Rkt Stop test sometimes fails for QEMU on semaphore (tracking in TestRktStop on stage1-kvm often fails on SemaphoreCI #3091)
Fixed Ctrl+C problem

To be clarified / To do

Documentation needs to be changed. If no new flavor is needed, I would extend existing paper
Should we check dependencies for qemu or just notice user to take care of them?

cc @coreos/rkt-kvm-maintainers

iaguis · 2016-07-20T13:39:26Z

I can't compile it on my system.

First, my default python is 3.x so the build didn't work. I added a workaround:

diff --git a/stage1/usr_from_kvm/qemu.mk b/stage1/usr_from_kvm/qemu.mk
index 0b4a3cd..ac7160b 100644
--- a/stage1/usr_from_kvm/qemu.mk
+++ b/stage1/usr_from_kvm/qemu.mk
@@ -53,7 +53,7 @@ $(call generate-stamp-rule,$(QEMU_BUILD_STAMP),$(QEMU_CONF_STAMP),, \

 $(call generate-stamp-rule,$(QEMU_CONF_STAMP),$(QEMU_CLONE_STAMP),, \
        $(call vb,vt,CONFIG EXT,qemu) \
-       cd $(QEMU_SRCDIR); ./configure $(QEMU_CONFIGURATOR) $(QEMU_CONFIGURATION_OPTS) $(call vl2,>/dev/null))
+       cd $(QEMU_SRCDIR); ./configure --python=/usr/bin/python2 $(QEMU_CONFIGURATOR) $(QEMU_CONFIGURATION_OPTS) $(call vl2,>/dev/null))

 # Generate filelist of qemu directory (this is both srcdir and
 # builddir). Can happen after build finished.

And then the next failure is:

ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
       You probably need to set PKG_CONFIG_LIBDIR
       to point to the right pkg-config files for your
       build target

I tried setting up PKG_CONFIG_LIBDIR, GLIB_SIZEOF_SIZE_T, the patch mentioned in https://bugs.launchpad.net/qemu/+bug/1579565, adding --extra-cflags="-m64" but I couldn't make it work.

jjlakis · 2016-07-21T11:19:53Z

@iaguis What version of gcc you are using?

iaguis · 2016-07-21T11:20:38Z

$ gcc --version
gcc (GCC) 6.1.1 20160707
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

jellonek · 2016-07-21T11:25:16Z

Could You give there produced error string? AFAIR gcc4.9 "does the job".

iaguis · 2016-07-21T11:26:45Z

The error string is mentioned in #2952 (comment):

ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
       You probably need to set PKG_CONFIG_LIBDIR
       to point to the right pkg-config files for your
       build target

jellonek · 2016-07-21T11:35:02Z

ack.

jjlakis · 2016-07-21T13:59:28Z

Can't reproduce it locally with different versions of gcc. In the link you attached, on the bottom, the solution was to reinstall glib, have you tried it?
I also changed qemu version to 2.6.0-rc5.

alban · 2016-07-21T14:26:41Z

I get the following error when trying to compile it:

ERROR: zlib check failed
       Make sure to have the zlib libs and headers installed.

The error probably comes from ./build-rkt-1.11.0+git/tmp/usr_from_kvm/qemu/src/configure:

#########################################
# zlib check

if test "$zlib" != "no" ; then
    cat > $TMPC << EOF
#include <zlib.h>
int main(void) { zlibVersion(); return 0; }
EOF
    if compile_prog "" "-lz" ; then
        :
    else
        error_exit "zlib check failed" \
            "Make sure to have the zlib libs and headers installed."
    fi
fi
LIBS="$LIBS -lz"

But zlib-devel-1.2.8-10.fc24.x86_64 is installed and when I try manually, it works:

$ cat test-zlib.c 
#include <zlib.h>
int main(void) { zlibVersion(); return 0; }

$ gcc -o test-zlib -lz test-zlib.c 
$ ldd test-zlib
    linux-vdso.so.1 (0x00007ffe4edd2000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fdd92665000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fdd922a3000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555e63a21000)
$ gcc --version
gcc (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3)

Any ideas?

jjlakis · 2016-07-21T14:46:15Z

No idea about inconsistency in check of zlib inside and outside of configure. My zlib packages:

$ apt list --installed | grep zlib
zlib1g/xenial,now 1:1.2.8.dfsg-2ubuntu4 amd64 [installed]
zlib1g-dev/xenial,now 1:1.2.8.dfsg-2ubuntu4 amd64 [installed,automatic]

alban · 2016-07-21T15:04:41Z

@krnowak solved my issue by installing the package "zlib-static" (zlib and zlib-devel are not enough).

alban · 2016-07-21T15:19:10Z

dependencies.md needs to be updated with the zlib-static.

Then I have the same issue as #2952 (comment) with "sizeof(size_t)".

jjlakis · 2016-07-21T15:20:23Z

@alban Should I add checking zlib-static in rkt ./configure?

krnowak · 2016-07-21T19:17:15Z

I remember removing all the checks for 3rd party projects like kvm, kernel, etc from configure and writing a note to make sure that all the dependencies are met. Not dure anout mentioning such deps in documentation either.

iaguis · 2016-07-22T15:42:53Z

I could compile it by removing the --static option in QEMU_CONFIGURATION_OPTS.

iaguis · 2016-07-25T12:39:57Z

If I start an interactive container, pressing Ctrl+C will terminate qemu:

/ # #gonna press Ctrl+C
/ # qemu: terminating on signal 2
$

iaguis · 2016-07-25T12:41:10Z

Also, systemd output is printed even though we didn't specify --debug:

$ sudo rkt run kinvolk.io/aci/busybox --interactive
image: using image from file /home/iaguis/work/coreos/go/src/github.com/coreos/rkt/build-rkt/target/bin/stage1-kvm.aci
image: using image from local store for image name kinvolk.io/aci/busybox
networking: loading networks from /etc/rkt/net.d

Welcome to Linux!

[  OK  ] Created slice system.slice.
[  OK  ] Created slice system-prepare\x2dapp.slice.
[  OK  ] Listening on Journal Socket.
[  OK  ] Started Pod shutdown.
         Starting Network configuration for device: eth0...
         Starting Create /etc/passwd and /etc/group...
[  OK  ] Listening on OpenSSH Server Socket for entering purpose.
[  OK  ] Started busybox Reaper.
[  OK  ] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[  OK  ] Started Create /etc/passwd and /etc/group.
[  OK  ] Started Network configuration for device: eth0.
[  OK  ] Started Journal Service.
         Starting Prepare minimum environment for chrooted applications...
[  OK  ] Started Prepare minimum environment for chrooted applications.
[  OK  ] Started Application=busybox Image=kinvolk.io/aci/busybox.
[  OK  ] Reached target rkt apps target.
/ #

jjlakis · 2016-07-26T13:42:01Z

Added some fixes, TestDiagnostic is passing and there's no reduntant output while starting pod (#2952 (comment))

krnowak · 2016-07-26T15:10:36Z

stage1/usr_from_kvm/usr_from_kvm.mk

 $(call setup-stamp-file,UFK_CBU_STAMP,cbu)
 $(call setup-tmp-dir,UFK_TMPDIR)

+HV_BUILD_FILE :=


This may break make clean in a certain scenario (configure with kvm-qemu flavor, make, configure with kvm-lkvm flavor, make, make clean).

I would prefer if you included both lkvm.mk and qemu.mk unconditionally, but instead modify the S1_RF_SECONDARY_STAMPS conditionally, like as follows:

UFK_INCLUDES := \ kernel.mk \ files.mk \ lkvm.mk \ qemu.mk ... $(call inc-many,$(UFK_INCLUDES)) ifeq ($(KVM_HV_TAG),"hv_lkvm") S1_RF_SECONDARY_STAMPS += $(LKVM_STAMP) else ifeq ($(KVM_HV_TAG),"hv_qemu") S1_RF_SECONDARY_STAMPS += $(QEMU_STAMP) endif

For this to work, you should also exclude both QEMU_STAMP and LKVM_STAMP from the namespace undefine mechanism. You can do it like so:

in lkvm.mk at the bottom of the file:

$(call undefine-namespaces,LKVM,LKVM_STAMP)

in qemu.mk at the bottom of the file:

$(call undefine-namespaces,QEMU,QEMU_STAMP)

And remove the lines in lkvm.mk and qemu.mk modifying the S1_RF_SECONDARY_STAMPS variable.

squeed · 2016-07-26T17:38:54Z

I agree with @alban, I think that the kvm / qemu should be different flavors. For the formal release process, we're going to want to build and distribute both.

jjlakis · 2016-07-27T06:52:45Z

@squeed Same flavor but separate images also allow to build and distribute both.

jonboulle · 2016-09-12T10:52:48Z

Are we good to go here?

jellonek · 2016-09-12T11:00:54Z

👍

jjlakis · 2016-09-12T11:15:46Z

Doc files updated 8a17e58

s-urbaniak · 2016-09-12T11:35:56Z

I still get a compilation error as mentioned in #2952 (comment).

s-urbaniak · 2016-09-12T11:36:37Z

(And by removing --static fixes it), but again, this will fail in i.e. Arch upstream.

s-urbaniak · 2016-09-12T11:59:46Z

I also suggest to set --disable-werror in QEMU_CONFIGURATION_OPTS, since new compilers have more restrictions, like here with deprecation warnings:

$ make
  CONFIG EXT   qemu
  BUILD EXT    qemu
hw/9pfs/9p-local.c: In function ‘local_readdir_r’:
hw/9pfs/9p-local.c:398:5: warning: ‘readdir_r’ is deprecated [-Wdeprecated-declarations]
...
$ gcc --version
gcc (GCC) 6.2.1 20160830
...

s-urbaniak · 2016-09-12T12:10:21Z

I have started a qemu based container, locally, great work @jjlakis ! 👍 👍

So this LGTM modulo the --static issue ;-)

jellonek · 2016-09-12T12:21:25Z

Binaries built without --static can fail on other os distro/version (especially built on other than glibc libcs - alpine for example) so removal of this option would be IMO wrong.

s-urbaniak · 2016-09-12T12:34:31Z

FYI: apparently Arch won't include the static libraries for glib2: https://bugs.archlinux.org/task/35758

jjlakis · 2016-09-13T09:21:04Z

@s-urbaniak @iaguis If qemu static build for Arch is not possible and looks like it won't be, it has to be documented. Or maybe we can recommend using rocket-in-rocket build? Of course I can add rkt configure option for static/non-static qemu build, but it wouldn't be so pretty IMO.

jellonek · 2016-09-13T09:32:20Z

As @iaguis pointed on irc - there is hope also for Arch - https://aur.archlinux.org/packages/glib2-static/
Still this is issue IMO out of scope of ... this issue and i think that option to build non-static qemu is unnecessary.

IMO this PR is complete and ready to be merged.

s-urbaniak · 2016-09-13T10:17:44Z

ok, this LGTM then, do you want to create follow-up issues for #2952 (comment), and #2952 (comment), or is this also resolved here already?

lucab · 2016-09-13T10:31:31Z

So, we can keep the static version for the moment and merge it as-is. This looks still rough around the edges, so a bit more exposure will help finding and fixing more issues. Arch downstream (and anybody else missing some .a) will probably not opt-in for the moment and just use our .aci if they want to play with it. If we have other issues related to linking, we may explore the dynamic linking way as a last resort. As a non-default kvm flavor, LGTM too.

Side question: how does https://github.com/01org/qemu-lite relate to this? I see it is not used here, but it is part of the ClearContainers brand.

jellonek · 2016-09-13T10:57:01Z

@lucab when i was still at Intel there was a plan to switch from plain qemu to this qemu-lite, but at the moment this project was unreleased, so this was a goal for future, so the question if if Intel guys have this still in backlog.

jjlakis · 2016-09-13T11:13:51Z

Thanks! That was PR of my life 💃
@lucab Definetely we will work on optimized QEMU
@s-urbaniak Issues with Ctrl+C and redundant output are resolved :)

This change adds the ability to configure rkt to use qemu-lite (implementing the pc-lite machine) over qemu. This change comprises of two parts: * The build system to retrieve qemu-lite source, configure and build (derived from the exiting qemu.mk) * A fork of the hvqemu package to support the slightly different command line options required for qemu. Signed-off-by: Rob Bradford <[email protected]>

alban added this to the v1.12.0 milestone Jul 20, 2016

alban added the technology/kvm label Jul 20, 2016

jjlakis force-pushed the jlakis-hypervisor2 branch from 89f9c3f to bf2d258 Compare July 21, 2016 13:52

iaguis added the needs/review label Jul 21, 2016

jjlakis force-pushed the jlakis-hypervisor2 branch from bf2d258 to 4008ce8 Compare July 26, 2016 13:24

krnowak reviewed Jul 26, 2016
View reviewed changes

lucab added needs/some-changes and removed needs/review labels Aug 3, 2016

jjlakis force-pushed the jlakis-hypervisor2 branch from 8a17e58 to 035e392 Compare September 13, 2016 08:59

Jacek J. Łakis added 9 commits September 13, 2016 10:59

configure: rename functions for reuse

1368a92

configure: add multi value configure option for hypervisors

7a27e4d

makefile: add multi-image support

6b48e93

kvm: add qemu makefile

3a07f88

kvm: add qemu driver

80030ee

kvm/configure: remove none_driver and kv_hv tag

9a8de75

kvm: add hypervisor check in init

9bd8283

tests: bump version of gcc to build qemu

3af5dac

Documentation: add qemu to KVM stage1 doc

035e392

s-urbaniak added reviewed/lgtm and removed needs/review labels Sep 13, 2016

s-urbaniak merged commit fc228dc into rkt:master Sep 13, 2016

pskrzyns deleted the jlakis-hypervisor2 branch September 13, 2016 15:16

Introduce QEMU support #2952

Introduce QEMU support #2952

Uh oh!

Conversation

jjlakis commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To be clarified / To do

Uh oh!

iaguis commented Jul 20, 2016

Uh oh!

jjlakis commented Jul 21, 2016

Uh oh!

iaguis commented Jul 21, 2016

Uh oh!

jellonek commented Jul 21, 2016

Uh oh!

iaguis commented Jul 21, 2016

Uh oh!

jellonek commented Jul 21, 2016

Uh oh!

jjlakis commented Jul 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alban commented Jul 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjlakis commented Jul 21, 2016

Uh oh!

alban commented Jul 21, 2016

Uh oh!

alban commented Jul 21, 2016

Uh oh!

jjlakis commented Jul 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krnowak commented Jul 21, 2016

Uh oh!

iaguis commented Jul 22, 2016

Uh oh!

iaguis commented Jul 25, 2016

Uh oh!

iaguis commented Jul 25, 2016

Uh oh!

jjlakis commented Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krnowak Jul 26, 2016

Choose a reason for hiding this comment

Uh oh!

squeed commented Jul 26, 2016

Uh oh!

jjlakis commented Jul 27, 2016

Uh oh!

jonboulle commented Sep 12, 2016

Uh oh!

jellonek commented Sep 12, 2016

Uh oh!

jjlakis commented Sep 12, 2016

Uh oh!

s-urbaniak commented Sep 12, 2016

Uh oh!

s-urbaniak commented Sep 12, 2016

Uh oh!

s-urbaniak commented Sep 12, 2016

Uh oh!

s-urbaniak commented Sep 12, 2016

Uh oh!

jellonek commented Sep 12, 2016

Uh oh!

s-urbaniak commented Sep 12, 2016

Uh oh!

jjlakis commented Sep 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jellonek commented Sep 13, 2016

Uh oh!

s-urbaniak commented Sep 13, 2016

Uh oh!

lucab commented Sep 13, 2016

Uh oh!

jjlakis commented Jul 20, 2016 •

edited

Loading

jjlakis commented Jul 21, 2016 •

edited

Loading

alban commented Jul 21, 2016 •

edited

Loading

jjlakis commented Jul 21, 2016 •

edited

Loading

jjlakis commented Jul 26, 2016 •

edited

Loading

jjlakis commented Sep 13, 2016 •

edited

Loading

jjlakis commented Sep 13, 2016 •

edited

Loading