Link-LXC-USERCR

From Linux Checkpoint / Restart Wiki
(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
 +
 
= Overview =
 
= Overview =
 
The following instructions describe the process to build/insall components necessary to checkpoint/restart (C/R) LXC containers using the C/R implementation
 
The following instructions describe the process to build/insall components necessary to checkpoint/restart (C/R) LXC containers using the C/R implementation
Line 23: Line 24:
 
== Terminology ==
 
== Terminology ==
  
* '''statefile / checkpoint-image''' - this refers to the file in which the application state is saved after a successful checkpoint.
+
'''statefile / checkpoint-image''' - this refers to the file in which the application state is saved after a successful checkpoint.
* '''USERCR''' - The user-space commands/library components that utilize the kernel system calls to checkpoint/restart applications. Specifically this refers to the git tree: ''git://git.ncl.cs.columbia.edu/pub/git/user-cr.git''  
+
 
 +
'''USERCR''' - The user-space commands/library components that utilize the kernel system calls to checkpoint/restart applications. Specifically this refers to the git tree: ''git://git.ncl.cs.columbia.edu/pub/git/user-cr.git''  
  
 
= 1. Build C/R-enabled Linux kernel =
 
= 1. Build C/R-enabled Linux kernel =
  
 
== Get the C/R-enabled kernel sources ==
 
== Get the C/R-enabled kernel sources ==
{|
 
| $ cd /root
 
|-
 
| $ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
 
|-
 
| $ cd linux-cr
 
|-
 
| $ git-checkout -b test-1 3522c57a9ec6f08a129a78322318abcb4467db28
 
|-
 
| $ git-log --pretty=short HEAD^..
 
  
 +
    $ cd /root
 +
    $ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
 +
    $ cd linux-cr
 +
    $ git-checkout -b test-1 0fdca57255b8b5bc9a4f107bee7f1e47d2630cb3
 +
    $ git-log --pretty=short HEAD^..
 
     commit 0fdca57255b8b5bc9a4f107bee7f1e47d2630cb3
 
     commit 0fdca57255b8b5bc9a4f107bee7f1e47d2630cb3
 
     Author: Dan Smith <danms@us.ibm.com>
 
     Author: Dan Smith <danms@us.ibm.com>
 
     Disable softirqs when taking the socket queue lock
 
     Disable softirqs when taking the socket queue lock
 
|}
 
  
 
== Ensure following tokens are set in .config ==
 
== Ensure following tokens are set in .config ==
   {| border="1"
+
   CONFIG_CHECKPOINT_SUPPORT=y
  |-
+
   CONFIG_SYSVIPC_CHECKPOINT=y
  | CONFIG_CHECKPOINT_SUPPORT=y
+
   CONFIG_CHECKPOINT=y
   |-
+
   CONFIG_CHECKPOINT_NETNS=y
  | CONFIG_SYSVIPC_CHECKPOINT=y
+
   CONFIG_CHECKPOINT_DEBUG=y
   |-
+
   CONFIG_CGROUPS=y
  | CONFIG_CHECKPOINT=y
+
   CONFIG_CGROUP_FREEZER=y
   |-
+
   CONFIG_NAMESPACES=y
  | CONFIG_CHECKPOINT_NETNS=y
+
   CONFIG_CGROUP_NS=y
   |-
+
   CONFIG_UTS_NS=y
  | CONFIG_CHECKPOINT_DEBUG=y
+
   CONFIG_IPC_NS=y
   |-
+
   CONFIG_USER_NS=y
  | CONFIG_CGROUPS=y
+
   CONFIG_PID_NS=y
   |-
+
   CONFIG_NET_NS=y
  | CONFIG_CGROUP_FREEZER=y
+
   CONFIG_FREEZER=y
   |-
+
  | CONFIG_NAMESPACES=y
+
   |-
+
  | CONFIG_CGROUP_NS=y
+
   |-
+
  | CONFIG_UTS_NS=y
+
   |-
+
  | CONFIG_IPC_NS=y
+
   |-
+
  | CONFIG_USER_NS=y
+
   |-
+
  | CONFIG_PID_NS=y
+
   |-
+
  | CONFIG_NET_NS=y
+
   |-
+
  | CONFIG_FREEZER=y
+
  |}
+
  
 
== Build, install, reboot on new kernel ==
 
== Build, install, reboot on new kernel ==
 
After every reboot, ensure '-o newinstance' mount option to /dev/pts works (see Documentation/filesystems/devpts.txt for details). In short, run following commands on each reboot:
 
After every reboot, ensure '-o newinstance' mount option to /dev/pts works (see Documentation/filesystems/devpts.txt for details). In short, run following commands on each reboot:
* $ rm /dev/ptmx
+
 
* $ ln -s pts/ptmx /dev/ptmx
+
  $ rm /dev/ptmx
* $ chmod 666 /dev/pts/ptmx
+
  $ ln -s pts/ptmx /dev/ptmx
 +
  $ chmod 666 /dev/pts/ptmx
  
 
= 2. Build USERCR =
 
= 2. Build USERCR =
  
* $ cd /root
+
== Get latest USERCR sources ==
* $ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr
+
    cd /root
* $ cd user-cr
+
    $ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr
* $ git-checkout ckpt-v20-dev
+
    $ cd user-cr
** Tested with commit e275f77e4a82d228c1df14dbeb691342e32cdac2 as HEAD.
+
    $ git-checkout -b test-1 fa382a1758ef395858b1bea530ccdb4e1360b76f
* $ KERNELSRC=/root/linux-cr make
+
    $ git-log --pretty=short HEAD^..
** Build USERCR by pointing to corresponding kernel-source. This should create restart.o and checkpoint.o needed by LXC.
+
    commit fa382a1758ef395858b1bea530ccdb4e1360b76f
** You may need to compile checkpoint.o and restart.o with -fPIC compiler option.
+
    Author: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
* $ make install
+
    Date:  Tue Mar 30 23:14:53 2010 -0700
 +
 
 +
== Build and install USERCR binaries ==
 +
Build USERCR by pointing to corresponding kernel-source tree we used above and install binaries. This should create '''restart.o''' and '''checkpoint.o''' needed by LXC, which are left in the current directory (for now).
 +
 
 +
'''Note:''' You may need to compile checkpoint.o and restart.o with -fPIC compiler option.
 +
 
 +
    $ KERNELSRC=/root/linux-cr make
 +
    $ ls restart.o checkpoint.o
 +
    restart.o  checkpoint.o
 +
    $ make install
  
 
= 3. Build/install LXC =
 
= 3. Build/install LXC =
  
* $ cd /root
+
== Get latest LXC sources ==
* $ git-clone git://lxc.git.sourceforge.net/gitroot/lxc/lxc lxc.git
+
 
* $ cd lxc.git
+
    $ cd /root
* Apply attached patches to LXC
+
    $ git-clone git://lxc.git.sourceforge.net/gitroot/lxc/lxc lxc.git
**(I tested with these patches applied to commit 9ea8066aa67b808f71f46e346bd7a215e2a355f3)
+
    $ cd lxc.git
* $ autogen.sh
+
    $ git-checkout -b test-1 f78a1f32f41f6acbbf0b78e6498736dbd22e2301
* $ ./configure --with-libcr=/root/user-cr
+
    $ git-log --pretty=short HEAD^
** This will fail if /root/user-cr does not container checkpoint.o, restart.o and app-checkpoint.h files  
+
    commit f78a1f32f41f6acbbf0b78e6498736dbd22e2301
* $ make
+
    Author: Daniel Lezcano <daniel.lezcano@free.fr>
* $ make install
+
    Date:  Fri Apr 2 23:37:42 2010 +0200
 +
 
 +
== Apply LXC patches ==
 +
Get the patchset from http://lxc.sf.net/patches/lxc+usercr/. Untar the file
 +
and apply each of the patches in the ''broken-out'' directory to the LXC git tree
 +
 
 +
    $ git-am -3 broken-out/0001-patch-1
 +
    $ git-am -3 broken-out/0001-patch-2
 +
    etc
 +
 
 +
== Configure, build, install LXC binaries ==
 +
 
 +
    $ autogen.sh
 +
    $ ./configure --with-libcr=/root/user-cr
 +
    This will fail if /root/user-cr does not container checkpoint.o, restart.o and app-checkpoint.h files  
 +
    $ make
 +
    $ make install
  
 
= 4. Checkpoint/restart a simple LXC container =
 
= 4. Checkpoint/restart a simple LXC container =
  
* $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /bin/sleep 1000
+
    $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /bin/sleep 1000
* $ lxc-checkpoint --name foo --statefile /root/lxc-foo.ckpt
+
    $ lxc-checkpoint --name foo --statefile /root/lxc-foo.ckpt
* $ lxc-stop --name foo
+
    $ lxc-stop --name foo
* $ lxc-restart --name foo --statefile /root/lxc-foo.ckpt
+
    $ lxc-restart --name foo --statefile /root/lxc-foo.ckpt
* $ lxc-stop --name foo
+
    $ lxc-stop --name foo
  
 
= 5. Checkpoint/restart other LXC containers: =
 
= 5. Checkpoint/restart other LXC containers: =
  
* Similar to step 4 above checkpoint/restart other applications in containers. For example applications clone following git tree:
+
Similar to step 4 above checkpoint/restart other applications in containers. For example applications clone following git tree:
** cr-tests: git://git.sr71.net/~hallyn/cr_tests.git
+
  
* Then try checkpoint/restart of:
+
    cr-tests: git://git.sr71.net/~hallyn/cr_tests.git
** a file-io session (see run-fileio1 in cr-tests[1])
+
 
** process-tree (see run-ptree1 in cr-tests[1])
+
Then try checkpoint/restart of:
 +
 
 +
* a file-io session (see run-fileio1 in cr-tests[1])
 +
* process-tree (see run-ptree1 in cr-tests[1])
  
 
= 6. Checkpoint/restart an LXC container running a VNC server =
 
= 6. Checkpoint/restart an LXC container running a VNC server =
  
* Run a "vi" editing session inside a VNC server using "twm"
+
== VI editing session in a VNC session ==
* $ cat /root/.vnc/xstartup
+
Run a "vi" editing session inside a VNC server using "twm" window manager
{|
+
    $ cat /root/.vnc/xstartup
|-
+
    #!/bin/sh||
| #!/bin/sh||
+
    xsetroot -solid grey||
|-
+
    vncconfig -iconic &||
|xsetroot -solid grey||
+
    xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &|
|-
+
    twm &
|vncconfig -iconic &||
+
 
|-
+
    $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /usr/bin/vncserver :1
|xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &|
+
    $ vncviewer :1
|-
+
    # Open a vi session in vnc viewer
|twm &
+
 
|}
+
    $ lxc-checkpoint --name foo  --statefile /root/vnc.ckpt
 +
    $ lxc-stop --name foo
 +
 
 +
    $ lxc-restart --pause --name foo --statefile /root/vnc.ckpt
 +
    # Leaves the server frozen due to --pause
 +
    $ lxc-unfreeze --name foo
  
* $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /usr/bin/vncserver :1
+
    $ vncviewer :1
* $ vncviewer :1
+
    # Should bring up the old VNC session with vi window
* # Open a vi session in vnc viewer
+
* $ lxc-checkpoint --name foo  --statefile /root/vnc.ckpt
+
* $ lxc-stop --name foo
+
* $ lxc-restart --pause --name foo --statefile /root/vnc.ckpt
+
* # Leaves the server frozen due to --pause
+
* $ lxc-unfreeze --name foo
+
* $ vncviewer :1
+
* # Should bring up the old VNC session with vi window
+

Revision as of 06:06, 8 April 2010

Contents

Overview

The following instructions describe the process to build/insall components necessary to checkpoint/restart (C/R) LXC containers using the C/R implementation being pushed into mainline.

The three main components to be built/installed are:

  • C/R enabled Linux kernel
  • USERCR - the user-space component of checkpoint/restart
  • LXC

Note: This page is still under construction. Feel free to fix obvious mistakes or email sukadev@linux.vnet.ibm.com.

Commit id

The instructions below identify a commit-id or a git tag in each of the git trees and it is recommended that you create a branch on those commit ids and apply any of the listed patches on that branch, before building the component.

Basic debugging of C/R

There is a strong dependency between each of these components (USERCR and Linux kernel are tightly coupled as are USERCR and LXC) so it is important to ensure the versions match up correctly. Otherwise attempts to checkpoint/restart typically fail with terse -EINVAL or -EBUSY errors.

If this happens, usually more debug information is found in dmesg output. If lxc-checkpoint fails some additional error information can be found by running

  • $ /bin/ckptinfo -ev <checkpoint-statefile>

Terminology

statefile / checkpoint-image - this refers to the file in which the application state is saved after a successful checkpoint.

USERCR - The user-space commands/library components that utilize the kernel system calls to checkpoint/restart applications. Specifically this refers to the git tree: git://git.ncl.cs.columbia.edu/pub/git/user-cr.git

1. Build C/R-enabled Linux kernel

Get the C/R-enabled kernel sources

   $ cd /root
   $ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
   $ cd linux-cr
   $ git-checkout -b test-1 0fdca57255b8b5bc9a4f107bee7f1e47d2630cb3
   $ git-log --pretty=short HEAD^..
   commit 0fdca57255b8b5bc9a4f107bee7f1e47d2630cb3
   Author: Dan Smith <danms@us.ibm.com>
   Disable softirqs when taking the socket queue lock

Ensure following tokens are set in .config

  CONFIG_CHECKPOINT_SUPPORT=y
  CONFIG_SYSVIPC_CHECKPOINT=y
  CONFIG_CHECKPOINT=y
  CONFIG_CHECKPOINT_NETNS=y
  CONFIG_CHECKPOINT_DEBUG=y
  CONFIG_CGROUPS=y
  CONFIG_CGROUP_FREEZER=y
  CONFIG_NAMESPACES=y
  CONFIG_CGROUP_NS=y
  CONFIG_UTS_NS=y
  CONFIG_IPC_NS=y
  CONFIG_USER_NS=y
  CONFIG_PID_NS=y
  CONFIG_NET_NS=y
  CONFIG_FREEZER=y

Build, install, reboot on new kernel

After every reboot, ensure '-o newinstance' mount option to /dev/pts works (see Documentation/filesystems/devpts.txt for details). In short, run following commands on each reboot:

  $ rm /dev/ptmx
  $ ln -s pts/ptmx /dev/ptmx
  $ chmod 666 /dev/pts/ptmx

2. Build USERCR

Get latest USERCR sources

   cd /root
   $ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr
   $ cd user-cr
   $ git-checkout -b test-1 fa382a1758ef395858b1bea530ccdb4e1360b76f
   $ git-log --pretty=short HEAD^..
   commit fa382a1758ef395858b1bea530ccdb4e1360b76f
   Author: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
   Date:   Tue Mar 30 23:14:53 2010 -0700

Build and install USERCR binaries

Build USERCR by pointing to corresponding kernel-source tree we used above and install binaries. This should create restart.o and checkpoint.o needed by LXC, which are left in the current directory (for now).

Note: You may need to compile checkpoint.o and restart.o with -fPIC compiler option.

   $ KERNELSRC=/root/linux-cr make 
   $ ls restart.o checkpoint.o
   restart.o   checkpoint.o
   $ make install

3. Build/install LXC

Get latest LXC sources

   $ cd /root
   $ git-clone git://lxc.git.sourceforge.net/gitroot/lxc/lxc lxc.git
   $ cd lxc.git
   $ git-checkout -b test-1 f78a1f32f41f6acbbf0b78e6498736dbd22e2301
   $ git-log --pretty=short HEAD^
   commit f78a1f32f41f6acbbf0b78e6498736dbd22e2301
   Author: Daniel Lezcano <daniel.lezcano@free.fr>
   Date:   Fri Apr 2 23:37:42 2010 +0200

Apply LXC patches

Get the patchset from http://lxc.sf.net/patches/lxc+usercr/. Untar the file and apply each of the patches in the broken-out directory to the LXC git tree

   $ git-am -3 broken-out/0001-patch-1
   $ git-am -3 broken-out/0001-patch-2
   etc

Configure, build, install LXC binaries

   $ autogen.sh
   $ ./configure --with-libcr=/root/user-cr
   This will fail if /root/user-cr does not container checkpoint.o, restart.o and app-checkpoint.h files 
   $ make
   $ make install

4. Checkpoint/restart a simple LXC container

   $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /bin/sleep 1000
   $ lxc-checkpoint --name foo --statefile /root/lxc-foo.ckpt
   $ lxc-stop --name foo
   $ lxc-restart --name foo --statefile /root/lxc-foo.ckpt
   $ lxc-stop --name foo

5. Checkpoint/restart other LXC containers:

Similar to step 4 above checkpoint/restart other applications in containers. For example applications clone following git tree:

   cr-tests: git://git.sr71.net/~hallyn/cr_tests.git

Then try checkpoint/restart of:

  • a file-io session (see run-fileio1 in cr-tests[1])
  • process-tree (see run-ptree1 in cr-tests[1])

6. Checkpoint/restart an LXC container running a VNC server

VI editing session in a VNC session

Run a "vi" editing session inside a VNC server using "twm" window manager

   $ cat /root/.vnc/xstartup
   #!/bin/sh||
   xsetroot -solid grey||
   vncconfig -iconic &||
   xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &|
   twm &
   $ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /usr/bin/vncserver :1
   $ vncviewer :1
   # Open a vi session in vnc viewer
   $ lxc-checkpoint --name foo  --statefile /root/vnc.ckpt
   $ lxc-stop --name foo
   $ lxc-restart --pause --name foo --statefile /root/vnc.ckpt
   # Leaves the server frozen due to --pause
   $ lxc-unfreeze --name foo
   $ vncviewer :1
   # Should bring up the old VNC session with vi window
Personal tools