====== Cucumber Linux from Scratch ======


====== Chapter 1 - Introduction ======

This page details the procedure for building Cucumber Linux from scratch. This process was originally derived from Linux from Scratch and as such shares many similarities with it. The process is broken down into several "chapters" which mirror the chapters in the Linux from Scratch book.

The build process is also broken down into several phases. Before we begin, here is a brief overview of the phases and chapters, how they line up and what their purposes are:

^ Phase   ^ Chapter(s)     ^ Purpose      ^
| "0"     | 1, 2, 3 & 4    | Set up the environment used to build Cucumber Linux. Note that nothing is actually built during this stage. As a result, this is not //technically// considered a phase, hence the quotes around 0. |
| 1       | 5              | Construct a temporary system in /tools that will be used to build the final system. |
| 2       | 6              | Build an initial bootstrap system. At the end of phase 2, the system will be capable of running itself (without the /tools directory), but will not be capable of compiling itself. |
| 3       | 7              | Build the necessary packages for the /cucumber directory of the ports tree to become entirely self hosted. At the end of phase 3, the system will be capable of both running *and* compiling itself; however, it will not be complete yet. |
| 4       | 8              | Build the remaining packages in the /cucumber directory of the ports tree that were not built in phases 2 or 3. At the end of phase 4, the system will be complete. |

As a final note, some users may wish to use a different distro (besides Cucumber Linux) to build Cucumber Linux. If you do choose to do this, please know that this is not officially supported. Generally, the only distribution that is officially supported for building a given major version of Cucumber Linux (i.e. 2.x) is the previous stable major version (i.e. 1.x) and the version itself (i.e. 2.x).
====== Chapter 2 - Preparing the Build Location ======

===== Introduction =====

In this chapter, the build environment is prepared. We will create two subdirectories: one for building the temporary system (Chapter 5) and another for building the final system (Chapters 6 & 7).

===== Creating the Directories =====

The Cucumber Linux from scratch build will reside in /opt/culfs. Start by creating that directory and changing the working directory to it:

  mkdir -pv /opt/culfs
  cd /opt/culfs

The two subdirectories (lfscript and chroot) will be created when in subsequent chapters as they become necessary.

====== Chapter 3 - Downloading the Package Sources ======

===== Introduction =====

There are two source trees that must be downloaded: the LFScript source tree (which will be used to build chapter 5) and the Cucumber Linux source tree (which will be used to build chapters 6 & 7).

===== Downloading LFScript =====

Clone the Git repository for the Cucumber Linux version of LFScript:

  git clone https://github.com/cucumberlinux/lfscript.git

If you are building the current development version of Cucumber Linux, then no further action is necessary for this step and you may move on to the Downloading the Cucumber Linux Source Tree section; however, if you are building a different version of Cucumber Linux, it will be necessary to switch to the Git branch for that version. For example, if you are building any version of Cucumber Linux 2.x, run:

  cd lfscript
  git checkout 2.x
  cd ..

===== Downloading the Cucumber Linux Source Tree =====

Add the privilege separation user ''portuser''. This user will be the only user with write access the ports tree, as well as the user account used to download all of the sources. It is important that this is an unprivileged account, as it will be connecting to several untrusted servers.

  useradd -m portuser

Clone the Git repository for the official Cucumber Linux source tree using portstrap:

  install -m 755 -d chroot/usr
  wget https://raw.githubusercontent.com/cucumberlinux/ports/master/utilities/tools/portstrap
  chmod +x portstrap
  ROOT=/opt/culfs/chroot ./portstrap

When asked to choose the location to clone the ports tree from, press enter to accept the default.

When asked which branch of the ports tree to clone, do one of the following:
  * If you are building the current development version of Cucumber Linux, enter ''master''.
  * If you are building any other version of Cucumber Linux, enter the version number (i.e. ''2.0'').

When asked to choose the user that will be able to write to the ports tree, enter ''portuser''.

Temporarily change the portmake configuration so it expects to find the ports tree in /opt:

  if [ -e /etc/portmake.conf ]; then
    mv /etc/portmake.conf{,.orig}
  fi
  echo PORTS_TREE=/opt/culfs/chroot/usr/ports > /etc/portmake.conf

Download the source tarballs for the phase 2 and 3 packages. This must be done now, as the chroot environment will not have internet access until after phase 3 is built. This step will also verify the integrity of all the downloaded tarballs via checksums and PGP signatures.

  chroot/usr/ports/utilities/tools/portmake download-recursive phase2
  chroot/usr/ports/utilities/tools/portmake download-recursive phase3
  
Restore the original portmake.conf.

  if [ -e /etc/portmake.conf.orig ]; then
    mv /etc/portmake.conf{.orig,}
  else
    rm /etc/portmake.conf
  fi


===== Downloading a Couple of Additional Packages =====

There are two additional packages that must be downloaded now so they are available immediately inside the chroot environment: pkgtools and tar 1.13. Create a directory to store them in:

  cd chroot
  mkdir sources
  cd sources
  
Download the source archives for them:

  wget https://github.com/cucumberlinux/pkgtools/archive/c2.0.0.tar.gz
  wget https://ftp.gnu.org/gnu/tar/tar-1.13.tar.gz
  
Note that you *must* download tar version 1.13 specifically. The version of pkgtools that is downloaded should be the latest patch release available for the version of Cucumber Linux you are building. For example, if you are building Cucumber Linux 3.0, then you should use the latest version of pkgtools 3.0.x.

Finally cd back to /opt/culfs.

  cd ../..
====== Chapter 4 - Final Perparations ======

===== Introduction =====

In this chapter, we will perform a few additional tasks to prepare for building the temporary system.

===== Creating Necessary Directories =====
  
Create a directory to download the phase 1 source tarballs to:	 

  mkdir /opt/culfs/lfscript/sources
====== Chapter 5 - Building Phase 1 ======

===== Introduction =====

This phase is akin to Chapter 5 of Linux from Scratch: [[http://www.linuxfromscratch.org/lfs/view/development/chapter05/introduction.html|Constructing the Temporary System]]. This step will be automated using a modified version of [[https://www.lfscript.org/wiki/index.php/Main_Page|LFScript]].

This chapter has two parts: first, we will build a tarball for the [[http://www.linuxfromscratch.org/lfs/view/development/chapter04/creatingtoolsdir.html|/tools directory]]. Then, we will extract it to /opt/culfs/chroot/tools and prepare the chroot environment.

===== Building the Tools Tarball =====

CD back into the lfscript directory:

  cd lfscript

Run the following command to build the tools tarball:

  ./lfscript -B
  


===== Installing /tools to the Chroot =====

CD to the chroot environment and extract the tools tarball:

  cd ../chroot
  tar -xvJf ../lfscript/packages-*/toolchain.bak.txz
  
===== Preparing the Chroot =====

There are a few additional steps that must be performed before entering the chroot environment.

==== Set the LFS Variable ====

Set the LFS variable to the chroot directory:

  export LFS=/opt/culfs/chroot
  
==== Set up the Kernel File Systems ====

For details about this step see [[http://www.linuxfromscratch.org/lfs/view/8.0/chapter06/kernfs.html]].

Various file systems exported by the kernel are used to communicate to and from the kernel itself. Create the mount points for them:

  mkdir -pv $LFS/{dev,proc,sys,run}

Perform a bind mount for /dev:

  mount --bind /dev $LFS/dev
  
Mount the remaining virtual file systems:

  mount -vt devpts devpts $LFS/dev/pts -o gid=5,mode=620
  mount -vt proc proc $LFS/proc
  mount -vt sysfs sysfs $LFS/sys
  mount -vt tmpfs tmpfs $LFS/run


==== Important Note about the Chroot Environment ====

If at any point you reboot your build system or close out of the shell you are performing the build in, it will be necessary to repeat the steps in the "Preparing the Chroot" section before reentering the chroot environment.


====== Chapter 6 - Building Phase 2 ======

===== Introduction =====

This phase is akin to Chapter 6 of Linux from Scratch: Installing Basic System Software. The majority of this chapter will be automated using the official Cucumber Linux ports tree and the portmake utility; however, a few of the early packages will need to be built by hand.

At the end of this phase, the system will be capable of running itself (without the /tools directory), but will not be capable of compiling itself yet.



===== Enter the Chroot =====

After completing the "Preparing the Chroot" section of Chapter 5, enter the chroot environment:

  chroot "$LFS" /tools/bin/env -i \
    HOME=/root                  \
    TERM="$TERM"                \
    PS1='\u:\w\$ '              \
    PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin/:/tools/sbin \
    /tools/bin/bash --login +h
    
===== Final Preparations =====

==== Create Essential Directories ====

It is time to create some structure in the LFS file system. Create a standard directory tree by issuing the following commands:

  mkdir -pv /{bin,boot,etc/{opt,sysconfig},home,lib/firmware,mnt,opt}
  mkdir -pv /{media/{floppy,cdrom},sbin,srv,var}
  install -dv -m 0750 /root
  install -dv -m 1777 /tmp /var/tmp
  mkdir -pv /usr/{,local/}{bin,include,lib,sbin,src}
  mkdir -pv /usr/{,local/}share/{color,dict,doc,info,locale,man}
  mkdir -v  /usr/{,local/}share/{misc,terminfo,zoneinfo}
  mkdir -v  /usr/libexec
  mkdir -pv /usr/{,local/}share/man/man{1..8}
  
  case $(uname -m) in
   x86_64) mkdir -v /lib64 ;;
  esac
  
  mkdir -v /var/{log,mail,spool}
  ln -sv /run /var/run
  ln -sv /run/lock /var/lock
  mkdir -pv /var/{opt,cache,lib/{color,misc,locate},local}

==== Create Essential Symlinks ====

Some programs use hardwired paths to other programs that do not exist yet. Create some temporary symlinks to satisfy them:

  install -m 755 -d /bin /usr/{bin,lib} /etc /sbin /var/log
  ln -sv /tools/bin/{bash,cat,echo,pwd,stty} /bin
  ln -sv /tools/bin/perl /usr/bin
  ln -sv /tools/lib/libgcc_s.so{,.1} /usr/lib
  ln -sv /tools/lib/libstdc++.so{,.6} /usr/lib
  sed 's/tools/usr/' /tools/lib/libstdc++.la > /usr/lib/libstdc++.la
  ln -sv bash /bin/sh
  ln -sv /proc/self/mounts /etc/mtab
  ln -sv /tools/sbin/installpkg /sbin/
  ln -sv /tools/sbin/makepkg /sbin/
  ln -sv /tools/sbin/upgradepkg /sbin/
  ln -sv /tools/bin/m4 /usr/bin/
  
==== Create Essential Files ====

Create a basic /etc/passwd:

  cat > /etc/passwd << "EOF"
  root::0:0:root:/root:/bin/bash
  bin:x:1:1:bin:/dev/null:/bin/false
  daemon:x:6:6:Daemon User:/dev/null:/bin/false
  messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
  nobody:x:99:99:Unprivileged User:/dev/null:/bin/false
  EOF

Create a basic /etc/group:

  cat > /etc/group << "EOF"
  root:x:0:
  bin:x:1:daemon
  sys:x:2:
  kmem:x:3:
  tape:x:4:
  tty:x:5:
  daemon:x:6:
  floppy:x:7:
  disk:x:8:
  lp:x:9:
  dialout:x:10:
  audio:x:11:
  video:x:12:
  utmp:x:13:
  usb:x:14:
  cdrom:x:15:
  adm:x:16:
  messagebus:x:18:
  systemd-journal:x:23:
  input:x:24:
  mail:x:34:
  nogroup:x:99:
  users:x:999:
  EOF

==== Initialize Log Files ====

The login, agetty, and init programs (and others) use a number of log files to record information such as who was logged into the system and when. However, these programs will not write to the log files if they do not already exist. Initialize the log files and give them proper permissions: 

  touch /var/log/{btmp,lastlog,wtmp}
  chgrp utmp /var/log/lastlog
  chmod 664  /var/log/lastlog
  chmod 600  /var/log/btmp

==== Customize Portmake ====

If desired, you can customize the Portmake configuration. This will let you change the build tag from ''localport''. This step is optional.

  cp -v /usr/ports/utilities/templates/portmake.conf /etc/

If desired, edit the file /etc/portmake.conf to your liking.

===== Build a few Packages Manually =====

The following packages must be built manually: pkgtools, which and tar 1.13.

First, cd to the directory the sources are in:

  cd /sources

==== Install pkgtools ====

Install pkgtools to /tools. Note: substitute the version of pkgtools you downloaded in chapter 3 for "c2.0.0"

  tar -xzf c2.0.0.tar.gz
  cd pkgtools-*
  make DESTDIR=/tools install
  
==== Install Tar 1.13 ====

Install Tar 1.13 to /tools:

  cd ..
  tar -xzf tar-1.13.tar.gz
  cd tar-1.13
  ./configure --prefix=/tools --program-suffix=-1.13
  make
  make install
  
==== "Install" Which ====

The full fledged version of Which is not necessary at this point, so we will install a minimal script to emulate the essential functionality:

  cat > /tools/bin/which << "EOF"
  #!/bin/bash
  type -pa "$@" | head -n 1 ; exit ${PIPESTATUS[0]}
  EOF
  chmod -v 755 /tools/bin/which


===== Building the Phase 2 Bootstrap =====

It is necessary to build a small set of bootstrap packages manually before building the rest of phase 2.

First, create a dummy gpg to use as a workaround. We will not install gpg until later, but certain packages will not build without it installed. This is safe to do since we already verified the integrity of the downloaded sources when we initially downloaded them in chapter 3. It will be placed in /tools to ensure that it is deleted at the end of the build and will not leak into the final system.

  cat > /tools/bin/gpg << EOF
  #!/bin/bash
  
  exit 0
  EOF
  chmod 755 /tools/bin/gpg


Also create a dummy sudo to satisfy a dependency in Portmake:

  cat > /tools/bin/sudo << EOF
  #!/bin/bash
  
  while [ \$# -ne 0 ]; do
    case \$1 in
      '-u')
        shift 2
        ;;
      *)
        exec \$@
        ;;
    esac
  done
  EOF
  chmod 755 /tools/bin/sudo

Build and install them:

  /usr/ports/utilities/tools/portmake install linux-headers
  /usr/ports/utilities/tools/portmake install man-pages
  /usr/ports/utilities/tools/portmake install glibc

Run the test suites for glibc. This step is considered critical and should not be skipped. A few test failures is nothing to be alarmed about; it is usually cause for alarm only if a large percentage of the test fail.

  cd /tmp/glibc-*/src/glibc-*/build
  make -j $(nproc) check
  
===== Adjusting the Toolchain =====

This section is based off of [[http://www.linuxfromscratch.org/lfs/view/8.0/chapter06/adjusting.html]].

==== Perform the Adjustment ====

First, backup the /tools linker, and replace it with the adjusted linker we made in chapter 5. We'll also create a link to its counterpart in /tools/$(uname -m)-pc-linux-gnu/bin:

  mv -v /tools/bin/{ld,ld-old}
  mv -v /tools/$(uname -m)-pc-linux-gnu/bin/{ld,ld-old}
  mv -v /tools/bin/{ld-new,ld}
  ln -sv /tools/bin/ld /tools/$(uname -m)-pc-linux-gnu/bin/ld
  
Next, amend the GCC specs file so that it points to the new dynamic linker. Simply deleting all instances of “/tools” should leave us with the correct path to the dynamic linker. Also adjust the specs file so that GCC knows where to find the correct headers and Glibc start files. A sed command accomplishes this:

  case $(uname -m) in
    x86_64) export LIBDIRSUFFIX=64 ;;
  esac
  
  gcc -dumpspecs | sed -e 's@/tools@@g'                   \
    -e '/\*startfile_prefix_spec:/{n;s@.*@/usr/lib'${LIBDIRSUFFIX}'/ @}' \
    -e '/\*cpp:/{n;s@$@ -isystem /usr/include@}' >      \
    `dirname $(gcc --print-libgcc-file-name)`/specs

It is a good idea to visually inspect the specs file to verify the intended change was actually made.

==== Sanity Check the Toolchain ====

It is imperative at this point to ensure that the basic functions (compiling and linking) of the adjusted toolchain are working as expected. To do this, perform the following sanity checks:

  cd /tmp
  echo 'int main(){}' > dummy.c
  cc dummy.c -v -Wl,--verbose &> dummy.log
  readelf -l a.out | grep ': /lib'

There should be no errors, and the output of the last command will be (allowing for platform-specific differences in dynamic linker name):

  [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

Now make sure that we're setup to use the correct start files:

  grep -o '/usr/lib.*/crt[1in].*succeeded' dummy.log

The output of the last command should be (allowing for platform specific variations):

  /usr/lib/../lib/crt1.o succeeded
  /usr/lib/../lib/crti.o succeeded
  /usr/lib/../lib/crtn.o succeeded



Verify that the compiler is searching for the correct header files:

  grep -B1 '^ /usr/include' dummy.log

This command should return the following output:

  #include <...> search starts here:
   /usr/include

Next, verify that the new linker is being used with the correct search paths:

  grep 'SEARCH.*/usr/lib' dummy.log |sed 's|; |\n|g'

References to paths that have components with '-linux-gnu' should be ignored, but otherwise the output of the last command should be:

  SEARCH_DIR("/usr/lib")
  SEARCH_DIR("/lib")

Next make sure that we're using the correct libc:

  grep "/lib.*/libc.so.6 " dummy.log

The output of the last command should be:

  attempt to open /lib/libc.so.6 succeeded

Lastly, make sure GCC is using the correct dynamic linker:

  grep found dummy.log

The output of the last command should be (allowing for platform-specific differences in dynamic linker name):

  found ld-linux-x86-64.so.2 at /lib/ld-linux-x86-64.so.2

===== Building the Remainder of Phase 2 =====

Now it is time to build the rest of phase 2. Fortunately, the remainder of this process can be automated with portmake. Run the following (note that this command will take a //long// time):

  /usr/ports/utilities/tools/portmake install phase2

====== Chapter 7 - Building Phase 3 ======

This phase is akin to building a few select packages from the Beyond Linux from Scratch Book.

At the end of this phase the system will be capable of both running *and* compiling itself; however, it will not be complete yet.

The entirety this process can be automated with portmake. Run the following (note that this command will take a //long// time):

  /usr/ports/utilities/tools/portmake install phase3

====== Chapter 8 - Building Phase 4 ======

This phase is akin to building additional packages from the Beyond Linux from Scratch Book.

At the end of this phase the system will be complete.

===== Modifying the Chroot Environment =====

Before we can build the remainder of the system, we must make a few modifications to the chroot environment to enable the ports tree and portmake to function properly.

==== Set up DNS ====

Create a resolv.conf so the chroot can resolve DNS queries. Feel free to substitute in your favorite DNS servers here.

  cat > /etc/resolv.conf << EOF
  nameserver 208.67.222.222
  nameserver 208.67.220.220
  EOF

==== Remove the Tools Directory ====

The ''/tools'' directory is no longer needed, so remove it to ensure that the tools located there do not impact phase 4 and update the PATH variable accordingly.

  rm -rf /tools
  export PATH=/bin:/usr/bin:/sbin:/usr/sbin

===== Building the Packages =====

Proceed to build the phase 4 packages. Similarly to phase 3, this entire process can be automated with portmake. Note that this will take a //very long// time (even longer than phases 1. 2 & 3).

  /usr/ports/utilities/tools/portmake install phase4

====== Chapter 9 - Creating the Installer Image ======

We are done building the packages now, so exit the chroot environment.

  exit

===== Getting the Installer Buildscripts =====

Navigate to /opt/culfs and download the installer buildscripts from GitHub:

  cd /opt/culfs
  git clone https://github.com/cucumberlinux/installer.git

===== Building the Installer =====

Enter the installer directory.

  cd installer

Run the build-iso.sh script, setting some special variables.

  PORTSDIR=/opt/culfs/chroot/usr/ports PKGDIR=/opt/culfs/chroot/opt/packages ./build-iso.sh

This will the installer ISO file and place it in /tmp.

====== The End ======

Congrats, you made it!