Sometime ago chromium 60 was released. I decided to upgrade but it turns out also gcc needed to be upgraded. Usually that is not a big deal, but since version 5, they have upgraded the c-language standard from gnu89 to gnu11 which may require rebuilding some libraries in order for them to provide a 'strict' ABI or interface-ish type of thing thats used in C++ and C.

As far as I understand it's sometimes not enough to say that function x returns type y and takes parameters a, b, c of type d, e, f you need to aditionally enforce how the underlying dependencies and primitive types are implemented to avoid applications from breaking in cases where the author assumes for example that the default integer type is capable of storing such-and-such amount of bits, but the user compiles the application on a system where the default integer type is far less and the application breaks because there is not enough room to store what it needs to store (this example is probably too simplistic).

This strict ABI business desireable for interoperability between OSen / and standard-libraries. So if I build an application on MS Windows using only ABI X, then I can compile and run if with a toolchain that provides ABI X anywhere. Sounds almost like Java, but not quite so. I'm only writing this down to se how my knowledge expands in a few years.

The point of this article is I had a hard time with some libraries. I wound up rebuilding chromium in a "fresh" chroot and merging the binaries to the host OS.

First signs of trouble

Something tipped me off about error messages like

undefined reference to `sqlite::result::get_column_name[abi:cxx11](int)'

because I know that sqlite is present on the system, yet the compiler could not find a specific version of it, labeled "cxx11". I checked out the release page for gcc-5, and true enough, some things had changed. I decided to rebuild all the widely used libraries.

$ eix -Ic -C sys-devel  
[I] sys-devel/bc (1.06.95-r1{tbz2}@08/17/2015): Handy console-based calculator utility
[I] sys-devel/bin86 (0.16.21{tbz2}@04/13/2015): Assembler and loader used to create kernel bootsector
[I] sys-devel/binutils-config (5-r3{tbz2}@09/05/2017): Utility to change the binutils version being used
[I] sys-devel/clang (3.9.1-r100{tbz2}@09/09/2017): C language family frontend for LLVM
[I] sys-devel/clang-runtime (3.9.1{tbz2}@09/09/2017): Meta-ebuild for clang runtime libraries
[I] sys-devel/dev86 (0.16.19@06/30/2014): Bruce's C compiler - Simple C compiler to generate 8086 code
[I] sys-devel/gcc-config (1.7.3{tbz2}@12/13/2013): Utility to manage compilers
[I] sys-devel/gdb (7.12.1{tbz2}@09/01/2017): GNU debugger
[I] sys-devel/gettext ({tbz2}@09/10/2017): GNU locale utilities
[I] sys-devel/libtool (2.4.6-r3(2){tbz2}@09/08/2017): A shared library tool for developers
[I] sys-devel/llvm (3.9.1-r1{tbz2}@09/10/2017): Low Level Virtual Machine
[I] sys-devel/m4 (1.4.17{tbz2}@05/31/2016): GNU macro processor
[I] sys-devel/make (4.2.1{tbz2}@02/16/2017): Standard tool to compile source trees
[I] sys-devel/patch (2.7.5{tbz2}@01/19/2017): Utility to apply diffs to files

$ eix -Ic | grep libc 
[I] dev-cpp/libcmis (0.5.2_pre20160820-r1(0.5){tbz2}@09/10/2017): C++ client library for the CMIS interface
[I] dev-libs/libcdio (0.93{tbz2}@09/10/2017): A library to encapsulate CD-ROM reading and control
[I] dev-libs/libcdio-paranoia (0.93_p1{tbz2}@03/06/2017): an advanced CDDA reader with error correction
[I] dev-libs/libcgroup (0.41-r4{tbz2}@11/05/2016 -> 0.41-r4{tbz2}[3]): Tools and libraries to configure and manage kernel control groups
[I] dev-libs/libclc (0.2.0_pre20160921{tbz2}@09/09/2017): OpenCL C library
[I] dev-libs/libcroco (0.6.12-r1(0.6){tbz2}@09/08/2017): Generic Cascading Style Sheet (CSS) parsing and manipulation toolkit
[I] dev-python/pycurl (7.43.0{tbz2}@02/16/2017): python binding for curl/libcurl
[I] media-libs/libcanberra (0.30-r5{tbz2}@03/24/2016): Portable sound event library
[I] media-libs/libcdr (0.1.3{tbz2}@09/10/2017): Library parsing the Corel cdr documents
[I] media-libs/libcuefile (477-r1{tbz2}@10/25/2014): Cue File library from Musepack
[I] sys-libs/glibc (2.23-r4(2.2){tbz2}@09/09/2017): GNU libc6 (also called glibc2) C library
[I] sys-libs/libcap (2.24-r2{tbz2}@10/05/2015): POSIX 1003.1e capabilities
[I] sys-libs/libcap-ng (0.7.8{tbz2}@02/16/2017): POSIX 1003.1e capabilities
[I] virtual/libc (1{tbz2}@01/19/2017): Virtual for the C library

and also stuff like boost, llvm

$ equery g --depth=1 =sys-devel/llvm-3.9.1-r1
 * Searching for llvm3.9.1-r1 in sys-devel ...

 * dependency graph for sys-devel/llvm-3.9.1-r1
 `--  sys-devel/llvm-3.9.1-r1  [~amd64 keyword] 
   `--  sys-libs/zlib-1.2.11  (sys-libs/zlib) amd64 
   `--  dev-lang/python-2.7.12  (>=dev-lang/python-2.7.5-r2) amd64 
   `--  dev-lang/python-exec-2.4.4  (>=dev-lang/python-exec-2) amd64  [python_targets_python2_7(-)? -python_single_target_jython2_7(-) -python_single_target_pypy(-) -python_single_target_pypy3(-) -python_single_target_python3_4(-) -python_single_target_python3_5(-) -python_single_target_python3_6(-) python_single_target_python2_7(+)]
   `--  dev-lang/perl-5.24.1-r2  (dev-lang/perl) amd64 
   `--  dev-libs/libxml2-2.9.5  (dev-libs/libxml2) ~amd64  [abi_x86_32(-)? abi_x86_64(-)? abi_x86_x32(-)? abi_mips_n32(-)? abi_mips_n64(-)? abi_mips_o32(-)? abi_ppc_32(-)? abi_ppc_64(-)? abi_s390_32(-)? abi_s390_64(-)?]
   `--  sys-devel/binutils-2.28.1  (>=sys-devel/binutils-2.22) amd64  [cxx]
   `--  dev-libs/libedit-20130712.3.1  (dev-libs/libedit) amd64  [abi_x86_32(-)? abi_x86_64(-)? abi_x86_x32(-)? abi_mips_n32(-)? abi_mips_n64(-)? abi_mips_o32(-)? abi_ppc_32(-)? abi_ppc_64(-)? abi_s390_32(-)? abi_s390_64(-)?]
   `--  virtual/libffi-3.0.13-r1  (>=virtual/libffi-3.0.13-r1) amd64  [abi_x86_32(-)? abi_x86_64(-)? abi_x86_x32(-)? abi_mips_n32(-)? abi_mips_n64(-)? abi_mips_o32(-)? abi_ppc_32(-)? abi_ppc_64(-)? abi_s390_32(-)? abi_s390_64(-)?]
   `--  dev-python/six-1.10.0  (dev-python/six) amd64  [python_targets_python2_7(-)? -python_single_target_jython2_7(-) -python_single_target_pypy(-) -python_single_target_pypy3(-) -python_single_target_python3_4(-) -python_single_target_python3_5(-) -python_single_target_python3_6(-) python_single_target_python2_7(+)]
   `--  sys-libs/ncurses-6.0-r1  (>=sys-libs/ncurses-5.9-r3) amd64  [abi_x86_32(-)? abi_x86_64(-)? abi_x86_x32(-)? abi_mips_n32(-)? abi_mips_n64(-)? abi_mips_o32(-)? abi_ppc_32(-)? abi_ppc_64(-)? abi_s390_32(-)? abi_s390_64(-)?]
   `--  dev-lang/ocaml-4.04.2  (>=dev-lang/ocaml-4.00.0) amd64 
   `--  dev-ml/ocaml-ctypes-0.11.2  (dev-ml/ocaml-ctypes) amd64 
   `--  sys-devel/make-4.2.1  (>=sys-devel/make-3.81) amd64 
   `--  sys-devel/flex-2.6.1  (>=sys-devel/flex-2.5.4) amd64 
   `--  sys-devel/bison-3.0.4-r1  (>=sys-devel/bison-1.875d) amd64 
   `--  sys-devel/gcc-6.4.0  (>=sys-devel/gcc-3.0) ~amd64 
   `--  sys-devel/llvm-3.9.1-r1  (>=sys-devel/llvm-3.5) [~amd64 keyword] 
   `--  sys-freebsd/freebsd-lib-11.0-r1  (>=sys-freebsd/freebsd-lib-9.1-r10) [missing keyword] 
   `--  sys-libs/libcxx-3.9.1  (sys-libs/libcxx) amd64 
   `--  sys-devel/binutils-apple-8.2.1  (>=sys-devel/binutils-apple-5.1) [missing keyword] 
   `--  virtual/pkgconfig-0-r1  (virtual/pkgconfig) amd64 
   `--  dev-python/sphinx-1.6.3-r2  (dev-python/sphinx) amd64 
   `--  sys-libs/binutils-libs-2.28-r1  (sys-libs/binutils-libs) amd64 
   `--  dev-lang/swig-3.0.12  (dev-lang/swig) amd64 
   `--  dev-ml/findlib-1.7.1  (dev-ml/findlib) amd64 
   `--  dev-ml/ounit-2.0.0  (dev-ml/ounit) amd64 
   `--  dev-util/ninja-1.7.2  (dev-util/ninja) amd64 
   `--  dev-util/cmake-3.7.2  (>=dev-util/cmake-3.6.1-r1) amd64 
   `--  sys-devel/clang-3.9.1-r100  (=sys-devel/clang-3.9.1-r100) amd64 
   `--  sys-devel/clang-runtime-3.9.1  (~sys-devel/clang-runtime-3.9.1) amd64 
[ sys-devel/llvm-3.9.1-r1 stats: packages (31), max depth (1) ]

Before doing all of that I disabled ccache in /etc/portage/make.conf

Still I could not compile some packages:

    balmora: /var/log/portage 
    $ ack 'internal com' 
    8055:/usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include/g++-v6/bits/stl_tree.h:1643:5:  \
    internal compiler error: Segmentation fault
    25292:../../ui/aura/window_tree_host.h:67:38: \
    internal compiler error: Segmentation fault

    3506:/var/tmp/portage/www-client/firefox-52.3.0/work/firefox-52.3.0esr/docshell/shistory/nsSHistory.cpp:1136:1: \
    internal compiler error: Segmentation fault

    30170:gen/headless/public/devtools/domains/ \
    internal compiler error: in extract_insn, at recog.c:2287
    38561:/usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include/g++-v6/type_traits:120:12: \
    internal compiler error: in iterative_hash_template_arg, at cp/pt.c:1849

    42839:/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/g++-v5/functional:1464:55: \
    internal compiler error: in iterative_hash_template_arg, at cp/pt.c:1713

    1926:/var/tmp/portage/net-libs/webkit-gtk-2.16.6/work/webkitgtk-2.16.6/Source/WTF/wtf/Expected.h:329:7: \
    internal compiler error: Segmentation fault

    19056:/var/tmp/portage/www-client/firefox-52.3.0/work/firefox-52.3.0esr/ff/ipc/ipdl/_ipdlheaders/mozilla/dom/PBrowser.h:49:5: \
    internal compiler error: Segmentation fault

And many more like these, with more or less specific error messages. ( Something nuked my /var/log/portage, or I did by accedent - can't find older build logs right now.) I found out that it helps to rebuild the toolchain in some cases, but since I had already done that, I also concluded that trying to resolve ICEs is a poor use of my time.

After checking out the Gentoo bugzilla for similar issues and stating my problem on #gentoo ( I realized that the flaw must be in my system and not the new gcc or chromium code, and also that debugging all of it would be a silly endaveour.

Nuke the system

The beauty of Gentoo is flexibility, versatility and how powerful it is if you have some idea about what you are doing. I decided on the following plan;

  1. get the latest stage3 and chroot into that
  2. enable binary packages
  3. build chromium, firefox and other big packages that were failing
  4. merge the packages back into the host system
  5. rebuild all the packages present in stage 3 as binary packages
  6. rebuild the exact same list of packages on the HOST using the binary packages where possible

I've done similar operations before, where the simplest solution is to simply untar the latest stage3 in your root. It will work, but you risk breaking your init system and causing other "unforseen" side effects, like nuking your /etc folder :).

Since I started with gentoo in about 2006, I think I have done a "fresh install" about 4-5 times due to the system being b0rked. Even then, I was able to recover all of the settings and OS configurations. And whenever I would work in a VM or a new laptop from an employer I would just move my /etc and /home untar the apropriate-architechture stage3 and my "whole system" would be ready "exactly" as it were some 2-3 days later, and I'd be able to do work even before X was done compiling because I'm a terminal wizard :D.

With Debian that happened every 18 months or more often because I had no idea what I was doing and pulled packages from different places. With Windows a fresh install is a "must" every six months... My current Win10 installation is an upgrade from Win7 from about 2014, and it's a powerful PC (I even moved it to different hardware), but there is so much unpredictable garbage going on there... even with a paid anti-virus package and years of experience on how to clean and maintain Windows (amateur basis)...

Most people I've met who know what Gentoo is will generally say, How can you wait for all the stuff to compile? It seems like such a waste of time. And I challenge you (especially Windows users); Migrate your system to new hardware, while keeping all your personal setting AND your OS setting AND the settings of your development environment in 2-4 days.

Actually I realize that this is possible with most GNU / Linux distros, do you accept the challenge?

The fruits

Holy sheet it's fast! Even faster than the binary google-chrome-stable package. New Firefox is good too, but I started using Chromium frequently about a year ago and I have to say, it's like the Tesla of electric cars for web development... A bit of a broken analogy there... yaya