NOTE: I took the time to get to the bottom of the issue discussed in this post. There’s a new post here that explains the “right way” to use Makefiles with yocto. As always, the error in this post was mine š
I’ve officially “drank the Kool-Aid” and I’m convinced openembedde and Yocto are pretty awesome. I’ve had a blast building small Debian systems on PCEngines hardware in the past and while I’m waiting for my Raspberry Pi to arrive I’ve been trying to learn the ins and outs of Yocto. The added bonus is that the XenClient team at Citrix uses openembedded for our build system so this work can also fall under the heading of “professional development”.
Naturally the first task I took on was way too complicated so I made a bunch of great progress (more about that in a future post once I get it stable) but then I hit a wall that I ended up banging my head against for a full day. I posted a cry for help on the mailing list and didn’t get any responses so I set out to remove as many moving parts as possible and find the root cause.
First things first read the Yocto development manual and the Yocto reference for whatever release you’re using. This is essential because no one will help you till you’ve read and understand these š
So the software I’m trying to build is built using raw Makefiles, none of that fancy autotools stuff. This can be a bit of a pain because depending on the Makefiles, it’s not uncommon for assumptions to be made about file system paths. Openembedded is all about cross compiling so it wants to build and install software under all sorts of strange roots and some Makefiles just can’t handle this. I ran into a few of these scenarios but nothing I couldn’t overcome.
Getting a package for my target architecture wasn’t bad but I did run into a nasty problem when I tried to get a native package built. From the searches I did on the interwebs it looks like there have been a number of ways to build native packages. The current “right way” is simply to have your recipe extend the native class. Thanks to XorA for documenting his/her new package workflow for that nugget.
BBCLASSEXTEND = "native"
After having this method blow up for my recipe I was tempted to hack together some crazy work around. I really want to upstream the stuff I’m working on though and I figure having crazy shit in my recipe to work around my misunderstanding of the native class was setting the whole thing up for failure. So instead I went back to basics and made a “hello world” program and recipe (included at the end of this post) hoping to recreate the error and hopefully figure out what I was doing wrong at the same time.
It took a bit of extra work but I was able to recreate the issue with a very simple Makefile. First the error message:
NOTE: package hello-native-1.0-r0: task do_populate_sysroot: Started ERROR: Error executing a python function in /home/build/poky-edison-6.0/meta-test/recipes-test/helloworld/hello_1.0.bb: CalledProcessError: Command 'tar -cf - -C /home/build/poky-edison-6.0/build/tmp/work/i686-linux/hello-native-1.0-r0/sysroot-destdir///home/build/poky-edison-6.0/build/tmp/sysroots/i 686-linux -ps . | tar -xf - -C /home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux' returned non-zero exit status 2 with output tar: /home/build/poky-edison-6.0/build/tmp/work /i686-linux/hello-native-1.0-r0/sysroot-destdir///home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux: Cannot chdir: No such file or directory tar: Error is not recoverable: exiting now tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors ERROR: The stack trace of python calls that resulted in this exception/failure was: ERROR: File "sstate_task_postfunc", line 10, in ERROR: ERROR: File "sstate_task_postfunc", line 4, in sstate_task_postfunc ERROR: ERROR: File "sstate.bbclass", line 19, in sstate_install ERROR: ERROR: File "/home/build/poky-edison-6.0/meta/lib/oe/path.py", line 59, in copytree ERROR: check_output(cmd, shell=True, stderr=subprocess.STDOUT) ERROR: ERROR: File "/home/build/poky-edison-6.0/meta/lib/oe/path.py", line 121, in check_output ERROR: raise CalledProcessError(retcode, cmd, output=output) ERROR: ERROR: The code that was being executed was: ERROR: 0006: bb.build.exec_func(intercept, d) ERROR: 0007: sstate_package(shared_state, d) ERROR: 0008: ERROR: 0009: ERROR: *** 0010:sstate_task_postfunc(d) ERROR: 0011: ERROR: (file: 'sstate_task_postfunc', lineno: 10, function: ) ERROR: 0001: ERROR: 0002:def sstate_task_postfunc(d): ERROR: 0003: shared_state = sstate_state_fromvars(d) ERROR: *** 0004: sstate_install(shared_state, d) ERROR: 0005: for intercept in shared_state['interceptfuncs']: ERROR: 0006: bb.build.exec_func(intercept, d) ERROR: 0007: sstate_package(shared_state, d) ERROR: 0008: ERROR: (file: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc) ERROR: Function 'sstate_task_postfunc' failed ERROR: Logfile of failure stored in: /home/build/poky-edison-6.0/build/tmp/work/i686-linux/hello-native-1.0-r0/temp/log.do_populate_sysroot.30718 Log data follows: | NOTE: QA checking staging | ERROR: Error executing a python function in /home/build/poky-edison-6.0/meta-test/recipes-test/helloworld/hello_1.0.bb: | CalledProcessError: Command 'tar -cf - -C /home/build/poky-edison-6.0/build/tmp/work/i686-linux/hello-native-1.0-r0/sysroot-destdir///home/build/poky-edison-6.0/build/tmp/sysroots /i686-linux -ps . | tar -xf - -C /home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux' returned non-zero exit status 2 with output tar: /home/build/poky-edison-6.0/build/tmp/wo rk/i686-linux/hello-native-1.0-r0/sysroot-destdir///home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux: Cannot chdir: No such file or directory | tar: Error is not recoverable: exiting now | tar: This does not look like a tar archive | tar: Exiting with failure status due to previous errors | | | ERROR: The stack trace of python calls that resulted in this exception/failure was: | ERROR: File "sstate_task_postfunc", line 10, in | ERROR: | ERROR: File "sstate_task_postfunc", line 4, in sstate_task_postfunc | ERROR: | ERROR: File "sstate.bbclass", line 19, in sstate_install | ERROR: | ERROR: File "/home/build/poky-edison-6.0/meta/lib/oe/path.py", line 59, in copytree | ERROR: check_output(cmd, shell=True, stderr=subprocess.STDOUT) | ERROR: | ERROR: File "/home/build/poky-edison-6.0/meta/lib/oe/path.py", line 121, in check_output | ERROR: raise CalledProcessError(retcode, cmd, output=output) | ERROR: | ERROR: The code that was being executed was: | ERROR: 0006: bb.build.exec_func(intercept, d) | ERROR: 0007: sstate_package(shared_state, d) | ERROR: 0008: | ERROR: 0009: | ERROR: *** 0010:sstate_task_postfunc(d) | ERROR: 0011: | ERROR: (file: 'sstate_task_postfunc', lineno: 10, function: ) | ERROR: 0001: | ERROR: 0002:def sstate_task_postfunc(d): | ERROR: 0003: shared_state = sstate_state_fromvars(d) | ERROR: *** 0004: sstate_install(shared_state, d) | ERROR: 0005: for intercept in shared_state['interceptfuncs']: | ERROR: 0006: bb.build.exec_func(intercept, d) | ERROR: 0007: sstate_package(shared_state, d) | ERROR: 0008: | ERROR: (file: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc) | ERROR: Function 'sstate_task_postfunc' failed NOTE: package hello-native-1.0-r0: task do_populate_sysroot: Failed ERROR: Task 3 (virtual:native:/home/build/poky-edison-6.0/meta-test/recipes-test/helloworld/hello_1.0.bb, do_populate_sysroot) failed with exit code '1' ERROR: 'virtual:native:/home/build/poky-edison-6.0/meta-test/recipes-test/helloworld/hello_1.0.bb' failed
So even with the most simple Makefile I could cause a native recipe build to blow up. Here’s the Makefile:
.PHONY : all clean install uninstall PREFIX ?= $(DESTDIR)/usr BINDIR ?= $(PREFIX)/bin HELLO_src = hello.c HELLO_bin = hello HELLO_tgt = $(BINDIR)/$(HELLO_bin) all : $(HELLO_bin) $(HELLO_bin) : $(HELLO_src) $(HELLO_tgt) : $(HELLO_bin) install -d $(BINDIR) install -m 0755 $^ $@ clean : rm $(HELLO_bin) install : $(HELLO_tgt) uninstall : rm $(BINDIR)/$(HELLO_tgt)
And here’s the relevant install method from the bitbake recipe:
do_install () {
oe_runmake DESTDIR=${D} install
}
Notice I’m using the variable DESTDIR to tell the Makefile the root (not just /) to install things to. This should work right? It works for a regular package but not for a native one! This drove me nuts for a full day.
The solution to this problem lies in some weirdness in the Yocto native class when combined with the populate_sysroot method. The way I figured this out was by inspecting the differences in the environment when building hello vs hello-native. When building the regular package for the target architecture variables like bindir and sbindir were what I would expect them to be:
bindir="/usr/bin" sbindir="/usr/sbin"
but when building hello-native they get a bit crazy:
bindir="/home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux/usr/bin" sbindir="/home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux/usr/sbin"
This is a hint at the source of crazy path that staging is trying to tar up above in the error message. Further if you look in the build directory for a regular target arch package you’ll see your files where you expect in ${D}sysroot-destdir/usr/bin but for a native build you’ll see stuff in ${D}sysroot-destdir/home/build/poky-edison-6.0/build/tmp/sysroots/i686-linux/usr/bin. Pretty crazy right? I’m sure there’s a technical reason for this but it’s beyond me.
So the way you can work around this is by telling your Makefiles about paths like bindir through the recipe. A fixed do_install would look like this:
do_install () {
oe_runmake DESTDIR=${D} BINDIR=${D}${bindir} install
}
For more complicated Makefiles you can probably specify a PREFIX and set this equal to the ${prefix} variable but YMMV. I’ll be trying this out to keep my recipes as simple as possible.
If you want to download my example the recipe is here. This will pull down the hello world source code and build the whole thing for you.
technomasochism 
I don’t tink there are many more reliable ways of self inflicting pain than handwriting makefiles for a project you intend to cross-compile. Save youself lot of pain and use autotools from the beginning, lof of work over many years went into the Poky autotools support.
LikeLike
Makefiles have been my personal favorite for self inflicted pain for a while now … but seriously, I suffered through this exercise to get a recipe working for someone else’s software. It was either rewrite their build system to use autotools and get that accepted upstream, or figure out the ins and outs of the ‘native’ class / environment. I agree though, new projects should use autotools. But until every source tree out there is built with autotools someone will have to write recipes that can cope with hand written Makefiles.
LikeLike
This “craziness” does have a rational explaination. “native” targets are meant to run on the system they’re built on and run in the location they’re installed to. This means they install to a destination of “/” and PREFIX is inside the native sysroot directory.
We install them to a DESTDIR to allow us to manipulate them before they then get moved to a final DESTDIR of “/”.
Most Makefiles handle this correctly by doing:
DESTDIR ?= “”
prefix ?= “/usrr”
bindir ?= “$(prefix)/bin”
and then, importantly, install in the form:
install -d $(DESTDIR)$(bindir)
so both prefix and DESTDIR are used. Whilst this is a convention, its a widely adopted and followed one. As you’ve found, you can call into a custom makefile and set the variables manually if the makefile doesn’t follow the convention.
LikeLike
UPDATE: Months later but I finally realized the main point of Rich’s comments. I was using
PREFIXinstead ofprefix. Case sensitivity is a bitch!Thanks for the comment Rich! I’m with you on the Makefile convention. The “hello world” example I give follows this convention quite closely (I think).
My confusion came from the change in the bindir variable in the OE environment for “native” packages. The change being ${bindir} for a cross compiled package as /usr/bin but in the “native” environment it has ${exec_prefix} pretended to it. All the while ${D} remains unchanged. I had expected (incorrectly) to be able to simply pass DESTDIR=${D} in my do_install to oe_runmake in both environments without issue. Following the convention you identify makes working through this difference possible but it isn’t what caused my confusion.
Regardless the veil has been lifted! Yocto / OE + properly written Makefiles = awesomeness … though a distant second to Yocto / OE + autotools š
LikeLike