There comes a time in life of a developer when one gets this error

valgrind: failed to start tool 'memcheck' for platform 'x86-linux'

I do not see myself as an expert Valgrind user, but I understand output of Valgrind enough to catch memory leaks in my C programs.

The other day I launched a personal project in assembly language and used standard C memory routines (calloc, realloc, free) for its
memory requirements. The program grew complex  and I wanted to be sure I am not leaking any memory (and also to be sure I am
leaking memory where I know I am leaking at that stage - amid development), and was also driven by the curiosity to see how well
Valgrind handled a program written in Assembly. After all, I have had never used Valgrind before for program written in any language
other than C.

So I launched Valgrind like so, and got that error - completely unexpected:

$ $ valgrind ./atail32_80 test.txt valgrind: failed to start tool 'memcheck' for platform 'x86-linux': No such file or directory $

Thought never crossed my mind that the error was due to Assembly language front-end - and I was on the right track with that!

I hypothesized that the error was due to the fact that the program being debugged was a 32-bit program^* on a 64-bit platform
(see below).

$ $ uname -i x86_64 $ file ./atail32_80 ./atail32_80: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0xf3e3af4712a9382cb2b793f580a26a6797d64d1f, not stripped $

^*I was building that program as a 32-bit executable on purpose (further discussion about that would be irrelevant to this post).

I launched an investigation console and performed this small experiment to confirm my hypothesis.

$ echo "void main(){}" > testme.c $ $ gcc  -m32  -o testme32  testme.c $ $ gcc  -m64  -o testme64  testme.c $ $ file testme32 testme32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x305f621662204c0289ee57e927c0221af84e4597, not stripped $ $ file testme64 testme64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x285da18d5e71346dfd958b260fccfe3717cd2b99, not stripped $ $ valgrind ./testme64 ==28701== Memcheck, a memory error detector ==28701== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==28701== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==28701== Command: ./testme64 ==28701== ==28701== ==28701== HEAP SUMMARY: ==28701==     in use at exit: 0 bytes in 0 blocks ==28701==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==28701== ==28701== All heap blocks were freed -- no leaks are possible ==28701== ==28701== For counts of detected and suppressed errors, rerun with: -v ==28701== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) $ $ valgrind ./testme32 valgrind: failed to start tool 'memcheck' for platform 'x86-linux': No such file or directory

The same error message on the last line above confirmed the cause of the error; but what was happening really ?

Knowing some tools helps; strace seemed liked the right one at this moment.

$ $ strace  -e trace=file valgrind  ./testme32 execve("/usr/bin/valgrind", ["valgrind", "./testme32"], [/* 67 vars */]) = 0 access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 open("./testme32", O_RDONLY)            = 3 readlink("/proc/self/exe", "/usr/bin/valgrind", 4096) = 17 execve("/usr/lib64/valgrind/memcheck-x86-linux", ["valgrind", "./testme32"], [/* 68 vars */]) = -1 ENOENT (No such file or directory) valgrind: failed to start tool 'memcheck' for platform 'x86-linux': No such file or directory +++ exited with 1 +++ $

The execve() line just above the error message, that is now familiar to us, gives it away. Valgrind is trying to launch
/usr/lib64/valgrind/memcheck-x86-linux which seems to be missing.

Let us confirm:

$ $ ls /usr/lib64/valgrind/memcheck* /usr/lib64/valgrind/memcheck-amd64-linux  /usr/lib64/valgrind/memcheck-x86-linux $

It is there ! But the file IS there - and at the very location expected by Valgrind ! Then why the failure ?

Wait, before concluding, recall that on *nix, we have all learned either the hard or the soft way, "When in doubt, think symbolic link".
On those lines:

$ $ cd /usr/lib64/valgrind/ $ $ pwd /usr/lib64/valgrind $ $ ls -l memcheck* -rwxr-xr-x. 1 root root 3449056 Apr 25  2013 memcheck-amd64-linux lrwxrwxrwx. 1 root root      37 Jun 10 04:42 memcheck-x86-linux -> ../../lib/valgrind/memcheck-x86-linux $ $ ls -l ../../lib/valgrind/memcheck-x86-linux ls: cannot access ../../lib/valgrind/memcheck-x86-linux: No such file or directory $

So, memcheck-86-linux is indeed a symbolic link, and the target of which is missing.

Looking further, we can confirm that symbolic link /usr/lib64/valgrind/memcheck-x86-linux is created by/belongs to Valgrind package.
No surprises there!

$ $ yum provides /usr/lib64/valgrind/memcheck-x86-linux Loaded plugins: langpacks, refresh-packagekit google-chrome/filelists                                                                                         | 1.8 kB  00:00:00     1:valgrind-3.8.1-15.fc19.x86_64 : Tool for finding memory management bugs in programs Repo        : fedora Matched from: Filename    : /usr/lib64/valgrind/memcheck-x86-linux 1:valgrind-3.8.1-15.fc19.x86_64 : Tool for finding memory management bugs in programs Repo        : installed Matched from: Filename    : /usr/lib64/valgrind/memcheck-x86-linux $

Digging further, we find that two memcheck* symbolic links created by valgrind.x86_64 are broken.

$ $ rpm -ql valgrind | grep memcheck /usr/lib64/valgrind/memcheck-amd64-linux /usr/lib64/valgrind/memcheck-x86-linux /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so /usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so $ $ $ rpm -ql valgrind | grep memcheck | xargs file /usr/lib64/valgrind/memcheck-amd64-linux:  ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=0x82b75aa9e8494556aa050869e5e7ed20e9522137, stripped /usr/lib64/valgrind/memcheck-x86-linux:  broken symbolic link to `../../lib/valgrind/memcheck-x86-linux' /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xdd994ac2a32bece59363fadc2003e49a1b8c3539, stripped /usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so:  broken symbolic link to `../.. /lib/valgrind/vgpreload_memcheck-x86-linux.so' $

Let us continue the expedition and find out what other Valgrind packages are available that could possibly bring in the missing files
that those symbolic links are pointing to:

$ $ yum list valgrind* Loaded plugins: langpacks, refresh-packagekit Installed Packages valgrind.x86_64                                       1:3.8.1-15.fc19                                            installed Available Packages valgrind.i686                                           1:3.8.1-15.fc19                                            fedora   valgrind-devel.i686                                 1:3.8.1-15.fc19                                            fedora   valgrind-devel.x86_64                             1:3.8.1-15.fc19                                            fedora   valgrind-openmpi.x86_64                       1:3.8.1-15.fc19                                            fedora $

It is evident that we have 64-bit Valgrind package installed on the system, and 32-bit counter part of the Valgrind package is available (but
not installed on the system).

The solution to my problem seems within arm's reach - Install the package valgrind.i686
But I want to confirm two things before I install that package.
(a) valgrind.i686 will bring in (at least) those two missing files
(b) valgrind.i686 will not overwrite important valgrind binaries with 32-bit versions and break testing of my 64-bit executables.

Let us confirm (a) first:

$ $ repoquery -l valgrind.i686 | grep memcheck /usr/lib/valgrind/memcheck-x86-linux /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so $

Sure, the targets of those two broken symbolic link matches the path and file names of these two files.
That confirms that valgrind.i686 will bring in missing files.

Now, let us confirm (b):

$ $ repoquery -l valgrind.x86_64 | grep bin /usr/bin/callgrind_annotate /usr/bin/callgrind_control /usr/bin/cg_annotate /usr/bin/cg_diff /usr/bin/cg_merge /usr/bin/ms_print /usr/bin/valgrind /usr/bin/valgrind-listener /usr/bin/vgdb $ $ repoquery -l valgrind.i686 | grep bin /usr/bin/callgrind_annotate /usr/bin/callgrind_control /usr/bin/cg_annotate /usr/bin/cg_diff /usr/bin/cg_merge /usr/bin/ms_print /usr/bin/valgrind /usr/bin/valgrind-listener /usr/bin/vgdb $ $ repoquery -l valgrind.x86_64 | grep bin | xargs file /usr/bin/callgrind_annotate: Perl script, ASCII text executable /usr/bin/callgrind_control:    Perl script, ASCII text executable /usr/bin/cg_annotate:            Perl script, ASCII text executable /usr/bin/cg_diff:                      Perl script, ASCII text executable /usr/bin/cg_merge:                 ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5228b41ce865c922a7948698c40a6ed1b7c7cf1c, stripped /usr/bin/ms_print:                  Perl script, ASCII text executable /usr/bin/valgrind:                   ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x19186524e13c9eaaacdc8bec0144b3d82bb0800a, stripped /usr/bin/valgrind-listener:     ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5c9a1aa7055c71fcb7cab739509940deb4594464, stripped /usr/bin/vgdb:                         ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0xcbcdb0a010739b8ebfbff0abd508683987646c30, stripped $

That seems like an issue.
There are duplicate executable files between the 64-bit and 32-bit versions, four of which are binary files.
I estimated - at worst, 32-bit package would overwrite these binary files and, as anticipated, break testing of 64-bit executables; but the damage would be easily reversible with two steps: (a) Uninstall valgrind.i686 (b) Reinstall valgrind.x86_64

With that I was ready to plunge on with a plan -  If things broke, I would to make copy of those two symbolic link's target files, uninstall 32-bit package, reinstall 64-bit package, and finally manually restore broken symbolic links to fix the issue I originally started with.

So I did (sudo yum install valgrind.i686), and as it happens so, valgrind.i686 does not overwrite 64-bit executables.

$ $ rpm -ql valgrind.i686 | grep bin | xargs file /usr/bin/callgrind_annotate: Perl script, ASCII text executable /usr/bin/callgrind_control:  Perl script, ASCII text executable /usr/bin/cg_annotate:          Perl script, ASCII text executable /usr/bin/cg_diff:                    Perl script, ASCII text executable /usr/bin/cg_merge:              ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5228b41ce865c922a7948698c40a6ed1b7c7cf1c, stripped /usr/bin/ms_print:                Perl script, ASCII text executable /usr/bin/valgrind:                 ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x19186524e13c9eaaacdc8bec0144b3d82bb0800a, stripped /usr/bin/valgrind-listener:   ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5c9a1aa7055c71fcb7cab739509940deb4594464, stripped /usr/bin/vgdb:                       ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0xcbcdb0a010739b8ebfbff0abd508683987646c30, stripped $

I can not explain this behaviour at this stage without looking into .rpm packaging script logic of this package.

Let us see if rest is as expected. Let us confirm that broken symbolic links are repaired:

$ $ rpm -ql valgrind.x86_64 | grep memcheck | xargs file /usr/lib64/valgrind/memcheck-amd64-linux:    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=0x82b75aa9e8494556aa050869e5e7ed20e9522137, stripped /usr/lib64/valgrind/memcheck-x86-linux:         symbolic link to `../../lib/valgrind/memcheck-x86-linux' /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xdd994ac2a32bece59363fadc2003e49a1b8c3539, stripped /usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so:   symbolic link to `../../lib/valgrind/vgpreload_memcheck-x86-linux.so' $

Sure they are.


Now, like you I was curious to see if valgrind could test 32-bit executables now (without breaking 64-bit executables tests):

$ $ valgrind ./testme32 ==7914== Memcheck, a memory error detector ==7914== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==7914== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==7914== Command: ./testme32 ==7914== ==7914== ==7914== HEAP SUMMARY: ==7914==     in use at exit: 0 bytes in 0 blocks ==7914==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==7914== ==7914== All heap blocks were freed -- no leaks are possible ==7914== ==7914== For counts of detected and suppressed errors, rerun with: -v ==7914== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) $ $ valgrind ./testme64 ==8176== Memcheck, a memory error detector ==8176== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==8176== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==8176== Command: ./testme64 ==8176== ==8176== ==8176== HEAP SUMMARY: ==8176==     in use at exit: 0 bytes in 0 blocks ==8176==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==8176== ==8176== All heap blocks were freed -- no leaks are possible ==8176== ==8176== For counts of detected and suppressed errors, rerun with: -v ==8176== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2) $

Good. Valgrind is now capable to test both: 64-bit as well as 32-bit executables.

$ valgrind ./atail32_80 test.txt <snip> ==13468== ==13468== HEAP SUMMARY: ==13468==     in use at exit: 266 bytes in 10 blocks ==13468==   total heap usage: 18 allocs, 8 frees, 623 bytes allocated ==13468== ==13468== LEAK SUMMARY: ==13468==    definitely lost: 262 bytes in 9 blocks <snip>

Testing my project binary with valgrind did catch expected memory leaks on the first run, along with couple of unexpected memory
access related bugs (not shown here). Now that is going to keep me busy for some time.

Leave any comments/suggestions below.