valgrind: failed to start tool 'memcheck' for platform
'x86-linux'
There comes a time in life of a developer when
one gets this error
valgrind: failed to start tool
'memcheck' for platform 'x86-linux'
I do not see myself as an expert
Valgrind user,
but I understand output of Valgrind enough to catch memory leaks in
my C programs.
The other day I launched a personal
project in
assembly language and used standard C memory routines (calloc,
realloc, free) for its
memory requirements. The program grew complex and I wanted to
be sure I am not leaking any memory (and also to be sure I
am
leaking memory where I know I am leaking at that stage - amid
development), and was also driven by the curiosity to see how well
Valgrind handled a program written in Assembly. After all, I have
had never used Valgrind before for program written in any language
other than C.
So I launched Valgrind like so, and got that error - completely
unexpected:
$
$ valgrind ./atail32_80 test.txt
valgrind: failed to start tool 'memcheck' for platform
'x86-linux': No such file or directory
$
|
Thought never crossed my mind that the error was due to Assembly
language front-end - and I was on the right track with that!
I hypothesized that the error was due to the fact that the program
being debugged was a 32-bit program
* on a 64-bit
platform
(see below).
$
$ uname -i
x86_64
$ file ./atail32_80
./atail32_80: ELF 32-bit LSB executable,
Intel 80386, version 1 (SYSV), dynamically linked
(uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0xf3e3af4712a9382cb2b793f580a26a6797d64d1f,
not stripped
$
|
*I was building that program as a 32-bit
executable on purpose (further discussion about that would be
irrelevant to this post).
I launched an investigation console and performed this small
experiment to confirm my hypothesis.
$
echo "void main(){}" > testme.c
$
$ gcc -m32 -o testme32 testme.c
$
$ gcc -m64 -o testme64 testme.c
$
$ file testme32
testme32: ELF 32-bit LSB
executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x305f621662204c0289ee57e927c0221af84e4597,
not stripped
$
$ file testme64
testme64:
ELF 64-bit LSB executable,
x86-64, version 1 (SYSV), dynamically linked (uses
shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0x285da18d5e71346dfd958b260fccfe3717cd2b99,
not stripped
$
$ valgrind ./testme64
==28701== Memcheck, a memory error detector
==28701== Copyright (C) 2002-2012, and GNU GPL'd, by
Julian Seward et al.
==28701== Using Valgrind-3.8.1 and LibVEX; rerun with -h
for copyright info
==28701== Command: ./testme64
==28701==
==28701==
==28701== HEAP SUMMARY:
==28701== in use at exit: 0 bytes
in 0 blocks
==28701== total heap usage: 0 allocs, 0 frees,
0 bytes allocated
==28701==
==28701== All heap blocks were freed -- no leaks are
possible
==28701==
==28701== For counts of detected and suppressed errors,
rerun with: -v
==28701== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 2 from 2)
$
$ valgrind ./testme32
valgrind: failed to start tool 'memcheck' for platform
'x86-linux': No such file or directory
|
The same error message on the last line above confirmed the cause of
the error; but what was happening really ?
Knowing some tools helps;
strace seemed
liked the right one at this moment.
$
$ strace -e trace=file valgrind ./testme32
execve("/usr/bin/valgrind", ["valgrind", "./testme32"],
[/* 67 vars */]) = 0
access("/etc/ld.so.preload",
R_OK) = -1 ENOENT (No such
file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("./testme32",
O_RDONLY)
= 3
readlink("/proc/self/exe", "/usr/bin/valgrind", 4096) = 17
execve("/usr/lib64/valgrind/memcheck-x86-linux",
["valgrind", "./testme32"], [/* 68 vars */]) = -1 ENOENT
(No such file or directory)
valgrind: failed to start tool 'memcheck' for platform
'x86-linux': No such file or directory
+++ exited with 1 +++
$
|
The execve() line just above the error message, that is now familiar
to us, gives it away. Valgrind is trying to launch
/usr/lib64/valgrind/memcheck-x86-linux which seems to be missing.
Let us confirm:
$
$ ls /usr/lib64/valgrind/memcheck*
/usr/lib64/valgrind/memcheck-amd64-linux
/usr/lib64/valgrind/memcheck-x86-linux
$
|
It is there ! But the file
IS there - and at the very
location expected by Valgrind ! Then why the failure ?
Wait, before concluding, recall that on *nix, we have all learned
either the
hard or the
soft way, "When in doubt,
think
symbolic link".
On those lines:
$
$ cd /usr/lib64/valgrind/
$
$ pwd
/usr/lib64/valgrind
$
$ ls -l memcheck*
-rwxr-xr-x. 1 root root 3449056 Apr 25 2013
memcheck-amd64-linux
lrwxrwxrwx. 1 root
root 37 Jun 10 04:42 memcheck-x86-linux ->
../../lib/valgrind/memcheck-x86-linux
$
$ ls -l ../../lib/valgrind/memcheck-x86-linux
ls: cannot access ../../lib/valgrind/memcheck-x86-linux:
No such file or directory
$
|
So, memcheck-86-linux is
indeed a symbolic link, and the target of which is missing.
Looking further, we can confirm that symbolic link
/usr/lib64/valgrind/memcheck-x86-linux is created by/belongs to
Valgrind package.
No surprises there!
$
$ yum provides /usr/lib64/valgrind/memcheck-x86-linux
Loaded plugins: langpacks, refresh-packagekit
google-chrome/filelists
| 1.8 kB 00:00:00
1:valgrind-3.8.1-15.fc19.x86_64
: Tool for finding memory management bugs in programs
Repo : fedora
Matched from:
Filename :
/usr/lib64/valgrind/memcheck-x86-linux
1:valgrind-3.8.1-15.fc19.x86_64
: Tool for finding memory management bugs in programs
Repo : installed
Matched from:
Filename :
/usr/lib64/valgrind/memcheck-x86-linux
$
|
Digging further, we find that two memcheck* symbolic links created
by valgrind.x86_64 are broken.
$
$ rpm -ql valgrind | grep memcheck
/usr/lib64/valgrind/memcheck-amd64-linux
/usr/lib64/valgrind/memcheck-x86-linux
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so
/usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so
$
$
$ rpm -ql valgrind | grep memcheck | xargs file
/usr/lib64/valgrind/memcheck-amd64-linux: ELF 64-bit
LSB executable, x86-64, version 1 (SYSV), statically
linked,
BuildID[sha1]=0x82b75aa9e8494556aa050869e5e7ed20e9522137,
stripped
/usr/lib64/valgrind/memcheck-x86-linux:
broken symbolic link to
`../../lib/valgrind/memcheck-x86-linux'
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV),
dynamically linked,
BuildID[sha1]=0xdd994ac2a32bece59363fadc2003e49a1b8c3539,
stripped
/usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so:
broken symbolic link to `../..
/lib/valgrind/vgpreload_memcheck-x86-linux.so'
$
|
Let us continue the expedition and find out what other Valgrind
packages are available that could possibly bring in the missing
files
that those symbolic links are pointing to:
$
$ yum list valgrind*
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
valgrind.x86_64
1:3.8.1-15.fc19
installed
Available Packages
valgrind.i686
1:3.8.1-15.fc19
fedora
valgrind-devel.i686
1:3.8.1-15.fc19
fedora
valgrind-devel.x86_64
1:3.8.1-15.fc19
fedora
valgrind-openmpi.x86_64
1:3.8.1-15.fc19
fedora
$
|
It is evident that we have 64-bit Valgrind package installed on the
system, and 32-bit counter part of the Valgrind package is available
(but
not installed on the system).
The solution to my problem seems within arm's reach -
Install
the package valgrind.i686
But I want to confirm two things before I install that package.
(a) valgrind.i686 will bring in (at least) those two missing files
(b) valgrind.i686 will not overwrite important valgrind binaries
with 32-bit versions and break testing of my 64-bit executables.
Let us confirm (a) first:
$
$ repoquery -l valgrind.i686 | grep memcheck
/usr/lib/valgrind/memcheck-x86-linux
/usr/lib/valgrind/vgpreload_memcheck-x86-linux.so
$
|
Sure, the targets of those two broken symbolic link matches the path
and file names of these two files.
That confirms that valgrind.i686 will bring in missing files.
Now, let us confirm (b):
$
$ repoquery -l valgrind.x86_64 | grep bin
/usr/bin/callgrind_annotate
/usr/bin/callgrind_control
/usr/bin/cg_annotate
/usr/bin/cg_diff
/usr/bin/cg_merge
/usr/bin/ms_print
/usr/bin/valgrind
/usr/bin/valgrind-listener
/usr/bin/vgdb
$
$ repoquery -l valgrind.i686 | grep bin
/usr/bin/callgrind_annotate
/usr/bin/callgrind_control
/usr/bin/cg_annotate
/usr/bin/cg_diff
/usr/bin/cg_merge
/usr/bin/ms_print
/usr/bin/valgrind
/usr/bin/valgrind-listener
/usr/bin/vgdb
$
$ repoquery -l valgrind.x86_64 | grep bin | xargs file
/usr/bin/callgrind_annotate: Perl script, ASCII text
executable
/usr/bin/callgrind_control: Perl script,
ASCII text executable
/usr/bin/cg_annotate:
Perl script, ASCII text executable
/usr/bin/cg_diff:
Perl script, ASCII text executable
/usr/bin/cg_merge:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x5228b41ce865c922a7948698c40a6ed1b7c7cf1c,
stripped
/usr/bin/ms_print:
Perl script, ASCII text executable
/usr/bin/valgrind:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x19186524e13c9eaaacdc8bec0144b3d82bb0800a,
stripped
/usr/bin/valgrind-listener: ELF
64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x5c9a1aa7055c71fcb7cab739509940deb4594464,
stripped
/usr/bin/vgdb:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0xcbcdb0a010739b8ebfbff0abd508683987646c30,
stripped
$
|
That seems like an issue.
There are duplicate executable files between the 64-bit and 32-bit
versions, four of which are binary files.
I estimated - at worst, 32-bit package would overwrite these binary
files and, as anticipated, break testing of 64-bit executables; but
the damage would be easily reversible with two steps: (a) Uninstall
valgrind.i686 (b) Reinstall valgrind.x86_64
With that I was ready to plunge on with a plan - If things
broke, I would to make copy of those two symbolic link's target
files, uninstall 32-bit package, reinstall 64-bit package, and
finally manually restore broken symbolic links to fix the issue I
originally started with.
So I did (
sudo yum install valgrind.i686), and as it happens
so, valgrind.i686
does not overwrite 64-bit executables.
$
$ rpm -ql valgrind.i686 | grep bin | xargs file
/usr/bin/callgrind_annotate: Perl script, ASCII text
executable
/usr/bin/callgrind_control: Perl script, ASCII text
executable
/usr/bin/cg_annotate:
Perl script, ASCII text executable
/usr/bin/cg_diff:
Perl script, ASCII text executable
/usr/bin/cg_merge:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x5228b41ce865c922a7948698c40a6ed1b7c7cf1c,
stripped
/usr/bin/ms_print:
Perl script, ASCII text executable
/usr/bin/valgrind:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0x19186524e13c9eaaacdc8bec0144b3d82bb0800a,
stripped
/usr/bin/valgrind-listener: ELF 64-bit LSB
executable, x86-64, version 1 (SYSV), dynamically
linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0x5c9a1aa7055c71fcb7cab739509940deb4594464,
stripped
/usr/bin/vgdb:
ELF 64-bit LSB executable, x86-64, version 1
(SYSV), dynamically linked (uses shared libs), for
GNU/Linux 2.6.32,
BuildID[sha1]=0xcbcdb0a010739b8ebfbff0abd508683987646c30,
stripped
$
|
I can not explain this behaviour at this stage without looking into
.rpm packaging script logic of this package.
Let us see if rest is as expected. Let us confirm that broken
symbolic links are repaired:
$
$ rpm -ql valgrind.x86_64 | grep memcheck | xargs file
/usr/lib64/valgrind/memcheck-amd64-linux:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
statically linked,
BuildID[sha1]=0x82b75aa9e8494556aa050869e5e7ed20e9522137,
stripped
/usr/lib64/valgrind/memcheck-x86-linux:
symbolic link to `../../lib/valgrind/memcheck-x86-linux'
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so: ELF
64-bit LSB shared object, x86-64, version 1 (SYSV),
dynamically linked,
BuildID[sha1]=0xdd994ac2a32bece59363fadc2003e49a1b8c3539,
stripped
/usr/lib64/valgrind/vgpreload_memcheck-x86-linux.so:
symbolic link to
`../../lib/valgrind/vgpreload_memcheck-x86-linux.so'
$
|
Sure they are.
Now, like you I was curious to see if valgrind could test 32-bit
executables now (without breaking 64-bit executables tests):
$
$ valgrind ./testme32
==7914== Memcheck, a memory error detector
==7914== Copyright (C) 2002-2012, and GNU GPL'd, by Julian
Seward et al.
==7914== Using Valgrind-3.8.1 and LibVEX; rerun with -h
for copyright info
==7914== Command: ./testme32
==7914==
==7914==
==7914== HEAP SUMMARY:
==7914== in use at exit: 0 bytes
in 0 blocks
==7914== total heap usage: 0 allocs, 0 frees,
0 bytes allocated
==7914==
==7914== All heap blocks were freed -- no leaks are
possible
==7914==
==7914== For counts of detected and suppressed errors,
rerun with: -v
==7914== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 0 from 0)
$
$ valgrind ./testme64
==8176== Memcheck, a memory error detector
==8176== Copyright (C) 2002-2012, and GNU GPL'd, by Julian
Seward et al.
==8176== Using Valgrind-3.8.1 and LibVEX; rerun with -h
for copyright info
==8176== Command: ./testme64
==8176==
==8176==
==8176== HEAP SUMMARY:
==8176== in use at exit: 0 bytes
in 0 blocks
==8176== total heap usage: 0 allocs, 0 frees,
0 bytes allocated
==8176==
==8176== All heap blocks were freed -- no leaks are
possible
==8176==
==8176== For counts of detected and suppressed errors,
rerun with: -v
==8176== ERROR SUMMARY: 0 errors from 0 contexts
(suppressed: 2 from 2)
$
|
Good. Valgrind is now capable to test both: 64-bit as well as 32-bit
executables.
$
valgrind ./atail32_80 test.txt
<snip>
==13468==
==13468== HEAP SUMMARY:
==13468== in use at exit: 266
bytes in 10 blocks
==13468== total heap usage: 18 allocs, 8
frees, 623 bytes allocated
==13468==
==13468== LEAK SUMMARY:
==13468== definitely lost: 262 bytes in
9 blocks
<snip>
|
Testing my project binary with valgrind did catch
expected
memory leaks on the first run, along with couple of unexpected
memory
access related bugs (not shown here). Now that is going to keep me
busy for some time.
Leave any comments/suggestions below.