[Valgrind-users] [vgdb - gdbserver] FIFO name mismatch...

[PLUG GULP]

Hi,

I am new to valgrind. I am trying to debug my web application
(testwtdbg.wt) using valgrind. But there is a mismatch between the
FIFO name valgrind generates and the FIFO vgdb tries to use. I am
using a FastCGI (Apache's mod_fcgid) application developed using Wt
(www.webtoolkit.eu). Valgrind is not directly invoked by me. Instead I
specify the valgrind command and its options in a Wt configuration
file (wt_config.xml) and Wt then prepends this command to the
application before launching it using execve system call. Here are the
commands issued to use valgrind and the corresponding output:

*A]
A.1. Command to start valgrind and web-application "testwtdbg.wt"
(this command is issued, using system call execve, by Wt framework in
the context of Apache webserver):

/usr/local/bin/valgrind -v --trace-children=yes --vgdb=full
--vgdb-error=0 /var/www/html/plug/docroot/testwtdbg.wt <<additional
parameters of the application/Wt>>


A.2. Log messages (Apache log messages): [***@playground build]# cat
/var/log/httpd/error_log

[Sat Apr 14 13:15:51 2012] [notice] SELinux policy enabled; httpd
running as context unconfined_u:system_r:httpd_t:s0
[Sat Apr 14 13:15:51 2012] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
[Sat Apr 14 13:15:51 2012] [notice] Digest: generating secret for
digest authentication ...
[Sat Apr 14 13:15:51 2012] [notice] Digest: done
[Sat Apr 14 13:15:51 2012] [notice] Apache/2.2.22 (Unix) DAV/2
mod_fcgid/2.3.6 configured -- resuming normal operations
[2012-Apr-14 13:16:35.383695] 6022 - [info] "WServer/wtfcgi:
initializing relay server"
[2012-Apr-14 13:16:35.387222] 6022 - [info] "config: reading Wt config
file: /etc/wt/wt_config.xml (location =
'/var/www/html/plug/docroot/testwtdbg.wt')"
[2012-Apr-14 13:16:35.389108] 6022 - [info] "wtfcgi: reading FastCGI
stream from stdin"
[2012-Apr-14 13:16:35.398416] 6022 - [info] "wtfcgi: spawned dedicated
process for ZHUb9FY39csyCLdY: pid=6164"
==6164== Memcheck, a memory error detector
==6164== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==6164== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==6164== Command: /var/www/html/plug/docroot/testwtdbg.wt client
ZHUb9FY39csyCLdY
==6164==
--6164-- Valgrind options:
--6164-- -v
--6164-- --trace-children=yes
--6164-- --vgdb=full
--6164-- --vgdb-error=0
--6164-- Contents of /proc/version:
--6164-- Linux version 3.3.1-3.fc16.x86_64
(***@x86-12.phx2.fedoraproject.org) (gcc version 4.6.3 20120306
(Red Hat 4.6.3-2) (GCC) ) #1 SMP Wed Apr 4 18:08:51 UTC 2012
--6164-- Arch and hwcaps: AMD64, amd64-sse3
--6164-- Page sizes: currently 4096, max supported 4096
--6164-- Valgrind library directory: /usr/local/lib/valgrind
--6164-- Reading syms from /var/www/html/plug/docroot/testwtdbg.wt (0x400000)
--6164-- Reading syms from
/usr/local/lib/valgrind/memcheck-amd64-linux (0x38000000)
--6164-- object doesn't have a dynamic symbol table
--6164-- Reading syms from /lib64/ld-2.14.90.so (0x3d26200000)
--6164-- Reading suppressions file: /usr/local/lib/valgrind/default.supp
==6164== (action at startup) vgdb me ...
==6164== embedded gdbserver: reading from
/tmp/vgdb-pipe-from-vgdb-to-6164-by-???-on-???
==6164== embedded gdbserver: writing to
/tmp/vgdb-pipe-to-vgdb-from-6164-by-???-on-???
==6164== embedded gdbserver: shared mem
/tmp/vgdb-pipe-shared-mem-vgdb-6164-by-???-on-???
==6164==
==6164== TO CONTROL THIS PROCESS USING vgdb (which you probably
==6164== don't want to do, unless you know exactly what you're doing,
==6164== or are doing some strange experiment):
==6164== /usr/local/lib/valgrind/../../bin/vgdb --pid=6164 ...command...
==6164==
==6164== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==6164== /path/to/gdb /var/www/html/plug/docroot/testwtdbg.wt
==6164== and then give GDB the following command
==6164== target remote | /usr/local/lib/valgrind/../../bin/vgdb --pid=6164
==6164== --pid is optional if only one valgrind process is running
==6164==


*B]
B.1. Check the FIFOs created by valgrind gdbserver: [***@playground
build]# ls -la /tmp/

drwxrwxrwt. 37 root root 4096 Apr 14 14:13 .
dr-xr-xr-x. 22 root root 4096 Apr 13 00:18 ..
drwxrwxrwt. 2 root root 4096 Apr 13 00:30 .font-unix
drwxrwxrwt. 2 root root 4096 Apr 14 11:56 .ICE-unix
-rw-------. 1 root root 0 Apr 13 00:41 tmpDjwfid
prw-r--r--. 1 apache apache 0 Apr 14 13:18
vgdb-pipe-from-vgdb-to-6164-by-???-on-???
-rw-r--r--. 1 apache apache 48 Apr 14 13:18
vgdb-pipe-shared-mem-vgdb-6164-by-???-on-???
prw-r--r--. 1 apache apache 0 Apr 14 13:18
vgdb-pipe-to-vgdb-from-6164-by-???-on-???
-r--r--r--. 1 root root 11 Apr 14 11:56 .X0-lock
drwxrwxrwt. 2 root root 4096 Apr 14 11:56 .X11-unix
drwxrwxrwt. 2 root root 4096 Apr 13 00:30 .XIM-unix


*C]
C.1 Start GDB to connect to the valgrind gdbserver: [***@playground
build]# gdb /var/www/html/plug/docroot/testwtdbg.wt

GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/www/html/plug/docroot/testwtdbg.wt...done.
(gdb) target remote | vgdb --pid=6164
Remote debugging using | vgdb --pid=6164
syscall failed: No such file or directory
error opening /tmp/vgdb-pipe-shared-mem-vgdb-6164-by-plug-on-playground.fc16
shared memory file
Remote communication error. Target disconnected.: Connection reset by peer.
(gdb)


*D]
D.1 Check the FIFO information using vgdb: [***@playground build]#
vgdb -d -d -d help --pid=6164

1334405920.781999 searching pid in directory /tmp/ format
/tmp/vgdb-pipe-from-vgdb-to-
1334405920.782175 check_trial 0
1334405920.782662 trying /tmp/.X0-lock
1334405920.783015 trying /tmp/vgdb-pipe-shared-mem-vgdb-6164-by-???-on-???
1334405920.786167 trying /tmp/.X11-unix
1334405920.786233 trying /tmp/tmpDjwfid
1334405920.786660 trying /tmp/.font-unix
1334405920.786901 trying /tmp/.ICE-unix
1334405920.787175 trying /tmp/vgdb-pipe-from-vgdb-to-6164-by-???-on-???
1334405920.787228 trying /tmp/vgdb-pipe-from-vgdb-to-6164-by-???-on-???
1334405920.789632 trying /tmp/.XIM-unix
1334405920.792089 trying /tmp/vgdb-pipe-to-vgdb-from-6164-by-???-on-???
1334405920.792142 trying /tmp/vgdb-pipe-to-vgdb-from-6164-by-???-on-???
1334405920.792265 vgdb: using
/tmp/vgdb-pipe-from-vgdb-to-6164-by-plug-on-playground.fc16
/tmp/vgdb-pipe-to-vgdb-from-6164-by-plug-on-playground.fc16
/tmp/vgdb-pipe-shared-mem-vgdb-6164-by-plug-on-playground.fc16
syscall failed: No such file or directory
error opening /tmp/vgdb-pipe-shared-mem-vgdb-6164-by-plug-on-playground.fc16
shared memory file

[Philippe Waroquiers]

Thanks for the precise information you have put in your question,
it helps to see what is going on.

==6164== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-6164-by-???-on-???
==6164== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-6164-by-???-on-???
==6164== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-6164-by-???-on-???

The above is the FIFOs and shared mem created by the Valgrind gdbserver.
They are created in the environment in which valgrind is launched.

[Plug Gulp]

On Sat, Apr 14, 2012 at 4:52 PM, Philippe Waroquiers
<***@skynet.be> wrote:

<snip>

The FIFOs and shared mem used by vgdb are based on the user name and the
host name.
 First try the env variable LOGNAME
 if not set, try the env variable USER
 if still not set, then take ???.
 HOST
 then HOSTNAME
 then ???
You should start your gdb and vgdb in the same "context/environment" as the
gdbserver
you are trying to connect to. Then both the Valgrind gdbserver and vgdb
should agree
on user and host, and use the same vgdb-pipe files.
If you cannot do that for one reason or another, you might try the following
  export LOGNAME="???"
  export HOST="???"
and then launch gdb+vgdb.
If you launch your gdb/vgdb under another account, the launched vgdb will
not have
rw access to the vgdb-pipe* files. To change the permissions, you will have
to one way or another let the apache user either chmod a+rw these files
or else use apache user to launch gdb and vgdb
(or maybe use root ?).

I tried starting apache webserver and gdb from root login. Did not
help, because the valgrind is still launched from under apache user.
So I set the LOGNAME and HOST to "???" as suggested above and started
the webserver and gdb from root login. I also changed the permissions
of the FIFOs using a+rw. Here is the output of the commands:

*A]

A.1 Valgrind (Apache log message) - Note that the error messages shown
in the log start to appear after I connect to valgrind using gdb+vgdb:

[***@playground build]$ sudo tail -f /var/log/httpd/error_log
[Sat Apr 14 22:31:20 2012] [notice] SELinux policy enabled; httpd
running as context unconfined_u:system_r:httpd_t:s0
[Sat Apr 14 22:31:20 2012] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
[Sat Apr 14 22:31:20 2012] [notice] Digest: generating secret for
digest authentication ...
[Sat Apr 14 22:31:20 2012] [notice] Digest: done
[Sat Apr 14 22:31:20 2012] [notice] Apache/2.2.22 (Unix) DAV/2
mod_fcgid/2.3.6 configured -- resuming normal operations
[2012-Apr-14 22:36:07.303928] 13140 - [info] "WServer/wtfcgi:
initializing relay server"
[2012-Apr-14 22:36:07.306978] 13140 - [info] "config: reading Wt
config file: /etc/wt/wt_config.xml (location =
'/var/www/html/plug/docroot/testwtdbg.wt')"
[2012-Apr-14 22:36:07.308753] 13140 - [info] "wtfcgi: reading FastCGI
stream from stdin"
[2012-Apr-14 22:36:07.319148] 13140 - [info] "wtfcgi: spawned
dedicated process for cMbPKxS2lzXyceXq: pid=13151"
==13151== Memcheck, a memory error detector
==13151== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==13151== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==13151== Command: /var/www/html/plug/docroot/testwtdbg.wt client
cMbPKxS2lzXyceXq
==13151==
--13151-- Valgrind options:
--13151-- -v
--13151-- --trace-children=yes
--13151-- --vgdb=full
--13151-- --vgdb-error=0
--13151-- Contents of /proc/version:
--13151-- Linux version 3.3.1-3.fc16.x86_64
(***@x86-12.phx2.fedoraproject.org) (gcc version 4.6.3 20120306
(Red Hat 4.6.3-2) (GCC) ) #1 SMP Wed Apr 4 18:08:51 UTC 2012
--13151-- Arch and hwcaps: AMD64, amd64-sse3
--13151-- Page sizes: currently 4096, max supported 4096
--13151-- Valgrind library directory: /usr/local/lib/valgrind
--13151-- Reading syms from /var/www/html/plug/docroot/testwtdbg.wt (0x400000)
--13151-- Reading syms from
/usr/local/lib/valgrind/memcheck-amd64-linux (0x38000000)
--13151-- object doesn't have a dynamic symbol table
--13151-- Reading syms from /lib64/ld-2.14.90.so (0x3d26200000)
--13151-- Reading suppressions file: /usr/local/lib/valgrind/default.supp
==13151== (action at startup) vgdb me ...
==13151== embedded gdbserver: reading from
/tmp/vgdb-pipe-from-vgdb-to-13151-by-???-on-???
==13151== embedded gdbserver: writing to
/tmp/vgdb-pipe-to-vgdb-from-13151-by-???-on-???
==13151== embedded gdbserver: shared mem
/tmp/vgdb-pipe-shared-mem-vgdb-13151-by-???-on-???
==13151==
==13151== TO CONTROL THIS PROCESS USING vgdb (which you probably
==13151== don't want to do, unless you know exactly what you're doing,
==13151== or are doing some strange experiment):
==13151== /usr/local/lib/valgrind/../../bin/vgdb --pid=13151 ...command...
==13151==
==13151== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==13151== /path/to/gdb /var/www/html/plug/docroot/testwtdbg.wt
==13151== and then give GDB the following command
==13151== target remote | /usr/local/lib/valgrind/../../bin/vgdb --pid=13151
==13151== --pid is optional if only one valgrind process is running
==13151==
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc
==13151== putpkt(write) error: no write_remote_desc



*B]

B.1 The read-write permissions of the FIFOs. I update the permissions
after valgrind starts but before gdb+vgdb is launched.

[***@playground build]$ ls -la /tmp/vgdb*
prw-rw-rw-. 1 apache apache 0 Apr 14 22:36
/tmp/vgdb-pipe-from-vgdb-to-13151-by-???-on-???
-rw-rw-rw-. 1 apache apache 48 Apr 14 22:36
/tmp/vgdb-pipe-shared-mem-vgdb-13151-by-???-on-???
prw-rw-rw-. 1 apache apache 0 Apr 14 22:36
/tmp/vgdb-pipe-to-vgdb-from-13151-by-???-on-???


*C]

C.1 The output of gdb+vgdb (Note that I also tried setting the
"remotetimeout" attribute of gdb to very large value as suggested in
README_DEVELOPERS file. But the result is the same except that no
error is displayed on gdb console and the control just waits (stalls?)
after printing the message "Remote debugging using | vgdb --pid=13151"
and does not return to gdb prompt. On the valgrind side though the
same error, "putpkt(write) error: no write_remote_desc", is displayed
as before except there is just one error rather than same error
printed multiple times)


[***@playground build]# gdb /var/www/html/plug/docroot/testwtdbg.wt
GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/www/html/plug/docroot/testwtdbg.wt...done.
(gdb) show env LOGNAME
LOGNAME = ???
(gdb) show env HOST
HOST = ???
(gdb) target remote | vgdb --pid=13151
Remote debugging using | vgdb --pid=13151
relaying data between gdb and process 13151
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Malformed response to offset query, timeout
(gdb)


*D]
D.1 gdb, valgrind, apache and linux distro versions I am using:

[***@playground build]# gdb --version
GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.


[***@playground build]# valgrind --version
valgrind-3.7.0


[***@playground build]$ uname -a
Linux playground.fc16 3.3.1-3.fc16.x86_64 #1 SMP Wed Apr 4 18:08:51
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


Apache version is 2.2.22 and mod_fcgid is 2.3.6


*E]
E.1 The output of `vgdb -d -d -d help --pid=13151` command is a
continuous non-stop flow of following message:

written_by_vgdb_before_sleep 1 seen_by_valgrind_before_sleep 1



Does valgrind ignore, for security reasons, the FIFOs with universal
write access? I think I read something to this regard but am not able
to recall where!
Also, I am not able to login as user "apache" and instead have to
login as root. But all this has not helped me debug using valgrind a
process under apache's context. The constraint that gdb, vgdb and
gdbserver all should start under the same context/environment is
proving to be quite restrictive.


Thanks and regards,

~Plug

Hope this helps ...
Philippe

[Philippe Waroquiers]

Post by Plug Gulp
I tried starting apache webserver and gdb from root login. Did not
help, because the valgrind is still launched from under apache user.
So I set the LOGNAME and HOST to "???" as suggested above and started
the webserver and gdb from root login. I also changed the permissions

Here is another solution to try :
* set the needed env var (LOGNAME, HOST)
* make a copy of the vgdb executable
* change owner of this vgdb file to be the user apache
* set the user and group id bit
(must be root or apache for that, I guess)
chmod u+s vgdb
chmod g+s vgdb

The above worked for me to then connect a root gdb+this vgdb
to a Valgrind running under my user (I replaced apache
by my own user)

3 notes:
1. once you have a "apache" vgdb, I believe root user should not be
needed anymore. You should be able to use your own user, as
the launched vgdb will "belong" to apache.
2. with the above hack, it seems some "ptrace" syscall done
by vgdb are still not authorised. So, you might have
to disable the ptrace things in vgdb using --max-invoke-ms=0
(see description and consequences in Valgrind manual).
3. IMPORTANT: of course, anybody else having access to this "apache"
vgdb can then connect to any Valgrind gdbserver belonging to
apache.
In other words, other users on the same computer have
"full" debuggability of all your apache Valgrind gdbserver
as long as this "apache" vgdb copy exists.

Post by Plug Gulp

Hope this helps ...

Hope this (really) helps (now) ...

Philippe