[Valgrind-users] Handling ioctl 0x30000001 in valgrind?

Discussion:

Carl Ponder

2015-01-29 03:28:34 UTC

I ran into this output from valgrind

==21898== Warning: noted but unhandled ioctl 0x30000001 with no
size/direction hints.
==21898== This could cause spurious value errors to appear.
==21898== See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
==21898== at 0x59C98C7: ioctl (in /lib64/libc-2.12.so)
==21898== by 0x61D148D: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x62B7B94: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x6271A6F: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x61AE21C: cuInit (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x414D89: __pgi_uacc_cuda_init (cuda_init.c:225)
==21898== by 0x408ACC: __pgi_uacc_enumerate (init.c:420)
==21898== by 0x408E64: __pgi_uacc_initialize (init.c:580)
==21898== by 0x405C09: __pgi_uacc_dataenterstart (dataenterstart.c:45)
==21898== by 0x404A19: MAIN_ (test_synccheck.F:177)
==21898== by 0x4027F3: main (in /home-2/cponder/Temp.064/synccheck)

I'd like to "gracefully" handle this ioctl in valgrind, and looked at
the instructions posted here

http://valgrind.org/docs/manual/dist.readme-missing.html

Grep'ing through the files in these directories

/usr/src/
/usr/include
/shared/apps/cuda/CUDA-v7.0.18
/shared/apps/rhel-6.2/tools/$USER/ValGrind-3.10.1

I can't find anything that maps the address 30000001.
Are these mappings only present for syscalls but not ioctl's?
Or do we need to install a development version of the /usr/src files?
Thanks,

Carl

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Carl Ponder

2015-01-31 15:07:24 UTC

Permalink

Tom Hughes

2015-01-31 15:37:24 UTC

Permalink

Post by Carl Ponder
Grep'ing through the files in these directories
/usr/src/
/usr/include
/shared/apps/cuda/CUDA-v7.0.18
/shared/apps/rhel-6.2/tools/$USER/ValGrind-3.10.1
I can't find anything that maps the address 30000001.
Are these mappings only present for syscalls but not ioctl's?
Or do we need to install a development version of the /usr/src files?

Well you won't normally find a raw ioctl number in the source because
they are normally constructed with the _IO. _IOR, _IOW or _IOWR macros.

Specifically, and assuming this is amd64-linux, that number appears to
correspond to:

_IO(0x3000, 1)

But in all honest, given you presumably have the source to libcuda, the
easiest thing will be just to look at that source to see what the ioctl
is that is being called!

Tom

--
Tom Hughes (***@compton.nu)
http://compton.nu/

Carl Ponder

2015-03-16 01:04:09 UTC

Permalink

*On 01/31/2015 07:07 AM, Carl Ponder wrote:*

Post by Carl Ponder
I ran into this output from valgrind
*==21898== Warning: noted but unhandled ioctl 0x30000001 with no
size/direction hints.*
==21898== This could cause spurious value errors to appear.
==21898== See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
==21898== at 0x59C98C7: ioctl (in /lib64/libc-2.12.so)
==21898== by 0x61D148D: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x62B7B94: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x6271A6F: ??? (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x61AE21C: cuInit (in /usr/lib64/libcuda.so.346.29)
==21898== by 0x414D89: __pgi_uacc_cuda_init (cuda_init.c:225)
==21898== by 0x408ACC: __pgi_uacc_enumerate (init.c:420)
==21898== by 0x408E64: __pgi_uacc_initialize (init.c:580)
==21898== by 0x405C09: __pgi_uacc_dataenterstart
(dataenterstart.c:45)
==21898== by 0x404A19: MAIN_ (test_synccheck.F:177)
==21898== by 0x4027F3: main (in /home-2/cponder/Temp.064/synccheck)

From following the instructions in

Post by Carl Ponder
http://valgrind.org/docs/manual/dist.readme-missing.html

I added this entry to the filecoregrind/m_syswrap/syswrap-linux.c :

10166 PRE(0x30000001)
10167 {
10168 PRINT("NVIDIA UvmInitialize ioctl");
10169 }

It didn't affect the output, though, I still see the same messages and
it looks like the abovePRINT statement didn't get executed.
I saw this message in two places in thevalgrind build log:

m_syswrap/syswrap-linux.c:10166: warning: no previous prototype for
âvgSysWrap_linux_0x30000001_beforeâ

The instructions also say this regarding system-calls:

Also, add it to the syscall_table[] array; use one of GENX_, GENXY
LINX_, LINXY, PLAX_, PLAXY.
GEN* for generic syscalls (in syswrap-generic.c), LIN* for linux
specific ones (in syswrap-linux.c) and PLA* for the platform
dependant ones (in syswrap-$(PLATFORM)-linux.c).
The *XY variant if it requires a PRE() and POST() function, and
the *X_ variant if it only requires a PRE()
function.

There isn't anysyscall_table in the syswrap-linux.c file, also I'm not
sure if this applies to ioctl's anyway.
So the obvious questions are (a) whether I put thePRE(0x30000001) macro
in the right file, and if there are other edits that I have to make to
the valgrind sources as well.
This particular ioctl doesn't take any arguments and only returns an
integer status value.
For now I just want to remove the warning-message fromvalgrind, after
that I'll look more closely at changes it might possibly make to the
global data state.
Thanks,

Carl Ponder

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Julian Seward

2015-03-17 08:44:05 UTC

Permalink

Post by Carl Ponder
10166 PRE(0x30000001)
10167 {
10168 PRINT("NVIDIA UvmInitialize ioctl");
10169 }

The right place to add it is PRE(sys_ioctl) and POST(sys_ioctl).

A more fundamental question is, are you running a kernel with NVidia-
specific hacks? What is this ioctl 0x30000001 ? What does it do? Is
it in the mainline linux kernel sources?

J

Tom Hughes

2015-03-17 10:05:32 UTC

Permalink

Post by Julian Seward

Post by Carl Ponder
10166 PRE(0x30000001)
10167 {
10168 PRINT("NVIDIA UvmInitialize ioctl");
10169 }

The right place to add it is PRE(sys_ioctl) and POST(sys_ioctl).
A more fundamental question is, are you running a kernel with NVidia-
specific hacks? What is this ioctl 0x30000001 ? What does it do? Is
it in the mainline linux kernel sources?

There were a couple of previous threads back in January but I don't we
ever really got to the bottom of that question...

Tom

--
Tom Hughes (***@compton.nu)
http://compton.nu/

Carl Ponder

2015-05-18 14:12:16 UTC

Permalink

*On 01/28/2015 09:27 PM, Carl Ponder wrote:*

*On 03/17/2015 03:44 AM, Julian Seward wrote: *

Post by Carl Ponder
A more fundamental question is, are you running a kernel with NVidia-specific hacks? What is this ioctl 0x30000001 ? What does it do? Is it in the mainline linux kernel sources?

I had to postpone this thread while I was working on other things, but
am back looking atvalgrind output again.
We're running standard Centos 6.6 but loading-in custom kernel extensions.
The ioctl call is to UvmInitialize, which maps the GPU memory into the
process address-space.

Post by Carl Ponder
The right place to add it is PRE(sys_ioctl) and POST(sys_ioctl).

I tried inserting print-statements into this function in the
filecoregrind/m_syswrap/syswrap-linux.c

5406 PRE(sys_ioctl)
5407 {
5408 *flags |= SfMayBlock;
5409
5410 ARG2 = (UInt)ARG2;
5411
*5412 PRINT("ioctl ARG1=0x%lx\n", (unsigned long) ARG1 );**
**5413 PRINT("ioctl ARG2=0x%lx\n", (unsigned long) ARG2 );**
**5414 PRINT("ioctl ARG3=0x%lx\n", (unsigned long) ARG3 );*

but didn't see any output, it looks like this function doesn't get
called on ioctl0x30000001.
The Warning at the top looks like it came from the
functionPRE_unknown_ioctl in the filecoregrind/m_syswrap/syswrap-generic.c

3488 for (i = 0; i <
sizeof(unknown_ioctl)/sizeof(unknown_ioctl[0]); i++) {
3489 if (unknown_ioctl[i] == request)
3490 break;
3491 if (unknown_ioctl[i] == 0) {
3492 unknown_ioctl[i] = request;
3493 moans--;
*3494 VG_(umsg)("Warning: noted but unhandled ioctl
0x%lx"**
**3495 " with no size/direction hints.\n",
request); *
3496 VG_(umsg)(" This could cause spurious value
errors to appear.\n");
3497 VG_(umsg)(" See
README_MISSING_SYSCALL_OR_IOCTL for "
3498 "guidance on writing a proper
wrapper.\n" );
3499 //VG_(get_and_pp_StackTrace)(tid,
VG_(clo_backtrace_size));
3500 return;
3501 }
3502 }

So I still don't understand how to get it to handle 0x30000001, given
that it never gets to thePRE(sys_ioctl).
It must be getting intercepted somewhere higher in the call-chain and
passed down toPRE_unknown_ioctlinstead.
Can you tell me what else needs to be changed?
Thanks,

Carl

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Tom Hughes

2015-05-18 14:17:19 UTC

Permalink

Post by Carl Ponder
*On 01/28/2015 09:27 PM, Carl Ponder wrote:*

*On 03/17/2015 03:44 AM, Julian Seward wrote: *

Post by Carl Ponder
A more fundamental question is, are you running a kernel with NVidia-specific hacks? What is this ioctl 0x30000001 ? What does it do? Is it in the mainline linux kernel sources?

But this ioctl is implemented by that extra kernel module I assume?

Post by Carl Ponder

Post by Carl Ponder
The right place to add it is PRE(sys_ioctl) and POST(sys_ioctl).

You know PRINT only prints when --trace-syscalls=yes is used?

Post by Carl Ponder
The Warning at the top looks like it came from the
So I still don't understand how to get it to handle 0x30000001, given
that it never gets to thePRE(sys_ioctl).
It must be getting intercepted somewhere higher in the call-chain and
passed down toPRE_unknown_ioctlinstead.

Well you have clearly done something wrong, as unknown_ioctl is invoked
from the default clause in sys_ioctl, so it must be getting called.

Tom

--
Tom Hughes (***@compton.nu)
http://compton.nu/

Carl Ponder

2015-05-19 16:26:34 UTC

Permalink

*On 05/18/2015 09:12 AM, Carl Ponder wrote:*

Post by Carl Ponder
I tried inserting print-statements into this function in the
filecoregrind/m_syswrap/syswrap-linux.c
5406 PRE(sys_ioctl)
5407 {
5408 *flags |= SfMayBlock;
5409
5410 ARG2 = (UInt)ARG2;
5411
*5412 PRINT("ioctl ARG1=0x%lx\n", (unsigned long) ARG1 );**
**5413 PRINT("ioctl ARG2=0x%lx\n", (unsigned long) ARG2 );**
**5414 PRINT("ioctl ARG3=0x%lx\n", (unsigned long) ARG3 );*
but didn't see any output, it looks like this function doesn't get
called on ioctl0x30000001.

*On 05/18/2015 09:17 AM, Tom Hughes wrote:*

Post by Carl Ponder
You know PRINT only prints when --trace-syscalls=yes is used?

Ok got it.... here is what worked, in the
file*coregrind/m_syswrap/syswrap-linux.c*:

5406 PRE(sys_ioctl)
5407 {
5408 *flags |= SfMayBlock;
5409
5410 ARG2 = (UInt)ARG2;
5411
5412 // We first handle the ones that don't use ARG3 (even as a
5413 // scalar/non-pointer argument).
5414 switch (ARG2 /* request */) {
.....
* 5467 /* NVIDIA UvmInitialize */
5468 case 0x30000001:
5469 PRINT("sys_ioctl ( %ld, 0x%lx, 0x%lx )",ARG1,ARG2,ARG3);
5470 PRE_REG_READ3(long, "ioctl",
5471 unsigned int, fd, unsigned int, request,
unsigned long, arg);
5472 return;*
5473
5474 default:

So it shuts off the warning message

*==21898== Warning: noted but unhandled ioctl 0x30000001 with no
size/direction hints.*
==21898== This could cause spurious value errors to appear.
==21898== See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.

but doesn't do anything to manage the memory state.
The output I'm getting from--trace-syscalls=yes is

SYSCALL[20867,1](16) *sys_ioctl ( 38, 0x30000001, 0x0 *) --> [async]
...
SYSCALL[20867,1](16) ... [async] --> Success(0x0:0x0)

so it doesn't look like any memory-range is being passed in.
I'm checking with our engineers to see if there is any special
processing that needs to happen here, and we'll also have to watch and
see if any other NVIDIA-specific calls are happening.

One question to you: does it make sense to be hard-coding the
*0x30000001* case into the generalioctl handler, or should there be an
NVIDIA-specific file in the coregrind/m_syswrap directory?
Thanks,

Carl

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Tom Hughes

2015-05-19 23:34:34 UTC

Permalink

Post by Carl Ponder
So it shuts off the warning message
*==21898== Warning: noted but unhandled ioctl 0x30000001 with no
size/direction hints.*
==21898== This could cause spurious value errors to appear.
==21898== See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
but doesn't do anything to manage the memory state.
The output I'm getting from--trace-syscalls=yes is
SYSCALL[20867,1](16) *sys_ioctl ( 38, 0x30000001, 0x0 *) --> [async]
...
SYSCALL[20867,1](16) ... [async] --> Success(0x0:0x0)
so it doesn't look like any memory-range is being passed in.
I'm checking with our engineers to see if there is any special
processing that needs to happen here, and we'll also have to watch and
see if any other NVIDIA-specific calls are happening.

Yes you will need to find out if it reads/writes any memory, though it
looks unlikely in this case.

Post by Carl Ponder
One question to you: does it make sense to be hard-coding the
*0x30000001* case into the generalioctl handler, or should there be an
NVIDIA-specific file in the coregrind/m_syswrap directory?
Thanks,

Well I'm not sure there's much precedent as we don't normally take
system calls that aren't in the upstream kernel. There may be a few ARM
ones as precedent.

Tom

--
Tom Hughes (***@compton.nu)
http://compton.nu/

Julian Seward

2015-06-11 15:34:24 UTC

Permalink

One question to you: does it make sense to be hard-coding the *0x30000001*
case into the generalioctl handler, or should there be an NVIDIA-specific
file in the coregrind/m_syswrap directory?

There is already a mechanism for dealing with proprietary GPU ioctls,
because of the need to run on phones with various GPUs. Try grepping
the sources (trunk) for "KernelVariant_android_gpu_adreno3xx" and follow
your nose from there.

Presumably you'll need to add a KernelVariant_<blah>_nvidia_<blah> enum
and a --kernel-variant= sub-flag. Have a look at m_main.c:714.

J

------------------------------------------------------------------------------