We have a recurring problem in our crash analysis system, where remote users get disconnected and crash starts endlessly looping trying to write to stdout. An strace of a recent instance is looping on: write(1, " JIFFIES\n", 10) = -1 EIO (Input/output error) but that isn't always the output string. this is a problem in out shared environment because the orphaned crash tasks eat up the CPUs, and we don't have the privilege to kill each others tasks. thanks, --Guy

On 02/21/2012 10:44 AM, Dave Anderson wrote: > > > ----- Original Message ----- >> We have a recurring problem in our crash analysis system, where remote users >> get disconnected and crash starts endlessly looping trying to write to stdout. >> An strace of a recent instance is looping on: >> >> write(1, " JIFFIES\n", 10) = -1 EIO (Input/output error) >> >> but that isn't always the output string. >> >> this is a problem in out shared environment because the orphaned crash tasks >> eat up the CPUs, and we don't have the privilege to kill each others tasks. >> >> thanks, >> --Guy > > Hmmm, upon initial glance, this seemed to be related to the crash-5.0.2 > fix that you guys reported: > > - Fix to prevent a crash session that is run over a network connection > that is killed/removed from going into 100% cpu-time loop. Without > the patch, the behavior of the built-in readline() library call in > gdb-7.0 has changed such that the function returns when the EOF is > encountered on /dev/tty, and the crash session goes into an endless > loop; whereas in gdb-6.1, the readline() call never returns because > the crash session gets killed while running in the library code. > (anderson(a)redhat.com) > > But if the orphaned task is repetetively writing the same thing, it > would never get to the next readline() call, where it would kill > itself. Taking your example, the "JIFFIES" write() is part of a "timer" > command, but I'm trying to understand how/why the command is not just > completing a series of (failed) fprintf's, and then falling into > the next readline() -- where it should kill itself? By any chance > was the remote caller doing a "repeat" command on the live system, > or something like that? (sounds doubtful since you'd have to have > root privileges to do that...) > This is not a live system. This is the setup where we analyze vmcores sent in by our customers. I don't understand how it happens either, unless for some reason fprintf is re-trying the failed write(). This is not the only failure scenario. I just saw another one repeating on this sequence: rt_sigaction(SIGFPE, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, 8) = 0 rt_sigreturn(0x8) = -1 ENETDOWN (Network is down) --- SIGFPE (Floating point exception) @ 0 (0) --- rt_sigaction(SIGFPE, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, 8) = 0 rt_sigreturn(0x8) = -1 ENETDOWN (Network is down) --- SIGFPE (Floating point exception) @ 0 (0) --- rt_sigaction(SIGFPE, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, {0x550ff0, [FPE], SA_RESTORER|SA_RESTART, 0x370ac302d0}, 8) = 0 rt_sigreturn(0x8) = -1 ENETDOWN (Network is down) --- SIGFPE (Floating point exception) @ 0 (0) --- Perhaps it isn't a crash program issue at all. Maybe it's at the system library level.