Here is an update to the summary I sent to SunManagers on January, 15, 1997.
This new summary has two sections. The first one contains my original problem and
the first summary I sent to SunManagers, including all the suggestions I got and
my comments on them.
This first summary generated strong feedback from some people who didn't agree with some
of the suggestions included in it and, above all, with my conclusions regarding the
whole issue. This feedback is also included in the first section.
After this I decided to create a short mailing list with all the people that had shown
some interest in this problem and send them additional details.
The second section of this summary includes the additional messages exchanged between
me and the following people:
Alex Finkel <afinkel@pfn.com>
Jim Harmon <jim@telecnnct.com>
Rich Kulawiec <rsk@itw.com>
Jacques Rall <jacques.rall@za.eds.com>
Pell Emanuelsson <pell@lysator.liu.se>
Eric Anderson <keric@fv.com>
robin.landis@imail.exim.gov
Kevin.Sheehan@uniq.com.au (Kevin Sheehan)
"Paul O'Donnell" <pod@morgan.com>
cgo@dsi.elis.fr (Cyril GODON)
Torsten Metzner <tom@plato.uni-paderborn.de>
"Feeney, Tim" <Tim.Feeney@FMR.COM>
john benjamins <johnb@Soliton.COM>
Let me warn you that I have not found what the problem was with the scripts. After the
problem appeared I have made several additional tests trying to reproduce it and was
never able to make it happen. As the scripts exhibiting the problem were being created
and executed by a database named "Mapper", installed on a system that is being administered
by a friend of mine maybe, as some people suggested, the problem had something to do with
there being more than one instance of each script in execution at the same time.
I have not had enough time to check this on the system actually running the Mapper database
therefore I am unable to provide further information.
My special thanks go to Paul O'Donnell for still showing interest after all this time and
suggesting that all in all it would still be a good idea to post a second summary even
if I had no definite conclusions regarding the subject.
Thank you all for your help and support!
Fernando Dias
Ok, here are the two sections. Please check them both and draw your own conclusions...
-- FIRST SECTION - Original problem - My first summary - The replies I got to my summary
>From sun-managers-relay@ra.mcs.anl.gov Wed Jan 15 19:35 GMT 1997
Date: Wed, 15 Jan 1997 16:54:12 GMT
From: Sun Manager Account <sunmanager@argos.silvac.pt>
To: sun-managers@ra.mcs.anl.gov
Subject: SUMMARY: Weird script behaviour in Enterprise 4000
Here is the summary to the question I posted a a couple of days ago...
The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and,
in my opinion, would clearly explain why the script ends up not doing what it
was supposed to do.
Please check his message, which is included just after my original message.
All in all I got four replies, which are included below.
This is my original message:
> From sun-managers-relay@ra.mcs.anl.gov Mon Jan 13 19:34 GMT 1997
> Date: Mon, 13 Jan 1997 18:13:39 GMT
> From: Sun Manager Account <sunmanager@argos.silvac.pt>
> To: sun-managers@ra.mcs.anl.gov
> Subject: Weird script behaviour in Enterprise 4000
>
> Hello SMs,
>
> Can somebody have a look at this and please explain to me what is going on here?
>
> I have made a very simple shell script that does something similar to what is shown below:
>
> while (some_condition) do
> touch file[var]
> rm file[var]
> increment [var]
> done
>
> This can also be done with a braindead shell script that goes something like:
>
> touch file1
> rm file1
> [repeat the two lines above X number of times, substituting "file1" for "file2", etc, etc.
>
>
> If you execute one of these scripts inside an empty directory, after it finishes running
> you should end up with the same empty directory you started with.
> The problem is that sometimes, depending on the load of the server at the time the script was
> running, some of the files created by the script will be left in the directory.
>
> We are testing this on a Sun Enterprise 4000 with 6 CPUs and 1Gbyte RAM...
>
> The only thing that occurs to me to explain this behaviour is that sometimes the remove
> command will be executed and fail because the system still is in the process of creating
> the file.
>
> Could this be due to the fact that the system is MultiProcessor, and could therefore
> be executing each command on a different CPU, which would lead, depending on load conditions,
> to the behaviour described above?
>
>
> Thanks in advance for any help,
>
> Fernando Dias
>
These are the replies I got. I have added my comments to clarify or correct some of them.
> From jim@telecnnct.com Mon Jan 13 21:04 GMT 1997
> Date: Mon, 13 Jan 1997 15:48:34 -0500
> From: Jim Harmon <jim@telecnnct.com>
> Mime-Version: 1.0
> To: Sun Manager Account <sunmanager@argos.silvac.pt>
> Subject: Re: Weird script behaviour in Enterprise 4000
> Content-Transfer-Encoding: 7bit
>
> In scripts, each command spawns a child to execute the process
> requested.
>
> So, in your example below, if the touch process goes to sleep before
> completion (due to system use--for example) and the rm process begins
> execution,but doesn't go to sleep, you'll have the two processes
> reversed in execution, leaving the touched file after the remove.
>
> The way to control that would be to write the script so that the rm
> can't execute UNTIL the touch is returned "complete".
>
> One way to do that would be to put an "if" statement around the rm, that
> checks to see if the file is found, and if not, repeats until it finds
> it.
>
> Let me know if you would like some suggestions on how to build the loop,
> I'm afraid that right now I'm trying to get a bunch of things done...
>
> Cool?
>
>
As I have said before I think Jim's reply makes a lot of sense and would justify
the script behaviour.
> From afinkel@pfn.com Mon Jan 13 20:04 GMT 1997
> X-Sender: afinkel@sunrah.pfn.com
> Date: Mon, 13 Jan 1997 14:54:57 -0500
> To: Sun Manager Account <sunmanager@argos.silvac.pt>
> From: Alex Finkel <afinkel@pfn.com>
> Subject: Re: Weird script behaviour in Enterprise 4000
> Mime-Version: 1.0
>
>
> You could try using the wait command in the script which will cause the
> shell to wait for the completion of previous jobs.
>
> The man page for wait(1) gives a good description .
>
> - Alex
>
>From the manual pages it looks like the wait(1) only waits for completion of jobs
that are run in the BACKGROUND. This is not the case with my script as each command
issued is run in the foreground.
> From rsk@itw.com Tue Jan 14 01:04 GMT 1997
> From: Rich Kulawiec <rsk@itw.com>
> Subject: Re: Weird script behaviour in Enterprise 4000
> To: sunmanager@argos.silvac.pt
> Date: Mon, 13 Jan 1997 19:28:51 -0500 (EST)
> X-Last-River: Brandywine
> X-Last-CD: The Nields, "Abigail"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
>
>
> That really shouldn't happen, because once the "touch" is executed, the file should exist.
>
> Have you run this with "sh -x" and or "sh -xv" to ensure that the [var] is getting updated
> correctly and in a matching fashion for each command that's run inside the script?
>
> >Could this be due to the fact that the system is MultiProcessor, and could therefore
> >be executing each command on a different CPU, which would lead, depending on load conditions,
> >to the behaviour described above?
>
> It darn well had not better not be, because if it is, it's an indication of a serious
> bug in Sun's multiprocessor implementation!
>
> ---Rsk
> Rich Kulawiec
> rsk@itw.com
>
And the last reply:
> From jacques.rall@za.eds.com Tue Jan 14 13:05 GMT 1997
> From: Jacques Rall <jacques.rall@za.eds.com>
> To: "'Sun Manager Account'" <sunmanager@argos.silvac.pt>
> Subject: RE: Weird script behaviour in Enterprise 4000
> Date: Tue, 14 Jan 1997 14:57:32 +0200
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
>
> The file is propably only in memory yet and didn't have time to do a
> 'sync'.
>
>From what I know the system 'syncs' by default every 60 seconds. If the files
residing only in memory could not be seen by other processes until each 'sync'
run we would get a lot of problems regarding interactivity and coordination
among processes. (e.g. a user writing a file to disk and having to wait until
the next 'sync' to be able to do further work on it)
Am I making sense here?
Regards to you all, and thanks for the help!
Fernando Dias
-- THE REPLIES I GOT TO MY FIRST SUMMARY WERE:
>From pell@lysator.liu.se Wed Jan 15 19:35 GMT 1997
Date: Wed, 15 Jan 1997 19:52:07 +0100 (MET)
From: Pell Emanuelsson <pell@lysator.liu.se>
To: Sun Manager Account <sunmanager@argos.silvac.pt>
Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000
MIME-Version: 1.0
Sorry, but the correct answer really came from Rich Kulawiec.
When the touch returns, the file is _guaranteed_ to exist (on that local
machine) even if not all the contents has been written to permanent
storage yet. So the rm should definitely work.
It might very well be that you found a bug in Solaris, though. Sun has
had bugs of that kind before. What kind of file system are you writing
to? UFS? NFS? LOFS?
Cheers - Pell
-- Lysator Computer Society | email: pell@lysator.liu.se Linkoping University | WWW home at: http://www.lysator.liu.se/~pell/ Sweden | Life was easier when I didn't have one.>From keric@fv.com Wed Jan 15 19:35 GMT 1997 Date: Wed, 15 Jan 1997 10:47:07 -0800 (PST) From: Eric Anderson <keric@fv.com> To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 MIME-Version: 1.0
> > Can somebody have a look at this and please explain to me what is going on here? > > > > I have made a very simple shell script that does something similar to what is shown below: > > > > while (some_condition) do > > touch file[var] > > rm file[var] > > increment [var] > > done > > > > This can also be done with a braindead shell script that goes something like: > > > > touch file1 > > rm file1 > > [repeat the two lines above X number of times, substituting "file1" for "file2", etc, etc.
what about 'touch <filename> && rm <filename>'?
The second half cannot execute until the first half is done..
--
--------------------------------------------------------------- Eric Anderson <keric@fv.com> First Virtual Holdings Unix System Administrator 11975 El Camino Real, Suite 200 <http://www.fv.com/> San Diego, CA 92130 ---------------------------------------------------------------
>From rsk@itw.com Wed Jan 15 19:35 GMT 1997 From: Rich Kulawiec <rsk@itw.com> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 To: sunmanager@argos.silvac.pt, jim@telecnnct.com Date: Wed, 15 Jan 1997 14:01:53 -0500 (EST) X-Last-River: Brandywine X-Last-CD: Duncan Shiek, "Duncan Shiek" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit
>Here is the summary to the question I posted a a couple of days ago... > >The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, >in my opinion, would clearly explain why the script ends up not doing what it >was supposed to do. > >Please check his message, which is included just after my original message. > >All in all I got four replies, which are included below. > >This is my original message: > >These are the replies I got. I have added my comments to clarify or correct some of them. > > >> From jim@telecnnct.com Mon Jan 13 21:04 GMT 1997 >> Date: Mon, 13 Jan 1997 15:48:34 -0500 >> From: Jim Harmon <jim@telecnnct.com> >> Mime-Version: 1.0 >> To: Sun Manager Account <sunmanager@argos.silvac.pt> >> Subject: Re: Weird script behaviour in Enterprise 4000 >> Content-Transfer-Encoding: 7bit >> >> In scripts, each command spawns a child to execute the process >> requested. >> >> So, in your example below, if the touch process goes to sleep before >> completion (due to system use--for example) and the rm process begins >> execution,but doesn't go to sleep, you'll have the two processes >> reversed in execution, leaving the touched file after the remove. >> >> The way to control that would be to write the script so that the rm >> can't execute UNTIL the touch is returned "complete". >> >> One way to do that would be to put an "if" statement around the rm, that >> checks to see if the file is found, and if not, repeats until it finds >> it.
If I understand what you are saying, you're wrong. On the other hand, it could be that I'm completely looney. Let me explain, and then let's figure it out.
If I do something like this:
#!/bin/csh
foreach i (1 2 3 4 5 6 7 8 9) touch hose.$i rm hose.$i end
Then what I'm wanting to happen is that the files hose.1, hose.2, etc. will be created and removed one at a time. This is similar enough to the originallly-stated problem that I can think whatever we say about it will apply to the original problem.
Now, the "touch" command will be executed by the shell fork()ing and then exec()ing, since that's how the shell runs commands. And since it's *not* in the background, that is, we ran it like this:
touch hose.$i
instead of like this:
touch hose.$i &
then we are guaranteed that the shell will wait() for for that command to finish and exit before it goes on to the next command. If the "touch" goes to sleep -- let's say because it's waiting for a system resource like disk or CPU or whatever -- then the shell *will still wait for it*. In other words, there is NO WAY that the "rm" should run until the "touch" has exit()ed, because the shell is supposed to guarantee sequential execution. In fact, that's what coded into the shell -- at least in the sh, csh, and ksh shells, which are the ones that I've been through the source code for.
(BTW, if we ran it in the background, with the "&", then the shell will not wait() for it, and desynchronization problems like the ones you're seeing can easily happen.)
This leaves other possibilities -- one is that this is happening on an NFS-mounted filesystem and something is amiss there. Another is that the disk read/write queue has gotten scrozzled and the the operations are not happening in the correct order. Another is that perhaps there is a kernel bug related to the multiprocessor code. (Early versions of Sequent's Dynix multiprocessor OS had similar bugs when we were beta'ing them 10 years ago.) And so on. But a correctly-executing shell should *never* do what's described above, i.e. fail to wait for a command to finish.
Rich Kulawiec rsk@itw.com
>From jim@telecnnct.com Wed Jan 15 21:34 GMT 1997 Date: Wed, 15 Jan 1997 15:44:51 -0500 From: Jim Harmon <jim@telecnnct.com> Mime-Version: 1.0 To: rsk@itw.com, sunmanager@argos.silvac.pt Cc: sunmanager@argos.silvac.pt Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 Content-Transfer-Encoding: 7bit
Rich,
I believe you're right...if we're talking about a single-CPU workstation or server.
I was thinking in terms of BG and threaded processes, rather then sequential FG processes when I answered the question.
However, strict FG processing would leave the original question unanswered, which was Fernando Dias' problem with sequential "hiccups" during system loading.
In his description he originally didn't state that he was running the script interactively at all times. (In his followup he did say something to that effect, so BG processing doesn't seem to be a factor.)
Since that's the case, what other possibilities are there that could cause a sequential write-delete script to skip the erase without error messages or warnings?
Taking into account the MultiProcessor (6 CPU) nature of the system, wouldn't the process manager tend to treat even sequential commands as individual threads if you didn't specifically limit the process(es) to a single CPU?
Fernando,
Have you included any instructions in your script to redirect STDERR to /dev/null or anything of that nature, and have you checked your system errorlog for any possibly related messages?
Rich Kulawiec wrote: > > >Here is the summary to the question I posted a a couple of days ago... > > > >The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, > >in my opinion, would clearly explain why the script ends up not doing what it > >was supposed to do. > > > >Please check his message, which is included just after my original message. > > > >All in all I got four replies, which are included below. > > > >This is my original message: > > > >These are the replies I got. I have added my comments to clarify or correct some of them. > > > > > >> From jim@telecnnct.com Mon Jan 13 21:04 GMT 1997 > >> Date: Mon, 13 Jan 1997 15:48:34 -0500 > >> From: Jim Harmon <jim@telecnnct.com> > >> Mime-Version: 1.0 > >> To: Sun Manager Account <sunmanager@argos.silvac.pt> > >> Subject: Re: Weird script behaviour in Enterprise 4000 > >> Content-Transfer-Encoding: 7bit > >> > >> In scripts, each command spawns a child to execute the process > >> requested. > >> > >> So, in your example below, if the touch process goes to sleep before > >> completion (due to system use--for example) and the rm process begins > >> execution,but doesn't go to sleep, you'll have the two processes > >> reversed in execution, leaving the touched file after the remove. > >> > >> The way to control that would be to write the script so that the rm > >> can't execute UNTIL the touch is returned "complete". > >> > >> One way to do that would be to put an "if" statement around the rm, that > >> checks to see if the file is found, and if not, repeats until it finds > >> it. > > If I understand what you are saying, you're wrong. On the other hand, > it could be that I'm completely looney. Let me explain, and then let's > figure it out. > > If I do something like this: > > #!/bin/csh > > foreach i (1 2 3 4 5 6 7 8 9) > touch hose.$i > rm hose.$i > end > > Then what I'm wanting to happen is that the files hose.1, hose.2, etc. > will be created and removed one at a time. This is similar enough > to the originallly-stated problem that I can think whatever we say > about it will apply to the original problem. > > Now, the "touch" command will be executed by the shell fork()ing > and then exec()ing, since that's how the shell runs commands. > And since it's *not* in the background, that is, we ran it like this: > > touch hose.$i > > instead of like this: > > touch hose.$i & > > then we are guaranteed that the shell will wait() for for that command > to finish and exit before it goes on to the next command. If the "touch" > goes to sleep -- let's say because it's waiting for a system resource > like disk or CPU or whatever -- then the shell *will still wait for it*. > In other words, there is NO WAY that the "rm" should run until the > "touch" has exit()ed, because the shell is supposed to guarantee sequential > execution. In fact, that's what coded into the shell -- at least in > the sh, csh, and ksh shells, which are the ones that I've been through > the source code for. > > (BTW, if we ran it in the background, with the "&", then the shell > will not wait() for it, and desynchronization problems like the ones > you're seeing can easily happen.) > > This leaves other possibilities -- one is that this is happening > on an NFS-mounted filesystem and something is amiss there. Another > is that the disk read/write queue has gotten scrozzled and the the > operations are not happening in the correct order. Another is that > perhaps there is a kernel bug related to the multiprocessor code. > (Early versions of Sequent's Dynix multiprocessor OS had similar bugs > when we were beta'ing them 10 years ago.) And so on. But a > correctly-executing shell should *never* do what's described above, > i.e. fail to wait for a command to finish. > > Rich Kulawiec > rsk@itw.com
-- Jim Harmon The Telephone Connection jim@telecnnct.com Rockville, Maryland
>From robin.landis@imail.exim.gov Wed Jan 15 22:34 GMT 1997 From: robin.landis@imail.exim.gov Date: Wed, 15 Jan 97 16:58:50 EST Encoding: 222 Text, 19 Text To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000
Hi. I didn't see this question, but have a backup script that shuts down an Oracle database and the takes a ufsdump backup. It's important that the database really is down before the dump begins. I'm insure that the dump waits on the shutdown commands by evaluating a return code from the shutdown commands as a condition for doing the dump: main.scrpt stop (shuts the database down) rc=$? (move $? to rc variable) if [ $rc != 0 ]; then echo "page robin, oracle shutdown exited with return code" echo $rc exit $rc fi ufsdump commands start here Another way to do this in sh is to use the . command to source the next script into the current shell. .....Robin
>From kevin@uniq.com.au Wed Jan 15 23:35 GMT 1997 From: Kevin.Sheehan@uniq.com.au (Kevin Sheehan {Consulting Poster Child}) Date: Thu, 16 Jan 1997 10:09:06 EST To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000
[ Regarding "SUMMARY: Weird script behaviour in Enterprise 4000", sunmanager@argos.silvac.pt writes on Jan 15: ]
> The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, > in my opinion, would clearly explain why the script ends up not doing what it > was supposed to do.
Not to me, as you'll see below. > > > while (some_condition) do > > touch file[var] > > rm file[var] > > increment [var] > > done > > > > If you execute one of these scripts inside an empty directory, after it finishes running > > you should end up with the same empty directory you started with. > > The problem is that sometimes, depending on the load of the server at the time the script was > > running, some of the files created by the script will be left in the directory. > > > > We are testing this on a Sun Enterprise 4000 with 6 CPUs and 1Gbyte RAM... > > > > The only thing that occurs to me to explain this behaviour is that sometimes the remove > > command will be executed and fail because the system still is in the process of creating > > the file.
Nope.
> > > > Could this be due to the fact that the system is MultiProcessor, and could therefore > > be executing each command on a different CPU, which would lead, depending on load conditions, > > to the behaviour described above?
Nope.
> > In scripts, each command spawns a child to execute the process > > requested.
Strictly speaking, it forks and execs a child process. It also waits until the child process has exitted before it continues. > > > > So, in your example below, if the touch process goes to sleep before > > completion (due to system use--for example) and the rm process begins
if the touch process goes to sleep, then the shell will wait until it wakes up and exits before it will run the rm command. Unless you are running these commands in background (touch file$var &) then the system calls for creat() and unlink() will complete in touch/rm and the shell will wait till the process exits.
> > execution,but doesn't go to sleep, you'll have the two processes > > reversed in execution, leaving the touched file after the remove.
Nope - the shell can't run rm until touch exits. touch doesn't exit until it has done the system call. Same for rm. > > > > The way to control that would be to write the script so that the rm > > can't execute UNTIL the touch is returned "complete".
which is what you are doing by having the shell wait for touch to exit. > > > > One way to do that would be to put an "if" statement around the rm, that > > checks to see if the file is found, and if not, repeats until it finds > > it.
sorry - once touch has returned from the creat() system call, the file by definition is there. If it is not, you have a bug. The problem is left over files, so we know that is working. > > > > Let me know if you would like some suggestions on how to build the loop, > > I'm afraid that right now I'm trying to get a bunch of things done... > > > > Cool? > > > > > > > As I have said before I think Jim's reply makes a lot of sense and would justify > the script behaviour.
Nope - once either the unlink() or creat() system call (rm and touch boil down to this) then the file system is supposed to be consistent. There isn't a "create time" or "remove time" where things are changing afterwards.
My guess - you have found an MT bug, or you are dealing with a soft mounted NFS directory.
l & h, kev
>From pod@morgan.com Thu Jan 16 01:34 GMT 1997 From: "Paul O'Donnell" <pod@morgan.com> Date: Wed, 15 Jan 1997 18:43:05 -0500 (EST) X-Face: #mT^U,17J-aUFAO![bO5%!!8(!&pY+pxsx3W"6*}&"{36w_~[(4ov.NM6<\T82Y%zp$@Z>c>8%yV2+&"G`xsq.TH/}J7(SaAM=IC3XUMQA4>Gut4pF`z|)~rn!IqaV#HwH){R6I?Ue_2KSc/B\oWEuW+Z#5Oa\&,jH;V6jXmGcbS@ To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 Cc: jim@telecnnct.com, jacques.rall@za.eds.com Mime-Version: 1.0
Sun Manager Account reckons: > Here is the summary to the question I posted a a couple of days ago... > > The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, > in my opinion, would clearly explain why the script ends up not doing what it > was supposed to do.
Jim's message is 100% wrong - the commands are not backgrounded, and therefore will happen in strict order. The wait only applies for backgrounded processes. The last message about sync is also incorrect, the sync is irrelevant (it affects writing to actual disk, but it does not affect the file system access).
Do you get a failure from the rm (perhaps is can't execute because the machine is short on memory or some other resource). I have never seen your behaviour before, and I can't reproduce it on any of our E4000s.
>From rsk@itw.com Thu Jan 16 08:04 GMT 1997 From: Rich Kulawiec <rsk@itw.com> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 To: jim@telecnnct.com (Jim Harmon) Date: Thu, 16 Jan 1997 02:22:25 -0500 (EST) Cc: sunmanager@argos.silvac.pt X-Last-River: Brandywine X-Last-CD: Duncan Shiek, "Duncan Shiek" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit
>I believe you're right...if we're talking about a single-CPU workstation >or server. > >I was thinking in terms of BG and threaded processes, rather then >sequential FG processes when I answered the question. > >However, strict FG processing would leave the original question >unanswered, which was Fernando Dias' problem with sequential "hiccups" >during system loading. > >In his description he originally didn't state that he was running the >script interactively at all times. (In his followup he did say >something to that effect, so BG processing doesn't seem to be a factor.) > >Since that's the case, what other possibilities are there that could >cause a sequential write-delete script to skip the erase without error >messages or warnings? > >Taking into account the MultiProcessor (6 CPU) nature of the system, >wouldn't the process manager tend to treat even sequential commands as >individual threads if you didn't specifically limit the process(es) to a >single CPU?
Hmmm...<sound of head scratching and coffee slurping>...no, it should still be single-threaded, even on a multiprocessor machine and even if it he runs it in the background. Let me see if I can explain myself, starting with the second case.
If we take that script
#!/bin/csh
foreach i (1 2 3 4 5 6 7 8 9) touch hose.$i rm hose.$i end
and put in, let's "/tmp/fred" and run it like this:
/tmp/fred
then the interactive shell that we just typed that into fork()s and the child shell process examines "fred" to find out what kind of executable it is (binary, shell, perl, etc.). Since this is a C-shell script, it exec()s the appopriate interpreter, /bin/csh, and feeds the text of the script to it on stdin. This means that the first time through the loop, when the first "touch" command is running, there's (a) a login shell, which is wait()ing for fred to finish, (b) an instance of csh, wait()ing for touch to finish and (c) an instance of touch, doing whatever-it-is that touch does to create a zero length file. It's possible that all three processes are on different CPU's -- there's no way to tell in advance, because there's no way to tell what else is running and contending for resources. But let's say we run this on an unloaded machine with free CPU's. Then even if (a) is on CPU #2, (b) is on CPU #5, and (c) is on CPU #3, processes (a) and (b) will still be wait()ing -- they'll just be doing it on a different CPU, that's all.
Now here's the first case. If instead we run
/tmp/fred &
then our login shell does not wait() for the instance of csh that's interpreting the script; using the same nomenclature as in the last paragraph, the (a) process goes on its merry way, accepting more user commands and doing what login shells do. But just as above, the (b) process, though, behaves just the same, wait()ing on the commands that it runs, and the (c) process, which is "touch", does the same thing it's supposed to do.
Let's try another example that I think is pretty close to what you're thinking of. Let's try this:
#!/bin/csh
touch bozo & rm bozo &
This script will yield somewhat predictable results on a uniprocessor system, because touch runs *so* fast that by the time the shell does another fork() and exec() to run the "rm" command, the "touch" command will have finished and the "rm" will have something to remove.
But on multiprocessor systems, it's possible that the fork() and exec() will happen much faster, because the instance of csh that's making those calls isn't competing for the CPU with "touch". It's also possible that if "touch" lands in a CPU that's busy doing something else and if "rm" lands in a CPU that's idle, that the "rm" will actually execute first -- which means that it'll try to remove a file that hasn't been created yet.
But all of this, as you've no doubt noticed, depends on allowing non-sequential execution of the commands with the script via &. And none of it explains the original problem.
I can think of some possibilities:
1. This is being done via NFS, and something is screwy in the NFS implementation -- or else mis-optimized. I've known of buggy filesystems where the queue of operations (read, write, seek, delete, link, etc.) is managed in such a way that anomalies happen. For example, if the design decision is made that unlink() calls get processed first because they're likely to free up space that may be useful, then there can be problems if the unlink() which is done by "rm" happens before the creat() done by touch. But this really shouldn't be happening in a modern Unix filesystem; bugs like this are pretty old and should be long dead.
2. Someone has done a Bad Thing inside the multiprocessor kernel or maybe the shell in what might be another attempt to overoptimize. It might be that there's an attempt being made to shove each successive shell command onto the least-busy CPU, which on an idle machine, would definitely *not* be the one that just handled the last command, since it would show recent use. If this is being done in such a way that the last command isn't wait()ed for correctly, then the next command could be launched prematurely. (I sense a bad joke coming on...just let it go, let it go...)
3. It could be that there's a bug in the script, and that it isn't issuing the right commands. That's probably the easiest thing to check with "sh -x script" or "sh -xv script", just to make sure that if "touch file.1234" is issued, that "rm file.1234" is too.
4. It could be something else entirely that none of us have thought of. But we will find it and document it and Sun will write us all big checks for being such thoughtful end users.
:-)
---Rsk Rich Kulawiec rsk@itw.com
Return-Path: <cgo@dsi> Received: from aristote.dsi.elis.fr by gie6.dsi (SMI-8.6/SMI-SVR4) id IAA23362; Thu, 16 Jan 1997 08:31:59 +0100 Received: by aristote.dsi.elis.fr (SMI-8.6/SMI-SVR4) id IAA22353; Thu, 16 Jan 1997 08:29:30 +0100 Date: Thu, 16 Jan 1997 08:29:30 +0100 From: cgo@dsi (Cyril GODON) Message-Id: <199701160729.IAA22353@aristote.dsi.elis.fr> To: sunmanager@argos.silvac.pt Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 X-Sun-Charset: US-ASCII content-length: 7583
No I think there is no good explanation for your problem below.
The rm will never run as long as the touch has not be done. It's impossible that the multiple processors should explain this behaviour. The fact the host has multiple or single processor is (or should be) absolutly transparent to the user as long as you manage correctly the parallelism of your application. But you have no concurrency in your script . All the commands are run sequentialy, one after the other.
Don't look for such an explanation, check and recheck what your are doing on your host, are your running two instances of the script at the same time on the same directory, call the Sun holtine and ask them to look for a bug in ksh (if this the script you use) or else where but don't look for a logic behaviour anywhere in UNIX
if UNIX was not able to run commands sequentialy or use its disk cache correctly, you would not have bought an E4000 with Solaris but a pc win NT !!
Cyril GODON
>From tom@diophant.uni-paderborn.de Thu Jan 16 10:05 GMT 1997 Date: Thu, 16 Jan 1997 10:09:50 +0100 (MET) From: Torsten Metzner <tom@plato.uni-paderborn.de> To: sunmanager@argos.silvac.pt Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000
||Here is the summary to the question I posted a a couple of days ago... | |The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, |in my opinion, would clearly explain why the script ends up not doing what it |was supposed to do.
Hello Fernando, do you really think so ? See my remarks below.
| |> From jim@telecnnct.com Mon Jan 13 21:04 GMT 1997 |> Date: Mon, 13 Jan 1997 15:48:34 -0500 |> From: Jim Harmon <jim@telecnnct.com> |> Mime-Version: 1.0 |> To: Sun Manager Account <sunmanager@argos.silvac.pt> |> Subject: Re: Weird script behaviour in Enterprise 4000 |> Content-Transfer-Encoding: 7bit |> |> In scripts, each command spawns a child to execute the process |> requested. |> |> So, in your example below, if the touch process goes to sleep before |> completion (due to system use--for example) and the rm process begins |> execution,but doesn't go to sleep, you'll have the two processes |> reversed in execution, leaving the touched file after the remove. |> |> The way to control that would be to write the script so that the rm |> can't execute UNTIL the touch is returned "complete". |> |> One way to do that would be to put an "if" statement around the rm, that |> checks to see if the file is found, and if not, repeats until it finds |> it. |> |> Let me know if you would like some suggestions on how to build the loop, |> I'm afraid that right now I'm trying to get a bunch of things done... |> |> Cool? |> |> | |
|As I have said before I think Jim's reply makes a lot of sense and would justify |the script behaviour.
I think that Jims answer is wrong. It's correct that in a script, each command (which is not a internally shell command) spawns a child to execute the process. But his conclusion is wrong. If I have two commands first_command second_command the second_command command is only executed, if the first_command command returns and this is the case if the first_command is finished (Surely I can write a command which will do something in the background and returns, but this is unusual). BTW that's the reason why wait works (is necessary) only for commands which runs in the background. So the fact that each command spawns a child has nothing to do with your problem.
In my opinion sync isn't the answer too, because if the filesystem isn't synced then the file doesn't necessarily exist on disk, but it exist for the system (only in memory) and so rm should find the file.
So in my opinion the behaviour of your shell script is a bug. But here I am not 100% sure. I think you should report it to SUN.
Hope this helps,
Torsten. ------------------------------------------------------------------------------- My address : Torsten Metzner E-Mail: tom@uni-paderborn.de Universitaet-GH Paderborn Tel.: +49 5251 602621 FB 17 - Mathematik Fax : +49 5251 603836 Warburger Str. 100 Office: D2.207 33098 Paderborn Germany -------------------------------------------------------------------------------
>From Tim.Feeney@FMR.COM Thu Jan 16 15:44 GMT 1997 From: "Feeney, Tim" <Tim.Feeney@FMR.COM> To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: RE: SUMMARY: Weird script behaviour in Enterprise 4000 Date: Thu, 16 Jan 1997 10:21:00 -0500 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit
Fernando,
First let me outline in general what happens in your script, and then try to offer a possible explaination. When you do the touch <filename> and it is not there the system issues a create call and returns with a status of the call. The rm <filename> does an inode lookup in the filesystem inode table and then deletes the entry. What might be occuring is that due to a heavy system load and that you might have write back cache enable (ie. the system issues a return status without the actual device responding to the write) the actual creation of the file has not completed by the time the touch command returns. The two processes are not running at the same time, as the scripting language waits for one command to finish before going on to the next one. A couple of ways to debug this are put echo $? lines after the touch and rm commands to see what the exit codes are. If you are performing this on a constantly used disk try it on one that is not used by anyone else (should not make a difference though). This is a interesting problem that I have not seen before, and would be interested in any results you get from testing, or any other comments.
Tim ----------
>From johnb@Soliton.COM Thu Jan 16 16:35 GMT 1997 Date: Thu, 16 Jan 1997 10:50:14 -0500 From: john benjamins <johnb@Soliton.COM> X-Sender: johnb@m5 To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: SUMMARY: Weird script behaviour in Enterprise 4000 Mime-Version: 1.0
On Wed, 15 Jan 1997, Sun Manager Account wrote:
} Here is the summary to the question I posted a a couple of days ago... } } The feedback from Jim Harmon <jim@telecnnct.com> makes a LOT of sense to me and, } in my opinion, would clearly explain why the script ends up not doing what it } was supposed to do. } } Please check his message, which is included just after my original message. } } All in all I got four replies, which are included below. } } This is my original message: } } > From sun-managers-relay@ra.mcs.anl.gov Mon Jan 13 19:34 GMT 1997 } > Date: Mon, 13 Jan 1997 18:13:39 GMT } > From: Sun Manager Account <sunmanager@argos.silvac.pt> } > To: sun-managers@ra.mcs.anl.gov } > Subject: Weird script behaviour in Enterprise 4000 } > } > Hello SMs, } > } > Can somebody have a look at this and please explain to me what is going on here? } > } > I have made a very simple shell script that does something similar to what is shown below:
you say >>similar<< but does the following script exhibit the behaviour you describe?
#!/bin/sh
cnt=0 while [ $cnt < 1000 ] do touch file${cnt} rm file${cnt} cnt=`expr $cnt + 1` done
if this shell script leaves files lying around, then i would report it to sun as a bug. there may be some subtle semantics that are in your "similar" script that are not in the trivial script i've given.
} > while (some_condition) do } > touch file[var] } > rm file[var] } > increment [var] } > done } > } > This can also be done with a braindead shell script that goes something like: } > } > touch file1 } > rm file1 } > [repeat the two lines above X number of times, substituting "file1" for "file2", etc, etc. } > } > } > If you execute one of these scripts inside an empty directory, after it finishes running } > you should end up with the same empty directory you started with. } > The problem is that sometimes, depending on the load of the server at the time the script was } > running, some of the files created by the script will be left in the directory. } > } > We are testing this on a Sun Enterprise 4000 with 6 CPUs and 1Gbyte RAM... } > } > The only thing that occurs to me to explain this behaviour is that sometimes the remove } > command will be executed and fail because the system still is in the process of creating } > the file. } > } > Could this be due to the fact that the system is MultiProcessor, and could therefore } > be executing each command on a different CPU, which would lead, depending on load conditions, } > to the behaviour described above?
this would require the >>SHELL<< to be multithreaded, which is not the case as far as i know. if it's not multithreaded, then the touch command MUST complete BEFORE the rm command can run. however, the file will exist unless there is an UFS bug somewhere in the kernel/driver.
-- END OF REPLIES TO THE FIRST SUMMARY I POSTED TO SUN MANAGERS
-- SECOND SECTION - Additional data provided by me to selected people, and the answers I got back
>From sunmanager Fri Jan 17 12:39 GMT 1997 Date: Fri, 17 Jan 1997 12:37:42 GMT From: Sun Manager Account <sunmanager> To: afinkel@pfn.com, jim@telecnnct.com, rsk@itw.com, jacques.rall@za.eds.com, pell@lysator.liu.se, keric@fv.com, robin.landis@imail.exim.gov, Kevin.Sheehan@uniq.com.au, pod@morgan.com, cgo@dsi.elis.fr, tom@plato.uni-paderborn.de, Tim.Feeney@FMR.COM, johnb@Soliton.COM, sunmanager@argos.silvac.pt Subject: ADDITIONAL INFO REGARDING: Weird script behaviour in E4000
Hello again,
To the persons who are getting this message "out of the blue", with no forewarning, let me just say that since posting my "SUMMARY" regarding this subject I have received messages from some people pointing out that some of my comments and conclusions were completely wrong. From looking at their messages I can see that some of them seem to know a lot about the "Systems Programming" side of UNIX, of which I have no experience whatsoever. The issues raised by some of them regard the "sequential, non multi-threaded, nature of the UNIX shell. By definition if the shell script execution process is strictly sequential the behaviour I am getting SHOULD NOT happen. This behaviour would therefore have to be attributed to a system or script related subtlety, or to a bug...
Ok, I am now going to give you guys the full scoop regarding what is happening with the script problem I reported some time back.
First, some corrections to my additional message:
1 - The script was not created by me. It was created, in the way decribed below, by a friend of mine and was run on his E4000. I am writing this on his behalf because he does not have access to this forum
2 - Yesterday I was able to go there and do some testing on my own. Here are the results, but first let me give you some details regarding the system configuration:
- Ultra Enterprise 4000 with 6 170MHz CPUs and 512MB RAM - Solaris 2.5.1 with the standard patch cluster installed - Sparc Storage Array with Volume Manager 2.1.1 and all relevant patches installed. The firmware on the Storage Array Controller AND Optical SBus Module Card have been checked and upgraded to the last available revisions
- The system has Mapper installed (a database system from Unisys) and the the development work is being made on top of this database. The shell scripts exhibiting the problems are generated by Mapper as part of its normal operation
- The shell scripts are executed in a UFS filesystem (/ARE) on a Storage Array partition spanning multiple disks
Here is a partial copy of the script:
if [ ! -d /ARE/ARET/S00 ]; then #DE961113 mkdir /ARE/ARET/S00 #DE961113 fi #DE961113 cd /ARE/ARET/S00 error=0 cd /ARE/ARET set -x touch DYMTCH touch SORTOUT rm DYMTCH touch DYMTCH rm SORTOUT touch DYMTCH ln DYMTCH SORTOUT cd /ARE/ARET touch U01 rm U01 touch S01/DYMTCH ln S01/DYMTCH U01 cd /ARE/ARET touch U02 rm U02 touch S02/DYMTCH ln S02/DYMTCH U02 cd /ARE/ARET touch U03 rm U03 touch S03/DYMTCH ln S03/DYMTCH U03 cd /ARE/ARET touch U04 rm U04 touch S04/DYMTCH ln S04/DYMTCH U04 cd /ARE/ARET touch U05 rm U05 touch S05/DYMTCH ln S05/DYMTCH U05 cd /ARE/ARET touch U06 rm U06 touch S06/DYMTCH ln S06/DYMTCH U06 cd /ARE/ARET touch U07 rm U07 touch S07/DYMTCH ln S07/DYMTCH U07 cd /ARE/ARET touch U08 rm U08 [snip, snip...]
Before the script executes the directories ARET and S[XX] will already have been created.
If you have a look at the beginning of the script you will see that the 'set -x' directive was put there so that we can check what it is doing.
Part of the output of the script execution with ' set -x' is included below:
+ touch DYMTCH + touch SORTOUT + rm DYMTCH DYMTCH: No such file or directory + rm SORTOUT + touch DYMTCH + ln DYMTCH SORTOUT + cd /ARE/ARET [snip, snip...] + touch U03 + rm U03 + touch S03/DYMTCH + ln S03/DYMTCH U03 ln: S03/DYMTCH and U03 are identical + cd /ARE/ARET + touch U04 + rm U04 U04: No such file or directory + touch S04/DYMTCH + ln S04/DYMTCH U04 + cd /ARE/ARET [snip, snip...]
Some additional important things to know:
- This script can be executed several times without failing, but once in a while it will...
- As I said before, I was there yesterday and executed the script several times (a lot of them!) and it never failed (Sigh...) Then I changed the pathbit "/ARE" to point to "/TRASH", a directory created by me on the internal E4000 system disk, and ran this script a lot of times. It also never failed...
Still, my friend says that it has happened to him several times, and the printout of the script run with 'set -x' proves it.
Well guys, what do you think?
NOTE: I am going to send this message to all the people that have sent me mail regarding this subject. If you have additional questions or just want to be excluded form this "personal" mailing list just drop me a note.
Regards,
Fernando Dias
>From jim@telecnnct.com Mon Jan 20 03:34 GMT 1997 Date: Sun, 19 Jan 1997 21:13:14 -0500 From: Jim Harmon <jim@telecnnct.com> Mime-Version: 1.0 To: Sun Manager Account <sunmanager@argos.silvac.pt> Subject: Re: ADDITIONAL INFO REGARDING: Weird script behaviour in E4000 Content-Transfer-Encoding: 7bit
You say the script is created by the database engine?
Here's what I would start looking for:
Independant Database activity that could be locking the newly "touch'ed" files.
(Perhaps the testing is not accurately duplicating the actual system conditions at the time of failure?)
Are the scripts being executed by the database system in an independant child process, without human intervention, or are they ALWAYS run on demand?
(In other words, as automated parrent-child spawning, as originally suggested, rather than fork/exec?)
Are any OTHER scripts running on the system that might be giving the indications found
in other words, is there perhaps a duplicate script running that could possibly be slightly out-of-sync with the failing script, so that occasionally the file is simply being replaced after the interactive script removes them?
>From pod@morgan.com Mon Jan 20 18:05 GMT 1997 From: "Paul O'Donnell" <pod@morgan.com> Date: Mon, 20 Jan 1997 12:36:02 -0500 (EST) X-Face: #mT^U,17J-aUFAO![bO5%!!8(!&pY+pxsx3W"6*}&"{36w_~[(4ov.NM6<\T82Y%zp$@Z>c>8%yV2+&"G`xsq.TH/}J7(SaAM=IC3XUMQA4>Gut4pF`z|)~rn!IqaV#HwH){R6I?Ue_2KSc/B\oWEuW+Z#5Oa\&,jH;V6jXmGcbS@ To: Sun Manager Account <sunmanager@argos.silvac.pt>, afinkel@pfn.com, jim@telecnnct.com, rsk@itw.com, jacques.rall@za.eds.com, pell@lysator.liu.se, keric@fv.com, robin.landis@imail.exim.gov, Kevin.Sheehan@uniq.com.au, cgo@dsi.elis.fr, tom@plato.uni-paderborn.de, Tim.Feeney@FMR.COM, johnb@Soliton.COM Subject: Re: ADDITIONAL INFO REGARDING: Weird script behaviour in E4000 Mime-Version: 1.0
Sun Manager Account reckons: > > Well guys, what do you think?
I think you have more than one of these scripts running at a time.
>From rsk@itw.com Mon Jan 20 18:35 GMT 1997 From: Rich Kulawiec <rsk@itw.com> Subject: Re: ADDITIONAL INFO REGARDING: Weird script behaviour in E4000 To: pod@morgan.com (Paul O'Donnell) Date: Mon, 20 Jan 1997 13:04:19 -0500 (EST) Cc: sunmanager@argos.silvac.pt (Sun Manager Account), afinkel@pfn.com, jim@telecnnct.com, jacques.rall@za.eds.com, pell@lysator.liu.se, keric@fv.com, robin.landis@imail.exim.gov, Kevin.Sheehan@uniq.com.au, cgo@dsi.elis.fr, tom@plato.uni-paderborn.de, Tim.Feeney@FMR.COM, johnb@Soliton.COM X-Last-River: Brandywine X-Last-CD: Duncan Shiek, "Duncan Shiek" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit
Paul O'Donnell writes: >Sun Manager Account reckons: >> >> Well guys, what do you think? > >I think you have more than one of these scripts running at a time.
Yes!! <Sound of hand smacking forehead>
After reading this, I went back through the all the info that you sent along, and noticed that the script doesn't appear to tag any of the files with the PID in order to uniqify them; which means that if your database system decides to create two of these and run them at the same time, that odd results are likely.
(Digression: a common technique inside shell scripts is to use filenames like /tmp/fred$$ in order to tag them with the PID of the process that's busy using them; this allows multiple instances of the same program to run without stepping on each other. There's also a subroutine in the C library, mktemp(), which facilitates doing the same thing inside C programs.)
I think Paul may very well have hit the nail on the head; do you have a way of testing for multiple concurrent executions of the script?
---Rsk Rich Kulawiec rsk@itw.com
>From jacques.rall@za.eds.com Tue Jan 21 06:34 GMT 1997 From: Jacques Rall <jacques.rall@za.eds.com> To: "'Sun Manager Account'" <sunmanager@argos.silvac.pt> Subject: RE: ADDITIONAL INFO REGARDING: Weird script behaviour in E4000 Date: Tue, 21 Jan 1997 07:19:55 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit
Maybe it's something to do with the disk array; the way it stripes?
-- END OF UPDATED SUMMARY
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:11:46 CDT