SUMMARY: /dev/dtremote symbolic link is breaking /dev/null

Thank yous to Casper Dik and Crist Clark for providing me with some
valuable investigative techniques.  The cause of the incredible
/dev/null metamorphosis  turned out to be a configuration script for
HP OpenView Operations 8.0 for Solaris.  Specifically, the Microsoft
Exchange "SPI" (smart plug-in) contained a configuration application,
EXSPI Node Config (8.10), which was doing the deed.  I am awaiting
word from HP as concerns a patch.

My suspicion that /dev/dtremote was somehow involved (it has a symb
link to /dev/null) was debunked by both of my friendly advisers. 
Total red herring.  I would often first notice the problem when
someone would remote-CDE into the box.  This requires a functional
/dev/null.  Jumped to conclusions.  Oops.

Generally, the mistake is a line of script which tries to use "mv" or
"cp" to write to null, which of course does not work.  The trick is
revealing the culprit which, without the healing benefits of dtrace on
Solaris 10, is beyond hard to do.  If you run this command on any
Solaris box, you will see what a daunting task it is to find out what
script might cause /dev/null to be overwritten or deleted:

  # find / -type f | xargs file * | awk -F: '/script/ {print $1}' |
xargs grep /dev/null

And that's assuming it's not a binary.  Only root can overwrite
/dev/null, so what you are looking for is a script which runs as root.
 My situation was complicated by the fact that my client wanted to be
able to login to CDE as root, widening the pool of suspects: cron,
login scripts, /etc/dt CDE scripts.  Any repository of local or
third-party software should be scrutinized as well.  More tips:

Once you have determined that /dev/null has changed, run "ls -lc
/dev/null".  That will tell you the last time the permissions where
changed.

Turn on process accounting (man acct).  Concentrate on the processes
that ran just before /dev/null changed.  You can also look in
/var/adm/messages for coinciding events.  In my case, I wrote a script
which checked /dev/null and, if missing or altered, sent a logger
message to syslog with an "ls -lc" of /dev/null and recreated
/dev/null.  Keeps things working and helps you narrow down the time
frame of the event.
 
There are two ways to recreate /dev/null:

# cd /dev
# ln -s ../devices/psuedo/mm@0:null null
[Make sure the link is made relative to /dev]

# /usr/sbin/mknod /dev/null c 13 2
# chown root:sys /dev/null ; chmod 666 /dev/null

In the end, it was none of this, just a hunch, that lead us to
discover the evil HP OVO program.  Good times.

--Greg Chavez
  
---------- Original message ----------
From: Greg Chavez <greg.chavez@gmail.com>
Date: Fri, 11 Feb 2005 09:30:32 -0500
Subject: /dev/dtremote symbolic link is breaking /dev/null
To: sunmanagers@sunmanagers.org

I am working short-term at a client site that relies heavily on remote
CDE logins between their many Suns.  The system on which I am working
is a Sun Blade 2500 running Solaris 9, fully patched, JASS'd, and
running HP OpenView Operations .. I built the OS myself.  The gist is
that /dev/null keeps getting turned into an empty file with 664
permissions.  Attempts to relink /dev/null to the null pseudo device
last only until someone tries to perform a remote CDE login. The
culprit appears to be /dev/dtremote which is symlinked to /dev/null.
That would seem to be a problem to me, but this old post suggests
otherwise:

http://marc.theaimsgroup.com/?l=sun-managers&m=104009118018389&w=2

As long as I change /dev/null from 644 to 666, this seems to cause no
problems when the system is running.  But this causes huge problems
when the system is rebooted.  Nothing can "create" /dev/null and the
systems boots up in single-user, read only.  I have to mount the OS
from CD and relink /dev/null to the psuedo device to get it working
again.  Obviously, this is terrible.

Does anybodh have any insight into this?  If I can't figure it out,
I'm going to advise my client to not remote-CDE into this box which,
really, I should advise them to not do anyway.  Nevertheless.  They
like things to work.

Thanks, will sum.

--Greg Chavez
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers