---------- X-Sun-Data-Type: text X-Sun-Data-Description: text X-Sun-Data-Name: text X-Sun-Charset: us-ascii X-Sun-Content-Lines: 119 I started to answer this question concerning recent attacks on the Sun RPC daemons statd and/or ttdbserverd, when I realized that this would be important information for all Sun admins to have. I have attached a file which includes the email traffic concerning this problem on the campus of University of California, San Diego. Most attacks occured here April 8 and 9. Many systems were compromised (most were at least probed). The moral of the story is to have the Recommended Patches installed, and keep them updated. Some Solaris 2.6 sytems were compromised because they had an older version of the ttdbserverd patch (see summary below). Here is a summary from the attached file: (but I would recommend at least glancing at everything) ----------------------------------------------------------------------------- After checking around on the machines that were broken into here in ECE, it appears that the attack tried multiple exploits, namely against rpc.statd and rpc.ttdbserverd. No attacks appear to have been successful on machines that had the latest Sun rpc.ttdbserverd patch and the latest rpc.statd patch. Luckily, it appears that the intruder didn't have time to come back and actually *do* anything after the presumably automated attack. I guess there's safety in numbers. :-) I found no evidence of any kind of tampering on any of the compromised machines, other than the droppings from the initial intrusion. Did anyone else find further evidence that the intruder had returned? Some other observations... on new Solaris installs, we turn off ttdbserverd in inetd.conf as a matter of practice, but there are still lots of machines that we haven't retroactively turned it off on. On all machines that I inspected, a /core file from rpc.ttdbserverd was found if rpc.ttdbserverd was enabled. Of those machines, only one was compromised (cardinal.ucsd.edu, Solaris 2.6) and it was missing the proper patch, 105802-07 - it had 105802-03 installed. [Note that 105802--5 will break dtlogin for non-root users!!] On the Solaris 2.5 machines that were compromised (barclay.ucsd.edu and topcad.ucsd.edu), neither had the rpc.statd patch installed (103468-03). Likewise for the Solaris 2.5.1 system (yosemite.ucsd.edu), it was missing rpc.statd patch 104166-02 (or later). In summary - the Solaris 2.5/2.5.1 machines that were compromised were vulnerable to both the rpc.statd and rpc.ttdbserverd exploits, and the Solaris 2.6 machine compromised was vulnerable to the rpc.ttdbserverd exploit. Does anyone have a Solaris machine that was compromised that *was* up to the current patch level? I realize that the soeunix.ucsd.edu compromise of a couple of weeks ago was apparently against a machine that was current, and Casper Dik's posting in comp.unix.solaris suggests that the patch doesn't fix the vulnerability. However, it seems inconsistent that only the unpatched machines here in ECE appear to have been compromised. All the more reason to comment out rpc.ttdbserverd in /etc/inetd.conf on all machines, and run rpc.statd as daemon. Relevant links to CERT advisories on these two exploits: http://www.cert.org/advisories/CA-98.11.tooltalk.html http://www.cert.org/advisories/CA-97.26.statd.html Rolf Schreiber UCSD ECE Computing Support Group 619-534-3999 rolf@ucsd.edu ------------------------------------------------------------------------------ I have attached a script written by Rolf that implements a "fix" for the statd vulnerability, by making statd run as user "daemon", and changing ownership and permissions of /var/statmon, as per Casper Dik's post in comp.unix.solaris. I have run this on Solaris 2.5 and 2.5.1 systems with no problems. Although most systems seemed protected by the latest statd patch (Solaris 2.5.1 and "lower"), it is still recommended to make this change. Hope this is useful. Below is the original question I was replying to. Dave Foster > From sun-managers-relay@sunmanagers.ececs.uc.edu Tue Apr 13 11:59:45 1999 > Date: Tue, 13 Apr 1999 08:54:04 -0700 > From: "Paul H. Yoshimune" > To: sun-managers@sunmanagers.ececs.uc.edu > Subject: ttdb hack? > Mime-Version: 1.0 > > I had the following show up in my logs last night; seems obvious someone is up > to no good. My question is, is this a known hack/vulnerability? I don't > *think* the attempt was successful, although I'll be watching things quite > closely to make sure... > > Anyone know what exactly happened here? > > Apr 12 21:41:32 blade rpc.ttdbserverd[1706]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:41:32 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:41:33 blade rpc.ttdbserverd[1707]: iserase(): 78 > Apr 12 21:42:11 blade rpc.ttdbserverd[1707]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:42:11 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:42:12 blade rpc.ttdbserverd[1708]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:42:12 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:42:13 blade rpc.ttdbserverd[1709]: iserase(): 78 > Apr 12 21:42:13 blade rpc.ttdbserverd[1709]: iserase(): 78 > Apr 12 21:44:15 blade rpc.ttdbserverd[1709]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:44:15 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:44:16 blade rpc.ttdbserverd[1710]: iserase(): 78 > Apr 12 21:45:05 blade rpc.ttdbserverd[1710]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:45:05 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:45:06 blade rpc.ttdbserverd[1725]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... > Apr 12 21:45:06 blade inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped > Apr 12 21:45:07 blade rpc.ttdbserverd[1735]: iserase(): 78 > Apr 12 21:50:21 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" > Apr 12 21:50:21 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" > Apr 12 21:51:11 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" > Apr 12 21:51:12 blade last message repeated 3 times > Apr 12 22:06:48 blade statd[130]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" > Apr 12 22:08:38 blade last message repeated 5 times > > -- > Paul H. Yoshimune > paul@katana.com > ---------- X-Sun-Data-Type: default X-Sun-Data-Description: default X-Sun-Data-Name: sunrpc.doc X-Sun-Charset: us-ascii X-Sun-Content-Lines: 732 ######################################################################### # SUNRPC.DOC 4-12-99 DSFoster # # This is a collection of emails from "managers@ucsd" list, regarding the # "sunrpc" security problem. Most machines on campus were probed for a # security hole involving the rpc.statd and rpc.ttdbserverd daemons. # # In a nutshell, machines that were completely up to date on patches # have not been compromised in this latest round of attacks, although # an earlier attack (see first message from Jim Napier) involved a fully # patched system. So it remains unclear whether the vendor-supplied # patches are sufficient to protect a system. # # One system that was compromised (alive5) was an unpatches Solaris 2.5.1 # system, while another (natasha) was a fully patched Solaris 2.6 system # which had an older ttdbserverd patch. # # The attack begins with using the statd and/or ttdbserverd security # holes to execute code included in the remote-procedure-call (RPC) # request; this code creates a file /tmp/bob which is used as the # configuration file when starting a new INETD process. This process # can be used at a later time to provide network services to an # outsider using the compromised system. # # What follows is a roughly chronological listing of email traffic # among "managers@ucsd" members concerning this problem. The initial # security problems with statd were first noted in 1997, but the first # evidence of this problem here at UCSD was in March of 1999. # # Action to take: # # 1. Install all Recommended Security patches, and keep them updated! # 2. Comment out rpc.ttdbserverd in /etc/inetd.conf. # 3. Run the rpc.statd process as user "daemon" (see last message # in this file). # 4. Install and use the TCP_WRAPPERS package: # ftp://ftp.sdsc.edu/pub/mirrors/ftp.win.tue.nl/pub/security/ # ######################################################################### Date: Sat, 27 Mar 1999 15:24:00 -0800 (PST) From: Jim Napier To: managers@ucsd.edu Subject: UCSD Engineering system compromise We've done an initial assessment and cleanup of soeunix. We believe the exploit that was used was a relatively new statd exploit that was reported on some security lists at the beginning of this year. statd was tricked into executing a series of instructions that cause a second inetd to launch with privileged services. One of the symptoms of this exploit is the existence of a file called /tmp/bob, which the second inetd uses as its configuration file. The system that was broken into was patched to the latest level of available vendor supplied statd patch, and unfortunately at this point there is no new specific vendor patch for this hack. A workaround is to run statd as user daemon instead of as root, which we tested and it worked for us. As near as we can ascertain no other systems in our immediate area were compromised and the intruder was noticed before utilizing the access to do any serious damage. Checks of network logs indicate that a possible source of the attack was E115210.vtacs.com. We'll be following up with that organization. <*> Jim Napier jnapier@ucsd.edu <*> <*> Office of Engineering Computing 7116 EBU1, MC 0405 <*> <*> Jacobs School of Engineering 534-5212 <*> ######################################################################### From: Jim Napier Date: Mon, 29 Mar 1999 15:34:04 -0800 (PST) To: managers@ucsd.edu Subject: more on statd breakin followup This is a followup to my post over the weekend regarding a breakin on one of our Suns and to provide some references for others who may want to see more technical detail. The breakin mechanism was most likely based on a program that came out at the beginning of this year that exploits a variant of an old (late 1997) rpc.statd hole that the code's author claims Sun never fully fixed. Summarizing what I know about what OSes are susceptible- * Solaris 2.4, 2.5.1 - definitely vulnerable, even if fully patched * Solaris 2.6, 7 - supposed to be vulnerable according to the notes acccopanying * the exploit code, but we were unsuccessful in breaking into our 2.6 systems. It may be that our 2.6 systems did not appear susceptible because according to the exploit code, to break into the newer OSes you need to also do some DNS spoofing, which we may not have properly triggered. There is an existing bugid (4159085) that covers this particular security problem. I called Sun and they were noncommittal about whether they felt 2.6 and newer were safe. First the tech person said it was, then on further rereads of the bug reports wasn't sure. They say there are engineers working on a patch. To see the hacker's view on the technical side of the current vulnerabilities and a copy of the code look at- http://www.attrition.org/hosted/cop/index.html The specific exploit we saw was to use the rpc.statd hole to create a phony inetd configuration file named /tmp/bob with the line "ingreslock stream tcp nowait root /bin/sh sh -i" in it. ingreslock is typically defined in most /etc/services files on Unix systems. Then the exploit would then run "inetd -s /tmp/bob", then open a connection to the ingreslock service which would drop into a root shell. One member of our group found a reference on the net that describes a breakin exactly like we saw and had a workaround from Casper Dik of Sun (i.e. change rpc.statd to run as user daemon). This can be viewed at- http://x1.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=450921549&CONTEXT=922733143.1600716902&hitnum=1 If I get more info from Sun to clarify specific OS version vulnerabilities to this, I'll followup to managers. <*> Jim Napier jnapier@ucsd.edu <*> <*> Office of Engineering Computing 7116 EBU1, MC 0405 <*> <*> Jacobs School of Engineering 534-5212 <*> ######################################################################### Date: Fri, 9 Apr 99 07:57:14 PDT From: spl@szechuan.ucsd.edu (Steve Lamont) To: managers@ucsd.edu Subject: loner.ucsd.edu may have been penetrated Content-Length: 1453 X-Lines: 30 Status: RO > Date: Fri, 9 Apr 1999 07:39:45 -0700 (PDT) > To: nbaker@chemcca33.UCSD.EDU > Subject: Re: Lockd/Statd Exploits? (What's This?) > > This is the same exploit that was used to break into one of our Solaris > 2.5.1 machines a couple of weeks ago. Looks like all our systems were > probed with this exploit last night at the same time. None appear to > have been successful. We changed all our rpc.statd daemons to run as > user daemon and that seems to have helped. You should look for the > existence of the /tmp/bob file, more than one inetd process running, > other suspicious processes (which may be sniffers), and existing open > connections from any suspicious hosts (netstat -n will show this). The > fingerprints we found during our breakin were evidence of a sniffer > binary that was installed in /var/tmp, a .rhosts file in /usr/bin, and > a tampered /etc/shadow file that had a password installed for nobody4, > which usually does not have a password. > > <*> Jim Napier jnapier@ucsd.edu <*> > <*> Office of Engineering Computing 7116 EBU1, MC 0405 <*> > <*> Jacobs School of Engineering 534-5212 <*> I just found a /tmp/bob file on loner.ucsd.edu and three copies of inetd running on the system. I have shut down the system and will be pulling it off the net and looking at it ASAP. *sigh* ############################################################################ From: rolf@la-playa.ucsd.edu (Rolf Schreiber) Date: Fri, 9 Apr 1999 15:15:42 -0700 To: managers@ucsd.edu, probes-l@ucsd.edu Subject: Further followup notes on Solaris statd/ttdbserverd exploit Cc: ecehelp@ece.ucsd.edu After checking around on the machines that were broken into here in ECE, it appears that the attack tried multiple exploits, namely against rpc.statd and rpc.ttdbserverd. No attacks appear to have been successful on machines that had the latest Sun rpc.ttdbserverd patch and the latest rpc.statd patch. Luckily, it appears that the intruder didn't have time to come back and actually *do* anything after the presumably automated attack. I guess there's safety in numbers. :-) I found no evidence of any kind of tampering on any of the compromised machines, other than the droppings from the initial intrusion. Did anyone else find further evidence that the intruder had returned? Some other observations... on new Solaris installs, we turn off ttdbserverd in inetd.conf as a matter of practice, but there are still lots of machines that we haven't retroactively turned it off on. On all machines that I inspected, a /core file from rpc.ttdbserverd was found if rpc.ttdbserverd was enabled. Of those machines, only one was compromised (cardinal.ucsd.edu, Solaris 2.6) and it was missing the proper patch, 105802-05 - it had 105802-03 installed. On the Solaris 2.5 machines that were compromised (barclay.ucsd.edu and topcad.ucsd.edu), neither had the rpc.statd patch installed (103468-03). Likewise for the Solaris 2.5.1 system (yosemite.ucsd.edu), it was missing rpc.statd patch 104166-02 (or later). In summary - the Solaris 2.5/2.5.1 machines that were compromised were vulnerable to both the rpc.statd and rpc.ttdbserverd exploits, and the Solaris 2.6 machine compromised was vulnerable to the rpc.ttdbserverd exploit. Does anyone have a Solaris machine that was compromised that *was* up to the current patch level? I realize that the soeunix.ucsd.edu compromise of a couple of weeks ago was apparently against a machine that was current, and Casper Dik's posting in comp.unix.solaris suggests that the patch doesn't fix the vulnerability. However, it seems inconsistent that only the unpatched machines here in ECE appear to have been compromised. All the more reason to comment out rpc.ttdbserverd in /etc/inetd.conf on all machines, and run rpc.statd as daemon. Relevant links to CERT advisories on these two exploits: http://www.cert.org/advisories/CA-98.11.tooltalk.html http://www.cert.org/advisories/CA-97.26.statd.html Rolf Schreiber UCSD ECE Computing Support Group 619-534-3999 rolf@ucsd.edu ######################################################################### spl From: Brian Parent Subject: Re: Further followup notes on Solaris statd/ttdbserverd exploit To: rolf@ece.ucsd.edu Date: Fri, 9 Apr 1999 16:56:09 -0700 (PDT) Cc: managers@ucsd.edu, probes-l@ucsd.edu, ecehelp@ece.ucsd.edu To corroborate Rolf's contention that patched 2.5.1 systems are safe, I downloaded the exploit and ran it against one of my own patched 2.5.1 systems, but without the "statd as daemon" workaround, and the exploit failed, and was logged. However, I would still recommend using the "statd as daemon" work around. > From: rolf@la-playa.ucsd.edu (Rolf Schreiber) > Date: Fri, 9 Apr 1999 15:15:42 -0700 > To: managers@ucsd.edu, probes-l@ucsd.edu > Subject: Further followup notes on Solaris statd/ttdbserverd exploit > > Does anyone have a Solaris machine that was compromised that *was* up to the > current patch level? I realize that the soeunix.ucsd.edu compromise of a couple > of weeks ago was apparently against a machine that was current, and Casper > Dik's posting in comp.unix.solaris suggests that the patch doesn't fix the > vulnerability. However, it seems inconsistent that only the unpatched machines > here in ECE appear to have been compromised. > > All the more reason to comment out rpc.ttdbserverd in /etc/inetd.conf on > all machines, and run rpc.statd as daemon. > > Relevant links to CERT advisories on these two exploits: > http://www.cert.org/advisories/CA-98.11.tooltalk.html > http://www.cert.org/advisories/CA-97.26.statd.html > > > -- > Rolf Schreiber UCSD ECE Computing Support Group 619-534-3999 rolf@ucsd.edu > ######################################################################### Date: Fri, 9 Apr 1999 18:02:10 -0700 (PDT) From: Jerome Wanetick To: madden@ucsd.edu cc: probes-l@ucsd.edu, managers@ucsd.edu Subject: Re: Further followup notes on Solaris statd/ttdbserverd exploit He's baaaaack! We shutdown "Bob" on the three Solaris 2.6 machines hit in our domain (132.239.117.n). We restarted /tmp/bob as an unprivileged user and on one of the machines and lo and behold he came back. He installed a root kit in /tmp, untarred it and attempted to install a packet sniffer called "update" in /usr/man/tmp. The root kit then tried to replace "inetd", "ps" and add a file in /kernel called "pssys". We've saved all of these files for anyone interested. -Jerry ######################################################################### Date: Fri, 9 Apr 1999 18:35:39 -0700 (PDT) From: Dennis Fetterly To: managers@ucsd.edu Subject: Solaris 2.6 (sparc) rpc.ttdbserverd patch In the wake of all of the patch reports for rpc.ttdbserver, I wanted to mention that the current version of the Solaris 2.6 sparc patch for rpc.ttdbserverd is 105802-07. From the readme for that patch: Problem Description: 4172282 patch 105802-05 breaks dtlogin for non-root users so it may be wise to make sure that you install the -07 patch, not the -05 version. -Dennis Dennis Fetterly Center for Coastal Studies Scripps Institution of Oceanography (619) 534-0607 dennis@coast.ucsd.edu ######################################################################### From: rolf@la-playa.ucsd.edu (Rolf Schreiber) Date: Fri, 9 Apr 1999 10:13:28 -0700 To: managers@ucsd.edu Subject: ECE Suns hacked; using nmap to find compromised Suns Cc: probes-l@ucsd.edu, ecehelp@ece.ucsd.edu ECE also saw a large sweep of our subnets by an intruder looking to exploit the rpc.statd vulnerability under Solaris. The following machines were compromised, damage control in progress: 132.239.24.12 (cardinal.ucsd.edu) 132.239.134.142 (barclay.ucsd.edu) 132.239.134.222 (yosemite.ucsd.edu) 132.239.134.83 (hawaii.ucsd.edu) 132.239.134.110 (oahu.ucsd.edu) 132.239.134.152 (monaco.ucsd.edu) 132.239.134.153 (nice.ucsd.edu) 132.239.134.190 (tahiti.ucsd.edu) 132.239.228.83 (topcad.ucsd.edu) I also used nmap to scan our subnets using the following command, using our 132.239.134 subnet as an example: nmap -p 1524 -m /var/tmp/134_subnet.port1524.scan 132.239.134.5/24 If you see an open port 1524 and it's a Sun, it's a good bet the machine's been compromised. If you need nmap, I've put it in our anonymous FTP area: ftp://ecesis.ucsd.edu/pub/nmap-2.12.tgz It is a gzipped tar file of the source code. Only takes a few minutes to compile. It runs under a lot of platforms, though the BSD/Linux platforms seem to work the best. Have a look at: http://www.insecure.org/nmap/index.html for more info. Now back to the damage control, sigh... -- Rolf Schreiber Manager, Computing Support Group Internet: rolf@ucsd.edu Electrical and Computer Engineering Voice: (619)534-3999 University of California at San Diego FAX: (619)534-2486 ######################################################################### From: Diana Stockdale Date: Fri, 9 Apr 1999 13:43:25 -0700 (PDT) To: managers@ucsd.edu, probes-l@ucsd.edu Subject: More about sunrpc compromise (sideshow.cadabratec.com) Cc: jmadden@ucsd.edu Dear Jim (Madden), I've included some notes below. Since this is a campus-wide event, and hosts my group manages were (luckily) not affected (this time... I'm not gloating, since it is only a matter of time before we are cracked too) would someone in your group take the lead in contacting cadabratec.com? Thanks. For the rest on these lists, the takehome message about this episode is that Wietse's tcp_wrappers and portmap/rpcbind are your friend, see ftp://ftp.porcupine.org/pub/security/index.html. Note that it is important to check PGP signatures since, if I recall correctly, a forged version of tcp_wrappers was temporarily on this site. (It was quickly discovered and anyone that had downloaded the forged copy were notified.) - Diana Stockdale diana@mpl.ucsd.edu x45804 ######################################################################### I've been looking into why we at MPL didn't see any evidence of attempted rpc probes last night. It turns out that almost all of our campus machines were probed Apr 8 between 02:02:18 and 02:28:11 from sideshow.cadabratec.com (216.129.4.5) which Steve shows as the first sunrpc probe on loner. I'm still looking into why 2 of our hosts did not report the probe. All of our probes were refused connection by tcpwrappers and nobody came back so we did not see any of the follow on probes later in the day. I have a feeling that the initial recognizance was from sideshow, followed by the actual attack later in the day from various sites. Note that something odd is going on wrt dns and sideshow. Now you see it... and now you don't. ucsd.edu-35> nslookup cadabratech.com Server: ns1.ucsd.edu Address: 128.54.16.2 Name: cadabratech.com Address: 216.129.4.6 ucsd.edu-36> nslookup sideshow.cadabratech.com Server: ns1.ucsd.edu Address: 128.54.16.2 Name: sideshow.cadabratech.com Address: 216.129.4.5 later... ucsd.edu-38> nslookup 216.129.4.5 Server: ns1.ucsd.edu Address: 128.54.16.2 *** ns1.ucsd.edu can't find 216.129.4.5: Non-existent host/domain ucsd.edu-39> Example log entries of probes follow. Suns: Apr 8 02:24:26 ark.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:20:24 cassis.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:02:18 chestnut.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:10:07 harpa rpcbind: refused connect from 216.129.4.5 to dump() Apr 8 02:12:15 nautilus.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:21:22 pandora rpcbind: refused connect from unknown to dump() Apr 8 02:09:07 pismo.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:02:58 purple rpcbind: refused connect from unknown to dump() Apr 8 02:22:26 tegulas rpcbind: refused connect from unknown to dump() Apr 8 02:11:07 tulip.ucsd.edu rpcbind: refused connect from unknown to dump() Apr 8 02:08:05 turbo rpcbind: refused connect from unknown to dump() Apr 8 02:23:25 violet rpcbind: refused connect from unknown to dump() Linux: Apr 8 02:25:35 callista portmap[29537]: connect from 216.129.4.5 to dump(): request from unauthorized host Apr 8 02:25:34 dracula portmap[8482]: connect from 216.129.4.5 to dump(): request from unauthorized host Apr 8 02:25:35 franken portmap[3726]: connect from 216.129.4.5 to dump(): request from unauthorized host Apr 8 02:25:35 ghost portmap[1616]: connect from 216.129.4.5 to dump(): request from unauthorized host Apr 8 02:25:38 turnip portmap[17869]: connect from 216.129.4.5 to dump(): request from unauthorized host Apr 8 02:28:11 turrid portmap[9862]: connect from 216.129.4.5 to dump(): request from unauthorized host SGI: Apr 8 02:25:30 6E:angel /usr/etc/portmap[141]: rejected prog 100000 proc 4 call from 216.129.4.5 Apr 8 02:25:30 6E:astartes /usr/etc/portmap[161]: rejected prog 100000 proc 4 call from 216.129.4.5 Apr 8 02:25:35 6E:pectin /usr/etc/portmap[140]: rejected prog 100000 proc 4 call from 216.129.4.5 ############################################################################ Date: Fri, 9 Apr 1999 14:24:45 -0700 To: Diana Stockdale , managers@ucsd.edu, probes-l@ucsd.edu From: Jim Madden Subject: ALERT: More about sunrpc compromise (sideshow.cadabratec.com) Cc: jmadden@ucsd.edu We will contact sidseshow.cadabratec.com. In addition, I'm trying to determine what further action to take based on traffic records we may have that show the extent of these probes and the source of the various break-ins. I'd like to hear from anyone on campus who has not already reported to this list or the probes list that they were hit. Any records of source addresses involved in the activity are interesting. Just to emphasize the obvious to everyone on this list: Last night many SunOS machines on campus were compromised using a vulnerability in the sunrpc mechanisms on the machines. Sun's current set of security patches do not fix the problem, although the work-around of running rpc.statd as daemon closes the hole. ALL mangers of suns should check for this problem. I've added a note from Jim Napier in Engineering that seems to neatly summarize the symptoms associated with this hack. Recent discussions of attacks on campus machines have occurred on the mailing list probes-l@ucsd.edu. People responsible for the active support of computers on the UCSD campus should subscribe to that list by sending mail to listserve@ucsd.edu that includes subscribe probes-l or subscribe yourmailaddress@ucsd.edu probes-l in the body of the message. Jim Madden jmadden@ucsd.edu 619-534-2682 ########################################################################### Date: Fri, 9 Apr 1999 18:35:39 -0700 (PDT) From: Dennis Fetterly To: managers@ucsd.edu Subject: Solaris 2.6 (sparc) rpc.ttdbserverd patch In the wake of all of the patch reports for rpc.ttdbserver, I wanted to mention that the current version of the Solaris 2.6 sparc patch for rpc.ttdbserverd is 105802-07. From the readme for that patch: Problem Description: 4172282 patch 105802-05 breaks dtlogin for non-root users so it may be wise to make sure that you install the -07 patch, not the -05 version. Dennis Fetterly Center for Coastal Studies Scripps Institution of Oceanography (619) 534-0607 dennis@coast.ucsd.edu ########################################################################### Date: Fri, 9 Apr 1999 16:50:45 -0700 From: foster@bial1.ucsd.edu To: managers@ucsd.edu Subject: Solaris statd/ttdbserverd exploit Cc: tjernigan@ucsd.edu, kmc@ucsd.edu, rt@ucsd.edu, jmadden@ucsd.edu Several of our Solaris machines [2.5/2.5.1/2.6/7] in the Brain Image Analysis Laboratory and the Alzheimer's Disease Research Center, in the La Jolla Professional Building, were probed for the rpc.statd and rpc.ttdbserverd vulnerabilities on April 8, just after 10PM; only one was successful. The one breakin that was successful was on a Solaris 2.6 system, natasha.ucsd.edu, which *did* have the Solaris 2.6 Recommended patches but did not have the current rpc.statd patch 105802-05. I know of another Solaris 2.5.1 system at the VA which was compromised. This system was unpatched. All other systems that were probed but not compromised were fully patched AND had the latest rpc.statd and rpc.ttdbserverd patches installed. Not all our machines were probed, and I'm not sure why. Of the 4 machines that WERE probed, they shared the following unique characteristics: they either had Samba running, or they had an http server (Apache or Netscape) running. Not sure if that means anything, but it is interesting that only a few of the machines were targeted. I found the following "droppings" on natasha.ucsd.edu: - rpc.ttdbserverd core dumped - multiple inetd's running, using /tmp/bob - /tmp/bob file I did not find any other evidence of activity; no suspicious processes, no binaries in /var/tmp, no .rhosts in /usr/bin, no changes to /etc/shadow. I have the rpc.ttdbserverd core file handy in case anyone would know how to analyze it. I have taken natasha.ucsd.edu down and will re-install the OS. Sigh!... Hosts that were probed: 132.239.109.66 bial1 bial1.ucsd.edu [Solaris 2.5, Patched] 132.239.229.153 galen galen.ucsd.edu [Solaris 2.5.1, Patched] 132.239.229.225 boone boone.ucsd.edu [Solaris 2.5.1, Patched] 132.239.229.178 natasha natasha.ucsd.edu [Solaris 2.6, Patched but no current statd patch, COMPROMISED] The messages appearing in /var/adm/messages, on a system that was probed but NOT compromised, were: ------------------------------------------------------------------------------ Apr 8 22:09:15 rpc.ttdbserverd[328]: _Tt_file_system::findBestMountPoint -- max_match_entry is null, aborting... Apr 8 22:09:25 inetd[127]: /usr/dt/bin/rpc.ttdbserverd: Child Status Changed - core dumped Apr 8 22:09:20 rpc.ttdbserverd[20763]: iserase(): 78 Apr 8 22:09:27 statd[122]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" Apr 8 22:09:27 statd[122]: attempt to create "/var/statmon/sm/; echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; /usr/sbin/inetd -s /tmp/bob &" ------------------------------------------------------------------------------ I have the current recommended security patches installed on all machines, but I thought machines were still vulnerable even with these. It appears that those machines with the latest rpc.statd and/or rpc.ttdbserverd patches were not compromised. Any comments? Could someone give more details about having statd run as user daemon? Will this have any effect on systems that use NFS mounts routinely? Also, if there were any technical reports on these exploits I'd appreciate it if you could forward them to me...I must have missed them in the deluge of recent emails. Thanks! Don't you just love being a sysadmin sometimes!... Dave ############################################################################ From: rolf@la-playa.ucsd.edu (Rolf Schreiber) Date: Mon, 12 Apr 1999 11:15:25 -0700 To: managers@ucsd.edu, probes-l@ucsd.edu Subject: Script hack to fix Solaris statd vulnerability Here's a shell script that I threw together on Friday to fix the Solaris machines in ECE so that /usr/lib/nfs/statd runs as daemon. Hope it's helpful. ================================== CUT HERE =================================== #!/bin/csh -f if ("`uname -r | cut -d. -f 1`" != "5") then echo "Not a Solaris system" exit(1) endif cd /etc/init.d echo "+++ Fixing rpc.statd startup" # Necessary to do it this way to preserve hard links to nfs.client - this # copies the original file and overwrites it with the edits we want. cp -p nfs.client nfs.client.orig sed -e 's| /usr/lib/nfs/statd| su daemon -c /usr/lib/nfs/statd|' nfs.client.orig > nfs.client chmod 744 nfs.client chown -R daemon /var/statmon chmod -R og-w /var/statmon echo "+++ statd entries in /etc/init.d/nfs.client: " grep "/usr/lib/nfs/statd" /etc/init.d/nfs.client set statpid = `ps -ef | grep '/[u]sr/lib/nfs/statd' | awk '{print $2}'` if ($statpid) then echo "+++ Currently running statd: " ps -f -p $statpid echo "+++ Killing currently running statd $statpid" kill $statpid endif echo "" set nonomatch rm -f /var/statmon/sm/* /var/statmon/sm.bak/* su daemon -c /usr/lib/nfs/statd echo "+++ New statd process: " ps -ef | grep '/[u]sr/lib/nfs/statd' exit(0) ############################################################################ At 09:25 AM 4/9/99 -0700, jnapier wrote: >For what it's worth, when our system was broken into I used it as a >test to see what would happen if I went through the ritual of reporting >it to the authorities. I talked to a local FBI agent who was very >helpful, quite technically informed and able to understand the >technical jargon, and had some interesting input on what's possible and >not possible in prosecuting such cases. In particular he pointed out >that *any* evidence that a password sniffer was installed and >activated, even if no passwords were captured, constitutes wiretap >fraud, which they are very interested in pursuing. For some reason this >type of fraud comes under FBI jurisdiction even if the source of the >attack does not cross state lines. The agent's name is Terry Rankhorn, >and he can be reached at 565-1255. Since it appears this sweep involves >campuswide breakins, it might be worth it for those affected to >coordinate to make sure it's clear what the total cost of repair is. >That dollar figure is important in determining at what level the case >is prosecuted. > ><*> Jim Napier jnapier@ucsd.edu <*> ><*> Office of Engineering Computing 7116 EBU1, MC 0405 <*> ><*> Jacobs School of Engineering 534-5212 <*> ############################################################################ # Work-around from Casper Dik: # # Run statd process as user "daemon": # # http://x1.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=450921549&CONTEXT=922733143.1600716902&hitnum=1 # ############################################################################ Re: statd Fatbrain.com Author Casper H.S. Dik - Network Security Engineer {*} Date: 1999/03/03 Forum: comp.unix.solaris > Over the past weekend I received the the following message on the >console and in /var/adm/messages on one of the workstations. >Feb 28 22:23:16 machinename statd[122]: statd: open of /var/statmon/sm/; >echo "ingreslock stream tcp nowait root /bin/sh sh -i" >>/tmp/bob ; >/usr/sbin/inetd -s /tmp/bob &, error No such file or directory WHat OS release was this? It looks like you've been targeted by a known security exploit. Does /tmp/bob exist? Is there another inetd running? > When I looked I first looked around on this workstation. I first looked >to see If it had been rebooted or crashed over the weekend. It had not. I >then looked in /var/statmon/sm for the hosts to be contacted if the >workstation is rebooted and one of the machines was mine. I did not receive >any sort of notification from statd on my machine. I then did a ps -ef | >grep inetd and saw two inetd -s processes running the original and the one >with output going to /tmp/bob. I killed the /usr/sbin/inetd -d >>/tmp/bob >since there appeared no reason to have two running. I also looked in >/tmp/bob and the entry was of course "ingreslock stream tcp nowait root >/bin/sh sh -i". The last thing I did was to check the /etc/inetd.conf and >make sure there was no ingreslock entry in it. Yes, it appears you've been hacked. > As this was the weekend the user was not in the office. There was >several ftp sessions to this machine though. > > By the way this machine is an Ultra1 Creator running Solaris 2.5 >I guess my main questions I have are: >1) Why would I hear from statd when there is no reboot ( appears tome to be >reason for statd to notify lockd to try to reclaim locks) There is a problem in statd which allows it to abuse other services. 2.5 is a bit old, but you need to install the latest automountd patches and statd patches.. For extra safelty, you probably need to run statd as non-root: In /etc/init.d/nfs.client invoke statd as (make sure you keep all links to that file) su daemon -c /usr/lib/nfs/statd and chown -R daemon /var/statmon; chmod -R og-w /var/statmon >2) Is this something that would spawn off on it's own (ie dialing in from >home to another machine on the network and ftping to this machine and >improperly disconnecting) or would someone actaully have to kick this off. No, it happens when someone tries to hack your system. >3) And is there need to be overly concerned with this. Yes. Casper ---------- X-Sun-Data-Type: cshell-script X-Sun-Data-Description: cshell-script X-Sun-Data-Name: fix_statd X-Sun-Charset: us-ascii X-Sun-Content-Lines: 40 #!/bin/csh -f # # Here's a shell script that I threw together on Friday to fix the Solaris # machines in ECE so that /usr/lib/nfs/statd runs as daemon. Hope it's # helpful. # Rolf Schreiber UCSD ECE Computing Support Group 619-534-3999 rolf@ucsd.edu if ("`uname -r | cut -d. -f 1`" != "5") then echo "Not a Solaris system" exit(1) endif cd /etc/init.d echo "+++ Fixing rpc.statd startup" # Necessary to do it this way to preserve hard links to nfs.client - this # copies the original file and overwrites it with the edits we want. # (Note that there are characters after the first 2 '|' characters) cp -p nfs.client nfs.client.orig sed -e 's| /usr/lib/nfs/statd| su daemon -c /usr/lib/nfs/statd|' nfs.client.orig > nfs.client chmod 744 nfs.client chown -R daemon /var/statmon chmod -R og-w /var/statmon echo "+++ statd entries in /etc/init.d/nfs.client: " grep "/usr/lib/nfs/statd" /etc/init.d/nfs.client set statpid = `ps -ef | grep '/[u]sr/lib/nfs/statd' | awk '{print $2}'` if ($statpid) then echo "+++ Currently running statd: " ps -f -p $statpid echo "+++ Killing currently running statd $statpid" kill $statpid endif echo "" set nonomatch rm -f /var/statmon/sm/* /var/statmon/sm.bak/* su daemon -c /usr/lib/nfs/statd echo "+++ New statd process: " ps -ef | grep '/[u]sr/lib/nfs/statd' exit(0) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ David S. Foster Univ. of California, San Diego Programmer/Analyst Brain Image Analysis Laboratory foster@bial1.ucsd.edu Department of Psychiatry (619) 622-5892 8950 Via La Jolla Drive, Suite 2240 La Jolla, CA 92037 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~