From presslaber at ips.at Mon Jul 11 09:33:01 2011 From: presslaber at ips.at (=?iso-8859-1?Q?Martin_Pre=DFlaber?=) Date: Mon, 11 Jul 2011 13:33:01 +0000 Subject: [SUMMARY] millions of files in ZFS Message-ID: <1D408CFD2A54A846B470A4F37CD3D03E048EDA@MSX.ips.local> Hi! First of all, thank you for your mails! In short: It's no problem with ZFS. --- SUMMARY --- Got a lot of answers from admins who are running servers with more than 90TB and 200-300 million files in one pool (-> mail servers). Having a lot of files in one directory will have a poor performance using commands like "ls" and users could have a hard time browsing the directories with such "big folders". But generally this amount of files is no problem, you can have billions of files in ZFS. (128bit) Nearly in every mail it is mentioned to have only one pool; splitting up will always prove to have been a wrong decision unless you need different reliability/performance setups (e.g. mirror vs. raid6). It's recommended to have at least 48GB RAM for 100 million files and if possible a separate ZFS log and/or cache device. A separate log device is highly recommended when sharing the data using NFS. Using more "small" LUNs should have a much better performance than only one big LUN (ZFS-striping & I/O queues). BUT; do not let ZFS go over 80-85% utilization. -> works as designed -> copy-on-write Reported issue: If you NFS clients are traversing the directories too often, that might invalidate the DNLC cache which associates paths to vnodes. That could happen in UFS as well as in ZFS since it's a common OS facility (see vmstat -s | grep 'name lookups'). http://download.oracle.com/docs/cd/E19620-01/805-4448/6j47cnj0u/index.html This happened to us because people had 500k message in their Maildirs. And the IMAP software was scanning all messages every time someone checked their inbox. When that happened, the DNLC hit ratio would drop from 97% to 10% for a few seconds... that would put too much pressure on the disk subsystem. Tuning guide: http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide Tuning tip: Use NFS v4 with ZFS, raise the buffer limits: /etc/system set zfs:zfs_arc_max = 0x200000000 set ncsize=0x100000 set nfs:nfs4_bsize=0x100000 set ndd:tcp_recv_hiwat=1024000 set ndd:tcp_xmit_hiwat=1024000 set ndd:tcp_max_buf=4194304 set ndd:tcp_cwnd_max=2097152 set ndd:tcp_conn_req_max_q=1024 set ndd:tcp_conn_req_max_q0=4096 Note: Not all OS versions can set these values via /etc/system, on Osol we have to set the tcp* values in one of the netinit scripts in /lib/svc/method via ndd. more information on zfs and scalability could be found on the zfs-discuss list: http://mail.opensolaris.org/mailman/listinfo http://opensolaris.org/jive/forum.jspa?forumID=80 UFS support should be no problem in the next solaris releases, at least for the next 5 to 10 years, but everyone said, it's time to migrate and you will love ZFS. Definitely go ZFS, nothing can beat that filesystem, you never ever don't want anything else if you have worked with it! once again, thanks for all answers! greetings from Austria, - martin presslaber [Q] millions of files in ZFS Hallo together, I am looking for some recommendations or suggestions regarding ZFS and on my opinion a huge amount of files. What we have now, is a setup with Solaris9 and UFS. Around 40 File-Systems with 500GB mounted on a single Server and each holds nearly 2 million files (100k to 10MB) in one directory (application based -> medical software). That's about 20TB with 80 million files (still growing), so let's plan to migrate to Solaris10 and ZFS... We won't have a problem with 20TB in one pool, often seen with ZFS, BUT: * Any experiences with 80 million files and ZFS? 80M in a pool and 2M per ZFS-FS? * Should we use 1 pool, or split the data into 2 or more pools? * Will ZFS still perform with 2 million files in one directory? (UFS works good enough) * With UFS we have 500GB LUNs; we plan to use 1-2TB LUNs for ZFS, good idea? * Is there any tuning needed or will it work with the standard settings? * What's about the "80% used space - performance issue"; having 20TB data, 40 ZFS-FS filled with 90-95%, will we need additional 4TB free space inside the pool to guarantee a good performance??? Some side notes: * Don't worry about the hardware; the new server is a m5000 with 128GB RAM with a lot of I/O getting the LUNs from an HDS USPv with a bunch of 15k FC disks. * The 500GB FS is a legacy limit; in the past, we had to less inodes with more than 500GB (should be solved with a modern UFS+) * With more than 2 million files in one directory, we had several issues sharing the directories with NFS (v3). It takes ages to copy the data from UFS to ZFS, so it would be horrible to see, that ZFS isn't working and we must copy everything back to UFS. What do you think, is it a good idea to migrate to ZFS or should we stay on UFS? (how long will UFS be supported in Solaris 11 12 13 14...) Thanks in advance! If you have any more questions, don't hesitate to contact me. - Martin Presslaber _______________________________________________ sunmanagers mailing list sunmanagers at sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers From cbarnar1 at earthlink.net Thu Jul 14 22:04:00 2011 From: cbarnar1 at earthlink.net (Christopher Barnard) Date: Thu, 14 Jul 2011 21:04:00 -0500 Subject: SUMMARY: How do I check if noexec_user_stack active? In-Reply-To: <8E4787EE-8C85-4F78-96F4-F2600E3393BD@earthlink.net> References: <8E4787EE-8C85-4F78-96F4-F2600E3393BD@earthlink.net> Message-ID: I asked: > I know that setting noexec_user_stack=1 in the /etc/system is a Good Thing To > Do and a reboot is necessary to make that change take effect. > > What I have not found is a way to determine if that setting has been enabled > on a particular server. If it is in /etc/system it is set, but how do I know > if it is active? There must be a command to report the setting in the running > kernel, but I as yet have not found it. How would I do this? The answer: echo noexec_user_stack/D | mdb -k you can extract a lot of info about the running kernel with mdb (or adb, if you prefer) Playing around with it today I found that is installed on all of my Solaris 10 servers, a handful of my Solaris 9, and none of my Solaris 8. Thanks to David Foster Matthew Hannigan Rob McMahon Tim Wright Gerhard Wieck Donald McLachlan Scott M, Sorrentino Francisco Roque btw, there a lot more than 'noexec_user_stack' that you can get values for from mdb... Christopher L. Barnard ------------------- comment your code as if the maintainer is a homicidal maniac who knows where you live. _______________________________________________ sunmanagers mailing list sunmanagers at sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers From niall at makalumedia.com Fri Jul 15 12:19:52 2011 From: niall at makalumedia.com (Niall O Broin) Date: Fri, 15 Jul 2011 18:19:52 +0200 Subject: SUMMARY: Solaris 8, flash archive, & Disksuite In-Reply-To: References: Message-ID: <400E32C4-1ECB-46AF-ADF9-9D43D502B29B@makalumedia.com> I asked: > I am trying to clone a V880 running Solaris 8 2/02 which has 6 disks and > uses Disksuite. I create the flar, and do the install, which proceeds as > expected, until it's time to reboot, at which point it all goes sadly wrong, > as it fails to mount the root filesystem with this error message: > > panic[cpu2]/thread=10408000: vfs_mountroot: cannot mount root > > I guess it's trying to mount the root fs on a meta device which isn't there. > How do I go about fixing this? Can I boot from the install CD and create the > meta devices ? I got a couple of answers, but nobody quite got the problem, though one reply did inspire me to look at /etc/system, which was the source of my woes - there was a rootdev entry there pointing to a non existent (yet) metadevice. I removed that entry and I could then boot and start recreating the metadevices. I may have to turn this into a Jumpstart recipe, so it's good to know. __ Kindest regards, Niall O Broin MakaluMedia Group | http://makalumedia.com _______________________________________________ sunmanagers mailing list sunmanagers at sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers From Kennedy.Doss at sanofi-aventis.com Mon Jul 18 11:19:25 2011 From: Kennedy.Doss at sanofi-aventis.com (Kennedy.Doss at sanofi-aventis.com) Date: Mon, 18 Jul 2011 11:19:25 -0400 Subject: SUMMARY: Recognizing EMC Clariion LUNs on a SOLARIS 10 Server Message-ID: Sun Admins: I got it working with several useful suggestions from the group members. Thanks a billion for that. Here is what I had to do: # powermt display dev=all Pseudo name=emcpower2a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170022F679145BADE011 [LUN 36] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016141E01079d0s0 SP A1 active alive 0 0 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016941E01079d0s0 SP B1 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016041E01079d0s0 SP A0 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016841E01079d0s0 SP B0 active alive 0 0 Pseudo name=emcpower0a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170023F679145BADE011 [LUN 37] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3073 UNKNOWN unknown SP A0 active alive 0 0 3074 UNKNOWN unknown SP B1 active alive 0 0 3073 UNKNOWN unknown SP B0 active alive 0 0 3074 UNKNOWN unknown SP A1 active alive 0 0 Pseudo name=emcpower1a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170024F679145BADE011 [LUN 38] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3073 UNKNOWN unknown SP A0 active alive 0 0 3074 UNKNOWN unknown SP B1 active alive 0 0 3074 UNKNOWN unknown SP A1 active alive 0 0 3073 UNKNOWN unknown SP B0 active alive 0 0 # # cd /etc # mv powermt.custom powermt.custom.0rig # cd /kernel/drv # mv emcp.conf emcp.conf.0rig # devfsadm -C # powermt config # powercf -q [To rebuild Powerpath logical links] # powermt display dev=all Pseudo name=emcpower2a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170022F679145BADE011 [LUN 36] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016141E01079d0s0 SP A1 active alive 0 0 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016941E01079d0s0 SP B1 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016041E01079d0s0 SP A0 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016841E01079d0s0 SP B0 active alive 0 0 Pseudo name=emcpower0a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170023F679145BADE011 [LUN 37] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016141E01079d2s0 SP A1 active alive 0 0 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016941E01079d2s0 SP B1 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016041E01079d2s0 SP A0 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016841E01079d2s0 SP B0 active alive 0 0 Pseudo name=emcpower1a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170024F679145BADE011 [LUN 38] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016141E01079d1s0 SP A1 active alive 0 0 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016941E01079d1s0 SP B1 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016041E01079d1s0 SP A0 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016841E01079d1s0 SP B0 active alive 0 0 # powermt save # Kennedy G. Doss (C K-Force, Consultant for: Regional Service Delivery Americas)&Application Hosting Services GIS (C Global Infrastructure Services TEL: 908 (C 981 (C 5295 )&CELL: 732 (C 371 (C 2887 55 Corporate Drive (C Bridgewater (C New Jersey [cid:image001.jpg at 01CC453C.8B3878B0] (9 Please consider the environment before printing this email From: Doss, Kennedy G. PH/US/EXT Sent: Monday, July 18, 2011 8:33 AM To: sunmanagers at sunmanagers.org Subject: Recognizing EMC Clariion LUNs on a SOLARIS 10 Server Importance: High Sun Admins: This my first experience with configuring Solaris 10 with CLARiiON disks. I have presented 3 disks from the CLARiiON Array to the Sun Solaris Host. I have loaded EMC's PowerPath. I tried to run a "powermt display" and only 1 out of the 3 disks show valid paths - though all the three disks are marked active. Is there something that I am missing on the server side? Any help from your end would be most appreciated. # powermt display dev=all Pseudo name=emcpower2a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170022F679145BADE011 [LUN 36] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016141E01079d0s0 SP A1 active alive 0 0 3074 pci at 1e,600000/SUNW,emlxs at 2/fp at 0,0 c2t5006016941E01079d0s0 SP B1 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016041E01079d0s0 SP A0 active alive 0 0 3073 pci at 1e,600000/SUNW,emlxs at 2,1/fp at 0,0 c3t5006016841E01079d0s0 SP B0 active alive 0 0 Pseudo name=emcpower0a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170023F679145BADE011 [LUN 37] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3073 UNKNOWN unknown SP A0 active alive 0 0 3074 UNKNOWN unknown SP B1 active alive 0 0 3073 UNKNOWN unknown SP B0 active alive 0 0 3074 UNKNOWN unknown SP A1 active alive 0 0 Pseudo name=emcpower1a CLARiiON ID=APM00063105053 [SOLARIS10_SLC_TESTING] Logical device ID=600601601B50170024F679145BADE011 [LUN 38] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ============================================================================== 3073 UNKNOWN unknown SP A0 active alive 0 0 3074 UNKNOWN unknown SP B1 active alive 0 0 3074 UNKNOWN unknown SP A1 active alive 0 0 3073 UNKNOWN unknown SP B0 active alive 0 0 # Kennedy G. Doss (C K-Force, Consultant for: Regional Service Delivery Americas)&Application Hosting Services GIS (C Global Infrastructure Services TEL: 908 (C 981 (C 5295 )&CELL: 732 (C 371 (C 2887 55 Corporate Drive (C Bridgewater (C New Jersey [cid:image001.jpg at 01CC453C.8B3878B0] (9 Please consider the environment before printing this email [demime 1.01b removed an attachment of type image/jpeg which had a name of image001.jpg] _______________________________________________ sunmanagers mailing list sunmanagers at sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers From dariusz.dolecki at gmail.com Tue Jul 19 08:45:24 2011 From: dariusz.dolecki at gmail.com (Dariusz Dolecki) Date: Tue, 19 Jul 2011 07:45:24 -0500 Subject: SUMMARY:psu and pbrun Message-ID: Just looked up the man page of psu - thank God there was one and took it from there....... _______________________________________________ sunmanagers mailing list sunmanagers at sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers