Tracking down frequent (every 20 seconds) SCSI hard disk drive access in 10.7

In a Lion Server (10.7.3 and 10.7.4 tried) system with multiple drives attached to an HPT SATA controller, there is some process that keeps accessing the hard disk drives exactly every 20 seconds. As a result the drives will never sleep (and power down).

The drive access pattern is not in parallel, but sequential within a 1 - 2 second time frame.

Resolution

  1. Even when the disks are unmounted and ejected by diskutil, the disk access continues at a rate of 3 times a minute.
  2. iosnoop doesn't give a clue which process might be causing this frequent disk access. There are lots of lines like:

    0     1 W    47880   4096    launchd ??/<unknown (NULL v_parent)>/<unknown (NULL v_name)>
    dtrace: error on enabled probe ID 4 (ID 992: io:mach_kernel:buf_strategy:start): illegal operation in action #3 at DIF offset 0
    0     1 W  2566032   8192    launchd ??/<unknown (NULL v_parent)>/<unknown (NULL v_name)>
    0     1 W  2567040   8192    launchd ??/<unknown (NULL v_parent)>/<unknown (NULL v_name)>
    

S.M.A.R.T. reading is "Not Supported", see diskutil info:

$ diskutil info disk6
   Device Identifier:        disk6
   Device Node:              /dev/disk6
   Part of Whole:            disk6
   Device / Media Name:      HPT DISK 1_4 Media

   Volume Name:              Not applicable (no file system)

   Mounted:                  Not applicable (no file system)

   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SCSI
   SCSI Domain ID:           0
   SCSI Target ID:           4
   SCSI LUN:                 0
   SMART Status:             Not Supported

   Total Size:               1.0 TB (1000204886016 Bytes) (exactly 1953525168 512-Byte-Blocks)
   Volume Free Space:        Not applicable (no file system)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)
   Ejectable:                Yes

   Whole:                    Yes
   Internal:                 No
   OS 9 Drivers:             No
   Low Level Format:         Not supported

Strange is that iostat doesn't track any disk activity, even though drive activity LED's turn on, and hearing a tiny amount of drive activity. In the output below is disk6 unmounted ejected and disk7 mounted.

$ iostat -dw 1 disk{6,7}
         disk6           disk7 
    KB/t tps  MB/s     KB/t tps  MB/s 
    6.06   0  0.00     6.39   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
          disk6           disk7 
    KB/t tps  MB/s     KB/t tps  MB/s 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00 
    0.00   0  0.00     0.00   0  0.00

Launchctl list after stopping the most notorious disk writing daemons:

$ sudo launchctl list
Password:
PID Status  Label
3257    -   0x7fe6a3611ba0.anonymous.SecurityAgent
3252    -   0x7fe6a3611580.anonymous.ManagedClient
-   0   com.apple.ZoomWindow
-   0   com.apple.VoiceOver
-   0   com.apple.UserNotificationCenterAgent-LoginWindow
3253    -   com.apple.UserEventAgent-LoginWindow
-   0   com.apple.tiswitcher
-   0   com.apple.storeagent
-   0   com.apple.store_helper
-   0   com.apple.scrod
-   0   com.apple.screensharing.agent
-   0   com.apple.ScreenReaderUIServer
3254    -   com.apple.RemoteDesktop.agent
-   0   com.apple.netauth.user.gui
-   0   com.apple.isst
-   0   com.apple.imklaunchagent
-   0   com.apple.iCalPush
-   0   com.apple.coreservices.appleid.authentication
-   0   com.apple.btsa
-   0   com.apple.BezelUIServer
-   0   com.apple.AppleGraphicsWarning
-   0   com.apple.launchctl.LoginWindow
1   -   0x7fe6a360a900.anonymous.launchd
3246    -   0x7fe6a3605200.anonymous.loginwindow
-   0   com.apple.AppSandboxSMLoginItemEnabler
-   0   com.apple.security.pboxd
-   0   com.apple.cmio.registerassistantservice
-   0   com.apple.appkit.xpc.sandboxedServiceRunner
-   0   com.apple.automator.xpc.workflowServiceRunner
-   0   com.apple.hiservices-xpcservice
-   0   com.apple.cfpreferences-xpcservice
-   0   com.apple.DataDetectors.DataDetectorsActionService
-   0   com.apple.security.XPCKeychainSandboxCheck
-   0   com.apple.qtkitserver
-   0   com.apple.XType.FontHelper
-   0   com.apple.HasTRB
1   -   0x7fe6a342ed10.anonymous.launchd
3246    -   0x7fe6a342a450.anonymous.loginwindow
1   -   0x7fe6a3444940.anonymous.launchd
554 -   0x7fe6a3444630.anonymous.filecoordinatio
1   -   0x7fe6a342ea10.anonymous.launchd
38  -   0x7fe6a342e710.anonymous.distnoted
3485    -   0x7fe6a3435dd0.anonymous.sudo
3488    -   0x7fe6a3432e70.anonymous.launchctl
3484    -   0x7fe6a3609350.anonymous.sleep
3252    -   0x7fe6a3611890.anonymous.ManagedClient
3257    -   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186B2
3115    -   org.postgresql.postgres
-   0   org.postfix.master
1288    -   0x7fe6a3603dd0.anonymous.sshd
1289    -   0x7fe6a3609650.anonymous.bash
1285    -   0x7fe6a343bbf0.anonymous.sshd
1169    -   0x7fe6a3610520.anonymous.ipfwloggerd
-   0   com.apple.authorizationhost.00000000-0000-0000-0000-0000000186B1
-   0   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186B1
-   0   com.apple.launchd.peruser.93
-   0   com.apple.authorizationhost.00000000-0000-0000-0000-0000000186A8
-   0   com.apple.xpchelper.F5010000-0000-0000-0000-000000000000
3486    -   com.apple.launchd.peruser.501
-   0   com.apple.launchd.peruser.92
-   0   com.apple.xpchelper.59000000-0000-0000-0000-000000000000
-   0   com.apple.launchd.peruser.89
-   0   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186A8
-   0   com.apple.launchd.peruser.212
-   0   com.apple.launchd.peruser.70
1   -   0x7fe6a3438830.anonymous.launchd
230 -   0x7fe6a3438520.anonymous.raidman-httpsd
-   0   com.apple.launchd.peruser.222
-   0   com.apple.xpchelper.DE000000-0000-0000-0000-000000000000
174 -   0x7fe6a3600cc0.anonymous.sh
-   0   com.apple.launchd.peruser.84
163 -   0x7fe6a3433170.anonymous.s2s
161 -   0x7fe6a3432b70.anonymous.c2s
159 -   0x7fe6a3432570.anonymous.router
158 -   0x7fe6a3432270.anonymous.sm
-   0   com.apple.launchd.peruser.220
-   0   com.openssh.sshd
-   0   com.apple.screensharing
-   0   com.apple.odproxyd
-   0   com.apple.msrpc.wkssvc
-   0   com.apple.msrpc.srvsvc
-   0   com.apple.msrpc.netlogon
-   0   com.apple.msrpc.mdssvc
-   0   com.apple.msrpc.lsarpc
-   0   com.apple.launchd.peruser.0
-   0   com.apple.xpchelper.00000000-0000-0000-0000-000000000000
-   0   org.x.privileged_startx
81  -   org.openldap.slapd
82  -   org.ntp.ntpd
83  -   org.jabber.jabberd_notification
-   0   org.cups.cupsd
-   62  org.clamav.freshclam-init
-   0   com.vix.cron
-   0   com.apple.xserve.serial-ports
-   0   com.apple.xscertd
-   0   com.apple.xscertd-helper
-   0   com.apple.xscertadmin
-   0   com.apple.xpchelper
-   0   com.apple.xgridd.pcastserver
3249    -   com.apple.WindowServer
-   0   com.apple.webdavfs_load_kext
-   0   com.apple.warmd
-   0   com.apple.vsdbutil
-   0   com.apple.var-db-shadow-backup
-   0   com.apple.var-db-dslocal-backup
-   0   com.apple.UserNotificationCenter
30  -   com.apple.UserEventAgent-System
-   0   com.apple.unmountassistant.sysagent
-   0   com.apple.uninstalld
-   0   com.apple.ucupdate.plist
-   0   com.apple.TrustEvaluationAgent.system
-   0   com.apple.taskgated
-   0   com.apple.taskgated-helper
93  -   com.apple.SystemStarter
-   0   com.apple.systempreferences.writeconfig
-   0   com.apple.systempreferences.install
-   0   com.apple.systemkeychain
36  -   com.apple.syslogd
-   0   com.apple.suhelperd
-   0   com.apple.storereceiptinstaller
-   0   com.apple.statd.notify
95  -   com.apple.stackshot
-   0   com.apple.spindump_symbolicator
-   0   com.apple.spindump
-   0   com.apple.softwareupdatecheck.periodic
-   0   com.apple.softwareupdatecheck.initial
-   0   com.apple.smbfs_load_kext
-   0   com.apple.smb.preferences
-   0   com.apple.shutdown_monitor
96  -   com.apple.ServerPerfLog
46  -   com.apple.securityd
-   0   com.apple.SecurityAgent
-   0   com.apple.security.syspolicy
-   0   com.apple.scsid
-   0   com.apple.SCHelper
-   0   com.apple.sandboxd
-   0   com.apple.salearn
-   0   com.apple.rpcbind
-   0   com.apple.RFBEventHelper
-   0   com.apple.ReportCrash.Root
-   0   com.apple.RemoteDesktop.PrivilegeProxy
-   0   com.apple.racoon
-   0   com.apple.printtool.daemon
-   0   com.apple.preferences.timezone.auto
-   0   com.apple.preferences.timezone.admintool
37  -   com.apple.powerd
-   0   com.apple.platform.ptmd
-   0   com.apple.pfctl
-   0   com.apple.periodic-weekly
-   0   com.apple.periodic-monthly
-   0   com.apple.periodic-daily
-   0   com.apple.PCIELaneConfigTool
-   0   com.apple.pcastagentconfigd
101 -   com.apple.PasswordService
1089    -   com.apple.opendirectoryd
-   0   com.apple.opendirectorybackup
-   0   com.apple.ocspd
33  -   com.apple.notifyd
102 -   com.apple.NotificationServer
-   0   com.apple.nis.ypbind
-   0   com.apple.nfsd
-   0   com.apple.newsyslog
-   0   com.apple.nlcd
-   0   com.apple.netauth.sys.gui
-   0   com.apple.netauth.sys.auth
-   0   com.apple.metadata.mds.spindump
-   0   com.apple.metadata.mds.scan
-   0   com.apple.mDNSResponderHelper
31  -   com.apple.mDNSResponder
-   0   com.apple.mdmclient.daemon
-   0   com.apple.ManagedClient
3246    -   com.apple.loginwindow
249 -   com.apple.logind
-   0   com.apple.locum
-   0   com.apple.lockd
-   0   com.apple.locate
-   0   com.apple.kuncd
29  -   com.apple.kextd
108 -   com.apple.KernelEventAgent
109 -   com.apple.Kerberos.kpasswdd
110 -   com.apple.Kerberos.kdc
-   0   com.apple.Kerberos.kcm
111 -   com.apple.Kerberos.kadmind
-   0   com.apple.Kerberos.digest-service
-   0   com.apple.kcproxy
-   0   com.apple.installd
-   0   com.apple.IFCStart
114 -   com.apple.hidd
-   0   com.apple.HeadlessStartup
-   0   com.apple.hdiejectd
-   0   com.apple.gssd
41  -   com.apple.fseventsd
-   0   com.apple.FontWorker
-   0   com.apple.fontmover
3263    -   com.apple.fontd
-   0   com.apple.findmymacmessenger
-   0   com.apple.findmymacd
554 -   com.apple.FileCoordination
3489    -   com.apple.emlog
-   0   com.apple.efilogin-helper
-   0   com.apple.eapolcfg_auth
118 -   com.apple.dynamic_pager
-   0   com.apple.dvdplayback.setregion
-   0   com.apple.DumpPanic
-   0   com.apple.DumpGPURestart
-   0   com.apple.dspluginhelperd
-   0   com.apple.dpd
38  -   com.apple.distnoted.xpc.daemon
-   0   com.apple.diskmanagementd
34  -   com.apple.diskarbitrationd
-   0   com.apple.DiagnosticReportCleanUp
235 -   com.apple.cvmsServ
-   0   com.apple.corestorage.corestoragehelperd
-   0   com.apple.corestorage.corestoraged
50  -   com.apple.coreservicesd
-   0   com.apple.coreservices.appleid.passwordcheck
-   0   com.apple.configureLocalKDC
35  -   com.apple.configd
-   0   com.apple.collabd.quicklook
-   0   com.apple.collabd.expire
-   0   com.apple.cmio.VDCAssistant
-   0   com.apple.cmio.IIDCVideoAssistant
-   0   com.apple.cmio.AVCAssistant
-   0   com.apple.bsd.launchdadd
-   0   com.apple.bsd.dirhelper
-   0   com.apple.bnepd
-   0   com.apple.blued
-   0   com.apple.backupd
-   0   com.apple.awacsd
-   0   com.apple.avbdeviced
-   0   com.apple.automountd
124 -   com.apple.autofsd
-   0   com.apple.authorizationhost
-   0   com.apple.auditd
347 -   com.apple.audio.coreaudiod
-   0   com.apple.aslmanager
-   0   com.apple.applepushserviced
-   0   com.apple.appleprofilepolicyd
126 -   com.apple.APNBridge
-   0   com.apple.AOSNotification-FMM
-   0   com.apple.alf
-   0   com.apple.airport.updateprefs
-   0   com.apple.airportd
-   0   com.apple.AirPort.wps
-   0   com.apple.afpfs_checkafp
-   0   com.apple.afpfs_afpLoad
-   0   com.apple.afctl
-   0   com.apple.ActivityMonitor
-   0   com.apple.AccountsConfigService
-   0   com.fernlightning.fseventer
-   0   com.apple.launchctl.System

Fs_usage gives almost every 20 seconds these repeating patterns:

$ sudo fs_usage -w
...
19:57:12.973849  exit                                                                                                             0.000027   sleep.37168
19:57:12.974393  select                   S=1                                                                                     0.000239 W launchd.166
19:57:12.974779  close             F=255                                                                                          0.000003   .37199
19:57:12.974860  dup2              F=4    F=1                                                                                     0.000003   .37199
19:57:12.974861  close             F=4                                                                                            0.000001   .37199
19:57:12.974867  close             F=3                                                                                            0.000001   .37199
19:57:12.975414  close             F=4                                                                                            0.000002   .37199
19:57:12.975460  close             F=4  [  9]                                                                                     0.000001   .37199
19:57:12.975747  close             F=3                                                                                            0.000002   sh.37200
19:57:12.975748  close             F=3                                                                                            0.000002   .37199
19:57:12.975751  dup2              F=4    F=1                                                                                     0.000003   sh.37200
19:57:12.975752  close             F=4                                                                                            0.000001   sh.37200
19:57:12.976252  dup2              F=3    F=0                                                                                     0.000003   sh.37201
19:57:12.976254  close             F=3                                                                                            0.000001   sh.37201
19:57:12.976527  execve                                 /bin/ps                                                                   0.000444   sh.37200
...
19:57:13.001362  stat64                 [  2]           /AppleInternal                                                            0.000007   sleep.37202
19:57:33.002940  exit                                                                                                             0.000030   sleep.37202
19:57:33.002960  select                   S=1                                                                                    20.004119 W launchd.166
19:57:33.003463  select                   S=1                                                                                     0.000350 W launchd.166
19:57:33.003841  close             F=255                                                                                          0.000003   .37204
19:57:33.003921  dup2              F=4    F=1                                                                                     0.000003   .37204
19:57:33.003923  close             F=4                                                                                            0.000001   .37204
19:57:33.003929  close             F=3                                                                                            0.000001   .37204
19:57:33.004457  close             F=4                                                                                            0.000002   .37204
19:57:33.004502  close             F=4  [  9]                                                                                     0.000001   .37204
19:57:33.004788  close             F=3                                                                                            0.000002   sh.37205
19:57:33.004788  close             F=3                                                                                            0.000002   .37204
19:57:33.004791  dup2              F=4    F=1                                                                                     0.000003   sh.37205
19:57:33.004792  close             F=4                                                                                            0.000001   sh.37205
19:57:33.005270  dup2              F=3    F=0                                                                                     0.000003   sh.37206
19:57:33.005272  close             F=3                                                                                            0.000001   sh.37206
19:57:33.005571  execve                                 /bin/ps                                                                   0.000449   sh.37205
...
19:57:53.030653  exit                                                                                                             0.000027   sleep.37207
19:57:53.030668  select                   S=1                                                                                    14.079398 W launchd.166
19:57:53.031187  select                   S=1                                                                                     0.000360 W launchd.166
19:57:53.031576  close             F=255                                                                                          0.000003   .37225
19:57:53.031655  dup2              F=4    F=1                                                                                     0.000003   .37225
19:57:53.031657  close             F=4                                                                                            0.000001   .37225
19:57:53.031663  close             F=3                                                                                            0.000001   .37225
19:57:53.032191  close             F=4                                                                                            0.000002   .37225
19:57:53.032235  close             F=4  [  9]                                                                                     0.000001   .37225
19:57:53.032519  close             F=3                                                                                            0.000002   sh.37226
19:57:53.032522  dup2              F=4    F=1                                                                                     0.000002   sh.37226
19:57:53.032523  close             F=4                                                                                            0.000001   sh.37226
19:57:53.032526  close             F=3                                                                                            0.000002   .37225
19:57:53.033006  dup2              F=3    F=0                                                                                     0.000003   sh.37227
19:57:53.033008  close             F=3                                                                                            0.000001   sh.37227
19:57:53.033298  execve                                 /bin/ps                                                                   0.000443   sh.37226
...
19:58:13.058381  exit                                                                                                             0.000028   sleep.37228
19:58:13.058397  select                   S=1                                                                                     7.053934 W launchd.166
19:58:13.058916  select                   S=1                                                                                     0.000339 W launchd.166
19:58:13.059297  close             F=255                                                                                          0.000002   .37251
19:58:13.059378  dup2              F=4    F=1                                                                                     0.000003   .37251
19:58:13.059380  close             F=4                                                                                            0.000001   .37251
19:58:13.059386  close             F=3                                                                                            0.000001   .37251
19:58:13.059923  close             F=4                                                                                            0.000002   .37251
19:58:13.059966  close             F=4  [  9]                                                                                     0.000001   .37251
19:58:13.060247  close             F=3                                                                                            0.000002   .37251
19:58:13.060251  close             F=3                                                                                            0.000002   sh.37252
19:58:13.060254  dup2              F=4    F=1                                                                                     0.000003   sh.37252
19:58:13.060256  close             F=4                                                                                            0.000001   sh.37252
19:58:13.060723  dup2              F=3    F=0                                                                                     0.000004   sh.37253
19:58:13.060724  close             F=3                                                                                            0.000001   sh.37253
19:58:13.061024  execve                                 /bin/ps                                                                   0.000434   sh.37252
...

The drives are still being accessed every 20 seconds, even after having unloaded almost every launchd item:

$ sudo launchctl list
PID Status  Label
-   0   com.apple.appkit.xpc.sandboxedServiceRunner
-   0   com.apple.hiservices-xpcservice
-   0   com.apple.security.XPCKeychainSandboxCheck
-   0   com.apple.cfpreferences-xpcservice
-   0   com.apple.cmio.registerassistantservice
7194    -   0x7fe6a4c10190.anonymous.sudo
7195    -   0x7fe6a4c0fcf0.anonymous.launchctl
-   0   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186B2
-   0   org.postfix.master
1288    -   0x7fe6a3603dd0.anonymous.sshd
1289    -   0x7fe6a3609650.anonymous.bash
1285    -   0x7fe6a343bbf0.anonymous.sshd
-   0   com.apple.authorizationhost.00000000-0000-0000-0000-0000000186B1
-   0   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186B1
-   0   com.apple.launchd.peruser.93
-   0   com.apple.authorizationhost.00000000-0000-0000-0000-0000000186A8
-   0   com.apple.xpchelper.F5010000-0000-0000-0000-000000000000
7184    -   com.apple.launchd.peruser.501
-   0   com.apple.launchd.peruser.92
-   0   com.apple.xpchelper.59000000-0000-0000-0000-000000000000
-   0   com.apple.launchd.peruser.89
-   0   com.apple.SecurityAgent.00000000-0000-0000-0000-0000000186A8
-   0   com.apple.launchd.peruser.212
-   0   com.apple.launchd.peruser.70
1   -   0x7fe6a3438830.anonymous.launchd
-   0   com.apple.launchd.peruser.222
-   0   com.apple.xpchelper.DE000000-0000-0000-0000-000000000000
-   0   com.apple.launchd.peruser.84
-   0   com.apple.launchd.peruser.220
-   0   com.openssh.sshd
-   0   com.apple.screensharing
-   0   com.apple.odproxyd
-   0   com.apple.msrpc.wkssvc
-   0   com.apple.msrpc.srvsvc
-   0   com.apple.msrpc.mdssvc
-   0   com.apple.msrpc.lsarpc
-   0   com.apple.launchd.peruser.0
-   0   com.apple.xpchelper.00000000-0000-0000-0000-000000000000
-   0   org.x.privileged_startx
-   0   org.cups.cupsd
-   62  org.clamav.freshclam-init
-   0   com.vix.cron
-   0   com.apple.xserve.serial-ports
-   0   com.apple.xscertd
-   0   com.apple.xscertd-helper
-   0   com.apple.xscertadmin
-   0   com.apple.xpchelper
-   0   com.apple.xgridd.pcastserver
-   0   com.apple.webdavfs_load_kext
-   0   com.apple.warmd
-   0   com.apple.vsdbutil
-   0   com.apple.var-db-shadow-backup
-   0   com.apple.var-db-dslocal-backup
-   0   com.apple.UserNotificationCenter
-   0   com.apple.unmountassistant.sysagent
-   0   com.apple.uninstalld
-   0   com.apple.ucupdate.plist
-   0   com.apple.TrustEvaluationAgent.system
-   0   com.apple.taskgated-helper
-   0   com.apple.systempreferences.writeconfig
-   0   com.apple.systempreferences.install
-   0   com.apple.systemkeychain
-   0   com.apple.suhelperd
-   0   com.apple.storereceiptinstaller
-   0   com.apple.statd.notify
-   0   com.apple.spindump_symbolicator
-   0   com.apple.spindump
-   0   com.apple.softwareupdatecheck.periodic
-   0   com.apple.softwareupdatecheck.initial
-   0   com.apple.smbfs_load_kext
-   0   com.apple.smb.preferences
-   0   com.apple.shutdown_monitor
-   0   com.apple.SecurityAgent
-   0   com.apple.security.syspolicy
-   0   com.apple.scsid
-   0   com.apple.SCHelper
-   0   com.apple.sandboxd
-   0   com.apple.salearn
-   0   com.apple.rpcbind
-   0   com.apple.RFBEventHelper
-   0   com.apple.ReportCrash.Root
-   0   com.apple.RemoteDesktop.PrivilegeProxy
-   0   com.apple.racoon
-   0   com.apple.printtool.daemon
-   0   com.apple.preferences.timezone.auto
-   0   com.apple.preferences.timezone.admintool
-   0   com.apple.platform.ptmd
-   0   com.apple.pfctl
-   0   com.apple.periodic-weekly
-   0   com.apple.periodic-monthly
-   0   com.apple.periodic-daily
-   0   com.apple.PCIELaneConfigTool
-   0   com.apple.pcastagentconfigd
1089    -   com.apple.opendirectoryd
-   0   com.apple.opendirectorybackup
-   0   com.apple.ocspd
-   0   com.apple.nis.ypbind
-   0   com.apple.nfsd
-   0   com.apple.newsyslog
-   0   com.apple.nlcd
-   0   com.apple.netauth.sys.gui
-   0   com.apple.netauth.sys.auth
-   0   com.apple.metadata.mds.spindump
-   0   com.apple.metadata.mds.scan
-   0   com.apple.mDNSResponderHelper
-   0   com.apple.mdmclient.daemon
-   0   com.apple.ManagedClient
-   -9  com.apple.loginwindow
-   0   com.apple.locum
-   0   com.apple.lockd
-   0   com.apple.locate
-   0   com.apple.kuncd
-   0   com.apple.Kerberos.kcm
-   0   com.apple.Kerberos.digest-service
-   0   com.apple.kcproxy
-   0   com.apple.installd
-   0   com.apple.IFCStart
-   0   com.apple.HeadlessStartup
-   0   com.apple.hdiejectd
-   0   com.apple.gssd
-   0   com.apple.FontWorker
-   0   com.apple.fontmover
-   0   com.apple.fontd
-   0   com.apple.findmymacmessenger
-   0   com.apple.findmymacd
-   0   com.apple.emlog
-   0   com.apple.efilogin-helper
-   0   com.apple.eapolcfg_auth
-   0   com.apple.dvdplayback.setregion
-   0   com.apple.DumpPanic
-   0   com.apple.DumpGPURestart
-   0   com.apple.dspluginhelperd
-   0   com.apple.dpd
-   0   com.apple.diskmanagementd
-   0   com.apple.DiagnosticReportCleanUp
-   0   com.apple.corestorage.corestoragehelperd
-   0   com.apple.corestorage.corestoraged
-   0   com.apple.coreservices.appleid.passwordcheck
-   0   com.apple.configureLocalKDC
-   0   com.apple.collabd.quicklook
-   0   com.apple.collabd.expire
-   0   com.apple.cmio.VDCAssistant
-   0   com.apple.cmio.IIDCVideoAssistant
-   0   com.apple.cmio.AVCAssistant
-   0   com.apple.bsd.launchdadd
-   0   com.apple.bsd.dirhelper
-   0   com.apple.bnepd
-   0   com.apple.blued
-   0   com.apple.backupd
-   0   com.apple.awacsd
-   0   com.apple.avbdeviced
-   0   com.apple.automountd
-   0   com.apple.authorizationhost
-   0   com.apple.auditd
-   0   com.apple.aslmanager
-   0   com.apple.applepushserviced
-   0   com.apple.appleprofilepolicyd
-   0   com.apple.AOSNotification-FMM
-   0   com.apple.

Solution 1:

I think you're on the right track with iosnoop - but perhaps fs_usage will get to the nub of which specific files are being accessed.

Start with sudo fs_usage -w and you can grep for things or perhaps find a stable PID to filter on. If the processes are coming and going too rapidly to track, you might need to fire up instruments or dump the launchd database to know what is polling all disks or just that specific disk.

Sorry I can't guess at the cause - but this tool helps me nail down all sorts of IO mysteries on Lion.

Solution 2:

Tracking down the executed commands can bo done with sudo execsnoop -v.

In this case the results were:

2012 May 12 17:05:53     0  15465    161 sleep
2012 May 12 17:06:13     0  15467  15466 ps
2012 May 12 17:06:13     0  15468  15466 awk
2012 May 12 17:06:13     0  15469    161 sleep
2012 May 12 17:06:33     0  15472  15470 awk
2012 May 12 17:06:33     0  15471  15470 ps
2012 May 12 17:06:33     0  15473    161 sleep
2012 May 12 17:06:53     0  15475  15474 ps
2012 May 12 17:06:53     0  15476  15474 awk
2012 May 12 17:06:53     0  15477    161 sleep
2012 May 12 17:07:13     0  15480  15478 awk
2012 May 12 17:07:13     0  15479  15478 ps

Where PID 161 is identified as:

$ sudo ps 161
  PID   TT  STAT      TIME COMMAND
  161   ??  S      0:03.53 /bin/sh /Library/StartupItems/Hptsvr/Hptsvr-alive

After killing process 161 there is longer any output for execsnoop and drive activity LED's are still blinking at a 20 second interval and fs_usage no longer has output at a 20 second interval (only at 30 seconds).

Solution 3:

It is not just lion, it is on every OS I have installed and on multiple laptops and systems I have purchased from various vendors. I haven't actually had a PC that could let the disk sleep for 10 years now. Every PC I have owned has the same problem. Constant disk access at specific intervals but no software debugger can capture or show what is doing the accessing. Unkown...

Some hidden firmware glitch that has been copied a million times or a backdoor placed on all pc platforms sold to the public? I don't have the resources or the equipment to actually do line level tracing. If I had friends in a lab somewhere I would get to the bottom of this.

When all the tools and software developed keep you in the dark it is obviously by design and not a flaw. Another obscured agenda that makes everything worse.