Came in this morning to find several users experiencing long delays when logging in. Turns out it's my ZCM server causing the issue: the CPU stays continually at 100%, with the main culprits being jsvc and dbsrv12. This server has been in use for 2+ years and was behaving normally yesterday. (At least the login delays weren't happening.)
Plenty of RAM:
free -m
total used free shared buffers cached
Mem: 3979 2330 1648 0 91 542
-/+ b/c: 1696 2283
Swap: 2055 0 2055
Server's running 11.2.3 and has been - I did no upgrades or updates or anything at all on this server between yesterday and today, though it does have ZPM, so some stuff may have come down the pipe from that that's causing problems - my knowledge of ZCM and it's plumbing is still pretty shallow.
ZCC is also darn near unresponsive - 2-3 minutes before clicks respond, but I don't usually get errors in the browser...just an endlessly spinning progress meter.
I've searched around and found lots of similar messages, but they all led to issues that were solved before 11.2. Does anyone have any experience with sort of thing or an idea of something I could try? In case it helps, here's the end of my loader-messages.log - looks like some errors, but I'm not sure what matters and what doesn't:
[DEBUG] [11/22/13 3:00:11 PM] [] [PatchLicense] [230] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:00:12 PM] [] [PatchLicense] [230] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:00:14 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n: com.novell.zenworks.datamodel.exceptions.Connectio nException: edir.mvschool.com:636] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_e94057940cc21f4c9de87ea8bbb32258_1385129790 070.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_e94057940cc21f4c9de87ea8bbb32258_1385133390 122.xml] [] []
[DEBUG] [11/22/13 3:01:15 PM] [] [Location.Configuration] [140] [] [Sucessfully updated Location Configuration Response for ZENObject : e74cddd5adf380973aaa73b6641108a3] [] []
[DEBUG] [11/22/13 3:01:15 PM] [] [Location.Configuration] [140] [] [Finished location configuration computation in (ms): 64982for ZENObject : e74cddd5adf380973aaa73b6641108a3] [] []
[DEBUG] [11/22/13 3:01:17 PM] [] [Location.Configuration.Module] [140] [] [Successfully computed location responses] [] []
[DEBUG] [11/22/13 3:01:47 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_e94057940cc21f4c9de87ea8bbb32258_1385133390 122.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_1372f04b8406d3428f39cfc9530ed471_1385080801 000.xml] [] []
[DEBUG] [11/22/13 3:03:11 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385133521 242.xml] [] []
[DEBUG] [11/22/13 3:04:24 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385128120 821.xml] [] []
[DEBUG] [11/22/13 3:05:04 PM] [] [PatchLicense] [141] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:05:05 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n] [] []
[DEBUG] [11/22/13 3:05:06 PM] [] [PatchLicense] [141] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [deleting oldest file from failed directory:Status_fc73b65259f9b145ab29038d047b4914_ 1380028523418.xml] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_3abd6a308baf084f9052f997c832b148_1385128120 821.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385129921 177.xml] [] []
[DEBUG] [11/22/13 3:08:41 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385128413 854.xml] [] []
[DEBUG] [11/22/13 3:10:19 PM] [] [PatchLicense] [318] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:10:25 PM] [] [PatchLicense] [318] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:12:20 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385130213 906.xml] [] []
[DEBUG] [11/22/13 3:14:35 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385080800 675.xml] [] []
[DEBUG] [11/22/13 3:17:07 PM] [] [PatchLicense] [140] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:17:13 PM] [] [Content Datamodel] [318] [] [select distinct c.ZUID from zContent c where (c.Replicate=:repSpec and exists (select 1 from zContentServers cps where cps.
id = c.ZUID and cps.Servers = :serverUID) or c.Replicate=:repExcept and not exists (select cps1.Servers from zContentServers cps1 where cps1.id = c.ZUID and cps1.Servers = :serverUID) or
c.Replicate=:repAll or c.Replicate=:repPrimary) and c.AliasedContentUID is null and (c.MetaVersion>0 or c.AliasedContentUID IS NOT NULL) and not exists (select 1 from zContentSyncState
cs where cs.ServerUID=:serverUID and (cs.SyncState='AVAILABLE' or cs.SyncState='PACKAGING') and (cs.ContentUID=c.ZUID or cs.ContentUID=c.AliasedContentUID))] [] []
[DEBUG] [11/22/13 3:17:13 PM] [] [Content Datamodel] [318] [] [serverUID = a0a5e7baa9f2394e549abdcd70c2b5d5 , isSQLQuery = true] [] []
[DEBUG] [11/22/13 3:17:20 PM] [] [PatchLicense] [140] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:17:43 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385132013 921.xml] [] []
Plenty of RAM:
free -m
total used free shared buffers cached
Mem: 3979 2330 1648 0 91 542
-/+ b/c: 1696 2283
Swap: 2055 0 2055
Server's running 11.2.3 and has been - I did no upgrades or updates or anything at all on this server between yesterday and today, though it does have ZPM, so some stuff may have come down the pipe from that that's causing problems - my knowledge of ZCM and it's plumbing is still pretty shallow.
ZCC is also darn near unresponsive - 2-3 minutes before clicks respond, but I don't usually get errors in the browser...just an endlessly spinning progress meter.
I've searched around and found lots of similar messages, but they all led to issues that were solved before 11.2. Does anyone have any experience with sort of thing or an idea of something I could try? In case it helps, here's the end of my loader-messages.log - looks like some errors, but I'm not sure what matters and what doesn't:
[DEBUG] [11/22/13 3:00:11 PM] [] [PatchLicense] [230] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:00:12 PM] [] [PatchLicense] [230] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:00:14 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n: com.novell.zenworks.datamodel.exceptions.Connectio nException: edir.mvschool.com:636] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_e94057940cc21f4c9de87ea8bbb32258_1385129790 070.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:00:31 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_e94057940cc21f4c9de87ea8bbb32258_1385133390 122.xml] [] []
[DEBUG] [11/22/13 3:01:15 PM] [] [Location.Configuration] [140] [] [Sucessfully updated Location Configuration Response for ZENObject : e74cddd5adf380973aaa73b6641108a3] [] []
[DEBUG] [11/22/13 3:01:15 PM] [] [Location.Configuration] [140] [] [Finished location configuration computation in (ms): 64982for ZENObject : e74cddd5adf380973aaa73b6641108a3] [] []
[DEBUG] [11/22/13 3:01:17 PM] [] [Location.Configuration.Module] [140] [] [Successfully computed location responses] [] []
[DEBUG] [11/22/13 3:01:47 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_e94057940cc21f4c9de87ea8bbb32258_1385133390 122.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:02:05 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_1372f04b8406d3428f39cfc9530ed471_1385080801 000.xml] [] []
[DEBUG] [11/22/13 3:03:11 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385133521 242.xml] [] []
[DEBUG] [11/22/13 3:04:24 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385128120 821.xml] [] []
[DEBUG] [11/22/13 3:05:04 PM] [] [PatchLicense] [141] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:05:05 PM] [] [Loader.Status Storer Module] [44] [] [Error reading ObjectInfo Exception: com.novell.zenworks.datamodel.exceptions.Authorita tiveSourceConnectionExceptio
n] [] []
[DEBUG] [11/22/13 3:05:06 PM] [] [PatchLicense] [141] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Failed directory size:10.0 MB] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [deleting oldest file from failed directory:Status_fc73b65259f9b145ab29038d047b4914_ 1380028523418.xml] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Moved Status_3abd6a308baf084f9052f997c832b148_1385128120 821.xml to failed directory.] [] []
[DEBUG] [11/22/13 3:05:26 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_3abd6a308baf084f9052f997c832b148_1385129921 177.xml] [] []
[DEBUG] [11/22/13 3:08:41 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385128413 854.xml] [] []
[DEBUG] [11/22/13 3:10:19 PM] [] [PatchLicense] [318] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:10:25 PM] [] [PatchLicense] [318] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:12:20 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385130213 906.xml] [] []
[DEBUG] [11/22/13 3:14:35 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385080800 675.xml] [] []
[DEBUG] [11/22/13 3:17:07 PM] [] [PatchLicense] [140] [] [Entering VerifyLicense] [] []
[DEBUG] [11/22/13 3:17:13 PM] [] [Content Datamodel] [318] [] [select distinct c.ZUID from zContent c where (c.Replicate=:repSpec and exists (select 1 from zContentServers cps where cps.
id = c.ZUID and cps.Servers = :serverUID) or c.Replicate=:repExcept and not exists (select cps1.Servers from zContentServers cps1 where cps1.id = c.ZUID and cps1.Servers = :serverUID) or
c.Replicate=:repAll or c.Replicate=:repPrimary) and c.AliasedContentUID is null and (c.MetaVersion>0 or c.AliasedContentUID IS NOT NULL) and not exists (select 1 from zContentSyncState
cs where cs.ServerUID=:serverUID and (cs.SyncState='AVAILABLE' or cs.SyncState='PACKAGING') and (cs.ContentUID=c.ZUID or cs.ContentUID=c.AliasedContentUID))] [] []
[DEBUG] [11/22/13 3:17:13 PM] [] [Content Datamodel] [318] [] [serverUID = a0a5e7baa9f2394e549abdcd70c2b5d5 , isSQLQuery = true] [] []
[DEBUG] [11/22/13 3:17:20 PM] [] [PatchLicense] [140] [] [Returning valid ] [] []
[DEBUG] [11/22/13 3:17:43 PM] [] [Loader.Status Storer Module] [44] [] [Processing Status_c65f20e2dee6a94a93cddbabcb9f9caa_1385132013 921.xml] [] []