Hello,
We are having problems with one of our netware 6.5 sp8 servers (all patches up to day). We also have another with the same config (hard and soft) and its working fine.
Both are IBM system x3550 M3 with 8GB and 1 E5520 with hyperthreading disabled.
The server with the problems is configured with auto tuning off and file cache maximum size in 2GB. No -u parameter.
Also the command memory shows Total server memory 2GB and extended 6GB in both servers (the one with problems and the fine one).
The error only appears during peek hours (2 to 4~pm) and its only from ADS.NLM.
15/02/2012 4.01.22 : SERVER-5.70-0 [nmID=6001D]
Cache memory allocator out of available memory.
15/02/2012 4.01.22 : SERVER-5.70-0 [nmID=2000A]
Short term memory allocator is out of memory.
3 attempts to get more memory failed.
request size in bytes 33554441 from Module ADS.NLM
Thanks in advance,
segstats.txt:
*** Memory Pool Configuration for : LH-MTZ-SRV-04
Time and date : 08:49:44 PM 02/17/2012
Server version : NetWare 6.5 Support Pack 8
Server uptime : 3d 19h 28m 42s
SEG.NLM uptime : 3d 03h 01m 42s
SEG.NLM version : v2.00.17
Original Memory : 8,539,234,304 bytes (7.95 GB)
ESM Memory : 6,442,450,944 bytes (6.00 GB)
0xFFFFFFFF --------------------------------------------------------------
| Kernel Reserved Space |
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 --------------------------------------------------------------
| User Address Space (L!=P) |
| |
| User Pool Size : 387,973,120 bytes (370.0 MB) |
| High Water Mark : 281,686,016 bytes (268.6 MB) |
| PM Pages In Use : 0 bytes (0 KB) |
| |
0xDE200000 --------------------------------------------------------------
| Virtual Memory Address Space (L!=P) |
| |
| VM Address Space : 1,629,450,240 bytes (1.52 GB) |
| Available : 33,230,848 bytes (31.7 MB) |
| Total VM Pages : 991,232 bytes (968 KB) |
| Free Clean VM : 0 bytes (0 KB) |
| Free Cache VM : 991,232 bytes (968 KB) |
| Total LP Pages : 32,194,560 bytes (30.7 MB) |
| Free Clean LP : 32,129,024 bytes (30.6 MB) |
| Free Cache LP : 65,536 bytes (64 KB) |
| Free Dirty : 0 bytes (0 KB) |
| NLM Memory In Use : 777,904,128 bytes (741.9 MB) |
| NLM/VM Memory : 809,684,992 bytes (772.2 MB) |
| Largest Segment : 57,122,816 bytes (54.5 MB) |
| Lowest Kernel Page: 0 bytes (0 KB) |
| : [0x00000000] |
| High Water Mark : 891,912,192 bytes (850.6 MB) |
| Alloc Failures : 109 |
| |
0x7D009000 --------------------------------------------------------------
| File System Address Space (L==P or L!=P) |
| |
| FS Address Space : 2,090,790,912 bytes (1.95 GB) |
| Available : 121,495,552 bytes (115.9 MB) |
| Largest Segment : 5,472,256 bytes (5.2 MB) |
| |
| NSS Memory (85%) : 684,625,920 bytes (652.9 MB) |
| NSS (avail cache) : 680,878,080 bytes (649.3 MB) |
| |
0x0061A000 --------------------------------------------------------------
| DOS / SERVER.NLM |
| |
| Size : 6,397,952 bytes (6.1 MB) |
| |
0x00000000 --------------------------------------------------------------
Total NLMs loaded on the server: 241
Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== ===============================
1. ADS.NLM 10.10 30 Aug 2011 344,449,260 bytes (328.5 MB)
2. NSS.NLM 3.27.03 7 Jun 2010 194,755,664 bytes (185.7 MB)
3. SERVER.NLM 5.70.08 3 Oct 2008 92,575,840 bytes (88.3 MB)
4. DS.NLM 20217.07 30 Jan 2009 23,061,351 bytes (22.0 MB)
5. CIOS.NLM 1.60 12 Feb 2008 20,252,511 bytes (19.3 MB)
6. OWCIMOMD.NLM 3.02 27 Nov 2007 9,314,520 bytes (8.9 MB)
7. FATFS.NLM 1.24 27 Aug 2007 5,699,669 bytes (5.4 MB)
8. MM.NLM 3.22.08 24 Apr 2009 4,443,504 bytes (4.2 MB)
9. PKI.NLM 3.32 25 Aug 2008 4,069,957 bytes (3.9 MB)
10. WS2_32.NLM 6.24.01 14 Feb 2008 3,566,252 bytes (3.4 MB)
11. PORTAL.NLM 4.03 22 Sep 2008 3,380,000 bytes (3.2 MB)
12. NWMKDE.NLM 7.94 11 Dec 2001 3,077,337 bytes (2.9 MB)
13. JVM.NLM 1.43 16 Oct 2008 2,701,919 bytes (2.6 MB)
14. LIB0.NLM 5.90.15 10 Mar 2008 2,457,946 bytes (2.3 MB)
15. CDBE.NLM 6.01 21 Sep 2006 2,443,488 bytes (2.3 MB)
16. NDSIMON.NLM 20214.04 30 Jan 2009 2,284,509 bytes (2.2 MB)
17. NWENC103.NLM 7.90 24 Feb 2001 2,134,781 bytes (2.0 MB)
18. EMBOX.NLM 20215.23 24 Apr 2008 1,982,250 bytes (1.9 MB)
19. LIBC.NLM 9.00.05 3 Oct 2008 1,970,870 bytes (1.9 MB)
20. CONNMGR.NLM 5.60.01 7 Sep 2006 1,856,531 bytes (1.8 MB)
Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== ===============================
1. ADS.NLM 311,177,452 bytes 344,449,260 bytes 31.7 MB
2. NSS.NLM 96,934,992 bytes 194,755,664 bytes 93.3 MB
3. SERVER.NLM 87,754,848 bytes 92,575,840 bytes 4.6 MB
4. DS.NLM 22,659,943 bytes 23,061,351 bytes 392 KB
5. CIOS.NLM 20,252,511 bytes 20,252,511 bytes 0 KB
6. OWCIMOMD.NLM 9,314,520 bytes 9,314,520 bytes 0 KB
7. FATFS.NLM 5,699,669 bytes 5,699,669 bytes 0 KB
8. MM.NLM 3,407,216 bytes 4,443,504 bytes 1012 KB
9. PKI.NLM 4,135,493 bytes 4,069,957 bytes -64 KB
10. WS2_32.NLM 3,529,388 bytes 3,566,252 bytes 36 KB
11. PORTAL.NLM 3,380,000 bytes 3,380,000 bytes 0 KB
12. NWMKDE.NLM 3,077,337 bytes 3,077,337 bytes 0 KB
13. JVM.NLM 2,697,823 bytes 2,701,919 bytes 4 KB
14. LIB0.NLM 2,457,946 bytes 2,457,946 bytes 0 KB
15. CDBE.NLM 2,443,488 bytes 2,443,488 bytes 0 KB
16. NDSIMON.NLM 2,284,509 bytes 2,284,509 bytes 0 KB
17. NWENC103.NLM 2,134,781 bytes 2,134,781 bytes 0 KB
18. EMBOX.NLM 1,982,250 bytes 1,982,250 bytes 0 KB
19. LIBC.NLM 1,966,774 bytes 1,970,870 bytes 4 KB
20. CONNMGR.NLM 1,868,819 bytes 1,856,531 bytes -12 KB
Logical Memory Summary Information
================================================== ===============================
File System Cache Information
FS Cache Free : 9,392,128 bytes (9.0 MB)
FS Cache Fragmented : 112,103,424 bytes (106.9 MB)
FS Cache Largest Segment : 5,472,256 bytes (5.2 MB)
Logical System Cache Information
LS Cache Free : 380,108,800 bytes (362.5 MB)
LS Cache Fragmented : 1,231,609,856 bytes (1.15 GB)
LS OS Reserved Data : 299,900,928 bytes (286.0 MB)
LS Cache Largest Segment : 57,122,816 bytes (54.5 MB)
LS Cache Largest Position : 2DD10000
Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 389,500,928 bytes (371.5 MB)
Total Fragmented : 1,343,713,280 bytes (1.25 GB)
Highest Physical Address : 7D009000
User Space : 387,973,120 bytes (370.0 MB)
User Space (High Water Mark) : 281,686,016 bytes (268.6 MB)
NLM Memory (High Water Mark) : 891,912,192 bytes (850.6 MB)
Kernel Address Space In Use : 1,937,776,640 bytes (1.80 GB)
Available Kernel Address Space : 445,149,184 bytes (424.5 MB)
Memory Summary Screen (.ms)
================================================== ===============================
KNOWN MEMORY Bytes Pages Bytes Pages
Server: 2096711728 511892 Video: 8192 2
Dos: 71632 17 Other: 131072 32
FS CACHE KERNEL NLM MEMORY
Original: 2092503040 510865 Code: 43585536 10641
Current: 121495552 29662 Data: 22327296 5451
Dirty: 0 0 Sh Code: 49152 12
Largest seg: 5472256 1336 Sh Data: 20480 5
Non-Movable: 81920 20 Help: 139264 34
Other: 1970925568 481183 Message: 1126400 275
Avail NSS: 680878080 166230 Alloc L!=P: 721514496 176151
Movable: 8192 2 Alloc L==P: 20922368 5108
Total: 809684992 197677
VM SYSTEM
Free clean VM: 0 0
Free clean LP: 32129024 7844
Free cache VM: 991232 242
Free cache LP: 65536 16
Free dirty: 0 0
In use: 0 0
Total: 33230848 8113
Memory Configuration (set parameters)
================================================== ==============================
Auto Tune Server Memory = OFF
File Cache Maximum Size = 2147483650
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 387973120
NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
------------------------------
File does not exist,
or is zero byte in size.
DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
------------------------------
File does not exist,
or is zero byte in size.
TSAFS Memory Information/Configuration
================================================== ==============================
Cache Memory Threshold : 10%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4
NSS Memory Information/Configuration
================================================== ==============================
Current NSS Memory Settings
-------------------------------------------
Cache Balance Percentage : 85%
Cache Memory Allocated : 652.9 MB
Available Cache from NSS : 649.3 MB
Current NSS Caching Percentages
-------------------------------------------
Buffer cache hit percentage : 83%
Name Tree cache hit percentage : 69%
File cache hit percentage : 99%
NSS Flush Status: 2 GB of RAM or less in the server
Server High/Low Water Mark Values
================================================== ==============================
NLM Memory High Water Mark = 891,912,192 bytes
File System High Water Mark = 348,100 bytes
User Space Information:
User Space High Water Mark = 281,686,016 bytes
Committed Pages High Water Mark = 13 pages
Mapped VM Pages High Water Mark = 1,269 pages
Reserved Pages High Water Mark = 70,625 pages
Swapped Pages High Water Mark = 512 pages
Available Low Water Mark = 122,167,296
ESM Memory High Water Mark = 1,233 pages
We are having problems with one of our netware 6.5 sp8 servers (all patches up to day). We also have another with the same config (hard and soft) and its working fine.
Both are IBM system x3550 M3 with 8GB and 1 E5520 with hyperthreading disabled.
The server with the problems is configured with auto tuning off and file cache maximum size in 2GB. No -u parameter.
Also the command memory shows Total server memory 2GB and extended 6GB in both servers (the one with problems and the fine one).
The error only appears during peek hours (2 to 4~pm) and its only from ADS.NLM.
15/02/2012 4.01.22 : SERVER-5.70-0 [nmID=6001D]
Cache memory allocator out of available memory.
15/02/2012 4.01.22 : SERVER-5.70-0 [nmID=2000A]
Short term memory allocator is out of memory.
3 attempts to get more memory failed.
request size in bytes 33554441 from Module ADS.NLM
Thanks in advance,
segstats.txt:
*** Memory Pool Configuration for : LH-MTZ-SRV-04
Time and date : 08:49:44 PM 02/17/2012
Server version : NetWare 6.5 Support Pack 8
Server uptime : 3d 19h 28m 42s
SEG.NLM uptime : 3d 03h 01m 42s
SEG.NLM version : v2.00.17
Original Memory : 8,539,234,304 bytes (7.95 GB)
ESM Memory : 6,442,450,944 bytes (6.00 GB)
0xFFFFFFFF --------------------------------------------------------------
| Kernel Reserved Space |
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 --------------------------------------------------------------
| User Address Space (L!=P) |
| |
| User Pool Size : 387,973,120 bytes (370.0 MB) |
| High Water Mark : 281,686,016 bytes (268.6 MB) |
| PM Pages In Use : 0 bytes (0 KB) |
| |
0xDE200000 --------------------------------------------------------------
| Virtual Memory Address Space (L!=P) |
| |
| VM Address Space : 1,629,450,240 bytes (1.52 GB) |
| Available : 33,230,848 bytes (31.7 MB) |
| Total VM Pages : 991,232 bytes (968 KB) |
| Free Clean VM : 0 bytes (0 KB) |
| Free Cache VM : 991,232 bytes (968 KB) |
| Total LP Pages : 32,194,560 bytes (30.7 MB) |
| Free Clean LP : 32,129,024 bytes (30.6 MB) |
| Free Cache LP : 65,536 bytes (64 KB) |
| Free Dirty : 0 bytes (0 KB) |
| NLM Memory In Use : 777,904,128 bytes (741.9 MB) |
| NLM/VM Memory : 809,684,992 bytes (772.2 MB) |
| Largest Segment : 57,122,816 bytes (54.5 MB) |
| Lowest Kernel Page: 0 bytes (0 KB) |
| : [0x00000000] |
| High Water Mark : 891,912,192 bytes (850.6 MB) |
| Alloc Failures : 109 |
| |
0x7D009000 --------------------------------------------------------------
| File System Address Space (L==P or L!=P) |
| |
| FS Address Space : 2,090,790,912 bytes (1.95 GB) |
| Available : 121,495,552 bytes (115.9 MB) |
| Largest Segment : 5,472,256 bytes (5.2 MB) |
| |
| NSS Memory (85%) : 684,625,920 bytes (652.9 MB) |
| NSS (avail cache) : 680,878,080 bytes (649.3 MB) |
| |
0x0061A000 --------------------------------------------------------------
| DOS / SERVER.NLM |
| |
| Size : 6,397,952 bytes (6.1 MB) |
| |
0x00000000 --------------------------------------------------------------
Total NLMs loaded on the server: 241
Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== ===============================
1. ADS.NLM 10.10 30 Aug 2011 344,449,260 bytes (328.5 MB)
2. NSS.NLM 3.27.03 7 Jun 2010 194,755,664 bytes (185.7 MB)
3. SERVER.NLM 5.70.08 3 Oct 2008 92,575,840 bytes (88.3 MB)
4. DS.NLM 20217.07 30 Jan 2009 23,061,351 bytes (22.0 MB)
5. CIOS.NLM 1.60 12 Feb 2008 20,252,511 bytes (19.3 MB)
6. OWCIMOMD.NLM 3.02 27 Nov 2007 9,314,520 bytes (8.9 MB)
7. FATFS.NLM 1.24 27 Aug 2007 5,699,669 bytes (5.4 MB)
8. MM.NLM 3.22.08 24 Apr 2009 4,443,504 bytes (4.2 MB)
9. PKI.NLM 3.32 25 Aug 2008 4,069,957 bytes (3.9 MB)
10. WS2_32.NLM 6.24.01 14 Feb 2008 3,566,252 bytes (3.4 MB)
11. PORTAL.NLM 4.03 22 Sep 2008 3,380,000 bytes (3.2 MB)
12. NWMKDE.NLM 7.94 11 Dec 2001 3,077,337 bytes (2.9 MB)
13. JVM.NLM 1.43 16 Oct 2008 2,701,919 bytes (2.6 MB)
14. LIB0.NLM 5.90.15 10 Mar 2008 2,457,946 bytes (2.3 MB)
15. CDBE.NLM 6.01 21 Sep 2006 2,443,488 bytes (2.3 MB)
16. NDSIMON.NLM 20214.04 30 Jan 2009 2,284,509 bytes (2.2 MB)
17. NWENC103.NLM 7.90 24 Feb 2001 2,134,781 bytes (2.0 MB)
18. EMBOX.NLM 20215.23 24 Apr 2008 1,982,250 bytes (1.9 MB)
19. LIBC.NLM 9.00.05 3 Oct 2008 1,970,870 bytes (1.9 MB)
20. CONNMGR.NLM 5.60.01 7 Sep 2006 1,856,531 bytes (1.8 MB)
Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== ===============================
1. ADS.NLM 311,177,452 bytes 344,449,260 bytes 31.7 MB
2. NSS.NLM 96,934,992 bytes 194,755,664 bytes 93.3 MB
3. SERVER.NLM 87,754,848 bytes 92,575,840 bytes 4.6 MB
4. DS.NLM 22,659,943 bytes 23,061,351 bytes 392 KB
5. CIOS.NLM 20,252,511 bytes 20,252,511 bytes 0 KB
6. OWCIMOMD.NLM 9,314,520 bytes 9,314,520 bytes 0 KB
7. FATFS.NLM 5,699,669 bytes 5,699,669 bytes 0 KB
8. MM.NLM 3,407,216 bytes 4,443,504 bytes 1012 KB
9. PKI.NLM 4,135,493 bytes 4,069,957 bytes -64 KB
10. WS2_32.NLM 3,529,388 bytes 3,566,252 bytes 36 KB
11. PORTAL.NLM 3,380,000 bytes 3,380,000 bytes 0 KB
12. NWMKDE.NLM 3,077,337 bytes 3,077,337 bytes 0 KB
13. JVM.NLM 2,697,823 bytes 2,701,919 bytes 4 KB
14. LIB0.NLM 2,457,946 bytes 2,457,946 bytes 0 KB
15. CDBE.NLM 2,443,488 bytes 2,443,488 bytes 0 KB
16. NDSIMON.NLM 2,284,509 bytes 2,284,509 bytes 0 KB
17. NWENC103.NLM 2,134,781 bytes 2,134,781 bytes 0 KB
18. EMBOX.NLM 1,982,250 bytes 1,982,250 bytes 0 KB
19. LIBC.NLM 1,966,774 bytes 1,970,870 bytes 4 KB
20. CONNMGR.NLM 1,868,819 bytes 1,856,531 bytes -12 KB
Logical Memory Summary Information
================================================== ===============================
File System Cache Information
FS Cache Free : 9,392,128 bytes (9.0 MB)
FS Cache Fragmented : 112,103,424 bytes (106.9 MB)
FS Cache Largest Segment : 5,472,256 bytes (5.2 MB)
Logical System Cache Information
LS Cache Free : 380,108,800 bytes (362.5 MB)
LS Cache Fragmented : 1,231,609,856 bytes (1.15 GB)
LS OS Reserved Data : 299,900,928 bytes (286.0 MB)
LS Cache Largest Segment : 57,122,816 bytes (54.5 MB)
LS Cache Largest Position : 2DD10000
Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 389,500,928 bytes (371.5 MB)
Total Fragmented : 1,343,713,280 bytes (1.25 GB)
Highest Physical Address : 7D009000
User Space : 387,973,120 bytes (370.0 MB)
User Space (High Water Mark) : 281,686,016 bytes (268.6 MB)
NLM Memory (High Water Mark) : 891,912,192 bytes (850.6 MB)
Kernel Address Space In Use : 1,937,776,640 bytes (1.80 GB)
Available Kernel Address Space : 445,149,184 bytes (424.5 MB)
Memory Summary Screen (.ms)
================================================== ===============================
KNOWN MEMORY Bytes Pages Bytes Pages
Server: 2096711728 511892 Video: 8192 2
Dos: 71632 17 Other: 131072 32
FS CACHE KERNEL NLM MEMORY
Original: 2092503040 510865 Code: 43585536 10641
Current: 121495552 29662 Data: 22327296 5451
Dirty: 0 0 Sh Code: 49152 12
Largest seg: 5472256 1336 Sh Data: 20480 5
Non-Movable: 81920 20 Help: 139264 34
Other: 1970925568 481183 Message: 1126400 275
Avail NSS: 680878080 166230 Alloc L!=P: 721514496 176151
Movable: 8192 2 Alloc L==P: 20922368 5108
Total: 809684992 197677
VM SYSTEM
Free clean VM: 0 0
Free clean LP: 32129024 7844
Free cache VM: 991232 242
Free cache LP: 65536 16
Free dirty: 0 0
In use: 0 0
Total: 33230848 8113
Memory Configuration (set parameters)
================================================== ==============================
Auto Tune Server Memory = OFF
File Cache Maximum Size = 2147483650
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 387973120
NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
------------------------------
File does not exist,
or is zero byte in size.
DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
------------------------------
File does not exist,
or is zero byte in size.
TSAFS Memory Information/Configuration
================================================== ==============================
Cache Memory Threshold : 10%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4
NSS Memory Information/Configuration
================================================== ==============================
Current NSS Memory Settings
-------------------------------------------
Cache Balance Percentage : 85%
Cache Memory Allocated : 652.9 MB
Available Cache from NSS : 649.3 MB
Current NSS Caching Percentages
-------------------------------------------
Buffer cache hit percentage : 83%
Name Tree cache hit percentage : 69%
File cache hit percentage : 99%
NSS Flush Status: 2 GB of RAM or less in the server
Server High/Low Water Mark Values
================================================== ==============================
NLM Memory High Water Mark = 891,912,192 bytes
File System High Water Mark = 348,100 bytes
User Space Information:
User Space High Water Mark = 281,686,016 bytes
Committed Pages High Water Mark = 13 pages
Mapped VM Pages High Water Mark = 1,269 pages
Reserved Pages High Water Mark = 70,625 pages
Swapped Pages High Water Mark = 512 pages
Available Low Water Mark = 122,167,296
ESM Memory High Water Mark = 1,233 pages