针对glibc bug的Java+docker RSS调优
我有和问题类似的故事Growing resident memory usage (RSS) of Java Process
jcmd NMT没有显示任何问题。它有常数。 比如
Native Memory Tracking:
Total: reserved=1611305KB +19680KB, committed=347717KB +41440KB
同时命令docker exec eb97c7681baa ps -o rss,vsz,sz 7返回
RSS VSZ SZ
458368 2294800 573700
NMT的提交大小保持+/-不变，但RSS不断增长，这导致docker OOM killer停止带有OOM的容器。 我读了几篇关于内存碎片的好文章 ( https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_native_memory_fragmentation_and_process_size_growth?lang=en）
所以很多人报告说，像MALLOC_ARENA_MAX=2这样的修复应该可以解决这个问题，但似乎需要进行更多的调整。
当我设置该值时，RSS增长的问题显著减少，但仍然存在。
似乎我需要设置以下值：
MALLOC_MMAP_THRESHOLD_-定义何时从java堆中分配内存（默认值128k）
MALLOC_TRIM_THRESHOLD_当竞技场大小达到最大阈值（默认值128k）时，它会尝试修剪竞技场以将内存释放回操作系统-是吗
从文章中：
If you decide to change the mmap threshold, the first step is to determine the allocation pattern. This can be done through tools such as ltrace (on malloc) or SystemTap, or if you know what is causing most of the allocations (e.g. Java DirectByteBuffers), then you can trace just those allocations. Next, create a histogram of these sizes and choose a threshold just under the smallest yet most frequent allocation. For example, let's say you've found that most allocations are larger than 8KB. In this case, you can set the threshold to 8192
我使用pmap工具检查分配的大小：
7: java -XX:NativeMemoryTracking=detail -XX:+PrintFlagsFinal -XshowSettings:vm -jar /usr/share/jvmservice/service.jar
Address Kbytes RSS Dirty Mode Mapping
00000000020af000 560 432 432 rw--- [ anon ]
00000000f8000000 43648 39032 39032 rw--- [ anon ]
00000000faaa0000 87424 66532 66532 rw--- [ anon ]
0000000100000000 11136 11056 11056 rw--- [ anon ]
00007f2820000000 30912 25520 25520 rw--- [ anon ]
00007f2828000000 65512 58568 58568 rw--- [ anon ]
00007f282fb06000 1280 1176 1176 rw--- [ anon ]
00007f282fd06000 2048 2044 2044 rw--- [ anon ]
00007f282ff06000 2048 2048 2048 rw--- [ anon ]
00007f2830106000 2048 2048 2048 rw--- [ anon ]
00007f2830309000 3064 2064 2064 rw--- [ anon ]
00007f283060a000 1016 100 100 rw--- [ anon ]
00007f283070b000 3064 2068 2068 rw--- [ anon ]
00007f2830a09000 2048 2048 2048 rw--- [ anon ]
00007f2830c0c000 1016 108 108 rw--- [ anon ]
00007f2830d0d000 1016 92 92 rw--- [ anon ]
00007f2830e0e000 1016 124 124 rw--- [ anon ]
00007f2830f0f000 1016 128 128 rw--- [ anon ]
00007f2831111000 1016 124 124 rw--- [ anon ]
00007f2831212000 1016 128 128 rw--- [ anon ]
00007f2831313000 1016 144 144 rw--- [ anon ]
00007f2831414000 1016 128 128 rw--- [ anon ]
00007f2831515000 1016 144 144 rw--- [ anon ]
00007f2831616000 1016 128 128 rw--- [ anon ]
00007f2831717000 1016 136 136 rw--- [ anon ]
00007f2831818000 3064 2136 2136 rw--- [ anon ]
00007f2831b16000 2048 2044 2044 rw--- [ anon ]
00007f2831d19000 3064 2152 2152 rw--- [ anon ]
00007f2832017000 2048 2048 2048 rw--- [ anon ]
00007f283221a000 3064 2152 2152 rw--- [ anon ]
00007f2832518000 16388 164 164 rw--- [ anon ]
00007f283351c000 1016 136 136 rw--- [ anon ]
00007f283361d000 3064 2136 2136 rw--- [ anon ]
00007f283391b000 2048 2048 2048 rw--- [ anon ]
00007f2833d20000 1016 100 100 rw--- [ anon ]
00007f2833e21000 3064 2148 2148 rw--- [ anon ]
00007f283411f000 2048 2048 2048 rw--- [ anon ]
00007f2834b43000 3064 2136 2136 rw--- [ anon ]
00007f2835291000 2116 2052 2052 rw--- [ anon ]
00007f28354a2000 2048 2048 2048 rw--- [ anon ]
00007f28356a2000 2048 2048 2048 rw--- [ anon ]
00007f28358a2000 2048 2048 2048 rw--- [ anon ]
00007f2835aa2000 2048 2048 2048 rw--- [ anon ]
00007f2835ca2000 2048 2048 2048 rw--- [ anon ]
00007f2835ea2000 2048 2048 2048 rw--- [ anon ]
00007f28360e1000 2048 2048 2048 rw--- [ anon ]
00007f28365df000 3064 2164 2164 rw--- [ anon ]
00007f28368dd000 2048 2048 2048 rw--- [ anon ]
00007f2836add000 2048 2048 2048 rw--- [ anon ]
00007f2836cdd000 2048 2048 2048 rw--- [ anon ]
00007f2836ee0000 3064 2156 2156 rw--- [ anon ]
00007f28371de000 2048 2048 2048 rw--- [ anon ]
00007f28373de000 2048 2048 2048 rw--- [ anon ]
00007f28375de000 2048 2048 2048 rw--- [ anon ]
00007f2837c00000 2048 2048 2048 rw--- [ anon ]
00007f2837e00000 2048 2048 2048 rw--- [ anon ]
00007f2838000000 19828 19820 19820 rw--- [ anon ]
00007f283c000000 6296 6036 6036 rw--- [ anon ]
00007f2840000000 1004 952 952 rw--- [ anon ]
00007f2844000000 65520 62180 62180 rw--- [ anon ]
00007f2848000000 192 192 192 rw--- [ anon ]
00007f284c000000 3260 3228 3228 rw--- [ anon ]
00007f2850000000 5060 5060 5060 rw--- [ anon ]
00007f28544b7000 1024 12 12 rw--- [ anon ]
00007f28545ba000 1016 116 116 rw--- [ anon ]
00007f28546bc000 1012 24 24 rw--- [ anon ]
00007f28547bd000 1012 56 56 rw--- [ anon ]
00007f28548bd000 1016 92 92 rw--- [ anon ]
00007f2854b31000 1016 92 92 rw--- [ anon ]
00007f2854c32000 1016 88 88 rw--- [ anon ]
00007f2854d31000 1024 16 16 rw--- [ anon ]
00007f2855000000 45440 44788 44788 rwx-- [ anon ]
00007f2864000000 27756 26120 26120 rw--- [ anon ]
00007f28680d1000 8976 8976 8976 rw--- [ anon ]
00007f2868995000 712 712 712 rw--- [ anon ]
00007f286a0f7000 12612 4916 0 r-x-- libjvm.so
---------------- ------- ------- -------
total kB 2294804 461600 456396
因此，似乎大多数细胞的大小为2048K。 我还用JX射线分析了堆
这个空间似乎被java DirectByteBuffers使用。下面是一篇关于如何调整它的好文章https://www.evanjones.ca/java-bytebuffer-leak.html
问题1。设置
MALLOC_MMAP_THRESHOLD_ = 8192
问题2。设置MALLOC_TRIM_THRESHOLD_的值时需要考虑这一点
来自医生
M_TRIM_THRESHOLD When the amount of contiguous free memory at the top of the heap grows sufficiently large, free(3) employs sbrk(2) to release this memory back to the system. (This can be useful in programs that continue to execute for a long period after freeing a significant amount of memory.) The M_TRIM_THRESHOLD parameter specifies the minimum size (in bytes) that this block of memory must reach before sbrk(2) is used to trim the heap. The default value for this parameter is 128*1024. Setting M_TRIM_THRESHOLD to -1 disables trimming completely. Modifying M_TRIM_THRESHOLD is a trade-off between increasing the number of system calls (when the parameter is set low) and wasting unused memory at the top of the heap (when the parameter is set high).
共 (0) 个答案