java JVM在序列化过程中挂起

Question

我正在调试每两天挂起一次的服务器

java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

突然，一切都冻结了，服务器停止处理/接收任何请求（确切地说是任何工作）。捕获线程转储后，我注意到以下几点：

• 没有死锁，根据jstack
• 199个线程处于BLOCKED状态，5个线程处于IN_NATIVE状态，1个线程处于IN_VM状态
• 所有阻塞线程都在wait/sleep/receive/etc方法上阻塞

只有IN_VM中的线程实际显示了一些重要堆栈：

Thread 10453: (state = IN_VM)
 - java.lang.System.identityHashCode(java.lang.Object) @bci=0 (Compiled frame; information may be imprecise)
 - java.io.ObjectOutputStream$HandleTable.hash(java.lang.Object) @bci=1, line=2360 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.insert(java.lang.Object, int) @bci=2, line=2323 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.growSpine() @bci=57, line=2338 (Compiled frame)
 - java.io.ObjectOutputStream$HandleTable.assign(java.lang.Object) @bci=28, line=2279 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=105, line=1428 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject(java.lang.Object) @bci=16, line=348 (Compiled frame)
 - java.util.HashMap.internalWriteEntries(java.io.ObjectOutputStream) @bci=49, line=1793 (Compiled frame)
 - java.util.HashMap.writeObject(java.io.ObjectOutputStream) @bci=24, line=1363 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor13.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
 - java.io.ObjectStreamClass.invokeWriteObject(java.lang.Object, java.io.ObjectOutputStream) @bci=24, line=1140 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=125, line=1497 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteFields(java.lang.Object, java.io.ObjectStreamClass) @bci=18, line=1523 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteObject() @bci=41, line=441 (Compiled frame)
 - java.util.Collections$SynchronizedMap.writeObject(java.io.ObjectOutputStream) @bci=8, line=2691 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor12.invoke(java.lang.Object, java.lang.Object[]) @bci=40 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)
 - java.io.ObjectStreamClass.invokeWriteObject(java.lang.Object, java.io.ObjectOutputStream) @bci=24, line=1140 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=125, line=1497 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectStreamClass$FieldReflector.getPrimFieldValues(java.lang.Object, byte[]) @bci=313, line=2135 (Compiled frame)
 - java.io.ObjectStreamClass.getPrimFieldValues(java.lang.Object, byte[]) @bci=6, line=1389 (Compiled frame)
 - java.io.ObjectOutputStream.defaultWriteFields(java.lang.Object, java.io.ObjectStreamClass) @bci=70, line=1533 (Compiled frame)
 - java.io.ObjectOutputStream.writeSerialData(java.lang.Object, java.io.ObjectStreamClass) @bci=211, line=1509 (Compiled frame)
 - java.io.ObjectOutputStream.writeOrdinaryObject(java.lang.Object, java.io.ObjectStreamClass, boolean) @bci=137, line=1432 (Compiled frame)
 - java.io.ObjectOutputStream.writeObject0(java.lang.Object, boolean) @bci=506, line=1178 (Compiled frame)
 - java.io.ObjectOutputStream$BlockDataOutputStream.flush() @bci=1, line=1822 (Compiled frame)
 - java.io.ObjectOutputStream.flush() @bci=4, line=719 (Compiled frame)
 ....
 ....
 - few more frames where we invoke `flush()`

我们尝试序列化的对象如下所示：

public class DomainObjectCollection implements Serializable
{
    private static final long serialVersionUID = -2050273337993827124L;

    private Map<String, DomainObject> objects;
    private Map<String, List<DomainObject>> objectsForUsername;

    public DomainObjectCollection()
    {
        this.objects = Collections.synchronizedMap(new HashMap<>());
        this.objectsForUsername = Collections.synchronizedMap(new HashMap<>());
    }
}

我们序列化对象非常简单：

os = new ObjectOutputStream(new BufferedOutputStream(new FileOutputStream(file), 128 * 1024));
                os.writeObject(object);
                os.flush();

所以，除了同步，没有什么特别的

有人遇到过类似的情况吗？如果我没有提供一些细节，请随时询问

更新

大约一小时前，从上面的线程转储中，我捕获了混合堆栈跟踪：

----------------- 10453 -----------------
0x00007f0e5b65d709  __pthread_cond_timedwait + 0x129
0x00007f0e5a8bcbb7  _ZN18ObjectSynchronizer12FastHashCodeEP6ThreadP7oopDesc + 0x377
0x00007f0e5a5746df  JVM_IHashCode + 0x4f
0x00007f0e4577d13f  <Unknown compiled code>
Locked ownable synchronizers:
    - None