在Google DFS fold中运行jupyter笔记本时在“保存和检查点”期间出错

1条回答

网友

1楼 · 发布于 2024-09-23 13:38:04

deniz195的一些伟大的调试导致了一个伟大的答案！这个临时解决方案适用于我和其他许多github朋友！在

引用他最初的回答：

I debugged the issue a little bit and found the following solution:
Part 1 - Minimal working example The issue seems to originate from shutil.copyfile, which checks if source and destination file are the same (using os.path.samefile).
It appears that the combination of windows and google drive file stream yields invalid results. Here is the minimum example (assuming G:\My Drive\foo.txt exists, using python 3.6.2):
>>> f1 = 'G:\\My Drive\\foo.txt'
>>> f2 = 'G:\\My Drive\\foo2.txt'
>>> import shutil
>>> shutil.copyfile(f1, f2)
>>> shutil.copyfile(f1, f2)
> The last line throws the SameFileError although it clearly shouldnt!
Whereas:
>>> f1 = 'G:\\My Drive\\foo.txt'
>>> f3 = 'C:\\Scratch\\foo2.txt'
>>> import shutil
>>> shutil.copyfile(f1, f3)
>>> shutil.copyfile(f1, f3)
> Throws no error (correct)!
Part 2 - Why? How does this happen?
It turns out that shutil.copyfile uses os.path.samefile to determine if a file is being copied on itself: (from https://github.com/python/cpython/blob/master/Lib/genericpath.py)
# Are two filenames really pointing to the same file?
def samefile(f1, f2):
    """Test whether two pathnames reference the same actual file"""
    s1 = os.stat(f1)
    s2 = os.stat(f2)
    return samestat(s1, s2)

# Are two stat buffers (obtained from stat, fstat or lstat)
# describing the same file?
def samestat(s1, s2):
    """Test whether two stat buffers reference the same file"""
    return (s1.st_ino == s2.st_ino and
            s1.st_dev == s2.st_dev)
Now unfortunately st_ino depends on windows on the file system (Meaning of ST_INO (os.stat() output) in Windows OS) and specifically is st_ino==0 for all files on Google Drive File Stream.
Part 3 - Where to go and who to blame? It seems that this error is an unlucky combination of a lazy filesystem (why not report some kind of uid as the inode?) and a naive os python library (Checking for file identity this way seems not to generalize well... why not check if st_ino==0?)
Part 4 - The dirty fix
Find genericpath.py of your python library:
>>> import os
>>> os.path.genericpath.__file__
'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\genericpath.py'
In this file, replace the samestat function with the following patch:
# Are two stat buffers (obtained from stat, fstat or lstat)
# describing the same file?
def samestat(s1, s2):
    """Test whether two stat buffers reference the same file"""
    return (s1.st_ino != 0 and
          s2.st_ino != 0 and
          s1.st_ino == s2.st_ino and
            s1.st_dev == s2.st_dev)
Save the file. Restart python (and/or jupyter).
Be happy and wait until either google or the python fixes this issue properly...

相关问题更多 >

编程相关推荐

热门问题

热门文章

在Google DFS fold中运行jupyter笔记本时在“保存和检查点”期间出错

相关问题 更多 >

编程相关推荐

热门问题

热门文章

相关问题更多 >