Python readBin与R

Question

我试图用Python读取二进制文件。其他人已使用以下代码使用R读入数据：

x <- readBin(webpage, numeric(), n=6e8, size = 4, endian = "little")
      myPoints <- data.frame("tmax" = x[1:(length(x)/4)],
                             "nmax" = x[(length(x)/4 + 1):(2*(length(x)/4))],
                             "tmin" = x[(2*length(x)/4 + 1):(3*(length(x)/4))],
                             "nmin" = x[(3*length(x)/4 + 1):(length(x))])

对于Python，我尝试以下代码：

^{pr2}$

我得到的结果略有不同。例如，R中的第一行返回4列-999.9，0，-999.0，0。Python为所有四列返回-999.0（下图）。在

Python输出：

R输出：

我知道他们是用一些[]代码按文件的长度进行切片，但我不知道在Python中究竟如何做到这一点，也不清楚他们为什么要这样做。基本上，我想重新创建R在Python中所做的事情。在

如果需要，我可以提供更多的代码库。我不想用不必要的代码来压倒一切。在

Answer 1

这里有一个不太需要内存的方法来做同样的事情。可能也快一点。（但这对我来说很难确认）

我的计算机没有足够的内存来运行第一个包含这些大文件的程序。这个是这样，但是我仍然需要先创建一个ony tmax的列表（文件的前1/4），然后打印它，然后删除这个列表，以便有足够的内存来存储nmax、tmin和nmin

但这篇文章也说，2018年文件中的nmin都是-999.0。如果这不合理，你能检查一下R代码是怎么解释的吗？我怀疑这只是档案里的东西。另一种可能性当然是，我完全搞错了（我对此表示怀疑）。不过，我也尝试了2017年的文件，但这个文件没有这样的问题：所有tmax、nmax、tmin、nmin都有大约37%-999.0的值

不管怎样，下面是第二个代码：

import os
import struct

# load_data()
#   data_store : object to append() data items (floats) to
#   num        : number of floats to read and store
#   datafile   : opened binary file object to read float data from
#
def load_data(data_store, num, datafile):
    for i in range(num):
        data = datafile.read(4)  # process one float (=4 bytes) at a time
        item = struct.unpack("<f", data)[0]  # '<' means little endian
        data_store.append(item) 

# save_list() saves a list of float's as strings to a file
#
def save_list(filename, datalist):
    output = open(filename, "wt")
    for item in datalist:
        output.write(str(item) + '\n')
    output.close()

#### MAIN ####

datafile = open('data.bin','rb')

# Get file size so we can calculate number of points without reading
# the (large) file entirely into memory.
#
file_info = os.stat(datafile.fileno())

# Calculate number of points, i.e. number of each tmax's, nmax's,
# tmin's, nmin's. A point is 4 floats of 4 bytes each, hence number
# of points = file-size / (4*4)
#
num = int(file_info.st_size / 16)

tmax_list = list()
load_data(tmax_list, num, datafile)
save_list("tmax.txt", tmax_list)
del tmax_list   # huge list, save memory

nmax_list = list()
load_data(nmax_list, num, datafile)
save_list("nmax.txt", nmax_list)
del nmax_list   # huge list, save memory

tmin_list = list()
load_data(tmin_list, num, datafile)
save_list("tmin.txt", tmin_list)
del tmin_list   # huge list, save memory

nmin_list = list()
load_data(nmin_list, num, datafile)
save_list("nmin.txt", nmin_list)
del nmin_list   # huge list, save memory

Answer 2

从R代码推断，二进制文件首先包含一个特定的数tmax，然后是相同数量的nmax，然后是{}和{}。该代码所做的是读取整个文件，然后使用切片将其分成4个部分（tmax、nmax等）。在

要在python中执行相同操作，请执行以下操作：

import struct

# Read entire file into memory first. This is done so we can count
# number of bytes before parsing the bytes. It is not a very memory
# efficient way, but it's the easiest. The R-code as posted wastes even
# more memory: it always takes 6e8 * 4 bytes (~ 2.2Gb) of memory no
# matter how small the file may be.
#
data = open('data.bin','rb').read()

# Calculate number of points in the file.  This is 
# file-size / 16, because there are 4 numeric()'s per
# point, and they are 4 bytes each.
#
num = int(len(data) / 16)

# Now we know how much there are, we take all tmax numbers first, then
# all nmax's, tmin's and lastly all nmin's.

# First generate a format string because it depends on the number points
# there are in the file. It will look like: "fffff"
#
format_string = 'f' * num

# Then, for cleaner code, calculate chunk size of the bytes we need to 
# slice off each time.
#
n = num * 4     # 4-byte floats

# Note that python has different interpretation of slicing indices
# than R, so no "+1" is needed here as it is in the R code.
#
tmax = struct.unpack(format_string, data[:n])
nmax = struct.unpack(format_string, data[n:2*n])
tmin = struct.unpack(format_string, data[2*n:3*n])
nmin = struct.unpack(format_string, data[3*n:])

print("tmax", tmax)
print("nmax", nmax)
print("tmin", tmin)
print("nmin", nmin)

如果目标是将此数据结构化为点列表（？）像(tmax,nmax,tmin,nmin)，然后将其附加到代码中：

^{pr2}$