如何在不丢失大于32位的整数的情况下加载保存在pandas中的HDF5文件中的数据帧？

Question

当我试图加载保存在pandas中作为HDF5文件保存在R:

Warning message: In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem, : NAs produced by integer overflow while converting 64-bit integer or unsigned 32-bit integer from HDF5 to a 32-bit integer in R. Choose bit64conversion='bit64' or bit64conversion='double' to avoid data loss and see the vignette 'rhdf5' for more details about 64-bit integers.

例如，如果我在pandas中创建HDF5文件：

import pandas as pd

frame = pd.DataFrame({
    'time':[1234567001,1234515616515167005],
    'X2':[23.88,23.96]
},columns=['time','X2'])

store = pd.HDFStore('a.hdf5')
store['df'] =  frame
store.close()
print(frame)

返回：

^{pr2}$

试着在R中加载：

#source("http://bioconductor.org/biocLite.R")
#biocLite("rhdf5")
library(rhdf5)

loadhdf5data <- function(h5File) {
  # Function taken from [How can I load a data frame saved in pandas as an HDF5 file in R?](https://stackoverflow.com/a/45024089/395857)
  listing <- h5ls(h5File)
  # Find all data nodes, values are stored in *_values and corresponding column
  # titles in *_items
  data_nodes <- grep("_values", listing$name)
  name_nodes <- grep("_items", listing$name)

  data_paths = paste(listing$group[data_nodes], listing$name[data_nodes], sep = "/")
  name_paths = paste(listing$group[name_nodes], listing$name[name_nodes], sep = "/")

  columns = list()
  for (idx in seq(data_paths)) {
    print(idx)
    data <- data.frame(t(h5read(h5File, data_paths[idx])))
    names <- t(h5read(h5File, name_paths[idx],  bit64conversion='bit64'))
    #names <- t(h5read(h5File, name_paths[idx],  bit64conversion='double'))
    entry <- data.frame(data)
    colnames(entry) <- names
    columns <- append(columns, entry)
  }

  data <- data.frame(columns)

  return(data)
}

frame  = loadhdf5data("a.hdf5")

我收到警告信息：

> frame = loadhdf5data("a.hdf5")
[1] 1
[1] 2
Warning message:
In H5Dread(h5dataset = h5dataset, h5spaceFile = h5spaceFile, h5spaceMem = h5spaceMem,  :
  NAs produced by integer overflow while converting 64-bit integer or unsigned 32-bit integer from HDF5 to a 32-bit integer in R. Choose bit64conversion='bit64' or bit64conversion='double' to avoid data loss and see the vignette 'rhdf5' for more details about 64-bit integers.

我可以看到其中一个时间值变成了NA：

> frame
     X2       time
1 23.88 1234567001
2 23.96         NA

我如何解决这个问题？选择bit64conversion='bit64'或bit64conversion='double'不会改变任何东西。在

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          4.0                         
year           2017                        
month          04                          
day            21                          
svn rev        72570                       
language       R                           
version.string R version 3.4.0 (2017-04-21)
nickname       You Stupid Darkness         

Answer 1

HDF5 Dataset Interface's documentation表示：

bit64conversion: Defines, how 64-bit integers are converted. Internally, R does not support 64-bit integers. All integers in R are 32-bit integers. By setting bit64conversion='int', a coercing to 32-bit integers is enforced, with the risc of data loss, but with the insurance that numbers are represented as integers. bit64conversion='double' coerces the 64-bit integers to floating point numbers. doubles can represent integers with up to 54-bits, but they are not represented as integer values anymore. For larger numbers there is again a data loss. bit64conversion='bit64' is recommended way of coercing. It represents the 64-bit integers as objects of class 'integer64' as defined in the package 'bit64'. Make sure that you have installed 'bit64'. The datatype 'integer64' is not part of base R, but defined in an external package. This can produce unexpected behaviour when working with the data.

因此，您应该安装bit64（install.packages("bit64")）并加载它（library(bit64)）。您可以检查integer64是否已加载：

> integer64
Function (length = 0) 
{
    ret <- double(length)
    oldClass(ret) <- "integer64"
    ret
}
<bytecode: 0x000000001a7a95f0>
<environment: namespace :it64>

现在您可以运行：

^{pr2}$

它给出了：

> frame
     X2                time
1 23.88          1234567001
2 23.96 1234515616515167005