读取包含单词“upper”的.dat文件会导致问题

Question

这是我用来读入.dat文件的代码：

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.dat', sep='\s+')

plt.figure()
plt.plot(df['yh_center[2]'], df['tot_scale01[4]'], 'ro')
plt.xlabel('yh_center')
plt.ylabel('tot_scale01[4]')
plt.savefig('name.pdf')
plt.show()

当dat中没有单独的单词“upper”时，它会正确读入。文件。 然而，我将有许多.dat文件，其中涉及的“上”字，导致错误。你知道吗

.dat文件如下所示：

#labels: yh_lower[1]   yh_center[2]   yh_upper[3]   tot_scale01[4]
#neval: 200000
#overflow:lower center upper    0.0000000000E+000    0.0000000000E+000    0.0000000000E+000    0.0000000000E+000

然后在第4行输入实际数据

-4.4000000000E+000   -4.3000000000E+000   -4.2000000000E+000    0.0000000000E+000

我还有43行数据，最后一行是

#nx: 3

（实际上有超过100列的数据，但这应该会改变…原理应该用前4列显示）

完整的错误报告是

Traceback (most recent call last):

  File "<ipython-input-1-00512d6cb966>", line 1, in <module>
    runfile('Private, so deleted that one here')

  File "/usr/lib/python3.4/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 685, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3.4/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 85, in execfile
    exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

  File "/home/ttp/manbrunn/Documents/NNLOJETexe/HJtest/funktioniert.py", line 15, in <module>
    plt.plot(df['yh_center[2]'], df['tot_scale01[4]'], 'ro')

  File "/usr/lib64/python3.4/site-packages/matplotlib/pyplot.py", line 3099, in plot
    ret = ax.plot(*args, **kwargs)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_axes.py", line 1374, in plot
    self.add_line(line)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 1504, in add_line
    self._update_line_limits(line)

  File "/usr/lib64/python3.4/site-packages/matplotlib/axes/_base.py", line 1515, in _update_line_limits
    path = line.get_path()

  File "/usr/lib64/python3.4/site-packages/matplotlib/lines.py", line 874, in get_path
    self.recache()

  File "/usr/lib64/python3.4/site-packages/matplotlib/lines.py", line 575, in recache
    x = np.asarray(xconv, np.float_)

  File "/usr/lib64/python3.4/site-packages/numpy/core/numeric.py", line 462, in asarray
    return array(a, dtype, copy=False, order=order)

ValueError: could not convert string to float: 'upper'

Answer 1

您可以通过在将数据传递给pandas之前删除每行的任何前导文本来预分析数据，例如：

import pandas as pd
import re

with open('data.dat') as f_input:
    header = next(f_input).split()[1:]
    next(f_input)
    next(f_input)

    data = [line.strip().split() for line in f_input]

df = pd.DataFrame(data[:-1], dtype='float', columns=header)

此版本还可以从第一行提取列名。你知道吗

---

如果upper只出现在第三行（而不是在整个文件中随机出现），您可以简单地使用skiprows来告诉Pandas从第四行开始读取，并且由于您在最后一行中有一个#nx: 3，因此也可以使用skipfooter来跳过该行：

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.dat', sep='\s+', skiprows=3, skipfooter=1, header=None)

在熊猫身上测试0.22.0