<p>我有这个日志文件“互联网.log“大约是10GB。当我在python中解析它时,会得到一个异常“MemoryError”。日志文件如下所示。。。你知道吗</p>
<pre><code>Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.107
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi uno.gycpi.b.yahoodns.net is 216.115.100.123
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.124
Jun 15 16:26:21 dnsmasq[1979]: reply fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.106
Jun 15 16:26:21 dnsmasq[1979]: query[A] fd-geoycpi-uno.gycpi.b.yahoodns.net from 192.168.1.33
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.106
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.124
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 216.115.100.123
Jun 15 16:26:21 dnsmasq[1979]: cached fd-geoycpi-uno.gycpi.b.yahoodns.net is 74.6.160.107
Jun 15 16:26:23 dnsmasq[1979]: query[A] armdl.adobe.com from 192.168.1.24
</code></pre>
<p>我当前正在使用此方法分析日志文件:</p>
<pre><code>def parse():
Date = []
IPAddress = []
DomainsVisited = []
with open("internet.log", "r") as file:
content = file.readlines()
for items in content:
if 'query[A]' in items:
getDate(Date, items)
getIPAddress(IPAddress, items)
getDomainsVisited(DomainsVisited, items)
finalResult = [[i, j, k] for i, j, k in zip(Date, IPAddress, DomainsVisited)]
return display(finalResult)
</code></pre>
<p>如果我解析一个10MB的日志文件,输出就会显示出来,但是当我去解析10GB的日志文件时,我得到了一个错误。我怎样才能解决这个问题?非常感谢。你知道吗</p>
<p>您不应该使用<code>file.readlines()</code>。这样做会立即将整个文件读入内存,很可能会立即将其填满。相反,迭代文件:</p>
<pre><code>with open("internet.log", "r") as file:
for items in file:
</code></pre>
<p>(当然,这取决于您对数据的处理方式,在浏览文件时仍然可能中断。)</p>