<p>列“url”的唯一约束是在url上创建隐式索引。这就解释了尺寸的增加。在</p>
<p>我不认为您可以填充表,然后添加unique约束。在</p>
<p>你的瓶颈肯定是CPU。尝试以下操作:</p>
<ol>
<li>安装工具z:<code>pip install toolz</code></li>
<li><p>使用此方法:</p>
<pre><code>from toolz import partition_all
def add_blacklist_url(self, urls):
# print('add_blacklist_url:: entries = {}'.format(len(urls)))
start_time = datetime.now()
for batch in partition_all(100000, urls):
try:
start_commit = datetime.now()
self.cursor.executemany('''INSERT OR IGNORE INTO blacklist(url) VALUES(:url)''', batch)
end_commit = datetime.now() - start_commit
print('add_blacklist_url:: total time for INSERT OR IGNORE INTO blacklist {} entries = {}'.format(len(templist), end_commit))
except sqlite3.Error as e:
print("add_blacklist_url:: Database error: %s" % e)
except Exception as e:
print("add_blacklist_url:: Exception in _query: %s" % e)
self.db.commit()
time_elapsed = datetime.now() - start_time
print('add_blacklist_url:: total time for {} entries = {}'.format(records, time_elapsed))
</code></pre></li>
</ol>
<p>代码未经测试。在</p>