Python多进程数据选择和插入Psycopg2

2024-09-29 22:20:04 发布

您现在位置:Python中文网/ 问答频道 /正文

我试图选择大量数据并插入到另一个表中,因此使用python多处理模块 我尝试了多重处理,但看不到任何结果

import psycopg2, multiprocessing, time
from multiprocessing import pool
def pg_run(thetable):
conn = psycopg2.connect(database = 
"XXX",user="XXX",password="XXX",port="XXX")
conn.autocommit = True
cur = conn.cursor()
thesql = """INSERT  INTO public."table_A" (col1,col2,col3,col4,col5) 
SELECT col1,col2,col3,col4,col5 
FROM public."tableB" 
inner join public."tableC" VFCR on VFCRH.col= VFCR.col
inner join public."tableD" RSXSCX6 on VFCRH.col= RSXSCX6.col
or (VFCRH.col= RSXSCX6.col or VFCRH.col= RSXSCX6.col) limit 10
"""
cur.execute(thesql)
if __name__ == '__main__':
   t1 = time.time()
   p = multiprocessing.Pool(4)
   thetable1 = '1'
   thetable2 = '2'
   thetable3 = '3'
   thetable4 = '4'
 myresult = p.map(pg_run,[thetable1,thetable2,thetable3,thetable4])

thesql = """SELECT col1,col2,col3,col4,col5 
FROM public."tableB"
INTO public.tableB2
FROM public."tableB"
"""
conn = psycopg2.connect(database = "XXX",user="XXX",password="XXX",port="XXX")
conn.autocommit = True
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS tableB2")
cur.execute(thesql)
print (time.time() - t1)

数据应该更快地插入到目标表中


Tags: timecolpublicconnmultiprocessingpsycopg2col2col3

热门问题