python – Threads中的subprocess.Popen

概述我有很多文件(超过4000个),我想同时加载到PostgreSQL中.我已将它们分成4个不同的文件列表,我想要一个线程迭代加载数据的每个列表. 我遇到的问题是我使用os.system来调用加载程序但这会阻止其他线程同时运行.如果我使用subprocess.Popen然后它们同时运行但是线程认为它们已经完成了execeuting,所以移动到我的脚本的下一部分. 我这样做是对的吗？或者是否有更好的方法 我有很多文件(超过4000个),我想同时加载到Postgresql中.我已将它们分成4个不同的文件列表,我想要一个线程迭代加载数据的每个列表.

我遇到的问题是我使用os.system来调用加载程序但这会阻止其他线程同时运行.如果我使用subprocess.Popen然后它们同时运行但是线程认为它们已经完成了execeuting,所以移动到我的脚本的下一部分.

我这样做是对的吗？或者是否有更好的方法从线程内调用子进程.

def thread1Load(self,thread1fileList):    connectionstring = settings.connectionstring    postgreshost = settings.postgreshost    postgresdatabase = settings.postgresdatabase    postgresport = settings.postgresport    postgresusername = settings.postgresusername    postgrespassword = settings.postgrespassword    tablename = None    enCoding = None    connection = psycopg2.connect(connectionstring)    for filename in thread1fileList:        load_cmd = #load command        run = subprocess.Popen(load_cmd,shell=True)    print "finished loading thread 1"def thread2load(self,thread2fileList):    connectionstring = settings.connectionstring    postgreshost = settings.postgreshost    postgresdatabase = settings.postgresdatabase    postgresport = settings.postgresport    postgresusername = settings.postgresusername    postgrespassword = settings.postgrespassword    tablename = None    connection = psycopg2.connect(connectionstring)    for filename in thread2fileList:        load_cmd = #load command                    run = subprocess.Popen(load_cmd,shell=True)    print "finished loading thread 2"def thread3Load(self,thread3fileList):    connectionstring = settings.connectionstring    postgreshost = settings.postgreshost    postgresdatabase = settings.postgresdatabase    postgresport = settings.postgresport    postgresusername = settings.postgresusername    postgrespassword = settings.postgrespassword    tablename = None    connection = psycopg2.connect(connectionstring)    for shapefilename in thread3fileList:        load_cmd = #load command        run = subprocess.Popen(load_cmd,shell=True)    print "finished loading thread 3"def thread4Load(self,thread4fileList):    connectionstring = settings.connectionstring    postgreshost = settings.postgreshost    postgresdatabase = settings.postgresdatabase    postgresport = settings.postgresport    postgresusername = settings.postgresusername    postgrespassword = settings.postgrespassword    tablename = None    connection = psycopg2.connect(connectionstring)    for filename in thread4fileList:        load_cmd = #load command        run = subprocess.Popen(load_cmd,shell=True)    print "finished loading thread 4"def finishUp(self):    print 'finishing up'def main():load = Loader()thread1 = threading.Thread(target=(load.thread1Load),args=(thread1fileList,))thread2 = threading.Thread(target=(load.thread2load),args=(thread2fileList,))thread3 = threading.Thread(target=(load.thread3Load),args=(thread3fileList,))thread4 = threading.Thread(target=(load.thread4Load),args=(thread4fileList,))threads = [thread1,thread2,thread3,thread4]for thread in threads:    thread.start()    thread.join()load.finishUp(connectionstring)if __name__ == '__main__':main()

解决方法 > Don’t repeat yourself.一个threadLoad方法就足够了.这样,如果您需要修改方法中的某些内容,则无需在4个不同的位置进行相同的修改.
>使用run.communicate()阻止,直到子进程完成.
>这将启动一个线程,然后阻塞直到该线程完成,然后
启动另一个线程等：

for thread in threads:    thread.start()    thread.join()

相反,首先启动所有线程,然后加入所有线程：

for thread in threads:    thread.start()for thread in threads:    thread.join()

import subprocessimport threadingclass Loader(object):    def threadLoad(self,threadfileList):        connectionstring = settings.connectionstring        ...        connection = psycopg2.connect(connectionstring)        for filename in threadfileList:            load_cmd =  # load command            run = subprocess.Popen(load_cmd,shell=True)            # block until subprocess is done            run.communicate()        name = threading.current_thread().name        print "finished loading {n}".format(n=name)    def finishUp(self):        print 'finishing up'def main():    load = Loader()    threads = [threading.Thread(target=load.threadLoad,args=(fileList,))               for fileList in (thread1fileList,thread2fileList,thread3fileList,thread4fileList)]    for thread in threads:        thread.start()    for thread in threads:        thread.join()    load.finishUp(connectionstring)if __name__ == '__main__':    main()

总结

以上是内存溢出为你收集整理的python – Threads中的subprocess.Popen全部内容，希望文章能够帮你解决python – Threads中的subprocess.Popen所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。