从C并行调用python

Question

我需要从我的C代码调用一个Python函数。 它工作得很好，但是当我想并行化时，它就崩溃了。 请参阅以下最小C代码：

#include <Python.h>
#include <stdio.h>

int main(void)
{
  double Z = 1.;
  double k = 1.;
  double l = 1.;
  double eta = -Z/k;

  Py_Initialize();

  PyObject* pName = PyString_FromString("mpmath");
  PyObject* pModule = PyImport_Import(pName);
  PyObject* pFunc = PyObject_GetAttrString(pModule, "coulombf");

  PyObject* pl = PyFloat_FromDouble(l);
  PyObject* peta = PyFloat_FromDouble(eta);

  int i;
#pragma omp parallel for private(i)
  for(i=0; i<10000; i++)
  {
    double r = 0.01*i;
    PyObject* prho = PyFloat_FromDouble(k*r);
    PyObject* pArgs = PyTuple_Pack(3, pl, peta, prho);
    PyObject* pValue = PyObject_CallObject(pFunc, pArgs);
    double value = PyFloat_AsDouble(pValue);
    printf("r=%.2f\tf=%.6f\n",r,value);
  }

  Py_Finalize();
  return 0;
}

让我们将这个文件命名为testPython.c，您可以用gcc -fopenmp testPython.c -o testPython -I/usr/include/python2.7 -L/usr/lib64/python2.7/config -lpython2.7编译它。在

现在用./testPython运行它，会看到这样的错误：Fatal Python error: GC object already tracked。（有时，错误信息不同。）

但是如果您编译它时不使用-fopenmp，那么这个程序可以完美地工作。在

我怎样才能克服这个问题？谢谢！在

编辑：

正如Natecat、johnbollinger和Olaf所回答的那样，多线程处理不太可能加快处理速度，但多处理确实可以加快计算速度。纯python脚本如下所示：

^{pr2}$

但是在C语言中怎么做呢？我还没找到路。在

Answer 1

@Natecat的回答基本上是对的，如果在细节和细微差别上有点欠缺的话。The docs of Python's C API给出一个更完整的图片。假设这是您正在使用的Python实现，您需要注意以下几点：

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program [...].

Therefore, the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions. In order to emulate concurrency of execution, the interpreter regularly tries to switch threads (see sys.setswitchinterval()). The lock is also released around potentially blocking I/O operations like reading or writing a file, so that other Python threads can run in the meantime.

以及

when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.

注意：OpenMP就是这样。在

If you need to call Python code from these threads [...] you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API. When you are done, you should reset the thread state pointer, release the GIL, and finally free the thread state data structure.

The PyGILState_Ensure() and PyGILState_Release() functions do all of the above automatically. The typical idiom for calling into Python from a C thread is:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

为了允许多个OpenMP线程安全地对同一个CPython解释器进行并发调用，您必须实现该模式，但是您不太可能从并行化中获得太多好处，因为各种OpenMP线程将在很大程度上被阻止并发运行。在

Answer 2

在python中不可能有任何类型的真正的多线程（例如在一个进程中使用多个系统线程），至少在最常见的python实现中是不可能的。您既可以不使用任何类型的并行化，也可以切换到没有GIL的实现。以下是一篇关于这个主题的更多信息的文章：https://wiki.python.org/moin/GlobalInterpreterLock