java我应该如何并行化计算代价高昂的for循环并整理迭代结果？

1 月，2 周 Questions & Answers 143

我在一台8核机器上工作，正在执行一项计算繁重的任务。然而，任务的每次执行（即for循环的迭代）都与前一个任务相当独立。从一次执行到下一次执行，只有一些变量被“汇总”。我猜这是并行化/线程化的一个很好的例子，但我不知道该怎么做

下面是代码的外观。现在，它只是我的主executor类中主方法的一部分：

double testerPayoffSum = 0.0, developerPayoffSum = 0.0;
Random seed = new Random();

        try {
            for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
                EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
                Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();

                assert (gameStrategy != null);

                TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);                
                PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());

                testerPayoffSum += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
                developerPayoffSum += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
                System.out.print("Output: ERASER Games played; Number of developers caught");
                System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");

            } catch(Exception e){sendEmailAlert("Execution Failed with Exception");}

如果可能的话，我想并行化for-loop计算，并继续对testerPayoffSum和developerPayofffSum变量求和。我怎样才能做到这一点

注意：for循环的每次执行大约需要20-30分钟，具体取决于输入大小（由各种GameConstant设置）。即使对少数人来说，上述过程也需要将近2-3个小时

Tags:

共 (3) 个答案

# 1 楼答案

声明队列以收集结果并将任务提交到线程池：

    final ArrayBloclingQueue<PlayerPayoffs> queue=new ArrayBloclingQueue<PlayerPayoffs>();
    Executor exec=new Executors.newFixedThreadPool(N); // number of threads depends on hardware
    for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
        exec.execute(new Runnable(){          
            EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
            Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();

            assert (gameStrategy != null);

            TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);      
            PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());
            queue.put(payoffs);
        });
    }

然后收集并汇总结果：

    double testerPayoffSum = 0.0, developerPayoffSum = 0.0;
    for (int i = 0; i < GameConstants.MAX_GAMES; i++) {
            PlayerPayoffs payoffs=queue.take();
            testerPayoffSum += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
            developerPayoffSum += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
            System.out.print("Output: ERASER Games played; Number of developers caught");
            System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");
    }

# 2 楼答案

创建一个线程对象来实现Callable，它返回一个包含testerPayoffSum和developerPayoffSum的Future对象，开始计算并汇总从Future获得的结果（另请参见https://blogs.oracle.com/CoreJavaTechTips/entry/get_netbeans_6）

# 3 楼答案

你确定你没有依赖关系吗

一,。使用过的类不能共享任何变量

如果是这样，那么你必须添加锁
但它会影响性能
如果一些共享变量被广泛使用
即使在非并行执行下，性能也会显著下降

二,。使用过的课程不得使用任何类型的机器学习

没有解决办法
因为并行化会破坏你的结果

现在如何做（我不是java编码器，所以我坚持C++代码）。p>

// - globals and headers                           -

unsigned long __stdcall function(LPVOID p);
Random seed = new Random();
const int N=8;          // threads count (<=CPU count)
int id[N];          // thread id
int max[N];         // number of games per thread
double testerPayoffSum[N];  // sum to separate variables to avoid locks need
double developerPayoffSum[N];
volatile int run=0,stop=0;  // thread control variables run is number of running threads and stop force stop...

// - main code                                -

// init some variables ... may be the seed init will be better here too
int i;                      
for (i = 0; i < N; i++) 
    {
    id[i]=i;
    max[i]=GameConstants.MAX_GAMES / N;
    testerPayoffSum[i]=0.0;
    developerPayoffSum[i]=0.0;
    } 
max[0]=GameConstants.MAX_GAMES % N;

// create threads
for (i = 0; i < N; i++)
    {
    HANDLE hnd=CreateThread(0,0,function,&id[i],0,0); 
    if (hnd!=NULL) CloseHandle(hnd); // this line is important !!! 
    // because if you do not close Handle it will be allocated until the end of app
    // handle leaks are nasty and cause weird OS behaviour
    // I saw many times this bug in commercial drivers
    // it is a nightmare for 24/7 software
    }

// wait for them
while (run) Sleep(200);

// sum the results to [0]
for (i = 1; i < N; i++)
    {
    testerPayoffSum[0]   +=testerPayoffSum[i];
    developerPayoffSum[0]+=developerPayoffSum[i];
    }
// here do what you need to do with the results

// - thread function                             -
unsigned long __stdcall function(LPVOID p)
    {
    run++;
    int ix=((int*)p)[0];
    for (i = 0; i < max[ix]; i++)
        {
        if (stop) break;
        EraserSimulator eraser = new EraserSimulator(GameConstants.MAX_TARGETS, GameConstants.MAX_RESOURCES, GameConstants.NUM_ATTACKER_TYPES, seed.nextInt());
        Map<Set<SingleObjectiveTarget>, Double> gameStrategy = eraser.run();
        assert (gameStrategy != null);
        TestingGameSimulator testingGame = new TestingGameSimulator(GameConstants.MAX_TARGETS, gameStrategy, GameConstants.NUM_GAMES_TO_STORE_FOR_HISTORY, GameConstants.NUM_TESTING_GAMES_TO_PLAY);                
        PlayerPayoffs payoffs = testingGame.run(eraser.getEraserInstance());
        testerPayoffSum[ix] += payoffs.getAverageTesterPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
        developerPayoffSum[ix] += payoffs.getAverageDeveloperPayoff(GameConstants.NUM_TESTING_GAMES_TO_PLAY);
//      do not call any visual stuff from thread !!! sometimes it can cause a lot of problems ...
//      instead cretae some global string variable and set it to what shoud be printed out
//      and inside wait while loop in main code add if string != "" then System.out.print(string);
//      but in that case you should add lock to it.
//      System.out.print("Output: ERASER Games played; Number of developers caught");
//      System.out.print(", " + GameConstants.NUM_TESTING_GAMES_TO_PLAY + ", " + payoffs.getNumTimesCaught() + "\n");
        //Sleep(100); // well placed sleep
        }
    run ;
    }

[注]

根据你的代码我假设gamestants是共享变量
如果只是为了阅读，那就没问题了
但如果你也在线程内部写（我怀疑是的）
然后你有一个大问题，因为你需要在游戏类中添加锁，然后
如果没有机器学习，那么你可以避免这种情况
通过为每个线程创建单独的GameConstants变量，比如。。。游戏常数[N]
但是你需要重写代码，这样它才能访问GameStants[ix]，而不是GameStants

[锁]

不知道锁是如何在JAVA中实现的

但是你也可以用你自己的东西

class _lock
 {
public:
 volatile bool locked;
 _lock() { locked=false; }
 void lock() { while(locked) Sleep(1); locked=true; }
 void unlock() { locked=false; }
 };

// now for each shared variable (or group of variables) add one global _lock variable
_lock l1; int sv1; // shared variable 1 and her lock

// any write access and sometimes also read access needs lock
l1.lock();
sv1++;
l1.unlock();

请注意，锁有时会导致应用程序冻结，尤其是在重载使用时
不管它是自己的锁还是OS锁
这主要发生在线程内部而不是主线程中混合视觉内容或一些操作系统调用时
在这种情况下，有时适当的睡眠有助于避免线程内部的操作系统调用
因为它会导致很多其他问题
也要尽可能短的时间内锁定，因为一旦发生冲突，冲突的线程就会停止
因此，不能只在循环开始时添加锁，然后在循环结束时解锁
因为并行加速将丢失

Python中文网

有 Java 编程相关的问题?

java我应该如何并行化计算代价高昂的for循环并整理迭代结果？

共 (3) 个答案

# 1 楼答案

# 2 楼答案

# 3 楼答案