python单元测试的执行顺序由它们的声明决定

Question

我正在使用python unittests和selenium，在我的代码中，我有一个测试类和许多测试用例：

class BasicRegression(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.driver = webdriver.Chrome(executable_path=Data.driver)
        cls.driver.implicitly_wait(1)
        cls.driver.maximize_window()

    def testcase1_some_stuff(self):
        do_something()

    def testcase2_some_stuff(self):
        do_something()

    def testcase3_some_stuff(self):
        do_something()

    ...

    @classmethod
    def tearDownClass(cls):
        cls.driver.close()
        cls.driver.quit()

if __name__ == '__main__':
    unittest.main()

测试按字母顺序执行，即testcase1、testcase2和testcase3，直到testcase9。标准问题出现在testcase10等等，它首先执行

我的问题是我如何设定他们的执行顺序

Answer 1

首先，单元测试应该是独立的。所以一定是python-unittest。通过python-unittest执行的测试应设计为能够独立运行。纯单元测试提供了一个好处，即当它们失败时，它们通常会描述出到底出了什么问题。尽管如此，我们还是倾向于使用unittest框架编写功能测试、集成测试和系统测试，这些测试在没有排序的情况下是不可行的，因为Selenium会自动执行浏览上下文。为了实现排序，您至少需要对测试名使用更好的命名约定，例如：test_1、test_2、test_3，等等，这是有效的，因为测试是根据字符串的内置顺序排序的

然而，根据您的观察，问题出现在test_10等排序顺序似乎中断的地方。例如，在名为test_1、test_2和test_10的3个测试中，unittest似乎在test_2之前执行test_10：

• 代码：

  import unittest
  
  class Test(unittest.TestCase):
  
      @classmethod
      def setUp(self):
          print("I'm in setUp")
  
      def test_1(self):
          print("I'm in test 1")
  
      def test_2(self):
          print("I'm in test 2")
  
      def test_10(self):
          print("I'm in test 10")
  
      @classmethod
      def tearDown(self):
          print("I'm in tearDown")
  
  if __name__ == "__main__":
      unittest.main()
  

• 控制台输出：

  Finding files... done.
  Importing test modules ... done.
  
  I'm in setUp
  I'm in test 1
  I'm in tearDown
  I'm in setUp
  I'm in test 10
  I'm in tearDown
  I'm in setUp
  I'm in test 2
  I'm in tearDown
                                     
  Ran 3 tests in 0.001s
  
  OK
  

---

解决方案

在不同的讨论中提出了不同的解决方案，其中一些解决方案如下：

• @max在讨论中Unittest tests order建议将sortTestMethodsUsing设置为None，如下所示：

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = None
  

• @atomocopter在讨论changing order of unit tests in Python中建议将sortTestMethodsUsing设置为某个值，如下所示：

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
  

• @ElmarZander在讨论中Unittest tests order建议使用nose并将您的测试用例编写为函数（而不是作为某些测试用例派生类的方法）nose不处理顺序，而是使用文件中定义的函数顺序

• @Keiji在讨论中提到：

sortTestMethodsUsing expects a function like Python 2's cmp, which has no equivalent in Python 3 (I went to check if Python 3 had a <=> spaceship operator yet, but apparently not; they expect you to rely on separate comparisons for < and ==, which seems much a backwards step...). The function takes two arguments to compare, and must return a negative number if the first is smaller. Notably in this particular case, the function may assume that the arguments are never equal, as unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated functions appear in the file. As far as I know, this is intended, and I'd be surprised if it changed, but I don't actually know if it's guaranteed behavior. I think this solution will even work in Python 2 as well, for those who are unfortunately stuck with it, though I haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class definition, though how this can be used with sortTestMethodsUsing will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test order being fixed - and I fully understand that you shouldn't rely on test order, and this is the reason people use to avoid answering this question. The order of my tests could be randomised and it'd work just as well. However, there is one very good reason I'd like the order to be fixed to the order they're defined in the file: it makes it so much easier to see at a glance which tests failed.