pytest 测试输出和结果-管理pytest的输出

修改 Python 回溯打印

修改回溯打印示例：

pytest --showlocals # show local variables in tracebacks
pytest -l           # show local variables (shortcut)

pytest --tb=auto    # (default) 'long' tracebacks for the first and last
                     # entry, but 'short' style for the other entries
pytest --tb=long    # exhaustive, informative traceback formatting
pytest --tb=short   # shorter traceback format
pytest --tb=line    # only one line per failure
pytest --tb=native  # Python standard library formatting
pytest --tb=no      # no traceback at all

​--full-trace​ 导致在错误时打印很长的跟踪（长于 ​--tb=long​）。 它还确保在 ​KeyboardInterrupt(Ctrl+C) 上打印堆栈跟踪。 如果测试花费的时间太长并且您使用 Ctrl+C 中断它们以找出测试挂起的位置，这将非常有用。 默认情况下不会显示任何输出（因为 ​KeyboardInterrupt被 pytest 捕获）。 通过使用此选项，您可以确保显示跟踪。

冗长

​-v​ 标志在各个方面控制 pytest 输出的详细程度：测试会话进度、测试失败时的断言详细信息、带有 ​--fixtures​ 的固定详细信息等。

考虑这个简单的文件：

# content of test_verbosity_example.py
def test_ok():
    pass


def test_words_fail():
    fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"]
    fruits2 = ["banana", "apple", "orange", "melon", "kiwi"]
    assert fruits1 == fruits2


def test_numbers_fail():
    number_to_text1 = {str(x): x for x in range(5)}
    number_to_text2 = {str(x * 10): x * 10 for x in range(5)}
    assert number_to_text1 == number_to_text2


def test_long_text_fail():
    long_text = "Lorem ipsum dolor sit amet " * 10
    assert "hello world" in long_text

执行 pytest 通常会给我们这个输出：

$ pytest --no-header
=========================== test session starts ============================
collected 4 items

test_verbosity_example.py .FFF                                       [100%]

================================= FAILURES =================================
_____________________________ test_words_fail ______________________________

    def test_words_fail():
        fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"]
        fruits2 = ["banana", "apple", "orange", "melon", "kiwi"]
>       assert fruits1 == fruits2
E       AssertionError: assert ['banana', 'a...elon', 'kiwi'] == ['banana', 'a...elon', 'kiwi']
E         At index 2 diff: 'grapes' != 'orange'
E         Use -v to get more diff

test_verbosity_example.py:8: AssertionError
____________________________ test_numbers_fail _____________________________

    def test_numbers_fail():
        number_to_text1 = {str(x): x for x in range(5)}
        number_to_text2 = {str(x * 10): x * 10 for x in range(5)}
>       assert number_to_text1 == number_to_text2
E       AssertionError: assert {'0': 0, '1':..., '3': 3, ...} == {'0': 0, '10'...'30': 30, ...}
E         Omitting 1 identical items, use -vv to show
E         Left contains 4 more items:
E         {'1': 1, '2': 2, '3': 3, '4': 4}
E         Right contains 4 more items:
E         {'10': 10, '20': 20, '30': 30, '40': 40}
E         Use -v to get more diff

test_verbosity_example.py:14: AssertionError
___________________________ test_long_text_fail ____________________________

    def test_long_text_fail():
        long_text = "Lorem ipsum dolor sit amet " * 10
>       assert "hello world" in long_text
E       AssertionError: assert 'hello world' in 'Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ips... sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet '

test_verbosity_example.py:19: AssertionError
========================= short test summary info ==========================
FAILED test_verbosity_example.py::test_words_fail - AssertionError: asser...
FAILED test_verbosity_example.py::test_numbers_fail - AssertionError: ass...
FAILED test_verbosity_example.py::test_long_text_fail - AssertionError: a...
======================= 3 failed, 1 passed in 0.12s ========================

请注意：

• 文件中的每个测试都由输出中的单个字符显示：​.​为 通过，​F​ 为失败。
• ​test_words_fail ​失败，我们看到一个简短的摘要，表明两个列表的索引 2 不同。
• ​test_numbers_fail ​失败，我们会看到字典项目左右差异的摘要。 相同的项目被省略。
• ​test_long_text_fail ​失败，并且 ​in ​语句的右侧使用 ​...`​ 被截断，因为它比内部阈值长（当前为 240 个字符）。

现在我们可以增加 pytest 的详细程度：

$ pytest --no-header -v
=========================== test session starts ============================
collecting ... collected 4 items

test_verbosity_example.py::test_ok PASSED                            [ 25%]
test_verbosity_example.py::test_words_fail FAILED                    [ 50%]
test_verbosity_example.py::test_numbers_fail FAILED                  [ 75%]
test_verbosity_example.py::test_long_text_fail FAILED                [100%]

================================= FAILURES =================================
_____________________________ test_words_fail ______________________________

    def test_words_fail():
        fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"]
        fruits2 = ["banana", "apple", "orange", "melon", "kiwi"]
>       assert fruits1 == fruits2
E       AssertionError: assert ['banana', 'a...elon', 'kiwi'] == ['banana', 'a...elon', 'kiwi']
E         At index 2 diff: 'grapes' != 'orange'
E         Full diff:
E         - ['banana', 'apple', 'orange', 'melon', 'kiwi']
E         ?                      ^  ^^
E         + ['banana', 'apple', 'grapes', 'melon', 'kiwi']
E         ?                      ^  ^ +

test_verbosity_example.py:8: AssertionError
____________________________ test_numbers_fail _____________________________

    def test_numbers_fail():
        number_to_text1 = {str(x): x for x in range(5)}
        number_to_text2 = {str(x * 10): x * 10 for x in range(5)}
>       assert number_to_text1 == number_to_text2
E       AssertionError: assert {'0': 0, '1':..., '3': 3, ...} == {'0': 0, '10'...'30': 30, ...}
E         Omitting 1 identical items, use -vv to show
E         Left contains 4 more items:
E         {'1': 1, '2': 2, '3': 3, '4': 4}
E         Right contains 4 more items:
E         {'10': 10, '20': 20, '30': 30, '40': 40}
E         Full diff:
E         - {'0': 0, '10': 10, '20': 20, '30': 30, '40': 40}...
E
E         ...Full output truncated (3 lines hidden), use '-vv' to show

test_verbosity_example.py:14: AssertionError
___________________________ test_long_text_fail ____________________________

    def test_long_text_fail():
        long_text = "Lorem ipsum dolor sit amet " * 10
>       assert "hello world" in long_text
E       AssertionError: assert 'hello world' in 'Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet '

test_verbosity_example.py:19: AssertionError
========================= short test summary info ==========================
FAILED test_verbosity_example.py::test_words_fail - AssertionError: asser...
FAILED test_verbosity_example.py::test_numbers_fail - AssertionError: ass...
FAILED test_verbosity_example.py::test_long_text_fail - AssertionError: a...
======================= 3 failed, 1 passed in 0.12s ========================

现在请注意：

• 文件中的每个测试在输出中都有自己的行。
• ​test_words_fail ​现在完整显示两个失败列表，除了索引不同。
• ​test_numbers_fail ​现在显示两个字典的文本差异，被截断。
• ​test_long_text_fail ​不再截断in语句的右侧，因为截断的内部阈值现在更大（当前为 2400 个字符）。

现在，如果我们进一步增加详细程度：

$ pytest --no-header -vv
=========================== test session starts ============================
collecting ... collected 4 items

test_verbosity_example.py::test_ok PASSED                            [ 25%]
test_verbosity_example.py::test_words_fail FAILED                    [ 50%]
test_verbosity_example.py::test_numbers_fail FAILED                  [ 75%]
test_verbosity_example.py::test_long_text_fail FAILED                [100%]

================================= FAILURES =================================
_____________________________ test_words_fail ______________________________

    def test_words_fail():
        fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"]
        fruits2 = ["banana", "apple", "orange", "melon", "kiwi"]
>       assert fruits1 == fruits2
E       AssertionError: assert ['banana', 'apple', 'grapes', 'melon', 'kiwi'] == ['banana', 'apple', 'orange', 'melon', 'kiwi']
E         At index 2 diff: 'grapes' != 'orange'
E         Full diff:
E         - ['banana', 'apple', 'orange', 'melon', 'kiwi']
E         ?                      ^  ^^
E         + ['banana', 'apple', 'grapes', 'melon', 'kiwi']
E         ?                      ^  ^ +

test_verbosity_example.py:8: AssertionError
____________________________ test_numbers_fail _____________________________

    def test_numbers_fail():
        number_to_text1 = {str(x): x for x in range(5)}
        number_to_text2 = {str(x * 10): x * 10 for x in range(5)}
>       assert number_to_text1 == number_to_text2
E       AssertionError: assert {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4} == {'0': 0, '10': 10, '20': 20, '30': 30, '40': 40}
E         Common items:
E         {'0': 0}
E         Left contains 4 more items:
E         {'1': 1, '2': 2, '3': 3, '4': 4}
E         Right contains 4 more items:
E         {'10': 10, '20': 20, '30': 30, '40': 40}
E         Full diff:
E         - {'0': 0, '10': 10, '20': 20, '30': 30, '40': 40}
E         ?            -    -    -    -    -    -    -    -
E         + {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4}

test_verbosity_example.py:14: AssertionError
___________________________ test_long_text_fail ____________________________

    def test_long_text_fail():
        long_text = "Lorem ipsum dolor sit amet " * 10
>       assert "hello world" in long_text
E       AssertionError: assert 'hello world' in 'Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet '

test_verbosity_example.py:19: AssertionError
========================= short test summary info ==========================
FAILED test_verbosity_example.py::test_words_fail - AssertionError: asser...
FAILED test_verbosity_example.py::test_numbers_fail - AssertionError: ass...
FAILED test_verbosity_example.py::test_long_text_fail - AssertionError: a...
======================= 3 failed, 1 passed in 0.12s ========================

现在请注意：

• 文件中的每个测试在输出中都有自己的行。
• 在这种情况下，​test_words_fail给出与以前相同的输出。
• ​test_numbers_fail现在显示两个字典的全文差异。
• ​test_long_text_fail也不会像以前那样在右侧截断，但现在 pytest 根本不会截断任何文本，无论其大小。

这些是详细程度如何影响正常测试会话输出的示例，但详细程度也用于其他情况，例如，如果您使用 ​pytest --fixtures -v​，甚至会显示以_ 开头的​fixture​。

支持使用更高的详细级别（​-vvv​，​-vvvv​，...），但目前对 pytest 本身没有影响，但是某些插件可能会使用更高的详细级别。

制作详细的总结报告

​-r​ 标志可用于在测试会话结束时显示简短的测试摘要信息，从而在大型测试套件中轻松获得所有失败、跳过、​xfails等的清晰图片。

它默认为 ​fE以列出失败和错误。

例如：

# content of test_example.py
import pytest


@pytest.fixture
def error_fixture():
    assert 0


def test_ok():
    print("ok")


def test_fail():
    assert 0


def test_error(error_fixture):
    pass


def test_skip():
    pytest.skip("skipping this test")


def test_xfail():
    pytest.xfail("xfailing this test")


@pytest.mark.xfail(reason="always xfail")
def test_xpass():
    pass

$ pytest -ra
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
rootdir: /home/sweet/project
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
SKIPPED [1] test_example.py:22: skipping this test
XFAIL test_example.py::test_xfail
  reason: xfailing this test
XPASS test_example.py::test_xpass always xfail
ERROR test_example.py::test_error - assert 0
FAILED test_example.py::test_fail - assert 0
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

​-r​ 选项接受其后的多个字符，上面使用的 ​a​ 表示除通过之外的所有字符。

以下是可以使用的可用字符的完整列表：

• ​f- 失败
• ​E- 错误
• ​s- 跳过
• ​x- xfailed
• ​X- xpassed
• ​p- 通过
• ​P- 通过输出

取消选择组的特殊字符：

• ​a- 除 ​pP外的所有
• ​A- 全部
• ​N- 无，这可用于不显示任何内容（因为 ​fE是默认值）

可以使用多个字符，例如只查看失败和跳过的测试，您可以执行：

$ pytest -rfs
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
rootdir: /home/sweet/project
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
FAILED test_example.py::test_fail - assert 0
SKIPPED [1] test_example.py:22: skipping this test
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

使用 ​p列出通过的测试，而 ​P添加一个额外的部分“PASSES”，其中包含那些通过但已捕获输出的测试：

$ pytest -rpP
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-7.x.y, pluggy-1.x.y
rootdir: /home/sweet/project
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
================================== PASSES ==================================
_________________________________ test_ok __________________________________
--------------------------- Captured stdout call ---------------------------
ok
========================= short test summary info ==========================
PASSED test_example.py::test_ok
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

创建结果日志格式文件

要创建纯文本机器可读的结果文件，您可以：

pytest --resultlog=path

查看路径位置的内容。使用此类文件，例如通过 PyPy-test 网页显示多个修订版的测试结果。

创建 JUnitXML 格式文件

要创建 ​Jenkins或其他持续集成服务器可以读取的结果文件，请使用以下调用：

pytest --junitxml=path

创建一个XML文件在路径。

要设置root测试套件的名称，你可以在配置文件中配置​junit_suite_name​选项:

[pytest]
junit_suite_name = my_suite

​JUnit XML​ 规范似乎表明时间属性应该报告总测试执行时间，包括设置和拆卸 (1, 2)。 这是默认的 pytest 行为。 要仅报告通话持续时间，请配置 ​junit_duration_report ​选项，如下所示：

[pytest]
junit_duration_report = call

record_property

如果要记录测试的其他信息，可以使用 ​record_property fixture​：

def test_function(record_property):
    record_property("example_key", 1)
    assert True

这将为生成的测试用例标签添加一个额外的属性 ​example_key="1"​：

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="example_key" value="1" />
  </properties>
</testcase>

或者，您可以将此功能与自定义标记集成：

# content of conftest.py


def pytest_collection_modifyitems(session, config, items):
    for item in items:
        for marker in item.iter_markers(name="test_id"):
            test_id = marker.args[0]
            item.user_properties.append(("test_id", test_id))

在你的测试中：

# content of test_function.py
import pytest


@pytest.mark.test_id(1501)
def test_function():
    assert True

结果如下：

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="test_id" value="1501" />
  </properties>
</testcase>

请注意，使用此功能将破坏最新 ​JUnitXML模式的模式验证。 当与某些 ​CI服务器一起使用时，这可能是一个问题。

record_xml_attribute

要向​testcase​元素添加额外的xml属性，可以使用​record_xml_attribute fixture​。这也可以用来覆盖现有的值:

def test_function(record_xml_attribute):
    record_xml_attribute("assertions", "REQ-1234")
    record_xml_attribute("classname", "custom_classname")
    print("hello world")
    assert True

与 ​​record_property​不同，这不会添加新的子元素。 相反，这将在生成的测试用例标签内添加一个属性 ​​assertions="REQ-1234"​ ​并用 ​​"classname=custom_classname​"​ 覆盖默认类名：

<testcase classname="custom_classname" file="test_function.py" line="0" name="test_function" time="0.003" assertions="REQ-1234">
    <system-out>
        hello world
    </system-out>
</testcase>

​record_xml_attribute是一个实验性功能，在未来的版本中，它的界面可能会被更强大和更通用的东西所取代。 然而，功能本身将被保留。

在使用 ​ci工具解析 ​xml报告时，在 ​record_xml_property上使用它会有所帮助。 但是，一些解析器对允许的元素和属性非常严格。 许多工具使用 ​xsd模式来验证传入的 ​xml​。 确保您使用的是解析器允许的属性名称。

以下是 ​Jenkins用于验证 ​XML报告的方案：

<xs:element name="testcase">
    <xs:complexType>
        <xs:sequence>
            <xs:element ref="skipped" minOccurs="0" maxOccurs="1"/>
            <xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="assertions" type="xs:string" use="optional"/>
        <xs:attribute name="time" type="xs:string" use="optional"/>
        <xs:attribute name="classname" type="xs:string" use="optional"/>
        <xs:attribute name="status" type="xs:string" use="optional"/>
    </xs:complexType>
</xs:element>

请注意，使用此功能将破坏最新 ​JUnitXML模式的模式验证。 当与某些 ​CI服务器一起使用时，这可能是一个问题。

record_testsuite_property

如果你想在测试套件级别添加一个属性节点，它可能包含与所有测试相关的属性，你可以使用​record_testsuite_property​会话范围的​fixture​:

​record_testsuite_property​会话范围的​fixture​可用于添加与所有测试相关的属性。

import pytest


@pytest.fixture(scope="session", autouse=True)
def log_global_env_facts(record_testsuite_property):
    record_testsuite_property("ARCH", "PPC")
    record_testsuite_property("STORAGE_TYPE", "CEPH")


class TestMe:
    def test_foo(self):
        assert True

这个​fixture​是一个可调用对象，它接收在生成的​xml​的测试套件级别添加的<属性>标签的名称和值:

<testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.006">
  <properties>
    <property name="ARCH" value="PPC"/>
    <property name="STORAGE_TYPE" value="CEPH"/>
  </properties>
  <testcase classname="test_me.TestMe" file="test_me.py" line="16" name="test_foo" time="0.000243663787842"/>
</testsuite>

名称必须是一个字符串，值将转换为字符串和正确的​xml​转义。

生成的​XML​兼容最新的​xunit​标准，而不是​record_property​和​record_xml_attribute​。

将测试报告发送到online pastebin service

为每个测试失败创建一个 URL：

pytest --pastebin=failed

这会将测试运行信息提交到远程粘贴服务，并为每个失败提供一个 URL。 如果您只想发送一个特定的失败，您可以像往常一样选择测试或添加例如 ​-x​

为整个测试会话日志创建URL:

pytest——pastebin =

目前只实现了粘贴到​https://bpaste.net/service​

如果由于任何原因创建URL失败，则会生成一个警告，而不是导致整个测试套件失败。