码厩技术博客

#5 Pytest setup 和 teardown 方法

• 全局
• setup_module(module) / teardown_module(module) 引入包的时候执行
• setup_function(function) / teardown_function(function)
• 模块级别
• setup() / teardown() 测试模块载入的时候执行
• 类级别
• setup_class(cls) / teardown_class(cls)
• setup_method(self, method) / teardown_method(self, method)
• setup(self) / teardown(self) nose 语法，会被上面两个方法覆盖

Supported nose Idioms

• setup() and teardown() at module/class/method level: any function or method called setup will be called during the setup phase for each test, same for teardown.
• SkipTest exceptions and markers
• setup/teardown decorators
• __test__ attribute on modules/classes/functions
• general usage of nose utilities

#4 pytest 学习

给 tmuxp 提了一个 Issue, 作者 @tony (Tony 老师？😜) 让我选择是自己写测试，还是他来写。
我表达我对 pytest 不熟之后，他给我一点建议：

• https://tmuxp.git-pull.com/developing.html should let you set it up.
• pytest documentation

所以我先学习学习，然后抽空把 test 自己写了。

#3 Python 单元测试框架

• doctest 标准库
• unitest 标准库
• pytest，基于 unitest
  pytest-dev/pytest
• nose/nose2，基于 unitest
  nose-devs/nose
  nose-devs/nose2

其他项目：

1. openatx/uiautomator2
   Android Uiautomator2 Python Wrapper
2. pytest-dev/pytest
   The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
3. pytest-dev/pytest-testinfra
   Testinfra test your infrastructures
4. jeffzh3ng/fuxi
   Penetration Testing Platform
5. locustio/locust
   Scalable user load testing tool written in Python
6. svanoort/pyresttest
   Python Rest Testing
7. SECFORCE/sparta
   Network Infrastructure Penetration Testing Tool
8. sivel/speedtest-cli
   Command line interface for testing internet bandwidth using speedtest.net
9. infobyte/faraday
   Collaborative Penetration Test and Vulnerability Management Platform
10. robotframework/robotframework
    Generic automation framework for acceptance testing and RPA
11. Manisso/fsociety
    fsociety Hacking Tools Pack – A Penetration Testing Framework
12. cocotb/cocotb
    cocotb, a coroutine based cosimulation library for writing VHDL and Verilog testbenches in Python
13. wolever/parameterized
    Parameterized testing with any Python test framework
14. cujanovic/SSRF-Testing
    SSRF (Server Side Request Forgery) testing resources
15. liwanlei/FXTest
    接口自动化测试平台——python+flask版，支持http协议,java 版本开发完毕https://github.com/liwanlei/plan
16. kmmbvnr/django-jenkins
    Plug and play continuous integration with django and jenkins
17. xiaocong/uiautomator
    Python wrapper of Android uiautomator test tool.
18. cobrateam/splinter
    splinter - python test framework for web applications
19. FSecureLABS/needle
    The iOS Security Testing Framework
20. robotframework/SeleniumLibrary
    Web testing library for Robot Framework
21. githublitao/api_automation_test
    接口自动化测试平台，已停止维护，看心情改改
22. ndb796/python-for-coding-test
    [한빛미디어] "이것이 취업을 위한 코딩 테스트다 with 파이썬" 전체 소스코드 저장소입니다.
23. FactoryBoy/factory_boy
    A test fixtures replacement for Python
24. Blazemeter/taurus
    Automation-friendly framework for Continuous Testing by
25. jazzband/django-nose
    Django test runner using nose
26. flipkart-incubator/Astra
    Automated Security Testing For REST API's
27. nose-devs/nose
    nose is nicer testing for python
28. SecuraBV/CVE-2020-1472
    Test tool for CVE-2020-1472
29. localstack/localstack
    💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!
30. spulec/freezegun
    Let your Python tests travel through time
31. Tencent/FAutoTest
    A UI automated testing framework for H5 and applets
32. robotframework/RIDE
    Test data editor for Robot Framework
33. microsoft/playwright-python
    Python version of the Playwright testing and automation library.
34. autotest/autotest
    Autotest - Fully automated tests on Linux
35. ansible-community/molecule
    Molecule aids in the development and testing of Ansible roles
36. pytest-dev/pytest-xdist
    pytest plugin for distributed testing and loop-on-failures testing modes.
37. powerfulseal/powerfulseal
    A powerful testing tool for Kubernetes clusters.
38. google/nogotofail
    An on-path blackbox network traffic security testing tool
39. wntrblm/nox
    Flexible test automation for Python
40. mjhea0/flaskr-tdd
    Flaskr: Intro to Flask, Test-Driven Development (TDD), and JavaScript
41. aws-ia/taskcat
    Test all the CloudFormation things! (with TaskCat)
42. facebookarchive/huxley
    A testing system for catching visual regressions in Web applications.
43. pytest-dev/pytest-bdd
    BDD library for the py.test runner
44. dagrz/aws_pwn
    A collection of AWS penetration testing junk
45. boxed/mutmut
    Mutation testing system
46. vishnubob/wait-for-it
    Pure bash script to test and wait on the availability of a TCP host and port
47. ethereum/casper
    Casper contract, and related software and tests
48. jhlau/doc2vec
    Python scripts for training/testing paragraph vectors
49. fsociety-team/fsociety
    A Modular Penetration Testing Framework
50. 0xInfection/TIDoS-Framework
    The Offensive Manual Web Application Penetration Testing Framework.
51. praetorian-inc/pentestly
    Python and Powershell internal penetration testing framework
52. hizhangp/yolo_tensorflow
    Tensorflow implementation of YOLO, including training and test phase.
53. zhuifengshen/xmind2testcase
    XMind2TestCase基于python实现，提供了一个高效测试用例设计的解决方案！
54. AnasAboureada/Penetration-Testing-Study-Notes
    Penetration Testing notes, resources and scripts
55. jseidl/GoldenEye
    GoldenEye Layer 7 (KeepAlive+NoCache) DoS Test Tool
56. testcontainers/testcontainers-python
    Testcontainers is a Python library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.
57. spulec/moto
    A library that allows you to easily mock out tests based on AWS infrastructure.
58. buildbot/buildbot
    Python-based continuous integration testing framework; your pull requests are more than welcome!
59. fossasia/open-event-server
    The Open Event Organizer Server to Manage Events https://test-api.eventyay.com
60. AirtestProject/Poco
    A cross-engine test automation framework based on UI inspection
61. Yijunmaverick/CartoonGAN-Test-Pytorch-Torch
    Pytorch and Torch testing code of CartoonGAN [Chen et al., CVPR18]
62. ionelmc/pytest-benchmark
    py.test fixture for benchmarking code
63. aws/aws-sam-cli
    CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
64. crossbario/autobahn-testsuite
    Autobahn WebSocket protocol testsuite
65. sixpack/sixpack
    Sixpack is a language-agnostic a/b-testing framework
66. SofianeHamlaoui/Lockdoor-Framework
    🔐 Lockdoor Framework : A Penetration Testing framework with Cyber Security Resources
67. mongomock/mongomock
    Small library for mocking pymongo collection objects for testing purposes
68. OWASP/owasp-mstg
    The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing and reverse engineering. It describes the technical processes for verifying the controls listed in the OWASP Mobile Application Security Verification Standard (MASVS).
69. python-needle/needle
    Automated tests for your CSS.
70. gyoisamurai/GyoiThon
    GyoiThon is a growing penetration test tool using Machine Learning.
71. SeldomQA/seldom
    WebUI/HTTP automation testing framework based on unittest
72. joestump/python-oauth2
    A fully tested, abstract interface to creating OAuth clients and servers.
73. maltize/sublime-text-2-ruby-tests
    Sublime Text 2 plugin for running ruby tests! (Unit, RSpec, Cucumber)
74. emirozer/fake2db
    create custom test databases that are populated with fake data
75. owid/covid-19-data
    Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data
76. Teemu/pytest-sugar
    a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)
77. GeorgeSeif/Semantic-Segmentation-Suite
    Semantic Segmentation Suite in TensorFlow. Implement, train, and test new Semantic Segmentation models easily!
78. jamesls/fakeredis
    Fake implementation of redis API (redis-py) for testing purposes
79. HypothesisWorks/hypothesis
    Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
80. se2p/pynguin
    The PYthoN General UnIt Test geNerator is a test-generation tool for Python
81. luogu-dev/cyaron
    CYaRon: Yet Another Random Olympic-iNformatics test data generator
82. knownsec/pocsuite3
    pocsuite3 is an open-sourced remote vulnerability testing framework developed by the Knownsec 404 Team.
83. seleniumbase/SeleniumBase
    A Python framework that inspires developers to become better test automation engineers. 🧠💡
84. terraform-compliance/cli
    a lightweight, security focused, BDD test framework against terraform.
85. coqui-ai/TTS
    🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
86. facebookresearch/DomainBed
    DomainBed is a suite to test domain generalization algorithms
87. qubvel/ttach
    Image Test Time Augmentation with PyTorch!
88. RhinoSecurityLabs/pacu
    The AWS exploitation framework, designed for testing the security of Amazon Web Services environments.
89. doyensec/inql
    InQL - A Burp Extension for GraphQL Security Testing
90. revsys/django-test-plus
    Useful additions to Django's default TestCase
91. online-judge-tools/oj
    Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
92. ticarpi/jwt_tool
    :snake: A toolkit for testing, tweaking and cracking JSON Web Tokens
93. yashaka/selene
    User-oriented Web UI browser tests in Python
94. davidtvs/pytorch-lr-finder
    A learning rate range test implementation in PyTorch
95. kevinburke/hamms
    Malformed servers to test your HTTP client
96. tarpas/pytest-testmon
    Selects tests affected by changed files. Continous test runner when used with pytest-watch.
97. pluwen/awesome-testflight-link
    Collection of Testflight public app link（iOS/iPad OS/macOS）。
98. facebookarchive/bootstrapped
    Generate bootstrapped confidence intervals for A/B testing in Python.
99. devpi/devpi
    Python PyPi staging server and packaging, testing, release tool
100. eth-brownie/brownie
     A Python-based development and testing framework for smart contracts targeting the Ethereum Virtual Machine.
101. scanapi/scanapi
     Automated Integration Testing and Live Documentation for your API
102. kevin1024/vcrpy
     Automatically mock your HTTP interactions to simplify and speed up testing
103. sodadata/soda-core
     Data profiling, testing, and monitoring for SQL accessible data.
104. CleanCut/green
     Green is a clean, colorful, fast python test runner.
105. trailofbits/deepstate
     A unit test-like interface for fuzzing and symbolic execution
106. ashishb/adb-enhanced
     🔪Swiss-army knife for Android testing and development 🔪 ⛺
107. D4Vinci/One-Lin3r
     Gives you one-liners that aids in penetration testing operations, privilege escalation and more
108. newsapps/beeswithmachineguns
     A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).
109. nilboy/tensorflow-yolo
     tensorflow implementation of 'YOLO : Real-Time Object Detection'(train and test)
110. NVlabs/FUNIT
     Translate images to unseen domains in the test time with few example images.
111. kyclark/tiny_python_projects
     Code for Tiny Python Projects (Manning, 2020, ISBN 1617297518). Learning Python through test-driven development of games and puzzles.
112. XifengGuo/CapsNet-Keras
     A Keras implementation of CapsNet in NIPS2017 paper "Dynamic Routing Between Capsules". Now test error ＝ 0.34%.
113. defparam/smuggler
     Smuggler - An HTTP Request Smuggling / Desync testing tool written in Python 3
114. joeyespo/pytest-watch
     Local continuous test runner with pytest and watchdog.
115. s0md3v/Blazy
     Blazy is a modern login bruteforcer which also tests for CSRF, Clickjacking, Cloudflare and WAF .
116. chakki-works/sumeval
     Well tested & Multi-language evaluation framework for text summarization.
117. kiwitcms/Kiwi
     The leading open source test management system with over 1 million downloads!
118. chenjj/espoofer
     An email spoofing testing tool that aims to bypass SPF/DKIM/DMARC and forge DKIM signatures.🍻
119. MisterBianco/BoopSuite
     A Suite of Tools written in Python for wireless auditing and security testing.
120. cszn/KAIR
     Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR
121. rueckstiess/mtools
     A collection of scripts to set up MongoDB test environments and parse and visualize MongoDB log files.
122. MozillaSecurity/funfuzz
     A collection of fuzzers in a harness for testing the SpiderMonkey JavaScript engine.
123. schemathesis/schemathesis
     A modern API testing tool for web applications built with Open API and GraphQL specifications.
124. worstcase/blockade
     Docker-based utility for testing network failures and partitions in distributed applications
125. kvalle/diy-lang
     A hands-on, test driven guide to implementing a simple programming language
126. nidhaloff/igel
     a delightful machine learning tool that allows you to train, test, and use models without writing code
127. minimaxir/gpt-3-experiments
     Test prompts for OpenAI's GPT-3 API and the resulting AI-generated texts.
128. tonglei100/sweetest
     小而美的自动化测试解决方案，支持 Web UI 测试，Http 接口测试，DB 操作测试，App 测试，小程序测试，Windows GUI 测试，文件操作
129. sdispater/cleo
     Cleo allows you to create beautiful and testable command-line interfaces.
130. joedicastro/vps-comparison
     A comparison between some VPS providers. It uses Ansible to perform a series of automated benchmark tests over the VPS servers that you specify. It allows the reproducibility of those tests by anyone that wanted to compare these results to their own. All the tests results are available in order to provide independence and transparency.
131. ryansmcgee/seirsplus
     Models of SEIRS epidemic dynamics with extensions, including network-structured populations, testing, contact tracing, and social distancing.
132. karec/cookiecutter-flask-restful
     Flask cookiecutter template for builing APIs with flask-restful, including JWT auth, cli, tests, swagger, docker and more
133. facebookresearch/FixRes
     This repository reproduces the results of the paper: "Fixing the train-test resolution discrepancy" https://arxiv.org/abs/1906.06423
134. honeynet/droidbot
     A lightweight test input generator for Android. Similar to Monkey, but with more intelligence and cool features!
135. darrenburns/ward
     Ward is a modern test framework for Python with a focus on productivity and readability.
136. httpie/http-prompt
     An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie
137. codingo/Reconnoitre
     A security tool for multithreaded information gathering and service enumeration whilst building directory structures to store results, along with writing out recommendations for further testing.
138. deepmind/pycolab
     A highly-customisable gridworld game engine with some batteries included. Make your own gridworld games to test reinforcement learning agents!
139. mjpost/sacrebleu
     Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
140. pschanely/CrossHair
     An analysis tool for Python that blurs the line between testing and type systems.
141. microsoft/restler-fuzzer
     RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
142. RussBaz/enforce
     Python 3.5+ runtime type checking for integration testing and data validation
143. pyvisa/pyvisa
     A Python package with bindings to the "Virtual Instrument Software Architecture" VISA library, in order to control measurement devices and test equipment via GPIB, RS232, or USB.
144. garethdmm/gryphon
     Powerful, proven, and extensible framework for building trading strategies at any frequency, with a focus on crypto currencies. Battle-tested with billions traded.
145. taverntesting/tavern
     A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax
146. dephell/dephell
     :package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.
147. hash3liZer/WiFiBroot
     A Wireless (WPA/WPA2) Pentest/Cracking tool. Captures & Crack 4-way handshake and PMKID key. Also, supports a deauthentication/jammer mode for stress testing
148. GoVanguard/legion
     Legion is an open source, easy-to-use, super-extensible and semi-automated network penetration testing tool that aids in discovery, reconnaissance and exploitation of information systems.
149. EnableSecurity/sipvicious
     SIPVicious OSS has been around since 2007 and is actively updated to help security teams, QA and developers test SIP-based VoIP systems and applications.
150. deepchecks/deepchecks
     Test Suites for Validating ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.
151. microsoft/dowhy
     DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
152. YudeWang/deeplabv3plus-pytorch
     Here is a pytorch implementation of deeplabv3+ supporting ResNet(79.155%) and Xception(79.945%). Multi-scale & flip test and COCO dataset interface has been finished.
153. james-atkinson/speedcomplainer
     A python app that will test your internet connection and then complain to your service provider (and log to a data store if you'd like)
154. flyteorg/flyte
     Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.
155. galkan/crowbar
     Crowbar is brute forcing tool that can be used during penetration tests. It is developed to support protocols that are not currently supported by thc-hydra and other popular brute forcing tools.
156. bslatkin/dpxdt
     Make continuous deployment safe by comparing before and after webpage screenshots for each release. Depicted shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.
157. xtiankisutsa/MARA_Framework
     MARA is a Mobile Application Reverse engineering and Analysis Framework. It is a toolkit that puts together commonly used mobile application reverse engineering and analysis tools to assist in testing mobile applications against the OWASP mobile security threats.
158. Quitten/Autorize
     Automatic authorization enforcement detection extension for burp suite written in Jython developed by Barak Tawily in order to ease application security people work and allow them perform an automatic authorization tests
159. HXSecurity/DongTai
     DongTai is an interactive application security testing(IAST) product that supports the detection of OWASP WEB TOP 10 vulnerabilities, multi-request related vulnerabilities (including logic vulnerabilities, unauthorized access vulnerabilities, etc.), third-party component vulnerabilities, etc.
160. bhdresh/CVE-2017-0199
     Exploit toolkit CVE-2017-0199 - v4.0 is a handy python script which provides pentesters and security researchers a quick and effective way to test Microsoft Office RCE. It could generate a malicious RTF/PPSX file and deliver metasploit / meterpreter / other payload to victim without any complex configuration.
161. screetsec/BruteSploit
     BruteSploit is a collection of method for automated Generate, Bruteforce and Manipulation wordlist with interactive shell. That can be used during a penetration test to enumerate and maybe can be used in CTF for manipulation,combine,transform and permutation some words or file text :p
162. cirosantilli/linux-kernel-module-cheat
     The perfect emulation setup to study and develop the Linux kernel v5.4.3, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 19.10 host.

#2 转载：性能测试应该怎么做？

偶然间看到了阿里中间件Dubbo的性能测试报告，我觉得这份性能测试报告让人觉得做这性能测试的人根本不懂性能测试，我觉得这份报告会把大众带沟里去，所以，想写下这篇文章，做一点科普。

#1 ab 报告解析

安装

CentOS 下的安装：

sudo yum install -y httpd-utils

Debian/Ubuntu 下的安装：

sudo apt install -y apache2-utils

使用

ab -n 请求数 -c 并发书 URL

常用参数：

• -n 请求数
• -c 并发数
• -t 持续测试时间
• -k 启用 HTTP KeepAlive
• -H 设置请求头
• -m 请求方式
• -T 设置 Content-Type
• -p 设置 POST 数据文件

如果 POST 请求：

echo 'key1=value1&key2=value2' > /tmp/postdata
-T 'application/x-www-form-urlencoded' -p postdata

echo '{"key1":"value1","key2":"value2"}' > /tmp/postdata
-T 'application/json' -p postdata

报告解读

执行 ab -n100 -c10 https://www.baidu.com/ 可以得到以下结果：

需要重要关注的数据做了标记，分别是：

• 失败数 Failed requests
  应该尽最大可能保证失败数为 0, 也就是成功数 100%
• 吞吐量(QPS/TPS) (Requests per second)
• 平均响应时间 (第一个 Time per request)
  应该等于：测试持续时间 / 请求数
  应该等于：并发数 / 吞吐量
• 最长响应时间 (最后面那行，100% longest request)
  80%, 90%, 95% 响应时间, 也可作为重要参考。

测试方案

Update @ 2021-12-29

url="https://www.baidu.com/"

rm -rf /tmp/httpbench-*.log
for c in {1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100,150,200,250,300,350,400,450,500,600,700,800,900,1000};
do echo "==== Concurrency Level: $c ===="; let n=$((c * 100)); ab -n $n -c $c $url > /tmp/httpbench-$c.log; sleep 10; done;

然后用 Python 脚本解析 ab 输出生成报告。

Python 代码就不贴了, 如果感兴趣可以打开看看: abReport.py

如果觉得我的方案不妥当，非常欢迎提出意见或建议。

Update @ 2021-12-30:

又对 abReport.py 做了调整 (abReport2.py)，现在可以输出一个 ECharts 图表了（尽整些花里胡哨的东西）。