码厩技术博客

#39 Python 项目结构图

常用的代码风格检查工具 pylint 中集成了 pyreverse, 这是一个用于生成 Python 的类图的工具。

#38 Python 多进程共同监听同一个端口

import socket
import signal
import sys
import os

def handle_connection(conn):
    conn.close()

def worker(sock):
    while True:
        try:
            conn, addr = sock.accept()
            handle_connection(conn)
        except OSError as e:
            if e.errno == socket.ECONNABORTED:
                # 忽略 ECONNABORTED 错误
                pass
            else:
                raise

def main():
    port = 8080
    backlog = 10  # 连接队列长度（超出会拒绝或忽略）
    num_workers = 4 # 子进程数

    # 创建监听器
    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listener.bind(('localhost', port))
    listener.listen(backlog)
    sock.setblocking(False)
    print(f"Listening on port {port}...")

    # # “来一个连接，起一个进程”的模式
    # def sig_handler(sig, frame):
    #     listener.close()
    #     sys.exit(0)
    # signal.signal(signal.SIGINT, sig_handler)
    # signal.signal(signal.SIGTERM, sig_handler)
    # while True:
    #     conn, addr = listener.accept()
    #     pid = os.fork()
    #     if pid == 0:
    #         listener.close()
    #         handle_connection(conn)
    #         sys.exit(0)
    #     else:
    #         conn.close()

    # 子进程放到进程组中
    os.setpgrp()

    # 多个 worker 子进程一同监听端口的模式
    processes = []
    for i in range(num_workers):
        p = Process(target=worker, args=(sock,))
        processes.append(p)
        p.start()
    # 通过 os.killpg 向进程组发送信号
    signal.signal(signal.SIGINT, lambda signum, frame: os.killpg(0, signal.SIGINT))
    signal.signal(signal.SIGTERM, lambda signum, frame: os.killpg(0, signal.SIGTERM))
    signal.pause()
    for p in processes:
        p.terminate()

if __name__ == '__main__':
    main()

惊群效应是指事件发生的时候，多个进程或线程竞争处理这个事件，导致系统负载出现一个尖峰。严重的情况下可能导致系统瘫痪。
虽然 accept 会阻塞住，只有一个抢到，但是惊群的问题应该还是存在。

进程组

• os.setpgrp 设置进程组
• os.killpg 向进程组发送信号，如果没有设置进程组，这个操作没有意义
• os.set_inheritable 继承文件描述符，然后可以独立使用和关闭

#37 Python 内存管理

内存分配

1. 堆内存 heap
2. 垃圾回收：
3. 引用计数
4. 延迟回收（支持手动触发）

引用计数

• 标记清除

标记所有当前正在使用的对象，然后清除所有未被标记的对象来回收内存。
这个算法的缺点是当内存中有大量未被标记的垃圾时，清除过程可能会变得非常缓慢。

• 分代回收

将对象分为三代：

• 新创建的对象放在第 0 代
• 第 1 代包含一些活跃的对象
• 第 2 代包含一些非常稳定的对象

不同代执行不同的标记策略。

相关工具

Python 中有一些内置的工具可以用于检查内存使用、内存分配情况以及各类对象数量的统计，下面列举一些常用的方法：

• memory_profiler 模块可以检查内存使用情况
• @profile 装饰器 + -m memory_profiler 参数，输出每个代码行的内存占用情况，以及整个程序的内存使用情况。
• objgraph 模块可以检查内存中各类对象的数量
• objgraph.show_refs() 对象引用图
• objgraph.show_most_common_types() 对象数量统计
• gc 模块可以手动管理内存
• gc.collect() 手动触发垃圾回收
• gc.get_count() 垃圾回收相关信息
• pympler
• meliae

手动释放内存：

#36 Subprocess Popen

如果命令不存在就会报：

FileNotFoundError: [Errno 2] No such file or directory: 'pythonjit'

示例

运行命令

import subprocess
subprocess.run('ls')

import subprocess
from subprocess import PIPE

result = subprocess.run(['touch', '/tmp/abc'])
print(result.stdout)

result = subprocess.run(['ls'], stdout=PIPE, text=True)
print(result.stdout)

cmd = ["python", "-c", "import time; time.sleep(3); print('hello')"]
result = subprocess.run(cmd, capture_output=True, text=True)
# capture_output=True => stdout = stderr = PIPE
print(result.stdout)

直接使用 Popen，灵活一些，也就复杂一些：

import sys
import subprocess
cmd = ["python", "-c", "import time; time.sleep(3); print('hello')"]
index = 0
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# print(p.stdout.read())  # 这一句会阻塞，直到程序执行完成
while True:
    index += 1
    try:
        output, errors = p.communicate(timeout=0.1)
        print(output)
        break
    except subprocess.TimeoutExpired:
        print(str(index), end=' ')
        sys.stdout.flush()

只是一个演示，如果需要异步执行的话，Python 3 支持基于 asyncio 的 subprocess。
参见：AsyncIO 异步执行命令

获取输出

import subprocess
output = subprocess.check_output(['ls'], text=True)
print(output)

获取返回代码

import subprocess
from subprocess import DEVNULL
try:
    subprocess.check_call(['ls', '/etc/abc'], stdout=DEVNULL, stderr=DEVNULL)
except subprocess.CalledProcessError as e:
    print(f'命令返回代码: {e.returncode}')

stdin/stdout/stderr

import subprocess
from subprocess import PIPE, STDOUT
process = subprocess.Popen(['grep', 'o'], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
process.stdin.write(b'Hello, World.\nWhats up?\nI am fine.\n3Q!\nAnd you?\n')
process.stdin.close()
return_code = process.wait()
print(process.stdout.read())
# b'Hello, World.\nAnd you?\n'

import subprocess
from subprocess import PIPE

result = subprocess.run(['ls', '-xyz'], stderr=PIPE)
print(repr(result.stderr.decode('utf-8')))

result = subprocess.run(['ls', '-xyz'], stderr=PIPE, text=True)
print(repr(result.stderr))
# 'ls: 不适用的选项 -- y\n请尝试执行 "ls --help" 来获取更多信息。\n'

管道

import subprocess
cmd = [
    ['tail', '-n10000', '/var/log/server.log'],
    ['grep', 'upload file'],
    ['awk', '{print $3}'],
]
stdin = None
for _cmd in cmd:
    proc = subprocess.Popen(_cmd, stdin=stdin, stdout=subprocess.PIPE, text=True)
    stdin = proc.stdout
output, error = proc.communicate()
print(output.strip())
print(error)

属性和方法

• PIPE -1，管道，表示捕获标准输入，或标准输出，或标准错误
  默认 stdin，stdout，stderr 是 None，也就是说没有配置，使用正常的标准输入，标准输出，标准错误。
• STDOUT -2，为标准错误准备，表示捕获标准错误，将其和标准输出混在一起，相当于 2>&1

• DEVNULL -3，空设备，表示丢弃标准输入，或标准输出，或标准错误（重定向到 /dev/null）

• Popen 进程的 Python 封装

• CompletedProcess 程序执行结果，包含命令，参数，返回吗，标准输出，标准错误

• SubprocessError

• CalledProcessError 如果得到错误状态码（非零），并且执行参数中 check=True

• TimeoutExpired 等待子进程超时

• list2cmdline 将命令列表组合一个完整命令

其实标准库中有另一个库可以做这个事情：

import shlex
cmd = 'grep -F "hello world" /tmp/abc.log'
args = shlex.split(cmd)
print(args)
# ['grep', '-F', 'hello world', '/tmp/abc.log']
print(shlex.join(args))
# grep -F 'hello world' /tmp/abc.log

• call 极简版本

def call(*popenargs, timeout=None, **kwargs):
    with Popen(*popenargs, **kwargs) as p:
        try:
            return p.wait(timeout=timeout)
        except:
            p.kill()
            raise

• check_call call 的封装，如果得到错误状态，就抛出 CalledProcessError

def check_call(*popenargs, **kwargs):
    retcode = call(*popenargs, **kwargs)
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd)
    return 0

• check_output 运行程序，获取标准输出

def check_output(*popenargs, timeout=None, **kwargs):
    for kw in ('stdout', 'check'):
        if kw in kwargs:
            raise ValueError(f'{kw} argument not allowed, it will be overridden.')
    if 'input' in kwargs and kwargs['input'] is None:
        if kwargs.get('universal_newlines') or kwargs.get('text') or kwargs.get('encoding') or kwargs.get('errors'):
            empty = ''
        else:
            empty = b''
        kwargs['input'] = empty
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, **kwargs).stdout

• getoutput 运行程序，获取输出

def getoutput(cmd, *, encoding=None, errors=None):
    return getstatusoutput(cmd, encoding=encoding, errors=errors)[1]

• getstatusoutput 运行程序，获取状态码和输出（stdout + stderr）

def getstatusoutput(cmd, *, encoding=None, errors=None):
    try:
        data = check_output(cmd, shell=True, text=True, stderr=STDOUT,
                            encoding=encoding, errors=errors)
        exitcode = 0
    except CalledProcessError as ex:
        data = ex.output
        exitcode = ex.returncode
    if data[-1:] == '\n':
        data = data[:-1]
    return exitcode, data

• run 运行程序

def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs):
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE
    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE
    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                exc.stdout, exc.stderr = process.communicate()
            else:
                process.wait()
            raise
        except:
            process.kill()
            raise
        retcode = process.poll()
        if check and retcode:
            raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
    return CompletedProcess(process.args, retcode, stdout, stderr)

封装关系：

Popen -> call -> check_call
Popen -> run -> check_output -> getstatusoutput -> getoutput

Popen

https://github.com/python/cpython/blob/master/Lib/subprocess.py

class Popen:
    def __init__(self, args, bufsize=-1, executable=None,
                 stdin=None, stdout=None, stderr=None,
                 preexec_fn=None, close_fds=True,
                 shell=False, cwd=None, env=None, universal_newlines=None,
                 startupinfo=None, creationflags=0,
                 restore_signals=True, start_new_session=False,
                 pass_fds=(), *, user=None, group=None, extra_groups=None,
                 encoding=None, errors=None, text=None, umask=-1, pipesize=-1,
                 process_group=None): pass
        # 创建子进程，执行系统命令
        # args          命令和参数
        # bufsize
        # executable    替代
        # stdin
        # stdout
        # stderr
        # preexec_fn
        # close_fds
        # shell
        # cwd
        # env
        # universal_newlines
        # startupinfo
        # creationflags
        # restore_signals
        # start_new_session
        # pass_fds
        # user
        # group
        # extra_groups
        # encoding
        # errors
        # text
        #   如果指定不一样的 text 和 universal_newlines 值，则抛出 SubprocessError
        #   self.text_mode = encoding or errors or text or universal_newlines
        # umask
        # pipesize
        # process_group

    def __repr__(self): pass

    @property
    def universal_newlines(self):
        return self.text_mode

    @universal_newlines.setter
    def universal_newlines(self, universal_newlines):
        self.text_mode = bool(universal_newlines)

    def __enter__(self):
        return self
    def __exit__(self, exc_type, value, traceback): pass
    def __del__(self, _maxsize=sys.maxsize, _warn=warnings.warn): pass

    def communicate(self, input=None, timeout=None) -> (stdout, stderr): pass
    def poll(self) -> None|returncode: pass  # 检查进程是否结束，返回状态码或 None（没有结束）
    def wait(self, timeout=None) -> returncode: pass  # 等待进程结束，返回状态码
    def send_signal(self, sig):
        self.poll()
        if self.returncode is not None:  # 程序已经执行结束
            return
        try:  # 防止并发杀进程
            os.kill(self.pid, sig)
        except ProcessLookupError:
            pass
    def terminate(self):  # 结束进程 -15
        self.send_signal(signal.SIGTERM)
    def kill(self):  # 强杀进程 -9
        self.send_signal(signal.SIGKILL)

#35 Python 执行系统命令的方法

这里是指在本地执行命令的方法，通过 SSH 的方式这里不做讨论。

#34 Python 单元测试框架

• doctest 标准库
• unitest 标准库
• pytest，基于 unitest
  pytest-dev/pytest
• nose/nose2，基于 unitest
  nose-devs/nose
  nose-devs/nose2

其他项目：

1. openatx/uiautomator2
   Android Uiautomator2 Python Wrapper
2. pytest-dev/pytest
   The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
3. pytest-dev/pytest-testinfra
   Testinfra test your infrastructures
4. jeffzh3ng/fuxi
   Penetration Testing Platform
5. locustio/locust
   Scalable user load testing tool written in Python
6. svanoort/pyresttest
   Python Rest Testing
7. SECFORCE/sparta
   Network Infrastructure Penetration Testing Tool
8. sivel/speedtest-cli
   Command line interface for testing internet bandwidth using speedtest.net
9. infobyte/faraday
   Collaborative Penetration Test and Vulnerability Management Platform
10. robotframework/robotframework
    Generic automation framework for acceptance testing and RPA
11. Manisso/fsociety
    fsociety Hacking Tools Pack – A Penetration Testing Framework
12. cocotb/cocotb
    cocotb, a coroutine based cosimulation library for writing VHDL and Verilog testbenches in Python
13. wolever/parameterized
    Parameterized testing with any Python test framework
14. cujanovic/SSRF-Testing
    SSRF (Server Side Request Forgery) testing resources
15. liwanlei/FXTest
    接口自动化测试平台——python+flask版，支持http协议,java 版本开发完毕https://github.com/liwanlei/plan
16. kmmbvnr/django-jenkins
    Plug and play continuous integration with django and jenkins
17. xiaocong/uiautomator
    Python wrapper of Android uiautomator test tool.
18. cobrateam/splinter
    splinter - python test framework for web applications
19. FSecureLABS/needle
    The iOS Security Testing Framework
20. robotframework/SeleniumLibrary
    Web testing library for Robot Framework
21. githublitao/api_automation_test
    接口自动化测试平台，已停止维护，看心情改改
22. ndb796/python-for-coding-test
    [한빛미디어] "이것이 취업을 위한 코딩 테스트다 with 파이썬" 전체 소스코드 저장소입니다.
23. FactoryBoy/factory_boy
    A test fixtures replacement for Python
24. Blazemeter/taurus
    Automation-friendly framework for Continuous Testing by
25. jazzband/django-nose
    Django test runner using nose
26. flipkart-incubator/Astra
    Automated Security Testing For REST API's
27. nose-devs/nose
    nose is nicer testing for python
28. SecuraBV/CVE-2020-1472
    Test tool for CVE-2020-1472
29. localstack/localstack
    💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!
30. spulec/freezegun
    Let your Python tests travel through time
31. Tencent/FAutoTest
    A UI automated testing framework for H5 and applets
32. robotframework/RIDE
    Test data editor for Robot Framework
33. microsoft/playwright-python
    Python version of the Playwright testing and automation library.
34. autotest/autotest
    Autotest - Fully automated tests on Linux
35. ansible-community/molecule
    Molecule aids in the development and testing of Ansible roles
36. pytest-dev/pytest-xdist
    pytest plugin for distributed testing and loop-on-failures testing modes.
37. powerfulseal/powerfulseal
    A powerful testing tool for Kubernetes clusters.
38. google/nogotofail
    An on-path blackbox network traffic security testing tool
39. wntrblm/nox
    Flexible test automation for Python
40. mjhea0/flaskr-tdd
    Flaskr: Intro to Flask, Test-Driven Development (TDD), and JavaScript
41. aws-ia/taskcat
    Test all the CloudFormation things! (with TaskCat)
42. facebookarchive/huxley
    A testing system for catching visual regressions in Web applications.
43. pytest-dev/pytest-bdd
    BDD library for the py.test runner
44. dagrz/aws_pwn
    A collection of AWS penetration testing junk
45. boxed/mutmut
    Mutation testing system
46. vishnubob/wait-for-it
    Pure bash script to test and wait on the availability of a TCP host and port
47. ethereum/casper
    Casper contract, and related software and tests
48. jhlau/doc2vec
    Python scripts for training/testing paragraph vectors
49. fsociety-team/fsociety
    A Modular Penetration Testing Framework
50. 0xInfection/TIDoS-Framework
    The Offensive Manual Web Application Penetration Testing Framework.
51. praetorian-inc/pentestly
    Python and Powershell internal penetration testing framework
52. hizhangp/yolo_tensorflow
    Tensorflow implementation of YOLO, including training and test phase.
53. zhuifengshen/xmind2testcase
    XMind2TestCase基于python实现，提供了一个高效测试用例设计的解决方案！
54. AnasAboureada/Penetration-Testing-Study-Notes
    Penetration Testing notes, resources and scripts
55. jseidl/GoldenEye
    GoldenEye Layer 7 (KeepAlive+NoCache) DoS Test Tool
56. testcontainers/testcontainers-python
    Testcontainers is a Python library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.
57. spulec/moto
    A library that allows you to easily mock out tests based on AWS infrastructure.
58. buildbot/buildbot
    Python-based continuous integration testing framework; your pull requests are more than welcome!
59. fossasia/open-event-server
    The Open Event Organizer Server to Manage Events https://test-api.eventyay.com
60. AirtestProject/Poco
    A cross-engine test automation framework based on UI inspection
61. Yijunmaverick/CartoonGAN-Test-Pytorch-Torch
    Pytorch and Torch testing code of CartoonGAN [Chen et al., CVPR18]
62. ionelmc/pytest-benchmark
    py.test fixture for benchmarking code
63. aws/aws-sam-cli
    CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
64. crossbario/autobahn-testsuite
    Autobahn WebSocket protocol testsuite
65. sixpack/sixpack
    Sixpack is a language-agnostic a/b-testing framework
66. SofianeHamlaoui/Lockdoor-Framework
    🔐 Lockdoor Framework : A Penetration Testing framework with Cyber Security Resources
67. mongomock/mongomock
    Small library for mocking pymongo collection objects for testing purposes
68. OWASP/owasp-mstg
    The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing and reverse engineering. It describes the technical processes for verifying the controls listed in the OWASP Mobile Application Security Verification Standard (MASVS).
69. python-needle/needle
    Automated tests for your CSS.
70. gyoisamurai/GyoiThon
    GyoiThon is a growing penetration test tool using Machine Learning.
71. SeldomQA/seldom
    WebUI/HTTP automation testing framework based on unittest
72. joestump/python-oauth2
    A fully tested, abstract interface to creating OAuth clients and servers.
73. maltize/sublime-text-2-ruby-tests
    Sublime Text 2 plugin for running ruby tests! (Unit, RSpec, Cucumber)
74. emirozer/fake2db
    create custom test databases that are populated with fake data
75. owid/covid-19-data
    Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data
76. Teemu/pytest-sugar
    a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)
77. GeorgeSeif/Semantic-Segmentation-Suite
    Semantic Segmentation Suite in TensorFlow. Implement, train, and test new Semantic Segmentation models easily!
78. jamesls/fakeredis
    Fake implementation of redis API (redis-py) for testing purposes
79. HypothesisWorks/hypothesis
    Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
80. se2p/pynguin
    The PYthoN General UnIt Test geNerator is a test-generation tool for Python
81. luogu-dev/cyaron
    CYaRon: Yet Another Random Olympic-iNformatics test data generator
82. knownsec/pocsuite3
    pocsuite3 is an open-sourced remote vulnerability testing framework developed by the Knownsec 404 Team.
83. seleniumbase/SeleniumBase
    A Python framework that inspires developers to become better test automation engineers. 🧠💡
84. terraform-compliance/cli
    a lightweight, security focused, BDD test framework against terraform.
85. coqui-ai/TTS
    🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
86. facebookresearch/DomainBed
    DomainBed is a suite to test domain generalization algorithms
87. qubvel/ttach
    Image Test Time Augmentation with PyTorch!
88. RhinoSecurityLabs/pacu
    The AWS exploitation framework, designed for testing the security of Amazon Web Services environments.
89. doyensec/inql
    InQL - A Burp Extension for GraphQL Security Testing
90. revsys/django-test-plus
    Useful additions to Django's default TestCase
91. online-judge-tools/oj
    Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
92. ticarpi/jwt_tool
    :snake: A toolkit for testing, tweaking and cracking JSON Web Tokens
93. yashaka/selene
    User-oriented Web UI browser tests in Python
94. davidtvs/pytorch-lr-finder
    A learning rate range test implementation in PyTorch
95. kevinburke/hamms
    Malformed servers to test your HTTP client
96. tarpas/pytest-testmon
    Selects tests affected by changed files. Continous test runner when used with pytest-watch.
97. pluwen/awesome-testflight-link
    Collection of Testflight public app link（iOS/iPad OS/macOS）。
98. facebookarchive/bootstrapped
    Generate bootstrapped confidence intervals for A/B testing in Python.
99. devpi/devpi
    Python PyPi staging server and packaging, testing, release tool
100. eth-brownie/brownie
     A Python-based development and testing framework for smart contracts targeting the Ethereum Virtual Machine.
101. scanapi/scanapi
     Automated Integration Testing and Live Documentation for your API
102. kevin1024/vcrpy
     Automatically mock your HTTP interactions to simplify and speed up testing
103. sodadata/soda-core
     Data profiling, testing, and monitoring for SQL accessible data.
104. CleanCut/green
     Green is a clean, colorful, fast python test runner.
105. trailofbits/deepstate
     A unit test-like interface for fuzzing and symbolic execution
106. ashishb/adb-enhanced
     🔪Swiss-army knife for Android testing and development 🔪 ⛺
107. D4Vinci/One-Lin3r
     Gives you one-liners that aids in penetration testing operations, privilege escalation and more
108. newsapps/beeswithmachineguns
     A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).
109. nilboy/tensorflow-yolo
     tensorflow implementation of 'YOLO : Real-Time Object Detection'(train and test)
110. NVlabs/FUNIT
     Translate images to unseen domains in the test time with few example images.
111. kyclark/tiny_python_projects
     Code for Tiny Python Projects (Manning, 2020, ISBN 1617297518). Learning Python through test-driven development of games and puzzles.
112. XifengGuo/CapsNet-Keras
     A Keras implementation of CapsNet in NIPS2017 paper "Dynamic Routing Between Capsules". Now test error ＝ 0.34%.
113. defparam/smuggler
     Smuggler - An HTTP Request Smuggling / Desync testing tool written in Python 3
114. joeyespo/pytest-watch
     Local continuous test runner with pytest and watchdog.
115. s0md3v/Blazy
     Blazy is a modern login bruteforcer which also tests for CSRF, Clickjacking, Cloudflare and WAF .
116. chakki-works/sumeval
     Well tested & Multi-language evaluation framework for text summarization.
117. kiwitcms/Kiwi
     The leading open source test management system with over 1 million downloads!
118. chenjj/espoofer
     An email spoofing testing tool that aims to bypass SPF/DKIM/DMARC and forge DKIM signatures.🍻
119. MisterBianco/BoopSuite
     A Suite of Tools written in Python for wireless auditing and security testing.
120. cszn/KAIR
     Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR
121. rueckstiess/mtools
     A collection of scripts to set up MongoDB test environments and parse and visualize MongoDB log files.
122. MozillaSecurity/funfuzz
     A collection of fuzzers in a harness for testing the SpiderMonkey JavaScript engine.
123. schemathesis/schemathesis
     A modern API testing tool for web applications built with Open API and GraphQL specifications.
124. worstcase/blockade
     Docker-based utility for testing network failures and partitions in distributed applications
125. kvalle/diy-lang
     A hands-on, test driven guide to implementing a simple programming language
126. nidhaloff/igel
     a delightful machine learning tool that allows you to train, test, and use models without writing code
127. minimaxir/gpt-3-experiments
     Test prompts for OpenAI's GPT-3 API and the resulting AI-generated texts.
128. tonglei100/sweetest
     小而美的自动化测试解决方案，支持 Web UI 测试，Http 接口测试，DB 操作测试，App 测试，小程序测试，Windows GUI 测试，文件操作
129. sdispater/cleo
     Cleo allows you to create beautiful and testable command-line interfaces.
130. joedicastro/vps-comparison
     A comparison between some VPS providers. It uses Ansible to perform a series of automated benchmark tests over the VPS servers that you specify. It allows the reproducibility of those tests by anyone that wanted to compare these results to their own. All the tests results are available in order to provide independence and transparency.
131. ryansmcgee/seirsplus
     Models of SEIRS epidemic dynamics with extensions, including network-structured populations, testing, contact tracing, and social distancing.
132. karec/cookiecutter-flask-restful
     Flask cookiecutter template for builing APIs with flask-restful, including JWT auth, cli, tests, swagger, docker and more
133. facebookresearch/FixRes
     This repository reproduces the results of the paper: "Fixing the train-test resolution discrepancy" https://arxiv.org/abs/1906.06423
134. honeynet/droidbot
     A lightweight test input generator for Android. Similar to Monkey, but with more intelligence and cool features!
135. darrenburns/ward
     Ward is a modern test framework for Python with a focus on productivity and readability.
136. httpie/http-prompt
     An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie
137. codingo/Reconnoitre
     A security tool for multithreaded information gathering and service enumeration whilst building directory structures to store results, along with writing out recommendations for further testing.
138. deepmind/pycolab
     A highly-customisable gridworld game engine with some batteries included. Make your own gridworld games to test reinforcement learning agents!
139. mjpost/sacrebleu
     Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
140. pschanely/CrossHair
     An analysis tool for Python that blurs the line between testing and type systems.
141. microsoft/restler-fuzzer
     RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
142. RussBaz/enforce
     Python 3.5+ runtime type checking for integration testing and data validation
143. pyvisa/pyvisa
     A Python package with bindings to the "Virtual Instrument Software Architecture" VISA library, in order to control measurement devices and test equipment via GPIB, RS232, or USB.
144. garethdmm/gryphon
     Powerful, proven, and extensible framework for building trading strategies at any frequency, with a focus on crypto currencies. Battle-tested with billions traded.
145. taverntesting/tavern
     A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax
146. dephell/dephell
     :package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.
147. hash3liZer/WiFiBroot
     A Wireless (WPA/WPA2) Pentest/Cracking tool. Captures & Crack 4-way handshake and PMKID key. Also, supports a deauthentication/jammer mode for stress testing
148. GoVanguard/legion
     Legion is an open source, easy-to-use, super-extensible and semi-automated network penetration testing tool that aids in discovery, reconnaissance and exploitation of information systems.
149. EnableSecurity/sipvicious
     SIPVicious OSS has been around since 2007 and is actively updated to help security teams, QA and developers test SIP-based VoIP systems and applications.
150. deepchecks/deepchecks
     Test Suites for Validating ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.
151. microsoft/dowhy
     DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
152. YudeWang/deeplabv3plus-pytorch
     Here is a pytorch implementation of deeplabv3+ supporting ResNet(79.155%) and Xception(79.945%). Multi-scale & flip test and COCO dataset interface has been finished.
153. james-atkinson/speedcomplainer
     A python app that will test your internet connection and then complain to your service provider (and log to a data store if you'd like)
154. flyteorg/flyte
     Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.
155. galkan/crowbar
     Crowbar is brute forcing tool that can be used during penetration tests. It is developed to support protocols that are not currently supported by thc-hydra and other popular brute forcing tools.
156. bslatkin/dpxdt
     Make continuous deployment safe by comparing before and after webpage screenshots for each release. Depicted shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.
157. xtiankisutsa/MARA_Framework
     MARA is a Mobile Application Reverse engineering and Analysis Framework. It is a toolkit that puts together commonly used mobile application reverse engineering and analysis tools to assist in testing mobile applications against the OWASP mobile security threats.
158. Quitten/Autorize
     Automatic authorization enforcement detection extension for burp suite written in Jython developed by Barak Tawily in order to ease application security people work and allow them perform an automatic authorization tests
159. HXSecurity/DongTai
     DongTai is an interactive application security testing(IAST) product that supports the detection of OWASP WEB TOP 10 vulnerabilities, multi-request related vulnerabilities (including logic vulnerabilities, unauthorized access vulnerabilities, etc.), third-party component vulnerabilities, etc.
160. bhdresh/CVE-2017-0199
     Exploit toolkit CVE-2017-0199 - v4.0 is a handy python script which provides pentesters and security researchers a quick and effective way to test Microsoft Office RCE. It could generate a malicious RTF/PPSX file and deliver metasploit / meterpreter / other payload to victim without any complex configuration.
161. screetsec/BruteSploit
     BruteSploit is a collection of method for automated Generate, Bruteforce and Manipulation wordlist with interactive shell. That can be used during a penetration test to enumerate and maybe can be used in CTF for manipulation,combine,transform and permutation some words or file text :p
162. cirosantilli/linux-kernel-module-cheat
     The perfect emulation setup to study and develop the Linux kernel v5.4.3, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 19.10 host.

#33 Python 加密解密

• PyCrypto Crypto 曾经非常著名的项目，后来没有维护了，推荐后面两个。

• https://github.com/pycrypto/pycrypto

• https://www.pycrypto.org/

• PyCryptodome Crypto 从 PyCrypto 分叉出来的项目，继续维护中。

• https://github.com/Legrandin/pycryptodome

• https://www.pycryptodome.org/
• pycryptodomex Cryptodome
  • 和 PyCryptodome 是一套代码，只是包名不同。
  • PyCryptodome 继续使用 Crypto 做包名，用来替代 PyCrypto。
  • pycryptodomex 使用 Cryptodome 做包名，方便和老的 PyCrypto 共存。

• Cryptography cryptography 算 Python 官方项目（PyCA Python Cryptographic Authority）。

• https://github.com/pyca/cryptography

• https://cryptography.io/

#32 virtualenv 的使用

安装

pip install virtualenvwrapper

把以下内容加入 ~/.bashrc，~/.zshrc 等 Shell 初始化代码中：

export WORKON_HOME="~/.virtualenvs"
export VIRTUALENVWRAPPER_PYTHON="/usr/bin/python3"
source virtualenvwrapper.sh

1. source virtualenvwrapper_lazy.sh 也可以，只是不会自动补全了。
2. 这两个脚本默认会放到 /usr/local/bin/ 目录，或者 ~/.local/bin/ 目录，所以可以直接 source 加载。

简单使用

# 创建并进入虚拟环境
mkvirtualenv myproject

# 进入指定虚拟环境
workon myproject

# 退出虚拟环境
deactivate

# 列出虚拟环境
workon

# 删除虚拟环境
rmvirtualenv myproject

其他命令

virtualenvwrapper

virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv
tool.  The extensions include wrappers for creating and deleting
virtual environments and otherwise managing your development workflow,
making it easier to work on more than one project at a time without
introducing conflicts in their dependencies.

For more information please refer to the documentation:

    http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

Commands available:

  add2virtualenv: add directory to the import path

  allvirtualenv: run a command in all virtualenvs

  cdproject: change directory to the active project

  cdsitepackages: change to the site-packages directory

  cdvirtualenv: change to the $VIRTUAL_ENV directory

  cpvirtualenv: duplicate the named virtualenv to make a new one

  lssitepackages: list contents of the site-packages directory

  lsvirtualenv: list virtualenvs

  mkproject: create a new project directory and its associated virtualenv

  mktmpenv: create a temporary virtualenv

  mkvirtualenv: Create a new virtualenv in $WORKON_HOME

  rmvirtualenv: Remove a virtualenv

  setvirtualenvproject: associate a project directory with a virtualenv

  showvirtualenv: show details of a single virtualenv

  toggleglobalsitepackages: turn access to global site-packages on/off

  virtualenvwrapper: show this help message

  wipeenv: remove all packages installed in the current virtualenv

  workon: list or change working virtualenvs

#31 logging 时间格式

import logging
LOG_LEVEL = logging.DEBUG
LOG_FORMAT = '%(asctime)s %(levelname)s %(message)s'
logging.basicConfig(level=LOG_LEVEL, format=LOG_FORMAT)
logging.info('hello world')

默认时间格式是 yyyy-mm-dd hh:mm:ss,xxx，如果我们要改这个格式可以用 datefmt 参数，遵循 time.strftime 的格式化参数格式。

问题有一个，毫秒从哪里来？

方法一：使用 msecs 占位符

最简单的办法：在格式字符串中使用 msecs 占位符，比如：

import logging
LOG_LEVEL = logging.DEBUG
LOG_FORMAT = '%(asctime)s.%(msecs)03d %(levelname)s %(message)s'
LOG_DATEFMT = '%H:%M:%S'
logging.basicConfig(level=LOG_LEVEL, format=LOG_FORMAT, datefmt=LOG_DATEFMT)
logging.info('hello world')

方法二：改用 datetime 对象

import logging
from datetime import datetime

class MyFormatter(logging.Formatter):
    converter = datetime.fromtimestamp

    def formatTime(self, record, datefmt=None):
        ct = self.converter(record.created)
        if datefmt:
            # s = time.strftime(datefmt, ct)
            s = ct.strftime(datefmt)
        else:
            # t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
            t = ct.strftime("%Y-%m-%d %H:%M:%S")
            s = "%s,%03d" % (t, record.msecs)
        return s

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler()
logger.addHandler(console)
formatter = MyFormatter(fmt='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S.%f')
console.setFormatter(formatter)
logging.info('hello world')

#30 Supervisor 进程守护

子命令

• help
• help <action>
• add <name> […]
• remove <name> […]
• update
• update all
• update <gname> […]
• clear <name>
• clear <name> <name>
• clear all
• fg <process>
• pid
• pid <name>
• pid all
• reload
• reread
• restart <name> 不会重新加载配置文件
• 如果配置有更新，应该 reread 一下
• 如果有 section 增删，还应该 update 一下
• restart <gname>:*
• restart <name> <name>
• restart all
• signal
• start <name>
• start <gname>:*
• start <name> <name>
• start all
• status
• status <name>
• status <name> <name>
• stop <name>
• stop <gname>:*
• stop <name> <name>
• stop all
• tail [-f] <name> [stdout|stderr] (default stdout)