码厩开发者笔记

#32 virtualenv 的使用

安装

pip install virtualenvwrapper

把以下内容加入 ~/.bashrc，~/.zshrc 等 Shell 初始化代码中：

export WORKON_HOME="~/.virtualenvs"
export VIRTUALENVWRAPPER_PYTHON="/usr/bin/python3"
source virtualenvwrapper.sh

1. source virtualenvwrapper_lazy.sh 也可以，只是不会自动补全了。
2. 这两个脚本默认会放到 /usr/local/bin/ 目录，或者 ~/.local/bin/ 目录，所以可以直接 source 加载。

简单使用

# 创建并进入虚拟环境
mkvirtualenv myproject

# 进入指定虚拟环境
workon myproject

# 退出虚拟环境
deactivate

# 列出虚拟环境
workon

# 删除虚拟环境
rmvirtualenv myproject

其他命令

virtualenvwrapper

virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv
tool.  The extensions include wrappers for creating and deleting
virtual environments and otherwise managing your development workflow,
making it easier to work on more than one project at a time without
introducing conflicts in their dependencies.

For more information please refer to the documentation:

    http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

Commands available:

  add2virtualenv: add directory to the import path

  allvirtualenv: run a command in all virtualenvs

  cdproject: change directory to the active project

  cdsitepackages: change to the site-packages directory

  cdvirtualenv: change to the $VIRTUAL_ENV directory

  cpvirtualenv: duplicate the named virtualenv to make a new one

  lssitepackages: list contents of the site-packages directory

  lsvirtualenv: list virtualenvs

  mkproject: create a new project directory and its associated virtualenv

  mktmpenv: create a temporary virtualenv

  mkvirtualenv: Create a new virtualenv in $WORKON_HOME

  rmvirtualenv: Remove a virtualenv

  setvirtualenvproject: associate a project directory with a virtualenv

  showvirtualenv: show details of a single virtualenv

  toggleglobalsitepackages: turn access to global site-packages on/off

  virtualenvwrapper: show this help message

  wipeenv: remove all packages installed in the current virtualenv

  workon: list or change working virtualenvs

#31 logging 时间格式

import logging
LOG_LEVEL = logging.DEBUG
LOG_FORMAT = '%(asctime)s %(levelname)s %(message)s'
logging.basicConfig(level=LOG_LEVEL, format=LOG_FORMAT)
logging.info('hello world')

默认时间格式是 yyyy-mm-dd hh:mm:ss,xxx，如果我们要改这个格式可以用 datefmt 参数，遵循 time.strftime 的格式化参数格式。

问题有一个，毫秒从哪里来？

方法一：使用 msecs 占位符

最简单的办法：在格式字符串中使用 msecs 占位符，比如：

import logging
LOG_LEVEL = logging.DEBUG
LOG_FORMAT = '%(asctime)s.%(msecs)03d %(levelname)s %(message)s'
LOG_DATEFMT = '%H:%M:%S'
logging.basicConfig(level=LOG_LEVEL, format=LOG_FORMAT, datefmt=LOG_DATEFMT)
logging.info('hello world')

方法二：改用 datetime 对象

import logging
from datetime import datetime

class MyFormatter(logging.Formatter):
    converter = datetime.fromtimestamp

    def formatTime(self, record, datefmt=None):
        ct = self.converter(record.created)
        if datefmt:
            # s = time.strftime(datefmt, ct)
            s = ct.strftime(datefmt)
        else:
            # t = time.strftime("%Y-%m-%d %H:%M:%S", ct)
            t = ct.strftime("%Y-%m-%d %H:%M:%S")
            s = "%s,%03d" % (t, record.msecs)
        return s

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler()
logger.addHandler(console)
formatter = MyFormatter(fmt='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S.%f')
console.setFormatter(formatter)
logging.info('hello world')

#30 Supervisor 进程守护

子命令

• help
• help <action>
• add <name> […]
• remove <name> […]
• update
• update all
• update <gname> […]
• clear <name>
• clear <name> <name>
• clear all
• fg <process>
• pid
• pid <name>
• pid all
• reload
• reread
• restart <name> 不会重新加载配置文件
• 如果配置有更新，应该 reread 一下
• 如果有 section 增删，还应该 update 一下
• restart <gname>:*
• restart <name> <name>
• restart all
• signal
• start <name>
• start <gname>:*
• start <name> <name>
• start all
• status
• status <name>
• status <name> <name>
• stop <name>
• stop <gname>:*
• stop <name> <name>
• stop all
• tail [-f] <name> [stdout|stderr] (default stdout)

#29 Werkzeug WSGI 框架

• PyPI：https://pypi.org/project/Werkzeug/
• 文档：https://werkzeug.palletsprojects.com/en/latest/
• 代码：https://github.com/pallets/werkzeug/

示例

from werkzeug.wrappers import Request, Response
from werkzeug.routing import Map, Rule

class Application:
    def __init__(self):
        self.url_map = Map([
            Rule('/', endpoint='hello')
        ])

    def hello(self, request):
        text = 'Hello, World!'
        response = Response(text, content_type='text/plain')
        return response

    @Request.application
    def __call__(self, request):
        adapter = self.url_map.bind_to_environ(request.environ)
        endpoint, values = adapter.match()

        handler = getattr(self, endpoint)
        return handler(request, **values)

if __name__ == '__main__':
    from werkzeug.serving import run_simple
    app = Application()
    run_simple('localhost', 5000, app)

Werkzeug 与协程（async/await）

Werkzeug 1.x 开始已经提供了对协程的支持。
PS：WSGI 规范是基于同步的设计，没有较好的适配异步编程，所以后面一些更年轻的框架选择了 ASGI（Async Server Gateway Interface）。
PS：2014/03/01, WSGI
PS：2021/11/06, 体验 ASGI

import asyncio
from werkzeug.wrappers import Request, Response
from werkzeug.routing import Map, Rule

class Application:
    def __init__(self):
        self.url_map = Map([
            Rule('/', endpoint='hello')
        ])

    async def hello(self, request):
        text = 'Hello, World!'
        response = Response(text, content_type='text/plain')
        return response

    @Request.application
    async def __call__(self, request):
        adapter = self.url_map.bind_to_environ(request.environ)
        endpoint, values = adapter.match()

        handler = getattr(self, endpoint)
        response = await handler(request, **values)
        return response

if __name__ == '__main__':
    from werkzeug.serving import run_simple
    app = Application()
    run_simple('localhost', 5000, app)

附：版本历史

#28 The Pallets Projects

这个组织下的每个项目单拿出来都是响当当的存在：

• Click — Beautiful, composable command line interface creation kit.
• Flask — a flexible and popular web development framework
• ItsDangerous — Safely pass trusted data to untrusted environments and back.
• Jinja — a full featured template engine for Python
• MarkupSafe — Safely add untrusted strings to HTML/XML markup.
• Quart — an async reimplementation of flask
• Werkzeug — The comprehensive WSGI web application library.

• Click 命令行工具
• Flask 轻量级 Web 框架
• ItsDangerous
• Jinja 知名模板库
• MackupSafe
• Quart
• Werkzeug Web 工具箱（WSGI），Flask 的底层

#27 Python 输出 Unicode 字符到标准输出时遇到 UnicodeEncodeError

源代码：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
print u'中国'

报错：

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    print u'中国'
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

#26 Python bytes 类型

1. bytes 按索引取值得到的是整数。

   b'abc'[0]
   97
   

2. str 转 bytes 只需要编码一下就行了。反过来就是解码一下。

   s = '你好'
   b = bytes(s, 'utf-8')
   # b'\xe4\xbd\xa0\xe5\xa5\xbd'
   assert b.decode('utf-8') == s
   

   bytes('hello')
   TypeError: string argument without an encoding
   
   bytes('hello', 'ascii')
   b'hello'
   
   'hello'.encode()
   b'hello'
   
   bytes('hello', 'utf-16')
   b'\xff\xfeh\x00e\x00l\x00l\x00o\x00'
   
   bytes('hello', 'utf-32')
   b'\xff\xfe\x00\x00h\x00\x00\x00e\x00\x00\x00l\x00\x00\x00l\x00\x00\x00o\x00\x00\x00'
   

3. bytes 和 int 列表的转换。

   list(b'abc')
   [97, 98, 99]
   
   bytes([97, 98, 99])
   b'abc'
   
   bytes([256])
   ValueError: bytes must be in range(0, 256)
   
   # 大端序
   (2008).to_bytes(length=4, byteorder='big', signed=False)
   b'\x00\x00\x07\xd8'
   # 小端序
   (2008).to_bytes(length=4, byteorder='little', signed=False)
   b'\xd8\x07\x00\x00'
   struct.pack('<I', 2008)
   b'\xd8\x07\x00\x00'
   
   int.from_bytes(b'\x00\x00\x07\xd8', 'big')
   2008
   int.from_bytes(b'\xd8\x07\x00\x00', 'little')
   2008
   
   int.from_bytes(b'abc', 'little')
   6513249
   int.from_bytes(b'cba', 'big')
   6513249
   

   PS：2020/11/02, 字节顺序（大端序、小端序）

4. Python2 字符串

   # python2
   print(repr('你好'))
   '\xc4\xe3\xba\xc3'
   print('\xc4\xe3\xba\xc3')
   你好
   
   # python3
   print('\xc4\xe3\xba\xc3')
   'ÄãºÃ'
   print('\xc4\xe3\xba\xc3'.encode('latin-1'))
   b'\xc4\xe3\xba\xc3'
   print('\xc4\xe3\xba\xc3'.encode('latin-1').decode('gbk'))
   '你好'
   

#25 Python tempfile

import tempfile
import time

timestr = time.strftime("%Y%m%d%H%M%S")

临时文件

# tempfile.mktemp(suffix='', prefix='tmp', dir=None)
# tempfile.mkstemp(suffix=None, prefix=None, dir=None, text=False)

filepath_temp = tempfile.mktemp(suffix='.html', prefix='.cache_%s_' % timestr)
print(filepath_temp)
# /tmp/.cache_20210929205452_zlmi3pkf.html

filepath_temp = tempfile.mktemp(suffix='.html', prefix='.cache_%s_' % timestr,
                                dir='/opt/apps/markjour/tmp/')
print(filepath_temp)
# /opt/apps/markjour/tmp/.cache_20210929205452_mcr7nj3e.html

filepath_temp = tempfile.mkstemp(suffix='.html', prefix='.cache_%s_' % timestr, text=True)

mkstemp 和 mktemp 的区别：mkstemp 返回一个文件描述符，mktemp 返回一个文件路径。
mktemp 返回的路径理论上会被另一个进程使用，所以这是 UNSAFE 的，应该用 mkstemp 代替。
mktemp 从 Python 2.3 开始标记为 Deprecated，但是至今还是可以调用。

临时目录

# tempfile.mkdtemp(suffix=None, prefix=None, dir=None)
dirpath_temp = tempfile.mkdtemp()
print(dirpath_temp)
# /tmp/tmp1dj2cl0d

dirpath_temp = tempfile.mkdtemp(prefix='build_',
                                dir='/opt/apps/markjour/tmp/')
print(dirpath_temp)
# /opt/apps/markjour/tmp/build_9k5synh5

#24 Python zipfile

import zipfile
import os

zip_path = "/path/to/zip/file.zip"
extract_path = "/path/to/extract/directory"

if not os.path.exists(extract_path):
    os.makedirs(extract_path)

with zipfile.ZipFile(zip_path, 'r') as fp:
    # 直接解压
    zip_ref.extractall(target_dir)

    # 逐个解压
    for filename in fp.namelist():

        # 如果文件存在就先移除，避免 FileExistsError
        extract_file_path = os.path.join(extract_path, filename)
        if os.path.exists(extract_file_path):
            # print("跳过解压缩文件：", filename)
            # continue
            os.remove(extract_file_path)

        # 解压，会自动创建目录结构
        fp.extract(filename, extract_path)

#23 Python 设置系统默认编码

关于 sys.setdefaultencoding 的一些问题。