#62 logging 不输出中文日志的问题
Python logging 日志 2014-07-16一个小问题。
coding in a complicated world
一个小问题。
Advanced Message Queuing Protocol, 高级消息队列协议
久负盛名的投资公司摩根大通(JPMorgan Chase)在 2005 年前后设计了 AMQP,并和红帽一同采用 Java 实现了这个协议(没过多久就改用 C++ 重构了一遍),这就是后来的 Apache Qpid。后来的一些消息队列也都支持 AMQP 协议,比如 RabbitMQ(采用 Erlang 开发)、Apache ActiveMQ(Java)。
其他常见的 MQ 协议还有:STOMP [^stomp], MQTT [^mqtt],有时也会和 XMPP [^xmpp] 做对比。

MessageDeliveryTag 这个标记十分重要PropertiesHeaderBodyContent TypeContent EncodingMessage QueueMessage BrokerMessage-Oriented Middleware 消息中间件,有时简写 MOMConnection TCP 连接, 服务器永远不会主动关闭连接Channel 通道,或者叫信道,逻辑连接,不同通道之间是完全隔离的 (ChannelID)Server 消息队列服务器,就是指 BrokerVirtual Host 虚拟主机,消息队列中的逻辑隔离Publisher 消息生产者Exchange 交换机Queue 队列Binding 绑定,指定了队列和交换机之间的关系RoutingKey 路由键,Binding 的附加参数,对消息进行过滤Comsumer 消息消费者AMQP 协议中,基本数据单位是帧。有 9 种帧结构用来开启、控制、关闭两点之间的信息传输链路:
- 打开(连接)open
- 开始(会话)begin
- 附加(链路)attach
- 传输 transfer
- 流量控制 flow
- 状态通信 disposition
- 分离(链路)detach
- 结束(会话)end
- 关闭(连接)close
连接 Conection,会话 Session,链路 Link。
链路是单向的数据传输通道,消息(Transfer 帧)就在链路上传输。
+ OpenConnection
| + StartSession
| | + AttachLink
| | | Transfer
| | | Flow
| | | Disposition
| | + DetachLink
| + EndSession
+ CloseConnection
实现了 AMQP 0-9-1。
ps -ef | grep rabbitmq | grep -v grep
rabbitmq 966640 1 5 11:09 ? 00:00:12 /usr/lib/erlang/erts-11.1.8/bin/beam.smp -W w -K true -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa -noshell -noinput -s rabbit boot -boot start_sasl -lager crash_log false -lager handlers []
rabbitmq 966744 966640 0 11:09 ? 00:00:00 erl_child_setup 65536
rabbitmq 966775 1 0 11:09 ? 00:00:00 /usr/lib/erlang/erts-11.1.8/bin/epmd -daemon
rabbitmq 966802 966744 0 11:09 ? 00:00:00 inet_gethost 4
rabbitmq 966803 966802 0 11:09 ? 00:00:00 inet_gethost 4
sudo nmap -p 1-65535 localhost
4369/tcp open epmd
5672/tcp open amqp
15672/tcp open unknown
25672/tcp open unknown
4369 Erlang 端口映射器守护程序 (epmd)5672 服务端口15672 Web 接口25672 不知道干嘛的,来自 inet_dist_listen_min - inet_dist_listen_max, 我本机默认配置type exchangeDeclare struct {
reserved1 uint16
Exchange string
Type string
Passive bool
Durable bool
AutoDelete bool
Internal bool
NoWait bool
Arguments Table
}
type queueDeclare struct {
reserved1 uint16
Queue string
Passive bool
Durable bool
Exclusive bool
AutoDelete bool
NoWait bool
Arguments Table
}
type queueBind struct {
reserved1 uint16
Queue string
Exchange string
RoutingKey string
NoWait bool
Arguments Table
}
AMQP 中定义的 4 种 Exchange 类型:
direct 需要符合 RoutingKey 的完全匹配topic 支持模糊匹配:# 代表任意个单词,* 代表一个单词fanout 将消息广播给所有绑定到该 Exchange 的 Queue,忽略 RoutingKeyheaders 采用消息的 Header 与 Binding 的属性来进行匹配,忽略 RoutingKeyx- 开头的属性不会参与匹配。x-match 属性,any (默认值) 表示匹配中一个字段就行,all 表示所有字段都匹配。预置 Exchange:
(AMQP default) directamq.direct directamq.fanout fanoutamq.headers headersamq.match headersamq.rabbitmq.log topicamq.rabbitmq.trace topicamq.topic topic默认 Exchange (
AMQP default) 有点特殊,所有的队列都自动绑定在上面,然后又是 direct 类型,这样一来,生产者 publish 时如果将消息投递到名字为空字符串的这个 Exchange,RoutingKey 填写队列名字,就可以直接将消息投递到队列中。
具体下来就是:
basic_ack(delivery_tag=0, multiple=False) 可以一次确认多个消息basic_nack(delivery_tag=None, multiple=False, requeue=True)basic.reject(delivery_tag=None, requeue=True)Nack 是 RabbitMQ 对 AMQP 的拓展,和 Reject 作用相同,唯一不同的是,Nack 可以一次拒绝该通道之前(小于 delivery_tag)所有没有 ACK 的消息。
Connection.StartConnection.StartOkConnection.SecureConnection.SecureOkConnection.TuneConnection.TuneOkConnection.OpenConnection.OpenOkConnection.CloseConnection.CloseOkConnection.BlockedConnection.UnblockedChannel.OpenChannel.OpenOkChannel.FlowChannel.FlowOkChannel.CloseChannel.CloseOkAccess.RequestAccess.RequestOkExchange.DeclareExchange.DeclareOkExchange.DeleteExchange.DeleteOkExchange.Bind 交换器绑定 (RabbitMQ 拓展)Exchange.BindOkExchange.Unbind 交换器解绑 (RabbitMQ 拓展)Exchange.UnbindOkQueue.Declare 队列声明Queue.DeclareOkQueue.Bind 队列绑定 (到交换器)Queue.BindOkQueue.Purge 队列清空 (没分配的,也就是 unack 不会被清空)Queue.PurgeOkQueue.Delete 队列删除Queue.DeleteOkQueue.Unbind 队列解绑Queue.UnbindOkBasic.Qos Quality of Service 设置:prefetch_sizeprefetch_countBasic.QosOkBasic.Consume 消费者开始消费Basic.ConsumeOkBasic.Cancel 取消消费者订阅Basic.CancelOkBasic.Publish 发布消息Basic.ReturnBasic.DeliverBasic.Get 直接从指定队列获取消息Basic.GetOkBasic.GetEmptyBasic.Ack 确认Basic.RejectBasic.RecoverAsyncBasic.RecoverBasic.RecoverOkBasic.Nack 负确认 (RabbitMQ 拓展)Tx.SelectTx.SelectOkTx.CommitTx.CommitOkTx.RollbackTx.RollbackOkConfirm.SelectConfirm.SelectOk来自 AMQP 协议文件 amqp-0-9-1.xml 中 Basic 的字段定义:
<!-- These are the properties for a Basic content -->
<!-- MIME typing -->
<field name = "content-type" domain = "shortstr" label = "MIME content type" />
<!-- MIME typing -->
<field name = "content-encoding" domain = "shortstr" label = "MIME content encoding" />
<!-- For applications, and for header exchange routing -->
<field name = "headers" domain = "table" label = "message header field table" />
<!-- For queues that implement persistence -->
<field name = "delivery-mode" domain = "octet" label = "non-persistent (1) or persistent (2)" />
<!-- For queues that implement priorities -->
<field name = "priority" domain = "octet" label = "message priority, 0 to 9" />
<!-- For application use, no formal behaviour -->
<field name = "correlation-id" domain = "shortstr" label = "application correlation identifier" />
<!-- For application use, no formal behaviour but may hold the
name of a private response queue, when used in request messages -->
<field name = "reply-to" domain = "shortstr" label = "address to reply to" />
<!-- For implementation use, no formal behaviour -->
<field name = "expiration" domain = "shortstr" label = "message expiration specification" />
<!-- For application use, no formal behaviour -->
<field name = "message-id" domain = "shortstr" label = "application message identifier" />
<!-- For application use, no formal behaviour -->
<field name = "timestamp" domain = "timestamp" label = "message timestamp" />
<!-- For application use, no formal behaviour -->
<field name = "type" domain = "shortstr" label = "message type name" />
<!-- For application use, no formal behaviour -->
<field name = "user-id" domain = "shortstr" label = "creating user id" />
<!-- For application use, no formal behaviour -->
<field name = "app-id" domain = "shortstr" label = "creating application id" />
<!-- Deprecated, was old cluster-id property -->
<field name = "reserved" domain = "shortstr" label = "reserved, must be empty" />
来自 rabbitmq/amqp091-go@v1.10.0/types.go 中的消息类型定义:
// Publishing captures the client message sent to the server. The fields
// outside of the Headers table included in this struct mirror the underlying
// fields in the content frame. They use native types for convenience and
// efficiency.
type Publishing struct {
// Application or exchange specific fields,
// the headers exchange will inspect this field.
Headers Table
// Properties
ContentType string // MIME content type
ContentEncoding string // MIME content encoding
DeliveryMode uint8 // Transient (0 or 1) or Persistent (2)
Priority uint8 // 0 to 9
CorrelationId string // correlation identifier
ReplyTo string // address to to reply to (ex: RPC)
// Expiration represents the message TTL in milliseconds. A value of "0"
// indicates that the message will immediately expire if the message arrives
// at its destination and the message is not directly handled by a consumer
// that currently has the capacatity to do so. If you wish the message to
// not expire on its own, set this value to any ttl value, empty string or
// use the corresponding constants NeverExpire and ImmediatelyExpire. This
// does not influence queue configured TTL values.
Expiration string
MessageId string // message identifier
Timestamp time.Time // message timestamp
Type string // message type name
UserId string // creating user id - ex: "guest"
AppId string // creating application id
// The application specific payload of the message
Body []byte
}
OSI: Open System Interconnect,开放系统互联


按照 OSI 的设计, 应用层的数据可以在这一层经过编码转换,压缩,加密。
端口 Port。
IP 地址。
PDU (协议数据单元): Packet (包)
PDU (协议数据单元): Frame (帧)
Logical Link Control
Media Access Control
PDU (协议数据单元): Bit
比特转换为物理层的信号, 比如无线电信号、光纤信号、载波信号等。
varchar(14),存:yyyymmddHHMMSS 格式varchar(19),存:yyyy-mm-dd HH:MM:SS 格式varchar(23),存:yyyy-mm-dd HH:MM:SS.fff 格式varchar(26),存:yyyy-mm-dd HH:MM:SS.ffffff 格式varchar(24),存:yyyy-mm-ddTHH:MM:SS+0800 或 yyyy-mm-dd HH:MM:SS 0800 格式int 1970 - 2038time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime((1 << 31) - 1))
# 2038-01-19 03:14:07
- int unsigned 1970 - 2106time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime((1 << 32) - 1))
# 2106-02-07 06:28:15
- bigint 可以用到世界毁灭2020-01-01 00:00:00TIMESTAMP 类型int 类型,2038 问题DATETIME 类型PS: DATE, TIME, YEAR 三种类型:
DATETIMEYEARMAC 地址的正式名称应该是 Media Access Control Address, 翻译过来就是 “媒体接入控制层地址”。
Timestamp 是按 UTC 时间进行存储。

MySQL 的 timestamp 类型挺好的,不过有个严重的 2038 问题,我不知道到时候这个类型会如何处理。
The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.
(1 << 31) / (3600 * 24 * 365)
根据时间范围,这应该是用的一个有符号的 32 位整型数实现的,为什么不用无符号数呢,不是又可以往后续 68 年么?
我预测,MySQL 的某一个新版本,将会增加一个 TIMESTAMP64 类型,采用 64 位有符号整形数存时间戳。
或直接将 TIMESTAMP 拓展到 64 位,先支持配置和命令行参数开启这个特性,然后后续版本将其视作默认设置。
The Web Server Gateway Interface (WSGI, pronounced whiskey or WIZ-ghee) is a simple calling convention for web servers to forward requests to web applications or frameworks written in the Python programming language.
The current version of WSGI, version 1.0.1, is specified in Python Enhancement Proposal (PEP) 3333.
Web 服务器网关接口(WSGI,发音为威士忌或 WIZ-ghee)是一种简单的调用约定,用于将请求转发到用 Python 编写的 Web 应用程序或框架的 Web 服务器。
当前版本的 WSGI,即 1.0.1 版本,由 Python 增强提案(PEP)3333 指定。
WSGI was originally specified as PEP-333 in 2003.
PEP-3333, published in 2010, updates the specification for Python 3.
WSGI 最初是在 2003 年的 PEP-333 中指定的。
2010 年发布的 PEP-3333 更新了 Python 3 的规范。
In 2003, Python web frameworks were typically written against only CGI, FastCGI, mod_python, or some other custom API of a specific web server. To quote PEP 333:
2003 年,Python Web 框架通常只针对 CGI、FastCGI、mod_python 或某些特定 Web 服务器的其他自定义 API 编写。引用 PEP 333 的话:
Python currently boasts a wide variety of web application frameworks, such as Zope, Quixote, Webware, SkunkWeb, PSO, and Twisted Web -- to name just a few. This wide variety of choices can be a problem for new Python users, because generally speaking, their choice of web framework will limit their choice of usable web servers, and vice versa... By contrast, although Java has just as many web application frameworks available, Java's "servlet" API makes it possible for applications written with any Java web application framework to run in any web server that supports the servlet API.
Python 目前拥有众多 Web 应用程序框架,例如 Zope、Quixote、Webware、SkunkWeb、PSO 和 Twisted Web 等等。这种广泛的选择可能会成为新 Python 用户的问题,因为一般来说,他们选择的 Web 框架将限制他们可用的 Web 服务器的选择,反之亦然……相比之下,尽管 Java 也拥有同样多的 Web 应用程序框架,但 Java 的“servlet”API 使得使用任何 Java Web 应用程序框架编写的应用程序都能在支持 servlet API 的任何 Web 服务器中运行。
WSGI was thus created as an implementation-neutral interface between web servers and web applications or frameworks to promote common ground for portable web application development.
因此,WSGI 被创建为实现中立的 Web 服务器和 Web 应用程序或框架之间的接口,以促进可移植 Web 应用程序的开发和共同基础。
The WSGI has two sides:
WSGI 有两个方面:
Between the server and the application, there may be one or more WSGI middleware components, which implement both sides of the API, typically in Python code.
在服务器和应用程序之间,可能会有一个或多个WSGI 中间件组件,它们在 Python 代码中实现了 API 的两个方面。
WSGI does not specify how the Python interpreter should be started, nor how the application object should be loaded or configured, and different frameworks and webservers achieve this in different ways.
WSGI 不指定 Python 解释器应如何启动,也不指定应用程序对象如何加载或配置,不同的框架和 Web 服务器以不同的方式实现这些功能。
A WSGI middleware component is a Python callable that is itself a WSGI application, but may handle requests by delegating to other WSGI applications. These applications can themselves be WSGI middleware components.
一个 WSGI 中间件组件是一个 Python 可调用对象,它本身是一个 WSGI 应用程序,但可以通过委托给其他 WSGI 应用程序来处理请求。这些应用程序本身可以是 WSGI 中间件组件。
A middleware component can perform such functions as:
中间件组件可以执行以下功能:
A WSGI-compatible "Hello, World!" application written in Python:
def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
yield b'Hello, World!\n'
Where:
application, which takes two parameters, environ and start_response. environ is a dictionary containing CGI environment variables as well as other request parameters and metadata under well-defined keys. start_response is a callable itself, taking two positional parameters, status and response_headers.start_response, specifying "200 OK" as the HTTP status and a "Content-Type" response header.A full example of a WSGI network server is outside the scope of this article. Below is a sketch of how one would call a WSGI application and retrieve its HTTP status line, response headers, and response body, as Python objects. Details of how to construct the environ dict have been omitted.
完整的 WSGI 网络服务器示例不再本文范围之内。以下只是一个 Demo,说明如何调用 WSGI 应用程序并检索其 HTTP 状态行、响应头和响应正文,作为 Python 对象。如何构造 environ 字典的详细信息已被省略。
from io import BytesIO
def call_application(app, environ):
status = None
headers = None
body = BytesIO()
def start_response(rstatus, rheaders):
nonlocal status, headers
status, headers = rstatus, rheaders
app_iter = app(environ, start_response)
try:
for data in app_iter:
assert status is not None and headers is not None, \
"start_response() was not called"
body.write(data)
finally:
if hasattr(app_iter, 'close'):
app_iter.close()
return status, headers, body.getvalue()
environ = {...} # "environ" dict
status, headers, body = call_application(app, environ)
PEP 3333 引入了一些新特性和更严格的规范,以提高 Python Web 应用程序的可移植性和互操作性。其中一些变化包括:
此外,PEP 3333 还引入了一些新的建议,包括使用可选的服务器和客户端请求信息,提供更好的错误处理和日志记录,以及更好地处理请求和响应的二进制数据。
-, 普通文件l, 链接文件d, 目录c, 字符设备文件b, 块设备文件p, 管道文件s, Sockett, 特殊文件ls)cd)touch /tmp/testPerm
chmod 7777 /tmp/testPerm
ll /tmp/testPerm
-rwsrwsrwt 1 markjour markjour 0 2014-02-27 17:41:06 /tmp/testPerm
Linux 权限的两种表示方法:
Linux 中的权限一共用 12 位来表示,除了上面 u g o 各 3 位(rwx)一共 9 位之外,还有 3 位:
sticky bit ,1,显示为 T
suid, sgid 是用来给文件设置临时提权用的。如果设置, 所有用户都会继承文件所有者或所有组的权限。
| suid | sgid | sbit | |
|---|---|---|---|
| 文件 | |||
| 目录 |
$ ls -alh /bin/ | grep rws
-rwsr-sr-x 1 daemon daemon 55K 2022-04-14 08:23:36 at
-rwsr-xr-x 1 root root 72K 2022-11-24 20:05:18 chfn
-rwsr-xr-x 1 root root 44K 2022-11-24 20:05:18 chsh
-rwsr-xr-x 1 root root 35K 2022-03-23 21:53:14 fusermount3
-rwsr-xr-x 1 root root 71K 2022-11-24 20:05:18 gpasswd
-rwsr-xr-x 1 root root 47K 2023-04-07 06:21:06 ksu
-rwsr-xr-x 1 root root 47K 2022-02-21 09:49:57 mount
-rwsr-xr-x 1 root root 28K 2022-11-24 20:05:18 newgidmap
-rwsr-xr-x 1 root root 40K 2022-11-24 20:05:18 newgrp
-rwsr-xr-x 1 root root 28K 2022-11-24 20:05:18 newuidmap
-rwsr-xr-x 1 root root 31K 2022-02-07 21:05:06 nvidia-modprobe
-rwsr-xr-x 1 root root 59K 2022-11-24 20:05:18 passwd
-rwsr-xr-x 1 root root 31K 2022-02-26 19:11:57 pkexec
-rwsr-xr-x 1 root root 55K 2022-02-21 09:49:57 su
-rwsr-xr-x 1 root root 227K 2023-04-04 02:00:44 sudo
-rwsr-xr-x 1 root root 35K 2022-02-21 09:49:57 umount
$ find /bin/ -type f -perm /4000 -ls
7342874 36 -rwsr-xr-x 1 root root 35200 Mar 23 2022 /bin/fusermount3
7352563 28 -rwsr-xr-x 1 root root 28136 Nov 24 2022 /bin/newuidmap
7341794 32 -rwsr-xr-x 1 root root 30872 Feb 26 2022 /bin/pkexec
7347918 36 -rwsr-xr-x 1 root root 35192 Feb 21 2022 /bin/umount
7347747 48 -rwsr-xr-x 1 root root 47480 Feb 21 2022 /bin/mount
7344663 28 -rwsr-xr-x 1 root root 28136 Nov 24 2022 /bin/newgidmap
7349993 72 -rwsr-xr-x 1 root root 72072 Nov 24 2022 /bin/gpasswd
7350008 60 -rwsr-xr-x 1 root root 59976 Nov 24 2022 /bin/passwd
7347439 228 -rwsr-xr-x 1 root root 232416 Apr 4 02:00 /bin/sudo
7344105 32 -rwsr-xr-x 1 root root 30936 Feb 7 2022 /bin/nvidia-modprobe
7345135 56 -rwsr-xr-x 1 root root 55672 Feb 21 2022 /bin/su
7347241 72 -rwsr-xr-x 1 root root 72712 Nov 24 2022 /bin/chfn
7352704 40 -rwsr-xr-x 1 root root 40496 Nov 24 2022 /bin/newgrp
7347300 44 -rwsr-xr-x 1 root root 44808 Nov 24 2022 /bin/chsh
7382902 48 -rwsr-xr-x 1 root root 47416 Apr 7 06:21 /bin/ksu
7343017 56 -rwsr-sr-x 1 daemon daemon 55624 Apr 14 2022 /bin/at
$ find /bin/ -type f -perm /2000 -ls
7342246 16 -rwxr-sr-x 1 root root 15504 Jan 12 2022 /bin/dotlock.mailutils
7347219 72 -rwxr-sr-x 1 root shadow 72184 Nov 24 2022 /bin/chage
7346616 24 -rwxr-sr-x 1 root tty 22912 Feb 21 2022 /bin/write.ul
7340649 308 -rwxr-sr-x 1 root plocate 313904 Feb 17 2022 /bin/plocate
7341420 40 -rwxr-sr-x 1 root crontab 39568 Mar 23 2022 /bin/crontab
7349934 24 -rwxr-sr-x 1 root shadow 23136 Nov 24 2022 /bin/expiry
7347279 24 -rwxr-sr-x 1 root tty 22904 Feb 21 2022 /bin/wall
7345047 288 -rwxr-sr-x 1 root _ssh 293304 Nov 23 2022 /bin/ssh-agent
7343017 56 -rwsr-sr-x 1 daemon daemon 55624 Apr 14 2022 /bin/at
https://en.wikipedia.org/wiki/Setuid
https://en.wikipedia.org/wiki/Sticky_bit
https://web.archive.org/web/20130204053849/http://content.hccfl.edu/pollock/aunix1/filepermissions.htm
chown 更改文件所有者和文件所有组chgrp 更改文件所有组chmod 更改权限chmod 755 filePath
chmod 644 filePath
chmod u+x filePath
chmod g+r filePath
chmod o+w filePath
chmod a+r filePath

time 库
本文主要是说 time 库。
参考文档:https://docs.python.org/2/library/time.html