coding in a complicated world
Ubuntu 19.04 与 4/19 发布，生命周期 9 个月。我提前两个月就定着日程在等，总是克制不住追新的冲动，哈哈。
好像主要就是：
do-release-upgrade
由于审计过程中意外中断，导致出现了一些问题，这里做个记录。
声明：这些问题，是非正常升级流程出现的，不是说是 Ubuntu 的问题。
在开源中国每日动弹中看到这么一道题目，蛮有意思，还学到了一个新的 MySQL 语法：CTE。
线上某些基于 nodejs 的服务时不时看到这种进程，而且一直都在：
root 25997 1 0 Mar25 ? 00:00:00 /root/.nvm/versions/node/v10.14.2/bin/node /root/.nvm/versions/node/v10.14.2/lib/node_modules/npm/node_modules/update-notifier/check.js {"pkg":{"name":"npm","version":"6.4.1"}}
经过检查判断，这是 npm 带来的一个依赖模块，用来做版本检测。
| Date | Version |
|---|---|
| 2023-07-28 | 1.1.5 |
| 2023-05-12 | 1.1.4 |
| 2023-04-30 | 1.1.3 |
| 2023-04-09 | 1.1.2 |
| 2023-03-10 | 1.1.1 |
| 2023-02-25 | 1.1.0 |
| 2023-04-30 | 1.0.6 |
| 2020-08-09 | 1.0.5 |
| 2020-04-06 | 1.0.4 |
| 2020-01-15 | 1.0.3 |
| 2019-12-31 | 1.0.2 |
| 2019-12-15 | 1.0.1 |
| 2019-12-09 | 1.0.0 |
| 2019-12-24 | 0.9.6 |
| 2019-10-07 | 0.9.5 |
| 2019-09-25 | 0.9.4 |
| 2019-08-09 | 0.9.3 |
| 2019-04-15 | 0.9.2 |
| 2018-12-09 | 0.9.1 |
| 2018-10-30 | 0.9.0 |
默认签名字段（28 个）：
cc, content-description, content-id, content-transfer-encoding, content-type,
date,
from,
in-reply-to,
list-archive, list-help, list-id, list-owner, list-post, list-subscribe, list-unsubscribe,
message-id, mime-version,
references, reply-to, resent-cc, resent-date, resent-from, resent-message-id, resent-sender, resent-to,
sender, subject,
to
d = dkim.DKIM('')
print(b', '.join(d.should_sign | d.frozen_sign))
b'list-unsubscribe, content-id, list-id, mime-version, resent-date, sender, cc, reply-to, content-type, list-owner, resent-message-id, resent-cc, resent-from, to, content-description, date, list-post, in-reply-to, content-transfer-encoding, from, references, list-help, subject, list-archive, resent-sender, list-subscribe, message-id, resent-to'
print(b', '.join(sorted(d.should_sign | d.frozen_sign)))
b'cc, content-description, content-id, content-transfer-encoding, content-type, date, from, in-reply-to, list-archive, list-help, list-id, list-owner, list-post, list-subscribe, list-unsubscribe, message-id, mime-version, references, reply-to, resent-cc, resent-date, resent-from, resent-message-id, resent-sender, resent-to, sender, subject, to'
准备实验用的密钥对。
openssl genpkey -algorithm RSA -out /tmp/private_key.pem
openssl rsa -in /tmp/private_key.pem -check
openssl rsa -pubout -in /tmp/private_key.pem -out /tmp/public_key.pem
import dkim
domain = 'mail.markjour.com'
selector = 's20190416'
with open('/tmp/private_key.pem', 'rb') as f:
privkey = f.read().strip()
# dkim.parse_pem_private_key(privkey)
message = """
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=utf-8
Date: Mon, 24 Sep 2018 12:31:21 +0000 (UTC)
From: Admin <no-reply@mail.markjour.com>
Mime-Version: 1.0
Subject: Hello World
Message-ID: <n4F5zz24LXvYqPHVrZLPJokasT7MlLxYQx6g>
Reply-To: sender@mail.markjour.com
To: kwicoo@gmail.com
List-Unsubscribe: <mailto:unsubscribe@mail.markjour.com?p=Ahi2DRmdOnTdpsDzPClCPqbpwmFyjvGJV2xfJGWqw6eFEKRwI402QeoSsFrArTw1s48A59f60pLl0x71ojsQSWERnp3aMZA6YvEw>
X-SMTP-ID: c89cf6a5-22b7-4d1a-9bce-9f91a6be1bfb
HELLO WORLD
""".strip().encode()
# dkim.rfc822_parse(message)
# print(dkim.DKIM(message).default_sign_headers())
# [b'Content-Transfer-Encoding', b'Content-Type', b'Date', b'From', b'Mime-Version', b'Subject', b'Message-ID', b'Reply-To', b'To', b'List-Unsubscribe', b'From']
signature = dkim.sign(message, selector.encode(), domain.encode(), privkey)
print(signature.decode())
# DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mail.markjour.com;
# i=@mail.markjour.com; q=dns/txt; s=s20190416; t=1555392275;
# h=content-transfer-encoding : content-type : date : from :
# mime-version : subject : message-id : reply-to : to : list-unsubscribe
# : from; bh=qg03cTlGc4OH4uPv7BGgoUyhgh23r+o1O6qzYOLixvA=;
# b=sJ09G6hHPaP6AMp2mqUXjEZ+BfUFz0o6nbpXWxJ4/OG0o9ZwPSj8aJibZtJjTKP3k/TR/
# 6SD543V8iNw+JwwM+XLOUZa0iduK+QkedccqNl5Hcfc9UI/U11NoHz76B3csL9KE9tb40jF
# mlLCuVUjci4HlOfEoKF8Ame8yWDHXVoNS/YT9/OSSc5q5q+qp6OX6PvzzxDomCHC6kbhOdv
# Yc/KEXrMQ1JQ971pRUBNQK3eN7bV7g1BwXuMEuhdwDa4aZ4YYcakKywo4Oey7bIy1E7evZN
# 5rUitRExLH4dQNrhxoZd4c3QOjd4ROTwseAaMN10U/egzDXjcw2q0UUC1UKQ==
append_headers = [b'x-smtp-id']
d = dkim.DKIM(message)
include_headers = d.default_sign_headers()
include_headers.extend(append_headers)
signature = d.sign(selector.encode(), domain.encode(), privkey, include_headers=include_headers)
print(signature.decode())
# DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mail.markjour.com;
# i=@mail.markjour.com; q=dns/txt; s=s20190416; t=1555392275;
# h=content-transfer-encoding : content-type : date : from :
# mime-version : subject : message-id : reply-to : to : list-unsubscribe
# : from : x-smtp-id; bh=qg03cTlGc4OH4uPv7BGgoUyhgh23r+o1O6qzYOLixvA=;
# b=MTSeE8X3R+8bn+kkJaX5j/OKPMe+sdombmmwK5zME3SHBqiOLbxCwOGyh3qJKXdLpJlEg
# pBnsDmNEjgC/rtBoclvnlCsaN7OFcZIe6ehfjwGeaw41r38Y8IgUQCkuN+IiL8FN1IiMI2f
# kSayumwcOCAwmA4yJfu8n1v4W416jXt775YKR+1bt2Df1fNA6FnfoSMTqZl7rHn9zo76Efg
# yvm7M0uT3uz0NZbJtqOnMFzRri9TEj4jYiCgsNaBYA9prbZlA02svoJx9qIJ2mKA+EcVpxK
# IsEAY4ZXzXfhynKLeOYGK786ghiZrtsQYGbP6c1fAzTNy+fLJzRFozsV/wEQ==
headers = set(dkim.DKIM.SHOULD) | set(dkim.DKIM.FROZEN) | append_headers - set(dkim.DKIM.SHOULD_NOT)
signature = dkim.sign(message, selector.encode(), domain.encode(), privkey, include_headers=headers)
print(signature.decode())
# DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mail.markjour.com;
# i=@mail.markjour.com; q=dns/txt; s=s20190416; t=1555392275;
# h=in-reply-to : x-smtp-id : content-type : cc : content-id : list-post
# : date : resent-from : list-owner : list-id : to : content-description
# : list-subscribe : message-id : sender : mime-version :
# resent-message-id : list-help : content-transfer-encoding : resent-cc
# : resent-date : list-unsubscribe : references : resent-sender : from :
# list-archive : subject : resent-to : reply-to;
# bh=qg03cTlGc4OH4uPv7BGgoUyhgh23r+o1O6qzYOLixvA=;
# b=lLJafHJ8B/DoO4FncLp+BIHaPy4xsq7dRAWjzAvkRoDSwjcg3EloW0FsCXS45EkmwmBZC
# Vks7zeOR1CS8oxcpauhxj1XnlwfcwLWtAQ3pogQTzNh4EEUFiNfgJTdXefAh7cpGHolQmy7
# w2TBXDPx+Ikynw2tNnGOBduLWi+BH3Et8KGaskR4D9QHWSrk4pqeaNannNhDPUfE98d2fS3
# kKBvqiEaTubQBdi8VXcl8J4R1SfdJZR2NfBJkPjJejlwTJaSytF2zyberpgflj0sEc8iHvM
# 2UQRcpxqm8GMRyzzKAXBSTzQhmaTOHntGokDTunNlUc/izFFRJm9SFiVq64g==
使用 opendkim-genkey 生成签名私钥和 DNS 配置文件：
$ opendkim-genkey --verbose --domain=mail.markjour.com --selector=s20190416 --directory=/tmp/
opendkim-genkey: generating private key
opendkim-genkey: private key written to s20190416.private
opendkim-genkey: extracting public key
opendkim-genkey: DNS TXT record written to s20190416.txt
域名配置好之后，可以使用 opendkim-testkey 检查：
$ opendkim-testkey -d test.markjour.com -k /tmp/private_key.pem -s s20190416 -v
opendkim-testkey: 's20190416._domainkey.test.markjour.com' record not found
有空思考一下几种场景下的文件存储方案。
- 场景 1. Web 静态资源
- 场景 2. 虚拟化中的卷存储
- 场景 3. 邮件系统中的邮件
- 场景 4. 大量图片，音频，视频（二进制）
- 场景 5. 大量日志（文本）
突然想了解一下我们 SMTP 服务中的入栈邮件大小，于是写了点脚本分析一下。
先上结论：
上面的接口中，我只认识 3.5mm 线接口。
还有一种常见音频输入和输出接口没有在上图中，那就是万能的 USB 了。
AUX 线，就是两段都是 3.5mm 的音频线，也就是常说的公对公。
电脑上的音频接口：
蓝色接口：线路输入(LineIn)，接一些播放设备，比如 CD 播放器、录音台等
为什么需要一个线路输入？
音频输入接口会从物理层面对信号进行预处理（比如放大），因为麦克风的输入信号强度低（低电平、高阻抗）。
而线路输入是经过处理的高质量音频信号，更看重的是数据保真。
某用户委托安全公司对本司（SendCloud）短信发送业务做安全检测，发现咱们的上游通道某一环节的安全漏洞。
跟踪这个过程，真的十分有趣。
这是 XSS 第一次发生在我身边，怎么也不会想到有人会犯这么弱智的错误。最基本的页面输出转义都没做。
页面内容输出转义、SQL 防注入、表单的 CSRF token 校验，应该算是 Web 站点搭建的基础工作吧！
https://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_id
server {
listen 80;
server_name example.com;
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'Request-ID: $request_id';
add_header X-Request-ID $request_id;
location / {
proxy_pass http://backend_server;
proxy_set_header X-Request-ID $request_id;
}
}
后台服务接收到 X-Request-ID 头信息，并记录到日志中。
可以放到线程上下文，或者协程上下文中，在所有相关日志打印时都带上 X-Request-ID。
from flask import Flask, request
import logging
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
@app.route('/')
def hello():
request_id = request.headers.get('X-Request-ID')
logging.info(f"Request ID: {request_id} - Processing request...")
return 'Hello, World!'
if __name__ == '__main__':
app.run()
以我本地系统（Ubuntu）上的 supervisord 为例。
ps -ef | grep supervisord / pgrep supervisordsudo cat /proc/1553/mapssudo gdb attach 1553dump memory /tmp/1553-heap.mem 0x564a7e667000 0x564a7eb25000strings -n 10 /tmp/1553-heap.mem/proc/pid/maps 文件的格式内存映射（mmap）：
这里的 maps 文件就是内存映射的一个情况。
sudo head /proc/self/maps 试试。当前进程的内存映射信息表，软链接。vm_area_struct$ sudo head /proc/1553/maps
564a7d858000-564a7d8a5000 r--p 00000000 08:01 3018208 /usr/bin/python2.7
564a7d8a5000-564a7da56000 r-xp 0004d000 08:01 3018208 /usr/bin/python2.7
564a7da56000-564a7db65000 r--p 001fe000 08:01 3018208 /usr/bin/python2.7
564a7db66000-564a7db68000 r--p 0030d000 08:01 3018208 /usr/bin/python2.7
564a7db68000-564a7dbde000 rw-p 0030f000 08:01 3018208 /usr/bin/python2.7
564a7dbde000-564a7dc02000 rw-p 00000000 00:00 0
564a7e667000-564a7eb25000 rw-p 00000000 00:00 0 [heap]
7fe8e4308000-7fe8e4448000 rw-p 00000000 00:00 0
7fe8e4448000-7fe8e444c000 r--p 00000000 08:01 1840428 /lib/x86_64-linux-gnu/libexpat.so.1.6.8
7fe8e444c000-7fe8e446d000 r-xp 00004000 08:01 1840428 /lib/x86_64-linux-gnu/libexpat.so.1.6.8
r/w/x，不必说，第四位有两种值：p 或 s，分别表示私有 private 或共享 shared。0000000000000000[heap] 堆[stack] 栈，主线程（main process）[stack:1001] 栈，线程 ID 我还没见到过这样的情况[vdso][vvar][vsyscall][heap]，子线程申请则是匿名映射。[stack:1001] 情况的原因么？从 maps 文件记录上看，增加一个子线程，在 maps 文件中就增加了两条记录，分别是子线程的栈空间和栈保护页的记录。默认情况下，pthread为子线程预留的栈空间大小为1MB，栈保护页为4KB（这主要跟页大小相关）。
#!/usr/bin/env python
import re
maps_file = open('/proc/self/maps', 'r')
mem_file = open('/proc/self/mem', 'r', 0)
for line in maps_file.readlines(): # for each mapped region
m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])', line)
if m.group(3) == 'r': # if this is a readable region
start = int(m.group(1), 16)
end = int(m.group(2), 16)
mem_file.seek(start) # seek to region start
chunk = mem_file.read(end - start) # read region contents
print chunk, # dump contents to standard output
maps_file.close()
mem_file.close()
sudo awk -n -F '[- ]' '/\[heap\]/ {h="0x"$2-"0x"$1+1;printf("%d bytes (%.2f MB)\n",h,h/1024/1024)}' /proc/1553/maps
python -c "import sys;a,b=sys.argv[1].split('-');ai=int('0x'+a,16);bi=int('0x'+b,16);x=(bi-ai+1);y=x/(1024*1024);print('%d Bytes (%.2f MB)'%(x,y));" 55854b0f5000-55855a372000