TOC

Python 与 HTTP/2

HTTP/2 都出来三年半了,还没有一个好用的 Python 客户端库,真是的。

注:HTTP/3 都快出来了,而且改走 UDP...
相关新闻:下一代 HTTP/3 协议将弃用 TCP 改用 QUIC

检查了一下 requests 的一些信息,找到下面三个 issue:

issue#4156 http2 protocol support?
Some servers are moving http2 for better performance. What would it take to have that in the requests library?
Opened by kgk on 14 Jun 2017, 3 comments
2017/06/14 kennethreitz: In the works!
哎,距今又有一年半了,还是没有动静。
PS: 随后,被 Lukasa 关闭了,理由是从同步的基础之上做修改意味着架构必须发生重大调整。

issue#2452 Add HTTP2 support
The draft specification of HTTP 2.0 is out. As a long-term goal, there should be a way to make requests compatible with it.
Opened by AkshatM on 20 Feb 2015, 1 comment

issue#2082 Add HTTP 2.0 support
Lukasa 提出他弄了一个 hyper 实现了 HTTP/2,然后将这个 issue 关闭了。

requests 应该是使用最多的第三方 HTTP 库吧,可能大神私事繁忙,没有足够精力吧。
当然,也有客观原因:
一方面,由于 HTTP/2 的新特性,可能需要一些异步的支持,对之前的同步架构来说,会有很大的调整。这些变动以及可能产生的不兼容都需要反复掂量(与等待,等待技术换代的一个契机)。
另一方面,相比在 requests 这一层来做变更,更好的方案是和 urllib3 协同更新(urllib3 还没有加入 HTTP/2 支持)。

HTTP/2

  • https://developers.google.com/web/fundamentals/performance/http2/?hl=zh-cn
  • https://juejin.im/post/5aaf51daf265da239376dfde

变化:

  1. 二进制,不再明文传输了
  2. 帧(二进制分帧层)
  3. TSL,协议无要求,但已是事实标准了
  4. 数据流优先级
  5. 连接复用
  6. Header 压缩
  7. 服务器推送
curl -vsS 'https://cn.bing.com/' -H 'authority: cn.bing.com' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'accept-encoding: gzip, deflate' -H 'accept-language: zh-CN,zh;q=0.9' --compressed -o /tmp/cn.bing.com.html
*   Trying 202.89.233.100...
* TCP_NODELAY set
* Connected to cn.bing.com (202.89.233.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [209 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [94 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4555 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.bing.com
*  start date: Jul 20 17:47:08 2017 GMT
*  expire date: Jul 10 17:47:08 2019 GMT
*  subjectAltName: host "cn.bing.com" matched cert's "*.bing.com"
*  issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT TLS CA 5
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x556d97fb88e0)
} [5 bytes data]
> GET / HTTP/2
> Host: cn.bing.com
> authority: cn.bing.com
> upgrade-insecure-requests: 1
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
> accept-encoding: gzip, deflate
> accept-language: zh-CN,zh;q=0.9
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 200
< cache-control: private, max-age=0
< content-type: text/html; charset=utf-8
< content-encoding: gzip
< vary: Accept-Encoding
< p3p: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
< set-cookie: SRCHD=AF=NOFORM; domain=.bing.com; expires=Sun, 13-Dec-2020 03:22:48 GMT; path=/
< set-cookie: SRCHUID=V=2&GUID=3524B9B660A44C16ACC68814CD08E5B7&dmnchg=1; domain=.bing.com; expires=Sun, 13-Dec-2020 03:22:48 GMT; path=/
< set-cookie: SRCHUSR=DOB=20181213; domain=.bing.com; expires=Sun, 13-Dec-2020 03:22:48 GMT; path=/
< set-cookie: _SS=SID=2934048BFC616C332883084FFD4F6D93; domain=.bing.com; path=/
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< report-to: {"group":"network-errors","max_age":604800,"endpoints":[{"url":"https://nel.bingparachute.com/api/report?cat=binghp"}]}
< nel: {"report_to":"network-errors","max_age":604800,"success_fraction":0.01,"failure_fraction":1.0}
< x-msedge-ref: Ref A: 7C446E8F1EF94EF48520FC233A975DFD Ref B: BJ1EDGE0310 Ref C: 2018-12-13T03:22:48Z
< set-cookie: _EDGE_S=F=1&SID=2934048BFC616C332883084FFD4F6D93; path=/; httponly; domain=bing.com
< set-cookie: _EDGE_V=1; path=/; httponly; expires=Tue, 07-Jan-2020 03:22:48 GMT; domain=bing.com
< set-cookie: MUID=2C049E938AEF6583179F92578BC1641B; path=/; expires=Tue, 07-Jan-2020 03:22:48 GMT; domain=bing.com
< set-cookie: MUIDB=2C049E938AEF6583179F92578BC1641B; path=/; httponly; expires=Tue, 07-Jan-2020 03:22:48 GMT
< date: Thu, 13 Dec 2018 03:22:47 GMT
<
{ [2909 bytes data]
* Connection #0 to host cn.bing.com left intact

知识点:

  1. accept-encoding 中去掉 br 这个选项,否则:
    curl: (61) Unrecognized content encoding type. libcurl understands deflate, gzip content encodings.
  2. HTTP 压缩(accept-encoding) 参考:维基百科
    compress: 来自 Unix,被淘汰了
    gzip: GNU zip format, rfc1952
    deflate: 同时使用了LZ77算法与哈夫曼编码(Huffman Coding)的一个无损数据压缩算法,rfc1951
    br: Brotli, HTTP/2 引入
  3. 最下面这个 2909 bytes data,数字时大时小,不一致。300 ~ 3000,原因不明。
    难道有缓存?

python-hyper(h2)

网上一搜,看到一个 hyper 好像比较有名气。

hyper:客户端,不知道是不是废弃了,16 年之后就没有更新。
代码几乎不能用,需要重新梳理。

  • 文档:https://hyper.readthedocs.io/en/latest/
  • 代码:https://github.com/Lukasa/hyper

h2 及其它仓库:HTTP/2 协议相关底层库,积极开发中...

  • 文档:https://python-hyper.org
  • 代码:https://github.com/python-hyper

Tornado HTTP/2 Client

  • https://github.com/vladmunteanu/th2c 基于上面那个 h2
  • https://github.com/bdarnell/tornado_http2 基于 pycurl