分类 cURL 下的文章

下载 高速公路年度发展报告2021
001.jpg

curl 'https://book.yunzhan365.com/wkmya/muml/files/mobile/[1-124].jpg'   -H 'authority: book.yunzhan365.com'   -H 'sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"'   -H 'sec-ch-ua-mobile: ?0'   -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'   -H 'accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8'   -H 'sec-fetch-site: same-origin'   -H 'sec-fetch-mode: no-cors'   -H 'sec-fetch-dest: image'   -H 'referer: https://book.yunzhan365.com/wkmya/muml/mobile/index.html'   -H 'accept-language: zh-CN,zh;q=0.9'   -H 'cookie: sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%2217a0e582ca111ac-06e58c91b95692-3c710e58-1430715-17a0e582ca2110a%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%2217a0e582ca111ac-06e58c91b95692-3c710e58-1430715-17a0e582ca2110a%22%7D; Hm_lvt_7fafaf8b8dc670c2109d0bbddc8be7e0=1623738298; Qs_lvt_323391=1623738298; Qs_pv_323391=309891294092990200%2C1997975047369361700%2C477389241827979800%2C3537689015888066000%2C4525915336882614300'   --compressed -O

curl使用保留符号[]和{}进行通配,它们一般不是合法URL的组成部分(IPv6地址除外),同时可以使用-g或--globoff禁用它。

可以使用[N-M]语法来指定一个数值范围,N为起始,M为结束(包括M):

curl -O http://example.com/[1-100].png

可以使用0作为前缀,如三位数的数字:

curl -O http://example.com/[001-100].png

如果只想请求数字为偶数的图像,可以使用步进(step counter):

curl -O http://example.com/[0-100:2].png

同时也可以处理字母范围:

curl -O http://example.com/section[a-z].html

如果需要采用列表,则需要将其放在花括号中,而不是中括号中:

curl -O http://example.com/{one,two,three,alpha,beta}.html

同时可以混用不同的组合:

curl -O http://example.com/{Ben, Alice, Frank}-{100*100, 1000*1000}.jpg
curl -O http://example.com/chess-[0-7]x[0-7].jpg

如果正在下载多个文件,并希望将它们保存在不同的子目录中,或者以不同的名称保存文件。curl可以采用输出文件名变量。URL中的每个通配都对应一个单独的变量,可以通过'#[num]'来引用,即在‘#’后面跟上与通配符对应的数字,从1(对应第一个通配)开始,到最后一个结束,如:

curl http://{one, two}.example.com -o "file_#1.txt"
curl http://{site, host}.host[1-5].example.com -o "subdir/#1_#2"

linux 合并所有图片到pdf文件

sudo apt install imagemagick

修改/etc/ImageMagick-6/policy.xml文件中
<policy domain="coder" rights="none" pattern="PDF" />

<policy domain="coder" rights="read | write" pattern="PDF" />

convert *.jpg -auto-orient pictures.pdf

convert `ls -1v *.jpg` file.pdf

1、curl -w参数

curl -h |grep "write-out"

-w, --write-out <format> Use output FORMAT after completion

2、显示所有时间示例

curl -o /dev/null -s -w "time_namelookup:%{time_namelookup}\ntime_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"  "https://const.net.cn/"

输出:

time_namelookup:0.058759
time_connect: 0.069912
time_starttransfer: 0.147355
time_total: 0.167685

3、参数说明

time_namelookup: DNS 服务器域名解析的时间(以上时间单位都是s)
time_connect: client 发出请求,到 c/s 建立TCP 的时间;里面包括 DNS 解析的时间
time_starttransfer: client 发出请求;到 server 响应发出第一个字节开始的时间;包括前面的2个时间
time_total: 从请求发起到链接关闭的总耗时

4、官方英文说明

time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.
time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
time_total: The total time, in seconds, that the full operation lasted.

5、格式可以使用字符串,格式化文件以及标准输入

The format can be specified as a literal "string", or you can have curl read the format from a file with "@filename" and to tell curl to read the format from stdin you write "@-".

6、使用格式文件示例

cat curl-format.txt

   time_namelookup:%{time_namelookup}\n
      time_connect:%{time_connect}\n
   time_appconnect:%{time_appconnect}\n
     time_redirect:%{time_redirect}\n    
  time_pretransfer:%{time_pretransfer}\n  
time_starttransfer:%{time_starttransfer}\n
------------------------------\n
        time_total:  %{time_total}\n

使用示例:

curl -o /dev/null -s -w "@curl-format.txt"  "https://const.net.cn/"

输出结果:

   time_namelookup:0.165160
      time_connect:0.240436
   time_appconnect:0.326137
     time_redirect:0.000000
  time_pretransfer:0.326211
time_starttransfer:0.412927
------------------------------
        time_total:  0.486244

1、查看curl与时间相关的选项

curl -h |grep time

有用的输出

--connect-timeout <seconds> Maximum time allowed for connection
--expect100-timeout <seconds> How long to wait for 100-continue
--keepalive-time <seconds> Interval time for keepalive probes
-m, --max-time <seconds> Maximum time allowed for the transfer
-y, --speed-time <seconds> Trigger 'speed-limit' abort after this time

2、控制连接超时--connect-timeout 5

curl -v --connect-timeout 1 https://const.net.cn

当出现连接超时时,会提示如下信息

curl: (28) Connection timed out after 1001 milliseconds

3、总超时时间--max-time
 我们希望curl程序在运行一段时间后,不管有没有执行完成都能正常退出。就用这个max-time

curl -v -m 60 https://const.net.cn

4、--speed-time 的用法
speed-time是用来根据速率来控制连接是否断开的。
用法:

curl -o const.log "https://const.net.cn" --speed-time 5 --speed-limit 1

是说将url内容保存到const.log中, 如果传输速度小于1字节/秒的状态持续5秒,该连接就会断开,并退出程序。

0、curl 显示连接的常用方法

curl -d "hello=world" https://const.net.cn/test/ --verbose

输出结果:
...

  • Server certificate:
  • subject: CN=const.net.cn
  • start date: May 19 00:00:00 2021 GMT
  • expire date: May 18 23:59:59 2022 GMT
  • subjectAltName: host "const.net.cn" matched cert's "const.net.cn"
  • issuer: C=CN; O=TrustAsia Technologies, Inc.; OU=Domain Validated SSL; CN=TrustAsia TLS RSA CA
  • SSL certificate verify ok.

POST /test/ HTTP/1.1
Host: const.net.cn
User-Agent: curl/7.68.0
Accept: /
Content-Length: 11
Content-Type: application/x-www-form-urlencoded

  • upload completely sent off: 11 out of 11 bytes
    ...

1、curl 显示 form 数据

curl -d "hello=world" https://const.net.cn/test/ --trace-ascii /dev/stdout

输出结果:
0000: POST /test/ HTTP/1.1
0016: Host: const.net.cn
002a: User-Agent: curl/7.68.0
0043: Accept: /
0050: Content-Length: 11
0064: Content-Type: application/x-www-form-urlencoded
0095:
=> Send data, 11 bytes (0xb)
0000: hello=world
== Info: upload completely sent off: 11 out of 11 bytes

2、curl 显示 boundary 数据

curl -F "hello=world" -F file=@/etc/timezone https://const.net.cn/test/ --trace-ascii /dev/stdout

输出结果:

0000: POST /test/ HTTP/1.1 0016: Host: const.net.cn 002a: User-Agent:
curl/7.68.0 0043: Accept: / 0050: Content-Length: 313 0065:
Content-Type: multipart/form-data; boundary=-------------------- 00a5:
----584e7b2299bf345b 00bb:
=> Send SSL data, 5 bytes (0x5) 0000: ....J
=> Send SSL data, 1 bytes (0x1) 0000: .
=> Send data, 313 bytes (0x139) 0000: --------------------------584e7b2299bf345b 002c: Content-Disposition: form-data; name="hello" 005a: 005c: world 0063:
--------------------------584e7b2299bf345b 008f: Content-Disposition: form-data; name="file"; filename="timezone" 00d1: Content-Type:
application/octet-stream 00f9: 00fb: Asia/Shanghai. 010b:
--------------------------584e7b2299bf345b--
== Info: We are completely uploaded and fine

1. Go run shell script or Go run shell string

curl := "curl -v https://const.net.cn"
cmd := exec.Command("bash", "-c", curl)
stdout, err := cmd.CombinedOutput()
if err != nil {
    fmt.Println("curl " + err.Error())
    return
}
fmt.Println(string(stdout)) 

2.输出结果

  • Uses proxy env variable https_proxy == 'socks5://127.0.0.1:1080'
  • Trying 127.0.0.1:1080...
  • TCP_NODELAY set % Total % Received % Xferd Average Speed Time Time Time Current

                                 Dload  Upload   Total   Spent    Left  Speed   0     0    0     0    0     0      0      0 --:--:-- --:--:--

    --:--:-- 0

  • SOCKS5 communication to const.net.cn:443
  • SOCKS5 connect to IPv4 43.129.233.128:443 (locally resolved)
  • SOCKS5 request granted.
  • Connected to 127.0.0.1 (127.0.0.1) port 1080 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data]
  • TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
  • TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data]
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [25 bytes data]
  • TLSv1.3 (IN), TLS handshake, Certificate (11): { [2780 bytes data]
  • TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data]
  • TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data]
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data]
  • TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data]
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: CN=const.net.cn
  • start date: May 19 00:00:00 2021 GMT
  • expire date: May 18 23:59:59 2022 GMT
  • subjectAltName: host "const.net.cn" matched cert's "const.net.cn"
  • issuer: C=CN; O=TrustAsia Technologies, Inc.; OU=Domain Validated SSL; CN=TrustAsia TLS RSA CA
  • SSL certificate verify ok. } [5 bytes data]

GET / HTTP/1.1
Host: const.net.cn
User-Agent: curl/7.68.0
Accept: /

......