标签 curl 下的文章

“”

1、查看curl与时间相关的选项

curl -h |grep time

有用的输出

--connect-timeout <seconds> Maximum time allowed for connection
--expect100-timeout <seconds> How long to wait for 100-continue
--keepalive-time <seconds> Interval time for keepalive probes
-m, --max-time <seconds> Maximum time allowed for the transfer
-y, --speed-time <seconds> Trigger 'speed-limit' abort after this time

2、控制连接超时--connect-timeout 5

curl -v --connect-timeout 1 https://const.net.cn

当出现连接超时时,会提示如下信息

curl: (28) Connection timed out after 1001 milliseconds

3、总超时时间--max-time
 我们希望curl程序在运行一段时间后,不管有没有执行完成都能正常退出。就用这个max-time

curl -v -m 60 https://const.net.cn

4、--speed-time 的用法
speed-time是用来根据速率来控制连接是否断开的。
用法:

curl -o const.log "https://const.net.cn" --speed-time 5 --speed-limit 1

是说将url内容保存到const.log中, 如果传输速度小于1字节/秒的状态持续5秒,该连接就会断开,并退出程序。

1、curl -w参数

curl -h |grep "write-out"

-w, --write-out <format> Use output FORMAT after completion

2、显示所有时间示例

curl -o /dev/null -s -w "time_namelookup:%{time_namelookup}\ntime_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n"  "https://const.net.cn/"

输出:

time_namelookup:0.058759
time_connect: 0.069912
time_starttransfer: 0.147355
time_total: 0.167685

3、参数说明

time_namelookup: DNS 服务器域名解析的时间(以上时间单位都是s)
time_connect: client 发出请求,到 c/s 建立TCP 的时间;里面包括 DNS 解析的时间
time_starttransfer: client 发出请求;到 server 响应发出第一个字节开始的时间;包括前面的2个时间
time_total: 从请求发起到链接关闭的总耗时

4、官方英文说明

time_appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
time_connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
time_namelookup: The time, in seconds, it took from the start until the name resolving was completed.
time_pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
time_redirect: The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
time_starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
time_total: The total time, in seconds, that the full operation lasted.

5、格式可以使用字符串,格式化文件以及标准输入

The format can be specified as a literal "string", or you can have curl read the format from a file with "@filename" and to tell curl to read the format from stdin you write "@-".

6、使用格式文件示例

cat curl-format.txt

   time_namelookup:%{time_namelookup}\n
      time_connect:%{time_connect}\n
   time_appconnect:%{time_appconnect}\n
     time_redirect:%{time_redirect}\n    
  time_pretransfer:%{time_pretransfer}\n  
time_starttransfer:%{time_starttransfer}\n
------------------------------\n
        time_total:  %{time_total}\n

使用示例:

curl -o /dev/null -s -w "@curl-format.txt"  "https://const.net.cn/"

输出结果:

   time_namelookup:0.165160
      time_connect:0.240436
   time_appconnect:0.326137
     time_redirect:0.000000
  time_pretransfer:0.326211
time_starttransfer:0.412927
------------------------------
        time_total:  0.486244

代码如下:

/* Example code that uploads a file name 'foo' to a remote script that accepts
 * "HTML form based" (as described in RFC1738) uploads using HTTP POST.
 *
 * The imaginary form we will fill in looks like:
 *
 * <form method="post" enctype="multipart/form-data" action="examplepost.cgi">
 * Enter file: <input type="file" name="sendfile" size="40">
 * Enter file name: <input type="text" name="filename" size="30">
 * <input type="submit" value="send" name="submit">
 * </form>
 *
 */
 
#include <stdio.h>
#include <string.h>
 
#include <curl/curl.h>
 
int main(int argc, char *argv[])
{
  CURL *curl;
  CURLcode res;
 
  curl_mime *form = NULL;
  curl_mimepart *field = NULL;
  struct curl_slist *headerlist = NULL;
  static const char buf[] = "Expect:";
 
  curl_global_init(CURL_GLOBAL_ALL);
 
  curl = curl_easy_init();
  if(curl) {
    /* Create the form */
    form = curl_mime_init(curl);
 
    /* Fill in the file upload field */
    field = curl_mime_addpart(form);
    curl_mime_name(field, "sendfile");
    curl_mime_filedata(field, "postit2.c");
 
    /* Fill in the filename field */
    field = curl_mime_addpart(form);
    curl_mime_name(field, "filename");
    curl_mime_data(field, "postit2.c", CURL_ZERO_TERMINATED);
 
    /* Fill in the submit field too, even if this is rarely needed */
    field = curl_mime_addpart(form);
    curl_mime_name(field, "submit");
    curl_mime_data(field, "send", CURL_ZERO_TERMINATED);
 
    /* initialize custom header list (stating that Expect: 100-continue is not
       wanted */
    headerlist = curl_slist_append(headerlist, buf);
    /* what URL that receives this POST */
    curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/examplepost.cgi");
    if((argc == 2) && (!strcmp(argv[1], "noexpectheader")))
      /* only disable 100-continue header if explicitly requested */
      curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
    curl_easy_setopt(curl, CURLOPT_MIMEPOST, form);
 
    /* Perform the request, res will get the return code */
    res = curl_easy_perform(curl);
    /* Check for errors */
    if(res != CURLE_OK)
      fprintf(stderr, "curl_easy_perform() failed: %s\n",
              curl_easy_strerror(res));
 
    /* always cleanup */
    curl_easy_cleanup(curl);
 
    /* then cleanup the form */
    curl_mime_free(form);
    /* free slist */
    curl_slist_free_all(headerlist);
  }
  return 0;
}

一些说明,以及为什么要disable 100-continue

1)Expect: 100-continue的来龙去脉:

HTTP/1.1 协议里设计 100 (Continue) HTTP 状态码的的目的是,在客户端发送 Request Message 之前,HTTP/1.1 协议允许客户端先判定服务器是否愿意接受客户端发来的消息主体(基于 Request Headers)。

即, Client 和 Server 在Post (较大)数据之前,允许双方“握手”,如果匹配上了,Client 才开始发送(较大)数据。

这么做的原因是,如果客户端直接发送请求数据,但是服务器又将该请求拒绝的话,这种行为将带来很大的资源开销。

协议对 HTTP/1.1 clients 的要求是:

如果 client 预期等待“100-continue”的应答,那么它发的请求必须包含一个 " Expect: 100-continue" 的头域!

2)libcurl 发送大于1024字节数据时启用Expect:100-continue特性:

这也就是 Laruence 在 2011 年撰文所写的:

在使用 curl 做 POST 的时候,当要POST 的数据大于 1024 字节的时候,curl 并不会直接就发起 POST 请求,而是会分为两步:

  1. 发送一个请求,包含一个 "Expect:100-continue" 头域,询问 Server 是否愿意接收数据;
  2. 接收到 Server 返回的100-continue 应答以后,才把数据 POST 给Server;

这是 libcurl 的行为。

第一, libcurl在发送大于1024 字节的 POST 请求时采用了这种方法,但是相对的,它会引起请求延迟的加大。

第二,并不是所有的 web server 都能正确处理并应答“100-continue”,比如 lighttpd,就会返回417”Expectation Failed “,造成请求逻辑出错。

( 注1:lighttpd 1.4 版本有此严重问题,于1.5版本修复。注2:Resin 于 3.0.5 版本增加了对 Expect: 100-continue 的支持。)

3)PHP Curl-library 可以主动封禁此特性:

有人在 PHP手册::curl_setopt 下留言说:

PHP curl 遵从 libcurl 的特性。由于不是所有 web servers 都支持这个特性,所以会产生各种各样的错误。如果你遇到了,可以用下面的命令封禁"Expect"头域:

<?php
    curl_setopt($ch, CURLOPT_HTTPHEADER,array('Expect:'));
?>

如果一个连接服务器不断开,则curl_easy_perform会一直阻塞。这个时候就需要关闭或断开连接的方法。

常用的方法是设置连接超时,或者总超时,本处使用原始socket close

CURLINFO_ACTIVESOCKET - get the active socket

Example

CURL *curl = curl_easy_init();
if(curl) {
  curl_socket_t sockfd;
  curl_easy_setopt(curl, CURLOPT_URL, "https://example.com");
 
  /* Do not do the transfer - only connect to host */
  curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1L);
  res = curl_easy_perform(curl);
 
  /* Extract the socket from the curl handle */
  res = curl_easy_getinfo(curl, CURLINFO_ACTIVESOCKET, &sockfd);
 
  if(res != CURLE_OK) {
    printf("Error: %s\n", curl_easy_strerror(res));
    return 1;
  }
}

另一个获取sock的方法

curl_easy_setopt(curl, CURLOPT_OPENSOCKETFUNCTION, my_opensocketfunc);

curl_socket_t my_opensocketfunc(void *clientp,curlsocktype purpose,struct curl_sockaddr *address)
{
    curl_socket_t sock=socket(address->family, address->socktype, address->protocol);
    printf("my_opensocketfunc sock = %d\n", sock);
    return  sock;
}

超时的配置见
CURLOPT_CONNECTTIMEOUT

Timeout for the connection phase. See CURLOPT_CONNECTTIMEOUT

CURLOPT_CONNECTTIMEOUT_MS

Millisecond timeout for the connection phase. See CURLOPT_CONNECTTIMEOUT_MS

CURLOPT_TIMEOUT

Timeout for the entire request. See CURLOPT_TIMEOUT

cURL百科
cURL是一个开源项目,主要的产品是curl(命令行工具)和libcurl(C语言的API库),两者功能均是:基于网络协议,对指定URL进行网络传输。

cURL涉及是任何网络协议传输,不涉及对具体数据的具体处理。curl支持的通信协议有DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SBMS, SMTP, SMTPS, TELNET 和TFTP。

cURL 官方网站

https://curl.se/

cURL 例子
cURL 简单模式:

$ curl http://example.com

cURL 详细(verbose)模式:

$ curl --verbose http://example.com
$ curl -v http://example.com

cURL 下载(output):

$ curl --output output.html http://example.com/
$ curl -o output.html http://example.com/

cURL 重定向:(curl默认不会重定向)

$ curl --location output.html http://example.com/
$ curl -L output.html http://example.com/

使用 curl 在命令行获取网站的响应时间

curl -L -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" https://example.com/

结果:

<!doctype html>
<html>
...
</html>
time_namelookup: 0.020519
time_connect: 0.187338
time_appconnect: 0.546461
time_pretransfer: 0.546661
time_redirect: 0.000000
time_starttransfer: 0.710620
time_total: 0.710879

time_appconnect and time_connect time tells us how much time is spent in SSL/TLS handshake. For a cleartext connection without SSL/TLS, this number is reported as zero.
Also note that time_redirect is zero in both outputs above. That is because no redirection occurs while visiting example.com.

curl post file as body
curl POST 二进制文件
strip new lines

curl --data "@/path/to/filename" http://...

keep new lines

curl --data-binary "@/path/to/filename" http://...