PHP也玩并发,巧用curl 并发减少后端访问时间

 

说明:本人源自3篇博文

http://blog.csdn.net/zuiaituantuan/article/details/7048782

首先,先了解下 php中的curl多线程函数:

# curl_multi_add_handle
# curl_multi_close
# curl_multi_exec
# curl_multi_getcontent
# curl_multi_info_read
# curl_multi_init
# curl_multi_remove_handle
# curl_multi_select

一般来说,想到要用这些函数时,目的显然应该是要同时请求多个url,而不是一个一个依次请求,否则不如自己循环去调curl_exec好了。

步骤总结如下:

第一步:调用curl_multi_init
第二步:循环调用curl_multi_add_handle
这一步需要注意的是,curl_multi_add_handle的第二个参数是由curl_init而来的子handle。
第三步:持续调用curl_multi_exec
第四步:根据需要循环调用curl_multi_getcontent获取结果
第五步:调用curl_multi_remove_handle,并为每个字handle调用curl_close
第六步:调用curl_multi_close

这里有一个网上找的简单例子,其作者称为dirty的例子,(稍后我会说明为何dirty):
/*
Here's a quick and dirty example for curl-multi from PHP, tested on PHP 5.0.0RC1 CLI / FreeBSD 5.2.1
*/

$connomains = array(
"http://www.cnn.com/",
"http://www.canada.com/",
"http://www.yahoo.com/"
);

$mh = curl_multi_init();

foreach ($connomains as $i => $url) {
     $conn[$i]=curl_init($url);
      curl_setopt($conn[$i],CURLOPT_RETURNTRANSFER,1);
      curl_multi_add_handle ($mh,$conn[$i]);
}

do { $n=curl_multi_exec($mh,$active); } while ($active);

foreach ($connomains as $i => $url) {
      $res[$i]=curl_multi_getcontent($conn[$i]);
      curl_close($conn[$i]);
}

print_r($res);

 

整个使用过程差不多就是这样,但是,这个简单代码有个致命弱点,就是在do循环的那段,在整个url请求期间是个死循环,它会轻易导致CPU占用100%。

现在我们来改进它,这里要用到一个几乎没有任何文档的函数curl_multi_select了,虽然C的curl库对select有说明,但是,php里的接口和用法确与C中有不同。

把上面do的那段改成下面这样:
                do {
                        $mrc = curl_multi_exec($mh,$active);
                } while ($mrc == CURLM_CALL_MULTI_PERFORM);
                while ($active and $mrc == CURLM_OK) {
                        if (curl_multi_select($mh) != -1) {
                                do {
                                        $mrc = curl_multi_exec($mh, $active);
                                } while ($mrc == CURLM_CALL_MULTI_PERFORM);
                        }
                }

因为$active要等全部url数据接受完毕才变成false,所以这里用到了curl_multi_exec的返回值判断是否还有数据,当有数据的时候就不停调用curl_multi_exec,暂时没有数据就进入select阶段,新数据一来就可以被唤醒继续执行。这里的好处就是CPU的无谓消耗没有了。

另外:还有一些细节的地方可能有时候要遇到:

控制每一个请求的超时时间,在curl_multi_add_handle之前通过curl_setopt去做:
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);

判断是否超时了或者其他错误,在curl_multi_getcontent之前用:curl_error($conn[$i]);


这里我只是简单使用上述的dirty的例子(足够用了,并未发现cpu使用100%的情况)。

对“看点”(kandian.com)某一接口模拟并发,功能是向 memcache中读数据并写入数据。因为保密关系,相关数据及结果就不贴出了。

模拟了3次,第一次10线程同时请求1000次,第二次,100线程同时请求1000次,第三次,1000线程同时请求100次(已经相当费劲了,不敢在设置超过1000的多线程)。

看来curl多线程模拟并发还是有一定局限的。

另外还怀疑,可能会因为多线程延迟带来结果的大误差,对比数据发现。在初始化和set所用时间出入不大,差别处在get方法,因此可简单排除这点~~~

 

 

 

http://log.dongsheng.org/2008/07/16/curl-multiple-handlers/

通常情况下 PHP 中的 cURL 是阻塞运行的,就是说创建一个 cURL 请求以后必须等它执行成功或者超时才会执行下一个请求,curl_multi_* 系列函数使并发访问成功可能,PHP 文档对这个函数的介绍不太详细,用法如下:

 

$requests = array('http://www.baidu.com', 'http://www.google.com');
$main    = curl_multi_init();
$results = array();
$errors  = array();
$info = array();
$count = count($requests);
for($i = 0; $i < $count; $i++) 
{  
$handles[$i] = curl_init($requests[$i]);  
var_dump($requests[$i]);  
curl_setopt($handles[$i], CURLOPT_URL, $requests[$i]);  
curl_setopt($handles[$i], CURLOPT_RETURNTRANSFER, 1);  
curl_multi_add_handle($main, $handles[$i]);
}
$running = 0; 
do {  
curl_multi_exec($main, $running);
} 
while($running > 0); 
for($i = 0; $i < $count; $i++)
{  $results[] = curl_multi_getcontent($handles[$i]);  
$errors[]  = curl_error($handles[$i]);  
$info[]    = curl_getinfo($handles[$i]);  
curl_multi_remove_handle($main, $handles[$i]);
}
curl_multi_close($main);
var_dump($results);
var_dump($errors);
var_dump($info);  


 

http://www.searchtb.com/2010/12/using-multicurl-to-improve-performance.html

前言:在我们平时的程序中难免出现同时访问几个接口的情况,平时我们用curl进行访问的时候,一般都是单个、顺序访问,假如有3个接口,每个接口耗时500毫秒那么我们三个接口就要花费1500毫秒了,这个问题太头疼了严重影响了页面访问速度,有没有可能并发访问来提高速度呢?今天就简单的说一下,利用curl并发来提高页面访问速度,希望大家多指导。1、老的curl访问方式以及耗时统计

<?php function curl_fetch($url, $timeout=3){     
$ch = curl_init();     
curl_setopt($ch, CURLOPT_URL, $url);     
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);     
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);     
$data = curl_exec($ch);     
$errno = curl_errno($ch);     
if ($errno>0) {         
$data = false;     
}     
curl_close($ch);     
return $data; 
} 
function microtime_float() 
{    
list($usec, $sec) = explode(" ", microtime());    
return ((float)$usec + (float)$sec); 
} 
$url_arr=array(      
"taobao"=>"http://www.taobao.com",      
"sohu"=>"http://www.sohu.com",      
"sina"=>"http://www.sina.com.cn",      
);  
$time_start = microtime_float();  
$data=array();  
foreach ($url_arr as $key=>$val)  
{      
$data[$key]=curl_fetch($val);  
}  
$time_end = microtime_float();  
$time = $time_end - $time_start;  
echo "耗时:{$time}"; 
?> 

耗时:0.614秒
2、curl并发访问方式以及耗时统计
<?php 
function curl_multi_fetch($urlarr=array()){     
$result=$res=$ch=array();     
$nch = 0;     
$mh = curl_multi_init();     
foreach ($urlarr as $nk => $url) {         
$timeout=2;         
$ch[$nch] = curl_init();         
curl_setopt_array($ch[$nch], array(         
CURLOPT_URL => $url,         
CURLOPT_HEADER => false,         
CURLOPT_RETURNTRANSFER => true,         
CURLOPT_TIMEOUT => $timeout,         
));         
curl_multi_add_handle($mh, $ch[$nch]);         
++$nch;     }     
/* wait for performing request */    
do {         
$mrc = curl_multi_exec($mh, $running);     
} while (CURLM_CALL_MULTI_PERFORM == $mrc);       
while ($running && $mrc == CURLM_OK) {         
// wait for network         
if (curl_multi_select($mh, 0.5) > -1) {             
// pull in new data;             
do {                 
$mrc = curl_multi_exec($mh, $running);             
} while (CURLM_CALL_MULTI_PERFORM == $mrc);         
}     
}       
if ($mrc != CURLM_OK) {         
error_log("CURL Data Error");     
}       
/* get data */    
$nch = 0;     
foreach ($urlarr as $moudle=>$node) {         
if (($err = curl_error($ch[$nch])) == '') {             
$res[$nch]=curl_multi_getcontent($ch[$nch]);             $result[$moudle]=$res[$nch];         }         
else        
{             
error_log("curl error");         
}         
curl_multi_remove_handle($mh,$ch[$nch]);         
curl_close($ch[$nch]);         
++$nch;     
}     
curl_multi_close($mh);     
return  $result; 
} 
$url_arr=array(      
"taobao"=>"http://www.taobao.com",      
"sohu"=>"http://www.sohu.com",      
"sina"=>"http://www.sina.com.cn",      
); 
function microtime_float() 
{    
list($usec, $sec) = explode(" ", microtime());    
return ((float)$usec + (float)$sec); 
} 
$time_start = microtime_float(); 
$data=curl_multi_fetch($url_arr); 
$time_end = microtime_float(); 
$time = $time_end - $time_start; 
echo "耗时:{$time}"; 
?> 

耗时:0.316秒
帅气吧整个页面访问后端接口的时间节省了一半
3、curl相关参数
来自:http://cn2.php.net/manual/en/ref.curl.php
curl_close — Close a cURL session
curl_copy_handle — Copy a cURL handle along with all of its preferences
curl_errno — Return the last error number
curl_error — Return a string containing the last error for the current session
curl_exec — Perform a cURL session
curl_getinfo — Get information regarding a specific transfer
curl_init — Initialize a cURL session
curl_multi_add_handle — Add a normal cURL handle to a cURL multi handle
curl_multi_close — Close a set of cURL handles
curl_multi_exec — Run the sub-connections of the current cURL handle
curl_multi_getcontent — Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read — Get information about the current transfers
curl_multi_init — Returns a new cURL multi handle
curl_multi_remove_handle — Remove a multi handle from a set of cURL handles
curl_multi_select — Wait for activity on any curl_multi connection
curl_setopt_array — Set multiple options for a cURL transfer
curl_setopt — Set an option for a cURL transfer
curl_version — Gets cURL version information

前端开发中的性能那点事(三)php的opcode缓存

前端开发中的性能那点事(一)巧用xdebug

发布了1595 篇原创文章 · 获赞 1155 · 访问量 1212万+
展开阅读全文

PHP cURL多处理性能比普通cURL

05-23

<div class="post-text" itemprop="text"> <p>I am trying to use cURL curl_multi_init() as a means to speed up requests. What I am experiencing though is that the request with the multi handles initinally take longer than the same request made with curl_init().</p> <p>Below is an example of two identical requests. <strong>Consistently</strong> the multi request takes about 4/5 times longer than the single request. The cURL options in both requests are identical. In this example the multi cURL only makes one request.</p> <p>Additional info: PHP 5.3.3/cURL 7.21.0, Windows server 2008/IIS 7x.</p> <p>I am totally flabbergasted about what could be causing this slow response. The request is being made to a server which resides in the same network, years of usage of the normal cURL has given me the experience that this kind of request, to this particular backend with non multi handling, take on average between 0.2/0.3 seconds.</p> <p>The question is: what could be causing this slowness of the multi cURL request.</p> <p>Below are the results of a test, two requests, one being done with curl_init(), the other with curl_multi_init (both in the same script). Notice the cURL info, the requests being exactly the same in terms of header_size, request_size, size_upload, size_download and download_content_length.</p> <p>Test with normal cURL:</p> <pre><code>Array ( [url] => http://myurl.com [content_type] => text/xml;charset=UTF-8 [http_code] => 200 [header_size] => 261 [request_size] => 312 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.203 [namelookup_time] => 0 [connect_time] => 0.015 [pretransfer_time] => 0.015 [size_upload] => 174 [size_download] => 236 [speed_download] => 1162 [speed_upload] => 857 [download_content_length] => 236 [upload_content_length] => 0 [starttransfer_time] => 0.203 [redirect_time] => 0 ) </code></pre> <p>Test with multi cURL (notice the speed upload/download is lower, connect and name lookup time higher):</p> <pre><code>Array ( [client] => Array ( [url] => http://myurl.com [content_type] => text/xml;charset=UTF-8 [http_code] => 200 [header_size] => 261 [request_size] => 312 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 1.047 [namelookup_time] => 0.61 [connect_time] => 0.61 [pretransfer_time] => 0.61 [size_upload] => 174 [size_download] => 236 [speed_download] => 225 [speed_upload] => 166 [download_content_length] => 236 [upload_content_length] => 0 [starttransfer_time] => 1.047 [redirect_time] => 0 [multi_handle_info] => Array ( [msg] => 1 [result] => 0 [handle] => Resource id #5 ) ) ) </code></pre> <p>Example of the code for the multi cURL (options are same as in the code for normal cURL):</p> <pre><code>$curl = array(); $result = array(); $mh = curl_multi_init(); foreach (array_keys($queries) as $id) { $curl[$id] = curl_init(); curl_setopt($curl[$id], CURLOPT_URL,$queries[$id]['url']); curl_setopt($curl[$id], CURLOPT_RETURNTRANSFER,1); curl_setopt($curl[$id], CURLOPT_CONNECTTIMEOUT,60); curl_setopt($curl[$id], CURLOPT_DNS_CACHE_TIMEOUT,3600); curl_setopt($curl[$id], CURLOPT_TIMEOUT, 240); curl_setopt($curl[$id], CURLOPT_POSTFIELDS, $queries[$id]['post']); curl_multi_add_handle($mh, $curl[$id]); } $running = null; do { curl_multi_exec($mh, $running); } while($running > 0); // get content and remove handles foreach($curl as $id => $c) { $result[$id] = curl_multi_getcontent($c); curl_multi_remove_handle($mh, $c); } </code></pre> </div> 问答

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 深蓝海洋 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览