<address id="bztp3"><nobr id="bztp3"></nobr></address>

      <sub id="bztp3"><listing id="bztp3"><menuitem id="bztp3"></menuitem></listing></sub>

        <form id="bztp3"></form>

        <address id="bztp3"></address>
        <sub id="bztp3"></sub>

          <form id="bztp3"></form>

              Linux網絡編程“驚群”問題總結

              1、前言

              我從事Linux系統下網絡開發將近4年了,經常還是遇到一些問題,只是知其然而不知其所以然,有時候和其他人交流,搞得非常尷尬。如今計算機都是多核了,網絡編程框架也逐步豐富多了,我所知道的有多進程、多線程、異步事件驅動常用的三種模型。最經典的模型就是Nginx中所用的Master-Worker多進程異步驅動模型。今天和大家一起討論一下網絡開發中遇到的“驚群”現象。之前只是聽說過這個現象,網上查資料也了解了基本概念,在實際的工作中還真沒有遇到過。今天周末,結合自己的理解和網上的資料,徹底將“驚群”弄明白。需要弄清楚如下幾個問題:

              (1)什么是“驚群”,會產生什么問題?

              (2)“驚群”的現象怎么用代碼模擬出來?

              (3)如何處理“驚群”問題,處理“驚群”后的現象又是怎么樣呢?

              2、何為驚群

              如今網絡編程中經常用到多進程或多線程模型,大概的思路是父進程創建socket,bind、listen后,通過fork創建多個子進程,每個子進程繼承了父進程的socket,調用accpet開始監聽等待網絡連接。這個時候有多個進程同時等待網絡的連接事件,當這個事件發生時,這些進程被同時喚醒,就是“驚群”。這樣會導致什么問題呢?我們知道進程被喚醒,需要進行內核重新調度,這樣每個進程同時去響應這一個事件,而最終只有一個進程能處理事件成功,其他的進程在處理該事件失敗后重新休眠或其他。網絡模型如下圖所示:

              Linux網絡編程“驚群”問題總結

              簡而言之,驚群現象(thundering herd)就是當多個進程和線程在同時阻塞等待同一個事件時,如果這個事件發生,會喚醒所有的進程,但最終只可能有一個進程/線程對該事件進行處理,其他進程/線程會在失敗后重新休眠,這種性能浪費就是驚群。

              3、編碼模擬“驚群”現象

              我們已經知道了“驚群”是怎么回事,那么就按照上面的圖編碼實現看一下效果。我嘗試使用多進程模型,創建一個父進程綁定一個端口監聽socket,然后fork出多個子進程,子進程們開始循環處理(比如accept)這個socket。測試代碼如下所示:

              ?1 #include <stdio.h>

              2 #include <unistd.h>

              3 #include <sys/types.h>

              4 #include <sys/socket.h>

              5 #include <netinet/in.h>

              6 #include <arpa/inet.h>

              7 #include <assert.h>

              8 #include <sys/wait.h>

              9 #include <string.h>

              10 #include <errno.h>

              11

              12 #define IP ? “127.0.0.1”

              13 #define PORT ?8888

              14 #define WORKER 4

              15

              16 int worker(int listenfd, int i)

              17 {

              18 ? ? while (1) {

              19 ? ? ? ? printf(“I am worker %d, begin to accept connection.\n”, i);

              20 ? ? ? ? struct sockaddr_in client_addr;

              21 ? ? ? ? socklen_t client_addrlen = sizeof( client_addr );

              22 ? ? ? ? int connfd = accept( listenfd, ( struct sockaddr* )&client_addr, &client_addrlen );

              23 ? ? ? ? if (connfd != -1) {

              24 ? ? ? ? ? ? printf(“worker %d accept a connection success.\t”, i);

              25 ? ? ? ? ? ? printf(“ip :%s\t”,inet_ntoa(client_addr.sin_addr));

              26 ? ? ? ? ? ? printf(“port: %d \n”,client_addr.sin_port);

              27 ? ? ? ? } else {

              28 ? ? ? ? ? ? printf(“worker %d accept a connection failed,error:%s”, i, strerror(errno));

              close(connfd);

              29 ? ? ? ? }

              30 ? ? }

              31 ? ? return 0;

              32 }

              33

              34 int main()

              35 {

              36 ? ? int i = 0;

              37 ? ? struct sockaddr_in address;

              38 ? ? bzero(&address, sizeof(address));

              39 ? ? address.sin_family = AF_INET;

              40 ? ? inet_pton( AF_INET, IP, &address.sin_addr);

              41 ? ? address.sin_port = htons(PORT);

              42 ? ? int listenfd = socket(PF_INET, SOCK_STREAM, 0);

              43 ? ? assert(listenfd >= 0);

              44

              45 ? ? int ret = bind(listenfd, (struct sockaddr*)&address, sizeof(address));

              46 ? ? assert(ret != -1);

              47

              48 ? ? ret = listen(listenfd, 5);

              49 ? ? assert(ret != -1);

              50

              51 ? ? for (i = 0; i < WORKER; i++) {

              52 ? ? ? ? printf(“Create worker %d\n”, i+1);

              53 ? ? ? ? pid_t pid = fork();

              54 ? ? ? ? /*child ?process */

              55 ? ? ? ? if (pid == 0) {

              56 ? ? ? ? ? ? worker(listenfd, i);

              57 ? ? ? ? }

              58

              59 ? ? ? ? if (pid < 0) {

              60 ? ? ? ? ? ? printf(“fork error”);

              61 ? ? ? ? }

              62 ? ? }

              63

              64 ? ? /*wait child process*/

              65 ? ? int status;

              66 ? ? wait(&status);

              67 ? ? return 0;

              68 }

              編譯執行,在本機上使用telnet 127.0.0.1 8888測試,結果如下所示:

              Linux網絡編程“驚群”問題總結

              按照“驚群”現象,期望結果應該是4個子進程都會accpet到請求,其中只有一個成功,另外三個失敗的情況。而實際的結果顯示,父進程開始創建4個子進程,每個子進程開始等待accept連接。當telnet連接來的時候,只有worker2 子進程accpet到請求,而其他的三個進程并沒有接收到請求。

              這是什么原因呢?難道驚群現象是假的嗎?于是趕緊google查一下,驚群到底是怎么出現的。

              其實在Linux2.6版本以后,內核內核已經解決了accept()函數的“驚群”問題,大概的處理方式就是,當內核接收到一個客戶連接后,?只會喚醒等待隊列上的第一個進程或線程?。所以,如果服務器采用accept阻塞調用方式,在最新的Linux系統上,已經沒有“驚群”的問題了。

              但是,對于實際工程中常見的服務器程序,大都使用select、poll或epoll機制,此時,服務器不是阻塞在accept,而是阻塞在select、poll或epoll_wait,這種情況下的“驚群”仍然需要考慮。接下來以epoll為例分析:

              使用epoll非阻塞實現代碼如下所示:

              ??1 #include <sys/types.h>

              2 #include <sys/socket.h>

              3 #include <sys/epoll.h>

              4 #include <netdb.h>

              5 #include <string.h>

              6 #include <stdio.h>

              7 #include <unistd.h>

              8 #include <fcntl.h>

              9 #include <stdlib.h>

              10 #include <errno.h>

              11 #include <sys/wait.h>

              12 #include <unistd.h>

              13

              14 #define IP ? “127.0.0.1”

              15 #define PORT ?8888

              16 #define PROCESS_NUM 4

              17 #define MAXEVENTS 64

              18

              19 static int create_and_bind ()

              20 {

              21 ? ? int fd = socket(PF_INET, SOCK_STREAM, 0);

              22 ? ? struct sockaddr_in serveraddr;

              23 ? ? serveraddr.sin_family = AF_INET;

              24 ? ? inet_pton( AF_INET, IP, &serveraddr.sin_addr);

              25 ? ? serveraddr.sin_port = htons(PORT);

              26 ? ? bind(fd, (struct sockaddr*)&serveraddr, sizeof(serveraddr));

              27 ? ? return fd;

              28 }

              29

              30 static int make_socket_non_blocking (int sfd)

              31 {

              32 ? ? int flags, s;

              33 ? ? flags = fcntl (sfd, F_GETFL, 0);

              34 ? ? if (flags == -1) {

              35 ? ? ? ? perror (“fcntl”);

              36 ? ? ? ? return -1;

              37 ? ? }

              38 ? ? flags |= O_NONBLOCK;

              39 ? ? s = fcntl (sfd, F_SETFL, flags);

              40 ? ? if (s == -1) {

              41 ? ? ? ? perror (“fcntl”);

              42 ? ? ? ? return -1;

              43 ? ? }

              44 ? ? return 0;

              45 }

              46

              47 void worker(int sfd, int efd, struct epoll_event *events, int k) {

              48 ? ? /* The event loop */

              49 ? ? while (1) {

              50 ? ? ? ? int n, i;

              51 ? ? ? ? n = epoll_wait(efd, events, MAXEVENTS, -1);

              52 ? ? ? ? printf(“worker ?%d return from epoll_wait!\n”, k);

              53 ? ? ? ? for (i = 0; i < n; i++) {

              54 ? ? ? ? ? ? if ((events[i].events & EPOLLERR) || (events[i].events & EPOLLHUP) || (!(events[i].events &EPOLLIN))) {

              55 ? ? ? ? ? ? ? ? /* An error has occured on this fd, or the socket is not ready for reading (why were we notified then?) */

              56 ? ? ? ? ? ? ? ? fprintf (stderr, “epoll error\n”);

              57 ? ? ? ? ? ? ? ? close (events[i].data.fd);

              58 ? ? ? ? ? ? ? ? continue;

              59 ? ? ? ? ? ? } else if (sfd == events[i].data.fd) {

              60 ? ? ? ? ? ? ? ? /* We have a notification on the listening socket, which means one or more incoming connections. */

              61 ? ? ? ? ? ? ? ? struct sockaddr in_addr;

              62 ? ? ? ? ? ? ? ? socklen_t in_len;

              63 ? ? ? ? ? ? ? ? int infd;

              64 ? ? ? ? ? ? ? ? char hbuf[NI_MAXHOST], sbuf[NI_MAXSERV];

              65 ? ? ? ? ? ? ? ? in_len = sizeof in_addr;

              66 ? ? ? ? ? ? ? ? infd = accept(sfd, ∈_addr, ∈_len);

              67 ? ? ? ? ? ? ? ? if (infd == -1) {

              68 ? ? ? ? ? ? ? ? ? ? printf(“worker %d accept failed!\n”, k);

              69 ? ? ? ? ? ? ? ? ? ? break;

              70 ? ? ? ? ? ? ? ? }

              71 ? ? ? ? ? ? ? ? printf(“worker %d accept successed!\n”, k);

              72 ? ? ? ? ? ? ? ? /* Make the incoming socket non-blocking and add it to the list of fds to monitor. */

              73 ? ? ? ? ? ? ? ? close(infd);

              74 ? ? ? ? ? ? }

              75 ? ? ? ? }

              76 ? ? }

              77 }

              78

              79 int main (int argc, char *argv[])

              80 {

              81 ? ? int sfd, s;

              82 ? ? int efd;

              83 ? ? struct epoll_event event;

              84 ? ? struct epoll_event *events;

              85 ? ? sfd = create_and_bind();

              86 ? ? if (sfd == -1) {

              87 ? ? ? ? abort ();

              88 ? ? }

              89 ? ? s = make_socket_non_blocking (sfd);

              90 ? ? if (s == -1) {

              91 ? ? ? ? abort ();

              92 ? ? }

              93 ? ? s = listen(sfd, SOMAXCONN);

              94 ? ? if (s == -1) {

              95 ? ? ? ? perror (“listen”);

              96 ? ? ? ? abort ();

              97 ? ? }

              98 ? ? efd = epoll_create(MAXEVENTS);

              99 ? ? if (efd == -1) {

              100 ? ? ? ? perror(“epoll_create”);

              101 ? ? ? ? abort();

              102 ? ? }

              103 ? ? event.data.fd = sfd;

              104 ? ? event.events = EPOLLIN;

              105 ? ? s = epoll_ctl(efd, EPOLL_CTL_ADD, sfd, &event);

              106 ? ? if (s == -1) {

              107 ? ? ? ? perror(“epoll_ctl”);

              108 ? ? ? ? abort();

              109 ? ? }

              110

              111 ? ? /* Buffer where events are returned */

              112 ? ? events = calloc(MAXEVENTS, sizeof event);

              113 ? ? int k;

              114 ? ? for(k = 0; k < PROCESS_NUM; k++) {

              115 ? ? ? ? printf(“Create worker %d\n”, k+1);

              116 ? ? ? ? int pid = fork();

              117 ? ? ? ? if(pid == 0) {

              118 ? ? ? ? ? ? worker(sfd, efd, events, k);

              119 ? ? ? ? }

              120 ? ? }

              121 ? ? int status;

              122 ? ? wait(&status);

              123 ? ? free (events);

              124 ? ? close (sfd);

              125 ? ? return EXIT_SUCCESS;

              126 }

              父進程中創建套接字,并設置為非阻塞,開始listen。然后fork出4個子進程,在worker中調用epoll_wait開始accpet連接。使用telnet測試結果如下:

              Linux網絡編程“驚群”問題總結

              從結果看出,與上面是一樣的,只有一個進程接收到連接,其他三個沒有收到,說明沒有發生驚群現象。這又是為什么呢?

              在早期的Linux版本中,內核對于阻塞在epoll_wait的進程,也是采用全部喚醒的機制,所以存在和accept相似的“驚群”問題。新版本的的解決方案也是?只會喚醒等待隊列上的第一個進程或線程?,所以,新版本Linux??部分的?解決了epoll的“驚群”問題。所謂?部分的?解決,意思就是:對于部分特殊場景,使用epoll機制,已經不存在“驚群”的問題了,但是對于大多數場景,epoll機制仍然存在“驚群”。

              epoll存在驚群的場景如下:在worker保持工作的狀態下,都會被喚醒,例如在epoll_wait后調用sleep一次。改寫woker函數如下:

              void worker(int sfd, int efd, struct epoll_event *events, int k) {

              /* The event loop */

              while (1) {

              int n, i;

              n = epoll_wait(efd, events, MAXEVENTS, -1);

              /*keep running*/

              sleep(2);

              printf(“worker ?%d return from epoll_wait!\n”, k);

              for (i = 0; i < n; i++) {

              if ((events[i].events & EPOLLERR) || (events[i].events & EPOLLHUP) || (!(events[i].events &EPOLLIN))) {

              /* An error has occured on this fd, or the socket is not ready for reading (why were we notified then?) */

              fprintf (stderr, “epoll error\n”);

              close (events[i].data.fd);

              continue;

              } else if (sfd == events[i].data.fd) {

              /* We have a notification on the listening socket, which means one or more incoming connections. */

              struct sockaddr in_addr;

              socklen_t in_len;

              int infd;

              char hbuf[NI_MAXHOST], sbuf[NI_MAXSERV];

              in_len = sizeof in_addr;

              infd = accept(sfd, ∈_addr, ∈_len);

              if (infd == -1) {

              printf(“worker %d accept failed,error:%s\n”, k, strerror(errno));

              break;

              }

              printf(“worker %d accept successed!\n”, k);

              /* Make the incoming socket non-blocking and add it to the list of fds to monitor. */

              close(infd);

              }

              }

              }

              }

              測試結果如下所示:

              Linux網絡編程“驚群”問題總結

              終于看到驚群現象的出現了。

              4、解決驚群問題

              Nginx中使用mutex互斥鎖解決這個問題,具體措施有使用全局互斥鎖,每個子進程在epoll_wait()之前先去申請鎖,申請到則繼續處理,獲取不到則等待,并設置了一個負載均衡的?算法?(當某一個子進程的任務量達到總設置量的7/8時,則不會再嘗試去申請鎖)來均衡各個進程的任務量。后面深入學習一下Nginx的驚群處理過程。

              相關新聞

              聯系我們

              400-080-6560

              在線咨詢:點擊這里給我發消息

              郵件:work@magedu.com

              工作時間:周一至周日,09:00-18:30

              QR code
              656彩票平台656彩票主页656彩票网站656彩票官网656彩票娱乐 www.97655g.com | www.7415kk.com | bet365ok04.com | www.50999s.com | y68com | www.573512.com | 444000kk.com | www.83033d.com | feicai0555.com | www.5xct.com | www.849558.com | www.345839.com | www.80977m.com | feicai0471.com | www.caipiao88a.com | 7893w9.com | www.0088hgbb.com | www.703097.com | 3049m.com | www.8722msc.com | 111122ff.com | 66287g.com | www.55717i.com | 44332007.com | www.456586.com | www.88zz.cc | 6423o.com | www.4521l.com | 4340m.com | www.39500l.com | www.xpj1111.cc | www.108097.com | www.508555.com | 1253-6.com | www.980879.com | ddcp9.com | www.86889.com | www.jsh955.com | b762.xyz | www.e3410.com | 2306c.com | www.763605.com | www.55422e.com | 7726bbb.com | www.ampj.com | www.8494e.com | 500000572.com | www.55060g.com | 9068dd.com | www.5854k.cc | www.xpj9780.com | 58802e.com | www.575477.com | 66300vip27.com | www.258768.com | www.79500p.com | pj200.cc | www.35918v.com | www.4963vv.com | b61653.com | www.797884.com | www.166tk.com | 444000yy.com | www.766898.com | www.hg1266.com | 9068mm.com | www.995773.com | www.hg6880.com | 26119a.com | www.50052h.com | www.1138js.com | hfgszc580.com | www.15355r.com | www.69111n.com | 6033i.com | www.97655n.com | www.70msc.com | 3404d.cc | www.akw4.com | www.97cp345.com | hjcp188.com | www.616971.com | www.370257.com | 112v.net | 937004.com | www.99552nn.com | 3424d.com | zhcp47.com | www.pj56ii.com | www.25288t.com | 861224.com | www.1368q.cc | www.97596.com | 预览0747f.com | xpj377c.com | www.189a56.com | www.xin888111.com | js05.app | www.501346.com | www.3116j.com | 6594ll.com | 1705a.com | www.33997.com | www.73800.com | www.450955.com | www.hg6136.com | 7555v.com | www.186793.com | www.f063801.com | www.7435z.com | 31325o.com | www.hy6262.com | www.dzcp6666.com | www.85657a.com | 59522w.com | www.904830.com | www.38138r.com | www.bet1600.com | 8547gg.com | www.55238f.com | www.3002229.com | 6261h.com | 22291166.com | www.20016.cc | www.65066dd.com | www.1115xj.com | 1407.cc | www.810709.com | www.35252c.com | www.388365.net | 5360cc.com | www.138037.com | www.766777.cc | www.1168g.com | 0296.com | www.817824.com | www.k3410.com | www.87680f.com | yh77333.com | 6hcc99.com | www.78949t.com | 32555g.com | www.81678b.com | www.jjjj55555.com | 22296vv.com | yd12308.com | www.846662.com | www.12136p.com | www.3xh3.com | 9149n.com | www.711448.com | www.313965.com | www.rrrr0063.com | aipin99.me | shen3322.com | 2381jj.com | www.8667o.com | www.55060b.com | www.09569t.com | 8827gg.com | 3522nn.cc | www.87668u.com | www.2875x.com | www.06386688.com | www.105242.com | www.63877v.com | 9994cc16.com | ahhsxj.com | www.108013.com | www.66095.cc | www.78680f.com | www.4516.com | www.32329193.com | 30006n.com | 3559tttt.com | nn500g.com | qq365d.com | www.c9337.com | 3651266.com | 52520524.com | 73567tg13.com | www.rcw8811.com | www.80065w.com | www.115527t.com | www.797009.com | www.777444n.com | LXYL350.com | 40033mmm.com | 3805.com | hg1698.com | www.263520.com | www.5441x.com | www.833bbb.com | www.f3065.com | www.bj448.com | www.pj8819.com | 3467r.com | 28288cc.com | 59759i.vip | www.717699.com | www.866509.com | www.53900g.com | www.js58111.com | www.5099ww.com | www.734555.com | www.a27229.com | 1305444.com | 32126t.net | j01234.com | 6033i.com | 5099ll.com | www.266991.com | www.946277.com | www.33112l.com | www.9192666.com | www.877565.com | www.71071n.com | www.dzh1133.com | www.6146f.com | www.67797d.com | www.81866s.com | wfcp666777.com | wz4008.com | 88772105.com | 623670.com | 32355.com | 66000.com | 365666.com | 840pj.com | yth12.net | 2078o.com | 20188i.com | www.093827.com | www.bet81365.com | blr135.com | 8036m.com | www.088186.com | www.606643.com | www.602465.com | www.755901.com | www.914077.com | www.hy5503.com | www.21033.com | 883399a.com | 08530004.com | 20054488.com | 923250.com | 1077ppp.com | 4052aa.com | zx1432.com | 2682hd.com | 1288gg.com | 6594xx.com | 2381zzz.com | 5002sss.com | 1145.com | 66136j.com | 21488app.com | sxpj62622.com | www.1869g.com | www.f32126.cc | www.88993d.com | www.rycp665.com | www.msc22.com | 4632233.com | 3482c.com | 78997.com | so77777.cc | 922545.com | www.pj9087.com | www.8344567.com | www.xpj1507.com | www.hg8052.com | www.304902.com | www.458633.com | www.789583.com | www.80065x.com | www.ban57.com | www.530751.com | 6hcc99.com | 2247pp.com | 88770076.com | jidu0.com | www.pj8777.com | www.4938l.com | www.4583.com | www.hg8kk.com | www.906777.com | www.26878k.com | www.658997.com | 69111q.com | 37a700.com | 02.bet | 3844rr.com | www.2090944.com | www.2997701.com | www.ac775.com | www.xpj8896.com | www.50026j.com | www.561518.com | 5005q.com | 22207s.com | 998h.cc | www.kb840.com | www.tm2818.com | www.5981r.com | www.893267.com | www.763227.com | 77333885.com | 5856zzz.com | 3136bb.com | www.101717.com | www.163a4.com | www.28891a.com | www.725898.com | zhcp68.com | j444000.com | P35pp.com | www.778l.com | www.6687c.com | www.2350h.com | www.39500o.com | 55331xs.com | 37570d.com | www.77888js.com | www.44vn777.com | www.50054c.com | www.196903.com | 1294h.com | 55uu8331.com | www.a3a444.cc | www.pjc11.vip | www.5xcj.com | 777.com | 599hainan.com | www.yh57788.com | www.rycp163.com | www.88266f.com | www.093388.com | xpjxianlu.com | www.xjs8893.com | www.299488.com | www.90wb.top | 139975.com | 4488pp.com | www.8694q.com | www.89677u.com | www.782w.com | 3mgmvvv.com | dz829.com | www.959msc.com | www.js89f.vip | www.731066.com | 53262u.com | 33hh8331.com | www.pj1314.com | www.vns887.com | www.609716.com | 889.net | www.pj0567.com | www.65066cc.com | www.073wy.com | www.9908598.com | www.7777hv.com | 2096.com | www.77764400.com | www.678js777.com | www.364077.com | 23599v.com | www.ylg29.com | www.81608t.com | sha162.com | 80368rr.com | www.356309.com | www.60007c.com | 1288cc.com | v5581.com | www.js6829.com | www.642414.com | 1h-9.cc | www.betwd2.com | www.71233g.com | 88850vv.com | www.86611p.com | www.17k0048.com | www.561518.com | 3788h.com | www.24071.com | www.3416q.com | 2595v.com | www.980143.com | www.0601z.com | 112d.net | www.383050.com | www.6939g.com | 28824n.com | www.9844ii.com | www.130256.com | 89439922.com | b5369.com | www.38775ii.com | 2613v.com | www.dfs994.com | www.8577.com | 65005m.com | www.hg2566.net | www.452.com | 4123001.com | www.256333.com | www.yun879.com | nnn40033.com | www.11004100.com | www.5xci.com | 23599t.com | www.304902.com | www.322424.com | 3379s.com | www.31361.com | 5598510.com | www.bet544.com | www.71233l.com | 11177381.com | www.79500a.com | www.8839p.com | 678.com | www.299366.com | www.675630.com | ude197.com | www.v1432.com | ly88888881.com | www.6807788.com | www.60123k.com | 1468xxx.com | www.5504v.com | www.177888.com | vns8305.com | www.79966.com | 2459c.com | www.hjdc48.com | www.327526.com | 8052.am | www.556303.com | 9479d4.com | www.sj52288.com | 16690055.com | www.dy6611.com | www.1333ac.com | hg123311.com | www.3399nsb.com | oo38648.cc | www.yham999.com | www.206883.com | www.js50b.com | www.49956t.com | 2618m.com | www.w84.com | 6175t.com | www.suncity758.com | www.99788o.com | www.77755545.com | www.cb3788.com | 22294433.com | www.91233h.com | d67890.com | www.40288.com | 38850066.com | www.4963hh.com | pj88oo.com | www.761111.com | www.163135.com | www.8694r.com | www.807835.com | www.555xin2.com | www.3416q.com | 6830y.com | www.89338c.com | 8722xxxx.com | www.0028229.com | 8538m.com | www.8905r.com | 6112jj.com | www.03a88.com | 405087.com | www.16181d.com | 33678.com | www.16181l.com | 19991p.com | www.4727.com | 3522p.vip | www.bet63n.com | 3405yy.com | www.38138u.com | 86811ll.com | www.huangma28.com | h5429.com | www.hgdc700.com | 5360d.com | www.pjc22.vip | o3405.com | www.798345.com | 496.es | www.yh9877.com | 500000515.com | www.26299r.com | 8827ggg.com | www.gczj5.com | www.341188.com | www.568380.com | www.7676760099.com | 4116t.com | www.911955.com | www.80767t.com | 7792u.com | www.371r.cc | 76543l.com | www.75hk.com | 2698d.com | www.0169a.com | 3089s.com | www.0088hg.com | 5309d.com | www.97171.cc | www.89422.com | www.346977.com | www.6567.cc | 3189p.com | www.0042737.com | afcp10.com | www.amhg002.com | www.9737ww.com | 453.com | www.4331vip.com | 444ths.com | www.hf0866.com | 4182a.com | www.4331i.com | d8502.com | www.669537.com | www.bmdc6677.com | 78808l.com | www.73990x.com | qq365v.com | www.hx1162.com | www.pj3390.com | 2373h.com | www.880317.com | 20160913.pw | www.755908.com | www.665345.com | hgw168v.com | www.x55789.com | 3049l.com | www.wcp8898.com | www.js00033.com | 33432w.com | www.99677x.com | www.yddc01.com | wns2.net | www.2021m.com | heji271.com | www.828516.com | www.456099.com | 000hg.com | www.53900h.com | www.383050.com | 3550b.com | www.cb600.cc | www.m.8235n.com | hg89111.com | www.bwinyz36.com | m666.org | www.217380.com | www.668cp99.com | 67890v.com | www.443300a.com | www.234345.com | 0241x.com | www.9478e.com | www.8884j.com | 61652t.com | www.68682r.com | www.pj9666.com | 4289y.com | www.68568f.com | www.88gg940.com | 4288hh.com | www.66653l.com | www.y8867.com | dd63777.com | www.50054b.com | www.js6767g.com | 4997y.com | www.3126f.com | www.10340.com | 6556pk.com | www.lqc4.com | www.1222l.com | app3865.com | www.517075.com | www.5446ww.com | qq444000.com | www.943977.com | www.77731i.com | 22dd8331.com | www.51515t.com | www.w84o.com | www.9374n.com | szgj666.com | www.25688n.com | www.12102.com | yy3336.com | www.348918.com | www.32688.cc | www.7168803.com | 079.com | www.7782t.com | www.54555.com | hg7088.com | 59964cc.com | www.48330y.com | www.66615yy.com | 4116y.com | www.548338.com | www.z063801.com | www.32126f.net | 2595x.com | www.818827.com | www.bet63f.com | www.68666l.com | 7779r.cc | www.978231.com | www.8850.com | www.56733l.com | 1389j.com | www.911209.com | www.009159.com | www.76060r.com | 58535555.com | www.433112.com | www.pjjtzy.cc | www.pj88222.com | 3568ww.com | www.87668o.com | www.131095.com | www.kj88822.com | 2jsuuu.com | 51133hh.com | www.8999ka.com | www.dzj0666.com | www.6491c.com | 1339.com | www.67258e.com | www.ejylc19.com | www.000453.com | 2373x.com | 1669l.com | www.640118.com | www.3846hh.com | www.1113890.com | 6245.ltd | 8178522.com | www.572539.com | www.flb906.com | www.hg78789.com | 15920000.com | le888p.com | www.346277.com | www.624205.com | www.6a444.com | www.a8a8345.com | 063910.com | 11aa8332.com | www.0014t.com | www.390292.com | www.xb0021.com | 2844o.com | www.109027.com | www.52303q.com | www.4136c.com | www.299888.com | nn500c.com | hg0088.pw | www.199301.com | www.81233d.com | www.00772r.com | www.903253.com | oo3189.com | blr969.com | 111000rr.com | www.957806.com | www.68365b.com | www.799666g.com | www.hg3598.com | 7653nci.com | 88682.com | d61653.com | www.921599.com | www.hd8671.com | www.c63.com | www.ajs5.com | 4636699.com | 40033nnn.com | 97570yl.com | www.234269.com | www.9356g.com | www.8905u.com | www.68277444.com | www.js07999.com | 3136rr.com | 7240v.com | 2096s.com | 2418007.com | www.788257.com | www.26163i.com | www.30350q.com | www.881.ag | www.tbb003.com | www.70398.com | 2jssss.com | m08199.com | 8522kkkk.com | pj1314520.com | www.783090.com | www.54400t.com | www.55676y.com | www.xpj2008.co | www.53848.com | www.8814e3.com | j83377.com | ha080.com | 30019yy.com | 30688c.com | 668.bz | www.082709.com | www.953500.com | www.50026m.com | www.9971i.com | www.v01888.com | www.2805.com | www.hg8316.com | www.35553999.com | www.9374r.com | 47749u.com | hgwz33.com | d1915d.com | 287722.com | 1665kk.com | ooo3405.com | em999.net | www.186791.com | www.626747.com | www.okw0.com | www.8499s.com | www.99677z.com | www.918.com | www.vns9998.me | www.3066kk.com | www.9c456.com | www.38238x.com | www.986msc.com | www.vns1668.cc | www.2061270.com | www.365635.com | www.bmw0055.com | 5005j.com | 518cp00.com | aa99151.com | js57666.com | 62222d.com | r14666.com | 533177.com | 418691.com | 3614z.com | 3421c.com | 04666m.com | 07833.com | 3421h.com | 8159t.cc | 4488w.com | 3846aaa.com | kkk01234.com | 86226s.com | 4488y.com | www.2544g8.com | l277.com | n888559.com | xhtd.com | 2846w.com | 168cp-v.com | feicai0470.com | 3122zz.com | 2618zz.com | 55984s.com | www.hg0056.com | www.63877r.com | www.75878tt.com | www.7830z.com | www.32666f.com | www.110732.com | www.444167.com | www.456277.com | js92359.com | 3559vv.com | bet99939.com | feicai0557.com | 90856m.com | www.hg288.com | www.hg0110.com | www.hg1187.com | www.hg3348.com | www.bai4422.com | www.c25aaa.com | www.1851117.com | www.0622sss.com | www.525995.com | www.16878i.com | www.5086e.com | www.949405.com | www.379477.com | 90307h.com | 1259i.com | 422yh.com | www.81233v.com | www.50074e.com | 73999b.com | w9625.com | 61366d.com | hg7033.com | 6220kk.com | www.zzzz0266.com | www.hg7644.com | www.408477.com | www.30350g.com | www.hg6014.com | www.2632b.com | www.613069.com | 0004661.com | v9330.com | 7192004.com | wnsr8806.com | www.am3366.com | www.hg3373.com | www.b948b.com | www.40288g.com | www.96386p.com | www.920959.com | 6hcc44.com | 15a46.net | 4167.com | 18775v.com | www.26yh76.com | www.e4042.com | www.656133.com | www.7714f.com | www.901572.com | 20160913.tv | b05677.com | 08159p.com | www.s1043.com | www.e3838.com | www.673888u.com | www.xb88.com | www.903079.com | 505mmm.com | vns00.vip | 096.com | www.33355508.com | www.kb9996.com | www.xpj71333.com | www.2408e.cc | www.50052a.com | 73567tg1.com | 2127pp.com | www.yh03456.com | www.3777338.com | www.490807.com | www.3126t.com | www.36788i.com | 36989f.com | 28758k.com | www.68610022.com | www.bet91481.com | www.84499w.com | www.444059.com | 77201.com | 6150c.com | www.8814e3.com | www.9b004.com | www.83033b.com | www.310195.com | 10050533.com | 6220i.com | www.427567.com | www.308956.com | www.500cd.com | 79964a.com | uu8159.cc | www.hg45888.com | www.c01885.com | www.99094j.com | www.717229.com | 5309e.com | n888559.com | www.pj038888.com | www.673888b.com | www.9928g.com | 01019991.com | 3300hhgz.com | www.a4737.com | www.05bet8.com | www.5091x.com | 2535.com | 563567.com | www.xh7738.com | www.00778j.com | www.clh9.com | 2618e.com | 66876j.com | www.188588.com | www.792087.com | www.546599.com | 500000454.com | www.81866n.com | www.cs.bet | www.871828.com | 4880y.com | 994167.com | www.9679h.com | www.xpj11885.com | www.626507.com | xpj382.cn | www.hg6789.com | www.5099nn.com | www.077wy.com | 6423e.com | l35151.com | www.42msxfpt5.com | www.50024t.com | angei9gov.cn | feicai0757.com | www.339879.com | www.77210k.com | 40661113.com | 3009e.com | www.a9151e.com | www.9356a.com | 92928633.com | 9149c.com | www.rcw88.vip | www.hcjt7.com | 047106.com | www.hg0798.com | www.9997028.com | www.582977.com | vns8351.com | www.25288w.com | www.u98478.com | 2418007.com | 8381ff.com | www.63msxfptb.com | www.055p.cc | 20773388.com | www.38200p.com | www.83066.com | www.225075.com | 3506.com | www.036888.com | www.07163f.com | 1592.com | www.8768011.com | www.56011z.com | 776314.com | nn500c.com | www.51705.com | www.806721.com | 4036yy.com | www.448809.com | www.36166r.com | 1591hhh.com | www.j22365.com | www.37377d.com | 2019p.cc | www.vtm005.com | www.2848.com | 62205.com | ag6676.com | www.820025.com | 22296ab.com | 99151c.com | www.b35jj.com | www.335284.com | 80368tt.com | www.28281app.com | www.589879.com | xpj11558.com | www.w948w.com | www.652862.com | 53358n.com | www.79500x.com | www.506586.com | wlb444.com | www.7370.com | www.364511.com | 78118877.com | www.n69096.com | www.33997v.com | 68963999.com | www.9895u.com | spj5577.com | www.wn2008.com | www.win1236.com | 272cc.net | www.88837e.com | www.26878p.com | 1458i.com | www.560yy.com | www.805595.com | 6261mm.com | www.bet63q.com | 08530005.com | www.js1166.com | www.974209.com | 69990k.com | www.hg6058.com | www.981500.com | vip7022.com | www.55060x.com | 11aa8332.com | www.38200y.com | www.hg6767.com | 1407y.com | www.www-737475.com | www.755834.com | 7935.com | www.c3846.com | 8547r.com | www.ee2229.com | www.905866.com | 2222k09.com | www.14168f.com | 8722oooo.com | www.189555.com | www.810877.com | 3844d.com | www.cb500.cc | 3630.com | www.wi6666.com | www.249711.com | 9464009.com | www.99677g.com | 3846q.com | www.9737yy.com | 99662016.com | www.yhc573.com | www.813770.com | 2546.com | www.xj1996.com | acp0006.com | www.2yhyh.com | 2190s.com | www.7545q.com | www.630367.com | 1665p.com | www.50026w.com | 1669j.com | www.44118f.com | 1222r.com | www.2945c.com | alpk22.com | www.hg9556.com | www.658252.com | 1665bb.com | www.08588r.com | 7720i.com | www.78919c.com | 4508yy.com | www.m80288.com | 8884001.com | www.64.ag | jinlong06.com |