<address id="bztp3"><nobr id="bztp3"></nobr></address>

      <sub id="bztp3"><listing id="bztp3"><menuitem id="bztp3"></menuitem></listing></sub>

        <form id="bztp3"></form>

        <address id="bztp3"></address>
        <sub id="bztp3"></sub>

          <form id="bztp3"></form>

              Python面試真題 – 常見的反爬蟲和應對方法?

              【Python面試真題】- 常見的反爬蟲和應對方法?

              1).通過Headers反爬蟲

              從用戶請求的Headers反爬蟲是最常見的反爬蟲策略。很多網站都會對Headers的User-Agent進行檢測,還有一部分網站會對Referer進行檢測(一些資源網站的防盜鏈就是檢測Referer)。如果遇到了這類反爬蟲機制,可以直接在爬蟲中添加Headers,將瀏覽器的User-Agent復制到爬蟲的Headers中;或者將Referer值修改為目標網站域名。對于檢測Headers的反爬蟲,在爬蟲中修改或者添加Headers就能很好的繞過。

              2).基于用戶行為反爬蟲

              還有一部分網站是通過檢測用戶行為,例如同一IP短時間內多次訪問同一頁面,或者同一賬戶短時間內多次進行相同操作。

              大多數網站都是前一種情況,對于這種情況,使用IP代理就可以解決。可以專門寫一個爬蟲,爬取網上公開的代理ip,檢測后全部保存起來。這樣的代理ip爬蟲經常會用到,最好自己準備一個。有了大量代理ip后可以每請求幾次更換一個ip,這在requests或者urllib2中很容易做到,這樣就能很容易的繞過第一種反爬蟲。

              對于第二種情況,可以在每次請求后隨機間隔幾秒再進行下一次請求。有些有邏輯漏洞的網站,可以通過請求幾次,退出登錄,重新登錄,繼續請求來繞過同一賬號短時間內不能多次進行相同請求的限制。

              3).動態頁面的反爬蟲

              上述的幾種情況大多都是出現在靜態頁面,還有一部分網站,我們需要爬取的數據是通過ajax請求得到,或者通過JavaScript生成的。首先用Fiddler對網絡請求進行分析。如果能夠找到ajax請求,也能分析出具體的參數和響應的具體含義,我們就能采用上面的方法,直接利用requests或者urllib2模擬ajax請求,對響應的json進行分析得到需要的數據。

              能夠直接模擬ajax請求獲取數據固然是極好的,但是有些網站把ajax請求的所有參數全部加密了。我們根本沒辦法構造自己所需要的數據的請求。這種情況下就用selenium+phantomJS,調用瀏覽器內核,并利用phantomJS執行js來模擬人為操作以及觸發頁面中的js腳本。從填寫表單到點擊按鈕再到滾動頁面,全部都可以模擬,不考慮具體的請求和響應過程,只是完完整整的把人瀏覽頁面獲取數據的過程模擬一遍。

              用這套框架幾乎能繞過大多數的反爬蟲,因為它不是在偽裝成瀏覽器來獲取數據(上述的通過添加 Headers一定程度上就是為了偽裝成瀏覽器),它本身就是瀏覽器,phantomJS就是一個沒有界面的瀏覽器,只是操控這個瀏覽器的不是人。利selenium+phantomJS能干很多事情,例如識別點觸式(12306)或者滑動式的驗證碼,對頁面表單進行暴力破解等。

              相關新聞

              聯系我們

              400-080-6560

              在線咨詢:點擊這里給我發消息

              郵件:work@magedu.com

              工作時間:周一至周日,09:00-18:30

              QR code
              656彩票平台656彩票主页656彩票网站656彩票官网656彩票娱乐 JS1388p.com | 9193.cc | www.17711.com | 9995z.cc | www.792085.com | 61323355.com | www.737198.com | www.23386000.com | 99110040.com | www.000128522.com | hg07111.com | www.342677.com | www.422209.com | 422yh.com | www.90305f.com | www.yun388.com | 31325e.com | www.71071q.com | r1915r.com | www.67767.cc | www.zr918.com | 30019ff.com | www.9895i.com | 0011bc.com | www.917787.com | www.443344.com | 18438y.com | www.pjzxyl.cc | www.e50336.com | 9411fff.com | www.565548.com | 7792a.com | www.679000.com | www.319393a.com | 469dh.com | www.2373r.com | www.5566074.com | 66669193.com | www.6653i.com | www.4107o.com | 2329.com | www.12455o.com | www.hg0088.lv | 2851t.com | www.0343i.com | www.hg0088.lv | 8030f.com | www.siji01.com | 4812222.com | 32126b.net | www.44118v.com | 201833.com | cj807.com | www.83377t.com | 54540123.com | j15gg.com | www.77802l.com | www.j30226.com | 1106.com | www.6769b.com | www.yh391.com | 8037xxx.com | www.912482.com | www.3122vv.com | 00009193.com | 910088.com | www.68365h.com | www.333222i.com | 8520g.com | www.ya2019a.com | www.44116609.com | vvv67890.com | www.374977.com | www.bwinyz19.com | 81511x.com | 3242a.com | www.81678l.com | www.79138.com | 9506n.com | www.354977.com | www.390292.com | www.bmw555888.com | b778877.com | www.00clf.com | www.dzh1133.com | 2381d.com | 3522ww.com | www.710502.com | www.ra8804.com | 6487jjj.com | www.178211.com | www.7920b.com | www.3459z.com | 0485y.com | www.451956.com | www.06386644.com | www.00829c.com | 3131e.cc | www.188393.com | www.7876389.com | www.5882323.com | 159666k.com | 53201133.com | www.9478f.com | www.5099ii.com | www.ylg2222.com | 00773c.com | www.659789.com | www.k7764.com | www.hg226.cm | 838388d.com | 80368jj.com | www.904857.com | www.3066ee.com | www.js46123.com | ctxcp26.com | www.098303.com | www.8582bb.com | www.44488807.com | 06064688.com | 30175544.com | jlykk.com | www.66621j.com | www.c5c77.com | www.pj6653.com | 86226.com | shanghaia18.com | www.50064f.com | www.77731o.com | www.pj11588.com | 75991j.com | www.26878u.com | www.60108n.com | www.53096666.com | www.122yh.com | 111122rr.com | 0207206.com | 5443j.com | www.rcw678.com | www.1116yl.com | www.46768.com | 5446q.com | 2566h6.com | 2096p.com | www.7dzh.com | www.456669.com | www.klcp3jc.com | www.333133p.com | 11005w.com | 8547k.com | www.33598i.com | www.575hc.com | www.36yl.com | www.8k002.com | www.31896000.com | 748335.com | 40033zz.com | 889345.com | www.375983.com | www.36166c.com | www.7783066.com | www.56655t.com | www.777444f.com | 500000710.com | 3999xl.com | xd2017.com | www.081306.com | www.tang444.com | www.0967006.com | www.5099rr.com | www.135suncity.com | www.7720o.com | zhcp65.com | 77606q.com | jdb58.com | 6868mm.cc | www.312180.com | www.igcp2.com | www.3334331.com | www.2021e.com | www.732156.com | www.88807c.com | 3644u.com | LXYL370.com | 9506p.com | 838388c.com | 7811i.com | 0004661.com | www.673066.com | www.84577.com | www.xx888.cc | www.62979.com | www.ll55826.com | www.700067.com | www.mic999.com | www.hggj.cc | 44449193.com | 1064f.com | 3156qqq.com | 1888860.com | www.kj222888.com | www.4828668.com | www.998855d.com | www.10852b.com | www.5860n.com | www.32126p.net | 7777ag2.com | 2997707.com | sj7377.com | 3566jj.com | 4591001.com | 813ooo.com | 5622d.com | 1654v.com | k8381.com | 9420v.com | pj88oo.com | 3511.sbf822.com | xpj893.com | c89qe65.com | 3258k.com | www.191060.com | www.504133.com | www.507433.com | www.502990.com | www.606694.com | www.602266.com | www.483770.com | www.452806.com | www.233591.com | www.376517.com | www.290252.com | www.195880.com | www.289971.com | www.181599.com | www.126438.com | www.81233o.com | 4379w.com | www.3479o.com | www.2955.cc | www.926730.com | www.550249.com | www.358773.com | www.196280.com | www.36166a.com | 10222.com | 7742pp.com | 0343i.com | 33557.com | 3559cp.net | 00774bb.com | by309.com | www.a3a000.net | www.yh8333.com | www.00618r.com | www.kelake55.com | www.cao880.com | www.97060d.com | www.00778e.com | www.0622iii.com | www.hg16688.com | www.33997h.com | www.am7979.com | www.369965.com | 1408t.com | 3550z.com | 8668827.com | 4717.com | 0033b.cc | www.00618k.com | www.777348.com | www.79500e.com | www.45598v.com | www.806765.com | www.61655w.com | www.938906.com | www.156506.com | 7799833.com | 66066p.com | 51200.com | 365058.com | www.588778.com | www.577444.com | www.19019d.com | www.3775p.com | www.11888js.com | www.88065f.com | www.837771.com | 00nn8331.com | 33382xx.com | 99112007.com | www.hg2267.com | www.wnsr0055.com | www.xpj11882.com | www.9155e.cc | www.36787a.com | 4812h.com | 33382cc.com | www.wn2008.com | www.hd2894.com | dd8662.com | 3049j.com | www.2598567.com | www.yk222b.com | www.642486.com | www.tang444.com | xpj.com | 91019k.net | 2019k.cc | www.7726e.com | www.4996jx.com | www.66332f.com | www.356776.com | a7252.com | 55331b.com | www.qqqqkkkk.com | www.7415ii.com | www.80065n.com | www.187135.com | i8381.com | 2060099.com | www.6410.com | www.85656.com | www.996745.com | 66458k.com | 9339999u.com | www.20161144.com | www.9170011.com | www.11554.cc | yh7003.bet | youfa777.vip | www.900591.com | www.2002366.com | www.hy5355.com | 7196z.com | 4960077.com | www.5319r.com | www.469704.com | www.qml4.com | 2613k.com | qqq8827.com | www.556644.cc | www.1434c.com | www.582993.com | js6444.com | 599369.com | www.5966aaa.com | www.96386d.com | 55oo8332.com | 28839b.com | www.45992c.com | www.9994331.com | www.43131r.com | 9679m.com | www.88807o.com | www.ag3399.com | www.582977.com | 88ff940.com | www.32126o.net | www.bet63p.com | www.hm3988.com | 69522v.com | 1775ee.com | www.6661ylg.com | www.hx1184.com | 6701b.com | www.4759mm.com | www.xpj16681.com | www.508533.com | 30688x.com | www.yl0399.com | www.hg241.com | www.89656a.com | j3405.com | www.32666l.com | www.83033g.com | 518cp.com | 44995002.com | www.cn365m.com | www.621416.com | 98955x.com | www.xh4477.com | www.8887sy.com | 656992.com | 6245.bet | www.115496.com | www.347977.com | 4809h.com | www.wi6666.com | www.111xm.cc | 8547aa.com | www.66166s.com | www.954321u.com | 3379g.com | www.pj23333.com | www.99552kk.com | www.77801a.com | 87665e.com | www.00618h.com | www.581905.com | 231498.com | www.hengshengguoji.com | www.hw82.com | 36407799.com | www.287722.com | www.621316.com | 0207210.com | www.hg995.com | www.if57.com | 3662k.com | www.99828.com | www.910424.com | d15666.com | www.77731k.com | www.647977.com | 6xh.vip | www.89599b.com | www.029783.com | 500000371.com | www.88325o.com | vwin666.com | www.ylhg1818.com | www.bj6666.com | 3788b.com | www.9a003.com | www.tlcp4.com | 53262t.com | www.61130.com | www.081501.com | 457892.com | www.99yh21.com | j0088.com | www.sss998.com | www.911720.com | 73055u.com | www.344654.com | 6220oo.com | www.9068zz.com | www.26199.com | 7555w.com | www.2yhyh.com | 69440011.com | www.7025d.com | www.1248g.cc | 00jj8332.com | www.778545.com | 1347.com | www.0241h.com | www.068619.com | www.js72999.com | www.76520x.com | hjcp77.com | www.0002737.com | 28747.com | www.www--77658.cc | www.803241.com | www.zfcp8.com | 998u.cc | www.c94600.com | 9949i.com | www.33707d.com | hg2015.cc | www.1770q.com | www.599091.com | 1449e.com | www.77210f.com | 62222s.com | www.5599msc.com | a8381.com | www.6666yh.vip | 50000885.com | www.88jt99.com | 30007d.com | www.7196x.com | www.401345.com | www.333222x.com | www.jcai0.com | 500000735.com | www.51331b.com | 86811rr.com | www.001170.com | 4288zz.com | www.56011u.com | dzc13.com | www.158024.com | zhibet188.com | www.cn365e.com | agaa29.com | www.65066ee.com | 1408.com | www.biying920vip.com | 36989d.com | www.0072737.com | 53262ii.com | www.4hg6668.com | 80850z.com | www.9981331.com | 53059.vip | www.4058p.com | 3392958.com | www.377666i.com | blhvip222.com | www.8c444.com | 4488aaa.com | www.0343l.com | 365bet888.com | www.33311.com | 1679c.com | www.52001.cc | www.81866x.com | www.890402.com | www.7893w04.com | www.223075.com | www.a80065.com | 08530006.com | www.twcp05.com | 13222r.com | www.63606d.com | v82365.com | www.99677k.com | 2613u.com | www.az0908.com | www.55268j.vip | www.78949o.com | www.005596.com | zyrb.com.cn | www.932418.com | 3566cc.com | www.73736f.com | qq5144.com | www.587944.com | www.mm2229.com | ms38648ff.cc | www.c80288.com | 04666p.com | www.66332t.com | www.k88875.com | www.305995.com | www.4901b.com | 19991m.com | www.52062s.com | sha2017.com | www.915196.com | www.884554.com | 50067r.com | www.u063801.com | 86811zz.com | www.765703.com | 7792k.com | www.560529.com | www.6574d.com | a83377.com | www.55717g.com | 996622yy.com | www.187396.com | www.68277666.com | hhh40033.com | www.6364m.com | www.7025f.com | 123456hh.cc | www.69567u.com | feicai0751.com | www.715920.com | www.100449.com | 3136dd.com | www.83033m.com | www.6666.cm | 0033v.cc | www.d94600.com | 38989o.com | 3822f.com | www.mgm868002.com | 6261d.com | www.712887.com | www.684000.com | 53262m.com | www.080wy.com | www.4270ll.com | 68742.com | www.73166c.com | www.v30226.com | 49490524.com | www.sb4010.com | www.32329193.com | 883399z.com | www.6858666.com | www.223456y.com | 1784.com | www.yh08.com | www.117390.com | 55797.com | www.84499y.com | www.09569g.com | 3650744.com | www.50732h.com | www.yh33566.com | qy002.vip | www.092wy.com | www.zz00y.com | t21148.com | www.653355.com | www.163a3.com | vic.am | rbbet77.com | www.aaa2848.com | www.9544j.com | 31325a.com | www.208339.com | www.66939988.com | eee5682.com | www.hd5658.com | www.tyc188.com | 1hga.com | qq7742.com | www.hr1855.com | www.9068dd.com | 2677sss.com | www.c2953.com | www.85857s.com | 7141aa.com | 11995002.com | www.yh76o.com | www.279765.com | c4255.com | www.407872.com | www.887767.com | www.9374t.com | z45638.com | www.974306.com | www.7415ss.com | 00778f.com | hgw2777.com | www.126kc.com | www.DF83.com | 9411uuu.com | 00048w.com | www.77fzc.com | www.33707d.com | 18775k.com | 706803.net | www.927320.com | www.4996sp.com | www.936699.com | 55331zz.com | www.808731.com | www.377666b.com | www.2090900.com | 9101909.com | www.233896.com | www.48330t.com | www.pj8474.com | wnsr8808.com | 7742tt.com | www.66fzc.com | www.26123hh.com | www.3657a.com | 3405c.com | 84497700.com | www.73166l.com | www.c44ii.com | www.a3a666.cc | 6340.com | www.89894d.com | www.1432.com | www.20199ww.com | www.882236f.com | 2147002.com | www.90305a.com | www.08588e.com | www.c44ii.com | www.18xinli.cc | 3648.com | 4639966.com | www.zfcp2.com | www.73990l.com | www.hc8888.com | 111122bb.com | w69441122.com | cao568.com | www.2632u.com | www.466890.com | www.me6622.com | 7945xx.com | 223536.com | 1483jj.com | www.158cp.com | www.63606e.com | www.56830f.com | 62620029.com | kkkk19927.com | 6220xx.com | www.ct8833.com | www.632bbb.com | www.41hd.com | www.282xpj.com | mm99966.com | x21148.com | www.68568o.com | www.9149a.com | www.58908b.com | www.188614.com | www.76060e.com | ahg1088.com | 556677882.com | 18775u.com | www.cp994.com | www.ihg5544.com | www.58118e.com | www.hg7869.com | qbzb2b.net | 45637m.com | dd3189.com | 1468.com | www.533660.com | www.50080q.com | www.60886v.com | www.beb111.org | www.09569r.com | 1624hh.com | y68bb.com | 44335156.com | 6245.be | www.178625.com | www.0ttz.com | www.6613644.com | www.45598f.com | www.gssuu168.com | www.js55567.com | www.8520u.com | amjs8033111.com | 45637h.com | 0343e.com | x33v.vip | www.346011.com | www.9dwj.com | www.0270y.com | www.86339p.com | www.3157g.com | www.82799b.com | www.5y.cc | www.1112xj.com | mry345.com | 55331382.com | r1915.com | 3846nn.com | 1669c.com | 2222k33.com | www.178357.com | www.810575.com | www.3890h.com | www.60wb.top | www.0088hgbb.com | www.35252c.com | www.xy047.com | www.bs9088.com | www.09527w.com | www.g22365.com | www.6177a.com | xx888y.com | 6220ff.com | 1109758.com | 711066.com | 6704888.com | 5627.com | 0234vv.com | 61328899.com | 5856ff.com | 838388a.com | xhgx2.com | 34w35.com | dl.hg258.in | 2649rr.com | 61329999.com | www.35155i.com | www.263299.com | www.393411.com | www.764499.com | www.911888aa.com | www.yy63.com | www.qucw8.com | www.091wy.com | www.022yn.cc | www.3479j.com | www.3416c.com | www.6769x.com | www.5484f.com | www.4189b.com | www.3479v.com | www.333hy.cc | www.bxcp3333.com | www.hx1165.com | www.rcw8800.com | www.911924.com | www.766076.com | www.658770.com | www.545669.com | www.355264.com | www.168323.com | www.33112a.com | yf2888.com | 80850ff.com | 54443o.com | 4052j.com | 776053.com | 059dhw.com | hg11765.com | jj00558.com | 025132.com | 2649pp.com | www.pj0004.com | www.51990.com | www.yh8364.com | www.lhcbm.com | www.pj6668.com | www.7779566.com | www.sbd024.net | www.9895z.com | www.1764h.com | www.831666.com | www.43818g.com | www.005ac.com | www.770981.com | www.289379.com | 22hh8332.com | 07599d.com | 69990b.com | so33333.cc | 655661177.com | 3078z.com | www.bet1677.com | www.66318t.com | www.hg0817.com | www.50999d.com | www.xj6002.com | www.4447758.com | www.52303y.com | www.hy0038.com | www.354611.com | 8522hhhh.com | 86688100.com | 29533.com | 3078h.com | www.6999blr.com | www.69111n.com | www.854569.com | www.15595.com | www.jgj.205115.com | www.5095o.com | www.560121.com | 80878r.com | 4433634.com | 01234ttt.com | 22998332.com | www.653501.com | www.999450.com | www.47751.net | www.nx58qp.com | www.3416y.com | www.326265.com | 3614a.com | b45638.com | zhcp99.com | www.4625s.com | www.05899.com | www.776887.com | www.76520z.com | www.595953.com | 28288n.com | 444000ff.com | 883399q.com | www.88837s.com | www.85857b.com | www.350c6.com | www.3126aa.com | www.068639.com | 00339193.com | 112v.net | www.dh03035.com | www.7335y.com | www.pj55717.com | www.qmkl6.com | 69447711.com | 08450.com | 3544i.com | www.8000hm.com | www.99552ee.com | www.022qh.cc | 93922e.com | feicai0412.com | jr7777.com | www.810702.com | www.570981.com | jj67890.com | www.65199.com | www.1559501.com | www.961287.com | 00w99.vip | 79889b.com | www.50999q.com | www.333xm.cc | 112y.net | 9411rrr.com | www.677256.com | www.2875z.com | 1634q.com | 22556w.com | www.77699.com | www.987jc.com | 6150t.com | www.js388999.com | www.9737aa.com | www.919509.com | 44ii8331.com | www.11155545.com | www.738055.com | www.504633.com | 3726k.com | www.hg0399.com | www.2418l.com | bmw8002.com | 8899988.com | www.js16288.com | www.0014f.com | 88993b.com | www.7669f.net | www.60886u.com | www.175905.com | 33rr8331.com | www.w10085.com | www.3890q.com | 1bet2.com | www.hg0088.lv | www.664456.com | 官网捕鱼1.com | hh5443.com | www.jz298.com | www.662182.com | 28365.com | www.767.cc | www.5854p.cc | 3614c.com | www.4625r.com | www.37377r.com | 7508x.com | 1327z.com | www.3410.com | www.043552.com | 3665yh.com | www.88166q.com | www.529026.com | 26668o.com | www.3533msc.com | www.915655.com | sbd9.vip | www.z3285.com | www.c5907.com | bbb1915.com | www.3688mhd.com | www.ya958.com | 5309j.com | www.7777hx.com | www.658zf.com | 4776611.com | www.js7896.com | www.7782x.com | 22668448.com | www.hg995.com | www.ya077.com | 86226n.com | www.3029.com | www.810706.com | wns173.com | www.301393.com | www.343577.com | 44488d.com | www.m32031.com | 2381rrr.com | www.pj550088.com | www.81608l.com | 1382.cm | www.2077w.com | www.5095i.com | 32555z.com | www.t9478.com | www.389044.com | 2805r.com | www.z2894.com | bet999000.com | www.6020400.com | www.71399w.com | sbdjgj.vip | www.990654.com | www.514211.com | 2546h.com | www.78680o.com | 7894.com | www.32666a.com | www.2934k.com | 3778tt.com | www.19019b.com | ii4675.com | www.76060x.com | www.61655m.com | 4022p.com | www.7793r.com | www.175905.com | y00351.com | www.388158.com | 06006l.com | www.888999yh.com | www.61233q.com | j4255.com | www.5099oo.com | 51122.com | www.7435x.com | www.54400h.com | 3482j.com | www.low8937.com | www.303204.com | www.033033a.com | www.0270r.com | 7811ww.com | www.pj3377.com | www.261793.com | www.18735528058.com | www.7886265.com | 304daohang.com | www.rycp159.com | xinu10.com | www.vnsr500.com | www.2875e.com | ylzz2228.com | www.98488.com | 3424z.com | www.whmydsc.com | zzylc5.com | www.655666e.com | www.1035f.com | 26668k.com | www.862239.com | 47751.com | www.83568y.com | 1483f.com | www.caipiao037.com | www.407872.com | 2924444.com | www.34489.cc | 00773ee.com | www.48330d.com | 5745222.com | www.45682.com | 3122kk.com | www.4519.com | www.338035.com | www.32126h.net | www.ck9191.com | 22335156.com | www.99677y.com | 1407.am | www.v63568.com | 1443f.com | www.8473x.com | 1397.cc | www.kuaibet.com | 2934j.com | www.81776.com | www.43131i.com | www.y2205.com | www.330587.com | www.700y.com | www.ct8811.com | hg31310.com | www.60fr.com | 47749u.com | www.12455r.com | p8999.cc | www.50026f.com | 44077o.com | www.00840q.com | sbobet.com |