前言

這是個在寫計算機網絡課設的時候碰到的問題，卡了我一天，所以總結一下。

其實在之前就有用requests寫過python爬蟲，但是計算機網絡要求更底層的實現，剛好我看到了[這篇文章]1結果發現他就是用socket來實現的請求，所以就學習了。

本來也覺得應該不難，畢竟就是建立tcp連接。

原網站的例子如下：

 def fetch(url): sock = socket.socket() # 建立socket sock.connect(('xkcd.com', 80)) # 遠程連接 request = 'GET {} HTTP/1.0/r/nHost: xkcd.com/r/n/r/n'.format(url) # 構建請求 sock.send(request.encode('ascii')) # 向socket發送數據 response = b''  chunk = sock.recv(4096) # 從socket接收數據 while chunk: response += chunk chunk = sock.recv(4096) # Page is now downloaded. links = parse_links(response) q.add(links)

我選擇爬取的網站是鏈家，當然也看了很多其他例子,還用fiddler抓包，把headers整個放了上去，首先是參考了這個文章：https://segmentfault.com/a/1190000005126160 ，該文章介紹到：

Python通過socket發送http請求

我們以訪問百度主頁為案例，使用socket發送http請求。

import sockets=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect(('www.baidu.com',80))s.send('''GET https://www.baidu.com/ HTTP/1.1Host: www.baidu.comConnection: keep-aliveAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8Upgrade-Insecure-Requests: 1User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36Accept-Language: zh-CN,zh;q=0.8''')buf=s.recv(1024)while len(buf):print bufbuf = s.recv(1024)

基于socket進行http編程在于請求參數的可控性更好，但是難度相應就更大了。以上發送的數據是直接從fiddler抓包器里面復制的。

基于以上，寫出如下代碼：

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect(('www.baidu.com',80))s.send('''GET / HTTP/1.1Host: zh.lianjia.comConnection: keep-aliveCache-Control: max-age=0Upgrade-Insecure-Requests: 1User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8Referer: https://www.baidu.com/link?url=4J5Kx--GLdLFESJhkfRePU8Ac_0agnTcOtB-b3kfnX8VNdZ_6TPqOyJGKVXkTczg&ck=6140.3.83.296.315.287.208.155&shh=www.baidu.com&sht=94886267_hao_pg&wd=&eqid=af98b98700060b77000000065aef0524Accept-Encoding: gzip, deflate, brAccept-Language: zh-CN,zh;q=0.9,en-CA;q=0.8,en;q=0.7Cookie: lianjia_uuid=ce61c41c-25b0-46d6-a0a0-d57a75ee8706; UM_distinctid=1631f588055f9-0286722badd3ec-b34356b-1fa400-1631f58805657f; _ga=GA1.2.43397143.1525239286; _smt_uid=5ae94e02.558be516; _jzqx=1.1525248800.1525335927.1.jzqsr=zh%2Elianjia%2Ecom|jzqct=/ershoufang/xiangzhouqu/.-; _jzqc=1; _jzqckmp=1; _gid=GA1.2.1028411676.1525594529; select_city=440400; all-lj=c60bf575348a3bc08fb27ee73be8c666; _qzjc=1; CNZZDATA1254525948=963210960-1525238218-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608956; CNZZDATA1255633284=1054798284-1525238580-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608969; lianjia_ssid=c046ddb3-3e66-4809-998a-52ade335fdfc; _qzja=1.1070225156.1525239298260.1525603274282.1525613866775.1525609113492.1525613866775.0.0.0.92.9; _qzjto=29.3.0; _jzqa=1.3750161754444366000.1525239284.1525603274.1525613867.9; _jzqy=1.1525239284.1525613867.3.jzqsr=baidu.jzqsr=baidu; Hm_lvt_9152f8221cb6243a53c83b956842be8a=1525607433,1525607626,1525609113,1525613867; Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1525613867; _qzjb=1.1525613866775.1.0.0.0; _jzqb=1.1.10.1525613867.1; CNZZDATA1255604082=964175865-1525237915-https%253A%252F%252Fwww.lianjia.com%252F%7C1525612833''')