爬虫获取页面源码

 通过url获取网页源码，我们一般分为这几种情况

1.直接获取

网页是静态网页，直接使用

public static string HttpPost(string url, string paraJsonStr)
        {
            WebClient webClient = new WebClient();
            webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
            byte[] postData = System.Text.Encoding.UTF8.GetBytes(paraJsonStr);
            byte[] responseData = webClient.UploadData(url, "POST", postData);
            string returnStr = System.Text.Encoding.UTF8.GetString(responseData);
            return returnStr;
 
        }

就可以获取网页页面的源码

2.使用上面的方法，无法获取页面，得到Request unsuccessful

Request unsuccessful. Incapsula incident ID: 89300071...

这种情况通常是因为网站有反爬虫的处理

解决办法：

2.1.请求添加Headers里面的信息

通常需要添加  “User-Agent”的值，不行的话，可以多添加几个header的属性和 值进去

常用的 “User-Agent”如下：

"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",

    "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",

    "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",

    "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",

    "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36 Edg/101.0.1210.39",

    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54"

例如：

public static string HttpPost(string url, string paraJsonStr = "")
        {
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);         
            request.Headers["User-Agent"] = "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Mobile Safari/537.36 Edg/104.0.1293.54";
            request.Headers["Cookie"] = "visid_incap_161858=BlNpjA+qS9ucNza1";
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            Stream responseStream = response.GetResponseStream();
            StreamReader streamReader = new StreamReader(responseStream, Encoding.UTF8);
            string str = streamReader.ReadToEnd();
            return str;
        }

2.2.在访问页面时加上时间间隔

3.无法获取到页面源代码，获取到的源代码不全，只有基本的框架

在Response中无法获取到完整的页面源代码，跟element的元素不同。这种情况使用Selenium插件的办法

import re
import time
import json
import random
from turtle import pd
from xml.dom.minidom import Element
import requests
import csv
from selenium import webdriver
# from selenium.webdriver.chrome.options import Options
from selenium.webdriver.edge.options import Options
from time import strftime, localtime

#path to webdriver
//需要下载浏览器的driver，括号里面式msedgedriver.exe的下载地址
    driver = webdriver.Edge(executable_path = "C:\\...\\msedgedriver.exe")
//加载时窗口的大小
    driver.maximize_window()
//url时加载时的网址，获取信息的具体网页的地址
    driver.get(url)
    #The sleep time is the time for the web page to dynamically obtain information. Depending on the speed of the network, different adjustments can be made.
    time.sleep(10)
//网页自动往下滑动的大小    
    driver.execute_script("window.scrollBy(0,{})".format(1500)) 
    time.sleep(2)
    #The page has a button, click to get the information of the whole page
    
//具体网页button的xpath路径
if(driver.find_element_by_xpath("//div[@class='articlePage_button']/span[@class='buttonText']")):
        driver.find_element_by_xpath("//div[@class='articlePage_button']/span[@class='buttonText']").click()    
    time.sleep(5) 

#Store data to file
    f=open("C:\\...\\a.csv","a+",encoding='utf-8',newline="")
    a_writer=csv.writer(f)

#parsing html information
    aText=driver.find_element_by_xpath("//h1[@class='pageTitle']").text
    print("aText:",aText)    
    a_writer.writerow(["name",aText])
#输出空行需要这样
    a_writer.writerow(["",""])


    driver.quit()

总结：

前面两种办法，都可以通过Response中查看到完整的页面代码