添加cookie

来源:7-1 数据表结构设计

若羲

2022-06-10

老师:在selenium,获取到拉勾网的cookie后,保存在本地后,我怎么用这个cookie,就是带到每一个请求里面,它是list 包裹的dict

[
{
    "domain": ".lagou.com",
    "expirationDate": 1655007008,
    "hostOnly": false,
    "httpOnly": false,
    "name": "__lg_stoken__",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 1
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654876800,
    "hostOnly": false,
    "httpOnly": false,
    "name": "__SAFETY_CLOSE_TIME__21662680",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 2
},
{
    "domain": ".lagou.com",
    "expirationDate": 1717917211,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_ga",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 3
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654845764,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_gat",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 4
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654931611,
    "hostOnly": false,
    "httpOnly": false,
    "name": "_gid",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "GA1.170afd24d251de6cb94fed9b8fe8ad45743f6",
    "id": 5
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "_putrc",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "97547EC170afd24d251de6cb94fed9b8fe8ad45743f60EADC",
    "id": 6
},
{
    "domain": ".lagou.com",
    "expirationDate": 1655450009.568349,
    "hostOnly": false,
    "httpOnly": false,
    "name": "gate_login_token",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "9aa4aa8db4",
    "id": 7
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "1654845212",
    "id": 8
},
{
    "domain": ".lagou.com",
    "expirationDate": 1686381211,
    "hostOnly": false,
    "httpOnly": false,
    "name": "Hm_lvt_4233ea3d81c6ccf756e6",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "16548340526117",
    "id": 9
},
{
    "domain": ".lagou.com",
    "expirationDate": 1657437210.665005,
    "hostOnly": false,
    "httpOnly": false,
    "name": "index_location_city",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "%E5%B9%BF%E5%B7%9E",
    "id": 10
},
{
    "domain": ".lagou.com",
    "expirationDate": 3802328856.435914,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LG_HAS_LOGIN",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "1",
    "id": 11
},
{
    "domain": ".lagou.com",
    "expirationDate": 3802328856.435821,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LG_LOGIN_USER_ID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "abcs",
    "id": 12
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGRID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "20220610151331-5dfe84fb-07b5-462c-9472-b1c351ff1da1",
    "id": 13
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654847011.24849,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGSID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "20220610151244-48392d09-f248-49fc-94a3-2af0aaf9a69b",
    "id": 14
},
{
    "domain": ".lagou.com",
    "expirationDate": 1967514230.483062,
    "hostOnly": false,
    "httpOnly": false,
    "name": "LGUID",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "20220510114351-abdfed1d-9e4b-4f5d-9b8c-743e8d8f6c5d",
    "id": 15
},
{
    "domain": ".lagou.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "login",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "true",
    "id": 16
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854852,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_HOST",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 17
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.85488,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_LAND",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "https%3A%2F%2Fwww.lagou.com%2Fguangzhou%2F",
    "id": 18
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854867,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_SITE",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 19
},
{
    "domain": ".lagou.com",
    "expirationDate": 1654846964.854836,
    "hostOnly": false,
    "httpOnly": false,
    "name": "PRE_UTM",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "",
    "id": 20
},
{
    "domain": ".lagou.com",
    "expirationDate": 1655450009.944884,
    "hostOnly": false,
    "httpOnly": false,
    "name": "privacyPolicyPopup",
    "path": "/",
    "sameSite": "unspecified",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "false",
    "id": 21
}]
写回答

1回答

若羲

提问者

2022-06-10

  1. 在pycharm 找到scrapy的源码,看cookies的参数,初始化时,是可以传入list(dict)

  2. 在读取cookie文件后,将数据反序列化,转为list,里面每一个就是一个dict  

  3. 在中间件中(process_request) 方法中,把cookies 设置为上面读取出来的值 

  4. 记得在settings.py中配置,该下载中间件,不然不会生效 

ps:在其他地方自己揣摸……

//img.mukewang.com/szimg/62a3157a09a6a60014651177.jpg

0
1
bobby
如何将list类型的cookie转换为dict放入scrapy的请求中,课程中模拟登录就是讲解的这个内容啊
2022-06-12
共1条回复

Scrapy打造搜索引擎 畅销4年的Python分布式爬虫课

带你彻底掌握Scrapy,用Django+Elasticsearch搭建搜索引擎

5795 学习 · 6290 问题

查看课程