php中文网 | cnphp.com

 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
查看: 290|回复: 0

Xpath爬取当当网

[复制链接]

2708

主题

2715

帖子

9656

积分

管理员

Rank: 9Rank: 9Rank: 9

UID
1
威望
0
积分
6826
贡献
0
注册时间
2021-4-14
最后登录
2024-6-16
在线时间
680 小时
QQ
发表于 2022-12-8 15:52:44 | 显示全部楼层 |阅读模式
[mw_shl_code=python,true]import requests
import time
from lxml import etree
import json


def json_data_save(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
    }

    html = requests.get(url, headers=headers,timeout=2).content.decode('gbk')
    data = etree.HTML(html)
    lists = data.xpath('//ul[@class="bigimg"]/li')
    dicts = {}
    names = []
    athors = []
    publishs = []
    publish_times = []
    prices = []


    for lis in lists:
        name = lis.xpath('./a/@title')[0]
        names.append(name)
        athor = lis.xpath('./p[@class="search_book_author"]/span/a/@title')[0]
        athors.append(athor)
        publish = lis.xpath('./p[@class="search_book_author"]/span/a/@title')[-1]
        publishs.append(publish)
        publish_time = lis.xpath('./p[@class="search_book_author"]/span/text()')[-2].split('/')[-1]
        publish_times.append(publish_time)
        
        price = lis.xpath('./p[@class="price"]/span[@class="search_now_price"]/text()')[0].split('¥')[-1]
        prices.append(price)

    dicts['name'] = names
    dicts['athor'] = athors
    dicts['publish'] = publishs
    dicts['publish_time'] = publish_times
    dicts['price'] = prices
    # 保存数据为json格式
    try:
        with open('dsj_2002班_20200126057.json', 'a', encoding="utf-8") as f:
            f.write(json.dumps(dicts, ensure_ascii=False))
    except IOError as e:
        print(str(e))

    finally:
        f.close()




def open_json(path):
    try:
        with open(path, 'r', encoding='utf-8') as f:
            js_data = json.load(f)
            print('json数据:', js_data)

    except Exception as e:
        print(str(e))
    finally:
        f.close()


if __name__ == '__main__':
    url = 'http://search.dangdang.com/?key=python爬虫&act=input'
    json_data_save(url)
    time.sleep(2)
    path = 'dsj_2002班_20200126057.json'
    open_json(path)
[/mw_shl_code]





上一篇:卡尔曼滤波
下一篇:正则表达式爬取当当网并保存为csv
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

QQ|php中文网 | cnphp.com ( 赣ICP备2021002321号-2 )

GMT+8, 2024-6-18 14:37 , Processed in 0.428353 second(s), 36 queries , Gzip On.

Powered by Discuz! X3.4 Licensed

Copyright © 2001-2020, Tencent Cloud.

申明:本站所有资源皆搜集自网络,相关版权归版权持有人所有,如有侵权,请电邮(fiorkn@foxmail.com)告之,本站会尽快删除。

快速回复 返回顶部 返回列表