使用sqlalchemy的scrapy数据库加载程序包装
scrapy_loaders的Python项目详细描述
废料装载机
废管道装载机
- 自由软件：麻省理工学院许可证
安装
pip install scrapy_loaders
在你的scrapy项目中（例如：spiderproject）
型号.py
fromsqlalchemyimport(Column,String,Text,)fromsqlalchemy.ext.declarativeimportdeclarative_baseDeclarativeBase=declarative_base()classItemModel(DeclarativeBase):__tablename__='table_name'id=Column('id',String(10),primary_key=True)name=Column('name',String(60))description=Column('description',Text())url=Column('url',Text())md5sum=Column('md5sum',String(45))......
设置.py
...# Postgres settings, check other SQLAlchemy settings if you wishDATABASE={'drivername':'postgresql+psycopg2','host':'localhost','port':'5432','username':'username','password':'password','database':'attack_mitre',}DECLARATIVE_BASE='SpiderProject.models.DeclarativeBase'...ITEM_PIPELINES={'SpiderProject.pipelines.SpiderProjectDbPipeline':300,}...
管道.py
fromSpiderProject.modelsimportItemModelfromscrapy_loaders.db_loadersimportDBLoaderfromscrapy_loaders.pipelinesimportDbPipelineclassItemLoader(DBLoader):model=ItemModelhash_fields=['name','description']update_fields=hash_fields+['md5sum']...classSpiderProjectDbPipeline(DbPipeline):db_loaders={'Item':ItemLoader,}...
功能
测试
TOdo:测试
nosetests --with-coverage --cover-inclusive --cover-package=scrapy_loaders --cover-html
欢迎加入QQ群-->： 979659372
