Mac系统安装python3.7.3并安装Scrapy爬虫框架

作者: admin 分类: Scrapy 发布时间: 2019-03-31 16:49  阅读: 784 views

最近要做一些爬取数据的工作, 综合网上的信息python语言是不错的选择, 其中scrapy也是很多人的不二选择。所以这里学习记录下,做些数据爬取的准备

 

1先去下载python3.7.x版本(mac默认2.7安装报错)

地址在这里:https://www.python.org/downloads/mac-osx/

 

2.安装后在命令行输入  python3 ,出现以下信息表示已经安装成功

Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

3. 执行命令  pip3 install Scrapy, 进行scrapy爬虫框架的安装

Collecting Scrapy
  Using cached https://files.pythonhosted.org/packages/3e/45/414e87ac8209d537c91575538c5307c20217a6943f555e0ee39f6db4bb0f/Scrapy-1.6.0-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl
Collecting service-identity (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting six>=1.5.2 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting lxml (from Scrapy)
  Downloading https://files.pythonhosted.org/packages/45/6c/436a534dca42f7982ba793983353035d117ab70541266704974efa323ade/lxml-4.3.3-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (8.7MB)
    100% |████████████████████████████████| 8.7MB 27kB/s 
Collecting w3lib>=1.17.0 (from Scrapy)
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/w3lib/
  Using cached https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
Collecting parsel>=1.5 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy)
  Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting queuelib (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting Twisted>=13.1.0 (from Scrapy)
  Downloading https://files.pythonhosted.org/packages/5d/0e/a72d85a55761c2c3ff1cb968143a2fd5f360220779ed90e0fadf4106d4f2/Twisted-18.9.0.tar.bz2 (3.1MB)
    100% |████████████████████████████████| 3.1MB 15kB/s 
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/incremental/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056) -- Some packages may not be found!
    Couldn't find index page for 'incremental' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056) -- Some packages may not be found!
    No local packages or working download links found for incremental>=16.10.1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/n6/dv5ctl_s05s4z1x17t7c7n8c0000gn/T/pip-install-vdefl_eg/Twisted/setup.py", line 20, in <module>
        setuptools.setup(**_setup["getSetupArgs"]())
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
        replace_conflicting=True,
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/n6/dv5ctl_s05s4z1x17t7c7n8c0000gn/T/pip-install-vdefl_eg/Twisted/

原因是incremental 依赖组件没有安装,

4. 安装 incremental组件

pip3 install --upgrade incremental  -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com
Looking in indexes: http://pypi.douban.com/simple/
Collecting incremental
  Downloading http://pypi.doubanio.com/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl
Installing collected packages: incremental
Successfully installed incremental-17.5.0

注意这里是 pip3

5.  重新安装 Scrapy爬虫框架

pip3 install Scrapy
Collecting Scrapy
  Using cached https://files.pythonhosted.org/packages/3e/45/414e87ac8209d537c91575538c5307c20217a6943f555e0ee39f6db4bb0f/Scrapy-1.6.0-py2.py3-none-any.whl
Collecting parsel>=1.5 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting w3lib>=1.17.0 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl
Collecting Twisted>=13.1.0 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/5d/0e/a72d85a55761c2c3ff1cb968143a2fd5f360220779ed90e0fadf4106d4f2/Twisted-18.9.0.tar.bz2
Collecting cssselect>=0.9 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl
Collecting six>=1.5.2 (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting service-identity (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
Collecting lxml (from Scrapy)
  Using cached https://files.pythonhosted.org/packages/45/6c/436a534dca42f7982ba793983353035d117ab70541266704974efa323ade/lxml-4.3.3-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting zope.interface>=4.4.2 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/14/22/ca298e4c576981830be5e706049e9a3c091edf9dc489a00e3c1ec8c501d3/zope.interface-4.6.0-cp37-cp37m-macosx_10_6_intel.whl (140kB)
    100% |████████████████████████████████| 143kB 46kB/s 
Collecting constantly>=15.1 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/b9/65/48c1909d0c0aeae6c10213340ce682db01b48ea900a7d9fce7a7910ff318/constantly-15.1.0-py2.py3-none-any.whl
Requirement already satisfied: incremental>=16.10.1 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from Twisted>=13.1.0->Scrapy) (17.5.0)
Collecting Automat>=0.3.0 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/a3/86/14c16bb98a5a3542ed8fed5d74fb064a902de3bdd98d6584b34553353c45/Automat-0.7.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/a7/b6/84d0c863ff81e8e7de87cff3bd8fd8f1054c227ce09af1b679a8b17a9274/hyperlink-18.0.0-py2.py3-none-any.whl
Collecting PyHamcrest>=1.9.0 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl (52kB)
    100% |████████████████████████████████| 61kB 12kB/s 
Collecting attrs>=17.4.0 (from Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/23/96/d828354fa2dbdf216eaa7b7de0db692f12c234f7ef888cc14980ef40d1d2/attrs-19.1.0-py2.py3-none-any.whl
Collecting pyasn1 (from service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 19kB/s 
Collecting pyasn1-modules (from service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 14kB/s 
Collecting cryptography (from service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/12/cd/f2c4dac6e48add01d4575e8fd8ca9a16f4236afe321e6e5ffdc7a4725b55/cryptography-2.6.1-cp34-abi3-macosx_10_6_intel.whl (1.7MB)
    100% |████████████████████████████████| 1.7MB 19kB/s 
Requirement already satisfied: setuptools in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->Scrapy) (40.8.0)
Collecting idna>=2.5 (from hyperlink>=17.1.1->Twisted>=13.1.0->Scrapy)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 15kB/s 
Collecting asn1crypto>=0.21.0 (from cryptography->service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    100% |████████████████████████████████| 102kB 11kB/s 
Collecting cffi!=1.11.3,>=1.8 (from cryptography->service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/6b/af/e39e89ecbfb676c6aa5855b1795bb207f4005187495c8d40e26b5fb51bfa/cffi-1.12.2-cp37-cp37m-macosx_10_9_x86_64.whl (168kB)
    100% |████████████████████████████████| 174kB 11kB/s 
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->service-identity->Scrapy)
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    100% |████████████████████████████████| 163kB 11kB/s 
Installing collected packages: lxml, six, w3lib, cssselect, parsel, PyDispatcher, zope.interface, constantly, attrs, Automat, idna, hyperlink, PyHamcrest, Twisted, pyasn1, pyasn1-modules, asn1crypto, pycparser, cffi, cryptography, service-identity, queuelib, pyOpenSSL, Scrapy
  Running setup.py install for PyDispatcher ... done
  Running setup.py install for Twisted ... done
  Running setup.py install for pycparser ... done
Successfully installed Automat-0.7.0 PyDispatcher-2.0.5 PyHamcrest-1.9.0 Scrapy-1.6.0 Twisted-18.9.0 asn1crypto-0.24.0 attrs-19.1.0 cffi-1.12.2 constantly-15.1.0 cryptography-2.6.1 cssselect-1.0.3 hyperlink-18.0.0 idna-2.8 lxml-4.3.3 parsel-1.5.1 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.4 pycparser-2.19 queuelib-1.5.0 service-identity-18.1.0 six-1.12.0 w3lib-1.20.0 zope.interface-4.6.0

如果安装过程中出现类似 “TimeOut”的相关提示,说明网络不好,一些依赖组件没有下载成功,重新安装即可

6. 命令行 输入 scrapy,出现以下错误

scrapy
-bash: scrapy: command not found

7. 退出terminal,重新打开后输入  scrapy

deathearth:~ chenhailong$ scrapy
Scrapy 1.6.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

安装成功,开始你的爬虫之旅吧.


   原创文章,转载请标明本文链接: Mac系统安装python3.7.3并安装Scrapy爬虫框架

如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!

发表评论

电子邮件地址不会被公开。 必填项已用*标注

更多阅读