mpspider

Official account article crawling & generating kindle e-books

Crawl the historical articles of the official account, parse them into markdown files, generate gitbook projects, and finally generate kindle books.

PS :

1. ebook-convert dependency required
2. gitbook needs to be used in node 6.x version, 8.x cannot be used, and others have not been tested
3. To generate mobi, you need to configure book.json

Crawl method

Supports two crawling methods:

1. Starting with a summary article on the official account, some official accounts will have annual summary articles, such as this article
2. Use anyproxy as an agent to grab historical message articles of the official account, and ignore non-graphic and text articles or subtitles

PS : Summary article refers to the article page of a public account, such as the article "The Road to Architects"

Process introduction

1. Crawl articles
2. Analyze the external link "public account article" of the internal link of the article
3. Continue to grab external link articles
4. Replace external link articles to local relative address
5. Crawl pictures in the article
6. Replace the article picture to the local relative address
7. Generate gitbook project
8. Use gitbook+ebook-convert to generate a kindle file

Steps 1~6 are fully automatic, 7 depends on your own situation

Install

npm i mpspider -g

How to execute

 # 第一种方式
mpspider article https://mp.weixin.qq.com/s/CIPosICgva9haqstMDIHag -d dest_path
# 第二种方式，需要手动配置代理，点击公众号「查看历史文章」，详见下面介绍，支持手机微信和 pc 微信列表
mpspider proxy -d dest_path -p proxy_port

After crawling, a gitbook project will be created in dest_path

Generate e-books

Execute the command

 # 进入抓取后gitbook的地址
cd dest_path
# 创建readme.md，gitbook不创建会报错
touch README.md
# 有必要可以创建book.json，参考gitbook文档
gitbook serve
# 访问地址查看效果
# -------
# 生成电子书
gitbook mobi ./ name.mobi

How to configure anyproxy proxy to crawl https page

Configure anyproxy https certificate

Reference: http://anyproxy.io/cn/#%E8%AF%81%E4%B9%A6%E9%85%8D%E7%BD%AE

Start anyproxy

anyproxy --rule lib/anyproxyRule.js

Use the configuration file mpspider.config.js

Supported configuration items:

• book.json configuration items: 'author', 'title', 'description'
• summarySort: The article sorting function, the method is equivalent to the Array.sort writing method, and it is passed into item object, with options such as mid , title , content , release , uri , etc. release is the pinyin file name, and the default sorting is based on release
• filter : an article content filter function, filters the article list array items through items.filter(option.filter) . The item content includes: mid , title , content
• listFilter : List article filtering, only used in proxy mode, filter data according to the json object of the article list, the commonly used object fields are
  • app_msg_ext_info: author , title , copyright_stat , content_url , source_url , digest , content , cover , is_multi , etc.
  • comm_msg_info: datetime publish timestamp
• turndown : supports four options: keep , remove , rule , and plugins , corresponding to the four configuration items of turndown respectively
• afterConverter : turndown After converting html to markdown content, pass the content string into the function. After processing, return the processed string

Example:

 const turndownPluginGfm = require ( 'turndown-plugin-gfm' ) ;
module . exports = {
    filter : item => {
        if ( item . title . indexOf ( '广告' ) !== - 1 ) {
            return false ;
        }
        return true ;
    } ,
    turndown : {
        keep : 'span' ,
        remove : 'span' ,
        rule : {
            strikethrough : {
                filter : [ 'del' , 's' , 'strike' ] ,
                replacement : function ( content ) {
                    return '~' + content + '~' ;
                }
            }
        } ,
        plugins : [ turndownPluginGfm . gfm , turndownPluginGfm . tables ]
    } ,
    afterConverter : content => {
        return content . replace ( / <(.+?)> / g , ( i , m ) => {
            return `&lt; ${ m } >` ;
        } ) ;
    }
} ; 

Secondary development

After git clone source code, enter the folder and execute npm i

• index.js entry file, commander and ora for command processing
• getList.js extracts article list based on summary file
• proxySpider.js crawls according to anyproxy proxy
• dealMPList.js file used according to the proxy
• unfetchMids.js extracts the articles in the internal link in the article list
• getImages.js crawl the image address in the article and replace it with the local address
• createBook.js generates gitbook markdown file and summary.md to replace the content of the inner link

E-book dependency

• ebook-convert: brew install caskroom/cask/calibre
• gitbook: npm i gitbook-cli -g

Kindle effect screenshot

Add chicken legs to the author