goodreads user scraper

Goodreads User Scraper

Scrape Goodreads User Data: Profile, Book Shelves, Books, Authors

     

Contents

• Usage
• Arguments
  • --user_id
  • --output_dir
  • --skip_user_info
  • --skip_shelves
  • --skip_authors
• Troubleshooting
• Development
• Publishing

Usage

Using pip:

pip install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Using pipx:

pipx run goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Arguments

--user_id

• Description: The user whose data should be scraped. Find your user id using these directions.
• Required: Yes

--output_dir

• Description: The directory where all scraped data will be output.
• Required: No
• Default: goodreads-data

--skip_user_info

• Description: Whether the script should skip scraping user information.
• Required: No
• Default: False

--skip_shelves

• Description: Whether the script should skip scraping shelves.
• Required: No
• Default: False

--skip_authors

• Description: Whether the script should skip scraping authors.
• Required: No
• Default: False

Troubleshooting

Ensure that your profile is viewable by anyone:

1. Navigate to the Goodreads Account Settings page
2. Click on the Account & notifications tab
3. In the Privacy section, under Who Can View My Profile, select "anyone". Save your account settings.

Development

1. Clone the GitHub repository

   git clone https://github.com/YashTotale/goodreads-user-scraper.git

2. Run the install script

   sh scripts/install.sh

3. Make changes

4. Run the test script

   sh scripts/test.sh

Publishing

1. Create .env

   TWINE_USERNAME=<foo>
   TWINE_PASSWORD=<bar>
   

2. Run the publish script

   sh scripts/publish.sh <patch|minor|major>