Skip to content

A React application utilizing APIs and web scraping for gathering news and data on specific stocks with the help of Node.js

Notifications You must be signed in to change notification settings

zakattack9/stockup

Repository files navigation

A React application leveraging APIs and web scraping for gathering stock news and data into one place

News Sources

Sometimes websites change their HTML layout or have counteractive measures in place to prevent web scraping their sites which will often cause the Stockup scraper to fail in scraping that specific site.

Scraped Sites Status
MarketWatch working
Bloomberg need to fix
Seeking Alpha works sometimes
InvestorPlace working
TheStreet need to fix
Motley Fool working
Barron's working
Zacks need to fix
Investopedia not currently added
CNBC not currently added
Yahoo Finance not currently added

Running Locally

Some news sources are not able to be scraped when this application is run in production, running locally will bypass a lot of these issues when web scraping certain sites.

  • navigate to project's root folder and run npm i
  • in project's root folder run npm i --s concurrently
  • navigate into client folder and run npm i
  • follow instructions to create .env file before running project
  • from the root folder, start the project with yarn dev or npm run dev

Creating .env File

This project uses two APIs to retrieve market data on stocks. Finnhub is used to gather stock data for the ETFs displayed on the homepage including all the fields on searched stocks (ex. high, low, market cap, etc.). marketstack gathers stock data for all stocks saved to the homepage; this is done because marketstack allows for the querying of multiple stock symbols in one API call helping to prevent rate limiting for Finnhub which only allows one stock symbol per call.

Setup

  • in the root project folder create a .env file
  • head over to marketstack and sign up for a free API key
  • head over to finnhub and sign up for a free API key
  • after receiving both API keys paste the following code into your .env file substituting each variable with their respective values
API_KEY=marketstack-api-key
API_KEY_FINNHUB=finnhub-api-key

Note: this project used to use worldtradingdata API to retrieve all stock data, however they were rebranded as marketstack API. To migrate over, simply sign up for a marketstack API key and swap that out with the old worldtradingdata API key.

Finnhub.io

Below are some of the Finnhub API endpoints that Stockup uses or plans to use in the future for retrieving stock data. Implementation can be found in ./utils/api.js

  • BASE_URL: https://finnhub.io/api/v1/
  • Basic Financials
    • endpoint: /stock/metric?symbol={ticker}&metric=all
    • targeted fields:
      • 52 Week High
      • 52 Week Low
      • 10 Day Average Trading Volume
  • Quote
    • endpoint: /quote?symbol={ticker}
    • targeted fields:
      • Open Price
      • Close Price
      • Day High
      • Day Low
      • Current Price
      • Previous Close Price
  • Company Profile 2
    • endpoint: /stock/profile2?symbol={ticker}
    • targeted fields:
      • Company Name
      • Market Capitalization
      • Exchange
      • Logo Image
  • Company News
    • endpoint: /company-news?symbol={ticker}&from={yyyy-mm-dd}&to={yyyy-mm-dd}
    • targeted fields:
      • Source
      • Headline
      • URL

Additional Resources

APIs

Web Scraping w/Cheerio Tutorials

Cloudscraper

Puppeteer

About

A React application utilizing APIs and web scraping for gathering news and data on specific stocks with the help of Node.js

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published