Skip to content

Commit

Permalink
Add parser.md to api-services
Browse files Browse the repository at this point in the history
  • Loading branch information
TueeNguyen committed Apr 7, 2022
1 parent c2d7d63 commit 89bfae1
Showing 1 changed file with 11 additions and 0 deletions.
11 changes: 11 additions & 0 deletions src/web/docusaurus/docs/api-services/parser.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
### Parser Service

Telescope's back-end began as a single, monolithic node.js app. During the 2.0 release, much of the back-end was split into separate microservices. Parts of the legacy back-end code were still in use, see `src/backend/*` which were removed in 3.0 release, telescope now consists of only microservices/workspaces.

The current system uses the parser service in order to run the feed parser and feed queue, see [`src/api/parser/data/feed.js`](https://github.com/Seneca-CDOT/telescope/blob/master/src/api/parser/src/data/feed.js). The processed feeds and posts are then stored in `Redis` (cache) and `Elasticsearch` (indexing) databases, and various microservices use these in order to get their data.

Telescope's data model is built on Feeds and Posts. A feed represents an RSS/Atom feed, and includes metadata about a particular blog (e.g., URL, author, etc) as well as URLs to individual Posts. A Post includes metadata about a particular blog post (e.g., URL, date created, date updated, etc).

To run the service, use `pnpm services:start parser` or `pnpm services:start` command. When it runs, the logs show information about feeds being parsed in real-time, which continues forever.

The parser get all the feed urls and authors from Supabase database, parses them, creates `Feed` objects and puts them into a queue managed by [Bull](https://github.com/OptimalBits/bull) and backed by `Redis`. These are then processed in [`src/api/parser/feed/processor.js`](https://github.com/Seneca-CDOT/telescope/blob/master/src/api/parser/src/feed/processor.js) in order to download the individual Posts, which are also cached in Redis.

0 comments on commit 89bfae1

Please sign in to comment.