[GSoC'23] Welcome to OpenAPI Web Search Discussions!

[vinitshahdeo]

👋 Welcome!

We’re using Discussions as a place to connect with interested candidates for GSoC'23. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about how you plan to work on this project. Feel free to use this thread to ask your doubts.

PS: The previous discussion thread can be found here.

[priyanshu-kun]

@vinitshahdeo Let's build this thing as a potential builder for this project. you can count on me.

[destrex271]

So i was thinking that we should try implementing this service in rust. Recently github also migrated its search mechanism to rust and it has been amazing so far. Although there are some differences in what github implemented and what we want to implement but I think we can leverage the power that rust gives us to implement this system successfully.

[vinitshahdeo]

Hey @destrex271,

Feel free to use whatever tech stack you prefer—the ultimate goal is to develop an approach for discovering APIs from lesser-known sources.

[marcos5163]

Hi @vinitshahdeo ,
I'm Manish Kumar, interested in contributing to this project and to become a part of this amazing community, want to discuss that if we can use the existing framework like scrapy which has async support for web scrapping, or write it on our own...

[vinitshahdeo]

Hey @marcos5163,
We really don't believe in re-inventing the wheel. It's absolutely fine to start with an existing solution.

[priyanshu-kun]

@vinitshahdeo I'm just wondering about these things.
Question 1:

extracting API definitions from lesser-known sources.

should I find and list them on the proposal beforehand cause finding them on the lesser-known source could be a tedious task in my view? or should I only discuss the approach for finding API definitions in the proposal without mentioning the sources?

Question 2: Should I implement my own search algorithms from scratch I mean there are various types of search algorithms that could be used here, or can I use existing solutions like elasticsearch and solr for indexing and searching? as well as indexing the data?

[priyanshu-kun]

Yeah, writing your own search algorithm from scratch could be a tedious task it's better to use existing solutions that could save us a lot of time.

[vinitshahdeo]

Question 1: Should I find and list them on the proposal beforehand cause finding them on the lesser-known source could be a tedious task in my view? or should I only discuss the approach for finding API definitions in the proposal without mentioning the sources?

Discussing a thorough approach to finding lesser-known sources should be enough. Having a POC to find a few lesser-known sources is desirable.

Question 2: Should I implement my own search algorithms from scratch I mean there are various types of search algorithms that could be used here, or can I use existing solutions like elasticsearch and solr for indexing and searching? as well as indexing the data?

Using an existing search algorithm is completely fine. We can improve it later if needed.

[priyanshu-kun]

@vinitshahdeo

Having a POC to find a few lesser-known sources is desirable.

could you elaborate on that part?

[vinitshahdeo]

@priyanshu-kun Sharing a POC (Proof of concept) of your approach to finding lesser-known sources would be great. Rather than providing a pseudocode, it's better to have a working code that gives us some lesser-known sources which can later be extended to get more.

It's not necessary but it's desirable to provide a POC.

[priyanshu-kun]

I get it!

[marcos5163]

This is my understanding about the project (correct me if i'm wrong) and how i'm planing to work on the project,
so, first we need to write a service to find sources(seed URLs of lessor known sources) for focused crawling(openAPI defination) for user entered keywords for that we can use web analytics tools or content-based analytics..

Once we get the seedURLs, need to write one service to implement crawling and scraping in async manner, this can be simply implemented in python (becuase it provides wide range of libraries like beautifulSoup , scrapy etc). this service returns the openAPI definations, then the last part is to validate the definations and start indexing..

cc. @vinitshahdeo

[vinitshahdeo]

Hey @marcos5163, 👋

You understood it right except for the following:

user entered keywords — the user-entered keywords will come later. Once we have the dataset of APIs, users can search on them using keywords. The crawling needs to be done first without any user input.

[ankit-pn]

@vinitshahdeo will there be any sort of qualifications task for this project?

[Prajwalprakash3722]

Hello @vinitshahdeo, great to meet you here.

I think I have understood the problem statement pretty well, my questions are below (sorry if they sound dumb 🤣 )

• How are we going to find sources for focused crawling? (Do we have a dataset containing urls?)
• What do you exactly mean by Implement a search algorithm, maybe plug the dataset to elastic service?

I tried to make a architecture of this service roughly, correct me if I am wrong.

graph TD;
    fetchSources(Fetch Sources)-->crawlService(Crawl Service);
    crawlService --> scrapeService(Scrape Service);
    scrapeService --> validationService(Validation Service);
    validationService --> indexingService(Indexing Service);
    indexingService --> searchEngine(Search Engine);
    searchEngine --> displayResults(Display Results);

Loading

We fetch sources of OpenAPI definitions, which are then passed to a crawl service. The crawl service asynchronously crawls the web pages to extract OpenAPI definitions, which are then scraped by a separate scraping service. The extracted definitions are validated and then indexed by an indexing service. Finally, the search engine processes user queries and displays results through a user interface.

[Prajwalprakash3722]

Please reply as soon as possible, will help to track our applications.

[HimanshuS129]

How do you plan to - fetch these lesser known sources?

[vinitshahdeo]

Hey @Prajwalprakash3722

How are we going to find sources for focused crawling? (Do we have a dataset containing urls?)

There's no dataset available with us for API-related crawling.

What do you exactly mean by Implement a search algorithm, maybe plug the dataset to elastic service?

Yeah, Implementing a search algorithm means providing an API to search that can be powered by available solutions like Elasticsearch.

[siva-kumar-s]

@vinitshahdeo
Hi My Name is Sivakumar S, you can call me siva, i am a data collector in a Scrapy Team and i am quiet proficient in WebCrawler and WebScraping and a Junior full Stack web Developer.

• To create a crawler for Open API definition dataset from scratch to crawl the whole web it won't work, first every crawler need a Seed URL to crawl from the url we need to scrapy the valid and specified data , it takes more time to get the proper website that have the valid data and then to find the way to scrap not all the wesite allow the bot to scrap, we need to check the website is allow bot to scrape the data there is so many params need to check to scrap a website.

• So a better way to create a Open API definition dataset use Common Crawl dataset from the dataset we need to find which website provied Open API definition and then using common crawl api we need get Open API definition data availabel in the common crawl dataset and validate the data and then index it then using elastic search we can get data to the User.

if anything worng please correct me.

[vinitshahdeo]

Hey @siva-kumar-s

That's right. We will proceed using the Common Crawl dataset.

[saip007]

hi @vinitshahdeo | community,
My name is Prashanth, Is there any pre-tasks to complete or issues, how to start contributing the project

[vinitshahdeo]

Hello folks 👋

I have added a qualifying task for GSoC'23. Please check #2.

cc: @Prajwalprakash3722 @marcos5163 @priyanshu-kun @destrex271 @ankit-pn @saip007 @siva-kumar-s

[Prajwalprakash3722]

Hey @vinitshahdeo when is the deadline for #2, I'm slightly held up with university examinations

[vinitshahdeo]

@Prajwalprakash3722 Hey, all the best for your exams. Please try to complete it before you submit your application.

[vinitshahdeo]

Hello everyone,

I wrote a public blog post about this project idea - how OAWS can help in unleashing the power of OpenAPI.

Hope it helps—vinitshahdeo.dev/open-api-web-search

[ankit-pn]

Nice Blog!!

[ayan-de]

thanks for the blog

[ankit-pn]

Hey @vinitshahdeo , I have mailed you my draft proposal. Please have a look at it and provide any constructive criticism/feedback if needed.

[vinitshahdeo]

Hey @ankit-pn

Thanks for sharing, we're looking into this.

[priyanshu-kun]

Hey @vinitshahdeo, I mailed you my draft proposal, Please have a look at it and provide any constructive criticism/feedback if needed.

[priyanshu-kun]

@vinitshahdeo I mailed my updated proposal could you please review that? Please???

[vinitshahdeo]

Hey Priyanshu, Overall, the proposal looks good. Two of my teammates have reviewed it. Please make sure that you upload it on time.

[Prajwalprakash3722]

Shall I also send in my proposal? to this mail

[ankit-pn]

Hey vinit, please review my proposal also and send necessary feedback if required. I have mailed you and submitted it on GSoC portal also.

[vinitshahdeo]

Hey @ankit-pn, I have shared your proposal with the team. Thank you:)

[Prajwalprakash3722]

Hi @vinitshahdeo I sent my proposal over the email, please have a look whenever you get free :), Thank You

to this email : vinitshahdeo@gmail.com

[Prajwalprakash3722]

Hey @vinitshahdeo @HimanshuS129 @MikeRalphson

Today is the deadline for submission. I would appreciate it if any one of you could provide me with feedback on my proposal as soon as possible. Thank you.