example_jupyter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2b254c37",
   "metadata": {},
   "source": [
    "In this notebook, we import `end_to_end.py` to work with our end-to-end classifier. To run an example from the terminal, check `example_cmd.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "af07b6d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import end_to_end\n",
    "from end_to_end import math_classifier\n",
    "\n",
    "#Importing classes that we had defined ourselves.\n",
    "from Concat import ConcatModels\n",
    "from Reduction import clf_reduction\n",
    "\n",
    "#Instantiating \n",
    "clf=math_classifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca32c9bb",
   "metadata": {},
   "source": [
    "The user should only work with the method `predict` of the `math_classifier` class. This method returns 2-character and 3-character MSC classes as well as the primary arXiv category along with their respective probabilities based on an input to `predict` in one of the forms below: \n",
    "\n",
    "1) a string through the argument `text`; this typically should be the title or the abstract of a math-related paper<a name=\"cite_ref-a\"></a>[<sup>a</sup>](#cite_note-a) (or their concatenation); \n",
    "\n",
    "2) an arXiv identifier<a name=\"cite_ref-b\"></a>[<sup>b</sup>](#cite_note-b) through the argument `identifier`; the corresponding article is then fetched from arXiv and prediction is made based on its title and abstract;\n",
    "\n",
    "3) a positive integer through the argument `n_random`<a name=\"cite_ref-c\"></a>[<sup>c</sup>](#cite_note-c); the desired number of most recent math-related preprints are fetched, and their URL's are printed as well as the predictions made based on titles and abstracts.  \n",
    "\n",
    "If neither of these inputs is provided, five preprints will be scraped and their predicted classes will be printed along with their URL's.\n",
    "\n",
    "\n",
    "Loading appropriate classifiers, scraping arXiv preprints, preprocessing the text, vectorization etc. all take place inside the class. \n",
    "\n",
    "We have trained two neural network classifiers for predicting the primary arXiv category, a convolutional neural network and a transformer network. The latter is used by default. Inputting `use_transformer=False` to `math_classifier` upon instantiation results in loading the convolutional neural net instead. The only parts of the outputs of the `predict` method that change are the predicted primary arXiv category and its probability.  \n",
    "\n",
    "<a name=\"cite_note-a\"></a>[a](#cite_ref-a) Defined to be a preprint listed under one of the math archives or under the mathematical physics archive.\n",
    "\n",
    "<a name=\"cite_note-b\"></a>[b](#cite_ref-b) Both old and new [arXiv identifier schemes](https://info.arxiv.org/help/arxiv_identifier.html) are supported. Identifiers beginning with `arXiv:` or ending with the preprint version are supported too. If the input identifier is invalid or non-existent, a ValueError is raised.\n",
    "\n",
    "<a name=\"cite_note-c\"></a>[c](#cite_ref-c) We recommend not to set `n_random` larger than 20.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e76f1b8c",
   "metadata": {},
   "source": [
    "The cells below examine various modes of `predict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "60537627",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('11G', 0.7), ('14G', 0.18), ('14H', 0.6)],\n",
       " '2-character MSC': [('14', 0.6), ('11', 0.7)],\n",
       " 'Primary Category': ('math.NT', 0.89)}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(text=\"The Belyi Characterization of a Class of Modular Curves\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d50f38a",
   "metadata": {},
   "source": [
    "The same input text when the convolutional neural net is loaded instead of the transformer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2112116e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('11G', 0.7), ('14G', 0.18), ('14H', 0.6)],\n",
       " '2-character MSC': [('14', 0.6), ('11', 0.7)],\n",
       " 'Primary Category': ('math.NT', 0.59)}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_alternate=math_classifier(use_transformer=False)\n",
    "clf_alternate.predict(text=\"The Belyi Characterization of a Class of Modular Curves\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46f282cc",
   "metadata": {},
   "source": [
    "Switching back to the original classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eddbd5df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('35K', 0.92), ('35B', 0.41)],\n",
       " '2-character MSC': [('35', 0.92)],\n",
       " 'Primary Category': ('math.AP', 0.95)}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(text=\"Locally bounded, local weak solutions to a doubly nonlinear parabolic equation, which models the multi-phase transition of a material, is shown to be locally continuous. Moreover, an explicit modulus of continuity is given. The effect of the p-Laplacian type diffusion is also considered.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "68887cdf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('57N', 0.27),\n",
       "  ('52B', 0.15),\n",
       "  ('57M', 0.46),\n",
       "  ('57Q', 0.29)],\n",
       " '2-character MSC': [('57', 0.46), ('52', 0.15)],\n",
       " 'Primary Category': ('math.GT', 0.99)}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(text=\"Triangulated Manifolds with Few Vertices: Geometric 3-Manifolds We explicitly construct small triangulations for a number of well-known 3-dimensional manifolds and give a brief outline of some aspects of the underlying theory of 3-manifolds and its historical development.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "411685b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('32G', 0.68),\n",
       "  ('57M', 0.6),\n",
       "  ('37F', 0.84),\n",
       "  ('30F', 0.73)],\n",
       " '2-character MSC': [('57', 0.6), ('30', 0.73), ('37', 0.84), ('32', 0.68)],\n",
       " 'Primary Category': ('math.DS', 0.99)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(identifier=\"2202.00768\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "31f031aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'3-character MSC': [('35A', 0.47),\n",
       "  ('55N', 0.48),\n",
       "  ('32S', 0.15),\n",
       "  ('53D', 0.13),\n",
       "  ('14D', 0.12),\n",
       "  ('14F', 0.87)],\n",
       " '2-character MSC': [('14', 0.87),\n",
       "  ('32', 0.15),\n",
       "  ('53', 0.13),\n",
       "  ('35', 0.47),\n",
       "  ('55', 0.48)],\n",
       " 'Primary Category': ('math.AG', 0.92)}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(identifier=\"arXiv:math/0509440v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d4d40c14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paper 1: http://arxiv.org/abs/2407.03303v1\n",
      "{'3-character MSC': [('35J', 0.53), ('65N', 0.95), ('35B', 0.17), ('65M', 0.3)], '2-character MSC': [('65', 0.95), ('35', 0.53)], 'Primary Category': ('math.NA', 0.98)} \n",
      "\n",
      "Paper 2: http://arxiv.org/abs/2407.03083v1\n",
      "{'3-character MSC': [('49Q', 0.6), ('35Q', 0.34), ('35R', 0.97), ('65K', 0.16)], '2-character MSC': [('49', 0.6), ('65', 0.16), ('35', 0.97)], 'Primary Category': ('math.NA', 0.5)} \n",
      "\n",
      "Paper 3: http://arxiv.org/abs/2407.02656v1\n",
      "{'3-character MSC': [('62G', 0.14), ('90B', 0.21), ('62F', 0.09), ('90C', 0.37)], '2-character MSC': [('90', 0.37), ('62', 0.14)], 'Primary Category': ('math.OC', 0.54)} \n",
      "\n",
      "Paper 4: http://arxiv.org/abs/2407.02635v1\n",
      "{'3-character MSC': [('05C', 0.77)], '2-character MSC': [('05', 0.77)], 'Primary Category': ('math.CO', 0.92)} \n",
      "\n",
      "Paper 5: http://arxiv.org/abs/2407.02364v2\n",
      "{'3-character MSC': [('34A', 0.27), ('35R', 0.12)], '2-character MSC': [('34', 0.27), ('35', 0.12)], 'Primary Category': ('math.AP', 0.55)} \n",
      "\n",
      "Paper 6: http://arxiv.org/abs/2407.02102v1\n",
      "{'3-character MSC': [('05C', 0.99)], '2-character MSC': [('05', 0.99)], 'Primary Category': ('math.CO', 0.99)} \n",
      "\n",
      "Paper 7: http://arxiv.org/abs/2407.01935v1\n",
      "{'3-character MSC': [('05C', 0.96), ('68R', 0.15)], '2-character MSC': [('68', 0.15), ('05', 0.96)], 'Primary Category': ('math.CO', 0.67)} \n",
      "\n",
      "Paper 8: http://arxiv.org/abs/2407.01450v1\n",
      "{'3-character MSC': [('17B', 1.0), ('81R', 0.95), ('16T', 0.21)], '2-character MSC': [('81', 0.95), ('17', 1.0), ('16', 0.21)], 'Primary Category': ('math.QA', 0.87)} \n",
      "\n",
      "Paper 9: http://arxiv.org/abs/2407.01066v1\n",
      "{'3-character MSC': [('53D', 0.26)], '2-character MSC': [('53', 0.26)], 'Primary Category': ('math-ph', 0.59)} \n",
      "\n",
      "Paper 10: http://arxiv.org/abs/2407.00893v1\n",
      "{'3-character MSC': [('35R', 0.12), ('44A', 0.12), ('35B', 0.25), ('65M', 0.28)], '2-character MSC': [('65', 0.28), ('44', 0.12), ('35', 0.25)], 'Primary Category': ('math.NA', 0.59)} \n",
      "\n",
      "Paper 11: http://arxiv.org/abs/2407.00775v1\n",
      "{'3-character MSC': [('35A', 0.13), ('35J', 0.68), ('35B', 0.25)], '2-character MSC': [('35', 0.68)], 'Primary Category': ('math.AP', 0.85)} \n",
      "\n",
      "Paper 12: http://arxiv.org/abs/2407.00528v1\n",
      "{'3-character MSC': [('13D', 0.35), ('14M', 0.47), ('05E', 0.19), ('13F', 0.35), ('13H', 0.94), ('13C', 0.47), ('13A', 0.27), ('13P', 0.18), ('14H', 0.11)], '2-character MSC': [('14', 0.47), ('13', 0.94), ('05', 0.19)], 'Primary Category': ('math.AC', 0.97)} \n",
      "\n",
      "Paper 13: http://arxiv.org/abs/2407.01625v1\n",
      "{'3-character MSC': [('05C', 0.98)], '2-character MSC': [('05', 0.98)], 'Primary Category': ('math.CO', 0.99)} \n",
      "\n",
      "Paper 14: http://arxiv.org/abs/2407.00192v1\n",
      "{'3-character MSC': [('42B', 0.72), ('33C', 0.13), ('47G', 0.37), ('43A', 0.25), ('44A', 0.2), ('42A', 0.42), ('42C', 0.19)], '2-character MSC': [('42', 0.72), ('33', 0.13), ('47', 0.37), ('44', 0.2), ('43', 0.25)], 'Primary Category': ('math.CA', 0.78)} \n",
      "\n",
      "Paper 15: http://arxiv.org/abs/2406.19883v1\n",
      "{'3-character MSC': [('18B', 0.13), ('16D', 0.53), ('18A', 0.47), ('18E', 0.2), ('18G', 0.16)], '2-character MSC': [('18', 0.47), ('16', 0.53)], 'Primary Category': ('math.CT', 0.61)} \n",
      "\n",
      "Paper 16: http://arxiv.org/abs/2406.19865v2\n",
      "{'3-character MSC': [('47A', 0.16), ('32A', 0.35), ('46B', 0.43), ('30H', 0.1), ('30A', 0.58)], '2-character MSC': [('30', 0.58), ('47', 0.16), ('46', 0.43), ('32', 0.35)], 'Primary Category': ('math.CV', 0.61)} \n",
      "\n",
      "Paper 17: http://arxiv.org/abs/2406.19321v1\n",
      "{'3-character MSC': [('81T', 0.17), ('60G', 0.42), ('60H', 0.54)], '2-character MSC': [('60', 0.54), ('81', 0.17)], 'Primary Category': ('math.PR', 0.82)} \n",
      "\n",
      "Paper 18: http://arxiv.org/abs/2406.18798v2\n",
      "{'3-character MSC': [('94A', 0.48), ('11B', 0.93)], '2-character MSC': [('94', 0.48), ('11', 0.93)], 'Primary Category': ('math.PR', 0.84)} \n",
      "\n",
      "Paper 19: http://arxiv.org/abs/2406.18231v1\n",
      "{'3-character MSC': [('54H', 0.17), ('37B', 0.54), ('37A', 0.17)], '2-character MSC': [('37', 0.54), ('54', 0.17)], 'Primary Category': ('math.DS', 0.88)} \n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf.predict(n_random=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a1f8a43",
   "metadata": {},
   "source": [
    "Exception handling, an invalid arXiv identifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "138807ab",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "The provided identifier didn't match any arXiv preprint, make sure it is valid.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:621\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    620\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__try_parse_feed(\n\u001b[0;32m    622\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, try_index\u001b[38;5;241m=\u001b[39m_try_index\n\u001b[0;32m    623\u001b[0m     )\n\u001b[0;32m    624\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\n\u001b[0;32m    625\u001b[0m     HTTPError,\n\u001b[0;32m    626\u001b[0m     UnexpectedEmptyPageError,\n\u001b[0;32m    627\u001b[0m     requests\u001b[38;5;241m.\u001b[39mexceptions\u001b[38;5;241m.\u001b[39mConnectionError,\n\u001b[0;32m    628\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:664\u001b[0m, in \u001b[0;36mClient.__try_parse_feed\u001b[1;34m(self, url, first_page, try_index)\u001b[0m\n\u001b[0;32m    663\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m resp\u001b[38;5;241m.\u001b[39mstatus_code \u001b[38;5;241m!=\u001b[39m requests\u001b[38;5;241m.\u001b[39mcodes\u001b[38;5;241m.\u001b[39mOK:\n\u001b[1;32m--> 664\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(url, try_index, resp\u001b[38;5;241m.\u001b[39mstatus_code)\n\u001b[0;32m    666\u001b[0m feed \u001b[38;5;241m=\u001b[39m feedparser\u001b[38;5;241m.\u001b[39mparse(resp\u001b[38;5;241m.\u001b[39mcontent)\n",
      "\u001b[1;31mHTTPError\u001b[0m: Page request resulted in HTTP 400 (https://export.arxiv.org/api/query?search_query=&id_list=24013.07139&sortBy=relevance&sortOrder=descending&start=0&max_results=100)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:621\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    620\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__try_parse_feed(\n\u001b[0;32m    622\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, try_index\u001b[38;5;241m=\u001b[39m_try_index\n\u001b[0;32m    623\u001b[0m     )\n\u001b[0;32m    624\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\n\u001b[0;32m    625\u001b[0m     HTTPError,\n\u001b[0;32m    626\u001b[0m     UnexpectedEmptyPageError,\n\u001b[0;32m    627\u001b[0m     requests\u001b[38;5;241m.\u001b[39mexceptions\u001b[38;5;241m.\u001b[39mConnectionError,\n\u001b[0;32m    628\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:664\u001b[0m, in \u001b[0;36mClient.__try_parse_feed\u001b[1;34m(self, url, first_page, try_index)\u001b[0m\n\u001b[0;32m    663\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m resp\u001b[38;5;241m.\u001b[39mstatus_code \u001b[38;5;241m!=\u001b[39m requests\u001b[38;5;241m.\u001b[39mcodes\u001b[38;5;241m.\u001b[39mOK:\n\u001b[1;32m--> 664\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(url, try_index, resp\u001b[38;5;241m.\u001b[39mstatus_code)\n\u001b[0;32m    666\u001b[0m feed \u001b[38;5;241m=\u001b[39m feedparser\u001b[38;5;241m.\u001b[39mparse(resp\u001b[38;5;241m.\u001b[39mcontent)\n",
      "\u001b[1;31mHTTPError\u001b[0m: Page request resulted in HTTP 400 (https://export.arxiv.org/api/query?search_query=&id_list=24013.07139&sortBy=relevance&sortOrder=descending&start=0&max_results=100)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:621\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    620\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__try_parse_feed(\n\u001b[0;32m    622\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, try_index\u001b[38;5;241m=\u001b[39m_try_index\n\u001b[0;32m    623\u001b[0m     )\n\u001b[0;32m    624\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\n\u001b[0;32m    625\u001b[0m     HTTPError,\n\u001b[0;32m    626\u001b[0m     UnexpectedEmptyPageError,\n\u001b[0;32m    627\u001b[0m     requests\u001b[38;5;241m.\u001b[39mexceptions\u001b[38;5;241m.\u001b[39mConnectionError,\n\u001b[0;32m    628\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:664\u001b[0m, in \u001b[0;36mClient.__try_parse_feed\u001b[1;34m(self, url, first_page, try_index)\u001b[0m\n\u001b[0;32m    663\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m resp\u001b[38;5;241m.\u001b[39mstatus_code \u001b[38;5;241m!=\u001b[39m requests\u001b[38;5;241m.\u001b[39mcodes\u001b[38;5;241m.\u001b[39mOK:\n\u001b[1;32m--> 664\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(url, try_index, resp\u001b[38;5;241m.\u001b[39mstatus_code)\n\u001b[0;32m    666\u001b[0m feed \u001b[38;5;241m=\u001b[39m feedparser\u001b[38;5;241m.\u001b[39mparse(resp\u001b[38;5;241m.\u001b[39mcontent)\n",
      "\u001b[1;31mHTTPError\u001b[0m: Page request resulted in HTTP 400 (https://export.arxiv.org/api/query?search_query=&id_list=24013.07139&sortBy=relevance&sortOrder=descending&start=0&max_results=100)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "File \u001b[1;32m~\\Desktop\\MSC project\\end_to_end.py:179\u001b[0m, in \u001b[0;36mmath_classifier.predict\u001b[1;34m(self, n_random, text, identifier)\u001b[0m\n\u001b[0;32m    178\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 179\u001b[0m     paper\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mnext\u001b[39m(results)                         \u001b[38;5;66;03m#HTTP error may be encountered if the identifier is not valid.\u001b[39;00m\n\u001b[0;32m    180\u001b[0m     scraped_text\u001b[38;5;241m=\u001b[39mpaper\u001b[38;5;241m.\u001b[39mtitle\u001b[38;5;241m+\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m+\u001b[39mpaper\u001b[38;5;241m.\u001b[39msummary\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:574\u001b[0m, in \u001b[0;36mClient._results\u001b[1;34m(self, search, offset)\u001b[0m\n\u001b[0;32m    573\u001b[0m page_url \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_format_url(search, offset, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpage_size)\n\u001b[1;32m--> 574\u001b[0m feed \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_parse_feed(page_url, first_page\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m    575\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m feed\u001b[38;5;241m.\u001b[39mentries:\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:631\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    630\u001b[0m     logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGot error (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n\u001b[1;32m--> 631\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_parse_feed(\n\u001b[0;32m    632\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, _try_index\u001b[38;5;241m=\u001b[39m_try_index \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m    633\u001b[0m     )\n\u001b[0;32m    634\u001b[0m logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGiving up (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:631\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    630\u001b[0m     logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGot error (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n\u001b[1;32m--> 631\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_parse_feed(\n\u001b[0;32m    632\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, _try_index\u001b[38;5;241m=\u001b[39m_try_index \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m    633\u001b[0m     )\n\u001b[0;32m    634\u001b[0m logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGiving up (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:631\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    630\u001b[0m     logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGot error (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n\u001b[1;32m--> 631\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_parse_feed(\n\u001b[0;32m    632\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, _try_index\u001b[38;5;241m=\u001b[39m_try_index \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m    633\u001b[0m     )\n\u001b[0;32m    634\u001b[0m logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGiving up (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:635\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    634\u001b[0m logger\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGiving up (try \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m): \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m, _try_index, err)\n\u001b[1;32m--> 635\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m err\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:621\u001b[0m, in \u001b[0;36mClient._parse_feed\u001b[1;34m(self, url, first_page, _try_index)\u001b[0m\n\u001b[0;32m    620\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__try_parse_feed(\n\u001b[0;32m    622\u001b[0m         url, first_page\u001b[38;5;241m=\u001b[39mfirst_page, try_index\u001b[38;5;241m=\u001b[39m_try_index\n\u001b[0;32m    623\u001b[0m     )\n\u001b[0;32m    624\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (\n\u001b[0;32m    625\u001b[0m     HTTPError,\n\u001b[0;32m    626\u001b[0m     UnexpectedEmptyPageError,\n\u001b[0;32m    627\u001b[0m     requests\u001b[38;5;241m.\u001b[39mexceptions\u001b[38;5;241m.\u001b[39mConnectionError,\n\u001b[0;32m    628\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[1;32m~\\anaconda3\\Lib\\site-packages\\arxiv\\arxiv.py:664\u001b[0m, in \u001b[0;36mClient.__try_parse_feed\u001b[1;34m(self, url, first_page, try_index)\u001b[0m\n\u001b[0;32m    663\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m resp\u001b[38;5;241m.\u001b[39mstatus_code \u001b[38;5;241m!=\u001b[39m requests\u001b[38;5;241m.\u001b[39mcodes\u001b[38;5;241m.\u001b[39mOK:\n\u001b[1;32m--> 664\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(url, try_index, resp\u001b[38;5;241m.\u001b[39mstatus_code)\n\u001b[0;32m    666\u001b[0m feed \u001b[38;5;241m=\u001b[39m feedparser\u001b[38;5;241m.\u001b[39mparse(resp\u001b[38;5;241m.\u001b[39mcontent)\n",
      "\u001b[1;31mHTTPError\u001b[0m: Page request resulted in HTTP 400 (https://export.arxiv.org/api/query?search_query=&id_list=24013.07139&sortBy=relevance&sortOrder=descending&start=0&max_results=100)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[9], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m clf\u001b[38;5;241m.\u001b[39mpredict(identifier\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m24013.07139\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[1;32m~\\Desktop\\MSC project\\end_to_end.py:184\u001b[0m, in \u001b[0;36mmath_classifier.predict\u001b[1;34m(self, n_random, text, identifier)\u001b[0m\n\u001b[0;32m    182\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__predict_from_cleaned_text(cleaned_text)\n\u001b[0;32m    183\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m:                                         \u001b[38;5;66;03m#To catch cases where the iterable is empty.\u001b[39;00m\n\u001b[1;32m--> 184\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe provided identifier didn\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt match any arXiv preprint, make sure it is valid.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m    186\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m flag\u001b[38;5;241m==\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrandom\u001b[39m\u001b[38;5;124m'\u001b[39m:    \n\u001b[0;32m    187\u001b[0m     client \u001b[38;5;241m=\u001b[39m arxiv\u001b[38;5;241m.\u001b[39mClient()                     \u001b[38;5;66;03m#Construct the default API client.\u001b[39;00m\n",
      "\u001b[1;31mValueError\u001b[0m: The provided identifier didn't match any arXiv preprint, make sure it is valid."
     ]
    }
   ],
   "source": [
    "clf.predict(identifier=\"24013.07139\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}