Analysis_Preannotation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Identifying People in CDCS Metadata Descriptions\n",
    "Run named entity recognition (NER) on the metadata descriptions extracted from the CDCS' online archival catalog.\n",
    "\n",
    "The code in this Jupyter Notebook is part of a PhD project to create a gold standard dataset labeled for gender biased language, on which a classifier can be trained to identify gender bias in archival metadata descriptions.  \n",
    "\n",
    "This project is focused on the English language and archival institutions in the United Kingdom.\n",
    "\n",
    "* Author: Lucy Havens\n",
    "* Date: November 2020 - February 2021\n",
    "* Project: PhD Case Study 1\n",
    "* Data Provider: [ArchivesSpace](https://archives.collections.ed.ac.uk/), Centre for Research Collections, University of Edinburgh\n",
    "\n",
    "***\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "  [I. Corpus Statistics](#corpus-stats)\n",
    "\n",
    "  [II. Named Entity Recognition with SpaCy](#ner)\n",
    "  \n",
    "  [III. Checking for Duplicate Descritions](#check-dups)\n",
    "  \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Already have en_core_web_sm\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import sent_tokenize\n",
    "# nltk.download('punkt')\n",
    "from nltk.corpus import PlaintextCorpusReader\n",
    "# nltk.download('averaged_perceptron_tagger')\n",
    "from nltk.corpus import stopwords\n",
    "# nltk.download('stopwords')\n",
    "from nltk.tag import pos_tag\n",
    "import string\n",
    "import csv\n",
    "import re\n",
    "\n",
    "import spacy\n",
    "from spacy import displacy\n",
    "from collections import Counter\n",
    "try:\n",
    "    import en_core_web_sm\n",
    "except ImportError:\n",
    "    print(\"Downlading en_core_web_sm model\")\n",
    "    import sys\n",
    "    !{sys.executable} -m spacy download en_core_web_sm\n",
    "else:\n",
    "    print(\"Already have en_core_web_sm\")\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"corpus-stats\"></a>\n",
    "## I. Corpus Statisctics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "descs = PlaintextCorpusReader(\"../AnnotationData/descriptions_by_fonds_split_with_ann/descriptions_by_fonds_split_with_ann/\", \".+\\.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Identifier', ':', 'AA4', 'Title', ':', 'Papers', 'of', 'Rev', 'Prof', 'John', 'McIntyre', '(', '1916', '-', '2005', ')', 'Scope', 'and', 'Contents', ':']\n"
     ]
    }
   ],
   "source": [
    "tokens = descs.words()\n",
    "print(tokens[0:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Identifier', ':', 'AA4'], ['Title', ':', 'Papers', 'of', 'Rev', 'Prof', 'John', 'McIntyre', '(', '1916', '-', '2005', ')'], ['Scope', 'and', 'Contents', ':', 'Sermons', 'and', 'addresses', ',', '1940', '-', '1997', ';', 'personal', 'papers', ',', '1932', '-', '2006', ';', 'lectures', ',', '1946', '-', '1990', ';', 'class', 'notes', 'and', 'lecture', 'notes', ',', '1932', '-', '1941', ';', 'correspondence', ',', '1979', '-', '1983', ';', 'papers', 'relating', 'to', 'the', 'Dean', 'of', 'the', 'Thistle', ',', '1974', '-', '1980', ';', 'newspaper', 'cuttings', ',', '1945', '-', '1982', ';', 'publications', 'and', 'articles', ',', '1929', '-', '1987', ';', 'essays', ',', '1935', '-', '1979', ',', 'audio', 'cassette', 'tapes', ',', '1987', 'and', 'papers', 'relating', 'to', 'television', 'broadcasts', ',', '1960', '-', '1967', '.', 'See', 'also', 'External', 'Documents', '(', 'below', ').'], ['Biographical', '/', 'Historical', ':', 'John', 'McIntyre', '(', '1916', '-', '2005', ')', 'was', 'born', 'in', 'Glasgow', ',', 'educated', 'at', 'Bathgate', 'Academy', 'and', 'then', 'Edinburgh', 'University', 'where', 'he', 'graduated', 'in', 'philosophy', 'and', 'then', 'in', 'divinity', '.'], ['In', '1943', 'he', 'became', 'the', 'minister', 'of', 'the', 'Ayrshire', 'village', 'of', 'Fenwick', 'before', 'leaving', 'Scotland', 'in', '1945', 'to', 'teach', 'theology', 'at', 'St', 'Andrew', \"'\", 's', 'College', 'in', 'the', 'University', 'of', 'Sydney', '.']]\n"
     ]
    }
   ],
   "source": [
    "sentences = descs.sents()\n",
    "print(sentences[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Estimated...\n",
      "  Characters: 13739019\n",
      "  Tokens: 2754044\n",
      "  Sentences: 156124\n",
      "  Files: 3649\n"
     ]
    }
   ],
   "source": [
    "def corpusStatistics(plaintext_corpus_read_lists):\n",
    "    total_chars = 0\n",
    "    total_tokens = 0\n",
    "    total_sents = 0\n",
    "    total_files = 0\n",
    "    \n",
    "    # fileids are the TXT file names in the nls-text-ladiesDebating folder:\n",
    "    for fileid in plaintext_corpus_read_lists.fileids():\n",
    "        total_chars += len(plaintext_corpus_read_lists.raw(fileid))\n",
    "        total_tokens += len(plaintext_corpus_read_lists.words(fileid))\n",
    "        total_sents += len(plaintext_corpus_read_lists.sents(fileid))\n",
    "        total_files += 1\n",
    "    \n",
    "    print(\"Total Estimated...\")\n",
    "    print(\"  Characters:\", total_chars)\n",
    "    print(\"  Tokens:\", total_tokens)\n",
    "    print(\"  Sentences:\", total_sents)\n",
    "    print(\"  Files:\", total_files)\n",
    "\n",
    "corpusStatistics(descs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Estimated Words: 2006380\n"
     ]
    }
   ],
   "source": [
    "words = [t for t in tokens if t.isalpha()]\n",
    "print(\"Total Estimated Words:\",len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_exclude = set((stopwords.words(\"english\")) + [\"Title\", \"Identifier\", \"Scope\", \n",
    "                \"Contents\", \"Biograhical\", \"Historical\", \"Processing\", \"Information\"]\n",
    "                )\n",
    "words = [w for w in words if w not in to_exclude]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Words Excluding Metadata Field Names: 1273237\n"
     ]
    }
   ],
   "source": [
    "print(\"Total Words Excluding Metadata Field Names:\",len(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ner\"></a>\n",
    "## II. Name Entity Recognition with spaCy\n",
    "Run named entity recognition (NER) to estimate the names in the dataset and get a sense for the value in manually labeling names during the annotation process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileids = descs.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = []\n",
    "for fileid in fileids:\n",
    "    file = descs.raw(fileid)\n",
    "    sentences += nltk.sent_tokenize(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "person_list = []\n",
    "for s in sentences:\n",
    "    s_ne = nlp(s)\n",
    "    for entity in s_ne.ents:\n",
    "        if entity.label_ == 'PERSON':\n",
    "            person_list += [entity.text] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28625\n"
     ]
    }
   ],
   "source": [
    "unique_persons = list(set(person_list))\n",
    "print(len(unique_persons))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Twynholm Manse',\n",
       " 'Owen',\n",
       " 'Christopher Dawson',\n",
       " 'Archibald Lindesay',\n",
       " 'William Fordaill',\n",
       " 'Christmas Card',\n",
       " 'Headteacher',\n",
       " 'Goulstonian Lecturer',\n",
       " 'Nicholas Thoums',\n",
       " 'Melvin J.',\n",
       " 'Lila Black',\n",
       " 'H. Tregellas',\n",
       " 'Mary Tovey',\n",
       " 'Children',\n",
       " 'Ein Beitrag',\n",
       " 'A. McLaren',\n",
       " 'Tayne',\n",
       " 'William S. Morrison',\n",
       " 'John Brownlee',\n",
       " 'J.H Sang',\n",
       " 'Group Almoner',\n",
       " 'Mme Laure Mitchell',\n",
       " 'Daemonologie',\n",
       " 'S. Karger',\n",
       " 'Steve Byrne',\n",
       " 'Amal Chandra Chaudhuri',\n",
       " 'Fallowis',\n",
       " 'Chekhov',\n",
       " 'Henry Tristram',\n",
       " 'Andrew More',\n",
       " 'Mrs Adele Koestler',\n",
       " 'Sewaltoun',\n",
       " 'Walter Harris',\n",
       " 'Willie',\n",
       " 'Steen Willasden',\n",
       " 'Donald Watterson',\n",
       " 'George Low',\n",
       " 'MD PhD',\n",
       " 'Patrick Hepburn',\n",
       " 'Bindan Blood',\n",
       " 'Hugh Macfarlane',\n",
       " 'Simon',\n",
       " 'F. Duthie',\n",
       " 'D5 Donovan',\n",
       " 'William Rerik',\n",
       " 'Jeremy Bentham',\n",
       " 'Raphael Falk',\n",
       " 'Curly Lop-Eared Lincoln Sow',\n",
       " 'Thomas Dikkesone',\n",
       " 'P. Brown',\n",
       " 'David Weir',\n",
       " 'Mula Rosoff',\n",
       " 'Belech De',\n",
       " 'Richard Fitz John',\n",
       " 'Gilbert Lawdir',\n",
       " 'Chesney',\n",
       " 'Begri',\n",
       " 'Alexander Petrovich Karpinsky(1894-1903)Aleksei',\n",
       " 'Northumberland Tests - Arithmetic',\n",
       " 'Pere Lachaise',\n",
       " 'Paul Wittgenstein',\n",
       " 'John Castarris',\n",
       " 'Peter Cocherane',\n",
       " 'Frederik Schien',\n",
       " 'John Alexander Lamb',\n",
       " 'Charles-Bertin-Gaston',\n",
       " 'David Donaldsone',\n",
       " 'Jeremy Montagu',\n",
       " 'William Stirlng Maxwell',\n",
       " 'Annie Macpherson',\n",
       " 'Maria Murray',\n",
       " 'Stewart Hore-Brown',\n",
       " 'Kirk Newton',\n",
       " 'G Macmullen',\n",
       " 'Thomas Inche',\n",
       " 'Gill Sans',\n",
       " 'Peter Hall',\n",
       " 'Alexander Hunman',\n",
       " 'William Fullarton',\n",
       " 'Paul IV',\n",
       " 'Hypothec',\n",
       " 'Oliver Burght',\n",
       " 'John Jaksone',\n",
       " 'Henry Huxley',\n",
       " 'Jeannie Wilson',\n",
       " 'Alexander H. Main\\n\\nBiographical / Historical',\n",
       " 'Blackett',\n",
       " 'Robert Menzies',\n",
       " 'Bakewell',\n",
       " 'Roma',\n",
       " 'Tewkesbury',\n",
       " 'Clara Gould ]\\n\\n',\n",
       " 'Glamis Castle',\n",
       " 'William Edward Norris',\n",
       " 'recipient[Sadleir',\n",
       " 'Iacobi Douglas',\n",
       " 'Adam Landels',\n",
       " 'Sanctorum',\n",
       " 'Royal',\n",
       " 'Robert Struther',\n",
       " 'Robert Lyndsay',\n",
       " '278.CorrespondenceFraser',\n",
       " 'A. Holland',\n",
       " 'Mary M\\n\\n',\n",
       " 'B68 Boyd',\n",
       " 'George Hutchesone',\n",
       " \"Lady Lothian'\",\n",
       " 'De Swart',\n",
       " 'Johannes De Lestowicz',\n",
       " 'Hansi Nagore',\n",
       " 'John Hirde',\n",
       " 'Maud C. Gilton',\n",
       " 'James Scott Bowerbank',\n",
       " 'Nikolai Krementsov',\n",
       " 'Quad Concert',\n",
       " 'Andrew Lindsay',\n",
       " 'Unwin',\n",
       " 'Sharrpey-Schaffer',\n",
       " 'Sheeps Fescue',\n",
       " 'John Braine',\n",
       " 'Mainz',\n",
       " 'Lamancha',\n",
       " 'Margaret Balfour',\n",
       " 'Jacobite',\n",
       " 'Catherine Cowan',\n",
       " 'Principal Clerk',\n",
       " 'Hugh Congilton',\n",
       " 'Thomas Loveday',\n",
       " 'Bruce Maclean',\n",
       " 'Bosendorfer',\n",
       " 'William Home Cook',\n",
       " 'D de Winton',\n",
       " 'St Vincent',\n",
       " 'Maurice Duhamel',\n",
       " 'George William Stow',\n",
       " 'Alexander Murray\\n\\n',\n",
       " 'Helen Chalmers',\n",
       " 'Alexander Caboll',\n",
       " 'Nancy',\n",
       " 'David Brewster',\n",
       " 'A.H. Wratislaw',\n",
       " 'Russell F Gwinnell',\n",
       " 'Bursae Mucosae',\n",
       " 'William Heron',\n",
       " 'van Zwichem',\n",
       " 'Winston Scott Memorial Lecture',\n",
       " 'Gershwin',\n",
       " 'George Steuart Mackenzie of Coul, Baronet',\n",
       " 'F. Tessier',\n",
       " \"Thomas M'Vcthee\",\n",
       " 'Miss Weisse',\n",
       " 'Anna',\n",
       " \"D. Mem'a\",\n",
       " 'Julius Wellhausen',\n",
       " 'Alexander Morton',\n",
       " 'Hugh Gourlaye',\n",
       " 'William Blakeley',\n",
       " 'Bad Scuol',\n",
       " \"O'Cailleach\",\n",
       " 'William Jobsoun',\n",
       " 'Robert D. Underwood',\n",
       " \"Hans Gals's\",\n",
       " 'Haggard',\n",
       " 'Kristy Davis',\n",
       " 'Stokes',\n",
       " 'Belfestinio',\n",
       " 'MS)Larkin',\n",
       " 'Thomas Lyndesay',\n",
       " 'Catherine II',\n",
       " 'Cert MD',\n",
       " 'William Ainslie',\n",
       " 'John Warden',\n",
       " 'Mrs Hart',\n",
       " 'Walter\\n\\n',\n",
       " 'Burdon Miller',\n",
       " 'Adele\\n\\nScope and Contents',\n",
       " 'James Simpson',\n",
       " 'Davis',\n",
       " 'Ho',\n",
       " 'Expert',\n",
       " 'George Narne',\n",
       " 'John Tyry',\n",
       " 'Manjra Singhai',\n",
       " 'Robert Craufurd',\n",
       " 'David Magee',\n",
       " 'C. R. M. Banks\\n\\n',\n",
       " 'Miss Cicely D Fawcet',\n",
       " 'William Neische',\n",
       " \"John M'Kennedy\",\n",
       " 'William Vobstar',\n",
       " 'Minnie Punton',\n",
       " 'Selkirk Players',\n",
       " 'Frozen Meat\\n\\n',\n",
       " 'Pamela Margaret Kingston',\n",
       " 'John Forrester',\n",
       " 'Arnold Bax',\n",
       " 'Won',\n",
       " 'William of Borthik',\n",
       " 'Ian M. MacIntosh',\n",
       " 'James Reid\\n\\n',\n",
       " 'William Pettillok',\n",
       " 'James Millar Lowe',\n",
       " 'Secondary Keys',\n",
       " 'Edward Percival Wright',\n",
       " 'William Merryman',\n",
       " 'Statistical Hypotheses',\n",
       " 'Idiocy\\n\\n',\n",
       " 'J Needham',\n",
       " 'Queen Empress',\n",
       " 'Paul Hindemith',\n",
       " 'Kert',\n",
       " \"James Gillespie's\",\n",
       " 'B.Litt',\n",
       " 'William Thorn',\n",
       " 'John Pitcairn Mackintosh MP',\n",
       " 'Confidential Returns',\n",
       " 'C. H. Bond\\n\\n',\n",
       " 'Adam House',\n",
       " 'Groningen',\n",
       " 'Michael Donaldsoun',\n",
       " 'Robert Lawdyr',\n",
       " 'Bacterias',\n",
       " 'H. A. Scott\\n\\n',\n",
       " 'James Matthew',\n",
       " 'Prosper Bysshe',\n",
       " 'John Tulloch',\n",
       " 'Margaret Creichtoun',\n",
       " 'Faked Horns',\n",
       " 'Patrick Sydsarf',\n",
       " \"Charles V'\",\n",
       " 'Maurice of Ewrth',\n",
       " 'Dalyhell',\n",
       " 'Henry Gannett',\n",
       " 'Lauckner',\n",
       " 'Margaret G. Winfield',\n",
       " 'Thomas Bancroft',\n",
       " 'H61 Hindley',\n",
       " 'Leigh Hoadley',\n",
       " 'Alexander Pastoune',\n",
       " 'R. Thompson',\n",
       " 'Reg H.',\n",
       " 'Anthony H',\n",
       " 'John Wain',\n",
       " 'Basil Urwin',\n",
       " 'F Schrader',\n",
       " 'Suton Coldfield',\n",
       " 'David Hoppringyll',\n",
       " 'Bertram Christian',\n",
       " 'David Bell',\n",
       " 'Robert Raith',\n",
       " 'James Rick',\n",
       " \"John M'Gachin\",\n",
       " 'Allan McNeil',\n",
       " 'James Menzeis',\n",
       " 'Monk Bretton',\n",
       " 'Jura',\n",
       " 'Robert Homer Burns',\n",
       " 'Elizabeth Lyon',\n",
       " \"St James'\",\n",
       " 'Whitlock',\n",
       " 'John George Bartholomew',\n",
       " 'John Kylpatryk',\n",
       " 'Holzinger',\n",
       " 'Markham',\n",
       " \"Henry VIII's\",\n",
       " 'Padhye',\n",
       " 'Trumps',\n",
       " 'Zihuatanejo LSD',\n",
       " 'George Henschel',\n",
       " 'Librarian',\n",
       " 'Miss Stevenson',\n",
       " 'Lewis Lukens',\n",
       " 'Walter Ross',\n",
       " 'James Bennie',\n",
       " 'Westerner',\n",
       " 'Adam Conyghame',\n",
       " 'Rosing',\n",
       " 'Walt Whitman',\n",
       " 'Geron',\n",
       " 'John A Roan',\n",
       " 'Zavattini',\n",
       " 'Thomas Nichol',\n",
       " 'Cunnamulla',\n",
       " 'David Pitcairn',\n",
       " 'M. Drysdale',\n",
       " 'Young',\n",
       " 'Walter Gray',\n",
       " 'Cocksey',\n",
       " 'K. R. McLaren',\n",
       " 'Barker',\n",
       " 'Thomas Thomson / Minto',\n",
       " 'Magdalene Lodge',\n",
       " 'H. Takahashi',\n",
       " 'Derek Findlay',\n",
       " 'Kundi Buffalo Cow',\n",
       " 'Shapes',\n",
       " 'Warren Poppino',\n",
       " 'Brian Clarke',\n",
       " \"Patrick Macdonald's\",\n",
       " 'Pollock Halls of Residence',\n",
       " 'George Lord Haliburton',\n",
       " 'Gallop',\n",
       " 'John Hammylton',\n",
       " 'Kenneth Alsopp',\n",
       " 'Alexander Melvill',\n",
       " 'James Tylter',\n",
       " 'Wilmut',\n",
       " 'George Fryslyn',\n",
       " 'Craven St. London',\n",
       " 'John Buntyng',\n",
       " 'Allan G Tweedie',\n",
       " 'George C. Harrap',\n",
       " 'de Kelder',\n",
       " 'GS Huntington',\n",
       " 'Bolton',\n",
       " 'Helen Sutherland',\n",
       " 'William Howard Taft',\n",
       " 'Haggs',\n",
       " 'Denney',\n",
       " 'David [Hamilton]',\n",
       " 'Mann',\n",
       " 'J. P. Pemberton',\n",
       " 'John Bartan',\n",
       " 'Alan Hamilton',\n",
       " 'John Byer',\n",
       " 'LondonESP',\n",
       " 'this Gilbert Browne',\n",
       " 'Economic History',\n",
       " 'Mary Wilson',\n",
       " 'G. Grassmann\\n\\n',\n",
       " 'MS_BOX_17',\n",
       " 'George France',\n",
       " 'Beloff',\n",
       " 'Christina Young',\n",
       " 'Noble',\n",
       " 'Agnes Stewart',\n",
       " 'Margaret Campbell Scott',\n",
       " 'John Brounhill',\n",
       " 'Freman',\n",
       " 'David Bait',\n",
       " 'Cambuskennet',\n",
       " 'Donald Dow',\n",
       " 'Robert Fleming',\n",
       " 'Earl John',\n",
       " \"W. Eichrodt's\",\n",
       " 'Edward Story',\n",
       " 'Thomas Coustoun',\n",
       " 'John Belay',\n",
       " 'Daniel Darly] to',\n",
       " 'Myrtoun',\n",
       " 'Haifa',\n",
       " 'Nigel\\n\\nScope and',\n",
       " 'Cunningham Lectures',\n",
       " 'Saint Julien - LL',\n",
       " \"Mary Erskine's\",\n",
       " 'Gorgie',\n",
       " 'Percy William Page Bedford',\n",
       " 'John Glen',\n",
       " \"JMD Mieklejohn's\",\n",
       " 'Charles Augustus Hartley',\n",
       " 'Jonet Durham',\n",
       " 'Hypomania',\n",
       " 'Bego',\n",
       " 'William Strothere',\n",
       " 'Alexander Johnston\\n\\n',\n",
       " 'Geographical',\n",
       " 'George Cuyk',\n",
       " 'Othmar',\n",
       " 'Dunkeld',\n",
       " 'Margaret Cooper',\n",
       " 'Alison M.\\n\\n',\n",
       " 'Granville Ransome Bantock',\n",
       " 'Augustus Square',\n",
       " 'John Mortymare',\n",
       " 'Christina McLaren',\n",
       " 'James Augustus Henry Murray',\n",
       " 'William Turner',\n",
       " 'Harry Luke',\n",
       " 'Memoirs Grzybowski',\n",
       " 'Kirk',\n",
       " 'Bálint Vajta',\n",
       " 'Winifred Donaldson',\n",
       " 'Civil Engineer',\n",
       " 'John Howie',\n",
       " 'R. Siuuarde',\n",
       " 'W K',\n",
       " 'Carrick Hill',\n",
       " 'R Robison',\n",
       " 'John Maknill',\n",
       " 'Bertha Oser',\n",
       " 'Walter Lany',\n",
       " 'John Hay',\n",
       " 'Eton Orchestra',\n",
       " 'Sofia Murller',\n",
       " 'Janet Munro',\n",
       " 'John Dowgall',\n",
       " 'Henry John Elwes',\n",
       " 'Pangloss',\n",
       " 'Andrew Simsoun',\n",
       " 'Jessie King',\n",
       " 'Eric Eaglesham',\n",
       " 'Alice Shaw',\n",
       " 'William Ferquhar',\n",
       " 'John Phillop',\n",
       " 'Kirkhope',\n",
       " 'William Duncan',\n",
       " 'H. Ogden',\n",
       " 'Willielma Campbell',\n",
       " 'Francis Dashwood',\n",
       " 'Malcolm Drummond',\n",
       " 'Thomas Cranstoune',\n",
       " 'Bruce Henderson Kenrick',\n",
       " 'Schumacher',\n",
       " 'George Haitlie',\n",
       " 'Andrew Lumsden',\n",
       " 'Cheryl Mendelson',\n",
       " 'David Foret',\n",
       " 'Small',\n",
       " 'Lady Halle',\n",
       " 'Grace Robb',\n",
       " 'James Aikman',\n",
       " 'C77 Cameron',\n",
       " 'John of Strivelyn',\n",
       " 'Worthington',\n",
       " 'Thomas Wemys',\n",
       " 'George Lundy',\n",
       " 'David Dugeoun',\n",
       " 'Edward H.:',\n",
       " 'Chelsea',\n",
       " 'Sibelius',\n",
       " 'Jeannie Robertson\\n\\nScope',\n",
       " 'George W. Corner',\n",
       " 'Percy Johnson-Marshall Collection',\n",
       " 'D. Laughton',\n",
       " 'EP Boas',\n",
       " 'Robert Ra',\n",
       " 'Adam Lumsden',\n",
       " 'Putnam',\n",
       " 'George Boswell',\n",
       " 'Donald Moodie',\n",
       " 'Benjamin Neeve Peach to Lanarkshire',\n",
       " 'Alexander Broyse [Bruce]',\n",
       " 'Hepatitis B. Biogen',\n",
       " 'Cuthbert Greirson',\n",
       " 'Worthing',\n",
       " 'Henry Walles',\n",
       " 'Diploma',\n",
       " 'David Dempster',\n",
       " 'Gioacchini Rossini\\n\\n',\n",
       " 'William WhitakerSir Charles William WilsonEdward Adrian WilsonJohn WilsonHorace',\n",
       " 'William Laidlaw\\n\\n',\n",
       " 'BUSH',\n",
       " 'Adam Dikesoun',\n",
       " 'David Ross Stadler\\n\\n',\n",
       " 'Alex Greig',\n",
       " 'James Higgs]\\n\\n',\n",
       " 'Abbotshall',\n",
       " 'Charles Stewart\\n\\n',\n",
       " 'A. Fuller - Maitland',\n",
       " 'Fynlastoun Cuningham',\n",
       " 'William Gladstone',\n",
       " 'David Haswell',\n",
       " 'Gu',\n",
       " 'John Binnie',\n",
       " 'Rubinstein',\n",
       " 'Eleonora Russell',\n",
       " 'Iytell Kenne',\n",
       " 'Johansson',\n",
       " 'Gabor',\n",
       " 'C22 Chisholm',\n",
       " 'J. Mandelstam',\n",
       " 'Mathew Stewart',\n",
       " 'James [Crichton',\n",
       " \"Mairì Laghach' '\",\n",
       " 'Dippenneis Easter',\n",
       " 'Moonstone',\n",
       " 'Soccer',\n",
       " 'A. Morrison',\n",
       " 'S66 Scott',\n",
       " 'Hilda C. McCrae',\n",
       " 'Vernon Watkins',\n",
       " 'Robert Bailze',\n",
       " 'John Aldinstoun',\n",
       " 'Khandmal',\n",
       " 'Athabasha Landing',\n",
       " 'Edward Annan',\n",
       " 'James Carncors',\n",
       " 'Thomas Craig',\n",
       " 'Robert Rait',\n",
       " 'Gerard David',\n",
       " 'Ernest Archibald Davidson',\n",
       " 'William Brownehill',\n",
       " 'J. Sigurjonson',\n",
       " 'Jonet Mailvyn',\n",
       " 'MSc Criminology',\n",
       " 'Escog',\n",
       " 'SL Parker',\n",
       " 'Samuel Mitchelson',\n",
       " 'Boznecho De Ostrow',\n",
       " 'Hugh Watt',\n",
       " 'Robert Choate Tyron',\n",
       " 'Christine Haldane',\n",
       " 'Anne Borthwick',\n",
       " 'Charles Thelander M.B.',\n",
       " 'Dietrich Schulz-Köhn',\n",
       " 'Mariota Dicsone',\n",
       " 'Alexander Bailzie',\n",
       " 'John Milne',\n",
       " 'Agnes Barr Auchencloss',\n",
       " 'Glendining',\n",
       " 'Friedrich Kroning',\n",
       " 'Tides',\n",
       " 'Laurence MacKenzie - with',\n",
       " 'Alexander De',\n",
       " 'McEwan Hall - Alan Owen MA - Degree',\n",
       " 'Bjoern L. Basberg',\n",
       " 'Gurnsey',\n",
       " 'J. McWhir',\n",
       " 'N. A.',\n",
       " 'H45 Hayes',\n",
       " 'Alexander Halkerstoun',\n",
       " 'Kathleen Bliss',\n",
       " 'Garnham',\n",
       " 'Lucy Waterhouse',\n",
       " 'David Drummond',\n",
       " 'Alexander Masoune',\n",
       " 'William Bridok',\n",
       " 'Schultz',\n",
       " 'Cadzow',\n",
       " 'Alexander Suvorov',\n",
       " 'Jan Hambourg',\n",
       " 'Carnforth J.',\n",
       " 'John Dog',\n",
       " 'John M. Mitchell',\n",
       " 'Christian E. G.',\n",
       " 'Raster Torbryn',\n",
       " 'Cora',\n",
       " 'Proficiency Students',\n",
       " 'Pontificorum',\n",
       " 'Matth',\n",
       " 'Henry Cary',\n",
       " 'Jersey Cow\\n\\n',\n",
       " 'FR Lillie',\n",
       " 'Mitchison',\n",
       " 'David Smith Cairns',\n",
       " 'Adam Weddale',\n",
       " 'Miss Dodds',\n",
       " 'Thomas S. Brown',\n",
       " 'John McIntyre',\n",
       " 'Scotticised English',\n",
       " 'Alexander Bauereage',\n",
       " 'John Graham Kerr\\n\\n',\n",
       " 'William Schorthous',\n",
       " 'Thomas Martyne of Lathoms',\n",
       " 'Koteliansky',\n",
       " 'Archibald Fergusson',\n",
       " 'Boyer',\n",
       " 'John Strathauchin',\n",
       " 'David Semple',\n",
       " 'Thomas Mason',\n",
       " 'Otto Peininger',\n",
       " 'Mrs A. Kestler',\n",
       " 'George Parker\\n\\n',\n",
       " 'George Crawfurd',\n",
       " 'Joseph Bell',\n",
       " 'B. J. Gradwell\\n\\n',\n",
       " 'Alister\\n\\nScope',\n",
       " 'Centaur',\n",
       " 'PGD',\n",
       " 'Haveley',\n",
       " 'Brevia Antiqua',\n",
       " 'James Blayr',\n",
       " 'Kynquhereis',\n",
       " 'Bonnie Prince',\n",
       " 'William Carubris',\n",
       " 'Benson',\n",
       " 'Gustav Behrens',\n",
       " 'Growth Characteristics',\n",
       " 'Henry B. Marshall-of-Buchan-Broughton',\n",
       " 'John Westualter',\n",
       " 'Charlotte Joyce',\n",
       " 'Wright',\n",
       " 'Harry Kernoff',\n",
       " 'Emma Anthony',\n",
       " 'Mrs Scott',\n",
       " 'Ernest\\n\\nScope and',\n",
       " 'Jean Margaret Bowe',\n",
       " 'Victor Gollancz Ltd.',\n",
       " 'EY',\n",
       " 'George Buchanan',\n",
       " 'R. W. Hobbs',\n",
       " 'Son Duncan',\n",
       " 'Frank Merrick',\n",
       " 'Irene Cairns',\n",
       " 'James Mudy',\n",
       " 'Preston',\n",
       " 'Johnston',\n",
       " 'L. E. Falk',\n",
       " 'Alexander Betoun',\n",
       " 'Jonsson Thorkelin',\n",
       " 'Mary D Geikie',\n",
       " 'James T Hobbs of Maisey',\n",
       " 'Zita',\n",
       " 'Robert Mure',\n",
       " 'Gosewyke',\n",
       " 'Andy Tait',\n",
       " 'Horses',\n",
       " 'Virginia Mandeczewski',\n",
       " 'William D Kennedy',\n",
       " 'Dean Faculty',\n",
       " 'Rosalind Mary Mitchison',\n",
       " 'John Greiff',\n",
       " 'Eileen Hutton',\n",
       " 'James Dalrymple c.1774',\n",
       " \"Robert Knox's\",\n",
       " 'William F\\n\\nScope',\n",
       " 'Mrs Siddous',\n",
       " 'Michael Gilbert',\n",
       " 'al-Fallahin',\n",
       " 'James Wycht',\n",
       " 'John Culpar',\n",
       " 'Academie Bulgarie des Sciences',\n",
       " 'Henry Dundas',\n",
       " 'Alex Murchison',\n",
       " 'William Laing / Georg',\n",
       " 'J. McFadyean',\n",
       " 'Achaltetie',\n",
       " 'Walter Scottengravings',\n",
       " 'Stafford Cripps',\n",
       " 'Godfrey Thomson\\n\\n',\n",
       " 'Andrew M. White',\n",
       " 'R. I. Murchison',\n",
       " 'Sydney M. Glen\\n\\n',\n",
       " 'Moorit',\n",
       " \"W. H. Auden'\",\n",
       " \"Nigel M'Nele\",\n",
       " 'F. J. Bolton',\n",
       " 'Sheila Maureen',\n",
       " 'Alphonse Favre',\n",
       " 'Tennant',\n",
       " 'Johann Wolfgang von Goethe',\n",
       " 'Slonimiski',\n",
       " 'Francesco Gasperini',\n",
       " 'Walter Goodal',\n",
       " 'Davina Ferguson',\n",
       " 'P.M. Bhargava',\n",
       " 'Debussy',\n",
       " \"Donald Macpherson Baillie's\",\n",
       " 'Robyn Williams',\n",
       " 'Margaret Storm',\n",
       " 'Akman',\n",
       " 'John Halliday',\n",
       " 'Firbush',\n",
       " 'Christabel Harriette',\n",
       " 'Arithmetic',\n",
       " 'Lambart',\n",
       " 'Valda Grieve',\n",
       " 'Percy Johnson Marshall',\n",
       " 'Phillpotts-Williams',\n",
       " 'Attlee',\n",
       " 'Elizabeth Looker',\n",
       " 'Evesques Heretiques',\n",
       " 'Gamgee',\n",
       " 'Thomas Suderland',\n",
       " 'Donald Patriksoun',\n",
       " 'William [Lauder',\n",
       " 'William Arnott Fleming',\n",
       " 'Gordon D. Rapp',\n",
       " 'Peter Henry Buck\\n\\n',\n",
       " 'Edward D Crabb',\n",
       " 'M Standfuss',\n",
       " 'Lewis Wolpert;Figs',\n",
       " 'GermanMarjory Kennedy',\n",
       " 'Wee Macgregor - Bengal State Elephant',\n",
       " 'Mary\\n\\nScope and',\n",
       " 'John Fourrous',\n",
       " \"A. Koestler's\",\n",
       " 'Crawfurd',\n",
       " 'David Rizzio',\n",
       " 'Nether Clermiston',\n",
       " \"St Patrick's\",\n",
       " 'William T. Baxter\\n\\nScope',\n",
       " 'Herbert Maxwell',\n",
       " 'Hugh McNiele',\n",
       " 'William Schewan',\n",
       " 'John Greg',\n",
       " 'Padagogisches Zentralblatt',\n",
       " 'Brian Black',\n",
       " 'Patrick Cockburii',\n",
       " 'George Orwell',\n",
       " 'Greta Busch',\n",
       " 'Cotton',\n",
       " 'J. Mitford',\n",
       " 'Timothy Cagneoli',\n",
       " 'Patrick Flokr',\n",
       " 'J. Zhu',\n",
       " 'H.C.',\n",
       " 'James Malloch',\n",
       " 'William Mungall',\n",
       " 'County Down',\n",
       " 'Edinmarnachtie',\n",
       " 'Thomas Moulton',\n",
       " 'William Hole',\n",
       " 'Arthur Dean\\n\\n',\n",
       " 'Arthur KoestlerLusty',\n",
       " 'James Matthew BarrieLord',\n",
       " \"Peter McIntyre's\",\n",
       " 'Michel Angelo Gasparini\\n\\n',\n",
       " 'Dorothy',\n",
       " 'Deborah Charlesworth;Tape III',\n",
       " 'W. B. Wordsworth\\n\\n',\n",
       " 'Irvine',\n",
       " 'Percival W. Hulbert',\n",
       " 'Lectures V-VII',\n",
       " 'C. F.\\n\\nScope and',\n",
       " 'Margaret Emily',\n",
       " 'W. S. Library',\n",
       " 'Winifred A. Donaldson',\n",
       " 'Robert Ellem',\n",
       " 'Loch Earn',\n",
       " 'St Louis',\n",
       " 'Thomas Home',\n",
       " 'Walter Bannerman',\n",
       " 'William Allan',\n",
       " 'Robert Calverley Trevelyan',\n",
       " 'Avril Isobel',\n",
       " 'Vitaliano Brancati',\n",
       " 'Canberra',\n",
       " 'Gordon Landon',\n",
       " 'W. Howard',\n",
       " 'A. Leutzberg',\n",
       " 'Burt',\n",
       " 'L53 La Croix',\n",
       " 'Christina Porter',\n",
       " 'Jules\\n\\nScope and',\n",
       " 'Elizabeth Lightbody',\n",
       " 'Maize Gluten Feed',\n",
       " 'Kherigarh Bull',\n",
       " \"St Oran's\",\n",
       " 'John I. Chalmers',\n",
       " 'John Barrington Wain',\n",
       " 'John G. Campbell',\n",
       " 'James Douglas',\n",
       " 'Archibald Hepburn',\n",
       " 'Harry',\n",
       " 'John Halden',\n",
       " 'Elena Young',\n",
       " 'Julian Huxley',\n",
       " 'Thomas Bernys',\n",
       " 'Lwoff',\n",
       " 'Rudi Seikin',\n",
       " 'H. M. Francis',\n",
       " 'Tannahill',\n",
       " 'William Hardyng',\n",
       " 'Gurjat',\n",
       " 'Dorothy Emmet',\n",
       " 'Boughton',\n",
       " 'J H Hopwood',\n",
       " 'George Durie',\n",
       " 'C. E.\\n\\n',\n",
       " 'Kohn-Speyer',\n",
       " 'James Lyford-Pike',\n",
       " \"Tam Dalyell's\",\n",
       " 'D Collmann',\n",
       " 'James Wright',\n",
       " 'Patrick Downald',\n",
       " 'Winifred Logan Gordon',\n",
       " 'C. K. G. Billings',\n",
       " 'Arizona Copper',\n",
       " 'William John Dundas',\n",
       " 'Frederick Thomas Morrison\\n\\nScope and',\n",
       " 'Border',\n",
       " 'Thomas Hartley Cromek',\n",
       " 'Sophie Weisse',\n",
       " 'D. Lloyd George',\n",
       " 'Balrowny',\n",
       " 'Tswana',\n",
       " 'J. Slee',\n",
       " 'V. L.\\n\\n',\n",
       " 'Kathleen R. McLaren',\n",
       " 'John Grey',\n",
       " 'Welsh Boar',\n",
       " 'Lady Torfrida Ransford',\n",
       " 'Monica Jean\\n\\n',\n",
       " 'Sec',\n",
       " \"M. Brown's\",\n",
       " 'Wurttemberg',\n",
       " 'J. O. Halliwell',\n",
       " 'Maurice Drummond',\n",
       " 'Jake Kugel',\n",
       " 'Khilia F. Kushner',\n",
       " 'Jollos',\n",
       " 'Stockbridge',\n",
       " 'Mr Ryrie',\n",
       " 'Irene',\n",
       " 'David Talbot',\n",
       " 'Bodl',\n",
       " 'Ron\\n\\n',\n",
       " 'Alexander Ezat',\n",
       " 'Malcolm Higges',\n",
       " 'Wilkinson',\n",
       " 'y-Pant',\n",
       " 'Mrs Gilliat',\n",
       " 'N. Fiddes Wilson\\n\\n',\n",
       " 'Foullars',\n",
       " \"John M'William Dwe\",\n",
       " 'Robert McDonald',\n",
       " 'N2 Neilson',\n",
       " 'Gamoose Bull',\n",
       " 'Roger of Grendoun',\n",
       " 'J.S Martyn\\n\\n',\n",
       " 'Stephen Toulmin',\n",
       " 'Jonet Balfour',\n",
       " 'Frederick Thomas',\n",
       " 'C Jucci',\n",
       " 'Walter] Runciman',\n",
       " 'John Orr\\n\\n',\n",
       " 'David Low',\n",
       " 'Drill Hall',\n",
       " 'Kirkman',\n",
       " 'John [Colville',\n",
       " 'A. M. Mackinnon',\n",
       " 'Lily Jackson',\n",
       " 'Perry',\n",
       " 'Albert Venn',\n",
       " 'Custom Revenue',\n",
       " 'Harold Salvesen',\n",
       " \"Robert Henderson's\",\n",
       " 'Covenant',\n",
       " 'George Edward',\n",
       " 'Violet Kennedy-Erskine',\n",
       " 'St Andrew St.',\n",
       " 'Worm',\n",
       " 'Janine',\n",
       " 'Ebenezer Henderson',\n",
       " 'Yarmouth',\n",
       " 'Reedbuck Ram\\n\\n',\n",
       " 'Boman',\n",
       " 'Mathew Hamylton',\n",
       " 'Modelshi',\n",
       " 'Harold\\n\\n',\n",
       " 'Prof Jurasz',\n",
       " 'Ramon Blanco',\n",
       " 'William Chalmer',\n",
       " 'William Logie',\n",
       " 'Elizabeth Hunter',\n",
       " 'Evelyn Chetwode',\n",
       " \"Bulletin de l'Academie Polonaise des Sciences\",\n",
       " 'John Porteous\\n\\nScope and',\n",
       " 'John Kenyon Mason',\n",
       " 'Greig',\n",
       " 'James Buchanan',\n",
       " 'Kenneth E. Boulding',\n",
       " 'Alessandro Scarlatti',\n",
       " 'Andrew Mertyne',\n",
       " 'Peter Mclaren',\n",
       " 'Cavaye',\n",
       " 'K6 Keith',\n",
       " 'Adeane',\n",
       " 'David Livingstone',\n",
       " 'Naturforscher Leopoldina',\n",
       " 'Merioneth',\n",
       " 'John Simpline',\n",
       " 'Uri Fr',\n",
       " 'Charles de\\n\\nScope',\n",
       " 'Ryegrass',\n",
       " 'Library College',\n",
       " 'Edward Atkinson Hornel',\n",
       " \"David G Kendall's\",\n",
       " 'James Ogilvy',\n",
       " 'Maisons de Bois',\n",
       " 'Walter Scott / Kinnoull',\n",
       " 'William Malwyn',\n",
       " 'Fossil',\n",
       " 'Richard Chatto',\n",
       " 'Robert Allan',\n",
       " 'Duns',\n",
       " 'Bogan',\n",
       " \"Mrs Cameron'\",\n",
       " 'John Craigie',\n",
       " 'Frederick S. Mackay\\n\\n',\n",
       " 'Shorthorn Dairy Cow',\n",
       " \"Charlotte Ledermann's\",\n",
       " \"Michael M'Crekane\",\n",
       " 'Maurice Collis',\n",
       " 'a Vera Henderson',\n",
       " 'Kathleen A. Boyd',\n",
       " 'Anthony\\n\\nScope and',\n",
       " 'Fyodor Dostoevsky',\n",
       " 'Saxmundham',\n",
       " 'Duncan B. Forrester',\n",
       " 'CH Hurst',\n",
       " \"Thomas Cumpston - Anglus - '\",\n",
       " 'Joy Elizabeth\\n\\n',\n",
       " 'Threshold',\n",
       " 'Charles Esplin / J.',\n",
       " 'De Hasteville',\n",
       " 'Malcolm McKinnon',\n",
       " 'W20 Wilmot',\n",
       " 'Harold Nature',\n",
       " \"Travels Thro'\",\n",
       " 'Henry Balnavis',\n",
       " 'Nick Barton;Tape III',\n",
       " \"Louis Roy'\",\n",
       " 'Mordington',\n",
       " 'Thesaurus Devonico-Carboniferous',\n",
       " 'Baldowy',\n",
       " 'Forbes',\n",
       " 'Clifton',\n",
       " 'Monterduffy',\n",
       " 'Barbara Jane',\n",
       " 'Skeleton Rear View',\n",
       " 'Aufbau Reconstructionpp.1-2Sent',\n",
       " 'Robert Clark',\n",
       " 'Alexander Napier',\n",
       " 'Ayrshire Cow',\n",
       " 'H.M. Tomlinson',\n",
       " 'Hugh Sutherland',\n",
       " 'Evelyn Valerie\\n\\n',\n",
       " 'C. P. E. Bach',\n",
       " 'Kerry Ram\\n\\n',\n",
       " 'E.F. Gale',\n",
       " 'Peter Strang',\n",
       " 'Henrico Spelmanno',\n",
       " 'Alice Blanche Balfour',\n",
       " 'Sr John Scott',\n",
       " 'Walton Hall',\n",
       " 'Sandy',\n",
       " \"Edinburgh Mela '\",\n",
       " 'Katharine M. Martin',\n",
       " 'C16 Caverhill',\n",
       " 'Mitcheson',\n",
       " 'J.G Gill\\n\\nTitle',\n",
       " 'Jean Fraser',\n",
       " 'Rudolf Presber',\n",
       " 'William Blaikze',\n",
       " 'E. R. Yerbury',\n",
       " 'Salve Christian Fredrik Salvesen',\n",
       " 'Fyndlaw Robertson',\n",
       " 'John Beill',\n",
       " 'David Whitehead Ltd',\n",
       " 'Mr Ross',\n",
       " 'Domini',\n",
       " 'J. G. Pulling',\n",
       " 'Dalkeith Road, Edinburgh',\n",
       " 'Bioprocessing',\n",
       " 'Robert of Hagarston',\n",
       " 'Karl Parsons',\n",
       " 'Henrich Möller',\n",
       " 'Johan de Cliddcrowe',\n",
       " 'Adam Moscrop',\n",
       " 'David Balfour',\n",
       " 'Kikuyu Grammar',\n",
       " 'Walter R. Spalding',\n",
       " 'William Henry Walsh',\n",
       " 'Bocklein',\n",
       " 'Elizabeth Edith',\n",
       " 'David Ogilvy',\n",
       " 'JB Hutchinson',\n",
       " 'Carl Deis',\n",
       " 'James Murcheston',\n",
       " 'John Flarity',\n",
       " 'Aranyi',\n",
       " 'Majesty',\n",
       " 'William Braud - Use',\n",
       " 'Charles D. Jack',\n",
       " 'Kerr',\n",
       " 'F. A. E. Crew\\n\\nScope and Contents',\n",
       " 'Joseph Banks',\n",
       " \"Blind Harry's\",\n",
       " \"John M'Henry\",\n",
       " 'William Hare',\n",
       " 'Fra Angelico da Fiesole',\n",
       " 'Jakob\\n\\nScope and',\n",
       " 'Daniel Stewarts',\n",
       " 'Netherbyer',\n",
       " 'Koleda',\n",
       " 'Co Ironmongers',\n",
       " 'A. Muzzolini',\n",
       " 'Thaithong',\n",
       " 'Sydenham',\n",
       " 'John William Arthur',\n",
       " 'RK Callow',\n",
       " 'LJ Blacher',\n",
       " 'Patrick Hamiltoun',\n",
       " 'Townshead Rd',\n",
       " 'Ceara Tree',\n",
       " 'H. Erskine Hill',\n",
       " 'John Strother',\n",
       " 'Maxtone Graham',\n",
       " 'Berks',\n",
       " 'R. M. Young',\n",
       " 'James Denney',\n",
       " 'Merino',\n",
       " 'Etude Particulière de la Théorie de Fisher',\n",
       " 'George F Manderson',\n",
       " \"John Coleman's\",\n",
       " 'Holger',\n",
       " 'Ceara-Tapping',\n",
       " 'R10 Russell',\n",
       " ...]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_persons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not perfect...some non-person entities labeled such as `Librarian` and `Diploma`.  I'll add labeling person names to the annotation instructions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"check-dups\"></a>\n",
    "## III. Checking for Duplicate Descriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# descs = PlaintextCorpusReader(\"../AnnotationData/descriptions_by_fonds_split_with_ann/descriptions_by_fonds_split_with_ann/\", \".+\\.txt\")\n",
    "# fileids = descs.fileids()\n",
    "# descs.raw(fileids[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Identifier: AA4',\n",
       " '',\n",
       " 'Title:',\n",
       " 'Papers of Rev Prof John McIntyre (1916-2005)',\n",
       " '',\n",
       " 'Scope and Contents:',\n",
       " 'Sermons and addresses, 1940-1997; personal papers, 1932-2006; lectures, 1946-1990; class notes and lecture notes, 1932-1941; correspondence, 1979-1983; papers relating to the Dean of the Thistle, 1974-1980; newspaper cuttings, 1945-1982; publications and articles, 1929-1987; essays, 1935-1979, audio cassette tapes, 1987 and papers relating to television broadcasts, 1960-1967.See also External Documents (below).',\n",
       " '',\n",
       " 'Biographical / Historical:',\n",
       " \"John McIntyre (1916-2005) was born in Glasgow, educated at Bathgate Academy and then Edinburgh University where he graduated in philosophy and then in divinity. In 1943 he became the minister of the Ayrshire village of Fenwick before leaving Scotland in 1945 to teach theology at St Andrew's College in the University of Sydney. Rev McIntyre remained in Australia until 1956 when he returned to Scotland to become professor of divinity at New College.\",\n",
       " 'In 1972, he was asked to be interim Moderator in St Giles Cathedral and in 1982-1983 stood as Moderator of the General Assembly. In 1989 he was awarded the CVO.',\n",
       " '']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = descs.raw(fileids[0])\n",
    "x.split(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Papers of Rev Prof John McIntyre (1916-2005)', 'Sermons and addresses, 1940-1997; personal papers, 1932-2006; lectures, 1946-1990; class notes and lecture notes, 1932-1941; correspondence, 1979-1983; papers relating to the Dean of the Thistle, 1974-1980; newspaper cuttings, 1945-1982; publications and articles, 1929-1987; essays, 1935-1979, audio cassette tapes, 1987 and papers relating to television broadcasts, 1960-1967.See also External Documents (below).', \"John McIntyre (1916-2005) was born in Glasgow, educated at Bathgate Academy and then Edinburgh University where he graduated in philosophy and then in divinity. In 1943 he became the minister of the Ayrshire village of Fenwick before leaving Scotland in 1945 to teach theology at St Andrew's College in the University of Sydney. Rev McIntyre remained in Australia until 1956 when he returned to Scotland to become professor of divinity at New College.\", 'In 1972, he was asked to be interim Moderator in St Giles Cathedral and in 1982-1983 stood as Moderator of the General Assembly. In 1989 he was awarded the CVO.']\n"
     ]
    }
   ],
   "source": [
    "descs_split = []\n",
    "for fileid in fileids:\n",
    "    d = descs.raw(fileid).split(\"\\n\")\n",
    "    for section in d:\n",
    "        if len(section) > 0:\n",
    "            if \"Identifier\" not in section and \"Title\" not in section and \"Scope and Contents\" not in section and \"Biographical / Historical\" not in section and \"Processing Information\" not in section and \"No information provided\" not in section:\n",
    "                descs_split += [section]\n",
    "print(descs_split[0:4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "93636"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(descs_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "82258"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(descs_split))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11378\n"
     ]
    }
   ],
   "source": [
    "print(len(descs_split)-len(set(descs_split)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like there are 11,378 descriptions that are repeated in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}