Universal RNN for name generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating names with an RNN\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import LSTM, Dense, Input, concatenate, Reshape, Dropout\n",
    "from tensorflow.keras.models import Model, load_model\n",
    "from tp_datasets import countries_cities as cc\n",
    "from tp_datasets import pokemon\n",
    "from tp_datasets import names as tp_names\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step-by-step example with German city names "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Getting the city names\n",
    "*(Get tp_datasets [here](https://github.com/tadeaspaule/tp-datasets\"))*\n",
    "- First we get a list of names of ~1000 German city names<br>\n",
    "- To remove noise in the model, I turned the names to lowercase and removed ~50ish cities that have odd symbols in their name\n",
    "- We also want to know the length of the longest and shortest name, this will come up later\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total names: 1055\n",
      "Amount of names after removing those with unwanted characters\n",
      ": 998\n",
      "Using the following characters:\n",
      " [' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'ß', 'ä', 'ö', 'ü']\n",
      "Longest name is 30 characters long\n",
      "Shortest name is 3 characters long\n"
     ]
    }
   ],
   "source": [
    "def process_names(names,*,unwanted=['(', ')', '-', '.', '/']):\n",
    "    names = [name.lower() for name in names]\n",
    "    print(\"Total names:\",len(names))\n",
    "    chars = sorted(list(set(''.join(names))))\n",
    "\n",
    "    def has_unwanted(word):\n",
    "        for char in word:\n",
    "            if char in unwanted:\n",
    "                return True\n",
    "        return False\n",
    "    names = [name for name in names if not has_unwanted(name)]\n",
    "    print(\"Amount of names after removing those with unwanted characters\\n:\",len(names))\n",
    "    chars = [char for char in chars if char not in unwanted]\n",
    "    print(\"Using the following characters:\\n\",chars)\n",
    "\n",
    "    maxlen = max([len(name) for name in names])\n",
    "    minlen = min([len(name) for name in names])\n",
    "    print(\"Longest name is\",maxlen,\"characters long\")\n",
    "    print(\"Shortest name is\",minlen,\"characters long\")\n",
    "    \n",
    "    # enchar indicates the end of the word\n",
    "    # here it goes through unlikely-to-be-used characters to find one it can use\n",
    "    endchars = '!£$%^&*()-_=+/?.>,<;:@[{}]#~'\n",
    "    endchar = [ch for ch in endchars if ch not in chars][0]\n",
    "\n",
    "    # ensures the character isn't already used & present in the training data\n",
    "    assert(endchar not in chars)\n",
    "    chars += endchar\n",
    "    \n",
    "    return names,chars\n",
    "\n",
    "names = cc.get_city_list(['Germany'])\n",
    "names,chars = process_names(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Getting the X - long sequences\n",
    "- This model basically works by looking at X characters (in this case 4), and predicting what the next character will be\n",
    "- Changing this X value will affect what patterns the model learns, if we make X too big it can simply memorize names from the dataset, but if we make it too small, it won't be able to accurately predict the next character\n",
    "- I played around with it a bit and settled on 4, but feel free to try out different values (you should only have to change the value below in seqlen = 4 and the rest of the code will adjust itself based on that)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6342 sequences of length 4 made\n"
     ]
    }
   ],
   "source": [
    "def make_sequences(names,seqlen):\n",
    "    sequences, lengths, nextchars = [],[],[] # To have the model learn a more macro understanding, \n",
    "                                             # it also takes the word's length so far as input\n",
    "    for name in names:\n",
    "        if len(name) <= seqlen:\n",
    "            sequences.append(name + chars[-1]*(seqlen - len(name)))\n",
    "            nextchars.append(chars[-1])\n",
    "            lengths.append(len(name))\n",
    "        else:\n",
    "            for i in range(0,len(name)-seqlen+1):\n",
    "                sequences.append(name[i:i+seqlen])\n",
    "                if i+seqlen < len(name):\n",
    "                    nextchars.append(name[i+seqlen])\n",
    "                else:\n",
    "                    nextchars.append(chars[-1])\n",
    "                lengths.append(i+seqlen)\n",
    "\n",
    "    print(len(sequences),\"sequences of length\",seqlen,\"made\")\n",
    "    \n",
    "    return sequences,lengths,nextchars\n",
    "\n",
    "seqlen = 4\n",
    "sequences,lengths,nextchars = make_sequences(names,seqlen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. One hot encoding the sequences, word lengths, and next characters\n",
    "- One hot encoding means that, for example, if you have 5 characters that can appear, you turn the first character into [1 0 0 0 0], the second into [0 1 0 0 0], and so on\n",
    "- We do it because this format is easy for the model to read (and we need to somehow turn the sequence strings into number values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_onehots(*,sequences,lengths,nextchars,chars):\n",
    "    x = np.zeros(shape=(len(sequences),len(sequences[0]),len(chars)), dtype='float32') # sequences\n",
    "    x2 = np.zeros(shape=(len(lengths),max(lengths))) # lengths\n",
    "\n",
    "    for i, seq in enumerate(sequences):\n",
    "        for j, char in enumerate(seq):\n",
    "            x[i,j,chars.index(char)] = 1.\n",
    "\n",
    "    for i, l in enumerate(lengths):\n",
    "        x2[i,l-1] = 1.\n",
    "\n",
    "    y = np.zeros(shape=(len(nextchars),len(chars)))\n",
    "    for i, char in enumerate(nextchars):\n",
    "        y[i,chars.index(char)] = 1.\n",
    "    \n",
    "    return x,x2,y\n",
    "\n",
    "x,x2,y = make_onehots(sequences=sequences,\n",
    "                     lengths=lengths,\n",
    "                     nextchars=nextchars,\n",
    "                     chars=chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Method for generating random starting sequences\n",
    "- Looks at the probabilities letters appear after each other (for example, how often is 'a' third when 'f' is second, compared to other letters that occur after a second 'f')\n",
    "- We will use this later to make brand new names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dictchars(names,seqlen):\n",
    "    dictchars = [{} for _ in range(seqlen)]\n",
    "\n",
    "    for name in names:\n",
    "        if len(name) < seqlen:\n",
    "            continue\n",
    "        dictchars[0][name[0]] = dictchars[0].get(name[0],0) + 1\n",
    "        for i in range(1,seqlen):\n",
    "            if dictchars[i].get(name[i-1],0) == 0:\n",
    "                dictchars[i][name[i-1]] = {name[i]: 1}\n",
    "            elif dictchars[i][name[i-1]].get(name[i],0) == 0:\n",
    "                dictchars[i][name[i-1]][name[i]] = 1\n",
    "            else:\n",
    "                dictchars[i][name[i-1]][name[i]] += 1\n",
    "    return dictchars\n",
    "                \n",
    "dictchars = get_dictchars(names,seqlen)\n",
    "                \n",
    "'''\n",
    "What is dictchars?\n",
    "Basically, stores how often a letter occurs after another letter at a specific spot in a name\n",
    "\n",
    "dictchars[0] just stores how often each letter is first, {a: 3, b:4, etc}\n",
    "\n",
    "dictchars[1+] store which letters (and how often) come after a certain letter.\n",
    "For example, if dictchars[1]['a'] = {b:4,c:1}, that means that if 'a' was first, \n",
    "b followed 4 times, while c followed only once.\n",
    "\n",
    "This is used in the method below to generate plausible-sounding starting sequences.\n",
    "'''\n",
    "    \n",
    "\n",
    "def generate_start_seq(dictchars):\n",
    "    res = \"\" # The starting sequence will be stored here\n",
    "    p = sum([n for n in dictchars[0].values()]) # total amount of letter occurences\n",
    "    r = np.random.randint(0,p) # random number used to pick the next character\n",
    "    tot = 0\n",
    "    for key, item in dictchars[0].items():\n",
    "        if r >= tot and r < tot + item:\n",
    "            res += key\n",
    "            break\n",
    "        else:\n",
    "            tot += item\n",
    "\n",
    "    for i in range(1,len(dictchars)):\n",
    "        ch = res[-1]\n",
    "        if dictchars[i].get(ch,0) == 0:\n",
    "            l = list(dictchars[i].keys())\n",
    "            ch = l[np.random.randint(0,len(l))]\n",
    "        p = sum([n for n in dictchars[i][ch].values()])\n",
    "        r = np.random.randint(0,p)\n",
    "        tot = 0\n",
    "        for key, item in dictchars[i][ch].items():\n",
    "            if r >= tot and r < tot + item:\n",
    "                res += key\n",
    "                break\n",
    "            else:\n",
    "                tot += item\n",
    "    return res\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Methods for generating text\n",
    "- The methods below basically take care of 'I give X letters, I get the full name', so that we can easily monitor the progress of the model (and combined with the above-declared method that makes random starting sequences, we won't even need to provide anything to get brand new names)\n",
    "- There is one concept used below called <i>temperature</i>. Basically it's a measure randomness plays when selecting the next letter, with 0 being no randomness, always picking the most likely letter, and 1 being total randomness, and the letters are chosen based on their probability value\n",
    "- Adjusting this changes how your generated names look, typically the closer to 0 you are the more coherent and closely resembling the training data the output is, and the closer to 1 you are the more novel but sometimes also less coherent the output is. This mainly affects large generated texts though, not so much names. Nevertheless, I tend to go for ~0.4 temperature usually, but feel free to try out different values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample(preds,temperature=0.4):\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    if temperature == 0:\n",
    "        # Avoiding a division by 0 error\n",
    "        return np.argmax(preds)\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1,preds,1)\n",
    "    return np.argmax(probas)\n",
    "\n",
    "def generate_name(model,start,*,chars=chars,temperature=0.4):\n",
    "    maxlength = model.layers[3].input.shape[1]\n",
    "    seqlen = int(model.layers[0].input.shape[1])\n",
    "    result = start\n",
    "    \n",
    "    sequence_input = np.zeros(shape=(1,seqlen,len(chars)))\n",
    "    for i, char in enumerate(start):\n",
    "        sequence_input[0,i,chars.index(char)] = 1.\n",
    "    \n",
    "    length_input = np.zeros(shape=(1,maxlength))\n",
    "    length_input[0,len(result)-1] = 1.\n",
    "    \n",
    "    prediction = model.predict(x=[sequence_input,length_input])[0]\n",
    "    char_index = sample(prediction,temperature)\n",
    "    while char_index < len(chars)-1 and len(result) < maxlength:\n",
    "        result += chars[char_index]\n",
    "        \n",
    "        sequence_input = np.zeros(shape=(1,seqlen,len(chars)))\n",
    "        for i, char in enumerate(result[(-seqlen):]):\n",
    "            sequence_input[0,i,chars.index(char)] = 1.\n",
    "        \n",
    "        length_input[0,len(result)-2] = 0.\n",
    "        length_input[0,len(result)-1] = 1.\n",
    "        \n",
    "        prediction = model.predict(x=[sequence_input,length_input])[0]\n",
    "        char_index = sample(prediction,temperature)\n",
    "    \n",
    "    return result.title()\n",
    "\n",
    "def generate_random_name(model,*,chars=chars,dictchars=dictchars,temperature=0.4):\n",
    "    start = generate_start_seq(dictchars)\n",
    "    return generate_name(model,start,chars=chars,temperature=temperature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Building the model\n",
    "- Here is where you can experiment and try out different approaches\n",
    "- After some testing, I went with the below setup:\n",
    "    - 2 Inputs (the sequence, and the one-hot-encoded length of the name at the end of that sequence)\n",
    "    - 2 parallel LSTM layers, one normal with relu, the other backwards with tanh, both with dropout 0.3\n",
    "    - Concatenate the LSTM outputs with the one-hot-encoded length\n",
    "    - Dense output layer with softmax activation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_model(x,x2,chars):\n",
    "    inp1 = Input(shape=x.shape[1:]) # sequence input\n",
    "    inp2 = Input(shape=x2.shape[1:]) # length input\n",
    "    lstm = LSTM(len(chars),activation='relu',dropout=0.3)(inp1)\n",
    "    lstm2 = LSTM(len(chars),dropout=0.3,go_backwards=True)(inp1)\n",
    "    concat = concatenate([lstm,lstm2,inp2])\n",
    "    dense = Dense(len(chars),activation='softmax')(concat)\n",
    "\n",
    "    model = Model([inp1,inp2],dense)\n",
    "    model.compile(optimizer='adam',loss='binary_crossentropy')\n",
    "    return model\n",
    "\n",
    "model = make_model(x,x2,chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Method for training a model and monitoring its progress\n",
    "- Using this makes it easy to try out different model architectures and see what names they are able to generate\n",
    "- For fast prototyping, just build and compile a model, then simply:\n",
    "```python\n",
    "try_model(model)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def try_model(model,*,x=x,x2=x2,y=y,chars=chars,dictchars=dictchars,total_epochs=180,print_every=60,temperature=0.4,verbose=True):\n",
    "    for i in range(total_epochs//print_every):\n",
    "        history = model.fit([x,x2],y,\n",
    "                            epochs=print_every,\n",
    "                            batch_size=64,\n",
    "                            validation_split=0.05,\n",
    "                            verbose=0)\n",
    "        if verbose:\n",
    "            print(\"\\nEpoch\",(i+1)*print_every)\n",
    "            print(\"First loss:            %1.4f\" % (history.history['loss'][0]))\n",
    "            print(\"Last loss:             %1.4f\" % (history.history['loss'][-1]))\n",
    "            print(\"First validation loss: %1.4f\" % (history.history['val_loss'][0]))\n",
    "            print(\"Last validation loss:  %1.4f\" % (history.history['val_loss'][-1]))\n",
    "            print(\"\\nGenerating random names:\")\n",
    "            for _ in range(10):\n",
    "                print(generate_random_name(model,chars=chars,dictchars=dictchars,temperature=temperature)) \n",
    "    if not verbose:\n",
    "        print(\"Model training complete, here are some generated names:\")\n",
    "        for _ in range(20):\n",
    "            print(generate_random_name(model,chars=chars,dictchars=dictchars,temperature=0.4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. And finally, training the model and seeing how it does"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 60\n",
      "First loss:            0.1313\n",
      "Last loss:             0.0747\n",
      "First validation loss: 0.1214\n",
      "Last validation loss:  0.0834\n",
      "\n",
      "Generating random names:\n",
      "Lenkirchen\n",
      "Kochen\n",
      "Red Ingen\n",
      "Grerach\n",
      "Rerstadt\n",
      "Wanal\n",
      "Mörten\n",
      "Nilkenberg\n",
      "Rhenein\n",
      "Wilenberg\n",
      "\n",
      "Epoch 120\n",
      "First loss:            0.0742\n",
      "Last loss:             0.0668\n",
      "First validation loss: 0.0833\n",
      "Last validation loss:  0.0845\n",
      "\n",
      "Generating random names:\n",
      "Hoch\n",
      "Geede\n",
      "Herne\n",
      "Walischen\n",
      "Olein\n",
      "Gau Am Rach\n",
      "Eschen\n",
      "Wuxter\n",
      "Zeilerstadt\n",
      "Aimarsted\n",
      "\n",
      "Epoch 180\n",
      "First loss:            0.0670\n",
      "Last loss:             0.0622\n",
      "First validation loss: 0.0850\n",
      "Last validation loss:  0.0853\n",
      "\n",
      "Generating random names:\n",
      "Fistenburg\n",
      "Türburg\n",
      "Herstadt\n",
      "Velpen\n",
      "Wad Salz\n",
      "Laven\n",
      "Britz\n",
      "Wargen\n",
      "Welweiler\n",
      "Huch\n"
     ]
    }
   ],
   "source": [
    "try_model(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Putting it all together to showcase other datasets\n",
    "### This method returns a model and a method for generating more names\n",
    "```python\n",
    "# usage example\n",
    "names = load_name_data() # load your desired name dataset\n",
    "model, generate_name = train_model(names)\n",
    "\n",
    "new_names = []\n",
    "for _ in range(1000):\n",
    "    new_name = generate_name()\n",
    "    new_names.append(new_name)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(names,*, seqlen=4,unwanted=['(', ')', '-', '.', '/'],verbose=True):\n",
    "    names,chars = process_names(names,unwanted=unwanted)\n",
    "    \n",
    "    sequences,lengths,nextchars = make_sequences(names,seqlen)\n",
    "    \n",
    "    x,x2,y = make_onehots(sequences=sequences,\n",
    "                          lengths=lengths,\n",
    "                          nextchars=nextchars,\n",
    "                          chars=chars)\n",
    "        \n",
    "    dictchars = get_dictchars(names,seqlen)\n",
    "    \n",
    "    model = make_model(x,x2,chars)  \n",
    "    \n",
    "    try_model(model,x=x,x2=x2,y=y,chars=chars,dictchars=dictchars,verbose=verbose)\n",
    "    \n",
    "    def generate():\n",
    "        return generate_random_name(model,chars=chars,dictchars=dictchars,temperature=0.4)\n",
    "    \n",
    "    return model, generate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Pokemon name dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total names: 819\n",
      "Amount of names after removing those with unwanted characters\n",
      ": 814\n",
      "Using the following characters:\n",
      " ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "Longest name is 11 characters long\n",
      "Shortest name is 3 characters long\n",
      "3569 sequences of length 4 made\n",
      "Model training complete, here are some generated names:\n",
      "Vumerow\n",
      "Chation\n",
      "Lidicha\n",
      "Biggole\n",
      "Skavitar\n",
      "Gradovant\n",
      "Pirashar\n",
      "Gyomatios\n",
      "Laticout\n",
      "Picarig\n",
      "Joroke\n",
      "Noubra\n",
      "Pottone\n",
      "Hidorita\n",
      "Yvypario\n",
      "Gotwik\n",
      "Ninder\n",
      "Tynum\n",
      "Gelbire\n",
      "Kroperite\n"
     ]
    }
   ],
   "source": [
    "_,pokenames,_ = pokemon.load_data(return_full_names=False)\n",
    "pokemodel,pokegen = train_model(pokenames,unwanted=['.',\"'\",'-','0','(',')','2',' '],verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: US cities dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total names: 2698\n",
      "Amount of names after removing those with unwanted characters\n",
      ": 2682\n",
      "Using the following characters:\n",
      " [' ', '-', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "Longest name is 25 characters long\n",
      "Shortest name is 3 characters long\n",
      "17326 sequences of length 4 made\n",
      "Model training complete, here are some generated names:\n",
      "Valmington\n",
      "Buter\n",
      "Noora City\n",
      "Actland\n",
      "Cosel\n",
      "Fansey\n",
      "Cayner\n",
      "Lachas\n",
      "Ganerton\n",
      "Tubart\n",
      "Masque\n",
      "Menton\n",
      "Schester\n",
      "Peter\n",
      "Horaland\n",
      "Manta Park\n",
      "Nockesville\n",
      "Pucaina\n",
      "Smport\n",
      "Mcorle\n"
     ]
    }
   ],
   "source": [
    "us_cities = cc.get_city_list(['United States'])\n",
    "us_model,us_gen = train_model(us_cities,unwanted=[\"'\",'ñ', 'ā', 'ī', '‘','(',')','.','1'],verbose=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}