blob42
/
deep-learning-nano-degree
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
deep-learning-nano-degree/intro-to-rnns/Anna KaRNNa.ipynb

853 lines
64 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Anna KaRNNa\n",
"\n",
"In this notebook, I'll build a character-wise RNN trained on Anna Karenina, one of my all-time favorite books. It'll be able to generate new text based on the text from the book.\n",
"\n",
"This network is based off of Andrej Karpathy's [post on RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) and [implementation in Torch](https://github.com/karpathy/char-rnn). Also, some information [here at r2rt](http://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html) and from [Sherjil Ozair](https://github.com/sherjilozair/char-rnn-tensorflow) on GitHub. Below is the general architecture of the character-wise RNN.\n",
"\n",
"<img src=\"assets/charseq.jpeg\" width=\"500\">"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"import time\n",
"from collections import namedtuple\n",
"\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we'll load the text file and convert it into integers for our network to use. Here I'm creating a couple dictionaries to convert the characters to and from integers. Encoding the characters as integers makes it easier to use as input in the network."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"with open('anna.txt', 'r') as f:\n",
" text=f.read()\n",
"vocab = set(text)\n",
"vocab_to_int = {c: i for i, c in enumerate(vocab)}\n",
"int_to_vocab = dict(enumerate(vocab))\n",
"chars = np.array([vocab_to_int[c] for c in text], dtype=np.int32)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's check out the first 100 characters, make sure everything is peachy. According to the [American Book Review](http://americanbookreview.org/100bestlines.asp), this is the 6th best first line of a book ever."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'Chapter 1\\n\\n\\nHappy families are all alike; every unhappy family is unhappy in its own\\nway.\\n\\nEverythin'"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text[:100]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can see the characters encoded as integers."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([36, 17, 78, 77, 22, 32, 43, 37, 44, 48, 48, 48, 34, 78, 77, 77, 12,\n",
" 37, 47, 78, 63, 18, 27, 18, 32, 23, 37, 78, 43, 32, 37, 78, 27, 27,\n",
" 37, 78, 27, 18, 66, 32, 21, 37, 32, 64, 32, 43, 12, 37, 30, 68, 17,\n",
" 78, 77, 77, 12, 37, 47, 78, 63, 18, 27, 12, 37, 18, 23, 37, 30, 68,\n",
" 17, 78, 77, 77, 12, 37, 18, 68, 37, 18, 22, 23, 37, 4, 35, 68, 48,\n",
" 35, 78, 12, 42, 48, 48, 57, 64, 32, 43, 12, 22, 17, 18, 68], dtype=int32)"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chars[:100]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the network is working with individual characters, it's similar to a classification problem in which we are trying to predict the next character from the previous text. Here's how many 'classes' our network has to pick from."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"83"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.max(chars)+1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Making training and validation batches\n",
"\n",
"Now I need to split up the data into batches, and into training and validation sets. I should be making a test set here, but I'm not going to worry about that. My test will be if the network can generate new text.\n",
"\n",
"Here I'll make both input and target arrays. The targets are the same as the inputs, except shifted one character over. I'll also drop the last bit of data so that I'll only have completely full batches.\n",
"\n",
"The idea here is to make a 2D matrix where the number of rows is equal to the batch size. Each row will be one long concatenated string from the character data. We'll split this data into a training set and validation set using the `split_frac` keyword. This will keep 90% of the batches in the training set, the other 10% in the validation set."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def split_data(chars, batch_size, num_steps, split_frac=0.9):\n",
" \"\"\" \n",
" Split character data into training and validation sets, inputs and targets for each set.\n",
" \n",
" Arguments\n",
" ---------\n",
" chars: character array\n",
" batch_size: Size of examples in each of batch\n",
" num_steps: Number of sequence steps to keep in the input and pass to the network\n",
" split_frac: Fraction of batches to keep in the training set\n",
" \n",
" \n",
" Returns train_x, train_y, val_x, val_y\n",
" \"\"\"\n",
" \n",
" slice_size = batch_size * num_steps\n",
" n_batches = int(len(chars) / slice_size)\n",
" \n",
" # Drop the last few characters to make only full batches\n",
" x = chars[: n_batches*slice_size]\n",
" y = chars[1: n_batches*slice_size + 1]\n",
" \n",
" # Split the data into batch_size slices, then stack them into a 2D matrix \n",
" x = np.stack(np.split(x, batch_size))\n",
" y = np.stack(np.split(y, batch_size))\n",
" \n",
" # Now x and y are arrays with dimensions batch_size x n_batches*num_steps\n",
" \n",
" # Split into training and validation sets, keep the first split_frac batches for training\n",
" split_idx = int(n_batches*split_frac)\n",
" train_x, train_y= x[:, :split_idx*num_steps], y[:, :split_idx*num_steps]\n",
" val_x, val_y = x[:, split_idx*num_steps:], y[:, split_idx*num_steps:]\n",
" \n",
" return train_x, train_y, val_x, val_y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now I'll make my data sets and we can check out what's going on here. Here I'm going to use a batch size of 10 and 50 sequence steps."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"train_x, train_y, val_x, val_y = split_data(chars, 10, 50)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(10, 178650)"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_x.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looking at the size of this array, we see that we have rows equal to the batch size. When we want to get a batch out of here, we can grab a subset of this array that contains all the rows but has a width equal to the number of steps in the sequence. The first batch looks like this:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[36, 17, 78, 77, 22, 32, 43, 37, 44, 48, 48, 48, 34, 78, 77, 77, 12,\n",
" 37, 47, 78, 63, 18, 27, 18, 32, 23, 37, 78, 43, 32, 37, 78, 27, 27,\n",
" 37, 78, 27, 18, 66, 32, 21, 37, 32, 64, 32, 43, 12, 37, 30, 68],\n",
" [37, 78, 63, 37, 68, 4, 22, 37, 24, 4, 18, 68, 24, 37, 22, 4, 37,\n",
" 23, 22, 78, 12, 75, 49, 37, 78, 68, 23, 35, 32, 43, 32, 69, 37, 0,\n",
" 68, 68, 78, 75, 37, 23, 63, 18, 27, 18, 68, 24, 75, 37, 33, 30],\n",
" [64, 18, 68, 42, 48, 48, 49, 70, 32, 23, 75, 37, 18, 22, 9, 23, 37,\n",
" 23, 32, 22, 22, 27, 32, 69, 42, 37, 10, 17, 32, 37, 77, 43, 18, 73,\n",
" 32, 37, 18, 23, 37, 63, 78, 24, 68, 18, 47, 18, 73, 32, 68, 22],\n",
" [68, 37, 69, 30, 43, 18, 68, 24, 37, 17, 18, 23, 37, 73, 4, 68, 64,\n",
" 32, 43, 23, 78, 22, 18, 4, 68, 37, 35, 18, 22, 17, 37, 17, 18, 23,\n",
" 48, 33, 43, 4, 22, 17, 32, 43, 37, 35, 78, 23, 37, 22, 17, 18],\n",
" [37, 18, 22, 37, 18, 23, 75, 37, 23, 18, 43, 58, 49, 37, 23, 78, 18,\n",
" 69, 37, 22, 17, 32, 37, 4, 27, 69, 37, 63, 78, 68, 75, 37, 24, 32,\n",
" 22, 22, 18, 68, 24, 37, 30, 77, 75, 37, 78, 68, 69, 48, 73, 43],\n",
" [37, 15, 22, 37, 35, 78, 23, 48, 4, 68, 27, 12, 37, 35, 17, 32, 68,\n",
" 37, 22, 17, 32, 37, 23, 78, 63, 32, 37, 32, 64, 32, 68, 18, 68, 24,\n",
" 37, 17, 32, 37, 73, 78, 63, 32, 37, 22, 4, 37, 22, 17, 32, 18],\n",
" [17, 32, 68, 37, 73, 4, 63, 32, 37, 47, 4, 43, 37, 63, 32, 75, 49,\n",
" 37, 23, 17, 32, 37, 23, 78, 18, 69, 75, 37, 78, 68, 69, 37, 35, 32,\n",
" 68, 22, 37, 33, 78, 73, 66, 37, 18, 68, 22, 4, 37, 22, 17, 32],\n",
" [21, 37, 33, 30, 22, 37, 68, 4, 35, 37, 23, 17, 32, 37, 35, 4, 30,\n",
" 27, 69, 37, 43, 32, 78, 69, 18, 27, 12, 37, 17, 78, 64, 32, 37, 23,\n",
" 78, 73, 43, 18, 47, 18, 73, 32, 69, 75, 37, 68, 4, 22, 37, 63],\n",
" [22, 37, 18, 23, 68, 9, 22, 42, 37, 10, 17, 32, 12, 9, 43, 32, 37,\n",
" 77, 43, 4, 77, 43, 18, 32, 22, 4, 43, 23, 37, 4, 47, 37, 78, 37,\n",
" 23, 4, 43, 22, 75, 48, 33, 30, 22, 37, 35, 32, 9, 43, 32, 37],\n",
" [37, 23, 78, 18, 69, 37, 22, 4, 37, 17, 32, 43, 23, 32, 27, 47, 75,\n",
" 37, 78, 68, 69, 37, 33, 32, 24, 78, 68, 37, 78, 24, 78, 18, 68, 37,\n",
" 47, 43, 4, 63, 37, 22, 17, 32, 37, 33, 32, 24, 18, 68, 68, 18]], dtype=int32)"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_x[:,:50]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I'll write another function to grab batches out of the arrays made by `split_data`. Here each batch will be a sliding window on these arrays with size `batch_size X num_steps`. For example, if we want our network to train on a sequence of 100 characters, `num_steps = 100`. For the next batch, we'll shift this window the next sequence of `num_steps` characters. In this way we can feed batches to the network and the cell states will continue through on each batch."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def get_batch(arrs, num_steps):\n",
" batch_size, slice_size = arrs[0].shape\n",
" \n",
" n_batches = int(slice_size/num_steps)\n",
" for b in range(n_batches):\n",
" yield [x[:, b*num_steps: (b+1)*num_steps] for x in arrs]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building the model\n",
"\n",
"Below is a function where I build the graph for the network."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"def build_rnn(num_classes, batch_size=50, num_steps=50, lstm_size=128, num_layers=2,\n",
" learning_rate=0.001, grad_clip=5, sampling=False):\n",
" \n",
" # When we're using this network for sampling later, we'll be passing in\n",
" # one character at a time, so providing an option for that\n",
" if sampling == True:\n",
" batch_size, num_steps = 1, 1\n",
"\n",
" tf.reset_default_graph()\n",
" \n",
" # Declare placeholders we'll feed into the graph\n",
" inputs = tf.placeholder(tf.int32, [batch_size, num_steps], name='inputs')\n",
" targets = tf.placeholder(tf.int32, [batch_size, num_steps], name='targets')\n",
" \n",
" # Keep probability placeholder for drop out layers\n",
" keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
" \n",
" # One-hot encoding the input and target characters\n",
" x_one_hot = tf.one_hot(inputs, num_classes)\n",
" y_one_hot = tf.one_hot(targets, num_classes)\n",
"\n",
" ### Build the RNN layers\n",
" # Use a basic LSTM cell\n",
" lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
" \n",
" # Add dropout to the cell\n",
" drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
" \n",
" # Stack up multiple LSTM layers, for deep learning\n",
" cell = tf.contrib.rnn.MultiRNNCell([drop] * num_layers)\n",
" initial_state = cell.zero_state(batch_size, tf.float32)\n",
"\n",
" ### Run the data through the RNN layers\n",
" # This makes a list where each element is one step in the sequence\n",
" rnn_inputs = [tf.squeeze(i, squeeze_dims=[1]) for i in tf.split(x_one_hot, num_steps, 1)]\n",
" \n",
" # Run each sequence step through the RNN and collect the outputs\n",
" outputs, state = tf.contrib.rnn.static_rnn(cell, rnn_inputs, initial_state=initial_state)\n",
" final_state = state\n",
" \n",
" # Reshape output so it's a bunch of rows, one output row for each step for each batch\n",
" seq_output = tf.concat(outputs, axis=1)\n",
" output = tf.reshape(seq_output, [-1, lstm_size])\n",
" \n",
" # Now connect the RNN outputs to a softmax layer\n",
" with tf.variable_scope('softmax'):\n",
" softmax_w = tf.Variable(tf.truncated_normal((lstm_size, num_classes), stddev=0.1))\n",
" softmax_b = tf.Variable(tf.zeros(num_classes))\n",
" \n",
" # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch\n",
" # of rows of logit outputs, one for each step and batch\n",
" logits = tf.matmul(output, softmax_w) + softmax_b\n",
" \n",
" # Use softmax to get the probabilities for predicted characters\n",
" preds = tf.nn.softmax(logits, name='predictions')\n",
" \n",
" # Reshape the targets to match the logits\n",
" y_reshaped = tf.reshape(y_one_hot, [-1, num_classes])\n",
" loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped)\n",
" cost = tf.reduce_mean(loss)\n",
"\n",
" # Optimizer for training, using gradient clipping to control exploding gradients\n",
" tvars = tf.trainable_variables()\n",
" grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), grad_clip)\n",
" train_op = tf.train.AdamOptimizer(learning_rate)\n",
" optimizer = train_op.apply_gradients(zip(grads, tvars))\n",
" \n",
" # Export the nodes\n",
" # NOTE: I'm using a namedtuple here because I think they are cool\n",
" export_nodes = ['inputs', 'targets', 'initial_state', 'final_state',\n",
" 'keep_prob', 'cost', 'preds', 'optimizer']\n",
" Graph = namedtuple('Graph', export_nodes)\n",
" local_dict = locals()\n",
" graph = Graph(*[local_dict[each] for each in export_nodes])\n",
" \n",
" return graph"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Hyperparameters\n",
"\n",
"Here I'm defining the hyperparameters for the network. \n",
"\n",
"* `batch_size` - Number of sequences running through the network in one pass.\n",
"* `num_steps` - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.\n",
"* `lstm_size` - The number of units in the hidden layers.\n",
"* `num_layers` - Number of hidden LSTM layers to use\n",
"* `learning_rate` - Learning rate for training\n",
"* `keep_prob` - The dropout keep probability when training. If you're network is overfitting, try decreasing this.\n",
"\n",
"Here's some good advice from Andrej Karpathy on training the network. I'm going to write it in here for your benefit, but also link to [where it originally came from](https://github.com/karpathy/char-rnn#tips-and-tricks).\n",
"\n",
"> ## Tips and Tricks\n",
"\n",
">### Monitoring Validation Loss vs. Training Loss\n",
">If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:\n",
"\n",
"> - If your training loss is much lower than validation loss then this means the network might be **overfitting**. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.\n",
"> - If your training/validation loss are about equal then your model is **underfitting**. Increase the size of your model (either number of layers or the raw number of neurons per layer)\n",
"\n",
"> ### Approximate number of parameters\n",
"\n",
"> The two most important parameters that control the model are `lstm_size` and `num_layers`. I would advise that you always use `num_layers` of either 2/3. The `lstm_size` can be adjusted based on how much data you have. The two important quantities to keep track of here are:\n",
"\n",
"> - The number of parameters in your model. This is printed when you start training.\n",
"> - The size of your dataset. 1MB file is approximately 1 million characters.\n",
"\n",
">These two should be about the same order of magnitude. It's a little tricky to tell. Here are some examples:\n",
"\n",
"> - I have a 100MB dataset and I'm using the default parameter settings (which currently print 150K parameters). My data size is significantly larger (100 mil >> 0.15 mil), so I expect to heavily underfit. I am thinking I can comfortably afford to make `lstm_size` larger.\n",
"> - I have a 10MB dataset and running a 10 million parameter model. I'm slightly nervous and I'm carefully monitoring my validation loss. If it's larger than my training loss then I may want to try to increase dropout a bit and see if that helps the validation loss.\n",
"\n",
"> ### Best models strategy\n",
"\n",
">The winning strategy to obtaining very good models (if you have the compute time) is to always err on making the network larger (as large as you're willing to wait for it to compute) and then try different dropout values (between 0,1). Whatever model has the best validation performance (the loss, written in the checkpoint filename, low is good) is the one you should use in the end.\n",
"\n",
">It is very common in deep learning to run many different models with many different hyperparameter settings, and in the end take whatever checkpoint gave the best validation performance.\n",
"\n",
">By the way, the size of your training and validation splits are also parameters. Make sure you have a decent amount of data in your validation set or otherwise the validation performance will be noisy and not very informative.\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"batch_size = 100\n",
"num_steps = 100 \n",
"lstm_size = 512\n",
"num_layers = 2\n",
"learning_rate = 0.001\n",
"keep_prob = 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training\n",
"\n",
"Time for training which is pretty straightforward. Here I pass in some data, and get an LSTM state back. Then I pass that state back in to the network so the next batch can continue the state from the previous batch. And every so often (set by `save_every_n`) I calculate the validation loss and save a checkpoint.\n",
"\n",
"Here I'm saving checkpoints with the format\n",
"\n",
"`i{iteration number}_l{# hidden layer units}_v{validation loss}.ckpt`"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true,
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/20 Iteration 1/3560 Training loss: 4.4183 1.6397 sec/batch\n",
"Epoch 1/20 Iteration 2/3560 Training loss: 4.3788 0.7654 sec/batch\n",
"Epoch 1/20 Iteration 3/3560 Training loss: 4.2195 0.7784 sec/batch\n",
"Epoch 1/20 Iteration 4/3560 Training loss: 4.5335 0.7526 sec/batch\n",
"Epoch 1/20 Iteration 5/3560 Training loss: 4.4629 0.7196 sec/batch\n",
"Epoch 1/20 Iteration 6/3560 Training loss: 4.3735 0.7344 sec/batch\n",
"Epoch 1/20 Iteration 7/3560 Training loss: 4.2830 0.6926 sec/batch\n",
"Epoch 1/20 Iteration 8/3560 Training loss: 4.1975 0.6992 sec/batch\n",
"Epoch 1/20 Iteration 9/3560 Training loss: 4.1208 0.7158 sec/batch\n",
"Epoch 1/20 Iteration 10/3560 Training loss: 4.0544 0.7060 sec/batch\n",
"Epoch 1/20 Iteration 11/3560 Training loss: 3.9961 0.7527 sec/batch\n",
"Epoch 1/20 Iteration 12/3560 Training loss: 3.9460 0.7117 sec/batch\n",
"Epoch 1/20 Iteration 13/3560 Training loss: 3.9025 0.7060 sec/batch\n",
"Epoch 1/20 Iteration 14/3560 Training loss: 3.8645 0.6890 sec/batch\n",
"Epoch 1/20 Iteration 15/3560 Training loss: 3.8308 0.6893 sec/batch\n",
"Epoch 1/20 Iteration 16/3560 Training loss: 3.7992 0.6998 sec/batch\n",
"Epoch 1/20 Iteration 17/3560 Training loss: 3.7699 0.7144 sec/batch\n",
"Epoch 1/20 Iteration 18/3560 Training loss: 3.7447 0.7604 sec/batch\n",
"Epoch 1/20 Iteration 19/3560 Training loss: 3.7213 0.7003 sec/batch\n",
"Epoch 1/20 Iteration 20/3560 Training loss: 3.6979 0.6853 sec/batch\n",
"Epoch 1/20 Iteration 21/3560 Training loss: 3.6773 0.6831 sec/batch\n",
"Epoch 1/20 Iteration 22/3560 Training loss: 3.6585 0.7401 sec/batch\n",
"Epoch 1/20 Iteration 23/3560 Training loss: 3.6412 0.7318 sec/batch\n",
"Epoch 1/20 Iteration 24/3560 Training loss: 3.6250 0.7306 sec/batch\n",
"Epoch 1/20 Iteration 25/3560 Training loss: 3.6095 0.7362 sec/batch\n",
"Epoch 1/20 Iteration 26/3560 Training loss: 3.5959 0.7257 sec/batch\n",
"Epoch 1/20 Iteration 27/3560 Training loss: 3.5829 0.7072 sec/batch\n",
"Epoch 1/20 Iteration 28/3560 Training loss: 3.5698 0.7180 sec/batch\n",
"Epoch 1/20 Iteration 29/3560 Training loss: 3.5579 0.7113 sec/batch\n",
"Epoch 1/20 Iteration 30/3560 Training loss: 3.5468 0.7387 sec/batch\n",
"Epoch 1/20 Iteration 31/3560 Training loss: 3.5370 0.7235 sec/batch\n",
"Epoch 1/20 Iteration 32/3560 Training loss: 3.5269 0.7129 sec/batch\n",
"Epoch 1/20 Iteration 33/3560 Training loss: 3.5169 0.6921 sec/batch\n",
"Epoch 1/20 Iteration 34/3560 Training loss: 3.5079 0.7006 sec/batch\n",
"Epoch 1/20 Iteration 35/3560 Training loss: 3.4990 0.6973 sec/batch\n",
"Epoch 1/20 Iteration 36/3560 Training loss: 3.4912 0.7065 sec/batch\n",
"Epoch 1/20 Iteration 37/3560 Training loss: 3.4828 0.7117 sec/batch\n",
"Epoch 1/20 Iteration 38/3560 Training loss: 3.4750 0.7141 sec/batch\n",
"Epoch 1/20 Iteration 39/3560 Training loss: 3.4673 0.7443 sec/batch\n",
"Epoch 1/20 Iteration 40/3560 Training loss: 3.4600 0.7414 sec/batch\n",
"Epoch 1/20 Iteration 41/3560 Training loss: 3.4529 0.7337 sec/batch\n",
"Epoch 1/20 Iteration 42/3560 Training loss: 3.4460 0.7220 sec/batch\n",
"Epoch 1/20 Iteration 43/3560 Training loss: 3.4396 0.7769 sec/batch\n",
"Epoch 1/20 Iteration 44/3560 Training loss: 3.4333 0.7866 sec/batch\n",
"Epoch 1/20 Iteration 45/3560 Training loss: 3.4273 0.6979 sec/batch\n",
"Epoch 1/20 Iteration 46/3560 Training loss: 3.4218 0.6913 sec/batch\n",
"Epoch 1/20 Iteration 47/3560 Training loss: 3.4164 0.7000 sec/batch\n",
"Epoch 1/20 Iteration 48/3560 Training loss: 3.4114 0.6916 sec/batch\n",
"Epoch 1/20 Iteration 49/3560 Training loss: 3.4065 0.6946 sec/batch\n",
"Epoch 1/20 Iteration 50/3560 Training loss: 3.4017 0.7230 sec/batch\n",
"Epoch 1/20 Iteration 51/3560 Training loss: 3.3970 0.7899 sec/batch\n",
"Epoch 1/20 Iteration 52/3560 Training loss: 3.3922 0.7018 sec/batch\n",
"Epoch 1/20 Iteration 53/3560 Training loss: 3.3879 0.6916 sec/batch\n",
"Epoch 1/20 Iteration 54/3560 Training loss: 3.3835 0.6925 sec/batch\n",
"Epoch 1/20 Iteration 55/3560 Training loss: 3.3794 0.7105 sec/batch\n",
"Epoch 1/20 Iteration 56/3560 Training loss: 3.3751 0.7442 sec/batch\n",
"Epoch 1/20 Iteration 57/3560 Training loss: 3.3712 0.7132 sec/batch\n",
"Epoch 1/20 Iteration 58/3560 Training loss: 3.3674 0.6911 sec/batch\n",
"Epoch 1/20 Iteration 59/3560 Training loss: 3.3635 0.6949 sec/batch\n",
"Epoch 1/20 Iteration 60/3560 Training loss: 3.3600 0.6954 sec/batch\n",
"Epoch 1/20 Iteration 61/3560 Training loss: 3.3566 0.6968 sec/batch\n",
"Epoch 1/20 Iteration 62/3560 Training loss: 3.3534 0.6942 sec/batch\n",
"Epoch 1/20 Iteration 63/3560 Training loss: 3.3505 0.6893 sec/batch\n",
"Epoch 1/20 Iteration 64/3560 Training loss: 3.3470 0.7183 sec/batch\n",
"Epoch 1/20 Iteration 65/3560 Training loss: 3.3437 0.7065 sec/batch\n",
"Epoch 1/20 Iteration 66/3560 Training loss: 3.3408 0.6992 sec/batch\n",
"Epoch 1/20 Iteration 67/3560 Training loss: 3.3380 0.7020 sec/batch\n",
"Epoch 1/20 Iteration 68/3560 Training loss: 3.3344 0.7185 sec/batch\n",
"Epoch 1/20 Iteration 69/3560 Training loss: 3.3314 0.7007 sec/batch\n",
"Epoch 1/20 Iteration 70/3560 Training loss: 3.3286 0.6948 sec/batch\n",
"Epoch 1/20 Iteration 71/3560 Training loss: 3.3259 0.7042 sec/batch\n",
"Epoch 1/20 Iteration 72/3560 Training loss: 3.3234 0.7412 sec/batch\n",
"Epoch 1/20 Iteration 73/3560 Training loss: 3.3207 0.7508 sec/batch\n",
"Epoch 1/20 Iteration 74/3560 Training loss: 3.3182 0.7345 sec/batch\n",
"Epoch 1/20 Iteration 75/3560 Training loss: 3.3159 0.6922 sec/batch\n",
"Epoch 1/20 Iteration 76/3560 Training loss: 3.3136 0.7058 sec/batch\n",
"Epoch 1/20 Iteration 77/3560 Training loss: 3.3112 0.7150 sec/batch\n",
"Epoch 1/20 Iteration 78/3560 Training loss: 3.3088 0.7372 sec/batch\n",
"Epoch 1/20 Iteration 79/3560 Training loss: 3.3064 0.6926 sec/batch\n",
"Epoch 1/20 Iteration 80/3560 Training loss: 3.3039 0.6921 sec/batch\n",
"Epoch 1/20 Iteration 81/3560 Training loss: 3.3015 0.7069 sec/batch\n",
"Epoch 1/20 Iteration 82/3560 Training loss: 3.2993 0.6933 sec/batch\n",
"Epoch 1/20 Iteration 83/3560 Training loss: 3.2972 0.7065 sec/batch\n",
"Epoch 1/20 Iteration 84/3560 Training loss: 3.2949 0.6919 sec/batch\n",
"Epoch 1/20 Iteration 85/3560 Training loss: 3.2925 0.6986 sec/batch\n",
"Epoch 1/20 Iteration 86/3560 Training loss: 3.2902 0.6895 sec/batch\n",
"Epoch 1/20 Iteration 87/3560 Training loss: 3.2880 0.6895 sec/batch\n",
"Epoch 1/20 Iteration 88/3560 Training loss: 3.2858 0.6850 sec/batch\n",
"Epoch 1/20 Iteration 89/3560 Training loss: 3.2839 0.7076 sec/batch\n",
"Epoch 1/20 Iteration 90/3560 Training loss: 3.2819 0.6936 sec/batch\n",
"Epoch 1/20 Iteration 91/3560 Training loss: 3.2800 0.7680 sec/batch\n",
"Epoch 1/20 Iteration 92/3560 Training loss: 3.2780 0.7508 sec/batch\n",
"Epoch 1/20 Iteration 93/3560 Training loss: 3.2761 0.7522 sec/batch\n",
"Epoch 1/20 Iteration 94/3560 Training loss: 3.2743 0.7540 sec/batch\n",
"Epoch 1/20 Iteration 95/3560 Training loss: 3.2723 0.7220 sec/batch\n",
"Epoch 1/20 Iteration 96/3560 Training loss: 3.2703 0.7042 sec/batch\n",
"Epoch 1/20 Iteration 97/3560 Training loss: 3.2684 0.7808 sec/batch\n",
"Epoch 1/20 Iteration 98/3560 Training loss: 3.2664 0.7194 sec/batch\n",
"Epoch 1/20 Iteration 99/3560 Training loss: 3.2645 0.6983 sec/batch\n"
]
},
{
"ename": "KeyboardInterrupt",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-38-780c3a10820d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 33\u001b[0m model.initial_state: new_state}\n\u001b[1;32m 34\u001b[0m batch_loss, new_state, _ = sess.run([model.cost, model.final_state, model.optimizer], \n\u001b[0;32m---> 35\u001b[0;31m feed_dict=feed)\n\u001b[0m\u001b[1;32m 36\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mbatch_loss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 37\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 765\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 766\u001b[0m result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 767\u001b[0;31m run_metadata_ptr)\n\u001b[0m\u001b[1;32m 768\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 769\u001b[0m \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 963\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 964\u001b[0m results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 965\u001b[0;31m feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m 966\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 967\u001b[0m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 1013\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1014\u001b[0m return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1015\u001b[0;31m target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m 1016\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1017\u001b[0m return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m 1020\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1021\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1022\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1023\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1024\u001b[0m \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m 1002\u001b[0m return tf_session.TF_Run(session, options,\n\u001b[1;32m 1003\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1004\u001b[0;31m status, run_metadata)\n\u001b[0m\u001b[1;32m 1005\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1006\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
]
}
],
"source": [
"epochs = 20\n",
"# Save every N iterations\n",
"save_every_n = 200\n",
"train_x, train_y, val_x, val_y = split_data(chars, batch_size, num_steps)\n",
"\n",
"model = build_rnn(len(vocab), \n",
" batch_size=batch_size,\n",
" num_steps=num_steps,\n",
" learning_rate=learning_rate,\n",
" lstm_size=lstm_size,\n",
" num_layers=num_layers)\n",
"\n",
"saver = tf.train.Saver(max_to_keep=100)\n",
"with tf.Session() as sess:\n",
" sess.run(tf.global_variables_initializer())\n",
" \n",
" # Use the line below to load a checkpoint and resume training\n",
" #saver.restore(sess, 'checkpoints/______.ckpt')\n",
" \n",
" n_batches = int(train_x.shape[1]/num_steps)\n",
" iterations = n_batches * epochs\n",
" for e in range(epochs):\n",
" \n",
" # Train network\n",
" new_state = sess.run(model.initial_state)\n",
" loss = 0\n",
" for b, (x, y) in enumerate(get_batch([train_x, train_y], num_steps), 1):\n",
" iteration = e*n_batches + b\n",
" start = time.time()\n",
" feed = {model.inputs: x,\n",
" model.targets: y,\n",
" model.keep_prob: keep_prob,\n",
" model.initial_state: new_state}\n",
" batch_loss, new_state, _ = sess.run([model.cost, model.final_state, model.optimizer], \n",
" feed_dict=feed)\n",
" loss += batch_loss\n",
" end = time.time()\n",
" print('Epoch {}/{} '.format(e+1, epochs),\n",
" 'Iteration {}/{}'.format(iteration, iterations),\n",
" 'Training loss: {:.4f}'.format(loss/b),\n",
" '{:.4f} sec/batch'.format((end-start)))\n",
" \n",
" \n",
" if (iteration%save_every_n == 0) or (iteration == iterations):\n",
" # Check performance, notice dropout has been set to 1\n",
" val_loss = []\n",
" new_state = sess.run(model.initial_state)\n",
" for x, y in get_batch([val_x, val_y], num_steps):\n",
" feed = {model.inputs: x,\n",
" model.targets: y,\n",
" model.keep_prob: 1.,\n",
" model.initial_state: new_state}\n",
" batch_loss, new_state = sess.run([model.cost, model.final_state], feed_dict=feed)\n",
" val_loss.append(batch_loss)\n",
"\n",
" print('Validation loss:', np.mean(val_loss),\n",
" 'Saving checkpoint!')\n",
" saver.save(sess, \"checkpoints/i{}_l{}_v{:.3f}.ckpt\".format(iteration, lstm_size, np.mean(val_loss)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Saved checkpoints\n",
"\n",
"Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"tf.train.get_checkpoint_state('checkpoints')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sampling\n",
"\n",
"Now that the network is trained, we'll can use it to generate new text. The idea is that we pass in a character, then the network will predict the next character. We can use the new one, to predict the next one. And we keep doing this to generate all new text. I also included some functionality to prime the network with some text by passing in a string and building up a state from that.\n",
"\n",
"The network gives us predictions for each character. To reduce noise and make things a little less random, I'm going to only choose a new character from the top N most likely characters.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def pick_top_n(preds, vocab_size, top_n=5):\n",
" p = np.squeeze(preds)\n",
" p[np.argsort(p)[:-top_n]] = 0\n",
" p = p / np.sum(p)\n",
" c = np.random.choice(vocab_size, 1, p=p)[0]\n",
" return c"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def sample(checkpoint, n_samples, lstm_size, vocab_size, prime=\"The \"):\n",
" samples = [c for c in prime]\n",
" model = build_rnn(vocab_size, lstm_size=lstm_size, sampling=True)\n",
" saver = tf.train.Saver()\n",
" with tf.Session() as sess:\n",
" saver.restore(sess, checkpoint)\n",
" new_state = sess.run(model.initial_state)\n",
" for c in prime:\n",
" x = np.zeros((1, 1))\n",
" x[0,0] = vocab_to_int[c]\n",
" feed = {model.inputs: x,\n",
" model.keep_prob: 1.,\n",
" model.initial_state: new_state}\n",
" preds, new_state = sess.run([model.preds, model.final_state], \n",
" feed_dict=feed)\n",
"\n",
" c = pick_top_n(preds, len(vocab))\n",
" samples.append(int_to_vocab[c])\n",
"\n",
" for i in range(n_samples):\n",
" x[0,0] = c\n",
" feed = {model.inputs: x,\n",
" model.keep_prob: 1.,\n",
" model.initial_state: new_state}\n",
" preds, new_state = sess.run([model.preds, model.final_state], \n",
" feed_dict=feed)\n",
"\n",
" c = pick_top_n(preds, len(vocab))\n",
" samples.append(int_to_vocab[c])\n",
" \n",
" return ''.join(samples)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, pass in the path to a checkpoint and sample from the network."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "NotFoundError",
"evalue": "Unsuccessful TensorSliceReader constructor: Failed to find any matching files for checkpoints/____.ckpt\n\t [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_recv_save/Const_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]\n\nCaused by op 'save/RestoreV2_6', defined at:\n File \"/home/spike/.pyenv/versions/3.5.1/lib/python3.5/runpy.py\", line 170, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/home/spike/.pyenv/versions/3.5.1/lib/python3.5/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/__main__.py\", line 3, in <module>\n app.launch_new_instance()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n app.start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelapp.py\", line 474, in start\n ioloop.IOLoop.instance().start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n super(ZMQIOLoop, self).start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/ioloop.py\", line 887, in start\n handler_func(fd_obj, events)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n return fn(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n self._handle_recv()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n self._run_callback(callback, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n callback(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n return fn(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 276, in dispatcher\n return self.dispatch_shell(stream, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 228, in dispatch_shell\n handler(stream, idents, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 390, in execute_request\n user_expressions, allow_stdin)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/ipkernel.py\", line 196, in do_execute\n res = shell.run_cell(code, store_history=store_history, silent=silent)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/zmqshell.py\", line 501, in run_cell\n return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2717, in run_cell\n interactivity=interactivity, compiler=compiler, result=result)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2821, in run_ast_nodes\n if self.run_code(code, result):\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2881, in run_code\n exec(code_obj, self.user_global_ns, self.user_ns)\n File \"<ipython-input-40-a0557d3d71b6>\", line 2, in <module>\n samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime=\"Far\")\n File \"<ipython-input-39-5ff7366d054c>\", line 4, in sample\n saver = tf.train.Saver()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 1040, in __init__\n self.build()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 1070, in build\n restore_sequentially=self._restore_sequentially)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 675, in build\n restore_sequentially, reshape)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 402, in _AddRestoreOps\n tensors = self.restore_op(filename_tensor, saveable, preferred_shard)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 242, in restore_op\n [spec.tensor.dtype])[0])\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 668, in restore_v2\n dtypes=dtypes, name=name)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\", line 763, in apply_op\n op_def=op_def)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 2327, in create_op\n original_op=self._default_original_op, op_def=op_def)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 1226, in __init__\n self._traceback = _extract_stack()\n\nNotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for checkpoints/____.ckpt\n\t [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_recv_save/Const_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]\n",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m 1021\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1022\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1023\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m 1003\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1004\u001b[0;31m status, run_metadata)\n\u001b[0m\u001b[1;32m 1005\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/lib/python3.5/contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 67\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py\u001b[0m in \u001b[0;36mraise_exception_on_not_ok_status\u001b[0;34m()\u001b[0m\n\u001b[1;32m 465\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpywrap_tensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 466\u001b[0;31m pywrap_tensorflow.TF_GetCode(status))\n\u001b[0m\u001b[1;32m 467\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mNotFoundError\u001b[0m: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for checkpoints/____.ckpt\n\t [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_recv_save/Const_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;31mNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-40-a0557d3d71b6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mcheckpoint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"checkpoints/____.ckpt\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0msamp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2000\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlstm_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprime\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"Far\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msamp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-39-5ff7366d054c>\u001b[0m in \u001b[0;36msample\u001b[0;34m(checkpoint, n_samples, lstm_size, vocab_size, prime)\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0msaver\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSaver\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0msaver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrestore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcheckpoint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0mnew_state\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minitial_state\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mc\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mprime\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36mrestore\u001b[0;34m(self, sess, save_path)\u001b[0m\n\u001b[1;32m 1426\u001b[0m \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1427\u001b[0m sess.run(self.saver_def.restore_op_name,\n\u001b[0;32m-> 1428\u001b[0;31m {self.saver_def.filename_tensor_name: save_path})\n\u001b[0m\u001b[1;32m 1429\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1430\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 765\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 766\u001b[0m result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 767\u001b[0;31m run_metadata_ptr)\n\u001b[0m\u001b[1;32m 768\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 769\u001b[0m \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 963\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 964\u001b[0m results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 965\u001b[0;31m feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m 966\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 967\u001b[0m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m 1013\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1014\u001b[0m return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1015\u001b[0;31m target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m 1016\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1017\u001b[0m return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
"\u001b[0;32m/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m 1033\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1034\u001b[0m \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1035\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode_def\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1036\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1037\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mNotFoundError\u001b[0m: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for checkpoints/____.ckpt\n\t [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_recv_save/Const_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]\n\nCaused by op 'save/RestoreV2_6', defined at:\n File \"/home/spike/.pyenv/versions/3.5.1/lib/python3.5/runpy.py\", line 170, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/home/spike/.pyenv/versions/3.5.1/lib/python3.5/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/__main__.py\", line 3, in <module>\n app.launch_new_instance()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n app.start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelapp.py\", line 474, in start\n ioloop.IOLoop.instance().start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n super(ZMQIOLoop, self).start()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/ioloop.py\", line 887, in start\n handler_func(fd_obj, events)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n return fn(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n self._handle_recv()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n self._run_callback(callback, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n callback(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n return fn(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 276, in dispatcher\n return self.dispatch_shell(stream, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 228, in dispatch_shell\n handler(stream, idents, msg)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 390, in execute_request\n user_expressions, allow_stdin)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/ipkernel.py\", line 196, in do_execute\n res = shell.run_cell(code, store_history=store_history, silent=silent)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/ipykernel/zmqshell.py\", line 501, in run_cell\n return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2717, in run_cell\n interactivity=interactivity, compiler=compiler, result=result)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2821, in run_ast_nodes\n if self.run_code(code, result):\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2881, in run_code\n exec(code_obj, self.user_global_ns, self.user_ns)\n File \"<ipython-input-40-a0557d3d71b6>\", line 2, in <module>\n samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime=\"Far\")\n File \"<ipython-input-39-5ff7366d054c>\", line 4, in sample\n saver = tf.train.Saver()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 1040, in __init__\n self.build()\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 1070, in build\n restore_sequentially=self._restore_sequentially)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 675, in build\n restore_sequentially, reshape)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 402, in _AddRestoreOps\n tensors = self.restore_op(filename_tensor, saveable, preferred_shard)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/training/saver.py\", line 242, in restore_op\n [spec.tensor.dtype])[0])\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 668, in restore_v2\n dtypes=dtypes, name=name)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\", line 763, in apply_op\n op_def=op_def)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 2327, in create_op\n original_op=self._default_original_op, op_def=op_def)\n File \"/home/spike/.pyenv/versions/3.5.1/envs/ml/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 1226, in __init__\n self._traceback = _extract_stack()\n\nNotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for checkpoints/____.ckpt\n\t [[Node: save/RestoreV2_6 = RestoreV2[dtypes=[DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_recv_save/Const_0, save/RestoreV2_6/tensor_names, save/RestoreV2_6/shape_and_slices)]]\n"
]
}
],
"source": [
"checkpoint = \"checkpoints/____.ckpt\"\n",
"samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime=\"Far\")\n",
"print(samp)"
]
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink