{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Explorer Notebook Tutorial\n",
    "\n",
    "The InPhO Topic Explorer features a powerful interactive coding environment that enables direct manipulation of the corpus and models, in contrast to the web visualization.\n",
    "\n",
    "When you run the `vsm notebook` command, several things happen:\n",
    "1.  The Topic Explorer creates a new folder called `notebooks` and places several files inside it:\n",
    "     -  **corpus.py** contains Python code which imports the Corpus object and a function that will load models trained on your corpus. It gathers this information from your `CORPUS.ini` file, where `CORPUS` is the name of the folder you prepared. More information on this special file is below.\n",
    "     -  **Topic Explorer Tutorial.ipynb** is this file, which provides the skeleton documentation for interacting with the models.\n",
    "     -  Other **ipynb** files, containing analyses that can be re-run on your own corpora.\n",
    "2.  The Topic Explorer launches a [Jupyter Notebook](http://jupyter.org/). This allows you to program in Python on your local computer using the browser, rather than a terminal or other program. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducing Jupyter\n",
    "If you look in the address bar of your browser, it should start with something like `localhost:8888/notebooks/Topic%20Explorer%20Tutorial.ipynb`. If you open a new tab or browser window and type in `localhost:8888` you will open the same list of files and be able to edit multiple files at once. Note that editing the same file in multiple tabs may cause you to accidentally overwrite your data. Jupyter does not check if the file is already open.\n",
    "\n",
    "Also, note that if the first portion of the url (`localhost:8888`) is different, you will need to enter what it says on your computer to open a new tab. Most likely the \"port number\" (`8888`) will be different, and may indicate you have a second instance of `vsm notebook` or another Jupyter Notebook server running.\n",
    "\n",
    "\n",
    "### Running Cells\n",
    "The Jupyter notebook operates through individual *cells* that run Python code. To run a cell press the \"run cell\" (play) button or select the cell by clicking on it, then select in the Jupyter menu \"Cell > Run Cells\".\n",
    "\n",
    "Try running the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"Hello world!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Immediately below the cell, the `Hello world!` should be printed, and to the left of the cell it should say `In [1]:`.\n",
    "\n",
    "#### A note on kernels and brackets\n",
    "If the number `[1]` is different, nothing is wrong, so long as there is a number printed.\n",
    "\n",
    "The number in brackets (`[1]`) counts the number of times you have run a cell in this notebook session. A notebook session is tied to a *kernel*. The kernel runs the Python code. If you wish to reset the numbers and run your code step-by-step, starting from `[1]` again, go to the menu and select \"Kernel > Restart & Run All\".\n",
    "\n",
    "If the number appears as `In [*]:` that means that the cell is currently running. When it changes to a number (`[2]`), then the files have completed importing.\n",
    "\n",
    "If you feel this is taking an absurdly long time to load (in excess of a few seconds), please press the stop button and notify the package developers. There might be a bug in the modeling software."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Errors and Debugging\n",
    "Each cell automatically calls `print` on the last line of the cell. Run the cell below to see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use this to print `Hello world!` once again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Hello world!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**D'oh!** This should have raised a `SyntaxError: invalid syntax`.\n",
    "\n",
    "Note the message `File \"<ipython-input-5-59ca0efa9f56>\", line 1` (the portion after `ipython-input` may be different). This tells you which line in the program errored. If you have errors in more advanced code, the line number will be very helpful in diagnosing the problem.\n",
    "\n",
    "For now, change the cell above to `\"Hello world!\"` and run again to get the proper output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing `corpus.py`\n",
    "Now that you know how to run a cell, we can begin interacting with the topic models. First we will import your corpus objects. Select and run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from corpus import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will now have access to several variables, the most important of which are:\n",
    " -  `c` -- The `vsm.Corpus` object\n",
    " -  `lda_v` -- A dictionary containing each of the `vsm.LdaViewer` instances. You can access a particular model with `lda_v[k]`, substituting k for a particular number, like `lda_v[20]` for the 20-topic model. If the model for that number of topics has not been trained, it will error.\n",
    " -  `topic_range` -- A list of the trained models (e.g., `[20, 40, 60, 80]`)\n",
    " -  `context_type` -- A string containing the particular context type modeled (e.g., `\"sentence\"`, `\"document\"`, `\"article\"`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introducing the `vsm` module\n",
    "\n",
    "The InPhO Topic Explorer is comprised of two modules:\n",
    "1. The `topicexplorer` module contains code for the visualization and user interfaces.\n",
    "2. The `vsm` module contains code for modeling differnet corpora. \n",
    "\n",
    "In order to make use of the term frequency (TF), term frequency-inverse document frequency (TfIdf), and latent semantic analysis (LSA) models, we must import the main vsm module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from vsm import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interacting with the Corpus: Term Frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command above has loaded your `Corpus` object into the `c` variable. You can see the list of all words that are in your corpus by typing `c.words` into a code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c.words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that it only shows the first few and last few unique words in the corpus, alphabetically sorted. \n",
    "\n",
    "What if we want to get a list of how often each word occurs? For that, we can use the `vsm.model.TF` to build a frequency distribution over the terms in the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# train the model and create a TfViewer object\n",
    "tf = TF(c, context_type)\n",
    "tf.train()\n",
    "tf_v = TfViewer(c, tf)\n",
    "\n",
    "# print the most frequent terms in the document\n",
    "# remember that IPython automatically prints the last cell of a document\n",
    "tf_v.coll_freqs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the cell above, you should see a table with the 20 most frequently used words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Interacting with Topic Models\n",
    "\n",
    "The InPhO Topic Explorer doesn't just work with term frequencies though - it creates LDA topic models. Through the notebook interface these models can be powerfully manipulated to produce new analyses.\n",
    "\n",
    "First, let's select a primary model to investigate, and load it into the variable `v`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the number of topics in the first model\n",
    "print topic_range[0]\n",
    "# remember that list indexes start with 0 not 1!\n",
    "\n",
    "# replace 'topic_range[0]' with a specific number, if you like\n",
    "k = topic_range[0]\n",
    "\n",
    "# load the topic model\n",
    "v = lda_v[k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code loads the first topic model into a viewer object. We have used the `topic_range[0]` instead of simply stating a number so that this same demo notebook will work with any model settings you've prepared. This portability enables us to write analyses that can be replicated across any corpus, and is one of the real strengths of using the `from corpus import *` model of coding your notebooks. If others are using the Topic Explorer to generate their objects, they can run the exact same analysis on different corpora, so long as the variable names are consistent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `v.topics()`\n",
    "First, lets print a list of topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v.topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To change the number of words printed per topic, use the `print_len` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v.topics(print_len=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Document-topic probabilities\n",
    "The above code shows the topic-word distributions and allows us to estimate the quality of our topics.\n",
    "\n",
    "#### `v.labels`\n",
    "The property `v.labels` (without parentheses) returns a list of all documents in a corpus, and is useful for processing each document generically, without having to look up the identifiers on the file system.\n",
    "\n",
    "Below, we print the first 3 document labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for label in v.labels[:3]:\n",
    "    print label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `v.doc_topics(doc_or_docs)`\n",
    "Each document-topic distribution can be examined with `v.doc_topics()`, which takes as its argument either a single label or a list of labels. Below we view the distribution for the first 3 documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v.doc_topics(v.labels[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `v.aggregate_doc_topics(doc_or_docs, normed_sum=False)`\n",
    "While `v.doc_topics(doc_or_docs)` shows the distribution for each document, `v.aggregate_doc_topics()` shows the average distribution of a collection of documents. The `normed` argument tells the program whether to weight each document by its length (`normed_sum=True`) or to consider them all equally (`normed_sum=False`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v.aggregate_doc_topics(v.labels[:3], normed_sum=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing documents with `v.dist()`\n",
    "\n",
    "Topic models give us a way to compare the siimilarity between two documents. To do this, we use `v.dist()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v.dist(v.labels[0], v.labels[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alternative distance measures\n",
    "By default, the Topic Explorer uses the Jensen-Shannon Distance to calculate the distance between documents. The Jensen-Shannon Distance (JSD) is a symmetric measure based on information theory that characterizes the difference between two probability distributions.\n",
    "\n",
    "However, several alternate methods are built into the `vsm.spatial` module. These include the Kullback-Leibler Divergence, which is an asymmetric component of the JSD and is used in [Murdock et al. (in review)](http://arxiv.org/abs/1509.07175) to characterize the cognitive surprise of a new text, given previous texts.\n",
    "\n",
    "Rather than using the JSD and assuming symmetric divergence between items, we assume that the second document is encountered after the first, effectively measuring text-to-text divergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# first import KL divergence:\n",
    "from vsm.spatial import KL_div\n",
    "\n",
    "# calculate KL divergence from the first document to the second\n",
    "print \"First to second\", v.dist(v.labels[0], v.labels[1], dist_fn=KL_div)\n",
    "\n",
    "# calculate KL divergence from the second document to the first, highlighting asymmetry:\n",
    "print \"Second to first\", v.dist(v.labels[1], v.labels[0], dist_fn=KL_div)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Python's Help System\n",
    "\n",
    "There are many other functions in the InPhO Topic Explorer and the associated `vsm` library. These are extensively documented within the code. \n",
    "\n",
    "One little-known feature about Python is its capacity for introspection: by using the `help()` method, one can find out all methods and properties of an object. For example, if one wanted to know what methods could be called on their corpus object, you could run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get help on particular methods. For example, there are many arguments to `v.topics()` beyond `print_len`. These can be seen by calling `help(v.topics)` without parentheses after `v.topics`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(v.topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `help(v.topics())` *with* parentheses will return help for the object reutrned by `v.topics()`, which is a `DataTable`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(v.topics())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to emphasize that this functionality can be used with any python library, including the standard library. For example, one could look at all the functions included in the `math` library by using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import math\n",
    "help(math.log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For emphasis, calling `help(math.log(3))` will return the documentation for `float`. Why? First `math.log(3)` will be evaluated, then `help()` will be called on that result: `help(1.0986122886681098)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "help(math.log(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `help()` and `?`\n",
    "\n",
    "Alternatively, you can place a `?` before any function to receive help in a separate \"frame\". This allows you to view help while scrolling up and down the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?math.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additional Examples\n",
    "\n",
    "This notebook gives some basic building blocks for using the Topic Explorer. Additional examples can be found on GitHub in the [inpho/vsm-demo-notebooks repository](http://github.com/inpho/vsm-demo-notebooks).\n",
    "\n",
    "# Contact Information\n",
    "If you have additional questions regarding the InPhO Topic Explorer or have comments on this tutorial, please e-mail [tutorial@hypershelf.org](mailto:tutorial@hypershelf.org).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
