Couple of minor fixes to ease transition from v1 to v2. Moved v1 readme

elsewhere and added a totally new v2 one

Changelog.md

@@ -0,0 +1,13 @@
+=======
+Predictor Changelog
+=========
+2.0.0 (2014-03-07)
+---------------------
+**Rewrite of 1.0.0 and contains several breaking changes!**
+
+Version 1.0.0 (which really should have been 0.0.1) contained several issues that made compatability with v2 not worth the trouble. This includes:
+* In v1, similarities were cached per input_matrix, and Predictor::Base utilized those caches when determining similarities and predictions. This quickly ate up Redis memory with even a semi-large dataset, as each input_matrix had a significant memory requirement. v2 caches similarities at the root (Recommender::Base), which means you can add any number of input matrices with little impact on memory usage.
+* Added the ability to limit the number of items stored in the similarity cache (via the 'limit_similarities_to' option). Now that similarities are cached at the root, this is possible and can greatly help memory usage.
+* Removed bang methods from input_matrix (add_set!, and_single!, etc). These called process! for you previously, but since the cache is no longer kept at the input_matrix level, process! has to be called at the root (Recommender::Base)
+* Bug fix: Fixed bug where a call to delete_item! on the input matrix didn't update the similarity cache.
+* Other minor fixes.

README.md

@@ -13,6 +13,10 @@ Originally forked and based on [Recommendify](https://github.com/paulasmuth/reco
 
 At the moment, Predictor uses the [Jaccard index](http://en.wikipedia.org/wiki/Jaccard_index) to determine similarities between items. There are other ways to do this, which we intend to implement eventually, but if you want to beat us to the punch, pull requests are quite welcome :)
 
+Notice
+---------------------
+This is the readme for Predictor 2.0, which contains a few breaking changes from 1.0. The 1.0 readme can be found [here](https://github.com/Pathgather/predictor/blob/master/docs/READMEv1.md). See below on how to upgrade to 2.0
+
 Installation
 ---------------------
 ```ruby
@@ -32,6 +36,7 @@ Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"])
 # Or, to improve performance, add hiredis as your driver (you'll need to install the hiredis gem first)
 Predictor.redis = Redis.new(:url => ENV["PREDICTOR_REDIS"], :driver => :hiredis)
 ```
+
 Inputting Data
 ---------------------
 Create a class and include the Predictor::Base module. Define an input_matrix for each relationship you'd like to keep track of. This can be anything you think is a significant metric for the item: page views, purchases, categories the item belongs to, etc.
@@ -51,6 +56,7 @@ Below, we're building a recommender to recommend courses based off of:
 class CourseRecommender
   include Predictor::Base
 
+  limit_similarities_to 500   # Optional, but if specified, Predictor only caches the top x similarities for an item at any given time. Can greatly help with efficient use of Redis memory
   input_matrix :users, weight: 3.0
   input_matrix :tags, weight: 2.0
   input_matrix :topics, weight: 1.0
@@ -62,37 +68,21 @@ Now, we just need to update our matrices when courses are created, users take a
 recommender = CourseRecommender.new
 
 # Add a single course to topic-1's items. If topic-1 already exists as a set ID, this just adds course-1 to the set
-recommender.topics.add_single!("topic-1", "course-1")
-
-# If your matrix is quite large, add_single! could take some time, as it must calculate the similarity scores
-# for course-1 across all other courses. If this is the case, use add_single and process the item at a more
-# convenient time, perhaps in a background job
-recommender.topics.add_single("topic-1", "course-1")
-recommender.topics.process_item!("course-1")
-
-# Add an array of courses to tag-1. Again, these will simply be added to tag-1's existing set, if it exists.
-# If not, the tag-1 set will be initialized with course-1 and course-2
-recommender.tags.add_set!("tag-1", ["course-1", "course-2"])
+recommender.add_to_matrix!(:topics, "topic-1", "course-1")
 
-# Or, just add the set and process whenever you like
-recommender.tags.add_set("tag-1", ["course-1", "course-2"])
-["course-1", "course-2"].each { |course| recommender.topics.process_item!(course) }
+# If your dataset is even remotely large, add_to_matrix! could take some time, as it must calculate the similarity scores
+# for course-1 and other courses that share a set with course-1. If this is the case, use add_to_matrix and
+# process the items at a more convenient time, perhaps in a background job
+recommender.topics.add_to_set("topic-1", "course-1", "course-2") # Same as recommender.add_to_matrix(:topics, "topic-1", "course-1", "course-2")
+recommender.process_items!("course-1", "course-2")
 ```
 
-As noted above, it's important to remember that if you don't use the bang methods (add_set! and add_single!), you'll need to manually update your similarities (the bang methods will likely suffice for most use cases though). You can do so a variety of ways.
-* If you want to simply update the similarities for a single item in a specific matrix:
-  ````
-  recommender.matrix.process_item!(item)
-  ````
-* If you want to update the similarities for all items in a specific matrix:
-  ````
-  recommender.matrix.process!
-  ````
-* If you want to update the similarities for a single item in all matrices:
+As noted above, it's important to remember that if you don't use the bang method 'add_to_matrix!', you'll need to manually update your similarities. If your dataset is even remotely large, you'll probably want to do this:
+* If you want to update the similarities for certain item(s):
   ````
-  recommender.process_item!(item)
+  recommender.process_items!(item1, item2, etc)
   ````
-* If you want to update all similarities in all matrices:
+* If you want to update all similarities for all items:
   ````
   recommender.process!
   ````
@@ -135,18 +125,18 @@ recommender.predictions_for("user-1", matrix_label: :users)
 # Paginate too!
 recommender.predictions_for("user-1", matrix_label: :users, offset: 10, limit: 10)
 
-# Gimme some scores and ignore user-2....that user-2 is one sketchy fella
-recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["user-2"])
+# Gimme some scores and ignore course-2....that course-2 is one sketchy fella
+recommender.predictions_for("user-1", matrix_label: :users, with_scores: true, exclusion_set: ["course-2"])
 ```
 
 Deleting Items
 ---------------------
-If your data is deleted from your persistent storage, you certainly don't want to recommend that data to a user. To ensure that doesn't happen, simply call delete_item! on the individual matrix or recommender as a whole:
+If your data is deleted from your persistent storage, you certainly don't want to recommend it to a user. To ensure that doesn't happen, simply call delete_from_matrix! with the individual matrix or delete_item! if the item is completely gone:
 ```ruby
 recommender = CourseRecommender.new
 
 # User removed course-1 from topic-1, but course-1 still exists
-recommender.topics.delete_item!("course-1")
+recommender.delete_from_matrix!(:topics, "course-1")
 
 # course-1 was permanently deleted
 recommender.delete_item!("course-1")
@@ -155,28 +145,51 @@ recommender.delete_item!("course-1")
 recommender.clean!
 ```
 
-Memory Management
+Limiting Similarities
 ---------------------
-Predictor works by caching the similarities for each item in each matrix, then computing overall similarities off those caches. With an even semi-large dataset, this can really eat up Redis's memory. To limit the number of similarities cached in each matrix, specify a similarity_limit option when defining the matrix.
+By default, Predictor caches all similarities for all items, with no limit. That means if you have 10,000 items, and each item is somehow related to the other, we'll have 10,000 sets each with 9,999 items. That's going to use Redis' memory quite quickly. To limit this, specify the limit_similarities_to option.
 ```ruby
 class CourseRecommender
   include Predictor::Base
 
-  input_matrix :users, weight: 3.0, similarity_limit: 300
-  input_matrix :tags, weight: 2.0, similarity_limit: 300
-  input_matrix :topics, weight: 1.0, similarity_limit: 300
+  limit_similarities_to 500
+  input_matrix :users, weight: 3.0
+  input_matrix :tags, weight: 2.0
+  input_matrix :topics, weight: 1.0
 end
 ```
 
-This will ensure that only the top 300 similarities for each item are cached in each matrix. This can greatly reduce your memory usage, and if you're just using Predictor for scenarios where you maybe show the top 5 or so similar items, then this can be hugely helpful. But note, **don't set similarity_limit to 5 in that case**. This simply limits the similarities cached in each matrix, but does not limit the similarities for an item across all matrices. That is computed (and can be limited) on the fly, and uses the similarity cache in each matrix. So, you need a large enough cache in each matrix to determine an intelligent similarity list across all matrices.
+This can really save a ton of memory. Just remember though, predictions fetched with the predictions_for call utilzes the similarity caches, so if you're using predictions_for, make sure you set the limit high enough so that intelligent predictions can be generated. If you aren't using predictions and are just using similarities, then feel free to set this to the maximum number of similarities you'd possibly want to show!
 
-*Note*: This is a bit of a hack, and there are most certainly other ways to improve Predictor's memory usage for large datasets, but each appear to require a more significant change than the trivial implementation of similarity_limit above. PRs are quite welcome that experiment with these other ways :)
+Upgrading from 1.0 to 2.0
+---------------------
+As mentioned, 2.0.0 is quite a bit different than 1.0.0, so simply upgrading with no changes likely won't work. My apologies for this. I promise this won't happen in future releases, as I'm much more confident in this Predictor release than the last. Anywho, upgrading really shouldn't be that much of a pain if you follow these steps:
 
-Oh, and if you decide to tinker with your limit to try and find a sweet spot, I added a helpful method to ensure limits are obeyed to avoid regenerating all similarities. Of course, this only helps if you are decreasing the limit. If you're increasing it, you'll need to process similarities all over.
+* Change predictor.matrix.add_set! and predictor.matrix.add_single! calls to predictor.add_to_matrix!. For example:
+```ruby
+# Change
+predictor.topics.add_single!("topic-1", "course-1")
+# to
+predictor.add_to_matrix!(:topics, "topic-1", "course-1")
+
+# Change
+predictor.tags.add_set!("tag-1", ["course-1", "course-2"])
+# to
+predictor.add_to_matrix!(:tags, "tag-1", "course-1", "course-2")
+```
+* Change predictor.matrix.process! or predictor.matrix.process_item! calls to just predictor.process! or predictor.process_items!
+```ruby
+# Change
+predictor.topics.process_item!("course-1")
+# to
+predictor.process_items!("course-1")
+```
+* Change predictor.matrix.delete_item! calls to predictor.delete_from_matrix!. This will update similarities too, so you may want to queue this to run in a background job.
 ```ruby
-recommender.users.ensure_similarity_limit_is_obeyed!  # Remove similarities that disobey our current limit
-recommender.tags.ensure_similarity_limit_is_obeyed!
-recommender.topics.ensure_similarity_limit_is_obeyed!
+# Change
+predictor.topics.delete_item!("course-1")
+# to delete_from_matrix! if you want to update similarities to account for the deleted item (in v1, this was a bug and didn't occur)
+predictor.delete_from_matrix!(:topics, "course-1")
 ```
 
 Problems? Issues? Want to help out?