sbt >= 1.4.7
Should be available by default on the IC Cluster. Otherwise, refer to each project installation instructions.
Download the ml-100k.zip
dataset in the data/
folder:
> mkdir -p data
> cd data
> wget http://files.grouplens.org/datasets/movielens/ml-100k.zip
Check the integrity of the file with (it should give the same number as below):
> md5 -q ml-100k.zip
0e33842e24a9c977be4e0107933c0723
Unzip:
> unzip ml-100k.zip
Add your ratings in the data/personal.csv
file, by providing a numerical rating between [1,5] for at least 20 movies. For example, to rate the 'Toy Story' movie with '5', modify this line:
1,Toy Story (1995),
to this:
1,Toy Story (1995),5
Do include your own ratings in your final submission so we can check your answers against those provided in your report.
Important: Edit the data/personal.csv
file using your IDE/editor. Do not use applications such as Microsoft Excel, as there is the risk of changing the commas (,) to semicolons (;).
> sbt "runMain similarity.Predictor --train data/ml-100k/u1.base --test data/ml-100k/u1.test --json similarity.json"
> sbt "runMain knn.Predictor --train data/ml-100k/u1.base --test data/ml-100k/u1.test --json knn.json"
> sbt "runMain recommend.Recommender --data data/ml-100k/u.data --personal data/personal.csv --json recommendations.json"
Steps:
1. Ensure you only used the dependencies listed in ````build.sbt```` in this template, and did not add any other.
2. Remove ````project/project````, ````project/target````, and ````target/````.
3. Test that all previous commands for generating statistics, predictions, and recommendations correctly produce a JSON file (after downloading/reinstalling dependencies).
4. Remove the ml-100k dataset (````data/ml-100k.zip````, and ````data/ml-100k````), as well as the````project/project````, ````project/target````, and ````target/````.
5. Add your report and any other necessary files listed in the Milestone description (see ````Deliverables````).
6. Zip the archive.
7. Submit to the TA for grading.
Essential sbt: https://www.scalawilliam.com/essential-sbt/
Explore Spark Interactively (supports autocompletion with tabs!): https://spark.apache.org/docs/latest/quick-start.html
Scallop Argument Parsing: https://github.com/scallop/scallop/wiki
Spark Resilient Distributed Dataset (RDD): https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/rdd/RDD.html
JSON Serialization: https://github.com/json4s/json4s#serialization
Erick Lavoie (Design, Implementation, Tests)
Athanasios Xygkis (Requirements, Tests)