Skip to content
This repository has been archived by the owner on Dec 2, 2020. It is now read-only.

Infochimps merge #129

Closed
wants to merge 93 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
af74865
Upgrade mongodb driver and use native BSON parser
Marsup Aug 10, 2012
e0d6061
improved logging and meta-cubifying of progress reports.
Aug 27, 2012
b2e9ece
Added authentication: allow_all, read_only, or mongo_cookie
Aug 28, 2012
8b57375
applied metalog to rest of code, added comments on major plot lines
Aug 28, 2012
59ec553
Merge branch 'pr2_authentication'
Aug 28, 2012
414d234
Log evaluator queries when in verbose mode
Aug 28, 2012
9889b33
re-adding visualizer
Aug 28, 2012
29b5011
Save metrics with zero-counts back to the DB
Aug 28, 2012
0e369cd
Merge remote-tracking branch 'marsup/mongodb-native-parser' into working
Aug 28, 2012
06ac46d
vows no longer stalls trying to run test.js
Aug 28, 2012
46166ab
Fixed the dancing heisen-fails in test/metric-test
Aug 28, 2012
efd3f51
graceful shutdown of server and flushers
Aug 29, 2012
7435bff
richer tests for server components; test helpers for setting up, star…
Aug 29, 2012
e608edc
travis.yml
Aug 29, 2012
f64cdb7
added test 'script' to package.json, for make travis-ci happy
Aug 29, 2012
262bfa3
trying to make travis-ci happy
Aug 29, 2012
ab04068
More tests around server, including UDP, HTTP and static file.
Aug 29, 2012
8ca2235
Temporarily disable complex metric expressions
hustonhoburg Aug 30, 2012
e0083f1
Move configs into single config script
hustonhoburg Aug 30, 2012
4770ac2
Add mongo server config to cube config
hustonhoburg Aug 30, 2012
be66b98
Events and metrics collections are created from config file
hustonhoburg Aug 30, 2012
3645464
Configurable mongo connection
hustonhoburg Aug 30, 2012
aa4fafd
Add horizons for calculations and invalidations when processing event…
hustonhoburg Aug 31, 2012
730ffe1
Cascade tiers down to 10 seconds
hustonhoburg Aug 31, 2012
e592724
Add warmer to refresh metrics
hustonhoburg Aug 31, 2012
957222f
Remove paraenthesis limitation
hustonhoburg Aug 31, 2012
402f68b
use strict mode everywhere
Sep 3, 2012
740d26c
doing scale experiments, so reverting the workaround-y stuff (no dist…
Sep 3, 2012
4581841
Extracting metric and event code into objects (WIP)
Sep 3, 2012
2635db0
adding underscore lib
Sep 3, 2012
7881f54
separated invalidation logic from putter (preparation for implementat…
Sep 3, 2012
3b79995
JSLinted files, groomed away the lint
Sep 3, 2012
865f340
DB shouldn't reconnect in test suite
Sep 3, 2012
6578fe2
Reworking metric code -- objectified measurements into separate metho…
Sep 3, 2012
50f759a
Made lint shut up about 'export' keyword
Sep 4, 2012
b51881a
added a 'spy' helper to metalog -- wraps a callback, shows the call a…
Sep 4, 2012
84c47e4
Added mbox queue, to better schedule queries under load
Sep 5, 2012
4151aab
Extract database interactions into separate file. Add config to allow…
hustonhoburg Sep 6, 2012
f6967f5
cleaned up broker and metrics in advance of merging new DB code
Sep 7, 2012
c3a5870
adding tracing and queuing (WIP)
Sep 9, 2012
81086f0
Added a trace system -- tracks internal progress through whole call s…
Sep 10, 2012
f1c9715
getting tests to run non-isolated
Sep 10, 2012
e2be0b1
Misc bug fixes
hustonhoburg Sep 13, 2012
1112467
Fix stuff
hustonhoburg Sep 13, 2012
5fe8844
Add warmer test
hustonhoburg Sep 13, 2012
a1b7107
Update random streamer example to be real time
hustonhoburg Sep 13, 2012
f453426
Stop random streamer example from batching
hustonhoburg Sep 19, 2012
f6cfbc8
Update streamer, remove tracing from events
hustonhoburg Sep 19, 2012
a12ec7f
Events include their bins when they are saved. Tiers cascade down to …
hustonhoburg Sep 24, 2012
3e6bb52
Add array and object types to random (cromulator) example
hustonhoburg Sep 24, 2012
c629714
Modelize event, metric, and measurement objects
hustonhoburg Oct 2, 2012
8e8971e
Add semi pyramidal calculations for non-pyramidal reduces
hustonhoburg Oct 4, 2012
bef5e48
Prepend numeric event data keys with 'k'
hustonhoburg Oct 4, 2012
36c78fd
Add horizons back in, complete with tests
hustonhoburg Oct 8, 2012
48b0060
Fix metrics bugs.
hustonhoburg Oct 16, 2012
9e77305
Send parse errors back to user for malformed queries
hustonhoburg Oct 16, 2012
527d7e0
Fix random-streamer example model references
hustonhoburg Nov 2, 2012
1f6f78d
Remove commented out code sections
hustonhoburg Nov 2, 2012
438efa3
Modelize invalidator
hustonhoburg Nov 2, 2012
b6d3524
Fix error when running from_wire on row with no vs value
hustonhoburg Nov 6, 2012
8208012
Add "jake" task to remove metrics older than the expiration horizon
hustonhoburg Nov 6, 2012
057e392
Only store intermediate metric values for nonpyramidal reduces
hustonhoburg Nov 6, 2012
76873e1
Oops, metrics weren't saving/cascading correctly
hustonhoburg Nov 6, 2012
c5392aa
Merge branch 'production'
hustonhoburg Nov 6, 2012
347600e
Merge branch 'working'
hustonhoburg Nov 6, 2012
59ec91d
Events only need validated once. Event putter and event.save were bot…
hustonhoburg Nov 9, 2012
b030102
Add grouping to metric queries.
hustonhoburg Nov 30, 2012
9218417
Board URLs no longer include organization
hustonhoburg Dec 19, 2012
b8bef17
Fix bug for warmer for empty boards
hustonhoburg Dec 19, 2012
3628932
Merge remote-tracking branch 'infochimps/master'
Marsup Jul 24, 2013
129aff8
Add queue-async dependency instead of submodule
Marsup Aug 6, 2013
edfcbd4
Fix database name
Marsup Aug 6, 2013
6de91ee
Change configuration style
Marsup Aug 1, 2013
6ccef0a
Fix metric tests by rising the timeout
Marsup Aug 6, 2013
67c08b1
merge square/master into Marsup/infochimps-merge
RandomEtc Sep 8, 2013
5dc75c1
remove use of process.nextTick
RandomEtc Sep 8, 2013
bf2400f
remove unused broker
RandomEtc Sep 8, 2013
0b1a018
bump bcrypt
RandomEtc Sep 8, 2013
c0b8972
Fix collector loop, forEach provides numbers as cb
Marsup Aug 14, 2013
ad9a0d3
One connection per db, that fixes auth problems
Marsup Aug 13, 2013
137f278
Store metrics with undefined values as well
Marsup Aug 9, 2013
201104b
Improve collectors packets aggregation
Marsup Sep 9, 2013
16eb102
Remove unused files
Marsup Sep 9, 2013
976d681
Remove bins persistence
Marsup Sep 9, 2013
44b16fd
Allow group-based queries to have empty answers
Marsup Aug 20, 2013
7e5954f
Upgrade node-static
Marsup Sep 9, 2013
1963e47
Update parsers with latest peg
Marsup Sep 9, 2013
0b9c7e7
Fix Makefile
Marsup Sep 9, 2013
51f1290
Fix example's configuration key
Marsup Sep 9, 2013
f237776
Move configuration file into lib
Marsup Sep 13, 2013
05ab0b4
Revert configuration style to POJO
Marsup Sep 13, 2013
f8de945
Complete the interface binding work
Marsup Oct 24, 2013
e0b19e8
If udp is not configured then don't bind udp.
splbio Oct 25, 2013
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
node_modules
.jshintrc
Procfile
away/*
144 changes: 144 additions & 0 deletions .jslint.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
#
# Configuration File for JavaScript Lint 0.3.0
# Developed by Matthias Miller (http://www.JavaScriptLint.com)
#
# This configuration file can be used to lint a collection of scripts, or to enable
# or disable warnings for scripts that are linted via the command line.
#

### Warnings
# Enable or disable warnings based on requirements.
# Use "+WarningName" to display or "-WarningName" to suppress.
#
-no_return_value ## function {0} does not always return a value
+duplicate_formal # duplicate formal argument {0}
+equal_as_assign # test for equality (==) mistyped as assignment (=)?{0}
+var_hides_arg # variable {0} hides argument
+redeclared_var # redeclaration of {0} {1}
-anon_no_return_value ## anonymous function does not always return a value
+missing_semicolon # missing semicolon
+meaningless_block # meaningless block; curly braces have no impact
-comma_separated_stmts ## multiple statements separated by commas (use semicolons?)
-unreachable_code ## unreachable code
+missing_break # missing break statement
+missing_break_for_last_case # missing break statement for last case in switch
+comparison_type_conv # comparisons against null, 0, true, false, or an empty string allowing implicit type conversion (use === or !==)
-inc_dec_within_stmt ## increment (++) and decrement (--) operators used as part of greater statement
+useless_void # use of the void type may be unnecessary (void is always undefined)
+multiple_plus_minus # unknown order of operations for successive plus (e.g. x+++y) or minus (e.g. x---y) signs
+use_of_label # use of label
-block_without_braces ## block statement without curly braces
+leading_decimal_point # leading decimal point may indicate a number or an object member
+trailing_decimal_point # trailing decimal point may indicate a number or an object member
+octal_number # leading zeros make an octal number
+nested_comment # nested comment
+misplaced_regex # regular expressions should be preceded by a left parenthesis, assignment, colon, or comma
-ambiguous_newline ## unexpected end of line; it is ambiguous whether these lines are part of the same statement
+empty_statement # empty statement or extra semicolon
-missing_option_explicit ## the "option explicit" control comment is missing
+partial_option_explicit # the "option explicit" control comment, if used, must be in the first script tag
+dup_option_explicit # duplicate "option explicit" control comment
+useless_assign # useless assignment
+ambiguous_nested_stmt # block statements containing block statements should use curly braces to resolve ambiguity
+ambiguous_else_stmt # the else statement could be matched with one of multiple if statements (use curly braces to indicate intent)
-missing_default_case ## missing default case in switch statement
+duplicate_case_in_switch # duplicate case in switch statements
+default_not_at_end # the default case is not at the end of the switch statement
+legacy_cc_not_understood # couldn't understand control comment using /*@keyword@*/ syntax
+jsl_cc_not_understood # couldn't understand control comment using /*jsl:keyword*/ syntax
+useless_comparison # useless comparison; comparing identical expressions
+with_statement # with statement hides undeclared variables; use temporary variable instead
+trailing_comma_in_array # extra comma is not recommended in array initializers
+assign_to_function_call # assignment to a function call
+parseint_missing_radix # parseInt missing radix parameter

### Output format
# Customize the format of the error message.
# __FILE__ indicates current file path
# __FILENAME__ indicates current file name
# __LINE__ indicates current line
# __ERROR__ indicates error message
#
# Visual Studio syntax (default):
+output-format __FILE__(__LINE__): __ERROR__
# Alternative syntax:
#+output-format __FILE__:__LINE__: __ERROR__


### Context
# Show the in-line position of the error.
# Use "+context" to display or "-context" to suppress.
#
+context

### Semicolons
# By default, assignments of an anonymous function to a variable or
# property (such as a function prototype) must be followed by a semicolon.
#
+lambda_assign_requires_semicolon

### Control Comments
# Both JavaScript Lint and the JScript interpreter confuse each other with the syntax for
# the /*@keyword@*/ control comments and JScript conditional comments. (The latter is
# enabled in JScript with @cc_on@). The /*jsl:keyword*/ syntax is preferred for this reason,
# although legacy control comments are enabled by default for backward compatibility.
#
+legacy_control_comments


### JScript Function Extensions
# JScript allows member functions to be defined like this:
# function MyObj() { /*constructor*/ }
# function MyObj.prototype.go() { /*member function*/ }
#
# It also allows events to be attached like this:
# function window::onload() { /*init page*/ }
#
# This is a Microsoft-only JavaScript extension. Enable this setting to allow them.
#
-jscript_function_extensions


### Defining identifiers
# By default, "option explicit" is enabled on a per-file basis.
# To enable this for all files, use "+always_use_option_explicit"
-always_use_option_explicit

# Define certain identifiers of which the lint is not aware.
# (Use this in conjunction with the "undeclared identifier" warning.)
#
# Common uses for webpages might be:
+define exports
#+define document


### Files
# Specify which files to lint
# Use "+recurse" to enable recursion (disabled by default).
# To add a set of files, use "+process FileName", "+process Folder\Path\*.js",
# or "+process Folder\Path\*.htm".
#
+process lib/cube/authentication.js
+process lib/cube/bisect.js
+process lib/cube/collectd.js
+process lib/cube/collector.js
+process lib/cube/emitter-http.js
+process lib/cube/emitter-udp.js
+process lib/cube/emitter-ws.js
+process lib/cube/emitter.js
+process lib/cube/endpoint.js
+process lib/cube/evaluator.js
+process lib/cube/event.js
+process lib/cube/index.js
+process lib/cube/metalog.js
+process lib/cube/metric.js
+process lib/cube/models.js
+process lib/cube/reduces.js
+process lib/cube/server.js
+process lib/cube/tiers.js
+process lib/cube/types.js
+process lib/cube/visualizer.js
+process lib/cube/warmer.js

+process test/*.js
+process bin/*.js
3 changes: 3 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
language: node_js
node_js:
- 0.8
38 changes: 38 additions & 0 deletions Jakefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
var forced_metric_expiration = 1000 * 60 * 60 * 24 * 7, // 7 days
Db = require('./lib/cube/db');

namespace("db", function(){
namespace("metrics", function(){
desc("Remove metrics with times past the forced metric expiration horizon")
task("remove_expired", [], function(){
var db = new Db(),
expiration_date = new Date(new Date() - forced_metric_expiration);

function handle(err) {
if(err) throw err;
}

db.open(function(err, db){
handle(err);
var metrics_db = db._dbs.metrics;
metrics_db.collectionNames({namesOnly: true}, function(err, names){
handle(err);
var metric_names = names.filter(function(name){ return /_metrics$/.test(name); }),
remaining = metric_names.length;
metric_names.forEach(function(name){
var segments = name.split('.'),
collection_name = (segments.shift(), segments.join('.'));
metrics_db.collection(collection_name, function(err, collection){
handle(err);
collection.remove({'_id.t': {$lt: expiration_date }}, function(err){
handle(err);
console.log('Removing ' + collection_name.split('_').join(' ') + ' older than ' + expiration_date);
if(!--remaining) db.close();
});
});
});
})
})
});
});
});
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ all: \
lib/cube/metric-expression.js

test: all
@$(JS_TESTER)
NODE_ENV=test $(JS_TESTER) --isolate --spec
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Cube

[![build status](https://secure.travis-ci.org/infochimps-labs/cube.png)](http://travis-ci.org/infochimps-labs/cube)

**Cube** is a system for collecting timestamped events and deriving metrics. By collecting events rather than metrics, Cube lets you compute aggregate statistics *post hoc*. It also enables richer analysis, such as quantiles and histograms of arbitrary event sets. Cube is built on [MongoDB](http://www.mongodb.org) and available under the [Apache License](/square/cube/blob/master/LICENSE).

Want to learn more? [See the wiki.](https://github.com/square/cube/wiki)
Loading