Simple statistics from the command line interface (CLI), fast.
This is a lightweight, fast tool for calculating basic descriptive statistics from the command line. Inspired by https://github.com/nferraz/st, this project differs in that it is written in C++, allowing for faster computation of statistics given larger non-trivial data sets.
Additions include the choice of biased vs unbiased estimators and the option to use the compensated variant algorithm.
Given a file of 1,000,000 ascending numbers, a simple test on a 2.5GHz dual-core MacBook using Bash time showed sta takes less than a second to complete, compared to 14 seconds using st.
Run ./autogen.sh, ./configure, make, and make install.
sta [options] < file
Imagine you have this sample file:
$ cat numbers.txt
1
2
3
4
5
6
7
8
9
10
Running sta is simple:
$ sta < numbers.txt
N max min sum mean sd
100 10 1 55 5.5 2.87228
To extract individual bits of information:
$ sta --sum --sd --var < numbers.txt
sum sd var
55 2.87228 8.25
sta, by default, assumes you have a population of scores, and thus normalises with N. If in fact you have a sample of scores, and wish to know the expected population standard deviation/variance, i.e. normalise with N-1, then just add the --sample flag. See Standard deviation estimation, and Population variance and sample variance:
$ sta --sum --sd --var --sample < numbers.txt
sum sd var
55 3.02765 9.16667
Worried about precision? You can calculate variance instead using the compensated variant algorithm:
$ sta --var --sample --compensated < numbers.txt
Want to compute quartiles? Run:
$ sta --q < numbers.txt
N min Q1 median Q3 max
100 1 26 50.5 76 100
How about percentiles? Run:
$ sta --p 50,61 < numbers.txt
50th 61th
51 62
Don't want to see the column names? Run:
$ sta --q --brief < numbers.txt
100 1 100 5050 50.5 29.0115
To transpose the output, run:
$ sta --q --transpose < numbers.txt
N 100
min 1
Q1 26
median 50.5
Q3 76
max 100
To supply your own delimiter, run:
$ sta --delimiter $'\t\t' --sd --sum < numbers.txt
sum sd
55 2.87228
or
$ sta --delimiter XXX --sd --sum < numbers.txt
sumXXXsd
55XXX2.87228
sta works with long doubles, and can process numbers in the following formats:
4.7858757E-39
4.7858757e-39
4.7858757
To change the output notation to fixed, supply the --fixed flag.
$ sta --fixed < numbers.txt
N min max sum mean sd sderr
10.000000 1.000000 10.000000 55.000000 5.500000 2.872281 0.908295
--brief
--compensated
--fixed
--mean
--median
--min
--max
--percentiles
--q
--q1
--q3
--sample
--sd
--sderr
--sum
--transpose
The example directory contains 2 scripts to create some example files:
$./examples/create_example_asc.pl n > some_file_with_n_numbers_asc
and
$./examples/create_example_rand.pl n > some_file_with_n_numbers_rand
To see how long st or sta takes to output the various statistics, call:
$./examples/time_sta.sh examples/large_file_1m
and
$./examples/time_st.sh examples/large_file_1m
--Add online variant
--Add confidence intervals for mean, var, and sd. This should allow for a user supplied interval.
I've not written C++ in a long time, so please do send comments, suggestions and bug reports to:
https://github.com/simonccarter/sta/issues
Or fork the code on github:
https://github.com/simonccarter/sta
I've recently integrated a basic testing platform using CxxTest. You'll need to download CxxTest and set the required environment variables to run the tests.
To build the tests, run the buildTester.sh file in the test directory.
Then run $ ./tester to run the tests.
Tests are not as extensive as they could be, and contributions are welcome here as well as the main code.