Skip to content
This repository has been archived by the owner on Nov 17, 2020. It is now read-only.
/ go-nlp Public archive

Utilities for working with discrete probability distributions and other tools useful for doing NLP work

Notifications You must be signed in to change notification settings

nuance/go-nlp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GNLP

A few structures for doing NLP analysis / experiments.

Basics

  • counter.Counter

A map-like data structure for representing discrete probability distributions. Contains an underlying map of event -> probability along with a probability for all other events. Supports some element-wise mathematical operations with other counter.Counter objects.

// Create a counter with 0 probability for unknown events (and with ""
// corresponding to the unknown event)
balls := counter.New(0.0)
	
// Add some observations
balls.Incr("blue")
balls.Incr("blue")
balls.Incr("red")

// Normalize into a discrete distribution
balls.Normalize()

// blue => 0.666666
balls.Get("blue")

// purple => 0.0
balls.Get("purple")

preference = counter.New(0.0)
preference.Set("red", 2.0)
preference.Set("blue", 1.0)
preference.Normalize()

expected_with_preference = counter.Multiply(balls, preference)
expected_with_preference.Normalize()

// blue => 0.5
expected_with_preference.Get("blue")
// red => 0.5
expected_with_preference.Get("red")

// You can also use log probabilities
balls.LogNormalize()
preferences.LogNormalize()

// And do in-place operations
balls.Add(preferences)

// Log-normalize expects counters with positive counts, so
// exponentiate-then-normalize
balls.Exp()
balls.LogNormalize()

// blue => -1 (== lg(0.5))
balls.Get("blue")
  • frozencounter.Counter

Similar to counter.Counters, but with a fixed set of keys and no default value. Represented under the hood as an array of doubles (with order fixed according to the set of keys). Supports element-wise math operations with other frozencounter.Counters that share the same set of keys. Some mathematical operations are accelerated by the BLAS library.

fBalls := frozencounter.Freeze(balls)
fPrefs := frozencounter.Freeze(preference)

fExpectedWithPreference := frozencounter.Multiply(fBalls, fPrefs)

About

Utilities for working with discrete probability distributions and other tools useful for doing NLP work

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published