Sync to head (#2034)

* Refactor of oauth code (#1993)

Make cookie and JWT expiry configurable

* Send a System.Upload.Completion event on server artifact upload (#1995)

* Fixed CSS to make column selector more visible (#1996)

* Added new GUI column type for tree (#1997)

* Used by process_tracker_tree() to build a process tree
* Fixed linux pslist() which was very slow due to including a lot of
  unnecessary and expensive fields. We now only return the commonly
  used fields

* Collect domain role info on interrogate (#1998)

* Collect domain role info on interrogate

If populated on check in, domainrole can be used to auto-tag or filter down for certain hunts (ei: Domain controllers)

Ref: https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/win32-computersystem#:~:text=workgroup%20is%20returned.-,DomainRole,-Data%20type%3A

* Cleaned up domain role lookup and added a notebook suggestion

Co-authored-by: Mike Cohen <mike@velocidex.com>

* api/authenticators: fix handling of missing oauthstate cookie for OAUTH2 (#2000)

I was able to crash Velociraptor by requesting the github authenticator
callback URL directly with e.g. curl https://vrrserver/auth/github/callback

It turns out that there was no error handling if there is no 'oauthstate'
cookie provided as part of the request and we hit a nil pointer
dereference panic.  The Google and Azure authenticators had the same
issue.

This commit fixes all three and resolves #1999.

* Fixed bug in UserAccessLog artifact (#2008)

- IP field was not properly parsed - replaced with a parse_binary()
  version to ensure backwards compatibility.
- By default parse_ese() was using the "file" accessor which in 0.6.5
  was changed to not fallback to NTFS parsing. This means that since
  UAL files are locked, the parser was unable to access them. This PR
  sets the accessor to be "auto" explicitly thereby forcing the ntfs
  parsing if needed.

* Update UserAccessLogs.yaml (#2009)

Added rolename mappings and updated details.

* Fixed crash in api_client command (#2010)

Also allow the query command to specify an org id.

* Capitalize 'i' in config generation output (#2012)

* Added all() and any() VQL functions (#2013)

This makes it more efficient and simpler to filter by large number of
regex without adding a lot of AND clauses to the query.

* Fix sense of multiple regexp in all() function (#2014)

Now all regex must match all items.

* Cater for unknown parents in process tracker. (#2015)

When performing a full sync (e.g. pslist), some of the processes have
no valid parent at this time (because the parent e.g. exited).

We need to mark those unknown parents in case a new process reuses
those pids - in this case the process call chain can accidentally
include those parents.

* Bugix: Raw file accessor had different behaviour on Windows (#2018)

* Refactor code to propagate the context in more cases. (#2019)

* Refactor code to propagate the context in more cases.

* Fixed tests

* update to clean up null fields (#2020)

* update to clean up null fields
* update to clean up null fields tests

* Add embedded stager parse usecase (#34) (#2023)

* Add embedded stager parse usecase

* Add some test fixes

* Add test results

* Add test fix

* Update Linux pslist() to use CommandLine column (#2024)

This brings it in line with the same column name on Windows.

Also fixed a crash in user_grant() due to insufficient error
checking.

Fixes: #2022

* Bugfix: Switch GUI to first available org (#2025)

When a user is created with no access to the root org, the GUI did not
automatically switch the user to their own org. This caused an issue
where the user was rejected (because by default they were trying to
access the root org) but there was no way to switch even manually to
the correct org.

This PR updates the user's preferences to the first available org
automatically allowing the user to log in and select other orgs
manually.

* Refactor client monitoring API to use service (#2027)

Also made maximum VFS directory size configurable.

Fixes: #2005

* Added regex protocols for int, float etc. (#2028)

* Bugfix: Maintain field order in sysmon based tracker (#2030)

When following ETW the EventData is an unordered map so we need to
explicitly build a dict() to maintain consistent ordering.

Also fixed bug in USN artifact

* Bugfix: watch_usn() was not flushing the mft LRU properly (#2032)

This caused it to stop emitting rows after a while because it was
unable to see new data.

* [Snyk] Upgrade ace-builds from 1.8.1 to 1.9.3 (#2033)

fix: upgrade ace-builds from 1.8.1 to 1.9.3

Snyk has created this PR to upgrade ace-builds from 1.8.1 to 1.9.3.

See this package in npm:
https://www.npmjs.com/package/ace-builds

See this project in Snyk:
https://app.snyk.io/org/scudette/project/76f4d127-566b-42ef-86f4-bdcbc92b90b4?utm_source=github&utm_medium=referral&page=upgrade-pr

* Prepare for the 0.6.6-rc2 release

Co-authored-by: svch0stz <8684257+svch0stz@users.noreply.github.com>
Co-authored-by: Jeff Mahoney <jeffm@suse.com>
Co-authored-by: baileys20055 <81445894+baileys20055@users.noreply.github.com>
Co-authored-by: weslambert <wlambertts@gmail.com>
Co-authored-by: Matthew Green <mgreen27@users.noreply.github.com>
Co-authored-by: Snyk bot <snyk-bot@snyk.io>

accessors/manipulators.go

@@ -611,3 +611,41 @@ func NewFileStorePath(path string) (*OSPath, error) {
 	err := manipulator.PathParse(path, result)
 	return result, err
 }
+
+// The OSPath object for raw files is unchanged - We must pass exactly
+// the same form as given to the underlying filesystem APIs. On
+// Windows this is some kind of device description like
+// \\?\GLOBALROOT\Device\Harddisk0\DR0 for example, but we never
+// attempt to parse it - just forward to the API as is.
+type RawFileManipulator struct{}
+
+func (self RawFileManipulator) AsPathSpec(path *OSPath) *PathSpec {
+	result := &PathSpec{}
+	if len(path.Components) == 0 {
+		return result
+	}
+
+	result.Path = path.Components[0]
+	return result
+}
+
+func (self RawFileManipulator) PathJoin(path *OSPath) string {
+	if len(path.Components) == 0 {
+		return ""
+	}
+	return path.Components[0]
+}
+
+func (self RawFileManipulator) PathParse(
+	path string, result *OSPath) error {
+	result.Components = []string{path}
+	return nil
+}
+
+func NewRawFilePath(path string) (*OSPath, error) {
+	manipulator := &RawFileManipulator{}
+	return &OSPath{
+		Components:  []string{path},
+		Manipulator: manipulator,
+	}, nil
+}

accessors/raw_file/raw_file.go

@@ -26,7 +26,7 @@ import (
 type RawFileSystemAccessor struct{}
 
 func (self RawFileSystemAccessor) ParsePath(path string) (*accessors.OSPath, error) {
-	return accessors.NewPathspecOSPath(path)
+	return accessors.NewRawFilePath(path)
 }
 
 func (self RawFileSystemAccessor) New(scope vfilter.Scope) (
@@ -54,7 +54,7 @@ func (self RawFileSystemAccessor) ReadDirWithOSPath(
 
 func (self RawFileSystemAccessor) OpenWithOSPath(
 	filename *accessors.OSPath) (accessors.ReadSeekCloser, error) {
-	return self.Open(filename.String())
+	return self.Open(filename.Path())
 }
 
 func (self RawFileSystemAccessor) Open(filename string) (accessors.ReadSeekCloser, error) {

acls/proto/acl.proto

@@ -49,7 +49,6 @@ message ApiClientACL {
     // A list of roles in lieu of the permissions above. These will be
     // interpolated into this ACL object.
     repeated string roles = 9;
-
 }
 
 // A role is a named sets of ACL permissions. A user may possess

actions/events_test.go

@@ -121,11 +121,12 @@ func (self *EventsTestSuite) TestEventTableUpdate() {
 	// We definitely need to update the table on this client.
 	assert.True(self.T(),
 		client_manager.CheckClientEventsVersion(
+			context.Background(),
 			self.ConfigObj, self.client_id, version))
 
 	// Get the new table
 	message := client_manager.GetClientUpdateEventTableMessage(
-		self.ConfigObj, self.client_id)
+		context.Background(), self.ConfigObj, self.client_id)
 
 	// Only one query will be selected now since no label is set
 	// on the client.
@@ -151,7 +152,7 @@ func (self *EventsTestSuite) TestEventTableUpdate() {
 	// We no longer need to update the event table - it is up to date.
 	assert.False(self.T(),
 		client_manager.CheckClientEventsVersion(
-			self.ConfigObj, self.client_id,
+			context.Background(), self.ConfigObj, self.client_id,
 			actions.GlobalEventTableVersion()))
 
 	// Now we set a label on the client. This should cause the
@@ -163,18 +164,18 @@ func (self *EventsTestSuite) TestEventTableUpdate() {
 	label_manager.(*labels.Labeler).Clock = self.Clock
 
 	require.NoError(self.T(),
-		label_manager.SetClientLabel(self.ConfigObj, self.client_id,
-			"Foobar"))
+		label_manager.SetClientLabel(
+			context.Background(), self.ConfigObj, self.client_id, "Foobar"))
 
 	// Setting the label will cause the client_monitoring manager
 	// to want to upgrade the event table.
 	assert.True(self.T(),
 		client_manager.CheckClientEventsVersion(
-			self.ConfigObj, self.client_id,
+			context.Background(), self.ConfigObj, self.client_id,
 			actions.GlobalEventTableVersion()))
 
 	new_message := client_manager.GetClientUpdateEventTableMessage(
-		self.ConfigObj, self.client_id)
+		context.Background(), self.ConfigObj, self.client_id)
 
 	assert.True(self.T(), new_message.UpdateEventTable.Version >
 		message.UpdateEventTable.Version)
@@ -206,19 +207,19 @@ func (self *EventsTestSuite) TestEventTableUpdate() {
 
 	// Now lets set the label to Label1
 	require.NoError(self.T(),
-		label_manager.SetClientLabel(self.ConfigObj, self.client_id,
-			"Label1"))
+		label_manager.SetClientLabel(
+			context.Background(), self.ConfigObj, self.client_id, "Label1"))
 
 	// We need to update the table again (takes a while for the
 	// client manager to notice the label change).
 	vtesting.WaitUntil(5*time.Second, self.T(), func() bool {
 		return client_manager.CheckClientEventsVersion(
-			self.ConfigObj, self.client_id,
+			context.Background(), self.ConfigObj, self.client_id,
 			actions.GlobalEventTableVersion())
 	})
 
 	new_message = client_manager.GetClientUpdateEventTableMessage(
-		self.ConfigObj, self.client_id)
+		context.Background(), self.ConfigObj, self.client_id)
 
 	// The new table has 2 event queries - one for the All label
 	// and one for Label1 label.