Feature request : RedELK logging integration

[MarcOverIP]

It would be awesome to have integration with RedELK(https://github.com/outflanknl/RedELK)

It seems all there is required is to have:

• RedWarden to produce detailed logging
• RedELK to have an connector/installer for RedWarden.

If you can produce the first, we can do the second.

Details on required logformat are detailed here: https://github.com/outflanknl/RedELK/wiki/Redirector-installation#Apache%20specifics.

Maybe add a config.yml directive to enable detailed logging? For example:

logging:
  - logging_type: redelk | basic
  - logging_file: $path

[mgeeky]

Hi @MarcOverIP , thanks for opening this issue and suggesting such an integration.
Of course I'm looking forward to add required changes to adapt to RedElk needs.

Can you please provide me with a few, anonymized sample log lines in a format expected by RedElk? Although I do have the format in a generalized syntax, a few examples would be helpful as well.

Cheers,
M.

[MarcOverIP]

Got you served here: https://github.com/outflanknl/RedELK/blob/master/example-data-and-configs/ExampleData/redira1_access-redelk.log

[mgeeky]

So I have the prototype implemented in redelk-integration branch:
https://github.com/mgeeky/RedWarden/tree/redelk-integration

The following options are required to be supplied in YAML configuration file for RedWarden to emit RedElk accepted log lines:

access_log: redelk.log
access_log_format: redelk
redelk_frontend_name: my-redir
redelk_backend_name: decoy

The resulting log can look like this:

[30/Aug/2021:20:00:57 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:199.19.224.165:38814 xforwardedfor:- headers:{Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0|213.32.20.232:80|-|-|-|-|-|} statuscode:301 request:GET /config/getuser?index=0 HTTP/1.1
[30/Aug/2021:20:01:37 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:443 backend:decoy client:144.86.173.74:32937 xforwardedfor:- headers:{Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com|binary-offensive.com|-|-|-|-|-|} statuscode:301 request:GET / HTTP/1.1
[30/Aug/2021:20:01:58 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:183.136.225.14:2805 xforwardedfor:- headers:{Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE|213.32.20.232:80|-|-|-|-|-|} statuscode:301 request:GET / HTTP/1.1
[30/Aug/2021:20:02:18 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:183.136.225.14:45583 xforwardedfor:- headers:{Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE|213.32.20.232:80|-|-|-|-|-|} statuscode:301 request:GET /favicon.ico HTTP/1.1
[30/Aug/2021:20:02:38 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:183.136.225.14:18777 xforwardedfor:- headers:{Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE|213.32.20.232:80|-|-|-|-|-|} statuscode:301 request:GET /robots.txt HTTP/1.1
[30/Aug/2021:20:03:36 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:183.136.225.56:14758 xforwardedfor:- headers:{Baiduspider+(+http://www.baidu.com/search/spider.htm);googlebot|baiduspider|baidu|spider|sogou|bingbot|bot|yahoo|soso|sosospider|360spider|youdaobot|jikeSpider;)|213.32.20.232|-|-|-|-|-|} statuscode:301 request:GET / HTTP/1.1
[30/Aug/2021:20:19:40 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:443 backend:decoy client:61.135.15.196:33718 xforwardedfor:- headers:{Mozilla/5.0 (Linux; Android 7.0; HUAWEI P20 Build/816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4472.114 Mobile Safari/537.36|binary-offensive.com|-|-|-|-|-|} statuscode:301 request:GET / HTTP/1.1
[30/Aug/2021:20:19:54 +0000] binary-offensive.com apache[12160]: frontend:my-redir/213.32.20.232:80 backend:decoy client:61.135.15.196:33396 xforwardedfor:- headers:{Mozilla/5.0 (Linux; Android 10.0; VIVO find 816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Mobile Safari/537.36|binary-offensive.com|-|-|-|-|-|} statuscode:301 request:GET / HTTP/1.1

@MarcOverIP can you give RedWarden a test go with RedElk?

Cheers,
M.

[MarcOverIP]

Looking good at first glance! I'll do some testing and let you know.

[PhoenixNP]

It's currently not logging the actions correctly.
It lists a static value towards RedElk which prevents parts of the filters and searches from working correctly. (You don't really know what happend with the traffic)

I reckon it should either use the static value you supply based on what action is taken.
Ie. Pass to C2-backend, Reject, Block, etc

[MarcOverIP]

I was under the impression that this was happening correctly. If traffic is not meant for the c2 server it is sent to backend decoy. If traffic is meant for the c2 it is sent to backend c2. Is it not @mgeeky ?

[mgeeky]

Currently I didn't yet added (because frankly I didn't understand what do we mean by frontend/backend labels) proper request labelling based on whether it was passed to C2 or was dropped.

Currently:

1. If packet matches Malleable C2 profile, it is redirected to Teamserver. Log is emitted with frontend/backend labels as configured in YAML.
2. If packet doesn't match Malleable C2 profile, it is (dropped | redirected elsewhere | proxied elsewhere) denoting it was fraudulent and we didn't want it reaching our Teamserver. Log is emitted with same values for frontend/backend labels.

I need a few minutes to add more options specifying labels for redirected/dropped requests.

If I understand correctly, we need two distinguished labels:

• one for accepted request
• one for dropped request

Is there anything else required? Still having bad time grasping this frontend/backend concept tbh. Didn't yet play with RedELK.

[MarcOverIP]

frontend label = the name of the web interface that the traffic was arriving on. This label is free format as far as RedELK concerns (well, technically, please don't use spaces). In practice we use names like http, http-main, http-trackingpixel1, phishingwebsitex
backend label = the name of the destination the traffic was sent to after the reverse proxy/RedWarden/whatever made its decision. RedELK requires either it to start either with c2 or decoy. Roughly said: was the traffic valid -> backend: c2*, was it not valid (backgorund internet scanners, or blue team poking investigation) -> decoy*

If you make the backend label fixed, you loose all possibilities for RedELK to support you with visibility and with alarming you when blue is poking at your infra.

A bit more info here: https://github.com/outflanknl/RedELK/wiki/Naming-requirements-within-RedELK

[mgeeky]

Perfect, now it's all clear to me!

Please find the following commit 131607e introducing necessary changes.

Now in order to properly setup RedWarden for RedELK integration, following options are required to be specified:

# Switches between one of the following pre-defined log formats:
# - 'apache2' combined access_log
# - 'redelk' log format
access_log_format: redelk

#
# ===================================
# RedELK Integration
#
#
# If RedWarden is to be integrated with RedElk, following three variables will have to be set
# according to this redirector server role.
#

# Label marking packets coming from this specific Proxy server.
# Can be anything, but nice candidates are:
# - http, http-proxy, http-trackingpixel1, phishingwebsite, etc
redelk_frontend_name: http-redwarden

# Label for packets that are passed to the C2 server.
# This value MUST start with "c2" and cannot contain spaces.
redelk_backend_name_c2: c2

# Label for packets that are NOT passed to the C2 (they either dropped, redirected, proxied away).
# This value MUST start wtih "decoy" and cannot contain spaces.
redelk_backend_name_decoy: decoy

And the log spool properly differentiates between c2 and decoy packets:

[24/Sep/2021:11:16:49 +0000] binary-offensive.com apache[29592]: frontend:http-redwarden/213.32.20.232:443 backend:c2 client:11.22.33.44:25310 xforwardedfor:- headers:{Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)|127.0.0.1:5555|-|-|-|-|-|} statuscode:200 request:GET /__utm.gif?utmac=UA-2202604-2&utmcc=__utmadlikfkbbcibeonakhcomijcdfcfjgjaieklbfoniiadkpaajlkfigijecofoemlcnmbdojciknknahmbmiboegcngfpbcpialkmkkbccamabifjihniojfkchimfejbkndpcpleffdiimcpaojmjahijfiemmhckhjalifjhdogemnaajblfjmhkkcallbdkacpkgmjcmnfkabagbaiphengiiodhfalingnilpfbpngidappoaplpiiolcblihd HTTP/1.1
[24/Sep/2021:11:16:55 +0000] binary-offensive.com apache[29592]: frontend:http-redwarden/213.32.20.232:443 backend:c2 client:11.22.33.44:25229 xforwardedfor:- headers:{Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)|127.0.0.1:5555|-|-|-|-|-|} statuscode:200 request:GET /__utm.gif?utmac=UA-2202604-2&utmcc=__utmadlikfkbbcibeonakhcomijcdfcfjgjaieklbfoniiadkpaajlkfigijecofoemlcnmbdojciknknahmbmiboegcngfpbcpialkmkkbcc
[24/Sep/2021:11:16:46 +0000] binary-offensive.com apache[29592]: frontend:http-redwarden/213.32.20.232:443 backend:decoy client:11.22.33.44:25303 xforwardedfor:- headers:{curl/7.78.0|b1o.it|-|-|-|-|-|} statuscode:301 request:GET /drop-me-like-im-mic HTTP/1.1
[24/Sep/2021:11:16:47 +0000] binary-offensive.com apache[29592]: frontend:http-redwarden/213.32.20.232:443 backend:decoy client:11.22.33.44:25318 xforwardedfor:- headers:{curl/7.78.0|b1o.it|-|-|-|-|-|} statuscode:301 request:GET /drop-me-like-im-mic HTTP/1.1

Hopefully that will work now.

Thanks for clarifying the concept around frontend and backend labels to me! I couldn't really get grasp of it up to now.

Regards,
M.

[MarcOverIP]

Looking good. First quick check on parsing the log lines is looking good. Now it's up to us (the RedELK dev team) to adjust the RedELK installers and to include a log line parser for RedWarden.