Add nosearch argument to skip auto search lib path

[vincentmli]

When run ecapture in Kubernetes pod, ecapture is given
--libssl, --pthread to specific lib location, but ecapture
continues to auto search gnutls lib and result in error

2022/06/05 16:46:48 pid info :3305486
2022/06/05 16:46:48 start to run EBPFProbeOPENSSL module
2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module <====
2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <===
2022/06/05 16:46:48 invalid argument <====

see #69

add nosearch argument to skip the auto search of tls libs when
in container that /etc/ld.so.conf and standard lib path like
/lib64, /usr/lib64 do not exist

Signed-off-by: Vincent Li vincent.mc.li@gmail.com

[vincentmli]

test to capture another container TLS traffic from container netshoot with ecapture

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k" --pthread="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0" --hex --single-mode=true
2022/06/07 17:39:30 pid info :1009316
2022/06/07 17:39:30 start to run EBPFProbeOPENSSL module
2022/06/07 17:39:30 single mode
2022/06/07 17:39:30 HOOK type:2, binrayPath:/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k
2022/06/07 17:39:30 libPthread so Path:/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0
2022/06/07 17:39:30 target all process. 


2022/06/07 17:39:49 PID:6846, Comm:bigipconfigdriv, TID:7001, Version:TLS1_2_VERSION, Send 353 bytes to [ADDR_NOT_FOUND], Payload:
GET /mgmt/tm/auth/partition/Common HTTP/1.1
Host: 10.169.72.34
User-Agent: python-requests/2.20.0 f5-icontrol-rest-python/1.3.13 k8s-bigip-ctlr-2.8.1-azure-2252-afaf2312050d8c16df43084b3d4382ce00125676
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: application/json
X-F5-Auth-Token: R2MLNGKBU3PI3PXTVHD3QAU5NX


2022/06/07 17:39:49 PID:6846, Comm:bigipconfigdriv, TID:7001, Version:TLS1_2_VERSION, Recived 639 bytes from [ADDR_NOT_FOUND], Payload:
HTTP/1.1 200 OK
Date: Tue, 07 Jun 2022 17:39:54 GMT
Server: Jetty(9.2.22.v20170606)
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/json; charset=UTF-8
Allow: 
Pragma: no-cache

[cfc4n]

I think that root cause must be found to solve the problem.
today, I will try to debug.

[vincentmli]

oh, I should explain it better, in https://github.com/ehids/ecapture/blob/master/cli/cmd/tls.go#L60-L120, the for _, modName := range modNames loop for each TLS modules, it calls each TLS modules Check() to find the lib for each TLS modules

		if e := conf.Check(); e != nil {
			logger.Printf("%v", e)
			continue
		}

when --libssl and --pthread is given to ecapture, in https://github.com/ehids/ecapture/blob/master/user/config_openssl.go#L30-L64, openssl found the right libssl so and pthread so, that is good, but above mentioned TLS modules for loop continues to look for gnutls modules library.

let's look at gnutls Check() https://github.com/ehids/ecapture/blob/master/user/config_gnutls.go#L26-L70, since we I did not give --gnutls to ecapture, gnutls Check() will find libgnutls.so by following line:

soPath, e := getDynPathByElf(this.Curlpath, "libgnutls.so")

and

111 // getDynPathByElf found soPath by soName from elfName
112 func getDynPathByElf(elfName, soName string) (string, error) {
113     
114         sos, e := getDynsFromElf(elfName)
115         if e != nil {
116                 return "", e
117         }   
118                     
119         // search dynamic library form ld.so.conf
120         var searchPath = GetDynLibDirs()
121         realSoName := recurseDynStrings(sos, searchPath, soName)
122             
123         // if not found soName from elfName
124         if len(realSoName) == 0 {
125                 return "", errors.New(fmt.Sprintf("cant found so lib from %s", elfName))
126         }   
127         return realSoName, nil
128 }

so in line 120, ecapture tries to search ibgnutls.so through /etc/ld.so.conf dynamic library directory, but the problem is /etc/ld.so.conf does not exist in the container that ecapture runs in, and the error reported as:

2022/06/05 16:46:48 start to run EBPFProbeGNUTLS module <====
2022/06/05 16:46:48 lstat /etc/ld.so.conf: no such file or directory <===
2022/06/05 16:46:48 invalid argument <====

after I take a further look, lstat /etc/ld.so.conf: no such file or directory should not be a fatal error to stop ecapture running, the invalid argument seems to be the fatal error result

 25 func GetDynLibDirs() []string {
 26         dirs, err := ParseDynLibConf("/etc/ld.so.conf")
 27         if err != nil {
 28                 log.Println(err.Error())
 29                 return []string{"/usr/lib64", "/lib64"}
 30         }
 31         return append(dirs, "/lib64", "/usr/lib64")
 32 }

so when /etc/ld.so.conf does not exist in container, it will get container /lib64, /usr/lib64

but in the container, /lib64 is symbolic link to /lib/libc.musl-x86_64.so.1, /usr/lib64 does not exist

bash-5.1# ls -l /lib64 /usr/lib64
ls: /usr/lib64: No such file or directory
/lib64:
total 0
lrwxrwxrwx    1 root     root            26 Jun  3 14:41 ld-linux-x86-64.so.2 -> /lib/libc.musl-x86_64.so.1

my guess is the fatal/panic error may eventually coming from line 135 loop for above /lib64 and /usr/lib64 that are not directory, but a symbolic link `` /lib/libc.musl-x86_64.so.1and non-existing/usr/lib64`

130 func recurseDynStrings(dynSym []string, searchPath []string, soName string) string {
131         var realSoName string
132         for _, el := range dynSym {
133                 // check file path here for library if it doesnot exists panic
134                 var fd *os.File
135                 for _, entry := range searchPath {

let me know if I am missing anything, so my solution is to skip getDynPathByElf() that started the whole lib searching path for each TLS modules when either --libssl, or/and --gnutls, or/and --nspr is/are given, does it make sense?

[vincentmli]

I pushed new changes, and I think this is better approach to solve the problem, it works for each TLS modules and solves the container auto lib search issue.

specifying --gnutls and --nosearch

ecapture tls --pthread=/lib/x86_64-linux-gnu/libpthread.so.0 --gnutls=/lib/x86_64-linux-gnu/libgnutls.so.30 --hex --nosearch=true
2022/06/08 15:40:37 pid info :254849
2022/06/08 15:40:37 start to run EBPFProbeOPENSSL module
2022/06/08 15:40:37 NoSearch requires specifying lib path
2022/06/08 15:40:37 start to run EBPFProbeGNUTLS module
2022/06/08 15:40:37 start to run EBPFProbeNSPR module
2022/06/08 15:40:37 NoSearch requires specifying lib path
2022/06/08 15:40:37 HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
2022/06/08 15:40:37 target all process. 

specifying both --libssl and --gnutls and --nosearch

root@cilium-demo-1:~# ecapture tls --pthread=/lib/x86_64-linux-gnu/libpthread.so.0 --gnutls=/lib/x86_64-linux-gnu/libgnutls.so.30 --libssl=/lib/x86_64-linux-gnu/libssl.so.1.1 --hex --nosearch=true
2022/06/08 15:41:35 pid info :255168
2022/06/08 15:41:35 start to run EBPFProbeOPENSSL module
2022/06/08 15:41:35 start to run EBPFProbeGNUTLS module
2022/06/08 15:41:35 start to run EBPFProbeNSPR module
2022/06/08 15:41:35 NoSearch requires specifying lib path
2022/06/08 15:41:35 HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
2022/06/08 15:41:35 target all process. 
2022/06/08 15:41:35 HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libssl.so.1.1
2022/06/08 15:41:35 libPthread so Path:/lib/x86_64-linux-gnu/libpthread.so.0
2022/06/08 15:41:35 target all process. 

normal run without --nosearch

root@cilium-demo-1:~# ecapture tls --hex
2022/06/08 15:52:07 pid info :258519
2022/06/08 15:52:07 start to run EBPFProbeOPENSSL module
2022/06/08 15:52:07 start to run EBPFProbeGNUTLS module
2022/06/08 15:52:07 HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libssl.so.1.1
2022/06/08 15:52:07 libPthread so Path:/lib/x86_64-linux-gnu/libpthread.so.0
2022/06/08 15:52:07 target all process. 
2022/06/08 15:52:07 start to run EBPFProbeNSPR module
2022/06/08 15:52:07 stat /lib/x86_64-linux-gnu/libnspr4.so: no such file or directory
2022/06/08 15:52:07 HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
2022/06/08 15:52:07 target all process. 

[cfc4n]

Thanks for your ideas. I found rootcause.

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/user/common.go#L161

fd value was nil ,so coredump with invalid argument

root cause of fd nil value, the reason is the same as you said.

ldd /usr/bin/wget
	/lib/ld-musl-x86_64.so.1 (0x7f365a47c000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f365a47c000)

wget require two lib so. but /etc/ld.so.conf does not exist. balabala.....

[cfc4n]

I send a commit in 97d8216 .

can you test it ,and merge into this PR ?

[vincentmli]

ok I will test your commit

[vincentmli]

your commit also solves the problem, so I will include your commit and my commit together in this PR, correct? I still prefer to keep my commit because it gives user option to not rely on lib auto search, what do you think @cfc4n

[root@cilium-dev home]# kubectl exec -it netshoot-ecap -- /bin/bash

bash-5.1# 
bash-5.1#  ecapture tls --pthread=/mnt/lib64/libpthread.so.0 --libssl=/mnt/lib64/libssl.so.1.1.1g --hex
2022/06/08 17:20:13 pid info :1932050
2022/06/08 17:20:13 start to run EBPFProbeOPENSSL module
2022/06/08 17:20:13 start to run EBPFProbeGNUTLS module
2022/06/08 17:20:13 lstat /etc/ld.so.conf: no such file or directory
2022/06/08 17:20:13 read file :/etc/ld.so.conf error .
2022/06/08 17:20:13 start to run EBPFProbeNSPR module
2022/06/08 17:20:13 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/08 17:20:13 HOOK type:2, binrayPath:/mnt/lib64/libssl.so.1.1.1g
2022/06/08 17:20:13 libPthread so Path:/mnt/lib64/libpthread.so.0
2022/06/08 17:20:13 target all process. 
2022/06/08 17:20:13 HOOK type:2, binrayPath:/usr/lib/libgnutls.so.30
2022/06/08 17:20:13 target all process. 

[vincentmli]

I included both commits in this PR :)

[cfc4n]

LGTM,thanks.