Table of Contents

1. Git
2. NPM helpers
3. Shell usefulness
4. Free Up Disk Space
5. Samba setup
6. Samba share access (unrestricted)
7. Samba share access (restricted)
8. Network goodies
9. Diff between files/folders
10. Protected archives
11. Pumping .bash_aliases
12. Encrypt/decrypt a file
13. Stress test via DoS attack
14. cURL cheatsheet
15. Wget basics
16. Installing programs from sources
17. Installing Oracle Java 8 / 9
18. FFmpeg sweets
19. Getting file info
20. Getting hardware info
21. Generate a random password
22. Amazon Web Services (AWS)
23. Docker
24. Cool cheatsheets / tutorials / helper tools
25. Bash-Snippets

---

Git

Documentation

expand section 🔻🔺

Check if merge conflicts will occur before actual merging:

git merge <branch> --no-ff --no-commit
git merge --abort

Discard all unstaged changes:

git checkout -- .

Create branch from particular commit:

git checkout -b <branch name> <commit hash>

Create a local branch that tracks a remote branch:

# starting to work on an existing remote branch (e.g. upstream/develop)
git checkout --track <remote branch name>

Add/stage (modified and deleted files only, new files are not affected) and commit in one command:

git commit -am "commit message"

Automate formation of release announcements:

git shortlog -s | awk -F\\t '{print $2}' > contributors.md
git shortlog --no-merges | awk -F '[[:alnum:] ]+ \\([0-9]' '{print $1}' | grep . > changelog.md

Get a nice list of actual different commits not shared between the branches:

git log --left-right --graph --cherry-pick --oneline master..develop

Generate a "top ten list" of most commited files, the "heat map" of the code changes. It could be used (along with other indicators, of course) to make informative decision about extracting certaint components into separate services.

git log --pretty=format: --name-only | sort | uniq -c | sort -rg | head -10

Housekeeping tools:

# cleans up unreachable or "orphaned" Git objects

# what is set to be pruned but not actually prune it
git prune --dry-run --verbose

# prune and display output of all objects and actions taken by it
git prune --progress

Ignore changes to a file that's already tracked in the repository. It's a common task and is very helpful in case of adding some local tokens, for example, to configuration file with boilerplate:

git update-index --assume-unchanged <file>
# tracking changes again
git update-index --no-assume-unchanged <file>

NPM

expand section 🔻🔺

Get debug info (useful for reports, GitHub issues, etc.):

npx envinfo --binaries --languages --system --utilities

Shell

expand section 🔻🔺

Check which init system your platform uses (systemd (systemctl command), or older System V (which uses the service command)):

ps --no-headers -o comm 1

Remove multiple sub-folders:

find . -type d -name node_modules -prune -exec rm -rf '{}' \;

Find patterns:

# find files containing a given text
find . -type f -print0 | xargs -0 grep -l "search string"
# "l" means that only the name of each input file with matched content will print
grep -rl "search string" /

# handle multiple search patterns
grep -e hacker -e root -e admin /etc/passwd

Sometimes process (Apache, for example) prevents to start service on the same port (nginx, for example):

# end the conflict process
sudo fuser -k 80/tcp

APT

# list all installed packages
apt list --installed | less

# list all ready-to-upgrade packages
apt list --upgradeable | less

# update list of available packages
sudo apt update

# upgrade the system by installing/upgrading packages
sudo apt upgrade

# search in package descriptions
apt search <needle_name>

# show package details
apt show <package_name>

# remove automatically all unused packages
sudo apt autoremove

# all-in-one upgrade system command
sudo apt update && sudo apt upgrade -y

Change/setup bash custom prompt (PS1) with Git branch displaying (if exists). Specify this in ~/.bashrc and run source ~/.bashrc for applying changes:

git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/[\1]/'
}

COLOR_USER='\[\e[1;32m\]'
COLOR_PATH='\[\e[00;36m\]'
COLOR_RESET='\[\e[0m\]'

PS1="${COLOR_RESET}${COLOR_USER}\u▶${COLOR_RESET} ${COLOR_PATH}\w ${COLOR_USER}\[\033[00;32m\]\$(git_branch)\[\033[00m\] $ ${COLOR_RESET}"

Result of the above is something like:

Fast checking of PHP SSL support:

echo '<?php phpinfo(); ?>' | php 2>&1 |grep -i ssl

Installing and basic setup of SSH Server:

sudo apt install openssh-server

# check status
service --status-all | grep ssh
# or
systemctl list-units | grep ssh

# config in /etc/ssh/sshd_config

# help
man sshd_config

# after making changes to the /etc/ssh/sshd_config file, save the file, and restart the sshd server to effect the changes using the following command:
sudo systemctl restart sshd.service

# in case of using SSH keys don't forget to copy the id_rsa.pub file to the remote host and append it to ~/.ssh/authorized_keys, then give it the right permissions:
chmod 600 .ssh/authorized_keys

Free Up Disk Space

expand section 🔻🔺

You can check the size of interested directory with du -sh <directory> preliminarily.

# clean the thumbnail cache
rm -rf ~/.cache/thumbnails/*

# remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed
sudo apt autoremove

# clean the apt cache
sudo apt-get clean

# remove old, unused kernels, except for the last two - the current and the previous
sudo purge-old-kernels

Samba setup

expand section 🔻🔺

sudo apt-get install -y samba samba-common python-glade2 system-config-samba
sudo mv /etc/samba/smb.conf /etc/samba/smb.conf.bak
sudo vi /etc/samba/smb.conf

[global]
workgroup = WORKGROUP
server string = Samba Server %v
netbios name = srvr1
security = user
map to guest = bad user
name resolve order = bcast host
wins support = no
dns proxy = no

Samba share access (unrestricted)

expand section 🔻🔺

sudo mkdir -p /samba/share
cd /samba
sudo chmod -R 0755 share
sudo chown -R nobody:nogroup share/

[share]
path = /samba/share
browsable = yes
writable = yes
guest ok = yes
read only = no

sudo service smbd restart

Samba share access (restricted)

expand section 🔻🔺

sudo mkdir -p /samba/share/secured
sudo addgroup securedgroup
cd /samba/share
sudo chown -R zhibirc:securedgroup secured
sudo chmod -R 0770 secured/
sudo usermod -a -G securedgroup zhibirc
sudo smbpasswd -a zhibirc

sudo vi /etc/samba/smb.conf

[secured]
path = /samba/share/secured
# valid users = zhibirc
valid users = @securedgroup
guest ok = no
writable = yes
browsable = yes

sudo service smbd restart

Network goodies

expand section 🔻🔺

Retrieve list of Samba master browser(s):

nmblookup -M -- -

Show NFS exports, like the showmount -e command:

nmap -sV --script=nfs-showmount 127.0.0.1

Mapping processes to system ports they listen for:

sudo netstat -tpln

Serve folder:

python -m SimpleHTTPServer 8080
# or
python3 -m http.server 8080
# or
sudo npm install http-server -g
http-server

Find out MAC address by using IP address:

arping -I eth0 -c 2 destination_ip

Using arp-scan allows to discover all IP hosts on the local network, including those that block all IP traffic such as firewalls and systems with ingress filters. It works on Ethernet and 802.11 wireless networks. Requires root privilege.

# "eth0" is used for example, in reality the network interface name depends on the OS, the network type and other factors
sudo arp-scan --interface=eth0 --localnet
# or
sudo arp-scan --localnet

Diff between files/folders

expand section 🔻🔺

# install "Meld", visual diff and merge tool for files, folders and VCS
sudo apt install meld
# diff between files
meld file1 file2
# diff between folders
meld dir1 dir2

Also it's possible and widely used to set Meld as a Git difftool and mergetool.

Protected archives

expand section 🔻🔺

Create encrypted ZIP archive (password as a plain text):

zip -P s0me_paSS -r protected.zip /home/sites/*/www/

Create encrypted ZIP archive (request to enter password), different choices:

zip --encrypt protected.zip file_name
zip --encrypt protected.zip file1 file2 file3
zip --encrypt -r protected.zip /home/user/folder/
zip --encrypt -r protected.zip /folder1/ /folder2/

ZIP supports a simple password-based symmetric encryption system, which is documented in the ZIP specification, and known to be seriously flawed, so don't use it for data with limited access.

Pumping .bash_aliases

expand section 🔻🔺

It's possible to put a lot of useful shortcuts in ~/.bash_aliases which can improve work effectiveness:

# General aliases
alias df="df -h"
alias du="du -c -h"
alias mkdir="mkdir -pv"
alias ls="ls --color=auto --group-directories-first"
alias ll="ls -lA"
alias lx="ll -BX"   # sort by extension
alias lz="ll -rS"   # sort by size
alias lt="ll -rt"   # sort by date
alias l.="ll -d .*" # show only hidden files
alias ..="cd .."
alias mnt="mount | column -t"
alias pwdgen="openssl rand -base64 30"
alias ports="netstat -tulanp" # quickly list all TCP/UDP port on the server
alias ping="ping -c 5" # stop after sending count ECHO_REQUEST packets
alias wget="wget -c" # can resume downloads
alias i="ifconfig"
alias net="netstat -tunlep"

# Git
alias ga='git add'
alias gp='git push'
alias gl='git log --pretty=format:"%h %ad | %s%d [%an]" --graph --date=short'
alias gs='git status'
alias gd='git diff'
alias gm='git commit'
alias gb='git branch'
alias gc='git checkout'
alias gf='git reflog'
alias gma='git commit -am'
alias gra='git remote add'
alias grr='git remote rm'
alias gpu='git pull'
alias gcl='git clone'
alias gta='git tag -a -m'

# Install NPM packages in Docker container to prevent security flaws
alias dnpm='docker run -it --rm -u=$UID:$(id -g $USER) -v "$PWD":/npm -w /npm node npm'
alias dnpx='docker run -it --rm -u=$UID:$(id -g $USER) -v "$PWD":/npm -w /npm node npx'
alias dnode='docker run -it --rm -u=$UID:$(id -g $USER) -v "$PWD":/npm -w /npm node node'
alias dyarn='docker run -it --rm -u=$UID:$(id -g $USER) -v "$PWD":/npm -w /npm node yarn'

Encrypt/decrypt a file

expand section 🔻🔺

Use the built-in gpg tool:

# encrypt
gpg -c important.data.txt
# decrypt
gpg important.data.txt.gpg

Stress test via DoS attack

expand section 🔻🔺

Using ab (Apache HTTP server benchmarking tool). Official docs: link

ab -k -c 350 -n 20000 example.com

For testing multiple URL's concurrently create a shell script with multiple ab calls:

#!/bin/sh

ab -n 100 -c 10 example.com/login > test1.txt &
ab -n 100 -c 10 example.com/news > test2.txt &

Using Siege:

siege -d10 -c50 example.com

cURL

expand section 🔻🔺

Debug options --verbose (-v), --trace, --trace-ascii, --trace-time allow to get more details as they show EVERYTHING curl sends and receives.

# use "-" as filename to have the output sent to stdout
curl --trace-ascii - http://www.example.com/

Make GET request, only print the response headers and display the time it took:

curl -sIX GET -w "Total time: %{time_total} s\n" www.example.com
# or
curl -o /dev/null -D- www.example.com

Typical usage, send GET request with headers:

curl -X GET 'http://www.example.com' -H 'Accept-Language: en' -H 'Authorization: Bearer A0v7mf98JJvWQTEbpEYNTt0uw2q0yl6P' -H 'Content-Type: application/json'

POST request format depends on content type (application/x-www-form-urlencoded is the default):

# or simply -d
curl --data "param1=value1&param2=value2" -X POST https://example.com/resource.cgi
curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://www.example.com

Identify the HTTP options available on the target URL, including the various types of allowed HTTP methods:

curl -v -X OPTIONS http://www.example.com/

Wget

expand section 🔻🔺

Downloading an entire Web Site:

# download the entire Web site
# convert links so that they work locally, off-line
# download all the files that are necessary to properly display a given HTML page
# guarantee that only the files below a certain hierarchy will be downloaded
# wait the specified number of seconds between the retrievals
# cause the time between requests to vary between 0.5 and 1.5 * wait (see above) seconds
wget --recursive --convert-links --page-requisites --no-parent --wait=5 --random-wait http://www.example.com/

You may want to specify --user-agent option which allows you to change the "User-Agent" line. Specifying empty user agent with --user-agent="" instructs Wget not to send the "User-Agent" header in HTTP requests.

Installing programs from sources

expand section 🔻🔺

tar xzvf program.sources.tar.gz
cd program.sources
# configure and compile
# if README is present, read it first
./configure
make
sudo make install
# clean up any temp files, optional
make clean

Installing Oracle Java 8 / 9

expand section 🔻🔺

sudo add-apt-repository ppa:webupd8team/java
sudo apt update; sudo apt install oracle-java8-installer
# or replace oracle-java8-installer with oracle-java9-installer to install Java 9
# check the Java version
javac -version
# set Java environment variables
sudo apt install oracle-java8-set-default

FFmpeg

expand section 🔻🔺

Get metadata information from media file:

# work on any file FFmpeg supports
ffmpeg -i video.mp4 -hide_banner
# advanced method using FFprobe, multimedia stream analyzer
ffprobe -v error -show_format -show_streams video.mp4

Convert MP4 video to MP3 audio:

ffmpeg -i video.mp4 audio.mp3
# or, with additional options
ffmpeg -i video.mp4 -b:a 192k -vn audio.mp3

Convert RTSP stream to HLS:

ffmpeg -i rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov -fflags flush_packets -max_delay 2 -flags -global_header -hls_time 2 -hls_list_size 3 -vcodec copy -y video.m3u8

Split a video into images:

mkdir video; ffmpeg -i video.mp4 image%d.jpg

Reduce the file size of MP4 file:

# get file information
ffmpeg -i video.mp4
# 497 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
# reduce bitrate by approximately half
ffmpeg -i video.mp4 -b 248k video.out.mp4

Crop video file:

ffmpeg -i video.mp4 -ss 00:00:03 -t 00:00:08 -async 1 fragment.mp4

Getting file info

expand section 🔻🔺

# display file or file system status
stat file.name
# get basic file info, recognize the type of data contained in
file file.name
# or, for getting mime type
file -i file.name
# read image metadata, ImageMagick is required
# get format and characteristics of one or more image files
identify -verbose file.name 

In case of media file container used by a multimedia stream use information from FFmpeg.

Getting hardware info

expand section 🔻🔺

Overall:

uname -a

sudo dmidecode | less

sudo lshw | less

# pretty print
sudo lshw -html > system_info.html

Specific:

# list USB devices
lsusb

# list all PCI devices
lspci

# CPU
cat /proc/cpuinfo
# or more precise form
lscpu

# RAM
cat /proc/meminfo
free

# SSD/HDD
sudo fdisk -l

Or, if you prefer some GUI tool, use hardinfo (sudo apt install hardinfo).

Generate a random password

expand section 🔻🔺

Amazon Web Services (AWS)

expand section 🔻🔺

Setup:

sudo apt install awscli
aws --version

# configuring
aws configure
AWS Access Key ID [None]: <associated with an IAM user>
AWS Secret Access Key [None]: <associated with an IAM user>
Default region name [None]: <any available region>
Default output format [None]: <json|text|table>

# or configure the same for another user
aws configure --profile <username> 

# enable command-completion feature

# locate the AWS Completer script, use this path in command below
which aws_completer

# in ~/.bashrc
complete -C '/usr/bin/aws_completer' aws
source ~/.bashrc 

Docker

Documentation

expand section 🔻🔺

Clean-up (containers, images, networks, cache):

alias d='docker' && d stop $(d ps -q) && d rm $(d ps -qa) && d rmi $(d images -q) && d network prune -f && d builder prune -f

Remove exited containers:

docker rm $(docker ps -a -q -f status=exited)

Build image in verbose mode (auto, plain, tty modes are available):

docker build . --tag <tag_name> --no-cache --progress=plain

Cool cheatsheets / tutorials / helper tools

• Bash-hackers wiki (bash-hackers.org)
• Shell vars (bash-hackers.org)
• Learn bash in y minutes (learnxinyminutes.com)
• Bash Guide (mywiki.wooledge.org)
• ShellCheck (shellcheck.net)

Bash-Snippets

A collection of small bash scripts for heavy terminal users with no dependencies: link