shell脚本

find multiple filename(extensions)

find . -type f $ -name "*.js" -o -name "*.css" $

find . -type f -name "*.js" -o -name "*.css"

unix-find-multiple-file-types

cut csv with comma value

sed -e 's/","\|",\|,"\|,,/@/g' /path/to/file | cut -d@ -f2

awk -F ',,|^"|","|",|,"|"$' '{print $3}' /path/to/file

How to use the command “cut” to cut out fields in a CSV file when fields contain commas?

find duplicate line in a file

sort <file> | uniq -c

sort <file> | uniq -c -d

sort: Write sorted concatenation of all FILE(s) to standard output

uniq: Filter adjacent matching lines from INPUT (or standard input),writing to OUTPUT (or standard output).

-c, –count prefix lines by the number of occurrences

-d, –repeated only print duplicate lines, one for each group

adjacent => 隣 => 邻近

find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated

create random string

tr -dc A-Za-z0-9 </dev/urandom | head -c 13 ; echo ''

how-to-generate-a-random-string

find exclude speicific dir,exclude dir in find

find . -name directory_to_exclude -prune -o ...

find . $ -name dir1_to_exclude -o -name dir2 ... $ -prune -o ...

exclude-a-directory-from-find-linux

unicode decode Java Property File unicode decode

cat file | xargs -0 printf

printf '%u4E2D'

output is 中

数学运算

add=$((one + two))

按照指定字符分割字符串以后去重

echo "ab,bc,cd,bc,ab,cd,ef" | awk -F"," '{for (i=1;i<=NF;i++) print $i}' | sort -u | sed '/^$/d' | awk '{print}' ORS=","

output:ab,bc,cd,ef

remove-all-duplicate-word-from-string-using-shell-script

how-to-split-a-delimited-string-into-an-array-in-awk

delete-empty-lines-using-sed

字符串正则比较

string="this is a string"
reg="this is a*"
if [[ $string =~ $reg ]]
then
    echo "matched"
else
    echo "not match"
fi

tips:要使用[[才能生效，使用[则不生效，$string和$reg不能加双引号

补位操作

#!/bin/bash
list=(localhost google.com nowhere)
C=1
for M in "${list[@]}"
do
    machine_indented=$(printf '%-20s' "$M")
    machine_indented=${machine_indented// /.}

    if ping -q -c 1  "$M" &>/dev/null ;  then
        printf "(%2d) %s CONNECTION OK\n" "$C" "$machine_indented"
    else
        printf "(%2d) %s CONNECTION FAIL\n" "$C" "$machine_indented"
    fi
    ((C=C+1))
done

输出结果

( 1) localhost........... CONNECTION OK
( 2) google.com.......... CONNECTION OK
( 3) nowhere............. CONNECTION FAIL

bash-using-printf-in-order-to-print-in-special-format

table格式输出打印table格式结果

for num in 1 10 100 1000 10000 100000 1000000; do printf "%10s %s\n" $num "foobar"; done
         1 foobar
        10 foobar
       100 foobar
      1000 foobar
     10000 foobar
    100000 foobar
   1000000 foobar

how-can-i-align-the-columns-of-tables-in-bash

column -t [file]

# or from stdin
cat file | column -t

# For a quick demonstration, format the output of mount
mount | column -t

how-can-i-align-the-columns-of-a-space-separated-table-in-bash

``和$()的区别

backticks-vs-braces-in-bash

合并多行输出为一行

ls -1 | paste -sd "," -

how-to-join-multiple-lines-of-file-names-into-one-with-custom-delimiter

ls -1 | awk '{print}' ORS=','

how-to-concatenate-multiple-lines-of-output-to-one-line

awk相比于paste可以避免在不同操作系统下换行符的问题，paste合并后会可能包含^M换行标志

nohup用法后台运行命令 excute command background and without no output

nohup /path/to/your/script.sh > /dev/null 2>&1 &

将输出重定向到/dev/null可以防止nohup输出”nohup: ignoring input and appending output to ‘nohup.out’“，改输出会导致终端session不切断，需要手动输入回车才行

how-to-run-a-command-in-the-background-and-get-no-output

find的exec结尾的`/`和`+`的区别

find dirname -type f -iname 'keyword' -exec vi -p {} \;

find dirname -type f -iname 'keyword' -exec vi -p {} +;

上面两个命令的区别是末尾的/和+

执行结果的差异是，前条命令相当于把find的所有结果一个一个的用vi打开，一次打开一个

后条命令相当于把find的所有结果当成一个参数，即用vi的tab模式打开所有的文件

即/是遍历的方式执行exec，+只执行一次exec，{}引用的所有文件列表

将shell脚本的内容同时输出到文件和控制台

file.sh

#!/bin/bash
LOG_FILE=/tmp/both.log
exec > >(tee ${LOG_FILE}) 2>&1
echo "this is stdout"
chmmm 77 /makeError

上述脚本必须使用bash的方式执行，使用sh的方式执行会报错即使用bash file.sh或者./file.sh

参考这个答案

write-outpus-to-log-file-and-console

bash和sh的区别

大体上理解bash是sh的超集

./file.sh这种方式执行文件是使用bash进行执行的，和bash file.sh的效果一样

difference-between-sh-and-bash

grep匹配多个关键字

通过使用-e选项来实现匹配多个关键字

echo "abcdef" | grep -e "e" -e "a"

match-two-strings-in-one-line-with-grep

Basic

Linux初心者のシェルスクリプト入門

Shellの基本を学ぶ

explainshell

the-art-of-command-line

cheat.sh

https://news.ycombinator.com/news

https://www.shellcheck.net/

不保存文件对比两个标准输出

diff <(some command with standard output) <(another command with standard output)

参考：https://stackoverflow.com/questions/3800202/diff-output-from-two-programs-without-temporary-files

脚本有错误时都自动退出 auto exit when on error

set -e

参考：https://stackoverflow.com/questions/2870992/automatic-exit-from-bash-shell-script-on-error

zip 不解压查看zip中的文件

zipinfo -l file.zip

参考： https://unix.stackexchange.com/questions/128303/how-to-list-files-in-a-zip-without-extra-information-in-command-line

按时间查找文件

find dir -iname '*.log' -mtime 0
find <path> -daystart -ctime 0 -print

参考:https://stackoverflow.com/questions/801095/how-do-i-find-all-the-files-that-were-created-today-in-unix-linux

通过tar压缩传输指定目录文件到server

tar zcf - some-dir | ssh some-server "cd /; tar xvzf -"

cut,uniq使用技巧分析csv指定列

cut -d',' -f4-7 myfilename.csv | uniq -D #以`,`分割文本，取第4-7列内容，然后输出不重复的内容
cut -d',' -f1,3 myfilename.csv | uniq -D #以`,`分割文本，取第1,3列内容，然后输出不重复的内容

confirm prompt

read -p "Are you sure? " -n 1 -r
echo    # (optional) move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
    # do dangerous stuff
fi

参考:how-do-i-prompt-a-user-for-confirmation-in-bash-script

read paramater 读取命令行参数

while [ "$1" != "" ]; do
    case $1 in
        -s  )   shift  
        SERVER=$1 ;;  
        -d  )   shift
        DATE=$1 ;;
    --paramter|p ) shift
        PARAMETER=$1;;
        -h|help  )   usage # function call
                exit ;;
        * )     usage # All other parameters
                exit 1
    esac
    shift
done

参考：酷壳网(打造高效的工作环境 – SHELL 篇)

#!/bin/bash
# Bash Menu Script Example
 
PS3='Please enter your choice: '
options=("Option 1" "Option 2" "Option 3" "Quit")
select opt in "${options[@]}"
do
    case $opt in
        "Option 1")
            echo "you chose choice 1"
            ;;
        "Option 2")
            echo "you chose choice 2"
            ;;
        "Option 3")
            echo "you chose choice $REPLY which is $opt"
            ;;
        "Quit")
            break
            ;;
        *) echo "invalid option $REPLY";;
    esac
done

参考：酷壳网(打造高效的工作环境 – SHELL 篇)

当选项比较多的时候，select默认会换行显示，可以通过设置COLUMNS=1来实现选项只显示一列

how-to-make-select-display-the-options-in-a-single-column

文件对比去重

comm -23 file1 file2

参考：

how-to-remove-the-lines-which-appear-on-file-b-from-another-file-a

deleting-lines-from-one-file-which-are-in-another-file

按照名字查找文件

find . -name "foo*"

注意要使用双引号

how-can-i-recursively-find-all-files-in-current-and-subfolders-based-on-wildcard

how-to-output-text-to-both-screen-and-file-inside-a-shell-script 将echo同时输出到屏幕和日志文件

command | tee -a "$log_file"

how-to-output-text-to-both-screen-and-file-inside-a-shell-script

查找文本

grep -rnw '/path/to/somewhere/' -e 'pattern'

how-do-i-find-all-files-containing-specific-text-on-linux

按分隔符拆分字符串并获取指定字符串

$ s='one_two_three_four_five'

$ A="$(cut -d'_' -f2 <<<"$s")"
$ echo "$A"
two

$ B="$(cut -d'_' -f4 <<<"$s")"
$ echo "$B"
four

split-string-by-delimiter-and-get-n-th-element

cd到上一个目录 how-to-go-to-the-previous-working-directory-in-terminal

cd -

cd $OLD_PWD

how-to-go-to-the-previous-working-directory-in-terminal

删除有乱码的文件或目录

文件或目录有乱码导致无法通过屏幕输入文件或目录名，可以通过incode进行删除

方法，找到inode并删除

ls -i

find ./ -inum nodenum -print -exec rm {} -rf \;

nodenum为ls -i看到inode号

取当前目录

#!/bin/bash

SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null && pwd )"
SOURCE="$(readlink "$SOURCE")"
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null && pwd )"

getting the source directory of as bash script from whithin

端口号占用

windows:

netstat -ano | findstr {查询端口号}

/dev/null 2>&1

/dev/null redirects standard output (stdout) to /dev/null, which discards it.

(The » seems sort of superfluous, since » means append while > means truncate and write, and either appending to or writing to /dev/null has the same net effect. I usually just use > for that reason.)

2>&1 redirects standard error (2) to standard output (1), which then discards it as well since standard output has already been redirected.

查看端口情况

netstat -ntlp

按名称查找进程

ps -ef | grep tomcat

统计tcp状态

netstat -n | awk '/^tcp/ {++y[$NF]} END {for(w in y) print w, y[w]}'

截取文本文件中指定范围的文本内容并输出到新文本，日志分析，文本内容查找

sed, a stream editor

通过grep找到要截取的文本所在行号cat -l | grep "要查找的文本内容"
通过sed命令，截取并输出到新文本sed -n 'start-line_num,end_line_nump' input.txt > output.txt

例如：sed -n '100,200p' input.txt > output.txt

上述命令将input.txt的第100行到200行截取后输出到output.txt

常见命令

grep -n 'text' filename.suffix 在filename.suffix中搜索’text’
\cp -rf test test1 拷贝test到test1目录下文件存在直接覆盖，命令前加入斜杠不会提示是否覆盖，否则由于cp命令一般有默认的-i选项所以会出现是否覆盖的提示
端口监听netstat

循环

declare -i i=1

while ((i<1000))

这里放要执行的命令

let i++

done

进程查看及操作

windows：

netstat -nao //列出所有进程以及其占用的端口

记住进程号

taskkill -PID 进程号 -F //强制关闭某个进程

linux：

netstat -naop

kill 进程号 -f