Shell and Basic Commands
A shell is a program1 that inputs Unix commands from the keyboard and relays them to the Unix system for execution. Shells typically include various shortcuts for users to use in stating their commands, and also a programming feature, in which users can make programs out of sets of their commands.
There are many different shells available for Unix systems. Here we focus on two of the most popular ones, tcsh and bash. Other shells tend to be very similar to one or both of these.
Switching from One Shell to Another
1
2
3
4
|
$ bash # temporarily use another shell
$ exit # end shell session
$ bash x.sh # run a script using bash
$ chsh -s bash # make bash as a default shell
|
Browsing the Directory Tree and Managing Files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
$ pwd # print working directory, displays the current location
/home/rhertzog
$ cd Desktop # change directory
$ pwd
/home/rhertzog/Desktop
$ cd . # the current directory is also known as . (one dot)
$ pwd
/home/rhertzog/Desktop
$ cd .. # the parent directory is always called .. (two dots)
$ pwd
/home/rhertzog
$ ls # listing the contents of a directory
Desktop Downloads Pictures Templates
Documents Music Public Videos
$ cd ~ # the home directory is also known as ~
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
$ mkdir test # create a directory
$ touch test.txt # create an empty file
$ ls
Desktop Downloads Pictures Templates Videos
Documents Music Public test test.txt
$ mv test new # moving and/or renaming files and directories
$ ls
Desktop Downloads new Public Videos
Documents Music Pictures Templates test.txt
$ rmdir new # remove a existing (empty) directory, or use `rm -r new`
$ cp test.txt test2.txt
$ cp -r source_dir dest_dir
$ rm test.txt test2.txt # remove file
$ ls
Desktop Downloads Pictures Templates Videos
Documents Music Public
|
Displaying and Modifying Text Files
1
2
3
4
5
6
7
|
$ cat file # reads a file and displays its contents on the terminal.
$ cat -n file # show number
$ head -10 file # show the first 10 lines
$ tail -5 file # show last 5 lines
$ tail -f crawler.log # dynamic view the file up to data
$ less file # If the file is too big to fit on a screen, display it page by page. It can scroll backwards through a file.
$ more file # If the file is too big to fit on a screen, display it page by page. It can not scroll backwards through a file.
|
Redirection
1
2
|
$ echo "text" >file # create a simple file named file with “text” as its contents.
echo "moretext" >>file # Adding a line at the end of this file
|
Similarly, if a command expects input from the keyboard, you can have it read from a file instead, by using <.
Pipes
Often it is useful to pipe the output of one command as input to another command.
The editor command starts a text editor (such as vi or nano) and allows creating, modifying and reading text files.
Searching for Files and within Files
1
|
$ find directory criteria # looks for files in the hierarchy under directory according to several criteria
|
1
2
|
$ find ./ -name "*log" -exec rm{}; # equal to `rm *log`
$ find ./ | wc - 1 # to checkout the number of file in the directory
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
find ./ \(-name "*.txt" -o -name "*.pdf" \) -print
find . -regex ".*\(\.txt|\.pdf\)$"
-iregex
find . ! -name "*.txt" -print
find . -maxdepth 1 -type f
find . -type d -print
find . -atime -7/+7 type f -print
find . -type f -size +2k
find . -type f -perm 644 -print
find . -type f -user weber -print
find . -type f -name "*.swp" -delete
find . -type f -name "*.swp" | xargs rm
find .type f -user root -exec chown weber { } \;
# {} will be each file matched
find . -type f -mtime +10 -name "*.txt" -exec cp {} Dir \;
-exec ./commands.sh { } \;
|
The most commonly used criterion is -name filename: that allows looking for a file by its name.
1
|
$ grep expression files # searches the contents of the files and extracts the lines matching the regular expression
|
1
2
3
4
5
6
|
grep [OPTIONS] PATTERN [FILE...]
grep [OPTIONS] [-e PATTERN]... [-f FILE]... [FILE...]
-E --extended-regexp
-F --fixed-strings
-G
-v no contain
|
Adding the -r option enables a recursive search on all files contained in the directory passed as a parameter.
Managing Processes
1
|
$ ps aux # lists the processes currently running and helps identifying them by showing their pid (process id)
|
Once the pid of a process is known
1
|
$ kill -signal pid # sending it a signal (if the process belongs to the current user)
|
Several signals exist:
- most commonly used are
TERM (a request to terminate gracefully)
KILL (a forced kill).
1
|
$ some command & # run programs in the background if the command is followed by a &.
|
1
|
$ jobs # list the processes running in the background
|
1
|
$ fg %job-number # restore a job to the foreground
|
When a command is running in the foreground (either because it was started normally, or brought back to the foreground with fg), the Control+Z key combination pauses the process and resumes control of the command-line. The process can then be restarted in the background with bg %job-number (for background).
1
2
3
4
5
|
$ free # displays information on memory
total used free shared buffers cached
Mem: 1028420 1009624 18796 0 47404 391804
-/+ buffers/cache: 570416 458004
Swap: 2771172 404588 2366584
|
The free command supports the -m and -g options, and displays its data either in mebibytes or in gibibytes, respectively.
1
2
3
4
5
6
7
|
$ df # (disk free) reports on the available disk space on each of the disks mounted in the filesystem
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 9614084 4737916 4387796 52% /
tmpfs 514208 0 514208 0% /lib/init/rw
udev 10240 100 10140 1% /dev
tmpfs 514208 269136 245072 53% /dev/shm
/dev/sda5 44552904 36315896 7784380 83% /home
|
The df command supports -h option (for human readable) converts the sizes into a more legible unit (usually mebibytes or gibibytes).
1
|
$ id # displays the identity of the user running the session
|
The id command displays the identity of the user running the session, along with the list of groups they belong to. Since access to some files or devices may be limited to group members, checking available group membership may be useful.
Managing Rights
Each file or directory has specific permissions for three categories of users:
- its owner (symbolized by
u as in “user”);
- its owner group (symbolized by
g as in “group”), representing all the members of the group;
- the others (symbolized by
o as in “other”).
All user (symbolized by a as in “all”).
Three types of rights can be combined:
- reading (symbolized by
r as in “read”);
- writing (or modifying, symbolized by
w as in “write”);
- executing (symbolized by
x as in “eXecute”).
Three commands control the permissions associated with a file:
chown user file changes the owner of the file;chgrp group file alters the owner group;- `chmod rights file changes the permissions for the file.
There are two ways of presenting rights:
-
the symbolic representation is probably the easiest to understand and remember. It involves the letter symbols mentioned above. You can define rights for each category of users (u/g/o), by setting them explicitly (with =), by adding (+), or subtracting (-). Thus the u=rwx,g+rw,o-r formula gives the owner read, write, and execute rights, adds read and write rights for the owner group, and removes read rights for other users.
1
|
chmod a+x script.sh (add execute attribute of file for all users)
|
-
The (octal) numeric representation associates each right with a value: 4 for read, 2 for write, and 1 for execute. We associate each combination of rights with the sum of the figures. Each value is then assigned to different categories of users by putting them end to end in the usual order (owner, group, others).
The 0 means no rights; thus chmod 600 file allows for read/write rights for the owner, and no rights for anyone else.
The most frequent right combinations are 755 for executable files and directories, and 644 for data files.
To represent special rights, you can prefix a fourth digit to this number according to the same principle, where the setuid, setgid and sticky bits are 4, 2 and 1, respectively. chmod 4754 will associate the setuid bit with the previously described rights.
Recursive operation
All the commands above have a -R option to operate recursively in sub-directories.
the “X” letter represents a right to execute which applies only to directories (and not to files lacking this right).
Thus, chmod -R a+X *directory* will only add execute rights for all categories of users (a) for all of the sub-directories and files for which at least one category of user (even if their sole owner) already has execute rights.
Changing the user and group
change the group of a file at the same time that you change the owner.
1
|
$ chown user:group file
|
Symbolic links
A symbolic link is a pointer to another file. When you access it, the file to which it points is opened. Removal of the link will not cause deletion of the file to which it points. Likewise, it does not have its own set of permissions, but rather retains the permissions of its target. Finally, it can point to any type of file: directories, special files (sockets, named pipes, device files, etc.), even other symbolic links.
1
|
$ ln -s target link-name # creates a symbolic link, named link-name, pointing to target.
|
sed
Online Help
1
2
3
|
$ which command # to see the install path of command
$ whereis command # to see the search path of command
$ man command # manual
|
Linux基础
学会使用命令帮助
1
2
3
4
5
6
|
man -k segment_command
whatis command
info command
man command
which command (to see the install path of command)
whereis command (to see the search path of command)
|
文件及目录管理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
mkdir
rm
rm -rf directory/
rm *log == $find ./ -name "*log" -exec rm{};
rmdir
mv
cp -rf
$find ./ | wc - 1 (to checkout the number of file in the directory)
cp -r source_dir dest_dir
cd
cd -
cd ~
pwd
ls -lrt
alias alias_name=command
>ls | cat -n (add index before all file name)
|
查找目录及文件 find/locate
1
2
3
4
|
find ./ -name "core*" | xargs file
find ./ -name "*.o" -exec rm {} \; （delete .o files recursively)
locate string (build index datebase to search file， find is timed, locate is not)
updatedb
|
查看文件内容
1
2
3
4
5
6
7
|
cat -n 显示行号
ls -al | more
head - 10 **
head -1 filename
tail -5 filenaem
diff file1 file2
tail -f crawler.log (dynamic view the file up to data)
|
查找文件内容
1
2
3
4
5
|
grep [OPTIONS] PATTERN [FILE...]
grep [OPTIONS] [-e PATTERN]... [-f FILE]... [FILE...]
-E --extended-regexp
-F --fixed-strings
-G
|
文件与目录权限修改
文件类型：[d]、[-]、[l]、[b]、[c]、[s]、[p]、
权限：文件类型、所有者三位、用户组三位、其他人三位
权限分数对 r:4，w:2，x:1， 相加有 777，644等等
1
2
3
4
5
6
|
chgrp group file 改变用户组 users必须在/etc/group内
chown user file (change the owner of file) use必须在/etc/passwd内
chown -R file directory/ 递归子目录进行修改
chmod 777 file (change the read write exect attributes of file)
chmod a+x script (add execute attribute of file) (user, group, other, all; + - =; r w x)
su - user 切换身份
|
2.8. 给文件增加别名
1
2
|
ln cc ccAgain 硬链接
ln -s cc ccTo 软/符号连接
|
管道和重定向
1
2
3
4
5
6
7
|
| 连续执行
; 串联
&& 前面成功则执行后面一条，否则不执行
|| 前面失败，则后面一条执行
if condition; then expression; else expression; fi
\>重定向
:> file 清空文件
|
设置环境变量
启用账号后自动执行的./profile文件，在里面设置环境变量
安装软件后添加环境变量
1
|
PATH=$APPDIR:/opt/app/soft/bin:$PATH:/usr/local/bin:$TUXDIR/bin:$ORACLE_HOME/bin;export PATH
|
Bash快捷输入或删除
1
2
3
4
|
Ctl-u 删除光标到行首的所有字符
Ctl-w 删除光标前边的最近一个空格之间的字符
Ctl-H 删除光标前边的一个字符
Ctl-R 匹配最相近的一个文件，然后输出
|
综合应用
1
2
|
cat -v record.log | grep AAA | grep -v BBB | wc -l
查找log中包含AAA但不包含BB的记录的总数
|
文本处理
find 文件查找
1
2
3
4
5
6
7
8
9
10
|
find . \( -name "*.txt" -o -name "*.pdf" \) -print
find . -regex ".*\(\.txt|\.pdf\)$"
-iregex
find . ! -name "*.txt" -print
find . -maxdepth 1 -type f
find . -type d -print
find . -atime -7/+7 type f -print
find . -type f -size +2k
find . -type f -perm 644 -print
find . -type f -user weber -print
|
后续动作：
1
2
3
|
find . -type f -name "*.swp" -delete
find . -type f -name "*.swp" | xargs rm
find .type f -user root -exec chown weber { } \;
|
对于每个匹配的文件，{}会被替换为相应的文件名
1
2
3
|
find . -type f -mtime +10 -name "*.txt" -exec cp {] Dir \;
-exec ./commands.sh { } \;
-print的定界符，默认使用'\n'，-printO使用‘\O'作为定界符
|
grep 文本搜索
1
2
3
4
|
grep math_pattern file
grep "string" . -R -n
grep -e "string1" -e "string2" file
********** man grep **************
|
xargs 命令行参数转换
1
2
3
4
5
|
cat test.txt | xargs 多行输入单行输出
cat test.txt | xargs -n3 -n 多行输出，3为一行字段
| xargs -dX -d自定义一个定界符
| xargs - I { } ./script.sh { } - I 指定一个替换字符{}
find . -type f -name "*.log" -print0 | xargs -0 rm -f print和xargs定界符'/0'
|
sort 排序
-n 按数字进行排序 VS -d 按字典序进行排序
-r 逆序排序
-k N 指定按第N列排序
uniq 消除重复行
sort unsort.txt | uniq -c 统计每行出现次数
sort unsort.txt | uniq -d 找出重复行
可指定每行中需要比较的重复内容：-s 开始位置 -w 比较字符数
用tr进行转换
1
|
tr [option] set1 [set2]
|
-c取代所有不属于第一字符集的字符；
-d删除所有属于第一字符集的字符；
-s把连续重复的字符以单独一个字符表示；
-t先删除第一字符集较第二字符集多出的字符。
cut 按列切分文本
1
2
3
4
|
cut -f2,4 filename 2，4字段
cut -f3 --complement filename 除了第3字段的所有字段
cut -b 字节
cut -c 字符
|
paste 按列拼接文本
1
|
paste file1 file2 [-d ","] 按列拼接，以，分隔
|
wc 统计行和字符的工具
1
2
3
|
wc -l file 行
wc -w file 字
wc -c file 字符
|
sed 文本替换利器
1
2
3
4
5
6
7
|
sed 's/text/replace_text/' file //替换每一行的第一处匹配的text
sed 's/text/replace_text/g' file // 全局替换
sed -i 's/text/repalce_text/g' file // 替换后不显示替换的内容
sed '/^$/d' file // 移除空白行
sed 's/hello\([0-9]\)/\1/' // 第一个匹配的括号内容使用标记 1 来引用
sed通常用单引号来引用；也可使用双引号，使用双引号后，双引号会对表达式求值
******** man sed *********
|
awk 数据流处理工具
迭代文件中的行、单词和字符
磁盘管理
查看磁盘空间
打包/ 压缩
解包/解压缩
进程管理工具
查询进程
终止进程
进程监控
分析线程栈
综合运用
性能监控
监控CPU
查询内存
查询页面交换
查询硬盘使用
综合应用
网络工具
查询网络服务和端口
网络路由
镜像下载
ftp sftp lftp ssh
网络复制
用户管理工具
用户
用户的组
用户权限
环境变量
系统管理及IPC资源管理
系统管理
IPC资源管理
Linux工具进阶
程序构建
程序调试
性能优化
工具参考篇
gdb 调试利器
ldd 查看程序依赖库
lsof 一切皆文件
ps 进程查看器
pstack 跟踪进程栈
strace 跟踪进程中的系统调用
ipcs 查询进程间通信状态
top linux下的任务管理器
free 查询可用内存
vmstat 监视内存使用情况
iostat 监视I/O子系统
sar 找出系统瓶颈的利器
readelf elf文件格式分析
objdump 二进制文件分析
nm 目标文件格式分析
size 查看程序内存映像大小
wget 文件下载
scp 跨机远程拷贝
crontab 定时任务
crontab
在ubuntu下安装crontab后，系统默认的是不开启crontab的日志记录，如何开始crontab的日志：
修改rsyslog文件，将/etc/rsyslog.d/50-default.conf 文件中的#cron.*前的#删掉；
重启rsyslog服务
1
|
service rsyslog restart
|
重启cron服务
1
2
3
4
5
|
sudo apt-get install sendmail
sudo apt-get install sendmail-cf
sudo apt-get install mailutils
sudo apt-get install sharutils
ps aux |grep sendmail
|
1
2
3
|
\# vi /etc/mail/sendmail.mc
dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
|
将上面两行的dnl去掉。在sendmail文件中，dnl表示该行为注释行，是无效的，因此通过去除行首的dnl字符串可以开启相应的设置行
sendmail 默认只会为本机用户发送邮件，只有把它扩展到整个Internet，才会成为真正的邮件服务器。
打开sendmail的配置宏文件：/etc/mail/sendmail.mc
1
|
vim /etc/mail/sendmail.mc
|
1
|
DAEMON_OPTIONS(`Family=inet, Name=MTA-v4, Port=smtp, Addr=127.0.0.1')dnl
|
把127.0.0.1修改为0.0.0.0 ，表明可以连接到任何服务器。
生成新的配置文件：
1
2
3
|
cd /etc/mail
mv sendmail.cf sendmail.cf-bak //做一个备份
m4 sendmail.mc > sendmail.cf // ">"的左右有空格。若提示错误说明你没有安装sendmail-cf 软件包
|
1
2
3
4
5
|
/var/log/cron.log
/var/mail/$user
crontab -e
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
|
References
- https://debian-handbook.info/browse/stable/
- https://linuxtools-rst.readthedocs.io/zh_CN/latest/