I n this article, I explore commands that every system administrator should know for troubleshooting, general housekeeping, and daily activities that you must perform.
When you practice commands that can be harmful to a production system, have a virtual machine running somewhere that you can torture and restore should something go wrong. For some reason, people generally frown on having to repair or reinstall production systems because someone practiced a new command that went awry. Plus, it’s cool to show up one day armed with new sysadmin skills to impress (school) your coworkers. Remember to say, “Watch this,” to be sure they’re paying attention before you hit the Enter key so it’s more dramatic and awe-inspiring.
NOTE: You don’t have to be the root user to run any of these commands. To change system parameters or to edit system files, though, you will have to be root.
Show whois logged in
As a system administrator, it’s your job to keep track of who logs into your systems, either through automation or when you’re in the system yourself. A quick check can tell you a lot about what’s going on at that point. For example, if you have a system whose performance is “in the red” and you’re not sure why issue the whocommand to find out who is logged in. If you see a developer or group of developers, they might be testing a new application that is grabbing all the resources. Or you might have the occasional rogue user running a poorly constructed Nmap command.
The whocommand tells you who is logged in, when they logged in, where they’re logged in from, and even which type of connection they’re using:
$ who
root tty1 2019-07-23 07:58
khess pts/0 2019-07-23 07:59 (192.168.1.81)
The ttyXlogins are from the console and the pts/Xones are over the network from a computer via SSH. An acronym for Pseudo Terminal Slave, most sysadmins refer to the ptsentries as pseudoterminals. The important thing is to note the difference between TTY (local console) and PTS (remote SSH) logins.
Another reason to run whois if you’re about to perform system maintenance. A quick check will tell you who you have to contact to advise them to log out of the system because your maintenance might include a reboot or other activity that will disrupt their work.
echoa line of text
Believe it or not, echois one of the most powerful commands at your disposal. With this command, you can do things like create files, append to them, check return codes, and view system variables.
To create a new file this command, use echowith some text, and then redirect the output to the file you want to create:
$ echo "This is a test file" > test.txt
You don’t have to use quotes around the text, but I always do—I worry that the text I redirect to the file won’t look right if I don’t. To be sure it’s correct, cat the file:
$ cat test.txt
This is a test file
To append some text on the next line, use the append redirect operator ( >>):
$ echo "This is how to add text to a file" >> test.txt
$ cat test.txt
This is a test file
This is how to add text to a file
Check the return code from the last command you ran with echo:
$ echo $? 0
A 0response typically means success. You can also use echoto check your environment variables:
$ echo $SHELL
/bin/bash
The echoman page gives you many more options and capabilities, such as how to use tabs, backspace, carriage returns, and more.
Display the topLinux processes
The topcommand does much more than simply display Linux processes, but it’s a start. Run topat the command line to observe for yourself all the information that this command provides:
top - 10:14:04 up 5 days, 48 min, 2 users, load average: 0.00, 0.00, 0.02
Tasks: 233 total, 1 running, 232 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.9 us, 5.9 sy, 0.0 ni, 88.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 1829.4 total, 191.2 free, 1066.0 used, 572.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 538.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1651 khess 20 0 64016 4936 4056 R 11.8 0.3 0:00.02 top
1 root 20 0 179492 12076 6804 S 0.0 0.6 0:40.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.17 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
The listing above shows the first few lines from my Red Hat 8.0 server’s topdisplay. This command runs continuously, so to exit, type q. This command is named topin the first place because it provides a real-time display of the top processes by CPU and memory usage. To see an exhaustive explanation of the topcommand, refer to the man page.
Other than q, the most beneficial key command for me is k, which prompts a process ID (PID) to kill (terminate). As a system administrator, it is your job to protect system health for the general user population of that system. In other words, killing processes to release resources is one of the things you do, within reason, of course. It’s career-limiting to kill processes in a haphazard fashion, but it’s something that has to be done from time to time—killing processes—not killing them haphazardly.
The topcommand gives you a real-time snapshot of system performance. Typically, you run topwhen a performance problem is reported. When a system is idle, running topisn’t exciting and often results in showing topas the most resource-consuming process on the system. Don’t be alarmed by this, but do realize that it’s possible.
Use topas much as you like, but realize that its information is not necessarily indicative of overall system performance. It is a snapshot and not a measure of long-term activity.
killa process
Although I wrote in the section above that it’s your job to sometimes kill processes, exercise caution when doing so. There’s a good chance that abruptly ending a process will cause data corruption, data loss, and even job loss for you if you haven’t cleared such actions through the proper channels.
The two most often used signals or options for the killcommand are -15and -9. Issuing a kill -15 <PID>is known as a soft, or polite kill. The -15(also known as SIGTERM) signal kills the process but allows it to finish any pending processing:
$ kill -15 <PID>
The -9signal ( SIGKILL) immediately terminates the program with no regard for current processing. The -9signal kills it. End of story. End of process:
$ kill -9 <PID>
There are two specific times to use the -9signal. The first is when you have a runaway process that can’t be killed with the -15signal, and the second is when you need to free system resources immediately without regard for data loss or corruption. This second scenario is rare, but it does happen. In that situation, the only other option might be to reboot the system. Even after killing the process, you might have to reboot anyway—killing certain processes can leave the system in an unstable state.
The takeaway here is to use killsparingly and only with permission.
Closely associated with the killcommand is the killallcommand. If you have a process such as the Chrome web browser that can consume more than its share of resources, you can issue the killall <processname>command to rid the system of all its spawned processes. The killallcommand doesn’t require you to know the PID, nor do you have to kill each individual process. Doing so can become way too tedious, and system administrators haven’t the patience for such things.
$ killall chrome
This command terminates all instances of Chrome owned by this user. You can issue the same command as root, but read the previous dialog about exercising caution when doing so, because issuing such a command as root terminates the program for everyone on the system.
Note: If your system doesn’t have the
killallcommand available, then you’ll have to add it by installing thepsmiscpackage as shown below.
$ sudo yum -y install psmisc
I know, we haven’t discussed the yumor dnfcommands yet. Take this one as a “just do it” lesson at this point.
View files moreor less
If you’ve used commands such as ps, you know that file listings can be long, and a lot of the information flows right off the screen. Sure, you can page up or scroll, but it’s not very efficient.
The commands moreand lesslimit the amount of data you see to one “page.” As with many things Linux-related, users are in two camps: the morecamp and the lesscamp. I’m in the morecamp. I never use less. And, no, lessisn’t more. Even the lessman page reads, “The opposite of more.”
From a usage standpoint, these two commands are similar. However, the differences surface when interacting with these commands. It’s impossible to show effectively in a static article but lesshas a few more navigation options than more. The morecommand’s options are:
- Advance one line using the Enter key.
- Advance a full page using the Spacebar .
- Quit by entering
q.
You cannot move backward using more. Less, being more Wonkavator-esque , allows you to move backward, search for strings, and much more. Use the man pages for moreand lessto decide which of these commands is right for you.
To make things even more complex, there are two ways to use moreand less. You can pipe (|) output to moreand lessor you can use these commands to operate directly on files. Here are some examples (without their output):
$ more /etc/passwd
$ cat /etc/passwd | more
$ ps -ef | more
$ less /etc/passwd
$ cat /etc/passwd | less
$ ps -ef | less
Update user passwdauthentication tokens
Standard users use the passwdcommand to change their passwords. It’s quick and simple to do. Issue the passwdcommand and you’re prompted to change your password:
$ passwd
Changing password for user khess.
Current password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
When changing your password, you’ll notice that the system does not respond with any dots, stars, or even blank spaces. This feature is far more secure in situations where someone is shoulder surfing during a password change. There is also no option for showing the password. Again, very secure.
There are additional passwdcommand options for the root user. For example, if you issue the following command as yourself, check your system’s response:
$ passwd -S
Only root can do that.
If the root user issues this command with a username, the command displays user information:
$ sudo passwd -S khess
khess PS 2019-07-29 0 99999 7 -1 (Password set, SHA512 crypt.)
The real power for system administrators is being able to set a user’s password without knowing the current one:
$ sudo passwd khess
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
As root, you can optionally lock and unlock user accounts:
$ sudo passwd -l john
Locking password for user john.
passwd: Success
$ sudo passwd -u john
Unlocking password for user john.
passwd: Success
Use passwdresponsibly. And when offboarding a user, you should lock the account rather than deleting it: The user might have important data saved in their home directory, or have a process running that requires the account to be functional. Locking is good enough to prevent further interactive logins, and will also inform you about any automated tasks that require a password to perform.
ifconfiga network interface
There are tasks that as a sysadmin you don’t do every day, but when you do them you need a power command like ifconfig. I classify this command in the power category because it does many things, but with simple syntax.
Note: While a user can look at network interface configurations and settings with, you must be root to make changes.
$ ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.96 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 2600:1702:a40:88b0:581f:ea48:4e1a:6711 prefixlen 64 scopeid 0x0<global>
inet6 fe80::3d1d:ee56:9c1c:33b prefixlen 64 scopeid 0x20<link>
ether 08:00:27:a7:47:25 txqueuelen 1000 (Ethernet)
RX packets 1153803 bytes 230635486 (219.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 78485 bytes 8389458 (8.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 48 bytes 5616 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 48 bytes 5616 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:7a:a9:b2 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
You can also use ifconfigto assign IP addresses to interfaces, change an interface’s IP addresses, take an interface offline, bring one online, and more.
grepa pattern
Use the greputility to search for a particular pattern inside a file or group of files. For example, say that you have a file in your home directory that contains the IP address of a remote system that you worked on a few months ago, but you can’t recall the exact address. You know it was something like 192.168.10.???. The problem is that you have 50 files in your home directory and it would take hours to search through them all by hand.
Well, fret no more, grepis here to help. In this example, you can grepfor the 192.168.10.pattern in your home directory:
$ grep 192.168.10. *
grep: data: Is a directory
grep: docs: Is a directory
grep: documents: Is a directory
grep: form: Is a directory
grep: forms: Is a directory
Notice that several of the entries state that you’re attempting to look into files that are directories, and that your search came up negative for a file containing the IP address. Use the recursive option ( -R) to search subdirectories:
$ grep -R 192.168.10. *
documents/systems_list.txt: 192.168.10.45 pumba
Here, your search was successful. The grepcommand returned the entire line that matches your pattern.
When system administrators mention grepor “grepping” something, they usually refer to piping in the same sentence, as in “Pipe it to grep.” You don’t always need to pipe to grepas you can see from the example above. But, piping to grepworks in a similar way. To search for systemdin your process list:
$ ps -ef |grep systemd
root 1 0 0 Aug07 ? 00:00:40 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
root 476 1 0 Aug07 ? 00:00:11 /usr/lib/systemd/systemd-journald
root 505 1 0 Aug07 ? 00:00:02 /usr/lib/systemd/systemd-udevd
root 632 1 0 Aug07 ? 00:00:02 /usr/lib/systemd/systemd-machined
dbus 653 1 0 Aug07 ? 00:01:45 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 712 1 0 Aug07 ? 00:00:07 /usr/lib/systemd/systemd-logind
gdm 1209 1 0 Aug07 ? 00:00:02 /usr/lib/systemd/systemd --user
gdm 1301 1209 0 Aug07 ? 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
khess 2423 29513 0 10:25 pts/1 00:00:00 grep --color=auto systemd
khess 8088 1 0 Aug07 ? 00:00:03 /usr/lib/systemd/systemd --user
khess 8113 8088 0 Aug07 ? 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
As you can see, piping to grepis the only way you can find all instances of systemdfrom the process list. Note the first entry with my username on it. That is my grepcommand searching for systemd. If you don’t want to see that entry, use the -v option to exclude the grepcommand itself from your results:
$ ps -ef | grep systemd | grep -v grep
The other grepoption that I find helpful is the ignore case option( -i):
$ grep -iR bob *
This command searches recursively through all files for the string bob, regardless of case, which could match all of the following: Bob, Spongebob, bilbobaggins, and BObrice.
Grep is very useful and can be used on text files, and in conjunction with other commands via piping. You can also grepfor complex patterns using regular expressions (regex) but that is a topic for other articles.
Scan and process patterns with awk
I feel like awkis one of those tools that few people use because they don’t understand the full power and possibilities of this little dynamo. I will jump right in with some examples. Say that I want a list of all processes that are systemd-related, but I only want the PIDs, not all of the other information that you get with ps:
$ ps -ef | grep systemd | grep -v grep | awk '{print $2}'
1
471
608
631
7449
7494
32681
To explain the command above: I ran a ps, grepped for systemd, removed my own grepcommand, and then piped the output to awkand printed the second column. It is the second column because by default awkuses a space as a field separator. The formal awkpart of the command would look like this: awk -F " " '{print $2}', where the -Foption defines the field separator. For comma-separated values, you’d use: awk -F "," '{print $2}'.
If you have a text file ( test.txt) containing the following:
one,two,three,four,five
1,2,3,4,5
6,7,8,9,10
a,b,c,d,e
And you run awkagainst that file to extract the third column of data, it displays the following:
$ cat test.txt | awk -F "," '{print $3}'
three
3
8
c
I think you can see what’s going on with awkhere. It’s handy for automation scripting as you can probably tell from these examples. You can extract data and operate on it dynamically with awk.
Edit text with vi
The vi(visual) text editor was a clever developer’s (Bill Joy) answer to updating the old line editor ex, which Bill Joy also wrote. This program 40+ years later is still the most used Linux command line text editor.
The vieditor is small, with the latest incarnation ( vimaka viimproved) weighing at just over 3MB in size. These days viis often a symbolic link to vim(in RHEL 8, for example). Its enhancements include multi-level undo, multiple windows and buffers, syntax highlighting, command line editing, file name completion, online help, and visual selection. Open vimand use the following command for a summary of the differences between vimand vi:
:help vi_diff.txt
vi has so many options and features that I’m only mentioning it as one of the commands you need to know in this article. Please refer to my vi : An introduction article for a more extensive look at vi.
Wrapping up
Surprisingly, out of more than 200 possible Linux commands, most system administrators only use about two dozen on a regular basis. If you know those, system administration becomes easier and far more elegant. Struggling with commands and syntax makes the job harder. Learn these popular and highly-used commands and you’ll have the power to make a difference in your environment.
