Run Shell Commands in Python and Get Output

Python is a powerful language for automation. Often, you need to run system commands. You can capture their output for further processing. This is a common task for developers and system administrators.

This guide explains the best methods. We will focus on the subprocess module. It is the modern and recommended way. We will cover key functions and best practices.

Why Run Shell Commands from Python?

Python scripts can automate repetitive tasks. Sometimes these tasks involve external programs. You might need to list files, check system status, or process data.

Running shell commands bridges Python and the operating system. It lets you leverage existing tools. You can then use Python's logic to handle the results.

Introducing the Subprocess Module

The subprocess module is the standard solution. It replaces older modules like os.system. It provides more control and security.

You can run commands, connect to their input/output streams, and get return codes. It is designed to be safe and flexible.

The subprocess.run() Function

The subprocess.run() function is the primary tool. It runs a command and waits for it to finish. It returns a CompletedProcess object.

This object contains the output and the return code. It is simple and works for most cases.

import subprocess

# Run a simple command
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)

# Print the standard output
print("STDOUT:", result.stdout)
# Print the return code (0 means success)
print("Return Code:", result.returncode)
    

STDOUT: total 0
-rw-r--r--  1 user  staff  0 Jan  1 10:00 file1.txt
-rw-r--r--  1 user  staff  0 Jan  1 10:00 file2.py
Return Code: 0
    

Key arguments for run() are capture_output=True and text=True. The first captures output. The second returns strings instead of bytes.

Handling Errors and Return Codes

Commands can fail. You must check the return code. A non-zero code usually indicates an error.

You can use check=True to raise an exception automatically. This makes error handling easier.

import subprocess

try:
    # This command will fail because 'nonexistentcommand' doesn't exist
    result = subprocess.run(['nonexistentcommand'], capture_output=True, text=True, check=True)
except subprocess.CalledProcessError as e:
    print(f"Command failed with code {e.returncode}")
    print(f"Error output: {e.stderr}")
    

Command failed with code 127
Error output: /bin/sh: nonexistentcommand: command not found
    

Capturing Output Separately: stdout and stderr

Programs have two main output streams. Standard output (stdout) is for normal results. Standard error (stderr) is for errors and diagnostics.

You can capture them separately. This helps in debugging and logging.

import subprocess

result = subprocess.run(
    ['ls', 'real_file.txt', 'ghost_file.txt'],
    capture_output=True,
    text=True
)

print("STDOUT:", result.stdout)
print("STDERR:", result.stderr)
print("Return Code:", result.returncode)
    

STDOUT: real_file.txt

STDERR: ls: ghost_file.txt: No such file or directory
Return Code: 1
    

Using Shell=True: A Word of Caution

The run() function can execute commands through the system shell. You use shell=True for this. It allows using shell features like pipes and wildcards.

This is a security risk if the command string comes from user input. It can lead to shell injection attacks. Avoid it when possible.

If you must use it, never pass unsanitized user input.

import subprocess

# Using shell=True to use a pipe
result = subprocess.run('echo "Hello" | tr "e" "a"', shell=True, capture_output=True, text=True)
print(result.stdout)  # Output: Hallo
    

Older Methods: os.system and os.popen

Before subprocess, Python used os.system and os.popen. They are still in the os module but are not recommended for new code.

os.system() runs a command but gives limited control. os.popen() can capture output but is deprecated. Use subprocess instead.

Practical Example: Checking Disk Usage

Let's build a useful script. It checks disk usage and alerts if it's too high. This shows a real-world application.

import subprocess

def check_disk_usage(threshold=80):
    """Check disk usage and alert if above threshold percent."""
    # Run the df command to get disk info
    result = subprocess.run(
        ['df', '-h', '/'],
        capture_output=True,
        text=True,
        check=True
    )
    
    # Parse the output (second line, fifth column)
    lines = result.stdout.strip().split('\n')
    # Get the line for the root filesystem
    data_line = lines[1]
    # Split by whitespace and get the usage percentage (without %)
    usage_percent = int(data_line.split()[4].replace('%', ''))
    
    print(f"Current disk usage: {usage_percent}%")
    
    if usage_percent > threshold:
        print(f"WARNING: Disk usage is above {threshold}%!")
    else:
        print("Disk usage is acceptable.")
    
    return usage_percent

# Check with a threshold of 85%
check_disk_usage(85)
    

Current disk usage: 75%
Disk usage is acceptable.
    

Best Practices and Security

Follow these guidelines for safe and effective command execution.

• Prefer subprocess.run(): It is the most versatile and secure function.
• Avoid shell=True: Use it only when necessary for shell features. Never with untrusted input.
• Use full command paths: For critical scripts, use absolute paths (e.g., /bin/ls) to avoid PATH issues.
• Handle errors: Always check return codes or use check=True.
• Decode bytes to text: Use text=True or decode output manually for string processing.

Conclusion

Running shell commands from Python is straightforward with the subprocess module. The subprocess.run() function is your main tool.

Remember to capture output, handle errors, and prioritize security. This skill is key for writing powerful automation and system administration scripts.

Start by integrating simple commands into your Python workflows. You will soon automate complex tasks with ease.