cURL & Wget — Complete Guide

A comprehensive practical guide covering command-line data transfer tools: cURL (Client URL) and Wget (World Wide Web Get).

1. Overview & Comparison

Both cURL and Wget are command-line utilities used for downloading data and interacting with servers over network protocols (HTTP, HTTPS, FTP, etc.).

Key Differences

2. cURL (curl) In-Depth

curl is designed to transfer data to or from a server using one of the supported protocols.

Common Flags & Options

HTTP Request Examples

1. Basic GET Request

curl https://api.example.com/users

2. Save Output to File

# Save to specific filename
curl -o user_data.json https://api.example.com/users

# Save using remote file's original name
curl -O https://example.com/downloads/archive.tar.gz

3. POST Request with JSON Data

curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "John Doe", "email": "john@example.com"}'

4. POST Request with Data from File

curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d @payload.json

5. PUT Request (Update Resource)

curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name": "Jane Doe", "email": "jane@example.com"}'

6. PATCH Request (Partial Update)

curl -X PATCH https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"email": "newemail@example.com"}'

7. DELETE Request

curl -X DELETE https://api.example.com/users/123

8. Follow Redirects (301 / 302)

curl -L http://example.com

Authentication & Headers

1. Bearer Token Authentication

curl -H "Authorization: Bearer YOUR_API_TOKEN" https://api.example.com/protected

2. Basic Authentication

curl -u username:password https://api.example.com/dashboard

3. Custom Headers

curl -H "X-Custom-Header: CustomValue" \
  -H "Accept-Language: en-US" \
  https://api.example.com/data

4. API Key Header

curl -H "x-api-key: secret-api-key-12345" https://api.example.com/v1/metrics

File Uploads & Forms

1. Multipart Form Upload (multipart/form-data)

curl -F "file=@/path/to/photo.jpg" \
  -F "description=Profile Photo" \
  https://api.example.com/upload

2. Standard Form URL-Encoded (application/x-www-form-urlencoded)

curl -d "username=admin" \
  -d "password=secret" \
  https://example.com/login

Cookies & Sessions

1. Save Cookies to a File

curl -c cookies.txt -d "user=admin&pass=123" https://example.com/login

2. Send Cookies with Request

curl -b cookies.txt https://example.com/dashboard

3. Inline Cookie String

curl -b "session_id=abc123xyz; dark_mode=true" https://example.com/preferences

Debugging, Metrics & Performance

1. Inspect Response Headers Only

curl -I https://example.com

2. Include Headers with Response Body

curl -i https://api.example.com/users

3. Verbose Network Tracing

curl -v https://api.example.com/users

4. Formatted Timing Metrics (-w)

curl -s -w "\
HTTP Code:       %{http_code}\n\
DNS Lookup:      %{time_namelookup}s\n\
TCP Connect:     %{time_connect}s\n\
TLS Handshake:   %{time_appconnect}s\n\
Pre-transfer:    %{time_pretransfer}s\n\
Start Transfer:  %{time_starttransfer}s\n\
Total Time:      %{time_total}s\n" \
-o /dev/null https://google.com

5. Rate Limiting & Resuming Downloads

# Limit download speed to 500 KB/s
curl --limit-rate 500k -O https://example.com/largefile.zip

# Resume interrupted download
curl -C - -O https://example.com/largefile.zip

3. Wget (wget) In-Depth

wget is a free, non-interactive network downloader that supports HTTP, HTTPS, and FTP protocols.

Common Flags & Options

Downloading Files

1. Download a Single File

wget https://example.com/files/document.pdf

2. Download and Save with Custom Name

wget -O custom_name.pdf https://example.com/files/document.pdf

3. Save into Specific Folder

wget -P /path/to/downloads https://example.com/files/document.pdf

4. Resume an Interrupted Download

wget -c https://example.com/files/large_archive.iso

5. Download in the Background

wget -b https://example.com/files/large_archive.iso

# View download progress in log
tail -f wget-log

6. Download Multiple Files from a URL List

wget -i urls.txt

Website Crawling & Mirroring

1. Mirror an Entire Website for Offline Viewing

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://example.com/

• --mirror (-m): Enables recursion, timestamps, and infinite depth.
• --convert-links (-k): Converts links to point to local files.
• --adjust-extension (-E): Appends .html to files if needed.
• --page-requisites (-p): Downloads all CSS, JS, and image assets.
• --no-parent (-np): Restricts downloads to the current subpath.

2. Download Specific File Types (e.g., all PDFs)

wget -r -A.pdf -np https://example.com/documents/

3. Reject Specific File Types

wget -r -R.zip,.tar.gz https://example.com/files/

Authentication & Rate Limiting

1. Basic HTTP Authentication

wget --http-user="myuser" --http-password="mypassword" https://example.com/secure/file.zip

2. Custom Request Header

wget --header="Authorization: Bearer YOUR_TOKEN" https://example.com/api/export.csv

3. Limit Download Speed & Retries

wget --limit-rate=1m --tries=10 --timeout=30 https://example.com/video.mp4

4. Side-by-Side Command Cheatsheet

5. Practical Automation Recipes

1. API Health Check Script (curl)

#!/usr/bin/env bash

ENDPOINT="https://api.example.com/health"
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$ENDPOINT")

if [ "$HTTP_STATUS" -eq 200 ]; then
  echo "✅ Endpoint is healthy (HTTP $HTTP_STATUS)"
else
  echo "❌ Endpoint failed with status HTTP $HTTP_STATUS"
  exit 1
fi

2. Formatted JSON API Query with jq (curl)

curl -s -H "Accept: application/json" https://jsonplaceholder.typicode.com/todos/1 | jq .

3. Bulk Document Downloader (wget)

#!/usr/bin/env bash

# Downloads all PDFs from a target directory with retry and rate limit
TARGET_URL="https://example.com/whitepapers/"

wget -r -l 2 -np -nc \
  -A "*.pdf" \
  --limit-rate=2m \
  --tries=5 \
  --wait=1 \
  -P ./whitepapers/ \
  "$TARGET_URL"

6. Summary: When to Choose Which?