Consuming Web API JSON Data Using curl and jq: Complete Guide 2026
Whether you are testing endpoints, building shell scripts, or debugging production APIs, the combination of curl and jq is the most powerful two-tool stack in a developer's terminal. This guide takes you from zero to confidently querying real APIs, parsing nested JSON, transforming data pipelines, and automating workflows — entirely from the command line.
1B+
curl downloads per year
~50ms
typical jq parse time on 1MB JSON
200+
built-in jq filter functions
0
runtime dependencies for jq
What Are curl and jq?
curl (Client URL) is a command-line tool and library for transferring data with URLs. It supports HTTP, HTTPS, FTP, and dozens of other protocols. Developers use it daily to send GET, POST, PUT, and DELETE requests to REST APIs.
jq is a lightweight, portable command-line JSON processor written in C. It reads JSON from stdin or a file, applies a filter expression, and outputs transformed JSON or plain text. Think of it as sed and awk but designed specifically for JSON.
curl strengths
HTTP requests, custom headers, authentication, file uploads, cookie handling, redirect following, TLS certificates.
jq strengths
Pretty-printing, field extraction, array iteration, conditional logic, grouping, sorting, math, and string interpolation.
Why pipe them together
curl delivers raw JSON; jq shapes it. The Unix pipe model means zero temporary files and instant feedback.
Where they run
macOS, Linux, Windows (WSL / Git Bash), CI/CD pipelines, Docker containers — anywhere bash runs.
Installation
Install curl
curl ships with macOS and most Linux distros. On Ubuntu/Debian run: sudo apt install curl. On macOS with Homebrew: brew install curl. On Windows install WSL2 or Git for Windows.
# Ubuntu / Debian
sudo apt update && sudo apt install -y curl
# macOS
brew install curl
# Verify
curl --versionInstall jq
jq is a single static binary — no dependencies. Download from jqlang.github.io/jq or use a package manager.
# Ubuntu / Debian
sudo apt install -y jq
# macOS
brew install jq
# Alpine (Docker)
apk add jq
# Verify
jq --versionTest the pipeline
Run a quick sanity check against a public API.
curl -s "https://jsonplaceholder.typicode.com/todos/1" | jq
# Expected pretty-printed output:
# {
# "userId": 1,
# "id": 1,
# "title": "delectus aut autem",
# "completed": false
# }Essential curl Flags for API Work
Raw curl https://api.example.com works, but the following flags turn it into a proper API client.
| Item | Flag | What it does |
|---|---|---|
| -s | -s | Silent mode — suppresses progress bar. Essential when piping to jq. |
| -S | -sS | Show errors even in silent mode. Combine with -s as -sS. |
| -X | -X POST | Set HTTP method. Default is GET. |
| -H | -H "Accept: application/json" | Add a request header. Repeat for multiple headers. |
| -d | -d '{"key":"val"}' | Send request body (implies POST). |
| -o | -o response.json | Save output to file instead of stdout. |
| -w | -w "%{http_code}" | Print extra info like HTTP status code after transfer. |
| -L | -L | Follow redirects (3xx responses). |
| -u | -u user:pass | Basic authentication. |
| --fail | --fail | Exit with non-zero code on HTTP errors (4xx, 5xx). Great for scripts. |
# Full production-grade GET request
curl -sS \
-H "Authorization: Bearer $TOKEN" \
-H "Accept: application/json" \
-L --fail \
"https://api.example.com/v2/users" \
| jq
# POST with JSON body
curl -sS \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name":"Alice","role":"admin"}' \
"https://api.example.com/v2/users" \
| jqjq Basics: Filters, Fields, and Arrays
Quick fact
The identity filter . in jq means "pass input through unchanged" — it is how you pretty-print any JSON.
Every jq expression is a filter. Filters read input, transform it, and produce output. You chain filters with | (the pipe operator inside jq).
# Pretty-print (identity filter)
curl -s "https://jsonplaceholder.typicode.com/users/1" | jq '.'
# Extract a single field
curl -s "https://jsonplaceholder.typicode.com/users/1" | jq '.name'
# "Leanne Graham"
# Nested field
curl -s "https://jsonplaceholder.typicode.com/users/1" | jq '.address.city'
# "Gwenborough"
# Array element by index
curl -s "https://jsonplaceholder.typicode.com/users" | jq '.[0].name'
# "Leanne Graham"
# All elements of an array (iterator)
curl -s "https://jsonplaceholder.typicode.com/users" | jq '.[].name'
# Prints each name on a separate line
# Array length
curl -s "https://jsonplaceholder.typicode.com/users" | jq '. | length'
# 10Remove string quotes with -r
-r flag (raw output) to strip them: jq -r '.name'. This is essential when piping jq output into other shell commands.Filtering and Selecting Data
The select() function filters array elements based on a condition. Combined with the iterator .[] it is the jq equivalent of WHERE in SQL.
# Filter todos that are completed
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq '[.[] | select(.completed == true)]'
# Filter users by userId
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq '[.[] | select(.userId == 1)]'
# Filter posts where title contains "qui"
curl -s "https://jsonplaceholder.typicode.com/posts" \
| jq '[.[] | select(.title | contains("qui"))]'
# Count matching items
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq '[.[] | select(.completed == true)] | length'Transforming and Reshaping JSON
jq can reconstruct JSON into any shape using object construction {} and array construction [].
# Pick specific fields (projection)
curl -s "https://jsonplaceholder.typicode.com/users" \
| jq '[.[] | {id, name, email}]'
# Rename fields
curl -s "https://jsonplaceholder.typicode.com/users" \
| jq '[.[] | {userId: .id, fullName: .name, contactEmail: .email}]'
# Add computed fields
curl -s "https://jsonplaceholder.typicode.com/posts" \
| jq '[.[] | . + {titleLength: (.title | length)}]'
# Flatten nested structure
curl -s "https://jsonplaceholder.typicode.com/users" \
| jq '[.[] | {id, name, city: .address.city, lat: .address.geo.lat}]'Aggregation: group_by, sort_by, unique_by
# Group todos by userId
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq 'group_by(.userId)'
# Count todos per user
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq 'group_by(.userId) | map({userId: .[0].userId, count: length})'
# Sort posts by title length
curl -s "https://jsonplaceholder.typicode.com/posts" \
| jq 'sort_by(.title | length)'
# Get unique userIds
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq '[.[].userId] | unique | sort'
# Min and max
curl -s "https://jsonplaceholder.typicode.com/todos" \
| jq 'group_by(.userId) | map({userId: .[0].userId, count: length}) | (max_by(.count), min_by(.count))'Working with Headers and HTTP Status Codes
Quick fact
Always check HTTP status codes in scripts. A 200 OK with an error body is the most common API gotcha.
# Print HTTP status code alongside response
curl -sS -w "\nHTTP_STATUS: %{http_code}\n" \
"https://jsonplaceholder.typicode.com/posts/1" | jq '.'
# Store status code and body separately
HTTP_BODY=$(curl -sS -w "%{http_code}" -o /tmp/api_response.json \
"https://api.example.com/data")
HTTP_CODE=$?
cat /tmp/api_response.json | jq '.'
echo "Status: $HTTP_BODY"
# Exit script on non-2xx (using --fail)
curl -sS --fail \
"https://api.example.com/endpoint" \
| jq '.' || { echo "API request failed"; exit 1; }
# Capture response headers with -D
curl -sS -D /tmp/headers.txt "https://api.example.com/data" | jq '.'
cat /tmp/headers.txtAuthentication Patterns
Bearer Token (OAuth 2.0 / JWT)
Most modern REST APIs use Bearer tokens in the Authorization header.
export TOKEN="eyJhbGciOiJSUzI1NiJ9..."
curl -sS \
-H "Authorization: Bearer $TOKEN" \
"https://api.example.com/me" | jqAPI Key in Header
Some APIs use a custom header like X-API-Key.
export API_KEY="sk-live-abc123"
curl -sS \
-H "X-API-Key: $API_KEY" \
"https://api.example.com/data" | jqBasic Authentication
Legacy APIs or internal tools sometimes use HTTP Basic auth.
curl -sS -u "$USERNAME:$PASSWORD" \
"https://api.internal.com/v1/status" | jqAPI Key in Query String
Some public APIs embed the key in the URL. Less secure — avoid for production.
curl -sS "https://api.example.com/data?api_key=$API_KEY" | jqReal-World Examples
Let us walk through five common real-world scenarios.
# 1. List your GitHub repos and show name + star count
curl -sS \
-H "Authorization: Bearer $GITHUB_TOKEN" \
-H "Accept: application/vnd.github+json" \
"https://api.github.com/user/repos?per_page=100" \
| jq '[.[] | {name, stars: .stargazers_count}] | sort_by(-.stars)'# 2. Get current temperature from OpenWeatherMap
curl -sS "https://api.openweathermap.org/data/2.5/weather?q=London&appid=$OWM_KEY&units=metric" \
| jq '{city: .name, temp: .main.temp, feels_like: .main.feels_like, description: .weather[0].description}'# 3. Handle paginated APIs (fetch all pages)
PAGE=1
ALL_DATA="[]"
while true; do
RESPONSE=$(curl -sS "https://api.example.com/items?page=$PAGE&limit=100")
ITEMS=$(echo "$RESPONSE" | jq '.items')
COUNT=$(echo "$ITEMS" | jq 'length')
if [ "$COUNT" -eq 0 ]; then break; fi
ALL_DATA=$(echo "$ALL_DATA $ITEMS" | jq -s '.[0] + .[1]')
PAGE=$((PAGE + 1))
done
echo "$ALL_DATA" | jq 'length'# 4. Create a resource and capture the new ID
NEW_ID=$(curl -sS -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"title":"New Post","body":"Content here","userId":1}' \
"https://jsonplaceholder.typicode.com/posts" \
| jq -r '.id')
echo "Created post with ID: $NEW_ID"# 5. Use shell variables inside jq with --arg
USER_ID=3
curl -s "https://jsonplaceholder.typicode.com/posts" \
| jq --argjson uid "$USER_ID" '[.[] | select(.userId == $uid)]'Error Handling in Scripts
Silent failures are dangerous
--fail or explicit status checking, curl returns exit code 0 even on 404 or 500 responses. Your script will happily process an error body as if it were valid data.Unsafe (no error handling)
# BAD: no error handling
curl -s "https://api.example.com/data" | jq '.users[]'
# If API returns {"error":"unauthorized"}, jq silently outputs nothingSafe (proper error handling)
# GOOD: check exit codes and validate response
RESPONSE=$(curl -sS --fail "https://api.example.com/data" 2>&1)
if [ $? -ne 0 ]; then
echo "curl failed: $RESPONSE" >&2
exit 1
fi
ERROR=$(echo "$RESPONSE" | jq -r '.error // empty')
if [ -n "$ERROR" ]; then
echo "API error: $ERROR" >&2
exit 1
fi
echo "$RESPONSE" | jq '.users[]'Performance Tips
Use -s flag always
Suppress progress output when piping to jq to avoid corrupting the JSON stream.
Stream large responses
For very large JSON arrays use jq --stream to process elements one at a time without loading the whole file into memory.
Save to file first
When applying multiple jq filters to the same API response, save to a temp file and run jq multiple times rather than making repeated API calls.
Use -c for compact output
jq -c outputs compact (minified) JSON, useful when passing output to another tool or saving space in logs.
Parallel requests with xargs
Combine curl with xargs -P to make multiple API requests in parallel: cat ids.txt | xargs -P 10 -I{} curl -s "https://api.example.com/item/{}"
Cache responses during dev
Save API responses to files during development and jq against the file: jq '.users' response.json. Saves rate limit quota.
Debugging Tips
Verbose mode for request inspection
Use -v to see request headers, TLS handshake, and response headers.
curl -v "https://api.example.com/data" 2>&1 | head -50Check what jq receives
If jq gives unexpected output, first inspect the raw curl response.
# Step 1: see raw response
curl -sS "https://api.example.com/data"
# Step 2: validate it is JSON
curl -sS "https://api.example.com/data" | jq 'type'
# Step 3: apply your filter
curl -sS "https://api.example.com/data" | jq '.data.items'Test jq expressions interactively
Use jqplay.org or echo a sample payload to test filters before scripting.
echo '{"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}' \
| jq '.users[] | select(.id == 2) | .name'
# "Bob"curl + jq vs Alternatives
| Item | curl + jq | Python requests + json |
|---|---|---|
| Setup | Zero deps, works in any shell | Needs Python + requests installed |
| Speed | Fastest for one-liners | Faster for complex logic |
| Readability | Terse, powerful | More readable for complex transformations |
| Error handling | Manual with exit codes | Structured exceptions |
| CI/CD suitability | Excellent — minimal footprint | Good — more setup required |
| Learning curve | jq syntax takes time | Familiar Python syntax |
Frequently Asked Questions
You are now ready to consume any REST API from the terminal