Skip to main content

How To Handle Large Response From An API In Python

Handling large responses from APIs in Python can be challenging, especially when dealing with substantial amounts of data. Here are some strategies to efficiently manage such scenarios:


If the API supports it, use pagination to retrieve data in smaller chunks. Fetch a limited number of records per request, and then iterate through subsequent pages. This approach prevents overwhelming your system with a massive response all at once.

import requests
def fetch_data_from_api(url, page_size=50):
    all_data = []
    page = 1
    while True:
        response = requests.get(url, params={"page": page, "per_page": page_size})
        if not response.ok:
        data = response.json()
        page += 1
    return all_data

# Example usage:
api_url = ""
result = fetch_data_from_api(api_url)

Asynchronous Processing

Consider an asynchronous approach to handle large responses. Use libraries like Tenacity to optimize your software for maximum API calls within your current plan’s limits. When you encounter a quota-exceeded response, your application can gracefully handle it without crashing.

import asyncio
import aiohttp

async def fetch_data_async(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            data = await response.json()
            return data

# Example usage:
api_url = ""
loop = asyncio.get_event_loop()
result = loop.run_until_complete(fetch_data_async(api_url))

Event Storage

Convert each item in the API response to a domain event and store it immediately in an event storage. Later, transform these events into the desired format for your read database (i.e, MySQL) using eventual consistency.

class EventStorage:
    def __init__(self): = []

    def store_event(self, event):

# Example usage:
event_storage = EventStorage()
api_response = [{"id": 100, "name": "Value1"}, {"id": 200, "name": "Value2"}]
for item in api_response:

Range Requests

When downloading large files from an API, set the Range header to retrieve specific byte ranges. In Python, you can achieve this using the requests library with streaming

Here is the quick code to handle large response from an API:

import requests

def download_large_file(url, output_file):
    headers = {"Range": "bytes=0-999999"}  # mention the byte range as needed
    response = requests.get(url, headers=headers, stream=True)
    with open(output_file, "wb") as f:
        for chunk in response.iter_content(chunk_size=1024):

# Example usage:
large_file_url = ""
output_filename = ""
download_large_file(large_file_url, output_filename)
print(f"Downloaded {output_filename}")

One more example:

import requests
response = requests.get('',
for chunk in response.iter_content(chunk_size=1024):
    # process your chunk here

Hope you find this useful. Happy learning!