+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Coding (https://sinister.li/Forum-Coding--71)
+--- Thread: Python script that utilizes asynchronous programming to concurrently fetch data (/Thread-Python-script-that-utilizes-asynchronous-programming-to-concurrently-fetch-data)
Python script that utilizes asynchronous programming to concurrently fetch data - vluzzy - 01-02-2024
Requirements:
Install aiohttp, pandas, and plotly libraries:
bash
pip install aiohttp pandas plotly
Here's the script:
python
import aiohttp
import asyncio
import pandas as pd
import plotly.express as px
async def fetch_data(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
data = await response.json()
return data
async def fetch_and_process_data(urls):
tasks = [fetch_data(url) for url in urls]
results = await asyncio.gather(*tasks)
return results
def process_data(results):
combined_data = []
for data in results:
# Perform any necessary processing on the data
combined_data.extend(data)
return combined_data
def visualize_data(data):
df = pd.DataFrame(data, columns=['Date', 'Value', 'Category'])
fig = px.line(df, x='Date', y='Value', color='Category', markers=True, title='Data Visualization')
fig.show()
async def main():
urls = [
'https://api.example.com/data1',
'https://api.example.com/data2',
'https://api.example.com/data3'
]
results = await fetch_and_process_data(urls)
processed_data = process_data(results)
visualize_data(processed_data)
if __name__ == "__main__":
asyncio.run(main())
Explanation:
The script uses aiohttp for asynchronous HTTP requests, allowing multiple requests to be made concurrently.
It defines an asynchronous function fetch_data to fetch data from a given URL.
fetch_and_process_data asynchronously fetches data from multiple URLs concurrently.
The process_data function processes the data as needed.
Finally, the visualize_data function uses Plotly to create an interactive line plot.