{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Asynchronous Python\n", "\n", "- [RealPython - Async IO in Python: A Complete Walkthrough](https://realpython.com/async-io-python/)\n", "- [Gregory Saunders - A Really Gentle Introduction to Asyncio](https://www.youtube.com/watch?v=3mb9jFAHRfw)\n", "- [Miguel Grinberg - Asynchronous Python for the Complete Beginner](https://speakerdeck.com/pycon2017/miguel-grinberg-asynchronous-python-for-the-complete-beginner)\n", " - [ [Video](https://www.youtube.com/watch?v=iG6fr81xHKA) ]\n", "- [Yury Selivanov - asyncawait and asyncio in Python 3 6 and beyond PyCon 2017](https://github.com/PyCon/2017-slides/blob/master/Yury%20Selivanov%20-%20async-await%20and%20asyncio%20in%20Python%203.6%20and%20beyond.pptx)\n", " - [ [Video](https://www.youtube.com/watch?v=2ZFFv-wZ8_g) ]\n", "- [Morvan - 加速爬虫: 异步加载 Asyncio](https://morvanzhou.github.io/tutorials/data-manipulation/scraping/4-02-asyncio/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key terms\n", "\n", "- **Coroutines**\n", "- **callbacks, events, transports, protocols**\n", "- **Futures**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Steps\n", "\n", "1. `async` both worker and main functions\n", "2. `loop = get_event_loop()`\n", "3. `loop.run_until_complete()`\n", "4. `future_result = loop.create_task()`\n", "5. `await asyncio.wait([future_result1, future_result2, ...])`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Async IO Design Patterns" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Asyncio Chaining\n", "\n", "```python\n", "import asyncio\n", "\n", "async def compute(x, y):\n", " print(\"Compute %s + %s ...\" % (x, y))\n", " await asyncio.sleep(1.0)\n", " return x + y\n", "\n", "async def print_sum(x, y):\n", " result = await compute(x, y)\n", " print(\"%s + %s = %s\" % (x, y, result))\n", "\n", "loop = asyncio.get_event_loop()\n", "loop.run_until_complete(print_sum(1, 2))\n", "loop.close()\n", "```\n", "\n", "[](https://docs.python.org/3.6/library/asyncio-task.html#example-chain-coroutines)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "import asyncio\n", "\n", "\n", "async def find_divisibles(inrange, div_by):\n", " \"\"\"A time-comsuming job\n", "\n", " [description]\n", " \"\"\"\n", " print(f'Finding nums in range {inrange} divisible by {div_by} ...')\n", "\n", " located = []\n", " for i in range(inrange):\n", " if i % div_by == 0:\n", " located.append(i)\n", " if i % 50000 == 0:\n", " # the magic trick to make function really async:\n", " # hang over the processing time to the event loop\n", " await asyncio.sleep(0.000001)\n", "\n", " print(f'(Done with nums in range {inrange} divisible by {div_by})')\n", " return located\n", "\n", "\n", "async def main():\n", " '''event loop'''\n", "\n", " div1 = loop.create_task(find_divisibles(5000000, 42713))\n", " div2 = loop.create_task(find_divisibles(1000000, 112))\n", " div3 = loop.create_task(find_divisibles(6000, 22))\n", " await asyncio.wait([div1, div2, div3])\n", "\n", " return div1, div2, div3" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finding nums in range 50000000 divisible by 42713 ...\n", "Finding nums in range 1000000 divisible by 112 ...\n", "Finding nums in range 6000 divisible by 22 ...\n", "(Done with nums in range 6000 divisible by 22)\n", "(Done with nums in range 1000000 divisible by 112)\n", "(Done with nums in range 50000000 divisible by 42713)\n", "[0, 42713, 85426, 128139, 170852]\n" ] } ], "source": [ "if __name__ == '__main__':\n", " try:\n", " # loop = asyncio.get_event_loop()\n", " # loop.set_debug(True)\n", " \n", " # https://stackoverflow.com/a/55409674/3744499\n", " # Jupyter is already running an event loop, \n", " # so you don't need to start the event loop yourself in jupyter\n", " # and you can directly call await main(url)\n", "\n", " # d1, d2, d3 = loop.run_until_complete(main())\n", " d1, d2, d3 = await main()\n", " \n", " print(d1.result()[:5])\n", " except Exception as e:\n", " raise e\n", " finally:\n", " # You always want to close the loop.\n", " # loop.close()\n", " pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Start job 1\n", "Start job 2\n", "Start job 3\n", "Job 1 takes 1 sec\n", "Job 2 takes 4 sec\n", "Job 3 takes 9 sec\n", "Async total time : 9.007589101791382\n" ] } ], "source": [ "import asyncio\n", "import time\n", "\n", "\n", "async def job(t): # async 形式的功能\n", " print(f'Start job {t}')\n", " await asyncio.sleep(t**2) # 等待 \"t\" 秒, 期间切换其他任务\n", " print(f'Job {t} takes {t**2} sec')\n", "\n", "\n", "async def main(loop): # async 形式的功能\n", " tasks = [loop.create_task(job(t)) for t in range(1, 4)] # 创建任务, 但是不执行\n", " await asyncio.wait(tasks) # 执行并等待所有任务完成\n", "\n", "t1 = time.time()\n", "loop = asyncio.new_event_loop() # 建立 loop\n", "loop.run_until_complete(main(loop)) # 执行 loop\n", "loop.close() # 关闭 loop\n", "\n", "print(\"Async total time : \", time.time() - t1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### aiohttp" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }