{ "info": { "author": "Giorgio Basile", "author_email": "giorgiobasile@users.noreply.github.com", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Topic :: Software Development :: Libraries" ], "description": "# prefect-planetary-computer\n\n

\n \n \n \n \"PyPI\"\n \n \n \n \n \n \n
\n \n \n \n \n

\n\nVisit the full docs [here](https://giorgiobasile.github.io/prefect-planetary-computer) to see additional examples and the API reference.\n\nPrefect integrations with the Microsoft [Planetary Computer](https://planetarycomputer.microsoft.com/) (PC).\n\n\n\n## Overview\n\nThis collection includes a [Credentials Block \ud83d\udd11](https://github.com/giorgiobasile/prefect-planetary-computer/) to store and retrieve a PC subscription key and Jupyter Hub token, with convenience methods to easily interact with the PC [Data Catalog \ud83c\udf0d](https://planetarycomputer.microsoft.com/catalog) and [Dask Gateway \ud83d\ude80](https://planetarycomputer.microsoft.com/docs/quickstarts/scale-with-dask/) server.\n\nFor more information about:\n\n- using Azure services with Prefect and the Planetary Computer, check out the [`prefect-azure`](https://github.com/PrefectHQ/prefect-azure/) collection.\n- the integration between Prefect and Dask, check out the [`prefect-dask`](https://github.com/PrefectHQ/prefect-dask/) collection.\n- taking advantage of the Planetary Computer data catalog and compute resources, check out the [Planetary Computer documentation](https://planetarycomputer.microsoft.com/docs/).\n\n## Resources\n\nFor more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://docs.prefect.io/collections/usage/)!\n\n### Installation\n\nInstall `prefect-planetary-computer` with `pip`:\n\n```bash\npip install prefect-planetary-computer\n```\n\nRequires an installation of Python 3.8+.\n\nWe recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.\n\nThese tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).\n\n### Usage\n\n!!! note\n * The following Examples are adapted from [Planetary Computer - Scale with Dask](https://planetarycomputer.microsoft.com/docs/quickstarts/scale-with-dask/).\n\n - Require the following additional packages:\n ```\n pip install xarray zarr adlfs netcdf4 prefect_azure\n ```\n - Make sure to share the same python dependencies - in particular `dask` and `distributed` - between the flow execution environment, the Dask Scheduler and Workers, [as explained in the Dask docs](https://docs.dask.org/en/stable/deployment-considerations.html#consistent-software-environments).\n\n#### Computing Dask Collections\n\nDask collection computations, such as Dask DataFrames, can be supported from within a Prefect task by creating a Dask Gateway cluster using the credentials block within the main flow or task itself.\n\n\n```python\n# Prefect tasks are executed using the default ConcurrentTaskRunner\n# Dask Collections tasks are executed on a new temporary Dask cluster \n\nimport xarray as xr\n\nfrom prefect import flow, task, get_run_logger\nfrom prefect_planetary_computer import PlanetaryComputerCredentials\n\nfrom prefect_azure import AzureBlobStorageCredentials\nfrom prefect_azure.blob_storage import blob_storage_upload\n\npc_credentials = PlanetaryComputerCredentials.load(\"PC_BLOCK_NAME\")\nbs_credentials = AzureBlobStorageCredentials.load(\"BS_BLOCK_NAME\")\n\n@task\ndef compute_mean(asset):\n logger = get_run_logger()\n\n with pc_credentials.new_gateway_cluster(\n name=\"test-cluster\",\n image=\"pangeo/pangeo-notebook:latest\"\n ) as cluster:\n\n cluster.adapt(minimum=2, maximum=10)\n client = cluster.get_client()\n\n ds = xr.open_zarr(\n asset.href,\n **asset.extra_fields[\"xarray:open_kwargs\"],\n storage_options=asset.extra_fields[\"xarray:storage_options\"]\n )\n logger.info(f\"Daymet dataset info\\n: {ds}\")\n \n timeseries = ds[\"tmin\"].mean(dim=[\"x\", \"y\"]).compute()\n logger.info(f\"Mean timeseries info\\n: {timeseries}\")\n\n return timeseries\n\n@flow\ndef pc_dask_flow():\n\n # get a configured PySTAC client\n catalog = pc_credentials.get_stac_catalog()\n\n # compute the minimum daily temperature averaged over all of Hawaii, \n # using the Daymet dataset\n asset = catalog.get_collection(\"daymet-daily-hi\").assets[\"zarr-abfs\"]\n prefect_future = compute_mean.submit(asset)\n timeseries = prefect_future.result()\n\n # save NetCDF timeseries file\n timeseries.to_netcdf(\"timeseries.nc\")\n\n # upload to 'my-container' blob storage container\n with open(\"timeseries.nc\", \"rb\") as f:\n blob = blob_storage_upload(\n data=f.read(),\n container=\"my-container\",\n blob=\"timeseries.nc\",\n blob_storage_credentials=bs_credentials,\n overwrite=False,\n )\n\n # return the blob name of the uploaded timeseries object\n return blob\n\npc_dask_flow()\n```\n\n#### Using the Dask Task Runner\n\nPrefect's [`prefect_dask.DaskTaskRunner`](https://prefecthq.github.io/prefect-dask/task_runners/#prefect_dask.task_runners.DaskTaskRunner) automatically instatiates a temporary Dask cluster at flow execution time, enabling submission of both Prefect and Dask Collections tasks.\n\n!!! warning\n - [`prefect-dask`](https://prefecthq.github.io/prefect-dask/) requires:\n ```\n distributed==2022.2.0; python_version < '3.8'\n distributed>=2022.5.0,<=2023.3.1\n ```\n - It requires [less configuration](https://discourse.prefect.io/t/how-can-i-resolve-this-error-prefecthttpstatuserror-on-prefect-2-3-1-dasktaskrunner/1541/4) on the Dask Workers side when using Prefect Cloud, you can [get started](https://docs.prefect.io/2.11.3/cloud/cloud-quickstart/) for free.\n\n```python\n# Both Prefect tasks and Dask Collections task are executed\n# on a new temporary Dask cluster \nimport xarray as xr\n\nfrom prefect import flow, task, get_run_logger\nfrom prefect_planetary_computer import PlanetaryComputerCredentials\n\nfrom prefect_azure import AzureBlobStorageCredentials\nfrom prefect_azure.blob_storage import blob_storage_upload\n\nfrom prefect_dask import get_dask_client \n\npc_credentials = PlanetaryComputerCredentials.load(\"PC_BLOCK_NAME\")\nbs_credentials = AzureBlobStorageCredentials.load(\"BS_BLOCK_NAME\")\n\npc_runner = pc_credentials.get_dask_task_runner(\n cluster_kwargs={\n \"image\": \"pangeo/pangeo-notebook:latest\",\n },\n adapt_kwargs={'minimum': 1, 'maximum': 10, 'active': True}\n)\n\n@task\ndef compute_mean(asset):\n logger = get_run_logger()\n\n with get_dask_client() as client:\n ds = xr.open_zarr(\n asset.hr\n **asset.extra_fields[\"xarray:open_kwargs\"],\n storage_options=asset.extra_fields[\"xarray:storage_options\"]\n )\n logger.info(f\"Daymet dataset info\\n: {ds}\")\n\n timeseries = ds[\"tmin\"].mean(dim=[\"x\", \"y\"]).compute()\n logger.info(f\"Mean timeseries info\\n: {timeseries}\")\n\n return timeseries\n\n@flow(task_runner=pc_runner)\ndef pc_dask_flow():\n \n # get a configured PySTAC client\n catalog = pc_credentials.get_stac_catalog()\n\n # compute the minimum daily temperature averaged over all of Hawaii, \n # using the Daymet dataset\n asset = catalog.get_collection(\"daymet-daily-hi\").assets[\"zarr-abfs\"]\n\n mean_task = compute_mean.submit(asset)\n timeseries = mean_task.result()\n\n # save NetCDF timeseries file\n timeseries.to_netcdf(\"timeseries.nc\")\n\n # upload to 'my-container' blob storage container\n with open(\"timeseries.nc\", \"rb\") as f:\n blob = blob_storage_upload(\n data=f.read(),\n container=\"my-container\",\n blob=\"timeseries.nc\",\n blob_storage_credentials=bs_credentials,\n overwrite=False,\n )\n\n # return the blob name of the uploaded timeseries object\n return blob\n\npc_dask_flow()\n```\n\n### Feedback\n\nIf you encounter any bugs while using `prefect-planetary-computer`, feel free to open an issue in the [prefect-planetary-computer](https://github.com/giorgiobasile/prefect-planetary-computer) repository.\n\nIf you have any questions or issues while using `prefect-planetary-computer`, you can find help in either the [Prefect Discourse forum](https://discourse.prefect.io/) or the [Prefect Slack community](https://prefect.io/slack).\n\nFeel free to star or watch [`prefect-planetary-computer`](https://github.com/giorgiobasile/prefect-planetary-computer) for updates too!\n\n### Contributing\n\nIf you'd like to help contribute to fix an issue or add a feature to `prefect-planetary-computer`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n\nHere are the steps:\n\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\npip install -e \".[dev]\"\n```\n4. Make desired changes\n5. Add tests\n6. Insert an entry to [CHANGELOG.md](https://github.com/giorgiobasile/prefect-planetary-computer/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\npre-commit install\n```\n8. `git commit`, `git push`, and create a pull request\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/giorgiobasile/prefect-planetary-computer", "keywords": "prefect", "license": "Apache License 2.0", "maintainer": "", "maintainer_email": "", "name": "prefect-planetary-computer", "package_url": "https://pypi.org/project/prefect-planetary-computer/", "platform": null, "project_url": "https://pypi.org/project/prefect-planetary-computer/", "project_urls": { "Homepage": "https://github.com/giorgiobasile/prefect-planetary-computer" }, "release_url": "https://pypi.org/project/prefect-planetary-computer/0.1.1/", "requires_dist": [ "prefect >=2.0.0", "prefect-dask", "planetary-computer", "dask-gateway", "pystac-client", "pytest ; extra == 'dev'", "black ; extra == 'dev'", "flake8 ; extra == 'dev'", "mypy ; extra == 'dev'", "mkdocs ; extra == 'dev'", "mkdocs-material ; extra == 'dev'", "mkdocstrings[python] ; extra == 'dev'", "isort ; extra == 'dev'", "pre-commit ; extra == 'dev'", "pytest-asyncio ; extra == 'dev'", "mkdocs-gen-files ; extra == 'dev'", "interrogate ; extra == 'dev'", "coverage ; extra == 'dev'", "pillow ; extra == 'dev'", "requests-mock ; extra == 'dev'", "importlib-resources ; extra == 'dev'", "numpy ; extra == 'dev'", "mock ; (python_version < \"3.8\") and extra == 'dev'" ], "requires_python": ">=3.8", "summary": "Prefect integrations with Microsoft Planetary Computer", "version": "0.1.1", "yanked": false, "yanked_reason": null }, "last_serial": 20273679, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "blake2b_256": "e48e670c647c1f40d78ad3a96b291e35a52fdb7d4747f89a6208db43cb97dc43", "md5": "6b2de89476a1db77388dcbdaf6d86766", "sha256": "dde2235879b8a7d072920c4b8b303a834e7e15ef23c81b586944fe57545957db" }, "downloads": -1, "filename": "prefect_planetary_computer-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "6b2de89476a1db77388dcbdaf6d86766", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.8", "size": 14600, "upload_time": "2023-08-11T13:12:42", "upload_time_iso_8601": "2023-08-11T13:12:42.068943Z", "url": "https://files.pythonhosted.org/packages/e4/8e/670c647c1f40d78ad3a96b291e35a52fdb7d4747f89a6208db43cb97dc43/prefect_planetary_computer-0.1.0-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "blake2b_256": "a6130351571d7d344bdd60c85d9f5e27ef7e43c75de05fe99a9fb54b7c999fb9", "md5": "3beb42453eb0579241ee9addac8ab713", "sha256": "b20250ba6bef212e9bc8cdaa7deba6de3358eec7a358042dc063dcdf516351d5" }, "downloads": -1, "filename": "prefect-planetary-computer-0.1.0.tar.gz", "has_sig": false, "md5_digest": "3beb42453eb0579241ee9addac8ab713", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.8", "size": 34996, "upload_time": "2023-08-11T13:12:44", "upload_time_iso_8601": "2023-08-11T13:12:44.923068Z", "url": "https://files.pythonhosted.org/packages/a6/13/0351571d7d344bdd60c85d9f5e27ef7e43c75de05fe99a9fb54b7c999fb9/prefect-planetary-computer-0.1.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.1": [ { "comment_text": "", "digests": { "blake2b_256": "c7e581f34c259c873bb8c97e0c3e3fec0a592fc5cddbce0d2ff005ec18de76c4", "md5": "050e058c41589af09216d1847e7c3f62", "sha256": "0f02d5c32018d8dcfa3b98391925dcc2227bdc4fbdd345e9aed7d73241b10bae" }, "downloads": -1, "filename": "prefect_planetary_computer-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "050e058c41589af09216d1847e7c3f62", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.8", "size": 14645, "upload_time": "2023-10-21T14:26:02", "upload_time_iso_8601": "2023-10-21T14:26:02.087466Z", "url": "https://files.pythonhosted.org/packages/c7/e5/81f34c259c873bb8c97e0c3e3fec0a592fc5cddbce0d2ff005ec18de76c4/prefect_planetary_computer-0.1.1-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "blake2b_256": "5a5ce30996fc768e2ba3dad56927f05814c6bf4c53c1efb2cf8af8d94b36390d", "md5": "365a6b5beadea63bfb9ad1df836e6154", "sha256": "d3df06c0134a532518829c5ee72bcf8f0ab918783c6374892a60ea79819ce4aa" }, "downloads": -1, "filename": "prefect-planetary-computer-0.1.1.tar.gz", "has_sig": false, "md5_digest": "365a6b5beadea63bfb9ad1df836e6154", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.8", "size": 35511, "upload_time": "2023-10-21T14:26:04", "upload_time_iso_8601": "2023-10-21T14:26:04.011954Z", "url": "https://files.pythonhosted.org/packages/5a/5c/e30996fc768e2ba3dad56927f05814c6bf4c53c1efb2cf8af8d94b36390d/prefect-planetary-computer-0.1.1.tar.gz", "yanked": false, "yanked_reason": null } ] }, "urls": [ { "comment_text": "", "digests": { "blake2b_256": "c7e581f34c259c873bb8c97e0c3e3fec0a592fc5cddbce0d2ff005ec18de76c4", "md5": "050e058c41589af09216d1847e7c3f62", "sha256": "0f02d5c32018d8dcfa3b98391925dcc2227bdc4fbdd345e9aed7d73241b10bae" }, "downloads": -1, "filename": "prefect_planetary_computer-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "050e058c41589af09216d1847e7c3f62", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.8", "size": 14645, "upload_time": "2023-10-21T14:26:02", "upload_time_iso_8601": "2023-10-21T14:26:02.087466Z", "url": "https://files.pythonhosted.org/packages/c7/e5/81f34c259c873bb8c97e0c3e3fec0a592fc5cddbce0d2ff005ec18de76c4/prefect_planetary_computer-0.1.1-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "blake2b_256": "5a5ce30996fc768e2ba3dad56927f05814c6bf4c53c1efb2cf8af8d94b36390d", "md5": "365a6b5beadea63bfb9ad1df836e6154", "sha256": "d3df06c0134a532518829c5ee72bcf8f0ab918783c6374892a60ea79819ce4aa" }, "downloads": -1, "filename": "prefect-planetary-computer-0.1.1.tar.gz", "has_sig": false, "md5_digest": "365a6b5beadea63bfb9ad1df836e6154", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.8", "size": 35511, "upload_time": "2023-10-21T14:26:04", "upload_time_iso_8601": "2023-10-21T14:26:04.011954Z", "url": "https://files.pythonhosted.org/packages/5a/5c/e30996fc768e2ba3dad56927f05814c6bf4c53c1efb2cf8af8d94b36390d/prefect-planetary-computer-0.1.1.tar.gz", "yanked": false, "yanked_reason": null } ], "vulnerabilities": [] }