{
    "info": {
        "author": "Based Labs",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "<div align=\"center\" id=\"top\"> \n  <img src=\"https://github.com/BasedLabs/aibenchmark/raw/4f774ab8ad881724103b69ecd328a1eb80a94d3b/media/aibenchmark-logo.png\" width=\"250px\" alt=\"aibencharmk\" />\n\n  &#xa0;\n\n  <!-- <a href=\"https://aibenchmark.netlify.app\">Demo</a> -->\n</div>\n\n<h1 align=\"center\">AIBenchmark</h1>\n<h2 align=\"center\">Benchmark your model against other models</h2>\n\n<p align=\"center\">\n  <img alt=\"Github top language\" src=\"https://img.shields.io/github/languages/top/BasedLabs/aibenchmark?color=56BEB8\">\n\n  <img alt=\"Github language count\" src=\"https://img.shields.io/github/languages/count/BasedLabs/aibenchmark?color=56BEB8\">\n\n  <img alt=\"Repository size\" src=\"https://img.shields.io/github/repo-size/BasedLabs/aibenchmark?color=56BEB8\">\n\n  <img alt=\"License\" src=\"https://img.shields.io/github/license/BasedLabs/aibenchmark?color=56BEB8\">\n\n  <!-- <img alt=\"Github issues\" src=\"https://img.shields.io/github/issues/BasedLabs/aibenchmark?color=56BEB8\" /> -->\n\n  <!-- <img alt=\"Github forks\" src=\"https://img.shields.io/github/forks/BasedLabs/aibenchmark?color=56BEB8\" /> -->\n\n  <!-- <img alt=\"Github stars\" src=\"https://img.shields.io/github/stars/BasedLabs/aibenchmark?color=56BEB8\" /> -->\n</p>\n\n<!-- Status -->\n\n<!-- <h4 align=\"center\"> \n  \ud83d\udea7  NoLabs \ud83d\ude80 Under construction...  \ud83d\udea7\n</h4> \n\n<hr> -->\n\n<p align=\"center\">\n  <a href=\"#dart-about\">About</a> &#xa0; | &#xa0; \n  <a href=\"#sparkles-features\">Features</a> &#xa0; | &#xa0;\n  <a href=\"#Technologies\">Technologies</a> &#xa0; | &#xa0;\n  <a href=\"#checkered_flag-starting\">Starting</a> &#xa0; | &#xa0;\n  <a href=\"#memo-license\">License</a> &#xa0; | &#xa0;\n  <a href=\"https://github.com/BasedLabs\" target=\"_blank\">Author</a>\n</p>\n\n<br>\n\n## Installation ##\n\nRun this script in your terminal:\n```bash\n$ pip install aibench\n```\n\n## About ##\n\nAIBenchmark is a package which lets you quickly get the benchmark of your model based on the popular datasets and compare with existing leaderboard. It also has a nice collection of metrics which you could easily import.\n\nWe currently support 14 text-based and 2 image-based datasets for AutoBenchmarking aiming for regression/classification tasks. Available datasets could be found in aibenchmark/dataset.py file. \n\nOr run the following code:\n\n```python\n\nfrom aibenchmark.dataset import DatasetsList\n\nprint(list(DatasetsList.get_available_datasets()))\n\n```\n\nCode example for benchmarking:\n\n```python\nfrom aibenchmark.benchmark import Benchmark\nfrom aibenchmark.dataset import DatasetInfo, DatasetsList\n\n\nbenchmark = Benchmark(DatasetsList.Texts.SST)\ndataset_info: DatasetInfo = benchmark.dataset_info\nprint(dataset_info)\n\ntest_features = dataset_info.data['Texts']\nmodel = torch.load(...)\n# Implement your code based on the type of model you use, your pre- and post-processing etc.\noutputs = model.predict(test_features)\n\n# Results of your model based on predictions\nbenchmark_results = benchmark.run(predictions=outputs, metrics=['accuracy', 'precision', 'recall', 'f1_score']) \n\n# Metrics\nprint(benchmark_results)\n# Existing leaderboard for this dataset\nprint(benchmark.get_existing_benchmarks())\n```\n\n## Features ##\n\n1) Fast comparison of metrics of your model and other SOTA models for particular dataset\n2) Supporting 16+ most populat datasets, the list is always updating. Soon we willl support more than 1000 datasets\n3) All metrics in one place and we are adding new ones in a standardised way\n\n## Technologies ##\n\nThe following tools were used in this project:\n\n- [Pytorch](https://pytorch.org/)\n- [Transformers](https://huggingface.co/transformers)\n- [Scikit-learn](https://scikit-learn.org/stable/)\n\n\n## :memo: License ##\n\nThis project is under license from MIT. For more details, see the [LICENSE](LICENSE.md) file.\n\n\nMade by <a href=\"https://github.com/jaktenstid\" target=\"_blank\">Igor</a> and <a href=\"https://github.com/timurishmuratov7\" target=\"_blank\">Tim</a>\n\n&#xa0;\n\n<a href=\"#top\">Back to top</a>\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/BasedLabs/aibenchmark/",
        "keywords": "ai benchmark metrics",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "aibench",
        "package_url": "https://pypi.org/project/aibench/",
        "platform": null,
        "project_url": "https://pypi.org/project/aibench/",
        "project_urls": {
            "Homepage": "https://github.com/BasedLabs/aibenchmark/"
        },
        "release_url": "https://pypi.org/project/aibench/0.0.5/",
        "requires_dist": [
            "datasets (==2.13.0)",
            "gdown (==4.7.1)",
            "lxml (==4.9.2)",
            "numpy (==1.25.0)",
            "pandas (==2.0.2)",
            "py7zr (==0.20.5)",
            "pytest (==7.3.2)",
            "Requests (==2.31.0)",
            "scikit-learn (==1.2.2)",
            "setuptools (==68.0.0)",
            "torch (==2.0.1)"
        ],
        "requires_python": "",
        "summary": "",
        "version": "0.0.5",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 18699441,
    "releases": {
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "8bf74d27bdc624fbb8c1da742f97887ca895c4c70b41b3b017ac180d3601b097",
                    "md5": "d2214bf820227322c5e765128601ce35",
                    "sha256": "427534a110f600f810d7f09ffafdbd3865722d57dce75dd632ee2955608ede44"
                },
                "downloads": -1,
                "filename": "aibench-0.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d2214bf820227322c5e765128601ce35",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9242,
                "upload_time": "2023-06-25T14:48:48",
                "upload_time_iso_8601": "2023-06-25T14:48:48.994740Z",
                "url": "https://files.pythonhosted.org/packages/8b/f7/4d27bdc624fbb8c1da742f97887ca895c4c70b41b3b017ac180d3601b097/aibench-0.0.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "ebbfcf39d8b6815a670003b8fda29d4f11b62c7817f08ca1d8ab88c94f108a0e",
                    "md5": "a8f8e6bce796332cbcb91242b8b627b1",
                    "sha256": "e56c40e192c7818760c87cf6fc5d8680897fbd2097aa00af621f4410ce548412"
                },
                "downloads": -1,
                "filename": "aibench-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "a8f8e6bce796332cbcb91242b8b627b1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10340,
                "upload_time": "2023-06-25T14:48:50",
                "upload_time_iso_8601": "2023-06-25T14:48:50.603422Z",
                "url": "https://files.pythonhosted.org/packages/eb/bf/cf39d8b6815a670003b8fda29d4f11b62c7817f08ca1d8ab88c94f108a0e/aibench-0.0.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "cb10ace3b7082115d226a9211ad7a9db6c417e328548703755ea884980baf8e7",
                    "md5": "a30435271968ae7ccc4d444957712919",
                    "sha256": "01c8b976a0fe81197e6b9aa7366e51be4de9a793dd8d8bec472a366ca23d2713"
                },
                "downloads": -1,
                "filename": "aibench-0.0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a30435271968ae7ccc4d444957712919",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9273,
                "upload_time": "2023-06-25T15:34:37",
                "upload_time_iso_8601": "2023-06-25T15:34:37.876862Z",
                "url": "https://files.pythonhosted.org/packages/cb/10/ace3b7082115d226a9211ad7a9db6c417e328548703755ea884980baf8e7/aibench-0.0.3-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "5b3a2d183e0ffd56f8a999768b5b3300f30b8562204a7cf3c94bf12b54e62f55",
                    "md5": "9661923c9b126acc8a23079fc84d8e06",
                    "sha256": "d223dddfe4f3d11252048e05b897f99e84ad436410c985ad5c2a53dd88020f18"
                },
                "downloads": -1,
                "filename": "aibench-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "9661923c9b126acc8a23079fc84d8e06",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9765,
                "upload_time": "2023-06-25T15:34:39",
                "upload_time_iso_8601": "2023-06-25T15:34:39.101310Z",
                "url": "https://files.pythonhosted.org/packages/5b/3a/2d183e0ffd56f8a999768b5b3300f30b8562204a7cf3c94bf12b54e62f55/aibench-0.0.3.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "dac9a3e9508400158a901860c630847d62f0b577fcbd937cf7662aa49772f04d",
                    "md5": "8ceb6f2d908f9d30a415c72f0fe38801",
                    "sha256": "3700929654ebb5357234c6463175d9135fee53ff982b1e7ead0689ad744b7d9a"
                },
                "downloads": -1,
                "filename": "aibench-0.0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8ceb6f2d908f9d30a415c72f0fe38801",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10708,
                "upload_time": "2023-06-25T15:38:09",
                "upload_time_iso_8601": "2023-06-25T15:38:09.123945Z",
                "url": "https://files.pythonhosted.org/packages/da/c9/a3e9508400158a901860c630847d62f0b577fcbd937cf7662aa49772f04d/aibench-0.0.4-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "22ba9598cb7de58b61d4d9ee93e504ad6b367a1fecb0a9e167edefc29a91485a",
                    "md5": "3d0f65a0a4b1f8d27c2d4838946fbbb6",
                    "sha256": "ab3c3d29902a1ad350472d2074f6ad1b63b4b7e50663af751eb7ede7b261ef50"
                },
                "downloads": -1,
                "filename": "aibench-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "3d0f65a0a4b1f8d27c2d4838946fbbb6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10653,
                "upload_time": "2023-06-25T15:38:10",
                "upload_time_iso_8601": "2023-06-25T15:38:10.671460Z",
                "url": "https://files.pythonhosted.org/packages/22/ba/9598cb7de58b61d4d9ee93e504ad6b367a1fecb0a9e167edefc29a91485a/aibench-0.0.4.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "5a916e4192a6e51248e148d6b1a6ef840937c0fa42b4b5e1840362fbe96af513",
                    "md5": "66776fcf7ab244d1862101073dec50dd",
                    "sha256": "71b62f519c22438acde8dc306a73c9ddc7bd6333456530fffeea65731d7c3b7e"
                },
                "downloads": -1,
                "filename": "aibench-0.0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "66776fcf7ab244d1862101073dec50dd",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10754,
                "upload_time": "2023-06-29T19:22:19",
                "upload_time_iso_8601": "2023-06-29T19:22:19.531287Z",
                "url": "https://files.pythonhosted.org/packages/5a/91/6e4192a6e51248e148d6b1a6ef840937c0fa42b4b5e1840362fbe96af513/aibench-0.0.5-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "blake2b_256": "14e8b9059615ad115e612f43c19f6290cea6410cd9273b20050d3135e3d9912a",
                    "md5": "abe81ba1f3931ca0ef789ce816eeb339",
                    "sha256": "b46b4cb6a6dd1d36fef7f57d1ded5a42d9a767f108d02ee87e7a99301ed1be2d"
                },
                "downloads": -1,
                "filename": "aibench-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "abe81ba1f3931ca0ef789ce816eeb339",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10382,
                "upload_time": "2023-06-29T19:22:20",
                "upload_time_iso_8601": "2023-06-29T19:22:20.905821Z",
                "url": "https://files.pythonhosted.org/packages/14/e8/b9059615ad115e612f43c19f6290cea6410cd9273b20050d3135e3d9912a/aibench-0.0.5.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5a916e4192a6e51248e148d6b1a6ef840937c0fa42b4b5e1840362fbe96af513",
                "md5": "66776fcf7ab244d1862101073dec50dd",
                "sha256": "71b62f519c22438acde8dc306a73c9ddc7bd6333456530fffeea65731d7c3b7e"
            },
            "downloads": -1,
            "filename": "aibench-0.0.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "66776fcf7ab244d1862101073dec50dd",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 10754,
            "upload_time": "2023-06-29T19:22:19",
            "upload_time_iso_8601": "2023-06-29T19:22:19.531287Z",
            "url": "https://files.pythonhosted.org/packages/5a/91/6e4192a6e51248e148d6b1a6ef840937c0fa42b4b5e1840362fbe96af513/aibench-0.0.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "14e8b9059615ad115e612f43c19f6290cea6410cd9273b20050d3135e3d9912a",
                "md5": "abe81ba1f3931ca0ef789ce816eeb339",
                "sha256": "b46b4cb6a6dd1d36fef7f57d1ded5a42d9a767f108d02ee87e7a99301ed1be2d"
            },
            "downloads": -1,
            "filename": "aibench-0.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "abe81ba1f3931ca0ef789ce816eeb339",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10382,
            "upload_time": "2023-06-29T19:22:20",
            "upload_time_iso_8601": "2023-06-29T19:22:20.905821Z",
            "url": "https://files.pythonhosted.org/packages/14/e8/b9059615ad115e612f43c19f6290cea6410cd9273b20050d3135e3d9912a/aibench-0.0.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}