{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fa0f3af1",
   "metadata": {},
   "source": [
    "# Performance notes\n",
    "\n",
    "In most cases, minimizing memory usage is Vaex' first priority, and performance comes seconds. This allows Vaex to work with very large datasets, without shooting yourself in the foot.\n",
    "\n",
    "However, this sometimes comes at the cost of performance.\n",
    "\n",
    "## Virtual columns\n",
    "\n",
    "When we add a new column to a dataframe based on existing, Vaex will create a virtual column, e.g.:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8b55a63f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:20:38.880601Z",
     "start_time": "2022-06-24T12:20:38.566851Z"
    }
   },
   "outputs": [],
   "source": [
    "import vaex\n",
    "import numpy as np\n",
    "x = np.arange(100_000_000, dtype='float64')\n",
    "df = vaex.from_arrays(x=x)\n",
    "df['y'] = (df['x'] + 1).log() - np.abs(df['x']**2 + 1).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c58ef3",
   "metadata": {},
   "source": [
    "In this dataframe, `x` uses memory, while `y` does not, it will be evaluate in chunks when needed. To demonstate the performance implications, let us compute with the column, to force the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3f7fa961",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:21:23.390840Z",
     "start_time": "2022-06-24T12:21:23.315840Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.74 s, sys: 12.3 ms, total: 2.75 s\n",
      "Wall time: 71.2 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(49999999.5)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df.x.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b7f9288a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:21:23.795472Z",
     "start_time": "2022-06-24T12:21:23.488106Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.88 s, sys: 635 ms, total: 4.52 s\n",
      "Wall time: 304 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(-17.42068049)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df.y.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f925d5ae",
   "metadata": {},
   "source": [
    "From this, we can see that a similar computation (the mean), with a virtual column can be much slower, a penalty we pay for saving memory.\n",
    "\n",
    "## Materializing the columns\n",
    "\n",
    "We can ask Vaex to materialize a column, or all virtual column using [df.materialize](https://vaex.io/docs/api.html#vaex.dataframe.DataFrame.materialize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "9ecfc226",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:23:22.878087Z",
     "start_time": "2022-06-24T12:23:22.521315Z"
    }
   },
   "outputs": [],
   "source": [
    "df_mat = df.materialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ceef7140",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:23:30.666133Z",
     "start_time": "2022-06-24T12:23:30.594355Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.54 s, sys: 14 ms, total: 2.56 s\n",
      "Wall time: 68.1 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(49999999.5)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df_mat.x.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "62e5d1e6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:23:35.163175Z",
     "start_time": "2022-06-24T12:23:35.091353Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.64 s, sys: 18.7 ms, total: 2.66 s\n",
      "Wall time: 68.1 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(-17.42068049)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "df_mat.y.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "156a6d8e",
   "metadata": {},
   "source": [
    "We now get equal performance for both columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03bc0375",
   "metadata": {},
   "source": [
    "## Consideration in backends with multiple workers\n",
    "\n",
    "As often is the case with web frameworks in Python, we use multiple workers, e.g. using gunicorn. If all workers would materialize, it would waste a lot of memory, there are two solutions to this issue:\n",
    "\n",
    "### Save to disk\n",
    "\n",
    "Export the dataframe to disk in hdf5 or arrow format as a pre-process step, and let all workers access the same file. Due to memory mapping, each worker will share the same memory.\n",
    "\n",
    "e.g.\n",
    "```python\n",
    "df.export('materialized-data.hdf5', progress=True)\n",
    "```\n",
    "\n",
    "\n",
    "### Materialize a single time\n",
    "\n",
    "\n",
    "Gunicorn has the following command line flag:\n",
    "```\n",
    "  --preload             Load application code before the worker processes are forked. [False]\n",
    "```\n",
    "\n",
    "\n",
    "This will let gunicorn first run you app (a single time), allowing you to do the materialize step. After your script run, it will fork, and all workers will share the same memory.\n",
    "\n",
    "\n",
    "### Tip: \n",
    "\n",
    "A good ida could be to mix the two, and use use Vaex' [df.fingerprint](https://vaex.io/docs/api.html#vaex.dataframe.DataFrame.fingerprint) method to cache the file to disk.\n",
    "\n",
    "E.g.\n",
    "```python\n",
    "import vaex\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "x = np.arange(100_000_000, dtype='float64')\n",
    "df = vaex.from_arrays(x=x)\n",
    "df['y'] = (df['x'] + 1).log() - np.abs(df['x']**2 + 1).log()\n",
    "\n",
    "filename = \"vaex-cache-\" + df.fingerprint() + \".hdf5\"\n",
    "if not os.path.exists(filename):\n",
    "    df.export(filename, progress=True)\n",
    "else:\n",
    "    df = vaex.open(filename) \n",
    "```\n",
    "\n",
    "In case the virtual columns change, rerunning will create a new cache file, and changing back will use the previously generated cache file. This is especially useful during development.\n",
    "\n",
    "In this case, it is still important to let gunicorn run a single process first (using the `--preload` flag), to avoid multiple workers doing the same work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "edb8d806",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-24T12:35:18.260847Z",
     "start_time": "2022-06-24T12:35:16.989161Z"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c21e4f2f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}