Data Types¶
Vaex is a hybrid DataFrame - it supports both numpy and arrow data types. This page outlines exactly which data types are supported in Vaex, and which we hope to support in the future. We also provide some tips on how to approach common data structures.
For some additional insight, you are welcome to look at this post as well.
Supported Data Types in Vaex¶
In the table below we define:
Supported: a column or expression of that type can exist and can be stored in at least one file format;
Unsupported: a column or expression of that type can currently not live within a Vaex DataFrame, but can supported be added in the future;
Will not support: This datatype will not be supported in Vaex going forward.
Framework |
Dtype |
Supported |
Remarks |
|---|---|---|---|
Python |
|
yes |
Will be converted to a |
Python |
|
yes |
Will be converted to a |
Python |
|
not yet |
|
Python |
|
not yet |
|
Python |
|
yes |
Will be converted to Arrow array |
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
Operations should be upcast to |
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
no |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
Operations should be upcast to |
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
|
|
|
yes |
Can’t be exported to HDF5 yet, but possible |
|
|
yes |
|
|
|
not yet |
General advice on data types in Vaex¶
Vaex requires that each column or expression be of a single data type, as in the case of databases. Having a column of different data type can result in a data type object, which is not supported, and can also give raise to various problems.
The general advice is to prepare your data to have a uniform data type per column prior to using Vaex with it.
[1]:
import vaex
import numpy as np
import pyarrow as pa
Higher dimensional arrays¶
Vaex support high dimensional numpy arrays. The one requirement the arrays must have the same shape. Currently DataFrames that contain higher dimensional numpy arrays can only be exported to HDF5. We hope that arrow will add support for this soon, so we can export such columns to the arrow and parquet formats also.
For example:
[2]:
x = np.random.randn(100, 10, 10)
df = vaex.from_arrays(x=x)
df
[2]:
| # | x |
|---|---|
| 0 | 'array([[ 1.83097431e+00, -9.90736404e-01, -8.85... |
| 1 | 'array([[ 1.99466370e+00, 8.92569841e-01, 2.11... |
| 2 | 'array([[-0.69977757, 0.61319317, 0.01313954, ... |
| 3 | 'array([[ 0.25304255, -0.84425097, -1.18806199, ... |
| 4 | 'array([[ 0.31611316, -1.36148251, 1.67342284, ... |
| ... | ... |
| 95 | 'array([[-0.60892972, -0.52389881, -0.92493729, ... |
| 96 | 'array([[ 1.10435809, 1.06875633, 1.45812865, ... |
| 97 | 'array([[-0.59311765, 0.10650056, -0.14413671, ... |
| 98 | 'array([[-0.24467361, -0.40743024, 0.6914302 , ... |
| 99 | 'array([[-1.0646038 , 0.53975242, -1.70715565, ... |
We can also pass a nested list of lists structure to Vaex. This will be converted on the fly to a numpy ndarray. As before, the condition is that the resulting ndarray must be regular.
For example:
[3]:
x = [[1, 2], [3, 4]]
df = vaex.from_arrays(x=x)
df
[3]:
| # | x |
|---|---|
| 0 | array([1, 2]) |
| 1 | array([3, 4]) |
If we happen to have a non-regular list of lists, i.e. a list of lists where the inner lists are of different lengths, we first need to convert it to an arrow array before passing it to vaex:
[4]:
x = [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10]]
x = pa.array(x)
df = vaex.from_arrays(x=x)
df
[4]:
| # | x |
|---|---|
| 0 | [1, 2, 3, 4, 5] |
| 1 | [6, 7] |
| 2 | [8, 9, 10] |
Note the arrow structs and lists can not be exported to HDF5 for the time being.
String support in Vaex¶
Vaex uses arrow under the hood to work with strings. Any strings passed to a Vaex DataFrame will be converted to an arrow type.
For example:
[5]:
x = ['This', 'is', 'a', 'string', 'column']
y = np.array(['This', 'is', 'one', 'also', None])
df = vaex.from_arrays(x=x, y=y)
print(df)
display(df.x.values)
display(df.y.values)
# x y
0 This This
1 is is
2 a one
3 string also
4 column --
<pyarrow.lib.StringArray object at 0x7f277b9b9040>
[
"This",
"is",
"a",
"string",
"column"
]
<pyarrow.lib.StringArray object at 0x7f277b9b9d60>
[
"This",
"is",
"one",
"also",
null
]
It is useful to know that string operations in Vaex also work on lists of lists of strings (and also on lists of lists of lists of strings, and so on).
[6]:
x = pa.array([['Reggie', 'Miller'], ['Michael', 'Jordan']])
df = vaex.from_arrays(x=x)
df.x.str.lower()
[6]:
Expression = str_lower(x)
Length: 2 dtype: list<item: string> (expression)
------------------------------------------------
0 ['reggie', 'miller']
1 ['michael', 'jordan']