# Data Types¶

Vaex is a hybrid DataFrame - it supports both numpy and arrow data types. This page outlines exactly which data types are supported in Vaex, and which we hope to support in the future. We also provide some tips on how to approach common data structures.

For some additional insight, you are welcome to look at this post as well.

## Supported Data Types in Vaex¶

In the table below we define:

• Supported: a column or expression of that type can exist and can be stored in at least one file format;

• Unsupported: a column or expression of that type can currently not live within a Vaex DataFrame, but can supported be added in the future;

• Will not support: This datatype will not be supported in Vaex going forward.

Framework

Dtype

Supported

Remarks

Python

int

yes

Will be converted to a numpy array

Python

float

yes

Will be converted to a numpy array

Python

datetime

not yet

Python

timedelta

not yet

Python

str

yes

Will be converted to Arrow array

numpy

int8

yes

numpy

int16

yes

numpy

int32

yes

numpy

int64

yes

numpy

float16

yes

Operations should be upcast to float64

numpy

float32

yes

numpy

float64

yes

numpy

datetime64

yes

numpy

timedelta64

yes

numpy

object ('O')

no

arrow

int8

yes

arrow

int16

yes

arrow

int32

yes

arrow

int64

yes

arrow

float16

yes

Operations should be upcast to float64

arrow

float32

yes

arrow

float64

yes

arrow

date32

yes

arrow

time64

yes

arrow

time32

yes

arrow

duration

yes

arrow

struct

yes

Can’t be exported to HDF5 yet, but possible

arrow

dictionary

yes

arrow

union

not yet

## General advice on data types in Vaex¶

Vaex requires that each column or expression be of a single data type, as in the case of databases. Having a column of different data type can result in a data type object, which is not supported, and can also give raise to various problems.

The general advice is to prepare your data to have a uniform data type per column prior to using Vaex with it.

[1]:

import vaex
import numpy as np
import pyarrow as pa


## Higher dimensional arrays¶

Vaex support high dimensional numpy arrays. The one requirement the arrays must have the same shape. Currently DataFrames that contain higher dimensional numpy arrays can only be exported to HDF5. We hope that arrow will add support for this soon, so we can export such columns to the arrow and parquet formats also.

For example:

[2]:

x = np.random.randn(100, 10, 10)
df = vaex.from_arrays(x=x)
df

[2]:

# x
0 'array([[ 1.83097431e+00, -9.90736404e-01, -8.85...
1 'array([[ 1.99466370e+00, 8.92569841e-01, 2.11...
2 'array([[-0.69977757, 0.61319317, 0.01313954, ...
3 'array([[ 0.25304255, -0.84425097, -1.18806199, ...
4 'array([[ 0.31611316, -1.36148251, 1.67342284, ...
... ...
95'array([[-0.60892972, -0.52389881, -0.92493729, ...
96'array([[ 1.10435809, 1.06875633, 1.45812865, ...
97'array([[-0.59311765, 0.10650056, -0.14413671, ...
98'array([[-0.24467361, -0.40743024, 0.6914302 , ...
99'array([[-1.0646038 , 0.53975242, -1.70715565, ...

We can also pass a nested list of lists structure to Vaex. This will be converted on the fly to a numpy ndarray. As before, the condition is that the resulting ndarray must be regular.

For example:

[3]:

x = [[1, 2], [3, 4]]
df = vaex.from_arrays(x=x)
df

[3]:

# x
0array([1, 2])
1array([3, 4])

If we happen to have a non-regular list of lists, i.e. a list of lists where the inner lists are of different lengths, we first need to convert it to an arrow array before passing it to vaex:

[4]:

x = [[1, 2, 3, 4, 5], [6, 7], [8, 9, 10]]
x = pa.array(x)
df = vaex.from_arrays(x=x)
df

[4]:

# x
0[1, 2, 3, 4, 5]
1[6, 7]
2[8, 9, 10]

Note the arrow structs and lists can not be exported to HDF5 for the time being.

## String support in Vaex¶

Vaex uses arrow under the hood to work with strings. Any strings passed to a Vaex DataFrame will be converted to an arrow type.

For example:

[5]:

x = ['This', 'is', 'a', 'string', 'column']
y = np.array(['This', 'is', 'one', 'also', None])

df = vaex.from_arrays(x=x, y=y)
print(df)

display(df.x.values)
display(df.y.values)

  #  x       y
0  This    This
1  is      is
2  a       one
3  string  also
4  column  --

<pyarrow.lib.StringArray object at 0x7f277b9b9040>
[
"This",
"is",
"a",
"string",
"column"
]

<pyarrow.lib.StringArray object at 0x7f277b9b9d60>
[
"This",
"is",
"one",
"also",
null
]


It is useful to know that string operations in Vaex also work on lists of lists of strings (and also on lists of lists of lists of strings, and so on).

[6]:

x = pa.array([['Reggie', 'Miller'], ['Michael', 'Jordan']])
df = vaex.from_arrays(x=x)
df.x.str.lower()

[6]:

Expression = str_lower(x)
Length: 2 dtype: list<item: string> (expression)
------------------------------------------------
0   ['reggie', 'miller']
1  ['michael', 'jordan']