io.warp10:warp10-ext-arrow

Installation

$ wf g -w /path/to/warp10 io.warp10 warp10-ext-arrow

/path/to/warp10/ is where Warp 10 is physically located.

Readme

Arrow extension for WarpScript

WarpScript functions for conversions from and to Apache Arrow columnar format.

From WarpScript to Arrow

->ARROW    // Encode input into Arrow columnar format (BYTES).

The supported input types are described in the conversion table below. The output is a BYTES representation of an Arrow table, containing a map of metadata and columns. The arrow metadata will contain the fields WarpScriptConversionMode, WarpScriptVersion and WarpScriptTimeUnitsPerSecond.

Input descriptionOutput descriptionMetadata descriptionWarpScriptConversionMode
List of GTS encoders or GTSone column for classname, one per label key, one for timestamp, one for latitude, one for longitude, one for elevation, one per value typeno additional output metadataENCODERS
A GTSone column for timestamp, one for latitude, one for longitude, one for elevation, one for valueoutput metadata includes full GTS metadataGTS
A pair list containing a map of metadata and a map of lists of equal sizeone column per listthe input map of metadata is the output metadataPAIR
A WarpFrame (if using WarpFrame extension)one column for timestamp, one column per GTSeach column is named with GTS classname and has GTS label metadataWARPFRAME

Empty columns are not encoded.

From Arrow to WarpScript

ARROW->    // Decode an Arrow stream (BYTES).

The function will try to infer the type of the result using the value of the metadata WarpScriptConversionMode, based on the conversion table above. If the input has no WarpScriptConversionMode, it will use the default WarpScriptConversionMode PAIR.

NOTE

Streaming Arrow format from memory or memory mapped files is not available yet. Contact us if you need this feature. SHM extension can be used as a workaround.

We made a blog article to present this extension: https://blog.senx.io/conversions-to-apache-arrow-format/

Build notes

Depends on rev >= 2.2.0

Author
Avatar

Jean-Charles Vialatte

jean-charles.vialatte@senx.io

Version

2.0.2-uberjar

Type

Extension

Last published

2020-11-06

License

Apache-2.0

Downloads
Repository

Github

Tags
arrow
Path Size Creation time
  ->ARROW 616 bytes 2020-11-06
  ARROW-> 1065 bytes 2020-11-06

Configuration

warpscript.extension.arrow = io.warp10.arrow.ArrowExtension
Versions Published
2.0.2-uberjar 2020-11-06
2.0.1-uberjar 2020-11-06
1.0.10-uberjar 2020-10-07
1.0.8-uberjar 2020-09-25
1.0.7-uberjar 2020-09-23
1.0.2-uberjar 2020-09-23
1.0.1-uberjar 2020-09-22
0.1.14-uberjar 2020-04-29
0.1.12-uberjar 2019-11-29
0.1.11-uberjar 2019-11-27
0.0.10-uberjar 2019-11-27
0.0.9-uberjar 2019-11-26
0.0.8-uberjar 2019-11-15
0.0.7 2019-11-15
0.0.6-uberjar 2019-11-15
0.0.5-uberjar 2019-11-15
0.0.4-uberjar 2019-11-15