WarpFleet

Introduction

WarpFleet aims to facilitate Warp 10™'s add-on sharing.

Security

To fulfill this goal, all authors contributing to WarpFleet must first register with the WarpFleet service via the filling of a form which aims at gathering some information about the authors, among which:

The WarpFleet team reserves the right to postpone or refuse any registration request.

Upon successful registration, the registering party will be assigned a group under which they will be able to publish artifacts.

Any publication of code will rely on PGP signature verification

Getting started

First be sure to have NodeJS v10+ installed.

$ npm install -g @senx/warpfleet
$ wf
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
Usage: wf [options] [command]

WarpFleet Command Line Interface

Options:
  -v, --version                                   output the version number
  -d, --debug                                     Debug mode
  -h, --help                                      output usage information

Commands:
  init|i [options]                                  Initialize a new Project
  list|l [group] [artifact]                         List available artifact and/or versions
  info   [version]                 Get information about a specific artifact
  install [options] [wfFile]                        Install your dependencies into your Warp 10
  get|g [options]  [artifact] [version]      Retrieve the packed macros, plugin or extension archive. If no version is specified, the most recent one is retrieved.
  publish|p [options] [version] [path] [repoUrl]    Publishes a plugin, macro or extension against WarpFleet
  unpublish|u [options] [version] [path] [repoUrl]  Unpublishes a plugin, macro or extension against WarpFleet

Create a sample extension:

$ wf i
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
? What do you want to develop? (Use arrow keys)
❯ Extension
  Macro package
  Plugin

Choose extension for example.

Some indications:

? What do you want to develop? Extension
? Enter project's group: io.warp10.ext.example
? Enter assets's name: my-awesome-module
? Enter assets's version: 0.0.1
? Enter project's description: A great extension for Warp 10
? Enter author's name: Your Name
? Enter email address: your.email@server.com
? Enter license: Whatever
? Enter GIT repo url: https://github.com/pseudo/my-awesome-module
? Enter Maven repo url root: https://dl.bintray.com/myCompany/maven/
? Coma separated list of tags
? Is all OK?
{
    "type": "ext",
    "group": "io.warp10.ext.example",
    "artifact": "my-awesome-module",
    "version": "0.0.1",
    "description": "A great extension for Warp 10",
    "author": "Your Name",
    "email": "your.email@server.com",
    "license": "Whatever",
    "git": "https://github.com/pseudo/my-awesome-module",
    "tags": []
} (Y/n)

Confirm. You can change everything later. It is stored in src/main/resources/wf.json.

? Is all OK?
{
    "type": "ext",
    "group": "io.warp10.ext.example",
    "artifact": "my-awesome-module",
    "version": "0.0.1",
    "description": "A great extension for Warp 10",
    "author": "Your Name",
    "email": "your.email@server.com",
    "license": "Whatever",
    "git": "https://github.com/pseudo/my-awesome-module",
    "repoUrl": "https://dl.bintray.com/myCompany/maven/",
    "tags": []
} Yes
[INFO] Retrieving starter kit
[INFO] Cleaning starter kit
[INFO] Generating artifacts
[INFO] Generating WarpFleet files
? my-awesome-module/README.md already exists, do you want to overwrite it? Yes
[SUCCESS] Successful
[SUCCESS] Asset generated

You now have a bootstrapped Warp 10 extension.

Publishing Process

The publication process involves heavy sanity checking of the published asset in order to ensure it reaches the quality standards that WarpFleet users can expect. As a consequence, an asset publication might be refused if such standards are not met. Should this be the case, the WarpFleet publication mechanism will do its best to expose the reasons of the rejection to the asset author.

First, you need to create a group, like io.warp10.ext.example in our sample. You will be the owner of the group

To create the group, generate a GPG key and export the public key:

$ gpg --full-generate-key
$ gpg --list-secret-keys --keyid-format LONG
/Users/picard/.gnupg/secring.gpg
------------------------------------
sec   4096R/3AA5C34371567BD2 2016-03-10 [expires: 2017-03-10]
uid                          J.L. Picard
ssb   4096R/42B317FD4BA89E7A 2016-03-10

$ gpg --export 3AA5C34371567BD2 > io.warp10.ext.example.gpg

Contact us in order to create the group onto WarpFleet. We will verify the uniqueness of the group and add your GPG key to it.

To publish on WarpFleet, you must tag your git repo with the same tag as the Maven repo. When it's done, you can publish your jar to Maven or Bintray:

$ git tag -a 0.0.4 -m "new version"
$ ./gradlew clean && ./gradlew -Duberjar shadowJar && ./gradlew -Duberjar bintrayUpload
$ wf p

Cookbook

Generate a wf.json into an existing project

$ wf i -w

Browse repo

$ wf l
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
> io.warp10
$ wf l io.warp10
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
> warp10-ext-barcode
 [ 0.1.0 ]
> warp10-ext-jdbc
 [ 0.1.0 ]
> warp10-ext-pcap
 [ 0.1.0 ]
> warp10-ext-pmml
 [ 0.2.0, 0.2.0-uberjar, 0.1.0 ]
> warp10-ext-polyglot
 [ 0.1.0 ]
> warp10-ext-s3
 [ 0.2.0-uberjar, 0.1.0 ]
> warp10-ext-tensorflow
 [ 0.1.0 ]
> warp10-plugin-mqtt
 [ 0.1.0, 0.1.0-uberjar ]
> warp10-plugin-zeppelin
 [ 1.0.1 ]

Retrieve an artifact

$ wf g io.warp10 --conf=/path/to/warp10/etc/my-conf-standalone.conf --macroDir=/path/to/warp10/macros/ --libDir=/path/to/warp10/libs
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
? Which asset do you want to retrieve? (Use arrow keys)
❯ warp10-ext-barcode
  warp10-ext-jdbc
  warp10-ext-pcap
  warp10-ext-pmml
  warp10-ext-polyglot
  warp10-ext-s3
  warp10-ext-tensorflow
(Move up and down to reveal more choices)

Since Warp 10™ 2.1.0, it supports multi-files configuration, so you can use:

$ wf g io.warp10 --confDir=/path/to/warp10/etc --macroDir=/path/to/warp10/macros/ --libDir=/path/to/warp10/libs

Retrieve a group of artifacts

In your Warp 10™ installation directory, create a file named wf.json:

{
  "dependencies": [
    "io.warp10:warp10-ext-barcode:0.2.0",
    "io.warp10:warp10-ext-jdbc",
    "io.warp10:warp10-ext-pmml:latest",
    "io.warp10:warp10-ext-s3:+"
  ]
}

The syntax is : <group>:<artifact>:<version>

If you want the latest version of the artifact, you can avoid to precise the version, or use + or latest.

wf install will ask you if it automatically patch your Warp 10™ configuration in order to activate extensions or plugins. It will also deploy artifacts and macros into right directories.

$ wf install --conf=/path/to/warp10/etc/my-conf-standalone.conf  --macroDir=/path/to/warp10/macros/ --libDir=/path/to/warp10/libs
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
    ✔ ext io.warp10:warp10-ext-pmml#0.2.0 retrieved
    ✔ ext io.warp10:warp10-ext-jdbc#0.1.0 retrieved
    ✔ ext io.warp10:warp10-ext-barcode#0.2.0 retrieved
    ✔ ext io.warp10:warp10-ext-s3#0.2.0-uberjar retrieved
    ✔ Download successful: gradle-wrapper.jar
    ✔ Download successful: gradlew
    ✔ Download successful: gradlew.bat
    ✔ Dependency commons-math3-3.6.1.jar successfully deployed
    ✔ Dependency core-3.3.3.jar successfully deployed
    ✔ Dependency guava-27.0-jre.jar successfully deployed
    ✔ Dependency pmml-agent-1.4.8.jar successfully deployed
    ✔ Dependency pmml-evaluator-1.4.6.jar successfully deployed
    ✔ Dependency pmml-model-1.4.8.jar successfully deployed
    ✔ Dependency warp10-ext-barcode-0.2.0.jar successfully deployed
    ✔ Dependency warp10-ext-jdbc-0.1.0.jar successfully deployed
    ✔ Dependency warp10-ext-pmml-0.2.0.jar successfully deployed
    ✔ Dependency warp10-ext-s3-0.2.0-uberjar.jar successfully deployed
    ✔ Please add this conf entries to /opt/warp10-1.2.11-rc4/etc/conf-standalone.conf
    warpscript.extension.jdbc=o.warp10.script.ext.jdbc.JDBCWarpScriptExtension
    ✔ Done

Since Warp 10™ 2.1.0, it supports multi-files configuration, so you can use:

$ wf install io.warp10 --confDir=/path/to/warp10/etc --macroDir=/path/to/warp10/macros/ --libDir=/path/to/warp10/libs

Get info about an artifact

$ wf info io.warp10 warp10-ext-pmml
       ___       __                     _______________         _____
       __ |     / /_____ __________________  ____/__  /___________  /_
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/
                                /_/                                        ™
✔ done
> type: "ext"
> group: "io.warp10"
> artifact: "warp10-ext-pmml"
> version: "0.2.0"
> description: "WarpScript™ PMML Extension"
> author: "SenX"
> email: "contact@senx.io"
> license: "AGPL-V3"
> git: "https://github.com/senx/warp10-ext-pmml"
> tags: ["WarpScript","Warp 10","ML","IA","PMML"]
> repoUrl: "https://dl.bintray.com/senx/maven/"
> dependencies: [{"group":"org.jpmml","artifact":"pmml-evaluator","version":"1.4.6","classifier":null}]
> macros: []
> jar: "https://dl.bintray.com/senx/maven//io/warp10/warp10-ext-pmml/0.2.0/warp10-ext-pmml-0.2.0.jar"
> pom: "https://dl.bintray.com/senx/maven//io/warp10/warp10-ext-pmml/0.2.0/warp10-ext-pmml-0.2.0.pom"
> name: "0.2.0"
> size: 4
> file: false
> lastModifiedTime: 1550683089340
> creationTime: 1550683089340
> link: "http://warpfleet.senx.io/api/io.warp10/warp10-ext-pmml/0.2.0"
> functions: []
> readme: "# WarpScript™ PMML Extension"