Genspio

I'm Seb Mondet:

• Now: Software Engineering @ Obsidian Systems, NYC
  • Working on Tezos ꜩ
• But previously: Software Engineering / Dev-Ops at Mount Sinai Health System, NYC
  • Hammer Lab → 3 compose talks ☺.
  • Also institute level: “consultancy.”

• Run big computational pipelines.
  • Servers with WebUIs, databases.
  • HPC scheduling (Torque, YARN, Google Cloud, AWS, …).
• Deal with precious human data.
  • HDFS, (broken) disks, S3, Gcloud Buckets, NFSs.
• Interactive exploration.
  • Direct access for the users (IPython, R, awk | wc, …).

• Need to setup local/cloud/datacenter-ish infrastructure for the lab.
• It's nobody's job.
• Nothing seems there for the “long term.”

→ Make composable tools that allow people to setup/monitor/clean-up their own infrastructure.
(+ it's more fun, + a better use of software people's time)

What's wrong with Unix.execve?

Unix.execv "/usr/bin/apt-get" [| "apt-get";"install"; "-y"; "postgresql" |]

👍 … but then,

let cmd =
  ["apt-get";"install"; "-y"; "postgresql"]
  |> List.map ~f:Filename.quote
  |> String.concat ~sep:" "
in
Unix.execv "/usr/bin/ssh" [| "ssh"; host_info ; cmd |]

Who failed?

Everybody ends-up reading some Stack-overflow answer

It's all strings after all:

gcloud compute create deprecates the already dysfunctional --wait option

sudo in some Debian version erases new lines …

1. Start writing simple combinators.
2. Add more typing info.
3. Hit portability / representation problems.
4. Go full-blown EDSL that compiles to pure POSIX shell.

• Simple, typed EDSL.
• Based-on a “classic” GADT + constructs built on top.
• Two Compilers to POSIX shell:
  • “Main:” Either one-liners, or multi-line scripts.
  • “Slow-flow” one: Output less complex but slower scripts; more portable to buggy shells (MacOSX's bash).
• Couple of Optimization Passes:
  • Classic Object-based Extensible AST Visitor.
  • Some constant propagation & involution elimination.

• Crazy internal representations: handle arbitrary strings.
• Proper fail escaping (trap + kill).
• POSIX-only use of sed, od, …
• Two “string” types: Byte-arrays and C-Strings.
• Focus on readability & testability, not much performance.

In the beginning there was C.

• In “Future Work” for Genspio 0.0.0.
• Two types accessible but only one (with conversions) in module EDSL.
• Optmization pass available.

Tests Generate a directory: Makefile + set of scripts

⇒ easy to tar c | ssh

• Generate (family of) shell scripts + README.md → Git-repo
• COSC: Compose Processes With Screen
  • Manage long running processes in detached Screen session.
  • https://github.com/smondet/cosc
• Multi-git: Scripts that deal with a bunch of git repositories
  • https://github.com/smondet/multi-git

Documentation website built by GitlabCI

• (Re-)builds for a given number of TREE-ISH.
• Re-run the tests.
• Builds the “Genspio Web Demo” for corresponding TREE-ISH.
  • Client-only webapp: js_of_ocaml, TyXML, react
  • OCaml REPL running in a “Webworker.”
  • Runs both compilers, provides “save to file” and “share URI” buttons.
  • https://gitlab.com/smondet/genspio-web-demo

• More library functions
  • SSH client integration.
  • Types over POSIX tools: Awk, Sed, …)
• Tests on MS-Windows.
• Webdemo:
  • Parse & display better error messages.
  • ReasonML “switch.”
  • Examples menu.

Thanks 🙏 — We welcome any contributions ☺