Distributing OCaml

Act One

Set the scene: Me

• Functional background? 20 years ago!

• Deja vu. Worked (briefly) on Windows side of CHICKEN Scheme

• Languages I used were a snapshot of the dominant languages …

Set the scene: Me

• 2000s: Fraud and credit analytics.

• Lots of data and security. Lots of machine learning. Telecom, credit cards, health insurance.

• C and Java. Solaris, z/OS and Linux servers. Windows just for GUIs.

Set the scene: Me

• 2010s: Amazon. Fraud, data warehousing and computer vision.

• Lots of data and security. Lots of machine and deep learning.

• Java, C, JS and Python. Linux servers and embedded hardware. Even less Windows b/c GUIs have become web apps.

Set the scene: Diskuv

• 2020. Proud of what I did at Amazon. But left Amazon to start Diskuv shortly after George Floyd.

• Goal was a suite of personal safety products; hardware and software. Fit with data, security and computer vision background.

• Most of my product designs had common element: communication between people. So first product was “chat”.

Set the scene: Diskuv

• Heavily forked version of Signal Server, with my own backend servers.

• Problem 1: Signal had not released their “open source” server code for a year while they were integrating the cryptocurrency “Mobile Coin”.

• Lesson: Saying “open-source” is not enough. Have to be prepared to not only release the code but make it runnable by end-users.

Set the scene: Diskuv

• Heavily forked version of Signal Server, with my own backend servers.

• Problem 2: Forking is expensive/slow. Developed rule-based code transformation tools in Python.

• Lesson: Avoid forks at all costs.

Set the scene: Diskuv

• Heavily forked version of Signal Server, with my own backend servers.

• Problem 3: Front-end code was lock heavy. And caused messages to be lost.

• Deep dive …

Set the scene - Locking

My bug report:

The bug is fairly general: I've waited for up to an hour waiting for
messages to be received. I've seen other people report similar things.
If I switch apps (ex. switch to a game, and then switch back to Signal)
my messages start going through.

Set the scene - Locking

On your phone

1. Go through the “registration” flow

2. Send a message “hi” to another Signal user (“other”)

Set the scene - Locking

On other phone

3. Accept the message sender

4. Reply to the message with “Hi”

Set the scene - Locking

On your phone

5. Wait for 1+ hrs

6. App in background then in foreground. Sometimes get message.

7. Close app and re-open, immediately get undelivered messages.

Set the scene - Locking

How would you know that you aren’t getting your message?

If you switch to a web page, and then back to your app, your message is there!

How do you support your product in the field?

Set the scene - Locking

My five-line hacky solution:

TextSecurePreferences.setWebsocketRegistered(context, true);
// ...
TextSecurePreferences.setPushRegistered(context, true);
// ...
IncomingMessageObserver incomingMessageObserver =
  ApplicationDependencies.getIncomingMessageObserver();
synchronized (incomingMessageObserver) {
  incomingMessageObserver.notifyAll();
}

Set the scene - Locking

My five-line hacky solution:

TextSecurePreferences.setWebsocketRegistered(context, true);
// ...
TextSecurePreferences.setPushRegistered(context, true);
// ...
/*
  CONTRASTS with how difficult it is

  to find locking problems!
*/

Set the scene - Locking

Root cause? Code did not guarantee that Object.notify sent when any of 5 conditions changed:

private synchronized boolean isConnectionNecessary() {
  boolean registered          = isPushRegistered(context);
  boolean websocketRegistered = isWebsocketRegistered(context);
  boolean isGcmDisabled       = isFcmDisabled(context);
  boolean hasNetwork          = isMet(context);
  boolean hasProxy            = isProxyEnabled();

  return registered                    && websocketRegistered &&
         (appVisible || isGcmDisabled) && hasNetwork          &&
         !networkAccess.isCensored(context);
}

private synchronized void waitForConnectionNecessary() {
  while (!isConnectionNecessary()) wait();
}

Set the scene - Locking

• I forked because I was introducing new behaviors to the Signal mobile application

• But new behaviors still need to interact with old behaviors, which means acquiring the same set of locks …

• But still guard against lock mis-ordering problems

• … means acquire your own locks in event-driven UI

• Nuclear arms race to ensure safe lock usage!

Set the scene - Locking

Lesson:

• Avoid locks at all costs. Prefer cooperative threading.

Set the scene - Alpha Launch

• Solved the problems, but limped to the starting line

• 6 months into company, demo of chat available! Ideal startup timeline.

• A couple months of Alpha testing with an invite-only app from the Google Play Store; getting good feedback

• But just before a public Beta launch, we enter Act Two

Act Two

Enter Google

From: Google Play Support <no-reply-googleplay-developer@google.com>
Sent: Sunday, May 9, 2021 5:39 PM
Subject: Notification from Google Play

This is a notification that your Google Play Publisher account has
been terminated.
REASON FOR TERMINATION: Prior violations of the Developer Program
Policies and Developer Distribution Agreement by this or associated
accounts as outlined in previous emails sent to the registered email
address(es) of the Publisher account(s).
...
If you've reviewed the policy and feel this termination may have been
in error, please reach out to our policy support team.
... We will not be restoring your account at this time.
The Google Play Team

Enter Google - Reason?

From: Google Play Support <no-reply-googleplay-developer@google.com>
Sent: Sunday, May 9, 2021 5:39 PM
Subject: Notification from Google Play

This is a notification that your Google Play Publisher account has
been terminated.
REASON FOR TERMINATION: Prior violations of the Developer Program
Policies and Developer Distribution Agreement by this or associated
accounts as outlined in previous emails sent to the registered email
address(es) of the Publisher account(s).
...
If you've reviewed the policy and feel this termination may have been
in error, please reach out to our policy support team.
... We will not be restoring your account at this time.
The Google Play Team

Enter Google - Appeal

From: Google Play Support <no-reply-googleplay-developer@google.com>
Sent: Sunday, May 9, 2021 5:39 PM
Subject: Notification from Google Play

This is a notification that your Google Play Publisher account has
been terminated.
REASON FOR TERMINATION: Prior violations of the Developer Program
Policies and Developer Distribution Agreement by this or associated
accounts as outlined in previous emails sent to the registered email
address(es) of the Publisher account(s).
...
If you've reviewed the policy and feel this termination may have been
in error, please reach out to our policy support team.
... We will not be restoring your account at this time.
The Google Play Team

Enter Google - Appeal

I have every reason to believe the app is in full compliance with the
policies and the agreement. But I've received no email indicating what the
policy violation was. Instead I received a "Prior violations of the
Developer Program Policies and Developer Distribution Agreement by this
or associated accounts as outlined in previous emails sent to the
registered email address(es) of the Publisher account(s)." However I have
received no "prior violations" in any "previous emails".

Enter Google - Appeal

From: googleplay-developer-support@google.com <googleplay-developer-support@google.com>
Sent: Wednesday, June 9, 2021 6:37 AM
Subject: Re: Your message about Google Play [5-XXXXXXXXXXXXX]

Hi Developers at Diskuv,
Thanks for contacting Google Play Team.

Due to adjusted work schedules at this time, you may experience longer
than usual process times for your appeal. We appreciate your patience
and understanding.

After further review, we've accepted your appeal and reinstated your
account. You'll need to sign in to your Play Console to modify and/or
republish any reinstated apps to make them available on Google Play.
...

Enter Google - Appeal

From: googleplay-developer-support@google.com <googleplay-developer-support@google.com>
Sent: Wednesday, June 9, 2021 6:37 AM
Subject: Re: Your message about Google Play [5-XXXXXXXXXXXXX]

...
If the option to resubmit is not available, please try making a small
change to your app's Store Listing page. For example, you can add and
remove a space at the end of your app description.

Before republishing your apps, you might want to review the Developer
Program Policies for additional policy guidance.
**Kindly keep in mind that any new policy violations may result in your
account's permanent termination from Google Play.**

Please let me know if you have any other questions.

Regards,
(someone's name)
The Google Play Team

Enter OCaml

Never again requirements:

• When writing a mobile application, must be easy to switch to a web application if a vendor messes with you

Enter OCaml

Personal requirements:

• Adding features like a new thread must not make existing code less safe or less understandable

• Minimize time it takes to deploy to embedded devices, desktops and iOS

• Make it as easy as possible to prove the software I develop is safe

Enter OCaml

Never again requirements:

• “When writing a mobile application, must be easy to switch to a web application if a vendor messes with you”

  Not many languages fit this. Highest rank first:

  1. JavaScript

  2. JavaScript code from OCaml (jsoo) backend

  3. WebAssembly code from Rust and C

Enter OCaml

Personal requirements:

• “Adding features like a new thread must not make existing code less safe or less understandable”

  Languages that supported cooperative threading were a good fit. Generalization hard to measure; stopped at cooperative threads as proxy.

  1. JavaScript and OCaml

  2. Rust (perceived difficulty of async, single-threaded tokio)

  3. C

Enter OCaml

Personal requirements:

• “Minimize time it takes to deploy to embedded devices, desktops and iOS”

  Hard to evaluate! Short-term time sinks (least first):

  1. Rust and C (large ecosystems; run anywhere)

  2. OCaml (learn new language) and JS (does not run everywhere)

Enter OCaml

Personal requirements:

• “Minimize time it takes to deploy to embedded devices, desktops and iOS”

Long-term time sinks (least time sink first):

1. C

2. JavaScript (interop mobile + embedded)

3. Rust (perceived drag on productivity; followup)

4. OCaml (hiring with low supply)

Enter OCaml

Rust Lang Roadmap for 2024 (written in 2022)

Companies building large teams of Rust users report that the typical onboarding time for a Rust engineer is around 3-6 months. Once folks learn Rust, they typically love it. Even so, many people report a sense of high “cognitive overhead” in using it, and “learning curve” remains the most common reason not to use Rust.

Enter OCaml

Personal requirements:

• “Make it as easy as possible to prove the software I develop is safe”

  Easy to prove first:

  1. OCaml (best-in-class with Coq, memory safe)

  2. Rust (memory and resource safety)

  3. JavaScript (memory safe)

  4. C

Enter OCaml

Not easy. No clear winner!

Enter OCaml

OCaml is chosen winner, but only with large risk

OCaml Risks

• Long Term: How to deal with low supply of OCaml-ers.

• Answer: Make OCaml easy to adopt and be productive for newcomers.

• Windows: Poor support today for Windows

• Answer: From my only interaction with OCaml (compiling Unison) knew that support existed for Windows.

OCaml Risks

• Long Term: How to deal with low supply of OCaml-ers.

• Windows: Poor support today for Windows

• Strategies?

  • Deal with top risks first.

  • Fund the cross-vendor/cross-platform development by releasing tools for peers who have similar vendor risks.

OCaml Risks

• Low Supply + Windows risks ⇒ Building OCaml projects on Windows should be easy for within my company and any contractors I hire.

• End goal: Open project, run a Windows script to install OCaml, edit OCaml source code, and press Build.

• For employees and contractors only.

Windows Issue #1

• The dominant compiler is Microsoft Visual Studio.

• OCaml on Windows used GCC.

• GCC and MSVC are often compatible, but sometimes not.

• They are different ABIs, especially wrt to DLLs.

• Rust: x86_64-pc-windows-msvc and x86_64-pc-windows-gnu

Windows Issue #1

libuv is typical Tier 3 support for GCC (MinGW): Node.js eventing library.

Windows Issue #1

Windows Issue #2

• Unix compatibility: MSYS2

• Pervasive UNIX commands in opam packages - conventional package manager for OCaml

Windows Issue #2

ocaml-base-compiler.4.14.0 package installs the OCaml compiler, but assumes there is a /bin/sh to run ./configure:

build: [
  [
    "./configure"           # <--- assumes /bin/sh
    "--prefix=%{prefix}%"
    "--docdir=%{doc}%/ocaml"
    "-C"
    "CC=cc" {os = "openbsd" | os = "macos"}
    "ASPP=cc -c" {os = "openbsd" | os = "macos"}
  ]
  [make "-j%{jobs}%"]
]

Windows Issue #1 and #2

Created with-dkml that proxies to a UNIX executable.

UNIX binaries in the PATH + MSVC compiler variables

Final                           Original
C:/Users/me/AppData/Local/Programs/DISKUV~1/bin
├── dune-real.exe       =       dune.exe
├── dune.exe            =       with-dkml.exe
├── opam-real.exe       =       opam.exe
├── opam.exe            =       with-dkml.exe
└── with-dkml.exe       =       with-dkml.exe

Windows Issue #3

• Sole maintainer of the only Windows distribution said they were discontinuing Windows

• Good time to release my Windows scripts publicly

• But … self-interest disclosure … just enough to make sure the Windows support did not die!

Enter OCSF

• OCaml Software Foundation approached me to see if I would continue supporting Windows

• My existing support was making sure OCaml on Windows would not die

• Implicit ask was to make sure Windows would thrive: others could contribute, and it be promoted

Enter OCSF

• Decision was whether extra effort on Windows made sense

• My perspective: change in priority so work I planned over the next three years could be brought in much sooner

• But also need clear lines between open-source and commercial …

Enter OCSF

Clear lines between open-source “DkML” and commercial “DkSDK”

Enter OCSF

• Agreed + thanks to OCSF. 1st project was single-click Windows installer

• Dogfooding caught bugs early; used DkML installed environment to develop my own chat software and DkSDK

Enter OCSF

• Biggest drawback: Slow! 2-cpu VM takes 90 minutes.

• Contrast with macOS M1 MacVM: 15 minutes

• Hardware is unequal, and Mac does not need to install Unix

• Observations from GitHub/GitLab CI: Windows compiles are 2X slower

Enter OCSF

• Biggest advantage: 5-7 clicks and 11 characters (y, ENTER, WinR, cmd, ENTER, utop, ENTER)

• Contrast to macOS: 4 non-interactive commands, 1 interactive command. Lots of reading, and common beginner pain with “switches” (analog to Python virtual environments)

Enter OCSF

Enter OCSF

• Biggest issues caught in the wild:

  • Flaky downloads

  • Unicode

  • Spaces in directories

Act Three: Happily ever after!

Status of DkML

• Biggest missing features?

  • More functionality without using switch (venv)

  • Lite “bytecode” installer without Visual Studio

Status of OCaml

• No OCaml 5 + MSVC. Could take a year for MSVC.

• Today is recommended Windows distribution, but will switch rapidly when opam 2.2 is released

• Testing Windows is not automatic, but is changing

• Deeply entrenched Unix-ism may be impossible to dislodge:

  • Makefiles not supporting spaces

  • Many many packages require pkg-config. And it is implicitly encouraged as default C resolver in dune

Status of OCaml

But it is quite reasonable to develop OCaml applications on Windows, for Windows.

My routine:

• One week macOS

• One week Windows (occasionally with Linux Docker or WSL2)

DkSDK and chat are the OCaml products I can develop on Windows …

Status on DkSDK

DkSDK: Development environment and libraries so you can manage your technology / vendor risk

• I’m building a chat app. I need it for my company’s product line, but chat is commodity; source code will be available when it is ready.

• Actors … stateful, distributed objects that communicate with message passing … is a common way to build a chat backend.

Status on DkSDK

DkSDK: Development environment and libraries so you can manage your technology / vendor risk.

I don’t foresee needing complexity of the big players: Erlang/OTP and Akka. But I might need it. Both are true:

• I don’t want to take the risk of writing large amounts of code and not being able to adopt a full-feature actor framework

• I don’t want to take the risk of committing to tech + vendor

Status on DkSDK

More important, I’ve built large, reliable compute systems on top of distributed memory (ex. Redis).

My hunch is others are fairly comfortable with Redis as well, and don’t need the reliability and deployment guarantees that come with the more complex frameworks.

Status on DkSDK

The project DkHelloWorldActor uses libraries that are “real” (from chat app) but top-level code is just a simple example from Akka.

Status on DkSDK

Since Redis is written in C, most of my development has been in CLion.

CLion - run actors-cli

Status on DkSDK

• Uses light weight threads in OCaml as the actor coordinator. OCaml-ers? Drop-downs and buttons to run OCaml code.

• Output is similar to what Akka has for its example:

  [05.814] [hello-akka.actor.dd-4] [akka://hello/user/greeter] Hello World!
  [05.815] [hello-akka.actor.dd-4] [akka://hello/user/greeter] Hello Akka!
  [05.815] [hello-akka.actor.dd-2] [akka://hello/user/World] Greeting 1 for World
  [05.815] [hello-akka.actor.dd-4] [akka://hello/user/Akka] Greeting 1 for Akka
  [05.815] [hello-akka.actor.dd-5] [akka://hello/user/greeter] Hello World!
  [05.815] [hello-akka.actor.dd-5] [akka://hello/user/greeter] Hello Akka!
  [05.815] [hello-akka.actor.dd-4] [akka://hello/user/World] Greeting 2 for World
  [05.815] [hello-akka.actor.dd-5] [akka://hello/user/greeter] Hello World!
  [05.815] [hello-akka.actor.dd-4] [akka://hello/user/Akka] Greeting 2 for Akka
  [05.816] [hello-akka.actor.dd-5] [akka://hello/user/greeter] Hello Akka!
  [05.816] [hello-akka.actor.dd-4] [akka://hello/user/World] Greeting 3 for World
  [05.816] [hello-akka.actor.dd-6] [akka://hello/user/Akka] Greeting 3 for Akka

Status on DkSDK

Really important point. The OCaml / C code is not tied to CLion.

Load it and run into another IDE (Visual Studio Code)

Status on DkSDK

On Monday April 24, 2023 will release the CMake/IDE build system. Not yet the Actor framework, but you can do C + OCaml + CMake on several targets:

https://gitlab.com/diskuv/DkHelloWorldActor/-/pipelines/819738886

Status on DkSDK

Enablers:

• You can do your normal development in Visual Studio, Xcode, Android Studio or Visual Studio Code.

Status on DkSDK

Enablers:

• Your SwiftUI (iOS) or JetPack (Android) contractor can build your mobile app … they copy a few lines of code into their projects … and press Build.

• Need a new field from the backend populated on the frontend? They can copy and paste a line, press Build, and continue front-end work.

• No concern for “do they know OCaml / Akka / Erlang / …”

Status on DkSDK

Who for?

• People who are pinched for tech resources.

• Startups, IT service orgs.

• Can visit https://diskuv.com and click on DkSDK to see documentation for how you would interact with it.

• Will be available Monday, April 26 with more complete docs

Status on Hiring

• DkSDK is mix of C and OCaml.

• Can choose (and did) for first FTE to be C programmer.

• Next (not hiring right now) will be front-end or an OCaml expert.

• “Risky” additions were high-school OCaml trainees.

Status on Hiring

Two purposes getting high school OCaml trainees:

1. Want someone to work on OCaml stuff until we are at point where we need a genuine OCaml expert.

2. Want to demonstrate to potential DkSDK customers that adopting OCaml is not going to break your team. If high school students with AP CS can train in ~150 hours and be productive, likely your team can too. Also, putting money where your mouth is good.

That second point justified the risk.

Status on Hiring

Expectations?

• Differences from college internship: High schoolers have far far less hours during school, and parents are in control of hours.

• College interns have experience doing large projects. Cannot expect a high school intern to break down large tasks.

• Default expectation is not to work during high school, even in summer.

Status on Hiring

Training Structure

• Caution: Minors! No specifics (what, why)

• Varied goals/situations; 2 interns and 2 contributors

• Trained for ~150 hours. Own rate dependent on course load and extracurriculars, and get tested.

• Goal for testing is to demonstrate they can repeatedly do 1 hour coding tasks correctly given 5 minutes of instruction; tasks same difficulty as a college intern. Expectation is during summer 1 hour grows to 1 day.

Status on Hiring

• One of four has completed hours and assessment. Getting a summer offer.

• Remaining three’s extracurriculars have eaten time (often down to 0 hours in a week). But very very confident most if not all of them will be passing the final test, and will work on a project or a return internship in the summer.

• Bottom line: Very proud of all my student trainees. Huge success