OCaml Planet

February 04, 2016

Dario Teixeira

Library authors: Don't forget the examples!

These are exciting times in the OCaml community. Compiler development is proceeding at a brisk pace, with several long-awaited features on the horizon (flambda, multicore, and modular implicits, just to name a few). The tooling has also improved dramatically in the past few years, making the time before OPAM and Merlin seem like a very distant and best forgotten dark age. Moreover, the community has grown to the point where it is very likely that you will find a library that tackles your particular needs, whatever they may be. Some pain points remain, however. In this post I'll address a particularly frustrating one: the issue of library documentation (or lack thereof). Frustrating not only because it is endemic to our community, but also because it can be mitigated with minimal burden to library authors.

Ideally, a library's documentation would consist of an introductory tutorial garnished with multiple examples, plus the API reference. While the latter is indisputably useful, for libraries with a large surface API it is not enough. In such cases, users faced solely with the API reference are likely to scratch their heads wondering where to begin and how the various pieces fit together. The API reference only becomes useful once users have built a proper mental model of the library, which is a lot easier to do after reading through a tutorial.

Unfortunately, if a project includes any documentation at all, it is very likely that consists solely of the API reference rather than a tutorial. The reason is fairly obvious to any developer: the API reference sits right there in the code (in the mli file, usually), and imposes a minimal burden in creation and maintenance. Writing a tutorial, on the other hand, is a whole extra task that robs time away from the actual coding. It's also not as fun.

There is however a middle ground between writing a full-fledged tutorial and not writing one at all. Moreover, it's a middle ground that imposes minimal inconvenience to developers while providing tremendous value to users. I'm talking about simply including some usage examples with your library.

What makes examples so special? Foremost, a good example provides one of the critical advantages of a tutorial: condensing a large API into a concrete starting point. Consider the case of Cohttp, which offers a relatively large API further complicated by the need to support both Lwt and Async backends. Despite this, getting started with Cohttp is actually fairly easy because it includes a couple of trivial examples. Any OCaml developer can look at the half-dozen lines of each example and immediately build a mental model of how the Cohttp library is structured.

Besides the advantages they bring to users, examples are not very burdensome to developers. For one, they can be derived from private examples used in testing. Moreover, it's fairly trivial to keep examples up-to-date with the latest library API, because in most cases you can rely on the compiler doing the heavy lifting of verifying that the example is still valid. All in all, there is an asymmetry at play: a minimal effort on the developer's part will make a tremendous difference to users. Therefore, my plea to OCaml library developers is to please include examples with your libraries!

Now that (hopefully) I've convinced you to exemplify your libraries, you should also consider a few common sense rules for maximising the effectiveness of examples:

  • Place your examples in a directory named examples. This is the de facto standard in the OCaml community.

  • Start simple, even if the example is dummy. Resist the temptation to demonstrate all the awesome features of your library in a single example.

  • Fully qualify identifiers. In other words, avoid global opens and over-using custom operators. Also, note that parenthesised local opens (the Module.(expr) syntax) are preferable to global opens anyway. And if you insist on showing off the powers of conciseness afforded by your library, please consider having two versions of the same example: one using full qualification, and one without.

  • Order your examples. Suppose you have a sundry collection of examples. Simply number them from the simplest to the more complex. Otherwise, users are faced with a directory full of examples, without knowing where to start. (This is the approach I have chosen for the examples for the Lambdoc library.)

  • Include a Makefile or build instructions for the examples, particularly if the building process is not obvious.

  • Keep your examples up-to-date.

by Dario at February 04, 2016 12:03 PM

February 03, 2016

Functional Jobs

OCaml server-side developer at Ahrefs Research (Full-time)

Who we are

Ahrefs Research is a San Francisco branch of Ahrefs Pte Ltd (Singapore), which runs an internet-scale bot that crawls whole Web 24/7, storing huge volumes of information to be indexed and structured in timely fashion. On top of that Ahrefs is building analytical services for end-users.

Ahrefs Research develops a custom petabyte-scale distributed storage to accommodate all that data coming in at high speed, focusing on performance, robustness and ease of use. Performance-critical low-level part is implemented in C++ on top of a distributed filesystem, while all the coordination logic and communication layer, along with API library exposed to the developer is in OCaml.

We are a small team and strongly believe in better technology leading to better solutions for real-world problems. We worship functional languages and static typing, extensively employ code generation and meta-programming, value code clarity and predictability, constantly seek out to automate repetitive tasks and eliminate boilerplate, guided by DRY and following KISS. If there is any new technology that will make our life easier - no doubt, we'll give it a try. We rely heavily on opensource code (as the only viable way to build maintainable system) and contribute back, see e.g. https://github.com/ahrefs . It goes without saying that our team is all passionate and experienced OCaml programmers, ready to lend a hand or explain that intricate ocamlbuild rule.

Our motto is "first do it, then do it right, then do it better".

What we need

Ahrefs Research is looking for backend developer with deep understanding of operating systems, networks and taste for simple and efficient architectural designs. Our backend is implemented mostly in OCaml and some C++, as such proficiency in OCaml is very much appreciated, otherwise a strong inclination to intensively learn OCaml in a short term will be required. Understanding of functional programming in general and/or experience with other FP languages (F#,Haskell,Scala,Scheme,etc) will help a lot. Knowledge of C++ and/or Rust is a plus.

The candidate will have to deal with the following technologies on the daily basis:

  • networks & distributed systems
  • 4+ petabyte of live data
  • OCaml
  • linux
  • git

The ideal candidate is expected to:

  • Independently deal with and investigate bugs, schedule tasks and dig code
  • Make argumented technical choice and take responsibility for it
  • Understand the whole technology stack at all levels : from network and userspace code to OS internals and hardware
  • Handle full development cycle of a single component, i.e. formalize task, write code and tests, setup and support production (devops)
  • Approach problems with practical mindset and suppress perfectionism when time is a priority

These requirements stem naturally from our approach to development with fast feedback cycle, highly-focused personal areas of responsibility and strong tendency to vertical component splitting.

What you get

We provide:

  • Competitive salary
  • Modern office in San Francisco SOMA (Embarcadero)
  • Informal and thriving atmosphere
  • First-class workplace equipment (hardware, tools)
  • No dress code

Get information on how to apply for this position.

February 03, 2016 07:36 PM

January 30, 2016

Psellos

OCaml on iOS Example Apps

January 30, 2016

Through the decades I’ve slowly realized that my heroes in the music are the bass players. It’s not too late for me to take up some kind of bass instrument, but in the meantime the equivalent in the programming universe is to help provide languages and systems for others to code on top of. Isn’t that so?

image of Portland app image of Gamut app

So, I guess that’s what I’m doing for OCaml on iOS. Most recently I’ve updated the two example apps Portland and Gamut. They’re packaged up to run on iOS devices and in the iOS Simulator, respectively, but of course either can run on either.

If you just want to download them and start them up in Xcode, you can get the sources here:

Portland 3.0.2, tested on iOS 9.2

Gamut 3.0.2, tested on iOS 9.2

I’ve also written out some instructions and principles of operation here:

Portland: Which Way is Up on iOS?

Gamut: Explore Colors in iOS Simulator

If you have any comments, suggestions, or encouragement, leave them below or email me at jeffsco@psellos.com.

Posted by: Jeffrey

<style type="text/css"> .flowaroundimg { float: left; margin: 0em 1em 0em 0em; } </style>

January 30, 2016 07:00 PM

January 26, 2016

OCamlCore Forge News

CCSS 1.6 released

For users, the salient point of this release is the improved support for media queries. Internally, this release features a migration from Ulex to Sedlex and from PCRE to OCaml-re. Complete unit tests based on Alcotest were also added.

by Dario Teixeira at January 26, 2016 05:16 PM

David Mentré

WE programming project: XKCD keyword index

With this post I'm starting a new kind of article: WE programming project. It's objective is to give ideas of programs that could be written over a WE, probably a lot more or lot less depending on your skill and willing to extend the idea.

Here is the first project proposal: an XKCD keyword index.

Objective

You probably known XKCD, "a webcomic of romance, sarcasm, math, and language". If not, I recommend it's reading. :-) All the XKCD comics are freely available, through an URL, e.g. "Asteroid" is available at URL https://imgs.xkcd.com/comics/asteroid.png.

Sometimes, I would like to illustrate a slide or blog item with one of XKCD comics. I know it exists, I saw it but I can no longer find it! So frustrating! So the main idea of this WE programming project is rather simple: make a keyword index of XKCD comics so that one could look for a comics over a theme or idea.

Requirements

Here is some requirements for this project, feel free to adapt to your needs or your own ideas: ;-)

  • It is a web site on the Internet that displays each XKCD comics (with proper reference to original web site) with associated keywords
  • One can search comics on a set of keywords
  • One can associate keywords (a-z- character set, space separated list) to a given comics
  • Already given keywords are suggested
  • No need of account by default, everybody is free to add new keywords (for easy contribution)
  • Moderators (with dedicated account) can make previously added keywords public, can remove keywords
  • No use of database (for easy installation), keywords are saved into a simple file
  • One can download the whole association of keywords to comics (e.g. as a JSON data structure)
  • There is an API to control the web site (for use by others and integration into other sites)

Needed programming technology

Any programming language you want which have a well designed web framework: Ruby with Rails, Python with Django, Javascript with Meteor, OCaml with Ocsigen, ...

If you tackle this project proposal, let me know! ;-)

by David Mentré at January 26, 2016 07:00 AM

January 23, 2016

Psellos

OCaml for iOS Bug Fix Releases

January 23, 2016

If you downloaded one of the OCaml compilers for iOS or the iOS Simulator, please download a new copy. Due to an error in the cross-compile build process, the compilers are looking for ocamlrun in my development tree rather than in the install location. As a result, they work for me but for nobody else. Unfortunately this means I didn’t see the problem in my testing.

Voronoi Image

You can download the updated compilers here:

OCamliOS 4.02.3 for 32-bit iOS (updated Jan 23, 2016)

OCamliOS 4.02.3 for 64-bit iOS (updated Jan 23, 2016)

OCamliOS 4.02.3 for 32-bit iOS Simulator (updated Jan 23, 2016)

OCamliOS 4.02.3 for 64-bit iOS Simulator (updated Jan 23, 2016)

In the big picture I guess this is an amusing type of bug to have. But in the short term it’s a bit of a drag. My apologies to anybody who was inconvenienced.

Thanks to Edgar Aroutiounian for finding the problem.

If you find more problems with the compilers, I’d be happy to hear about them. Or if you have any comments or encouragement, leave them below or email me at jeffsco@psellos.com.

Posted by: Jeffrey

<style type="text/css"> .flowaroundimg { float: left; margin: 0em 1em 0em 0em; } .rightoffloat li { position: relative; left: 1em; } pre { white-space: pre-wrap; width: 96%; margin-bottom: 24px; overflow: hidden; padding: 3px 10px; -webkit-border-radius: 3px; background-color: #fed; border: 1px solid #dcb; } pre code { white-space: pre-wrap; border: none; padding: 0; background-color: transparent; -webkit-border-radius: 0; } td { padding: 0em 1em 0em 1em; } th { padding: 0em 1em 0em 1em; } </style>

January 23, 2016 07:00 PM

January 21, 2016

Coq

Coq 8.5 is out!

The final release of Coq 8.5 is available! The 8.5 version brings several major features to Coq:
  • asynchronous edition of documents under CoqIDE to keep working on a proof while Coq checks the other proofs in the background (by Enrico Tassi);
  • universe polymorphism making it possible to reuse the same definitions at various universe levels (by Matthieu Sozeau);
  • primitive projections improving space and time efficiency of records, and adding eta-conversion for records (by Matthieu Sozeau);
  • a new tactic engine allowing dependent subgoals, fully backtracking tactics, as well as tactics which can consider multiple goals together (by Arnaud Spiwack);
  • a new reduction procedure called native_compute to evaluate terms using the OCaml native compiler, for proofs with large computational steps (by Maxime Dénès).
More information about the changes from 8.4 to 8.5 can be found in the CHANGES file. Feedback and bug reports are extremely welcome. Enjoy!

by Maxime Dénès at January 21, 2016 09:00 PM

January 08, 2016

Richard Jones

Half-baked ideas: C strings with implicit length field

For more half-baked ideas, see the ideas tag.

If you prefer just to see the code, then it’s here.

Chris Siebenmann wrote a couple of interesting articles about C’s null terminated strings and how they pre-date C.

Chris notes an alternative is a length + string representation, as used in Pascal. Although there are libraries for this in C, there are several drawbacks and approximately no one uses them.

However it’s possible to have the best of both worlds: Strings using an implicit length field that takes up no extra storage. These strings are backwards compatible with ordinary C strings — you can literally pass them to legacy functions or cast them to char * — yet the equivalent of a strlen operation is O(1).

There are two ideas here: Firstly, when you use the C malloc function, malloc stashes some extra metadata about your allocation, and with most malloc implementations there is a function to obtain the size of the allocation from a pointer. In glibc, the function is called malloc_usable_size. Note that because of alignment concerns, the amount allocated is usually larger than the amount you originally requested.

The second idea comes from OCaml. OCaml stores strings in a clever internal representation which is both backwards compatible with C (a fancy way to say they are null terminated), and it allows you to get the real length of the string even though OCaml — like C — allocates more than requested for alignment reasons.

So here’s how we do it: When allocating an “implicit length string” (ilenstr) we store extra data in the final byte of the “full” malloced space, in the byte marked B in the diagram below:

+-------------------------+----+------------+----+
| the string              | \0 |   ....     | B  |
+-------------------------+----+------------+----+
<----- malloc we requested ---->
<----------- malloc actually allocated ---------->

If malloc allocated exactly the same amount of space as is used by our string + terminating null, then B is simply the terminating \0:

+-------------------------+----+
| the string              | \0 |
+-------------------------+----+

If malloc allocated 1 spare byte, we store B = 1:

+-------------------------+----+----+
| the string              | \0 | 1  |
+-------------------------+----+----+

If malloc allocated 4 spare bytes, we store B = 4:

+-------------------------+----+----+----+----+----+
| the string              | \0 |   ....       | 4  |
+-------------------------+----+----+----+----+----+

Getting the true length of the string is simply a matter of asking malloc for the allocated length (ie. calling malloc_usable_size), finding the last byte (B) and subtracting it. So we can get the true string length in an O(1) operation (usually, although this may depend on your malloc implementation).

ilenstr strings can contain \0 characters within the string.

ilenstr strings are also backwards compatible, in that we can pass one to any “legacy” C function, and assuming the string itself doesn’t contain any \0 inside it, everything just works.

Alright. This is terrible. DO NOT USE IT IN PRODUCTION CODE! It breaks all kinds of standards, is unportable etc. There are security issues with allowing \0-containing strings to be passed to legacy functions. Still, it’s a nice idea. With proper cooperation from libc, standards authorities and so on, it could be made to work.

Here is my git repo:

http://git.annexia.org/?p=ilenstr.git;a=summary


by rich at January 08, 2016 05:12 PM