{"id":1461,"date":"2013-09-24T13:16:30","date_gmt":"2013-09-24T13:16:30","guid":{"rendered":"http:\/\/www.syslog.cl.cam.ac.uk\/?p=1461"},"modified":"2013-09-24T21:58:59","modified_gmt":"2013-09-24T21:58:59","slug":"liveblogging-ocaml-workshop-2013","status":"publish","type":"post","link":"https:\/\/www.syslog.cl.cam.ac.uk\/2013\/09\/24\/liveblogging-ocaml-workshop-2013\/","title":{"rendered":"Liveblogging OCaml Workshop 2013"},"content":{"rendered":"

\"photo\"<\/a>I'm here bright and early at the OCaml 2013<\/a> workshop at ICFP with Heidi Howard, Leo White, Jeremy Yallop and David Sheets! \u00c2\u00a0There's a packed session today with lots of us presenting, so we'll swap over editing this post as the day progresses. \u00c2\u00a0I also need to pop out to attend a panel on the future of Haskell so I'll be missing the bindings session, sadly!<\/p>\n

<\/p>\n

Accessing and using weather data in OCaml<\/h2>\n

Hez Carty from MDA Information Systems on doing large-scale weather data processing.\"photo\"<\/a><\/p>\n

He started with OCaml in graduate school and was looking for something better than Python, whereas they are building systems that need to run for many years with high stability. \u00c2\u00a0He's not from a formal CS background but from the physical sciences, and found that the static typing made it easy to explore the language features. \u00c2\u00a0The native code output is also important, and he finds predictability very important in day-to-day engineering. \u00c2\u00a0The simple FFI also made a big difference.<\/p>\n

They do weather data retrieval and use HTTP\/zeromq as a message bus to do language interop, and a workflow management tool to handle dependency graphs. \u00c2\u00a0They take weather models frmo all these sources, and do various analyses (principal component analysis, probability models for precipitation via Monte Carlo simulations with a first order Markov chains). \u00c2\u00a0More generally, there is a bunch of processing for climatology and data extraction and insertion.<\/p>\n

A NAM forecast plots the geo data of the USA, and overlays semitransparent layers (e.g. clouds) that are stacked up to get a fake satellite view of the earth, and then there are simulated radar returns where the model (e.g.) rains. \u00c2\u00a0In addition to the structured formats, there is raw streaming data from sensor stations that send back raw binary data such as precipitation events, which is written as a small OCaml library to parses them into more structured formats.<\/p>\n

The big data formats are HDF4 and GRIB, which are binary data formats. There are mature C libraries to parse them and lots of existing analysis tools (e.g. wgrib which can extract lat\/longs from a big file). \u00c2\u00a0However, the tools arent great for doing larger scale analysis, as shelling out all the time for thousands of points is not ideal -- therefore they bind to OCaml to do the analysis in-house.<\/p>\n

The GRIB bindings use int, float and float arrays and bindings are entirely hand-written <i>(I'm looking forward to the ctypes talk! --anil)<\/i>. \u00c2\u00a0They expose only an OCaml friendly interface that is safe for callers. \u00c2\u00a0The speaker now shows several nice graphs from CloudSat tracks from earth orbit, and how they take intersecting profiles to deduce things about what multiple satellites are seeing.<\/p>\n

They're mixing lots of different tools and satellite sources -- derived information from teleconnections (air surface pressure and how it varies in terms of oscillations between the north and south Atlantic). \u00c2\u00a0All the analysis is written in OCaml, and he had to work with several scientists who didn't have access to their initial source code in many cases, so the experience of working with his fresh system was very positive.<\/p>\n

Challenges:<\/p>\n