Loyc and Nemerle

I created this blog to be about Loyc, the Language of Your Choice project, which is about making general tools for creating and extending programming language, converting code between languages, and creating tools for IDEs. However, quite simply Loyc is too big a project for one person. I couldn't find the time for it, and I couldn't quite figure out how to make an extensible programming system.

Last August I decided to reboot with something much more modest: to design and implement a series of improvements to C#. That way I hoped to gain enough experience to do something bigger, later. To increase my chance of success, I changed to a part-time worker at work, leaving more time for Loyc.

I was going to start by modifying NRefactory, and my "Enhanced C#" would simply compile down to plain C#. I couldn't figure out how to accomplish my end goal--a fully extensible language--so I decided to simply improve the existing C# language with a series of specific and modest features such as the "null dot" or safe navigation operator (now denoted ??.), the quick-binding operator (::), forwarding clauses, return value covariance, C++-style templates, "static if", blah blah blah.

However, once I was done drafting the new language, I noticed that despite all the work I'd put into merely the draft alone, the new language still didn't address one of the most requested features from C# users: "Provide a way for INotifyPropertyChanged to be implemented for you automatically on classes".

INotifyPropertyChanged is a simple interface for allowing code to subscribe to an object to find out when it changes. For example:

interface INotifyPropertyChanged {
 event PropertyChangedEventHandler PropertyChanged;
}
class Person : INotifyPropertyChanged {
 public event PropertyChangedEventHandler PropertyChanged;
 
 string _name;
 public string Name
 {
  get { return _name; }
  set {
   if (_name != value) {
    _name = Value;
    Changed("Name");
   }
  }
 }
 string _address;
 public string Address
 {
  ... same thing ...
 }
 DateTime _dateOfBirth;
 public DateTime DateOfBirth
 {
  ... same thing ...
 }
 void Changed(string prop)
 {
  if (PropertyChanged != null)
   // The .NET "EventArgs" concept is stupid, but I digress
   PropertyChanged(this, new PropertyChangedEventArgs(prop));
 }
}

The problem is that you need 10 new lines of code for every new property in your class. Couldn't the language have some kind of feature to make this easier? But I didn't want to have a feature that was specifically designed for INotifyPropertyChanged and nothing else. Besides, there are different ways that users might want the feature to work: maybe some people want the event to fire even if the value is not changing. Maybe some people want to do a reference comparison while others want to do object.Equals(). Maybe it can't be fully automatic--some properties may need some additional processing besides just firing the event. How could a feature built into the language be flexible enough for all scenarios?

I decided at that point that it was time to learn LISP. After studying it briefly, I figured out that what would really help in this case is macros. Macros would allow users to provide a code "template" that is expanded for each property. To make a medium-length story short, I came up with an interchange format for syntax trees called "Loyc trees", and I dropped my EC# 1.0 draft--which was never made--to create EC# 2.0; many of the features of EC# 1.0 could be accomplished with macros anyhow. But since I wasn't satisfied with any of the readily-available parser generators, I also decided to make my own LL(k) parser generator. The initial version is about 75% complete right now (no, no, just the parser generator, sorry); the main missing features are syntactic "and" predicates (an optional feature that provides unlimited lookahead), and the ability for the parser generator to parse its own input (I'm using C# operator overloading for now.)

So the other day I finally started writing the parser for EC# 2.0, which would also be the parser generator's native language... when I rediscovered Nemerle.

I'd looked at Nemerle a few years back, but it didn't seem very impressive at the time; it just appeared to be a C#-like language with some type inference, algebraic data types, and pattern matching (and my memory is bad but I think it mainly targeted Linux (Mono)). Those are all good things, but my interest has always been in extensibility. I never knew until now that in fact Nemerle already has LISP-style macros (well, not quite LISP-style: a Nemerle macro cannot exist in the same project as the code to which it is applied; but that's okay because EC# wasn't going to support such an advanced feature either (not at first, although I want to support it eventually).)

I don't know whether Nemerle has always been extensible, and I simply didn't realize it, or whether macros are a new feature. In either case, this discovery will certainly impact my development of Loyc and EC#. In principle, Nemerle is so similar to EC# that I should perhaps even halt development of EC# and just use Nemerle.

My first impression is that Nemerle's macros seem to lack the simplicity of my design, but they are clearly more powerful. So far, I have only figured out how to support syntactic macros in EC#. I haven't figured out how to allow a macro to access semantic knowledge of a program--to look up types and members, or to introspect the "meaning" of some code and then to modify that code. Nemerle seems to have multiple "flavors" of macros that can accomplish different things, which is very exciting. Another neat fact is that both myself and Nemerle's people decided to use what I call a "tree parser"--the lexer output is grouped by parenthesis, square brackets and braces, before it is sent to the main parser. This makes it easier to support extensibility (although EC#'s parser will not be extensible), and makes it easy for the parser to look beyond anything in parentheses.

So where do I go from here? I'm not sure yet. I'll see what I can learn about the Nemerle compiler and macro system, before I can decide on a course of action. I am particularly interested in the base language--the language before any macros are added--but I have found only one page on that topic, and the beginning of that article is hard to understand. Oh well. (Argh! They keep saying that basic features of the language such as the "if" expression are defined by macros, but since Nemerle has no "goto" statement, I can't imagine how the "if" expression could decompose into anything simpler.) I also found this "extended course" in macros, and other information.

Wikipedia's Nemerle page claims that "the core developers of Nemerle were hired by the Czech software development company JetBrains." However, the core developers appeared to be Michał Moskal and Kamil Skalski, and Google seemed to indicate that those two worked for Microsoft and Google respectively, not JetBrains. So I contacted Michał, who told me that neither of them were working on Nemerle! He said there "were a few people, mostly from Russia, still working on it." But who exactly is left? I have no idea.

The webpage is in disarray. The first time I tried to download Nemerle, I found the download link on the front page of the wiki, but this points to downloads that are 2 years old, making me wonder if Nemerle was dead (I found the correct download link later). I wanted to talk to Nemerle's developers, but the link to the "Forum" on Nemerle's front page was broken, so next I found this contact page, but the links to the mailing list archives do not work, and the subscription link requires login credentials. Finally I found this "news" page, which hasn't been updated since 2004! Finally, Michał pointed me to Nemerle's Google Group, but clicking "New Topic" caused the error "An error occurred while communicating with the server." Yikes!

Finally, I was able to ask a couple of questions about Nemerle--but so far, the only response is from someone that cannot understand my English. It turns out that the first language of Nemerle is Russian! Could it be that I won't be able to work with the Nemerle people due to a language barrier?

In conclusion, we should all learn Esperanto or something.
Por finiĝi, ni ĉiuj devus lerni Esperanton aŭ ion.

(I used to be very interested in the language boo, but the boo people wouldn't talk to me, and in general did not document boo's advanced features at all. I don't know if the Nemerle people will talk to me, but at least Nemerle offers far more articles about its features than boo ever did, even if you only count the English articles (note, it's been a couple of years now since I checked on boo).)