Erlang: Things that make you go Hmmm…
October 28th, 2007 by
gloriajw
Today I had lofty plans of doing many things. But instead I learned Erlang. Someone approached me with a potential project, and thinks Erlang is the solution to the problem; how to achieve real time behavior with a flat process space in an application level program on a standard OS.
To my dismay, Python 3K won’t be offering this set of features, often referred to as concurrency. Very concise reasons are linked here, and actually make good sense:
http://ifacethoughts.net/2007/09/12/python-3k-discussions/#primary
But as an aside, that dreaded Global Interpreter Lock, which has bitten many of us in the ass, is unfortunately here to stay. The good news is that Python, even in it’s current form, allows you to recompile it and link directly to the OS thread libraries. What then happens to thread safety in Python with this configuration? It’s in your hands to lock and unlock semaphores or mutexes as you see fit, the way it should be.
Erlang is, well, an interesting alternative. It supports POSIX threads (now called pthreads since the last time I used them) and timer-based interrupts, which place most thread control in the developer’s hands. And like VxWorks, it offers a flat process space with a global method and variable name space.
Now, notice I compared VxWorks, a Real Time Operating System (RTOS) to Erlang, a language running on almost every operating system. Therein lay the power and intrigue of Erlang.
It also supports what are called green threads, which is essentially a way of running multiple interruptable tasks in the same process, without using OS resources to do the low level context switch. It’s neat, but I wonder if it’s necessary on a POSIX-based, lightweight process compliant OS such as LINUX. On Windows it seems like a necessary, handy tool, but it is not unique to Erlang, and is a bit of a tangent here.
Erlang’s quirky syntax seems to be, to some extent, a side effect of it’s features. For example, an Erlang programmer has to specify a module and function name when calling it. Similarly, one has to export what modules and functions one wishes to make visible. Parameters seem to be strictly positional, and overloaded functions are prevalent for this reason (not a bad thing).
In this example chunk, copied from here:
ring(_, N, _, _, _) when(N =
io:format(â€ÂEmpty ring~nâ€Â),
erlang:error(emptyRing);
ring(_, _, M, _, _) when(M =
io:format(â€ÂNo messages to send~nâ€Â),
erlang:error(noMessagesToSend);
ring(N, N, M, First_process, Main_process) ->
io:format(â€ÂRing process ~p (~p) created~nâ€Â, [N, self()]),
io:format(â€ÂSending ~p messages through the ring~nâ€Â, [M]),
First_process ! {send, main_process, Main_process},
loop(M, N, N, First_process, Main_process);
ring(I, N, M, First_process, Main_process) ->
io:format(â€ÂRing process ~p (~p) created~nâ€Â, [I, self()]),
Next_process = spawn(fun() ->
ring(I+1, N, M, First_process, Main_process) end),
loop(M, I, N, Next_process, Main_process).
Notice the lack of if-then-else or case statements to evaluate parameters. It either matches the prototype or it does not. It has assert() functions to evaluate parameters and anything else. And it has an exception handler, but throwing and catching exceptions is discouraged, for good reason. If the process control is at the thread level, threads should be conceptually treated like mini processes, and therefore should be cleanly exited.
Some level of type checking is done during compile time (yes, it’s compiled, not interpreted), which is definitely a nice alternative to the run time errors in Python which cause print statements to crash your entire program. But there’s something so quirky about having to specify
function(object)
instead of
object.function()
or
object->function()
And why does the global name space cause one to have to type:
spawn(fun() -> ring(1, N, M, self(), Main_process) end)
Can’t Erlang figure out that ring() is a function, and complain if there is an overload/name space violation when it occurs? Why must I tell Erlang: “ring is a function”?
And why, when importing a module, do I have to specify how many parameters the incoming functions expect? Again, another side effect of the global name space, or just quirky design?
-import(erunit, [test/2, assertEquals/2, run/1]).
I don’t remember having to do this in VxWorks, or any other language, for that matter. In C, the onus of prototyping functions was usually left to to creator of the module, and comes for free to the user via an #include statement. Erlang is quite odd in this way, but the good news is that it is also explicit. It is not an obscure oddity, and is easy to intuit.
I guess the bigger question is, in exchange for such a robust, neat idea of bringing the power of interrupts, flat name spaces and thread-level control to a programming language which runs on most operating systems, does it really have to be so quirky?
Should we be writing Erlang generators in Python? Maybe this is the answer (it’s nice to dream of simple answers).
This is what I have gathered in the span of about seven hours of Erlang study. I am sure I’m missing some finer details, but this is the jist of it from what I have seen so far. It leaves me with a feeling of hoping for more, yet being foolishly optimistic.
I can’t help but hope that Parallel Python could come to the rescue, but even it is limited for a certain scope of problem. All objects passed to the running job must be able to be pickled and unpickled. This makes passing of methods impossible, since they need to be instantiated on the “other side”, in the scope of the separate running process. For non-serializable complex objects, you’re stuck. For mutually exclusive manipulations of non-persistent serializable data, it is a good solution. So, essentially, Parallel Python could be used to circumvent some, but not all, threaded limitation problems which do not require persistence (or, which use a database to resolve their persistence needs). This isn’t very real-time-OS-like in any way, but resolves a subset of problems which out-of-the-box Python threading does not resolve.
One thing seems to be true, is that it’s nice to have choices, and to be able to compare and contrast the vast amount of freeware out there. The power of Erlang lay in it’s ability to provide RTOS-like features and functions on top any OS through a language interface. That in and of itself is a major leap forward for developers.
What I’d like to see in Erlang is an example of how to have event driven interrupts (where is the signal handler in this language? Am I stuck with just timers?), and how to interface/wrap functions from other languages, and pull them into Erlang.
Gloria
October 30th, 2007 at 2:57 pm
I posted a comment yesterday but just got a white page.
October 30th, 2007 at 3:06 pm
Erlang was designed in 1989, which could make for a quirky language. Mind you, so was Python. Ruby and Lua are both 1993. I like the way Lua has been revised over time, where as it’s unlikely that Erlang will become less quirky when there is so much code running on it. (though anyone could fork it)
I’m intrigued by Lua because, which though it has no pthreads built in, it’s ready to be embedded in a host that does, with an Erlang-ish share-nothing philosophy that programmers shouldn’t need to deal with mutexes and semaphores and all that. Of course, it doesn’t have OTP and all that other Erlang goodness. But the syntax, oh the syntax!
October 30th, 2007 at 3:14 pm
Nathan, thanks for being patient and reposting. We had a glitch yesterday, and Nola came to the rescue.
I have heard of LUA but had no reason to look at it. The person who asked me to evaluate Erlang needs green threads and concurrency, so I have a specific need that is hard to satisfy. This project, as it turns out, is so juicy, because of it’s unique problem set, that I REALLY hope I get to work on it.
But it looks like Erlang does exactly what this person needs.
Can you show me a project which uses LUA in production? I’d love to see how it’s being used.
Thanks again, Nathan!
~G~
October 31st, 2007 at 8:04 pm
If you simply wrote “spawn(ring(…))”, then Erlang would evaluate ring() immediately and pass the results to spawn(). This isn’t what you want!
When you write “spawn(fun() -> ring(…) end)”, you are creating a new, anonymous function and passing it to spawn() to be evaluated later. This is what you want. In Python you would do this with the “lambda” keyword, or by creating a named function:
def init_ring(): ring(1, N, M, self(), Main_process)
spawn(init_ring)
October 31st, 2007 at 8:11 pm
“And why, when importing a module, do I have to specify how many parameters the incoming functions expect?”
In functional languages, it’s common to have private “helper” functions that take extra arguments for the sake of tail-recursion. For example:
factorial(N) -> factorial(N, 1).
factorial(0, Acc) -> Result;
factorial(N, Acc) -> factorial(N-1, N*Acc).
In this example, you would export factorial/1 only. Then no one else can call factorial/2, which is just an implementation detail. Also the compiler can do extra optimization if it knows that factorial/2 is never called externally.
November 1st, 2007 at 8:41 am
Matt, thanks, this all makes sense.
I am assuming that fun() is not necessary in the case of non-lamda functions on the fly. But I guess I see quite a bit of functions constructed on the fly, similarly to the days of C/C++ inline functions. Is this a common tendency in Erlang, and if so, why?
The construct is still odd to me, because the scope of prototypes and functions is not cleanly contained within a class. It is very “C” like to overload functions and export only what is public. I can’t help but think that this quirkiness of Erlang, although efficient, did not have to be implemented this way, but is instead just a legacy design paradigm used by C programmers when constructing Erlang.
I’d like to see an RTOS-like OO language similar to the C++ POSIX compliant libraries supplied by VxWorks, but I understand how careful one would have to be to ensure it’s efficiency (beware of ‘heavy’ objects).
Where is the signal handler in Erlang? I see examples of timer based interrupts, but that is all. Also, I see examples of how to call Erlang from Python, but what about in the other direction?
November 1st, 2007 at 8:43 am
@gloria The “green threads” in Lua are called coroutines, and they aren’t preemptive. Instead the programmer uses a resume/yeild combination which gives complete control over when task switching happens. It’s a little like Ruby’s blocks with do |var| … end, and the method using yield. Except in Lua it is far more flexible, because the yield need not be directly in the function called… it can be somewhere down the line.
Adobe Lightroom uses Lua to good effect, such as when you rotate a photo there is a transition that runs while the interface remains responsive. Lightroom is 40% Lua, accounting for the UI, with the image crunching in C/C++. Outside of that Lua has mostly been used in games. As far as web, it seems pretty early for Lua. There is:
http://www.keplerproject.org/
The maturity of Erlang will probably win out for your project. But Lua is certainly an interesting up-and-comer, and the “Programming in Lua” book is one of the better-written books on computer programming. Enjoy!
November 1st, 2007 at 8:50 am
Wow Nathan, this sounds fascinating! I am not sure maturity of the language matters, especially if LUA is so premature that this project can help guide the feature set of LUA. This may work in our favor, actually.
I will check this, out, thank you for the tip!
November 1st, 2007 at 10:34 am
@gloria : to clarify, Lua as a language is 14 years old and is pretty refined. It’s the web side of things that’s still needing a lot of work.
The language is designed for especially for embedding and extending via C… originally developed for embedded software for an oil company, and having been popular in games it should have the performance you need. Please let me know how it goes…
http://www.lua.org/
November 1st, 2007 at 12:02 pm
“But I guess I see quite a bit of functions constructed on the fly, similarly to the days of C/C++ inline functions. Is this a common tendency in Erlang, and if so, why?”
Because Erlang is a functional language, many functions take other functions as arguments. It’s a common pattern to construct one function just to pass it to another function.
You don’t need to use anonymous functions; it’s just faster than giving them names. But you could create a named function instead, just like my Python example:
init_ring(N, M, Main_process) -> ring(1, N, M, self(), Main_process).
spawn(my_module, init_ring, [N, M, Main_process]).
“The construct is still odd to me, because the scope of prototypes and functions is not cleanly contained within a class.”
If you want to think in OO terms, the real “objects” in Erlang are processes. The process maintains its state, and responds to messages. (The messages are the “methods” in this analogy.)
Erlang is really very little like C. It has more in common with other functional languages (Lisp, Haskell, ML) and also with Prolog. Its use of higher-order functions is basically identical to those languages.
“Where is the signal handler in Erlang?”
I don’t know whether there’s a way to interrupt an Erlang process in the middle of a calculation. It seems like this would break some of the safety guarantees of the language. I think the Erlang way to do this would be to have the process poll its own message queue for “interrupts”. But I don’t have any actual experience in this area.
“Also, I see examples of how to call Erlang from Python, but what about in the other direction?”
Chapter 12 of “Programming Erlang” by Joe Armstrong (which I recommend highly) is all about how to interface Erlang with other languages. See also: http://www.erlang.org/doc/tutorial/part_frame.html
By the way, I just recently started using Erlang myself, so I was very happy to see the discussion you started here!
November 2nd, 2007 at 11:09 am
Signal handlers in Erlang are usually handled through the gen_server set of modules. OTP apps use a supervisor – worker model for processes. And the gen_server modules do the heavy lifting for this.
Really though sending a message to a process is dead simple in erlang so creating interrupts for a process is really easy.
November 3rd, 2007 at 10:51 am
This is really helpful, thanks for answering my questions.
If sending messages to processes creates interrupts, does this imply that there is an “administrative channel” of sorts, for each process, which would handle it’s message with a higher precedence than a standard message I would send from process to process?
For example, say I want to send a “kill” to a current thread, and I want it to have the highest precedence of all, like a ‘kill -9′ does. Is this inherent to the Erlang model? Is this something I need to build? Can you post an example of how you do this?
Thanks again.
November 3rd, 2007 at 6:17 pm
http://www.erlang.org/doc/design_principles/part_frame.html
talks about the the supervisor – worker architecture. Pay particular attention to the Gen_Event and Supervisor sections.
If on the other hand you just want a simple kill/interrupt of a process then the erlang:exit(Pid, reason) function is probably what you want. It will kill the process specified by Pid with the reason. if the process is trapping exits then you can handle the interrupt however you want if it isn’t then the process just exits with the reason.
Keep in mind that erlang is a functional language and interrupts are handled according to those principles. It makes sense if you think in functional terms but if you try to impose imperative paradigms on it then you get confused really quickly.
It’s worth it though if for no other reason than to stretch your mind.
November 4th, 2007 at 9:36 am
All of this has been helpful, thank you. I’ll be stretching my fingers and my comfort zone as well as my mind, on this next project. More questions will follow, I am sure