I have heard many times now that LRM doesn't scale. People have been telling it to me in all these different circumstances. They tell me that languages like Deep Throttle and Moulder have scaling but LRM doesn't. Even the Scally language has scaling - you could understand that from the name I guess. So why doesn't LRM has this feature?
Since I'm rather fond of LRM, and really don't want to go back to either Deep Throttle or Moulder, I decided to add scaling to LRM. I mean, how hard could it be if all these other languages have it? I have implemented langauges before, so this should just be a case of finding the correct implementation of the feature, and grafting it onto LRM's code base, and then provide a patch to the mailing lists.
Since I knew LRM didn't have scaling, I decided to investigate other languages that have it. I began with the original language, the one people write operating systems in. Obviously a language like Deep Throttle has scaling. So I started reading the books, looking through the syntax. But I couldn't find anything there. What was I missing? Was there some obscure syntactic element that wasn't documented, that provided scaling? Or was it hidden somewhere? Maybe the pointers were in reality another word for scaling? But no, it didn't seem that way. I decided to break open GDPC, the Gnu Deep Throttle Compiler, and see if this scaling thing was actually something the compiler did autoamtically for you - a bit like register allocation. But I didn't find any scaling allocation. There was lots of other neat stuff there, but all of it was quite mundane translations from the Deep Throttle code into machine code.
Feeling utterly dejected I turned my attention to Moulder. I assume that you all are familiar with the language - it's used in many places around the world and provides lots of really advanced features such as garbage collection, a virtual machine, and yes - scaling! Of course, this is where it would be. Maybe Deep Throttle had hidden the feature somewhere obscure, but I was sure Moulder would make it easier to find. I started out with the syntax again, looking through the things you could do with the language. Interestingly, Moulder is a really small language, when you get down to it. It's small but you need to say lots of things to get anything done. I couldn't find the syntax for scaling, so I started entertaining other ideas. Maybe all those extra words "protected final static Foo
You see where this is heading? Right, I had to investigate the bytecodes that Moulder use. Where is that precious scaling bytecode? I assume it would be called something like invoke_scaling or maybe ldscaled. But nothing. There was no scaling bytecode. All of the existing bytecodes just did the same mundane things that I know LRM already can do. I was now feeling ready to give up, but I came up with one final idea. Scally is famous for being a language with scaling. It's creators even named it after that feature. So if both Moulder and Scally has scaling, maybe it's somewhere in Moulders gigantic standard library?
But guess what? I didn't find it. I still haven't found anyone who knows how you implement Scaling in a language, so I guess that LRM will never have it... Anyone who care to enlighten me, please send me a detailed email with an implementation of Scaling. I really feel the need to know how this thing works.
22 kommentarer:
this scally discussion might help:
http://www.nabble.com/Scala's-scalability-claims-are-confusing-td15172045.html
best,
jherber
Generally speaking, a framework "scales" if, for most applications using that framework, you can compensate for an increased user count by a linear or close to linear increase in hardware.
Google's version of MapReduce, for instance, is scalable - if properly used, you can describe your problem in it, then, when your servers balk under the load, just throw more hardware at it and it works again.
Common causes for claims of "doesn't scale" is that a framework doesn't support spreading the load over a network of computers - you can make a single box only so powerful. Or there's some inherent limit in the framework that means that adding more users causes a disproportional slowdown.
I can't make any statements towards specific frameworks' scalabilities though.
Hope that explains things.
I don't see the point of obfuscating the language names, since your descriptions of them give them away. As a result, it's still apparent what your bias is anyway, just with the added delay of having to read a little further (perhaps that is the point of obfuscating).
Languages aren't scalable objects. Only applications are. This is particularly true because as you scale an application the language remains constant across the application. If scaling your application requires you to reengineer components of your app, that's an issue with having picked the wrong implementation.
That said, languages provide the terrain upon which you can scale your application. If you're using a framework, you're further constrained on that terrain. So when someone says "Ruby doesn't scale" what they really should be saying is "Ruby is slow". And as far as i can tell (and i say this as genuine ruby lover) that is a statement that would be true.
My replies to them are still a) "So what?" and b) "There are a lot of efforts to improve Ruby's performance, and i'm willing to hang around to see what happens."
"If scaling your application requires you to reengineer components of your app, that's an issue with having picked the wrong implementation."
Rubbish! Scaling is simply a problem to be solved when it arises, and not before. The effort to build a "scalable" app is more than to not, and why waste time solving a problem that might never exist?
Scaling is a problem whose solution (money and time) are available when you need them. From the start of a project, that isn't true.
solve the problem you have when you have it I say!
"If scaling your application requires you to reengineer components of your app, that's an issue with having picked the wrong implementation."
Rubbish! Scaling is simply a problem to be solved when it arises, and not before. The effort to build a "scalable" app is more than to not, and why waste time solving a problem that might never exist?
Scaling is a problem whose solution (money and time) are available when you need them. From the start of a project, that isn't true.
solve the problem you have when you have it I say!
Gold. Seriously tho, who actually thinks Ola needs scaling explained to him? I think somebody missed the joke ;-)
Maybe you sould try to add a module Scale ( which have a class method scale_it, with lot of vm stuff – or not ), a lot of documentations and pub.
LRM will be scalable too !!!
i do not think feepingcreature or notheory missed the joke, ola was sarcastically referring to the common belief: "rails does not scale" (see his previous post) and his point i guess is that scaling is nothing to do with the language (or framework for that matter).
while it's understandable ruby/rails people defend their toys, i find it ironic to argue that from one hand that execution speed of the interpreter (or let's say a query via AR) does not matter on the hand post about how the jruby team improved this or that in jruby.
if performance does not matter, why spend so much time and energy on improving it?
...on the other hand
dohzya is wrong. There's no "require Scaling" statement; it just automagically happens, no thought/design/planning necessary -- or it's a fault of the language and framework! :)
Maybe the scaling code is more obvious in Scala. After all, it's in the name. Alternatively, yeah, there's always that Scalable module you can mix into LRM... :-)
"while it's understandable ruby/rails people defend their toys, i find it ironic to argue that from one hand that execution speed of the interpreter (or let's say a query via AR) does not matter on the hand post about how the jruby team improved this or that in jruby."
Wow that comment really misses the point! Just because scaling is separate from the time it takes to execute of some instruction in an environment does not imply speed is not important.
I think too many people have learned to argue like politicians instead of trying to have objective discussions. Dear commenter, please claim that you were trying to be ironic and restore my faith in humanity.
matt m,
care to elaborate?
"Just because scaling is separate from the time it takes to execute of some instruction in an environment..."
while scalability is certainly a broad term, i do not think you can separate it completely from let's say framework or environment overhead/performance.
matt m,
just to give you a concrete example:
last year while twitter struggled with scaling issue 1.0 they ran into this problem:
http://romeda.org/blog/2007/06/select-from-everything-or-why-databases.html
because of this bug, AR generated horrible queries. so are you telling me that this AR overhead had nothing to do with scaling (because it's just slow execution, right)?
"while scalability is certainly a broad term, i do not think you can separate it completely from let's say framework or environment overhead/performance."
Hmmm...I don't think I tried to completely separate it. I was merely referring to the poor structure of the argument that was laid out. Ola says that scaling and execution speed are different things. You then criticize him for saying that execution speed isn't important. Which he didn't say.
And now the same mistake is repeated. No one said that the execution speed and scaling are "completely separate". So why must you make arguments against it?
However, do this thought experiment. Suppose all CPUs were 10x slower than today. Would that have any impact on whether a platform can scale? Obviously, it wouldn't, it would just mean you need more hardware to execute the same number of instructions. Now suppose CPUs were 10x faster- again, no impact on whether Platform X can scale, just an impact on the hardware configuration required to do so.
"Consider Twitter...because of this bug, AR generated horrible queries. so are you telling me that this AR overhead had nothing to do with scaling (because it's just slow execution, right)?"
Wow- by that illogic, just because I can write a bad Java program it means Java can't scale? You must have been the champion on your school debate team. Twitter briefly had database problems, Twitter doesn't have those problems now. So, does that mean it did or it didn't scale? Or it didn't and now it does? Or do you even have a point to make at all that is relevant to this discussion?
Just because things are separate doesn't mean they have nothing to do with one another. Consider your average Java memory leak bug in a database driver- usually connection pooling related. I have to work around that bug to preserve memory, but the app takes more memory than it would otherwise. So yes, that means I have to order more RAM, but it doesn't mean the application can't be scaled, it's just more expensive to do so.
Consider a commercial database product where the recommended scaling approach to a big database is to find the server with the most CPUs and with the most memory. At some point you buy the biggest Sun E25K or whatever. And it stops scaling. But few would say that "it doesn't scale." Maybe, it doesn't scale "infinitely", but who cares?
I think you can agree that the generic question- "does X scale?" is a pretty stupid one without numerous conditions attached, and really only makes sense in relation to some other thing. If you say, can I build a website that does serves n tps with this language or that framework, the answer is almost always going to be yes.
So what are you complaining about? And why?
"Hmmm...I don't think I tried to completely separate it."
so now you do agree that performance of the framework+interpreter and scalability are related? that was my only point.
" I was merely referring to the poor structure of the argument that was laid out."
thank you.
" Ola says that scaling and execution speed are different things. You then criticize him for saying that execution speed isn't important. Which he didn't say."
I did not say that they were the same, I just said that they are related.
"And now the same mistake is repeated. No one said that the execution speed and scaling are "completely separate". So why must you make arguments against it?"
all i said was that i found that ironic to argue that scalability is nothing to do with the environment (including the framework and the interpreter) and at the same time post about performance improvements.
and i did not criticize ola, i appreciate what he and Charles and co. have done with Jruby.
"However, do this thought experiment. Suppose all CPUs were 10x slower than today. Would that have any impact on whether a platform can scale? Obviously, it wouldn't, it would just mean you need more hardware to execute the same number of instructions. Now suppose CPUs were 10x faster- again, no impact on whether Platform X can scale, just an impact on the hardware configuration required to do so."
as i said scalability is a broad term. and yes, it is (mis)used many times to flag certain technologies including rails, oh, this does not scale (as dhh pointed that out in his twitter comment).
that being said, companies have limited money to buy hardware and they would like to spend less (surprise). and "throwing in" hardware is sometimes not that easy either even if money is available.
i use rails since late 2005 and maybe i am a rare type but i just happen to understand when people are referring to rails related scalability issues, and yes, maybe i am not religious enough since i believe that scaling rails it's still harder than it should be.
(on the other hand, the future looks bright in ruby land: rails 2.1, mod_rails, jruby or merb - just to name a few great improvements)
"Wow- by that illogic, just because I can write a bad Java program it means Java can't scale?
If, for example, a java *framework* adds unnecessary extra overhead for whatever reason, then yes, i believe that particular framework makes it harder to scale.
"You must have been the champion on your school debate team."
could you please stop your ad hominem attacks?
" Twitter briefly had database problems, Twitter doesn't have those problems now. So, does that mean it did or it didn't scale?"
if you read the post blaine also claims that that particular *framework* related problem was related to their *scaling* issues. so my point again: scaling is not independent from the environment as opposed to what ola claims.
"Just because things are separate doesn't mean they have nothing to do with one another."
please see above.
"Consider your average Java memory leak bug"
my average java memory leak bug? what are you talking about?
" I have to work around that bug to preserve memory, but the app takes more memory than it would otherwise. So yes, that means I have to order more RAM, but it doesn't mean the application can't be scaled, it's just more expensive to do so."
again the reality is that companies have limited resources, buying extra hardware sounds like a good solution to the problem but sometimes that's not an option (either because of money or technical reasons).
would not be your platform easier to scale if the bug did not exist in the first place?
"Consider a commercial database product where the recommended scaling approach to a big database is to find the server with the most CPUs and with the most memory. At some point you buy the biggest Sun E25K or whatever. And it stops scaling. But few would say that "it doesn't scale." Maybe, it doesn't scale "infinitely", but who cares?"
as i said, i agree with that "does X scale" arguments are oftentimes FUD and without context it really does not make any sense.
on the other hand, i do think that there are certain *rails specific* problems that do not necessary exist elsewhere and which problems might directly effect scalability as opposed to what ola says (ie frameworks/languages do not have baked in scalability related characteristics).
"I think you can agree that the generic question- "does X scale?" is a pretty stupid one without numerous conditions attached, and really only makes sense in relation to some other thing. If you say, can I build a website that does serves n tps with this language or that framework, the answer is almost always going to be yes."
"So what are you complaining about? And why?"
except the ad hominem attacks I liked the discussion. thanks.
Scalibity of a language or a platform has almost nothing to do with the runtime speed of the language.
So, the speed of Ruby is not a problem at all. The problem is Ruby doesn't give the developers any means of creating light-weight threads and/or processes that are translated into native OS threads or processes.
If I create 8 threads in a Scalable language running on a 8 core box, each one of those threads will be mapped to one these cores and they will all be running (almost) simultaneously.
You have been looking at wrong places for scalability Ola, you should be looking into the land of Wherlang to find it.
Cheers
So, as languages and tools do not seem to have anything to do with scalability I guess we can choose easily the culprit:
The idiot we pay peanuts to write our sw ...
The status-quo is happy
Beautiful... I hadn't even thought of it in such terms. I much appreciate your eloquent reality-check for everyone. :)
Do you know the Atlantica online Gold, in the game you need the Atlantica Gold. it can help you increase your level. My friends always asked me how to buy Atlantica online Gold, i do not know he spend how much money to buy the Atlantica online money, when i see him in order to play the game and search which the place can buy the cheap Atlantica online Gold . i also happy with him.
I do not know why I was very like to play the EVE, so I need eve isk and the eve gold , do you know the eve online isk.
Skicka en kommentar