Just saw this post by Scott Alexander. It's pretty cool. http://slatestarcodex.com/superintelligence-faq/

@Angle this makes some very weird unspoken assumptions about what "general intelligence" is and how it works; there's an analogy made to Go-playing (which isn't very much like general intelligence) and another one made to biological evolution (which isn't very much like AI development), and a lot of the argument seems to rest on extending those two analogies past breaking point

@aeonofdiscord Mmm, yeah - but I'm not certain he's wrong. We humans do seem to have something approximating general intelligence, even if we're kinda crap at it, and it seems reasonable to expect that AI could achieve the same thing. :/

@Angle for me that expectation is unreasonable; I don't think it's a matter of throwing more computers at the problem (because if it were, we'd be able to build strong AIs today, just very slow ones). I don't think there's a "general intelligence algorithm"

and even if there were, I think the kind of exponential growth presupposed by SAI fans is unlikely; there are hardware limits and energy costs, and it's not clear that a GIA would scale linearly

@aeonofdiscord I don't think it's a matter of throwing more computers at the problem either, I think it's a matter of inventing better Algorithms - and that doesn't make me feel any better at all. Inventing better algorithms is not a problem, we do that pretty steadily. And the more algorithms we invent, the better tools we have, and the more wealth we have, and the more people go into AI research... :/