Just saw this post by Scott Alexander. It's pretty cool. http://slatestarcodex.com/superintelligence-faq/

@Angle I'm a bit skeptical of some of their arguments THOUGH recent news has made me slightly less skeptical of some. I had been pretty sure that the lack of legibility would make machine-learning generated AIs unable to improve themselves, though the machine learning bot that learns to make machine learning bots seems at first blush to go beyond what I'd expect. I need to look into it in more detail.

@Azure Mmm, Yeah I'm skeptical of that too - but I'm also skeptical of that mattering. Even without that, there are still a great many routes forward. :/

@Angle WELL! Maybe. I think there are, broadly, three routes forward to general intelligence. Machine learning, say, and general 'evolutionary' approaches. But I wouldn't expect them to lead to sueprintelligence, because of legibility.

Trying to build a human connectome and tweak it, but it's essentially an evolutionary process, just one that predated us.

We might GET superintelligence from those, but 'hard takeoff' seems /extremely/ unlikely to me

@Azure Personally, I think the most likely and most worrying path to superintelligence isn't even computer centric - it's human centric. it's human beings acquiring more knowledge and more tools for building AI, and then using those AI to get more knowledge and more tools. And this path, I'm absolutely certain is possible, and expect to allow massive gains over a period of moths. Yes, it's a bit slower than the standard hard takeoff scenario, but not enough. Do you expect our current government, or even whatever comes after it, to be able to make any kind of a useful action on such a takeoff? I don't. :/

@Angle The CURRENT government? Possibly not, though the competence of the current government has surprised me. The bureaucracy actually seems to be running better than I'd expect with its terrible mismanagement. THOUGH! That's beside the point.

I actually think your acceleration scenario is /reasonably likely/ though it doesn't seem compatible with the usual AI doomsday that Mr. Alexander invokes.

@Azure Mmm, yeah the only attention he gives it is when he wonders about the AI being absolutely to the google C.E.O. I still think it's worth worrying about, though. :/

@Angle That's the kind of AI risk I /do/ worry about. WELL, that and machine learning overfitting things. So we end up depleting human expertise and turning various domains over to machine learning algorithms that only optimize for a /proxy/ for the value we actually care about and in some edge cases following them brings about non-obvious catastrophes.

@Azure Yeah, the whole domain has more possible catastrophes than you can shake a stick at. :/

@Angle THOUGH this reminds me of one thing that really bothers me about a lot of the rationalist aspirant set on AI. Namely that their good outcome is ‘AI God’.

Maybe I'm just fundamentally more democratic or antitheistic, but that sounds like so much crap compared to mutual progressive enhancement as a goal.

@Azure YES! I agree! Though I think part of that may just come down to them desperately looking for a possible way out. Mutual Progressive Enhancement might be very nice, but is it possible? In a hard take off scenario, it's almost certainly not, not unless you get yourself a friendly AI god first. :/