Just saw this post by Scott Alexander. It's pretty cool. http://slatestarcodex.com/superintelligence-faq/

@Angle I'm a bit skeptical of some of their arguments THOUGH recent news has made me slightly less skeptical of some. I had been pretty sure that the lack of legibility would make machine-learning generated AIs unable to improve themselves, though the machine learning bot that learns to make machine learning bots seems at first blush to go beyond what I'd expect. I need to look into it in more detail.

@Azure Mmm, Yeah I'm skeptical of that too - but I'm also skeptical of that mattering. Even without that, there are still a great many routes forward. :/

@Angle WELL! Maybe. I think there are, broadly, three routes forward to general intelligence. Machine learning, say, and general 'evolutionary' approaches. But I wouldn't expect them to lead to sueprintelligence, because of legibility.

Trying to build a human connectome and tweak it, but it's essentially an evolutionary process, just one that predated us.

We might GET superintelligence from those, but 'hard takeoff' seems /extremely/ unlikely to me

@Azure Personally, I think the most likely and most worrying path to superintelligence isn't even computer centric - it's human centric. it's human beings acquiring more knowledge and more tools for building AI, and then using those AI to get more knowledge and more tools. And this path, I'm absolutely certain is possible, and expect to allow massive gains over a period of moths. Yes, it's a bit slower than the standard hard takeoff scenario, but not enough. Do you expect our current government, or even whatever comes after it, to be able to make any kind of a useful action on such a takeoff? I don't. :/

@Angle The CURRENT government? Possibly not, though the competence of the current government has surprised me. The bureaucracy actually seems to be running better than I'd expect with its terrible mismanagement. THOUGH! That's beside the point.

I actually think your acceleration scenario is /reasonably likely/ though it doesn't seem compatible with the usual AI doomsday that Mr. Alexander invokes.

@Azure Mmm, yeah the only attention he gives it is when he wonders about the AI being absolutely to the google C.E.O. I still think it's worth worrying about, though. :/

@Angle That's the kind of AI risk I /do/ worry about. WELL, that and machine learning overfitting things. So we end up depleting human expertise and turning various domains over to machine learning algorithms that only optimize for a /proxy/ for the value we actually care about and in some edge cases following them brings about non-obvious catastrophes.

@Azure Yeah, the whole domain has more possible catastrophes than you can shake a stick at. :/

@Angle THOUGH this reminds me of one thing that really bothers me about a lot of the rationalist aspirant set on AI. Namely that their good outcome is ‘AI God’.

Maybe I'm just fundamentally more democratic or antitheistic, but that sounds like so much crap compared to mutual progressive enhancement as a goal.

@Azure YES! I agree! Though I think part of that may just come down to them desperately looking for a possible way out. Mutual Progressive Enhancement might be very nice, but is it possible? In a hard take off scenario, it's almost certainly not, not unless you get yourself a friendly AI god first. :/

@Angle WELL! Hard takeoff is problematic. Though I feel like if you're after friendly AI, then mutual enhancement should be the /goal/ of friendly AI, and I just don't hear them talk about it, which bothers me.

(Now, I admit, there are some /other/ things that seem to bother me about a lot of rationalfic. I don't mean just HP:MoR, but Glowfic seems to be populated by immortal superintelligent absolute monarchs, say.)

@Azure Not actually familiar with Glowfic. XD

As for human enhancement, it's pretty generally agreed that thats a lot harder than AI. So we'll probably get AI first, and it will largely eclipse human enhancement. :/

@Angle Well, yes. A superintelligent AI to figure out how to make better humans would be good. Or figure out uploading and some method of conversion.

@Azure But all that comes after you have your superintelligent AI, at which point the universe is your oyster anyway. XD

@Angle Well, yes. That's true. I have a scene about that progression I keep planning to slip into a book or story somewhere.

Incidentally, have you ever read The Book of the New Sun?

@Azure Nope, never heard of it. I'ma look it up now though. XD

@Angle It is, in all honesty, the best book series I have ever read. Also the concluding book, Urth of the New Sun, is relevant to the mutual improvement question.

The first four books have been described as a fantasy story pretending to be a science fiction story.

and the last as a science fiction story pretending to be a fantasy story.

@Azure @Angle yes! wow... yes. and there's a story that's told in the book about--it sounds like an AI uplift scenario gone possibly wrong, but *definitely* sideways, and then even more strangely. XD