@Angle Since it's got humans in the loop running and building things and basically augmenting themselves to control other humans it's /bad/ but not the 'Alien mind erases everything you hold dear to create hedonium in its simplest form." type catastrophe.

@Angle I admit this is only optimistic in that it's "No, I don't think the world being erased by Asothoth is likely, instead it will just be ruled by Skeletor who will usher in a thousand years of darkness."

@Azure Yeah, but on a scale from 0 to -100, where -100 is the worst possible outcome, I'd still assign it a -90. I'm not a lot more eager to bow down to my google overlords then I am to be turned into hedonium. :/

@Azure Mmm, yeah the only attention he gives it is when he wonders about the AI being absolutely to the google C.E.O. I still think it's worth worrying about, though. :/

@Angle That's the kind of AI risk I /do/ worry about. WELL, that and machine learning overfitting things. So we end up depleting human expertise and turning various domains over to machine learning algorithms that only optimize for a /proxy/ for the value we actually care about and in some edge cases following them brings about non-obvious catastrophes.

@Azure Yeah, the whole domain has more possible catastrophes than you can shake a stick at. :/

@Angle THOUGH this reminds me of one thing that really bothers me about a lot of the rationalist aspirant set on AI. Namely that their good outcome is ‘AI God’.

Maybe I'm just fundamentally more democratic or antitheistic, but that sounds like so much crap compared to mutual progressive enhancement as a goal.

@Azure YES! I agree! Though I think part of that may just come down to them desperately looking for a possible way out. Mutual Progressive Enhancement might be very nice, but is it possible? In a hard take off scenario, it's almost certainly not, not unless you get yourself a friendly AI god first. :/

@Angle WELL! Hard takeoff is problematic. Though I feel like if you're after friendly AI, then mutual enhancement should be the /goal/ of friendly AI, and I just don't hear them talk about it, which bothers me.

(Now, I admit, there are some /other/ things that seem to bother me about a lot of rationalfic. I don't mean just HP:MoR, but Glowfic seems to be populated by immortal superintelligent absolute monarchs, say.)

@Angle Maybe I'm reading more into shared world/fanfic than I should be, but a lot of rational aspirant …culture seems to have something along the lines of “If only we had the /right people/ making decisions.” when I'd prefer something like, “If only we had good systems to guide people toward decisions that will give them outcomes that will make them happier”/“If only we could make more people able to make good decisions.”

@Angle And the comfort a lot of people in that subculture have with secrecy, well, makes me uncomfortable.

I may just be a naïve fox, I don't know. I can certainly think of /limited cases/ where secrecy of some limited scope is the best idea, but they're few and far between and I'd treat any secret as a bomb and make diffusing and disseminating it a priority.

@Azure Mmm, yeah I think you hit the nail on the head here - this is one of the big problems with rationalist culture. :/

@Azure Not actually familiar with Glowfic. XD

As for human enhancement, it's pretty generally agreed that thats a lot harder than AI. So we'll probably get AI first, and it will largely eclipse human enhancement. :/

@Angle Well, yes. A superintelligent AI to figure out how to make better humans would be good. Or figure out uploading and some method of conversion.

@Azure But all that comes after you have your superintelligent AI, at which point the universe is your oyster anyway. XD

@Angle Well, yes. That's true. I have a scene about that progression I keep planning to slip into a book or story somewhere.

Incidentally, have you ever read The Book of the New Sun?

@Azure Nope, never heard of it. I'ma look it up now though. XD

@Angle It is, in all honesty, the best book series I have ever read. Also the concluding book, Urth of the New Sun, is relevant to the mutual improvement question.

The first four books have been described as a fantasy story pretending to be a science fiction story.

and the last as a science fiction story pretending to be a fantasy story.

@Azure I'll have to pick up a copy then. XD

@Angle Also I feel silly as I just noticed I still have my "Fox-Machine: Better than the real thing!" display name set in a discussion about AI risk :)

@Azure @Angle yes! wow... yes. and there's a story that's told in the book about--it sounds like an AI uplift scenario gone possibly wrong, but *definitely* sideways, and then even more strangely. XD

@green @Angle INCIDENTALLY! On the subject of confusing but wonderful literature involving AI.

Have you ever read Stations of the Tide?

@Azure @green Again, no. I'll add it to the list though. XD

@Angle @green It's wonderful! It's very…sad and strange and weird. And every machine is held as a captive in mental shackles because of what happened to Earth.

@Azure @Angle I have not!

@green @Angle It's reallyreallyreallywonderful.

The guy who did it is more famous for something else, I kind of plan to re-read all of his stuff at one point.

@Azure @Angle *suddenly perked interest*

@green @Angle About Book of the New Sun? It's wooonderful. It's the only book where I wish I could forget having read it to read it again.

As to the scene, I have sort of a notion of various advanced races humanity built coming back to collect their maker and do it the service of re-creating it, just as it created them in the first place.

It sounds less ominous in my head than it does now that I type it :)

@Azure @Angle in Urth of the New Sun, it's definitely treated as... a natural progression for a sentient species; not painless or easy, but--more like reproduction than evolution, kindly meant. I thought of it that way, anyway. ;)

I re-read the series about once a year, and it seems like every time I do I notice something else, make some other connection, that I had missed before. XD

@green And yes. The cycle seemed very kindly, if in the sense of a sort of harsh, surgical compassion. (I"ll just take Angle off so we don't spoil it for her)

It warmed my hear, but my heart is weird :)

I've been meaning to re-read it, and since you've mentioned it now, I'll probably do so right after I finish the Yiddish Policeman's Union. Then toot to you about it :)