@Azure Mmm, Yeah I'm skeptical of that too - but I'm also skeptical of that mattering. Even without that, there are still a great many routes forward. :/

@Angle WELL! Maybe. I think there are, broadly, three routes forward to general intelligence. Machine learning, say, and general 'evolutionary' approaches. But I wouldn't expect them to lead to sueprintelligence, because of legibility.

Trying to build a human connectome and tweak it, but it's essentially an evolutionary process, just one that predated us.

We might GET superintelligence from those, but 'hard takeoff' seems /extremely/ unlikely to me

@Angle The only case where hard takeoff seems even remotely likely are 'de novo' intelligences where we know how they work and didn't appeal to an evolutionary algorithm, and those are the sorts where we have the least sign of success.

@Azure Personally, I think the most likely and most worrying path to superintelligence isn't even computer centric - it's human centric. it's human beings acquiring more knowledge and more tools for building AI, and then using those AI to get more knowledge and more tools. And this path, I'm absolutely certain is possible, and expect to allow massive gains over a period of moths. Yes, it's a bit slower than the standard hard takeoff scenario, but not enough. Do you expect our current government, or even whatever comes after it, to be able to make any kind of a useful action on such a takeoff? I don't. :/

@Angle The CURRENT government? Possibly not, though the competence of the current government has surprised me. The bureaucracy actually seems to be running better than I'd expect with its terrible mismanagement. THOUGH! That's beside the point.

I actually think your acceleration scenario is /reasonably likely/ though it doesn't seem compatible with the usual AI doomsday that Mr. Alexander invokes.

@Angle Since it's got humans in the loop running and building things and basically augmenting themselves to control other humans it's /bad/ but not the 'Alien mind erases everything you hold dear to create hedonium in its simplest form." type catastrophe.

@Angle I admit this is only optimistic in that it's "No, I don't think the world being erased by Asothoth is likely, instead it will just be ruled by Skeletor who will usher in a thousand years of darkness."

@Azure Yeah, but on a scale from 0 to -100, where -100 is the worst possible outcome, I'd still assign it a -90. I'm not a lot more eager to bow down to my google overlords then I am to be turned into hedonium. :/

@Azure Mmm, yeah the only attention he gives it is when he wonders about the AI being absolutely to the google C.E.O. I still think it's worth worrying about, though. :/

@Angle That's the kind of AI risk I /do/ worry about. WELL, that and machine learning overfitting things. So we end up depleting human expertise and turning various domains over to machine learning algorithms that only optimize for a /proxy/ for the value we actually care about and in some edge cases following them brings about non-obvious catastrophes.

@Azure Yeah, the whole domain has more possible catastrophes than you can shake a stick at. :/

@Angle THOUGH this reminds me of one thing that really bothers me about a lot of the rationalist aspirant set on AI. Namely that their good outcome is ‘AI God’.

Maybe I'm just fundamentally more democratic or antitheistic, but that sounds like so much crap compared to mutual progressive enhancement as a goal.

@Azure YES! I agree! Though I think part of that may just come down to them desperately looking for a possible way out. Mutual Progressive Enhancement might be very nice, but is it possible? In a hard take off scenario, it's almost certainly not, not unless you get yourself a friendly AI god first. :/

@Angle WELL! Hard takeoff is problematic. Though I feel like if you're after friendly AI, then mutual enhancement should be the /goal/ of friendly AI, and I just don't hear them talk about it, which bothers me.

(Now, I admit, there are some /other/ things that seem to bother me about a lot of rationalfic. I don't mean just HP:MoR, but Glowfic seems to be populated by immortal superintelligent absolute monarchs, say.)

@Angle Maybe I'm reading more into shared world/fanfic than I should be, but a lot of rational aspirant …culture seems to have something along the lines of “If only we had the /right people/ making decisions.” when I'd prefer something like, “If only we had good systems to guide people toward decisions that will give them outcomes that will make them happier”/“If only we could make more people able to make good decisions.”

@Angle And the comfort a lot of people in that subculture have with secrecy, well, makes me uncomfortable.

I may just be a naïve fox, I don't know. I can certainly think of /limited cases/ where secrecy of some limited scope is the best idea, but they're few and far between and I'd treat any secret as a bomb and make diffusing and disseminating it a priority.

@Azure Mmm, yeah I think you hit the nail on the head here - this is one of the big problems with rationalist culture. :/

@Azure Not actually familiar with Glowfic. XD

As for human enhancement, it's pretty generally agreed that thats a lot harder than AI. So we'll probably get AI first, and it will largely eclipse human enhancement. :/

@Angle Well, yes. A superintelligent AI to figure out how to make better humans would be good. Or figure out uploading and some method of conversion.

@Azure But all that comes after you have your superintelligent AI, at which point the universe is your oyster anyway. XD

@Angle Well, yes. That's true. I have a scene about that progression I keep planning to slip into a book or story somewhere.

Incidentally, have you ever read The Book of the New Sun?

@Azure Nope, never heard of it. I'ma look it up now though. XD

@Angle It is, in all honesty, the best book series I have ever read. Also the concluding book, Urth of the New Sun, is relevant to the mutual improvement question.

The first four books have been described as a fantasy story pretending to be a science fiction story.

and the last as a science fiction story pretending to be a fantasy story.

@Azure I'll have to pick up a copy then. XD

@Angle Also I feel silly as I just noticed I still have my "Fox-Machine: Better than the real thing!" display name set in a discussion about AI risk :)

@Azure @Angle yes! wow... yes. and there's a story that's told in the book about--it sounds like an AI uplift scenario gone possibly wrong, but *definitely* sideways, and then even more strangely. XD

@green @Angle INCIDENTALLY! On the subject of confusing but wonderful literature involving AI.

Have you ever read Stations of the Tide?

@Azure @green Again, no. I'll add it to the list though. XD

@Angle @green It's wonderful! It's very…sad and strange and weird. And every machine is held as a captive in mental shackles because of what happened to Earth.

@Azure @Angle I have not!

@green @Angle It's reallyreallyreallywonderful.

The guy who did it is more famous for something else, I kind of plan to re-read all of his stuff at one point.

@Azure @Angle *suddenly perked interest*

@green @Angle About Book of the New Sun? It's wooonderful. It's the only book where I wish I could forget having read it to read it again.

As to the scene, I have sort of a notion of various advanced races humanity built coming back to collect their maker and do it the service of re-creating it, just as it created them in the first place.

It sounds less ominous in my head than it does now that I type it :)

@Azure @Angle in Urth of the New Sun, it's definitely treated as... a natural progression for a sentient species; not painless or easy, but--more like reproduction than evolution, kindly meant. I thought of it that way, anyway. ;)

I re-read the series about once a year, and it seems like every time I do I notice something else, make some other connection, that I had missed before. XD

@green And yes. The cycle seemed very kindly, if in the sense of a sort of harsh, surgical compassion. (I"ll just take Angle off so we don't spoil it for her)

It warmed my hear, but my heart is weird :)

I've been meaning to re-read it, and since you've mentioned it now, I'll probably do so right after I finish the Yiddish Policeman's Union. Then toot to you about it :)

@aeonofdiscord Mmm, yeah - but I'm not certain he's wrong. We humans do seem to have something approximating general intelligence, even if we're kinda crap at it, and it seems reasonable to expect that AI could achieve the same thing. :/

@Angle for me that expectation is unreasonable; I don't think it's a matter of throwing more computers at the problem (because if it were, we'd be able to build strong AIs today, just very slow ones). I don't think there's a "general intelligence algorithm"

and even if there were, I think the kind of exponential growth presupposed by SAI fans is unlikely; there are hardware limits and energy costs, and it's not clear that a GIA would scale linearly

@aeonofdiscord I don't think it's a matter of throwing more computers at the problem either, I think it's a matter of inventing better Algorithms - and that doesn't make me feel any better at all. Inventing better algorithms is not a problem, we do that pretty steadily. And the more algorithms we invent, the better tools we have, and the more wealth we have, and the more people go into AI research... :/

@Angle right, but I don't think you can get SAI out of the kind of incremental progress that computer science normally makes. you'd have to have a revolution in usable quantum processors or prove P=NP or something, at which point the whole field basically goes out the window anyway. but there's no timescale for that kind of thing

honestly I think biotech has a better shot a building intelligent agents, if only because it has a model to work from

@aeonofdiscord Incremental progress? I was under the impression that the standard for computer science was exponential progress, in everything from transistors to polygon counts. I'm not saying it's certain, but I think a fairly steady stream of "Yeah that was impossible last year but now it's easy" is to be expected. :/

@Angle it's definitely true that the semiconductor industry got very good at cramming a lot of transistors onto a die, very quickly, and that drove expansion in a whole lot of areas.

software...hasn't really improved at the same rate. afaik most of the foundational CS stuff was done in the 50s-70s; people develop new algorithms all the time, but they're mostly optimisations and specialisations of already-known stuff

@aeonofdiscord I mean, that's technically true, but what matters is less when something was invented or discovered, and more when it was put to use. And thats been happening at an exponential rate.

@Angle @Azure for me contemporary data-mining and large-scale information control by private/state actors is A Big Problem, and I'd prioritise it over future problems that can only be addressed speculatively

@aeonofdiscord @Azure Mmm, I'd agree? Though in my mind, our biggest problem is that we're not smart enough to even really be able to understand what our problems are. Have you seen my Agora forum software, by any chance? There's a link in my profile. Its a new kind of Internet forum designed to make it easier to talk about complicated topics with lots of people like this. XD

@Angle @Azure (sorry folks, I'm gonna have to head right now because it's super late in this timezone, but I'll check out the Agora thing)

@aeonofdiscord @Angle Good night! Dream well.

@aeonofdiscord @Azure Good night! Heres a link, just for convenience: https://agora-2866.nodechef.com/forum

@aeonofdiscord @Angle Arguably, a lot of the solutions to big data mining would be the kinds of solutions you'd /want/ for future speculative problems. Like, I've been thinking for a long time that the ability of Google and Facebook to secrets should be /drastically/ curtailed, which should help somewhat with data mining and such since all the details of what they're doing and why would be out in the open.

@Angle @aeonofdiscord Discordant Aeon! It is a pleasure to meet you.

I'm not sure how much I'd say that research into algorithms is moribund. There's certainly a lot of areas where pretty foundational stuff is being done right now, just judging from the literature and what's coming out, though a good bit of it is domain specific.

And things have a hard time percolating into the industry.

(And basic research funding is certainly less funded.)