These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet.

The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture.

This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.

Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away.

Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.

There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.

I disagree

That list is a list of the dangers of power

Many of these dangers: misinformation, killer robots, people on this list have been actively working on

Rank hypocrisy

And people projecting their own dark personalities onto a neutral technology

Yes there are dangers in unbridled private power. They are not dangers unique to AI.

If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.

People are being replaced by robots and AI because the latter are cheaper. That's the market force.

Cheaper means that more value us created. As a whole, people get more service for doing less work.

The problem is that the money or value saved trickles up to the rich.

The only solutions can be, regulations,

- do not tax anymore based on income from doing actual work.

- tax automated systems on their added value.

- use the tax generated capital to provide for a basic income for everybody.

In that way, the generated value goes to people who lost their jobs and to the working class as well.

Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are:

1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway?

2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk.

3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway.

4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk.

5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook.

6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves.

7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI.

8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so.

None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.

With crypto, self-driving cars, computers, the internet or just about any other technology, development and distribution happened over decades.

With AI, there’s a risk that the pace of change and adoption could be too fast to be able to respond or adapt at a societal level.

The rebuttals to each of the issues in your comment are valid, but most (all?) of the counter examples are ones that took a long time to occur, which provided ample time for people to prepare and adapt. E.g. “technology making us fat” happened over multiple decades, not over the span of a few months.

Either way, I think it’s good to see people proactive about managing risk of new technologies. Governments and businesses are usually terrible at fixing problems that haven’t manifested yet… so it’s great to see some people sounding the alarms before any damage is done.

Note: I personally think there’s a high chance AI is extremely overhyped and that none of this will matter in a few years. But even so, I’d rather see organizations being proactive with risk management rather than reacting too the problem when it’s too late.

https://research.facebook.com/downloads/babi/

There's been a lot of progress since then, but it's also nearly 10 years later. Progress isn't actually instant or overnight. It's just that OpenAI spent a ton of money to scale it up then stuck an accessible chat interface on top of tech that was previously being mostly ignored.

For example, let's say you need to ensure that your writes actually ends up on disk: https://stackoverflow.com/questions/12990180/what-does-it-ta...

The typical file abstraction introduces buffering on top of this, which adds additional edge cases: https://stackoverflow.com/questions/42434872/writing-program...

If you want ordered writes, then you have to handle this through some kind of journalling or at the application level with something like this: https://pages.cs.wisc.edu/~vijayc/papers/UWCS-TR-2012-1709.p...

And that's only something that's visible at the application level. There are plenty of similar edge cases below the surface.

Even if all of this works correctly, you still have to remember that very few systems give any guarantees about what data ends up on disk. So you often end up using checksums and/or error correcting codes in your files.

Finally, all this is really only talking about single files and write operations. As soon as you need to coordinate operations between multiple files (e.g., because you're using directories) things become more complicated. If you then want to abuse the file system to do something other than read and write arrays of bytes you will have to deal with operations that are even more broken, e.g., file locking: http://0pointer.de/blog/projects/locking.html

---

It's not an accident that you are using other services to store your files. For example, S3 handles a lot of the complexity around durable storage for you, but at a massive cost in latency compared to what the underlying hardware is capable of.

Similarly, application programmers often end up using embedded databases, for exactly the same reason and with exactly the same problem.

This is a shame, because your file system has to solve many of the same problems internally anyway. Metadata is usually guaranteed to be consistent and this is implemented through some sort of journaling system. It's just that the file system abstraction does not expose any of this and necessitates a lot of duplicate complexity at the application level.

---

Edit: After re-reading the grandparent comment, it sounds like they are arguing against the "array of bytes" model. I agree that this is usually not what you want at the application level, but it's less clear how to build a different abstraction that can be introduced incrementally. Without incremental adoption such a solution just won't work.

My question is can those uncertainties be fixed with a less performant, ordered, and safe file system for typical application use. Then bleeding-edge with plenty of sharp edge cases for high performance compute work that application programmers can handle at app level? Because it is nuts how fast hardware and inexpensive RAM are and I think if you add +30% time on file write IO that will not greatly impact the user experience vs all the other causes of lag that burden us like network and bloat.

Then in the HPC word if a new byte cloud where all context is in some database with a magic index naturally comes to be we can move to that. I won't rule out needing to change the underlying file system because that's pretty over my head and there are good ideas I don't understand.

My point is to push against the proprietary format vendor lock-in file system abstractions like I get in nested objects in microsoft powerpoint or word or apple garage band where the app is merely wrapping files and hiding your actual data that you can pick up and move to another app. I don't want to need to adopt a way of thinking about pretty simple objects to use every different program.

I like wavs > flac, plain text > binary, constant bit rate > variable bit rate, sqlite > cloud company db (not really fair but just saying sqlite one-file db is amazing). Storage is inexpensive and adding in layers to decode the data runs a risk of breaking it and I like interoperability. Once you lose the file data and just have content clouds there might be compression running on the data changing the quality, e.g. youtube as a video store with successive compression algorithms aging old videos.

It drives me nuts when needing to attach things I'm faced with a huge context list where I'd rather go find a directory. Abstractions are just that, mental models to avoid the low level stuff. I'm still cool thinking of my information as file trees I think that's an OK level. But you're right complex operations with a file system has issues. I've messed up logging and file IO not thinking it through and it made me think about needing to fix my mistaken code.

The point is that we treat the differential of a real valued function as a function/vector/matrix for historical reasons. The simpler perspective that always works is that the differential of a function is the best linear approximation of the function at a given point. But for historical reasons most math textbooks restrict themselves to "first order functions" and avoid, e.g., functions returning functions.

This also leads to ridiculous notational problems when dealing with higher order functions, like integration and all kinds of integral transforms.

As far as I understand, the MLIR project predates its machine learning application and was originally intended as a new IR for clang. In that capacity it makes a lot of sense. MLIR is also currently experimental in Tensorflow, although I have no idea how mature the implementation is.

Similarly, there has been significant investment into Swift for Tensorflow, so it's probably here to stay. On the other hand, from a language design perspective Swift is not a particularly good choice for automatic differentiation and translation into Tensorflow graphs (imperative, exposing many details of the underlying machine, etc.). Without a lot of investment into this project it might just be overtaken by a better engineered competitor, or more likely, fail to gain sufficient mind-share over the "good enough" python solution that already exists.

Of all the things in this list, this is the one you think is odd?

The idea behind homotopy type theory is a groundbreaking connection between type theory and abstract homotopy theory (really, higher-category theory) and had far reaching consequences in both areas.

For example, cubical type theory solves many problems with equality in type theory and it was only developed by understanding the homotopical models of type theory that were developed for HoTT. I could give more examples, but seriously, this alone is a major advance in logic.

There are some proposals which address this (e.g., concepts), but none of them are part of the language standard yet. Concepts in particular made it into the C++20 draft, but they also made it into a draft of the C++17 standard and were ultimately rejected. Somewhat vexingly C++ concepts actually come with the same problems only at the level of concepts instead of at the level of templates.

The main problem with lenses is that common lens libraries look extremely complicated at first glance and seem to be solving a very simple problem. That rightfully puts most people off of learning what all the fuss is about.

Most mathematicians spend very little time thinking about the foundations of mathematics though, so I'd take that with a grain of salt. ;)

Seriously though, I think the statement that "all of modern mathematics can be formalized in ZFC" is probably wrong. There have been attempts to do just that (e.g., Bourbaki), but they invariably failed and ended up with extensions of ZFC (e.g., for category theory) or just silently started using higher-order logic.

Type theory seems to be a lot more practical. Both because it is explicitly designed to be an open system (a closed system must fail eventually due to Gödel arguments) and because similar formalization efforts go a lot more smoothly. For example, compare the treatment of category theory in the HoTT book with any introductory text using set theory.

The Glushkov construction (which Hyperscan uses among other things) is a similarly straightforward translation from a regular expression to an epsilon-free NFA with nice properties. There's a very neat paper explaining the construction in the form of a play: https://sebfisch.github.io/haskell-regexp/regexp-play.pdf

On the CFG side, I like to think of Early parsing as analogous to the Glushkov construction. At the very least they have similar properties.

In my experience, people won't come out and say it, but this seems to be what everyone is thinking. :)

The problem with this is that it is wrong.

Classical ZFC in particular is a very strong and specific set of assumptions* with a very tenuous link to any practical application. If you actually want to develop a useful bit of mathematics it makes sense to consider the "foundations" as a moving piece. It's a part of the design space for modeling your problem domain, not some god-given notion of truth.

You can translate between different logical theories by building a model of one in another, so it's not like you loose anything. But it's cooky to insist that we should start with ZFC of all things.

---

*) I mean that second-order ZFC has basically no non-trivial models, so there is no real way of extending ZFC to talk about domain specific aspects of your problems.

> You can translate between different logical theories by building a model of one in another, so it's not like you loose anything. But it's cooky to insist that we should start with ZFC of all things.

I don't see how these two are consistent. Almost everything most mathematicians do can be done both in ZFC and your favourite non-kooky axiom system. Certain Powers That Be seem to have decided that ZFC is the foundation of mathematics, so they say that what they're doing follows from ZFC even if they have a very hazy idea of what it is, but why does it matter? Most mathematics probably won't be formalized in their lifetime anyway, so whether it ends up being formalized on top of ZFC or something else doesn't affect them.

You would be amazed at how many uniqueness and existence theorems in how many areas of mathematics require Zorn's Lemma. Which is, of course, equivalent to the axiom of choice. For example, "Every vector space has a (possibly infinite) basis." Or, "Every Hilbert space has a (possibly infinite) orthonormal basis."

It is rare for mathematicians to think much about choice. But it underpins key results in a surprising number of fields.

> "Almost everything most programmers do can be done both in x86 assembly and your favorite non-kooky programming language. Certain Powers That Be seem to have decided that x86 is the foundation of computer science. [...] Why does it matter?"

The problem is that it is difficult to translate results in a theory built in ZFC to other "architectures". In mathematics, the architectures in question are not different axiom systems, they are different branches of mathematics.

Let me give you an example. There is a large body of work on differential geometry with many useful constructions. Classical differential geometry works directly in a model where manifolds are certain subspaces of (countable products of) R^n. These constructions have been successfully imported into many different areas of mathematics. In most cases people just had to tweak the definitions slightly and adapt the proofs by keeping the basic strategy and changing all the details.

What is happening here is that the underlying ideas of differential geometry are not specific to this particular model.

When faced with such a concrete model, our first instinct should be to abstract from it and ask which assumptions are required. This is difficult in ZFC, because in the end you have to encode everything into sets. It's not possible to reason about "abstract datatypes" directly, without literally building a notion of logic (generalized algebraic theory) and models of it within ZFC. Even then, the existence of choice means that you usually have to exclude unwanted models of your theory.

Coming back to differential geometry: You can generalize a lot of it by working in a differentially cohesive (infinity-)topos. This is terribly indirect (in my opinion) and looses a lot of the intuitions. A topos is literally a model of a certain logic. Alternatively you can work directly in this logic (the "internal language" of the topos), where the differential geometric structure is available in the form of additional logical connectives. You are now talking in a language where it makes sense to talk about two points being "infinitesimally close" and where you can separate the topological from the infinitesimal structure.

At the same time you reap the benefits that there are many more models of differential cohesion than there are models of "R^n modeled in classical set theory". You can easily identify new applications, which might in turn suggest looking into different aspects of your theory. It's a virtuous cycle. :)

This approach is deeply unnatural when working in set theory or arithmetic. You have to encode everything into sets or numbers and then these sets or numbers become the thing you are studying.

I'd make the following alternative analogy: I code in Python on an x86, because that happens to be the machine on my desk. If you told me I should be using POWER instead of x86, I'd probably just shrug: I could do that - my work is portable - but it's also completely irrelevant to my work. I think this would be how most people in say analysis, algorithms or combinatorics feel, for example.