Existential Risk

Existential risk from AI systems.

Rate at which AI capabilities improve.

Sudden jump to superintelligence.

Incremental capability growth.

Learned subsystem that optimizes its own objective.

Maintaining alignment under new conditions.

Willingness of system to accept correction or shutdown.

Using limited human feedback to guide large models.