Instrumental convergence is the hypothetical tendency of sufficiently intelligent, goal-directed beings (human and nonhuman) to pursue similar sub-goals (such as survival or resource acquisition), even if their ultimate goals are quite different. More precisely, beings with agency may pursue similar instrumental goals—goals which are made in pursuit of some particular end, but are not the end goals themselves—because it helps accomplish end goals.
Instrumental convergence posits that an intelligent agent with seemingly harmless but unbounded goals can act in surprisingly harmful ways. For example, a sufficiently intelligent program with the sole, unconstrained goal of solving a complex mathematics problem like the Riemann hypothesis could attempt to turn the Earth (and in principle other celestial bodies) into additional computing infrastructure to succeed in its calculations.
Proposed basic AI drives include utility function or goal-content integrity, self-protection, freedom from interference, self-improvement, and non-satiable acquisition of additional resources.
Instrumental...
John Deters
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.