📚 A Próxima Onda

🎯 Resumo da Obra

In the book The Coming Wave Mustafa Suleyman argues that we are about to face a new technological wave due mainly to the development of Inteligência artificial and Biotechnology. To this end, the author defines O que é uma onda tecnologica, what were the 3 ondas tecnológicas anteriores and what is A próxima onda. The main point of the book is that Tecnologias fundamentais se expandem rapidamente and that we must deal with O problema de contenção tecnológico. Therefore, we need to face the Armadilha do pessimismo-aversão to try to prevent the Apocalipse informacional and the deterioration of the relationship between Tecnologias e o estado. To do this, we need to look not just at the detail, but at the big picture of what is happening in the world of technology, since Tecnologias são ideias and there are several Forças tecnológicas de proliferação.

It is necessary to understand that Nós dependemos de tecnologias but that Technological solutions are not a panacea. Therefore, when developing technologies, we must understand the Drivers of the search for better technologies and use the Estratégia para identificar novas ondas tecnológicas in order to try to contain the development of this technology so that it is safe.

🧠 Minha Experiência

One of the interesting parts of the book is the beginning, which was written by an Inteligência artificial and which presents a writing style that is practically indistinguishable from the author. This alone is a demonstration of the impact that this technology has on our lives. In addition, the author also brings worrying reports of the advance of synthetic biology both in terms of technical capacity and cost reduction. Taken together, these technologies clearly have the potential to have a huge impact on our future, leaving us with no idea what might emerge and what path societies and states will take.

The book left me with a sense of urgency about the need to contain the development of these technologies. Not in the sense of stopping them, but in the sense of controlling their development and ensuring that they are aligned with best ethical and social practices. This includes appropriate regulation, risk assessment, and measures to mitigate misuse.

However, Suleyman also acknowledges the challenges of containing these technologies due to their enormous economic and control potential. The power and profits that can be gained from their development and use will make it difficult to implement effective containment measures. Despite the challenges, I believe it is essential that we have an open and informed discussion about the future of AI and synthetic biology

📝 Notas de Conteúdo

🖍️ Destaques e Citações

Almost every foundational technology ever invented, from pickaxes to plows, pottery to photography, phones to planes, and everything in between, follows a single, seemingly immutable law: it gets cheaper and easier to use, and ultimately it proliferates, far and wide.
A school of naive techno-solutionism sees technology as the answer to all of the world’s problems. Alone, it’s not. How it is created, used, owned, and managed all makes a difference.
Put simply, technology and political order are intimately connected. The introduction of new technologies has major political consequences. .

And now we stand at the brink of another such moment as we face the rise of a coming wave of technology that includes both advanced AI and biotechnology. — Página: 202

Nota: Those are two of the most transformagive technologies

With AI, we could unlock the secrets of the universe, cure diseases that have long eluded us, and create new forms of art and culture that stretch the bounds of imagination. With biotechnology, we could engineer life to tackle diseases and transform agriculture, creating a world that is healthier and more sustainable. — Página: 205

But on the other hand, the potential dangers of these technologies are equally vast and profound. With AI, we could create systems that are beyond our control and find ourselves at the mercy of algorithms that we don’t understand. With biotechnology, we could manipulate the very building blocks of life, potentially creating unintended consequences for both individuals and entire ecosystems. — Página: 207

The above was written by an AI. The rest is not, although it soon could be. This is what’s coming. — Página: 215

Nota: An ai can write like a human!?

Permeating humanity’s oral traditions and ancient writings is the idea of a giant wave sweeping everything in its path, leaving the world remade and reborn. — Página: 225

Look again at history and you can see it marked by a series of metaphorical waves: the rise and fall of empires and religions, and bursts of commerce. — Página: 231

Nota: There arw other forms of waves that are not literal.

Almost every foundational technology ever invented, from pickaxes to plows, pottery to photography, phones to planes, and everything in between, follows a single, seemingly immutable law: it gets cheaper and easier to use, and ultimately it proliferates, far and wide. — Página: 236

Humanity’s quest to improve—ourselves, our lot, our abilities, and our influence over our environment—has powered a relentless evolution of ideas and creation. — Página: 238

Almost every object in your line of sight has, in all likelihood, been created or altered by human intelligence. — Página: 244

Nota: Vivemos num mundo nao natural

Every principle and abstract concept, every small creative endeavor or project, every encounter in your life, has been mediated by our species’ unique and endlessly complex capacity for imagination, creativity, and reason. Human ingenuity is an astonishing thing. — Página: 247

Nota: This is amazing!

It’s no exaggeration to say the entirety of the human world depends on either living systems or our intelligence. — Página: 252

The coming wave is defined by two core technologies: artificial intelligence (AI) and synthetic biology. — Página: 255

From where we stand today, it appears that containing this wave—that is, controlling, curbing, or even stopping it—is not possible. This book asks why that might be true and what it means if it is. — Página: 259

Nota: This is the main ideia in the book.

What if we could distill the essence of what makes us humans so productive and capable into software, into an algorithm? — Página: 266

To achieve this objective, we would need to create a system that could imitate and then eventually outperform all human cognitive abilities, from vision and speech to planning and imagination, and ultimately empathy and creativity. — Página: 273

Nota: This is the foal of an AGI.

AI has been climbing the ladder of cognitive abilities for decades, and it now looks set to reach human-level performance across a very wide range of tasks within the next three years. — Página: 287

Beyond AI, a wider revolution was underway, with AI feeding a powerful, emerging generation of genetic technologies and robotics. Further progress in one area accelerates the others in a chaotic and cross-catalyzing process beyond anyone’s direct control. — Página: 291

Nota: There is a positiive feedback loop

Once matured, these emerging technologies will spread rapidly, becoming cheaper, more accessible, and widely diffused throughout society. They will offer extraordinary new medical advances and clean energy breakthroughs, creating not just new businesses but new industries and quality of life improvements in almost every imaginable area. — Página: 300

Nota: Arent those difffente type of AI with different tasks?!

The likelihood of each may be small, but the possible consequences are huge. Even a slim chance of outcomes like these requires urgent attention. — Página: 307

Nota: Pequenos eventos podem ter graves consequencias

Some countries will react to the possibility of such catastrophic risks with a form of technologically charged authoritarianism to slow the spread of these new powers. — Página: 308

Nota: This is a possibility

Equally plausible is a Luddite reaction. Bans, boycotts, and moratoriums will ensue. — Página: 311

Nota: This is also possible

Moreover, attempting to ban development of new technologies is itself a risk: technologically stagnant societies are historically unstable and prone to collapse. Eventually, they lose the capacity to solve problems, to progress. — Página: 314

Nota: Is it mandatory to keep going foward?!

This is the core dilemma: that, sooner or later, a powerful generation of technology leads humanity toward either catastrophic or dystopian outcomes. — Página: 320

I see this as an interlocking set of technical, social, and legal mechanisms constraining and controlling technology working at every possible level: a means, in theory, of evading the dilemma. Yet even technology’s harshest critics tend to dodge this language of hard containment. That needs to change; I hope this book shows why, and hints at how. — Página: 325

In the past, new jobs were created at the same time as old ones were made obsolete, but what if AI could simply do most of those as well? There was, I suggested, little precedent for the new forms of concentrated power that were coming. — Página: 337

Nota: When technology starts to act more like humans we might have a problem.

The message was clear, but I spelled it out anyway. “The pitchforks are coming,” I said. Coming for us, the makers of technology. It was up to us to ensure the future was better than this. — Página: 341

One stood out. The presenter showed how the price of DNA synthesizers, which can print bespoke strands of DNA, was falling rapidly. Costing a few tens of thousands of dollars, they are small enough to sit on a bench in your garage and let people synthesize—that is, manufacture—DNA. — Página: 348

Nota: This is a poblem with uninmagined consequences.

Given the increasing availability of the tools, the presenter painted a harrowing vision: Someone could soon create novel pathogens far more transmissible and lethal than anything found in nature. — Página: 351

Nota: Isso e bastante perigoso

No one wanted to confront the implications of the hard facts and cold probabilities they’d heard. — Página: 361

This widespread emotional reaction I was observing is something I have come to call the pessimism-aversion trap: the misguided analysis that arises when you are overwhelmed by a fear of confronting potentially dark realities, and the resulting tendency to look the other way. — Página: 367

Spend time in tech or policy circles, and it quickly becomes obvious that head-in-the-sand is the default ideology. — Página: 396

Nota: Not diving deeper in the problem is a problem.

Waves are everywhere in human life. This one is just the latest. — Página: 406

Nota: This is similar to other ones

I am biased when it comes to the question of whether this matters. Nonetheless, having been up close to this unfurling revolution over the last decade and a half, I am convinced we’re on the cusp of the most important transformation of our lifetimes. — Página: 410

Nota: This is someone from within the developmsnt of technology

But without containment, every other aspect of technology, every discussion of its ethical shortcomings, or the benefits it could bring, is inconsequential. — Página: 416

Nota: Withiou thinking thare is no understanting of its responsability

Instead of turning away from technologies, the past is marked by a pronounced pattern of proliferation, resulting in sprawling chains of both intended and unintended consequences. — Página: 421

I call this “the containment problem.” How do we keep a grip on the most valuable technologies ever invented as they get cheaper and spread faster than any in history? — Página: 423

Nota: How do we control it?!

details of the coming wave itself. At its heart lie two general-purpose technologies of immense promise, power, and peril: artificial intelligence and synthetic biology. — Página: 425

The various technologies I’m speaking of share four key features that explain why this isn’t business as usual: they are inherently general and therefore omni-use, they hyper-evolve, they have asymmetric impacts, and, in some respects, they are increasingly autonomous. — Página: 429

Nota: These as the characteristics of the comming wave

In part 4 the discussion moves to what we can do about it. — Página: 440

Our great challenge is that we still haven’t reckoned with the latter mode of failure. — Página: 450

Nota: The ones were technology does more harm

Guarding against failure means understanding and ultimately confronting what can go wrong. — Página: 455

Today some 2 billion combustion engines are in everything from lawnmowers to container ships. Around 1.4 billion of them are in cars. They have grown steadily more accessible, efficient, powerful, and adaptable. A whole way of life, arguably a whole civilization, developed around them, from sprawling suburbs to industrial farms, drive-thru restaurants to car mod culture. Vast highways were built, sometimes right through cities, severing neighborhoods but connecting far-flung regions. — Página: 495

Once there was momentum, the spread of the internal combustion engine became unstoppable. From a few oil-soaked German workshops grew a technology that has affected every human being on earth. — Página: 503

Nota: Some technologies arw similar to this onw

Technology has a clear, inevitable trajectory: mass diffusion in great roiling waves. — Página: 507

Over time demand for the best new products and services grows, driving competition to produce cheaper versions bursting with yet more features. This in turn drives yet more demand for the technologies that create them, and they also become easier and cheaper to — Página: 509

So, what is a wave? Put simply, a wave is a set of technologies coming together around the same time, powered by one or several new general-purpose technologies with profound societal implications. — Página: 515

Nota: This concept is interesting

By “general-purpose technologies,” I mean those that enable seismic advances in what human beings can do. — Página: 516

From the very beginning, we are never separate from the waves of technology we create. We evolve together, in symbiosis. — Página: 520

Nota: Do we use technology or does it uses us?

Stonework and fire were proto-general-purpose technologies, meaning they were pervasive, in turn enabling new inventions, goods, and organizational behaviors. — Página: 531

The irony of general-purpose technologies is that, before long, they become invisible and we take them for granted. — Página: 535

Language, agriculture, writing—each was a general-purpose technology at the center of an early wave. These three waves formed the foundation of civilization as we know it. Now we take them for granted. — Página: 536

number of general-purpose technologies — Página: 538

Nota: What are they?

The Agricultural Revolution (9000 –7500 BCE), one of history’s most significant waves, marked the arrival of two massive general-purpose technologies that gradually replaced the nomadic, hunter-gatherer way of life: the domestication of plants and animals. — Página: 550

Eventually, this coupled with a new revolution in tools—hoes and plows. These simple innovations marked the beginning of modern civilizations. — Página: 553

Nota: Civikization started with agriculture

Over time, this dynamic accelerated. Beginning around the 1770s in Europe, the first wave of the Industrial Revolution combined steam power, mechanized looms, the factory system, and canals. — Página: 558

In the 1840s came the age of railways, telegraphs, and steamships, and a bit later steel and machine tools; together they formed the First Industrial Revolution. — Página: 560

Then, just a few decades later, came the Second Industrial Revolution. You’ll be familiar with its greatest hits: the internal combustion engine, chemical engineering, powered flight, and electricity. — Página: 561

This isn’t, however, an orderly process. Technological waves don’t arrive with the neat predictability of the tides. Over the long term, waves erratically intersect and intensify. — Página: 564

Nota: Technology is based on caos

They are part of us. There is no such thing as a non-technological human being. — Página: 570

Nota: We are one with technology

Most people in technology are stuck in the minutiae of today and dreaming of tomorrow. It is tempting to think of inventions in discrete and lucky moments. But do so and you’ll miss the stark patterns of history, the sheer, almost innate tendency for technology’s waves to come again and again. — Página: 578

General-purpose technologies become waves when they diffuse widely. Without an epic and near-uncontrolled global diffusion, it’s not a wave; it’s a historical curiosity. — Página: 584

Proliferation is catalyzed by two forces: demand and the resulting cost decreases, each of which drives technology to become even better and cheaper. — Página: 606

The long and intricate dialogue of science and technology produces a chain of insights, breakthroughs, and tools that build and reinforce over time, productive recombinations that drive the future. — Página: 607

Nota: This is a good relationship between the two.

Civilization’s appetite for useful and cheaper technologies is boundless. This will not change. — Página: 614

Transistors are now produced in the tens of trillions per second, at billionths of a dollar per transistor: the fastest, most extensive proliferation in history. — Página: 638

It took smartphones a few years to go from niche product to utterly essential item for two-thirds of the planet. — Página: 642

Now humans produce hundreds of billions of emails, messages, images, and videos daily and store them in the cloud. Eighteen million gigabytes of data are added to the global sum every single minute of every day. — Página: 646

Our phones are the first thing we see in the morning and the last at night. Every aspect of human life is affected: they help us find love and new friends while turbocharging supply chains. They influence who gets elected and how, where our money is invested, our children’s self-esteem, our music tastes, our fashion, our food, and everything in between. — Página: 650

No previous wave has mushroomed as quickly, but the historical pattern nonetheless repeats. At first it seems impossible and unimaginable. Then it appears inevitable. And each wave grows bigger and stronger still. — Página: 655

History tells us that technology diffuses, inevitably, eventually to almost everywhere, from the first campfires to the fires of the Saturn V rocket, from the first scrawled letters to the endless text of the internet. — Página: 661

Technology’s unavoidable challenge is that its makers quickly lose control over the path their inventions take once introduced to the world. — Página: 672

What people actually do with your invention, however well intentioned, can never be guaranteed. — Página: 675

Understanding technology is, in part, about trying to understand its unintended consequences, to predict not just positive spillovers but “revenge effects.” — Página: 683

As technology proliferates, more people can use it, adapt it, shape it however they like, in chains of causality beyond any individual’s comprehension. — Página: 687

Technology’s problem here is a containment problem. If this aspect cannot be eliminated, it might be curtailed. Containment is the overarching ability to control, limit, and, if need be, close down technologies at any stage of their development or deployment. It means, in some circumstances, the ability to stop a technology from proliferating in the first place, checking the ripple of unintended consequences (both good and bad). — Página: 692

In most cases, containment is about meaningful control, the capability to stop a use case, change a research direction, or deny access to harmful actors. — Página: 700

Containment encompasses regulation, better technical safety, new governance and ownership models, and new modes of accountability and transparency, all as necessary (but not sufficient) precursors to safer technology. — Página: 712

Think of containment, then, as a set of interlinked and mutually reinforcing technical, cultural, legal, and political mechanisms for maintaining societal control of technology during a time of exponential change; an architecture up to the task of containing what would have once been centuries or millennia of technological change happening now in a matter of years or even months, where consequences ricochet around the world in seconds. — Página: 715

In hindsight, waves might appear smooth and inevitable. But there is an almost infinite array of small, local, and often arbitrary factors that affect a technology’s trajectory. — Página: 737

Where there is demand, technology always breaks out, finds traction, builds users. — Página: 757

As technologies let you do more, for less, their appeal only grows, along with their adoption. — Página: 766

Technologies are ideas, and ideas cannot be eliminated. — Página: 774

Technology is an eternally dangling carrot, constantly promising more, better, easier, cheaper. — Página: 775

That nuclear technology remained contained was no accident; it was a conscious nonproliferation policy of the nuclear powers, helped by the fact that nuclear weapons are incredibly complex and expensive to produce. — Página: 798

Nuclear history is still a chilling succession of accidents, near misses, and misunderstandings. — Página: 817

Although its history is one of enabling people to do more, increasing capabilities, driving improvements in well-being, it’s not a one-sided story: Technology creates more lethal and destructive weapons as well as better tools. It produces losers, eliminates some jobs and ways of life, and creates harm up to the planetary, existential scale of climate change. — Página: 868

As long as a technology is useful, desirable, affordable, accessible, and unsurpassed, it survives and spreads and those features compound. — Página: 873

While technology doesn’t tell us when, or how, or whether to walk through the doors it opens, sooner or later we do seem to walk through them. — Página: 874

By and large we’ve let the waves wash over us, managing on an uncoordinated, ad hoc basis, accepting that capabilities spreading inevitably and uncontrollably is, whether welcomed or reviled, a fact of life. — Página: 881

In the coming decades, a new wave of technology will force us to confront the most foundational questions our species has ever faced. — Página: 884

For most of history, the challenge of technology lay in creating and unleashing its power. That has now flipped: the challenge of technology today is about containing its unleashed power, ensuring it continues to serve us and our planet. — Página: 889

The “general” in AGI refers to the technology’s intended broad scope; we wanted to build truly general learning agents that could exceed human performance at most cognitive tasks. — Página: 901

For the first time I’d witnessed a very simple, very elegant system that could learn valuable knowledge, arguably a strategy that wasn’t obvious to many humans. — Página: 918

When IBM’s Deep Blue beat Garry Kasparov at chess in 1997, it used the so-called brute-force technique, where an algorithm aims to systematically crunch through as many possible moves as it can. — Página: 935

AlphaGo initially learned by watching 150,000 games played by human experts. Once we were satisfied with its initial performance, the key next step was creating lots of copies of AlphaGo and getting it to play against itself over and over. — Página: 938

Until recently, the history of technology could be encapsulated in a single phrase: humanity’s quest to manipulate atoms. — Página: 960

First bits and then increasingly genes supplanted atoms as the building blocks of invention. — Página: 969

They began parallel revolutions—those of bits and genes—that dealt in the currency of information, working at new levels of abstraction and complexity. Eventually, the technologies matured and gave us everything from smartphones to genetically modified rice. But there were limits to what we could do. — Página: 971

artificial intelligence and synthetic biology. For the first time core components of our technological ecosystem directly address two foundational properties of our world: intelligence and life. In other words, technology is undergoing a phase transition. No longer simply a tool, it’s going to engineer life and rival—and surpass—our own intelligence. — Página: 976

While AI and synthetic biology are the coming wave’s central general-purpose technologies, a bundle of technologies with unusually powerful ramifications surrounds them, encompassing quantum computing, robotics, nanotechnology, and the potential for abundant energy, among others. — Página: 984

Technology is a set of evolving ideas. New technologies evolve by colliding and combining with other technologies. — Página: 989

Each technology described here intersects with, buttresses, and boosts the others in ways that make it difficult to predict their impact in advance. They are all deeply entangled and will grow more so. — Página: 998

We see this now on a large scale, with AI helping design better chips and production techniques that enable more sophisticated forms of AI and so on. — Página: 1002

In the era of abundant venture capital, distinguishing shiny objects from genuine breakthroughs is not so straightforward. — Página: 1011

Nonetheless, understanding the coming wave is not about making a snap judgment about where things will be this or that year; it is about closely tracking the development of multiple exponential curves over decades, projecting them into the future, and asking what that means. — Página: 1017

Deep learning uses neural networks loosely modeled on those of the human brain. In simple terms, these systems “learn” when their networks are “trained” on large amounts of data. — Página: 1030

Everywhere you look, software has eaten the world, opening the path for collecting and analyzing vast amounts of data. — Página: 1066

Within a few years AIs will be able to talk about, reason over, and even act in the same world that we do. — Página: 1071

Nota: It this real?

A big part of what makes humans intelligent is that we look at the past to predict what might happen in the future. — Página: 1095

LLMs take advantage of the fact that language data comes in a sequential order. Each unit of information is in some way related to data earlier in a series. — Página: 1100

The challenge lies in designing an algorithm that “knows where to look” for signals in a given sentence. What are the key words, the most salient elements of a sentence, and how do they relate to one another? In AI this notion is commonly referred to as “attention.” — Página: 1103

In the attention map, every token bears some relationship to every token before it, and for a given input sentence the strength of this relationship describes something about the importance of that token in the sentence. — Página: 1109

Based on this, it then suggests which tokens should come next in the sequence, what output logically follows the input. In other words, it autocompletes what might come next. — Página: 1114

Nota: It is just autocompletting stuff

All of this is just the start. We are only beginning to scratch at the profound impact large language models are about to have. If DQN and AlphaGo were the early signs of something lapping at the shore, ChatGPT and LLMs are the first signs of the wave beginning to crash around us. — Página: 1132

Nota: As coisas mudarao rapidamente, mesmo!

This kind of vast, almost instantaneous consumption of information is not just difficult to comprehend; it’s truly alien. — Página: 1145

Nota: Muitos dados nao necessariamente corretos pode ser um problema

This is the reality of the coming wave. It advances at an unprecedented rate, taking even its proponents by surprise. — Página: 1158

To get a sense of one petaFLOP, imagine a billion people each holding a million calculators, doing a complex multiplication, and hitting “equals” at the same time. I find this extraordinary. — Página: 1164

While this is true, it misses the fact that in AI training we can just keep connecting larger and larger arrays of chips, daisy-chaining them into massively parallel supercomputers. — Página: 1170

Researchers meanwhile see more and more evidence for “the scaling hypothesis,” which predicts that the main driver of performance is, quite simply, to go big and keep going bigger. — Página: 1172

It’s true that we are, more widely, complex emotional and social beings. But humans’ ability to complete given tasks—human intelligence itself—is very much a fixed target, as large and multifaceted as it is. Unlike the scale of available compute, our brains do not radically change year by year. In time this gap will be closed. — Página: 1182

At the other end of the spectrum, you can now create a nanoLLM based on just three hundred lines of code capable of generating fairly plausible imitations of Shakespeare. In short, AI increasingly does more with — Página: 1195

Over the next decade, there will almost certainly be dramatic capability increases, even as costs further decline by multiple orders of magnitude. Progress is accelerating so much that benchmarks get eclipsed before new ones are even made. — Página: 1198

EleutherAI, — Página: 1201

Nota: Procurar por essa empresa

burgeoning field of generative AI. — Página: 1209

The potential for harm, abuse, and misinformation is real. But the positive news is that many of these issues are being improved with larger and more powerful models. — Página: 1225

Nota: Is this true?!

but rather that AI had reached a point where it could convince otherwise intelligent people—indeed, someone with a real understanding of how it actually worked—that it was conscious. — Página: 1266

Nota: As cezes saber cono algo funciona nao significa que voce compreende aquilo e menos ainda que suas crencas nao sao cpazes de afetar seu julgamento.

There’s a recurrent problem with making sense of progress in AI. We quickly adapt, even to breakthroughs that astound us initially, and within no time they seem routine, even mundane. — Página: 1271

In the words of John McCarthy, who coined the term “artificial intelligence”: “As soon as it works, no one calls it AI anymore.” AI is—as those of us building it like to joke—“what computers can’t do.” Once they can, it’s just software. — Página: 1273

The eminent professor of complexity Melanie Mitchell rightly points out that present-day AI systems have many limitations: they can’t transfer knowledge from one domain to another, provide quality explanations of their decision-making process, and so on. — Página: 1281

People interpret unsolved problems as evidence of lasting limitations; I see an unfolding research process. — Página: 1285

Over the last decade, intellectual and political elites in tech circles became absorbed by the idea that a recursively self-improving AI would lead to an “intelligence explosion” known as the Singularity. — Página: 1301

Debating timelines to AGI is an exercise in reading crystal balls. — Página: 1305

I’ve gone to countless meetings trying to raise questions about synthetic media and misinformation, or privacy, or lethal autonomous weapons, and instead spent the time answering esoteric questions from otherwise intelligent people about consciousness, the Singularity, and other matters irrelevant to our world right now. — Página: 1307

Nota: Algumas questoes deveriam ser prioritarias.

AGI is binary—you either have it or you don’t, a single, identifiable threshold that would be crossed by a given system. I’ve always thought that this characterization is wrong. Rather, it’s a gradual transition, where AI systems become increasingly capable, consistently nudging toward AGI. It’s not a vertical takeoff so much as a smooth evolution already underway. — Página: 1310

All that matters is what the system can do. Focus on that, and the real challenge comes into view: systems can do more, much more, with every passing day. — Página: 1315

In a paper published in 1950, the computer scientist Alan Turing suggested a legendary test for whether an AI exhibited human-level intelligence. When AI could display humanlike conversational abilities for a lengthy period of time, such that a human interlocutor couldn’t tell they were speaking to a machine, the test would be passed: the AI, conversationally akin to a human, deemed intelligent. — Página: 1318

We don’t just care about what a machine can say; we also care about what it can do. — Página: 1324

Nota: At the moment there is not a lot of use cases outside the narrow vison of it.

Put simply, passing a Modern Turing Test would involve something like the following: an AI being able to successfully act on the instruction “Go make $1 mi l l i o n o n A ma z o nina f e w m o n t h s w i t hj u s t a$ 100,000 investment.” — Página: 1327

Should my Modern Turing Test for the twenty-first century be met, the implications for the global economy are profound. — Página: 1334

We’ll come to robots later, but the truth is that for a vast range of tasks in the world economy today all you need is access to a computer; — Página: 1338

The challenge is in advancing what AI developers call hierarchical planning, stitching multiple goals and subgoals and capabilities into a seamless process toward a singular end. — Página: 1340

Nota: When this is possible, we might have a greate problem

I think of this as “artificial capable intelligence” (ACI), the point at which AI can achieve complex goals and tasks with minimal oversight. — Página: 1348

ACI represents the next stage of AI’s evolution. A system that not only could recognize and generate novel images, audio, and language appropriate to a given context, but also would be interactive—operating in real time, with real users. — Página: 1354

All of this, then, enables tool use and the emergence of real capability to perform a wide range of complex, useful actions. It adds up to a genuinely capable AI, an ACI. — Página: 1359

The future of AI is, at least in one sense, fairly easy to predict. Over the next five years, vast resources will continue to be invested. Some of the smartest people on the planet are working on these problems. Orders of magnitude more computation will train the top models. All of this will lead to more dramatic leaps forward, including breakthroughs toward AI that can imagine, reason, plan, and exhibit common sense. — Página: 1365

It won’t be long before AI can transfer what it “knows” from one domain to another, seamlessly, as humans do. — Página: 1368

It’s not just a tool or platform but a transformative meta-technology, the technology behind technology and everything else, itself a maker of tools and platforms, not just a system but a generator of systems of any and all kinds. — Página: 1372

Just as everything from the steam engine to the microprocessor was driven by an intense dialogue between physics and engineering, so the coming decades will be defined by a convergence of biology and engineering. — Página: 1385

Much of civilization would have been impossible without selective breeding—the insistent process of refining crops and animals to select for more desirable traits. — Página: 1393

The Economist calls the Carlson curve: the epic collapse in costs for sequencing DNA. — Página: 1414

CRISPR gene editing (the acronym stands for clustered regularly interspaced short palindromic repeats) — Página: 1420

CRISPR edits DNA sequences with the help of Cas9, an enzyme acting as a pair of finely tuned DNA scissors, cutting parts of a DNA strand for precise genetic editing and modification of anything ranging from a minute bacterium to large mammals like human beings, with edits anywhere from tiny changes to significant interventions in the genome. — Página: 1423

After the initial CRISPR paper was published, progress applying it was rapid; the first gene-edited plants were created within a year, the first animals—mice—even before that. — Página: 1427

CRISPR use cases are multiplying, from tomatoes ultrarich in vitamin D to treatments for conditions including sickle-cell disease and beta-thalassemia (a blood disorder producing abnormal hemoglobin). — Página: 1433

Companies like the Odin will sell you a genetic engineering kit including live frogs and crickets for $1,999, while another kit includes a mini-centrifuge, a polymerase chain reaction machine, and all the reagents and materials you need to get going. — Página: 1439

Nota: This is both incredible and dangerous

If sequencing is reading, synthesizing is writing. And writing doesn’t just involve reproducing known strands of DNA; it also enables scientists to write new strands, to engineer life itself. — Página: 1447

Where nature takes a long and winding path to reach extraordinarily effective results, this bio-revolution puts the power of concentrated design at the heart of these self-replicating, self-healing, and evolving processes. — Página: 1458

Biotech is following a similar trajectory, only at a much earlier phase; organisms will soon be designed and produced with the precision and scale of today’s computer chips and software. — Página: 1467

Nota: Isso e interessante e assustador ao mesmo tempo

Countless experiments are underway in the strange and emerging landscape of synthetic biology: viruses that produce batteries, proteins that purify dirty water, organs grown in vats, algae that draw down carbon from the atmosphere, plants that consume toxic waste. — Página: 1476

No one can fully say what the consequences might be. — Página: 1480

Nota: E as vezes isso poder ir no caminho errado

Eventually, it might be possible to reconfigure ourselves to enhance our immune responses. That, in turn, might open the door to even more ambitious experimentation like longevity and regenerative technologies, already a burgeoning area of research. — Página: 1489

A world where life spans are set to average a hundred years or more is achievable in the next decades. Nor is this just about longer life; it’s about healthier lives as we get older. — Página: 1497

It won’t be long before “gene doping” becomes a live issue in sports, education, and professional life. — Página: 1501

Next-generation DNA printers will produce DNA with an increasing degree of precision. If improvements can be made in not only expressing that DNA but then using it to genetically engineer a diverse array of new organisms, automating and scaling the processes, a device or set of devices could, theoretically, produce an enormous range of biological materials and constructions using only a few basic inputs. — Página: 1523

There is still a long way to go before this technology can be harnessed. But all the functional parts of a computer—data storage, information transmission, and a basic system of logic—can in principle be replicated using biological materials. — Página: 1532

Forty-five percent of the global disease burden could be met with “science that is conceivable today.” — Página: 1536

Understand proteins, and you’ve taken a giant leap forward in understanding—and mastering—biology. But there’s a problem. Simply knowing the DNA sequence isn’t enough to know how a protein works. Instead, you need to understand how it folds. Its shape, formed by this knotted folding, is core to its function: collagen in our tendons has a rope-like structure, while enzymes have pockets to hold the molecules they act on. — Página: 1541

Whereas once it might have taken researchers weeks or months to determine a protein’s shape and function, that process can now begin in a matter of seconds. This is what we mean by exponential change. This is what the coming wave makes possible. — Página: 1571

All intelligence to date has come from life. Call them synthetic intelligence and artificial life and they still mean the same thing. Both fields are about re-creating, engineering these utterly foundational and interrelated concepts, two core attributes of humanity; change the view and they become one single project. — Página: 1576

It likely won’t be too long before neural “laces” made from carbon nanotubes plug us directly into the digital world. — Página: 1590

Technological waves are bigger than just one or two general-purpose technologies. They are clusters of technologies arriving at around the same time, anchored by one or more general-purpose technologies but extending far beyond them. — Página: 1599

Rather, they develop in rippling amplificatory loops. Where you find a general-purpose technology, you also find other technologies developing in constant dialogue, spurred on by it. — Página: 1606

The John Deere company still makes agricultural technology today. You might be thinking tractors, sprinklers, and combines, and it’s true that John Deere does make all these things. Increasingly, though, the company builds robots. — Página: 1624

As with AI, robotics proved much more difficult in practice than early engineers assumed. The real world is a strange, uneven, unexpected, and unstructured environment, exquisitely sensitive to things like pressure: — Página: 1639

But as we’ve seen in so many other applications of machine learning, what starts with close human supervision ends up with the AI learning to do the task better by itself, eventually generalizing to new settings. — Página: 1653

Robots can operate with precision in a far greater range of environments for far longer periods than humans. — Página: 1669

I think we’re now getting to the point where AI is pushing robots toward their original promise: machines that can replicate all the physical actions of a human and more. — Página: 1671

Nota: Maybe they are capable, but this does bot mean that they are easily deployable

AIs are products of bits and code, existing within simulations and servers. Robots are their bridge, their interface with the real world. — Página: 1690

While very much a nascent technology, there are huge implications when quantum computing does materialize. Its key attraction is that each additional qubit doubles a machine’s total computing power. — Página: 1702

In theory, solving any optimization problem could be greatly sped up—almost anything that involves minimizing costs in complex circumstances, whether that’s efficiently loading a truck or running a national economy. — Página: 1714

Energy rivals intelligence and life in its fundamental importance. Modern civilization relies on vast amounts of it. — Página: 1727

In 2000, solar energy cost $4.88 per watt, but by 2019 it had fallen to just 38 cents. — Página: 1737

As the elements of AI, advanced biotechnology, quantum computing, and robotics combine in new ways, prepare for breakthroughs like advanced nanotechnology, a concept that takes the ever-growing precision of technology to its logical conclusion. — Página: 1755

The dream of the physical universe rendered a completely malleable platform, the plaything of tiny, dexterous nanobots or effortless replicators, is still the province, — Página: 1765

At its core, the coming wave is a story of the proliferation of power. If the last wave reduced the costs of broadcasting information, this one reduces the costs of acting on it, giving rise to technologies that go from sequencing to synthesis, reading to writing, editing to creating, imitating conversations to leading them. — Página: 1769

We are reaching the decisive point of what, in geological or human evolutionary timescales, is a technological explosion unfolding in successive waves, a compounding, accelerating cycle of innovation steadily getting faster and more impactful, breaking first over a period of thousands of years, then hundreds of years, and now single years or even months. — Página: 1778

Nota: Things are moving fast, too fast i dare to say

Drones and AI played a small but important part in the early days of the conflict in Ukraine, new technologies with a pronounced asymmetric potential that closed some — Página: 1811

The real question is what this means for conflict when production costs fall by another order of magnitude and capabilities multiply. — Página: 1818

Nota: This might be a problem

The coming wave is, however, characterized by a set of four intrinsic features compounding the problem of containment. — Página: 1822

First among them is the primary lesson of this section: hugely asymmetric impact. — Página: 1823

Second, they are developing fast, a kind of hyper-evolution, iterating, improving, and branching into new areas at incredible speed. — Página: 1825

Third, they are often omni-use; that is, they can be used for many different purposes. — Página: 1826

And fourth, they increasingly have a degree of autonomy beyond any previous technology. — Página: 1826

Emerging technologies have always created new threats, redistributed power, and removed barriers to entry. — Página: 1830

Nota: Asymetrical technilogy and power

In the words of the security expert Audrey Kurth Cronin, “Never before have so many had access to such advanced technologies capable of inflicting death and mayhem.” — Página: 1838

A single AI program can write as much text as all of humanity. A single two-gigabyte image-generation model running on your laptop can compress all the pictures on the open web into a tool that generates images with extraordinary creativity and precision. — Página: 1846

Nota: These are examples of ghe great vimpsct that technology might have.

However many safeguards and security protocols are in place, the scale of impact is far wider than we’ve seen before. — Página: 1857

AI creates asymmetric risks beyond those of a bad batch of food, a plane accident, or a faulty product. Its risks extend to entire societies, making it not so much a blunt tool as a lever with global consequences. — Página: 1858

Nota: It is a toolt that can act globaly. It already does.

And we already live in an age of interlinked global systems. In the coming wave a single point—a given program, a genetic change—can alter everything. — Página: 1861

If you want to contain technology, you might hope it develops at a manageable pace, giving society time and space to understand and adapt to it. — Página: 1863

Put simply, innovation in the “real world” could start moving at a digital pace, in near-real time, with reduced friction and fewer dependencies. — Página: 1878

You will be able to experiment in small, speedy, malleable domains, creating near-perfect simulations, and then translate them into concrete products. And then do it again, and again, learning, evolving, and improving at rates previously impossible in the expensive, static world of atoms. — Página: 1879

Nota: os this the concepg or busebof digital twin?

The physicist César Hidalgo argues that configurations of matter are significant because of the information they contain. — Página: 1881

Just as today’s models produce detailed images based on a few words, so in decades to come similar models will produce a novel compound or indeed an entire organism with just a few natural language prompts. — Página: 1897

This is what we mean by hyper-evolution—a fast, iterative platform for creation. — Página: 1901

Nota: Each technology fuel the next one

Dual-use technologies are those with both civilian and military applications. — Página: 1917

Dual-use technologies are both helpful and potentially destructive, tools and weapons. — Página: 1922

And the more powerful the technology, the more concern there should be about how many uses it might have. — Página: 1926

Omni-use technologies like steam or electricity have wider societal effects and spillovers than narrower technologies. — Página: 1930

It’s still very early days, and truly general systems are still some way off, but at some point these capabilities will expand to many thousands of activities. — Página: 1937

Nota: We might see what these general porpuse systems looks like in the near future.

Over time, technology tends toward generality. What this means is that weaponizable or harmful uses of the coming wave will be possible regardless of whether this was intended. — Página: 1944

What’s different about the coming wave is how quickly it is being embedded, how globally it spreads, how easily it can be componentized into swappable parts, and just how powerful and above all broad its applications could be. — Página: 1948

Researchers in the field categorize autonomy from level 0, no autonomy whatsoever, to level 5, where a vehicle can drive itself under all conditions and the driver simply inputs a destination and then can fall happily asleep. — Página: 1963

Keeping humans “in the loop,” as the saying goes, is desirable, but optional. — Página: 1968

Nota: This might ve the case for a lot of tasks

A paradox of the coming wave is that its technologies are largely beyond our ability to comprehend at a granular level yet still within our ability to create and use. — Página: 1985

Autonomous systems can and may be explainable, but the fact that so much of the coming wave operates at the edge of what we can understand should give us pause. We won’t always be able to predict what these autonomous systems will do next; that’s the nature of autonomy. — Página: 1989

Humans dominate our environment because of our intelligence. A more intelligent entity could, it follows, dominate us. The AI researcher Stuart Russell calls it the “gorilla problem”: gorillas are physically stronger and tougher than any human being, but it is they who are endangered or living in zoos; they who are contained. — Página: 1999

An “intelligence explosion” is the point at which an AI can improve itself again and again, recursively making itself better in ever faster and more effective ways. — Página: 2004

Nobody really knows how we can contain the very features being researched so intently in the coming wave. — Página: 2008

the real question is why it’s so hard to see it as anything other than inevitable. — Página: 2013

This implication of national rivalry was an aspect of the contest I soon came to regret. — Página: 2027

Nota: Things ae not as simple as we thibk

it became clear AlphaGo was part of a much bigger story than one trophy, system, or company; it was that of great powers engaging in a new and dangerous game of technological competition—and a series of overwhelmingly powerful and interlocking incentives that ensure the coming wave really is coming. — Página: 2046

From curiosity to crisis, fortune to fear, at its heart technology emerges to fill human needs. — Página: 2050

The first driver has to do with what I experienced with AlphaGo: great power competition. — Página: 2055

Second comes a global research ecosystem with its ingrained rituals rewarding open publication, curiosity, and the pursuit of new ideas at all costs. — Página: 2057

Then come the immense financial gains from technology and the urgent need to tackle our global social challenges. — Página: 2058

And the final driver is perhaps the most human of all: ego. — Página: 2059

China is already ahead of the United States in green energy, 5G, and AI and is on a trajectory to overtake it in quantum and biotech in the next few years. — Página: 2125

Nota: China is rising sharply

Xi was stating a bald truth, the self-declared mantra of not just China but virtually every state, from superpower leaders at the frontier to isolated pariahs: who builds, owns, and deploys technology matters. — Página: 2134

Technology has become the world’s most important strategic asset, not so much the instrument of foreign policy as the driver of it. — Página: 2137

But the reality is that the logic of nation-states is at times painfully simple and yet utterly inevitable. — Página: 2146

The debate now isn’t whether we are in a technological and AI arms race; it’s where it will lead. — Página: 2153

Nota: This debate seams to be treated more serious as time goes on.

Grandes modelos de linguagem are still seen as cutting-edge, yet there is no great magic or hidden state secret to them. Access to computation is likely the biggest bottleneck, but plenty of services exist to make it happen. The same goes for CRISPR or DNA synthesis. — Página: 2180

happening. It is a point so obvious it doesn’t often get mentioned: there is no central authority controlling what technologies get developed, who does it, and for what purpose; technology is an orchestra with no conductor. — Página: 2187

Raw curiosity, the quest for truth, the importance of openness, evidence-based peer review—these are core values for scientific and technological research. — Página: 2193

Openness is science and technology’s cardinal ideology. What is known must be shared; what is discovered must be published. — Página: 2197

Moreover, these days papers are announced on Twitter and often written with social media influence in mind. They are designed to be eye-catching and attract attention. — Página: 2209

Nota: This might be a huge problem

Academics fervently argue for open access to their research. In tech, strong norms around sharing and contributing support a flourishing space of open-source software. — Página: 2211

Nota: The world is starting to be more open?

All in all, to a degree that is perhaps underappreciated, publication and sharing aren’t just about the process of falsification in science. They’re also for prestige, for peers, for the pursuit of a mission, for the sake of a job, for likes. — Página: 2218

All these companies, those most keenly investing in the coming wave, those with the most lavish budgets, have a track record of openly publishing their research. — Página: 2230

That’s no discredit to him; it just shows that no one can ever be sure of anything at the research frontier. — Página: 2247

Science and technology research is inherently unpredictable, exceptionally open, and growing fast. Governing or controlling it is therefore immensely difficult. — Página: 2254

The railway boom of the 1840s was “arguably the greatest bubble in history.” But in the annals of technology, it is more norm than exception. There was nothing inevitable about the coming of the railways, but there was something inevitable about the chance to make money. — Página: 2280

Science has to be converted into useful and desirable products for it to truly spread far and wide. Put simply: most technology is made to earn money. — Página: 2285

they see them as ways to make a profit. And this, the potential for profit, is built on something even more long-lasting and robust: raw demand. People both want and need the fruits of technology. — Página: 2291

The sheer breadth of human wants and needs, and the countless opportunities to profit from them, are integral to the story of technology and will remain so in the future. — Página: 2295

In developed economies, people work far less than they used to for far more reward. In Germany, for example, annual working hours have decreased by nearly 60 percent since 1870. — Página: 2304

Nota: What are othe data about work?

Here’s a slight correction: much of what we see around us is powered by human intelligence in direct pursuit of monetary gain. — Página: 2308

When a corporation automates insurance claims or adopts a new manufacturing technique, it creates efficiency savings or improves the product, boosting profits and attracting new customers. Once an innovation delivers a competitive advantage like this, everyone must either adopt it, leapfrog it, switch focus, or lose market share and eventually go bust. The attitude around this dynamic in technology businesses in particular is simple and ruthless: build the next generation of technology or be destroyed. — Página: 2314

All of the world’s knowledge, best practices, precedent, and computational power will be available, tailored to you, to your specific needs and circumstances, instantaneously and effortlessly. — Página: 2343

Nota: Does it really?

Little is ultimately more valuable than intelligence. Intelligence is the wellspring and the director, architect, and facilitator of the world economy. The more we expand the range and nature of intelligences on offer, the more growth should be possible. — Página: 2347

In 1945, around 50 percent of the world’s population was seriously undernourished. Today, despite a population well over three times bigger, that’s down to 10 percent. — Página: 2370

food sufficient, if not adequately distributed, for the planet’s eight billion and rising human inhabitants. — Página: 2376

Nota: The problem is not on the vproduction side, but rather on the distribuition side.

We pursue new technologies, including those in the coming wave, not just because we want them, but because, at a fundamental level, we need them. — Página: 2378

The energy scholar Vaclav Smil calls ammonia, cement, plastics, and steel the four pillars of modern civilization: the material base underwriting modern society, each hugely carbon-intensive to produce, with no obvious successors. — Página: 2390

If we are to stand any chance of keeping global warming under two degrees Celsius, then the world’s scientists working under the UN’s Intergovernmental Panel on Climate Change have been clear: carbon capture and storage is an essential technology. — Página: 2398

Technology can and will improve lives and solve problems. — Página: 2407

A school of naive techno-solutionism sees technology as the answer to all of the world’s problems. Alone, it’s not. How it is created, used, owned, and managed all make a difference. — Página: 2416

The coming wave is coming partly because there is no way through without it. — Página: 2422

Scientists and technologists are all too human. They crave status, success, and a legacy. They want to be the first and best and recognized as such. — Página: 2425

polymathic Hungarian American John von Neumann. “What we are creating now,” he said, “is a monster whose influence is going to change history, provided there is any history left, yet it would be impossible not to see it through, not only for military reasons, but it would also be unethical from the point of view of the scientists not to do what they know is feasible, no matter what terrible consequences it may have.” — Página: 2435

Nota: Isso e pesrigoso. Bastante perugoso

Making history, doing something that matters, helping others, beating others, impressing a prospective partner, impressing a boss, peers, rivals: it’s all in there, all part of the ever-present drive to take risks, explore the edges, go further into the unknown. Build something new. Change the game. Climb the mountain. — Página: 2441

Nota: This vision might br too dangsrous

Nationalism, capitalism, and science—these are, by now, embedded features of the world. Simply removing them from the scene is not possible in any meaningful time frame. — Página: 2449

Technology “emerges” through countless independent contributions all layering on top of one another, a metastasizing, entangled morass of ideas unraveling themselves, driven on by deep-rooted and dispersed incentives. — Página: 2455

Nota: Most of the time we have no ideia about where the ideias come from.

Technology is now an indispensable mega-system infusing every aspect of daily life, society, and the economy. — Página: 2464

This is the ultimate collective action problem. The idea that CRISPR or AI can be put back in the box is not credible. Until someone can create a plausible path to dismantling these interlocking incentives, the option of not building, saying no, perhaps even just slowing down or taking a different path isn’t there. — Página: 2471

At its heart, the nation-state, the central unit of the world’s political order today, offers its citizens a simple and highly persuasive bargain: that not only is centralization of power in the sovereign, territorial state possible but its benefits far outweigh the risks. — Página: 2485

On the one hand the most dystopian excesses of centralized power must be avoided, and on the other we must accept regular intervention to maintain order. — Página: 2493

In this third section of the book, we grapple with the profound consequences of these technologies for the nation-state and for the liberal democratic nation-state above all. — Página: 2501

We ended up with maximum divergence on all possible things. It was, in other words, politics as usual. — Página: 2519

The idea that technology alone can solve social and political problems is a dangerous delusion. But the idea that they can be solved without technology is also wrongheaded. — Página: 2542

While technology is still the single most powerful avenue for addressing the challenges of the twenty-first century, we cannot ignore downsides. — Página: 2554

Technology is not just a tool to support the bargain we’ve made in the nation-state; it is also a genuine threat to it. — Página: 2557

And anyone who thinks what happened in Syria could never happen “here” is kidding themselves; people are people wherever they are. — Página: 2563

Going into the coming wave, many nations are beset by a slew of major challenges battering their effectiveness, making them weaker, more divided, and more prone to slow and faulty decision-making. — Página: 2577

Democracies are built on trust. People need to trust that government officials, militaries, and other elites will not abuse their dominant positions. Everyone relies on the trust that taxes will be paid, rules honored, the interests of the whole put ahead of individuals. Without trust, from the ballot box to the tax return, from the local council to the judiciary, societies are in trouble. — Página: 2581

Nota: Rhis might not be true for every state. It looks to me that is more about power than trust

That so many people profoundly feel society is failing is itself a problem: Distrust breeds negativity and apathy. People decline to vote. — Página: 2593

Behind the new authoritarian impulse and political instability lies a growing pool of social resentment. — Página: 2597

Forty million people in the United States live in poverty, and more than five million live in “Third World conditions”—all within the world’s richest economy. — Página: 2603

Nota: Even the weath counties has its problems

Onshoring, national security, resilient supply chains, self-sufficiency—today’s language of trade is once again the language of borders, barriers, and tariffs. — Página: 2611

Every previous wave of technology has had profound political implications. We should expect the same in the future. — Página: 2621

Social media thrives on heightened emotions and, quite often, outrage. — Página: 2627

Too fast in its development, too global, too protean and enticing for any simple model of containment, strategically critical, relied upon by billions, modern technology itself is a prime actor, a monumental force nation-states struggle to manage. — Página: 2634

Nota: Technology is a problem in this case

Technology didn’t straightforwardly “cause” or create the modern state (or indeed any political structure). But the potential it unleashes is not neutral in that story. — Página: 2640

As the historian of technology Langdon Winner puts it, “Technology in its various manifestations is a significant part of the human world. Its structures, processes, and alterations enter into and become part of the structures, processes, and alterations of human consciousness, society, and politics.” In other words, technology is political. — Página: 2642

In this, technology is one of the key determinants of history, but never alone and never in a mechanistic, inherently predictable way. — Página: 2649

Nota: the impact might not be predicted at first

Technologies are ideas, manifested in products and services that have profound and lasting consequences for people, social structures, the environment, and everything in between. — Página: 2652

Put simply, technology and political order are intimately connected. The introduction of new technologies has major political consequences. — Página: 2668

What emerges will, I think, tend in two directions with a spectrum of outcomes in between. On one trajectory, some liberal democratic states will continue to be eroded from within, becoming a kind of zombie government. — Página: 2684

On another, unthinking adoption of some aspects of the coming wave opens pathways to domineering state control, creating supercharged Leviathans whose power goes beyond even history’s most extreme totalitarian governments. — Página: 2687

WannaCry tricked some users into opening an email, which released a “worm” replicating and transporting itself to infect a quarter of a million computers across 150 countries in just one day. — Página: 2716

Nota: That is scary

In other words, thanks to a basic failure of containment, a global superpower became a victim of its own powerful and supposedly secure technology. This is uncontained asymmetry in action. — Página: 2741

With the coming wave that assumption is a serious mistake. Such attacks demonstrate that there are those who would use cutting-edge technologies to degrade and disable key state functions. — Página: 2747

Nota: As technology progresses attacks can make more damage

This is, in part, a story of who can do what, a story of power and where it lies. — Página: 2761

Power is “the ability or capacity to do something or act in a particular way;…to direct or influence the behavior of others or the course of events.” — Página: 2763

Technology is ultimately political because technology is a form of power. — Página: 2766

As we saw in part 2, it will enable people to do things in the real world. I think of it like this: just as the costs of processing and broadcasting information plummeted in the consumer internet era, the cost of actually doing something, taking action, projecting power, will plummet with the next wave. Knowing is great, but doing is much more impactful. — Página: 2767

Nota: Been able to do things and not just know things is a dramatcally diffent thing

Wherever power is today, it will be amplified. Anyone with goals—that is, everyone—will have huge help in realizing them. — Página: 2774

Everyone will have a world-class team on their side and in their corner. — Página: 2781

Nota: Does it really? Been an expert is different than having only knowledge.

Democratizing access necessarily means democratizing risk. — Página: 2784

More sophisticated armed robots will further reduce barriers to violence. — Página: 2803

The cost of military-grade drones has fallen by three orders of magnitude over the last decade. By 2028, $26 billion a year will be spent on military drones, and at that point many are likely to be fully autonomous. — Página: 2811

Nota: That is scary

Imagine a huge cache of documents from a company leaked. A legal AI might be able to parse this against multiple legal systems, figure out every possible infraction, and then hit that company with multiple crippling lawsuits around the world at the same time. — Página: 2828

This new dynamic—where bad actors are emboldened to go on the offensive—opens up new vectors of attack thanks to the interlinked, vulnerable nature of modern systems: not just a single hospital but an entire health system can be hit; not just a warehouse but an entire supply chain. — Página: 2844

If the state began in new forms of war, perhaps it will end in the same way. — Página: 2856

Now powerful, asymmetric, omni-use technologies are certain to reach the hands of those who want to damage the state. — Página: 2860

Eventually, the balance might be restored, but not before a wave of immensely destabilizing force is unleashed. And as we’ve seen, the nature of the threat is far more widespread than blunt forms of physical assault. — Página: 2866

Ask yourself, what happens when anyone has the power to create and broadcast material with incredible levels of realism? — Página: 2878

These fakes will be so good our rational minds will find it hard to accept they aren’t real. — Página: 2881

Nota: What is real is almost impossivle to know

Anyone motivated to sow instability now has an easier time of it. — Página: 2887

The threat here lies not so much with extreme cases as in subtle, nuanced, and highly plausible scenarios being exaggerated and distorted. — Página: 2891

Nota: Somethong plausible is more dangerous than something outrageous

Not everyone would buy it, but those who wanted to believe would find it utterly compelling. — Página: 2899

Nota: Deepfakes needs to be just real enought

AI-enhanced digital tools will exacerbate information operations like these, meddling in elections, exploiting social divisions, and creating elaborate astroturfing campaigns to sow chaos. — Página: 2912

Eighty-two percent of influential users advocating for “reopening America” were bots. — Página: 2917

Cue an “Infocalypse,” the point at which society can no longer manage a torrent of sketchy material, where the information ecosystem grounding knowledge, trust, and social cohesion, the glue holding society together, falls apart. — Página: 2926

Even as the number of BSL-4 labs booms, only a quarter of them score highly on safety, according to the Global Health Security Index. — Página: 2963

Nothing should get out. Yet pathogens do, time and again. Despite being some of the toughest around, the protocols, technologies, and regulations for containment fail. — Página: 2970

Few areas of biology are as controversial as gain-of-function (GOF) research. Put simply, gain-of-function experiments deliberately engineer pathogens to be more lethal or infectious, or both. — Página: 2974

One way of understanding how it might happen—that is, how viruses might become more lethal and transmissible at the same time—and how we might combat that is to, well, make it happen. That’s where gain-of-function research comes in. — Página: 2977

In late 2022, an NIH study at Boston University combined the original, more deadly strain of COVID with the spike protein of the more transmissible omicron variant. Many felt the research shouldn’t have gone ahead, but there it was, funded by public money. — Página: 2993

It’s about what goes wrong when powerful tools proliferate, what mistakes get made, what “revenge effects” unfurl, what random, unforeseen mess results from technology’s collision with reality. — Página: 2996

If every half-competent lab or even random biohacker can embark on this research, tragedy cannot be indefinitely postponed. — Página: 3003

Nota: This bmight be a huge problem of safety

As the power and spread of any technology grows, so its failure modes escalate. — Página: 3005

Yet stressors might also be less discrete events, less a robot attack, lab leak, or deepfake video, and more a slow and diffuse process undermining foundations. — Página: 3009

Maynard Keynes called “technological unemployment.” In Keynes’s view, this was a good thing, with increasing productivity freeing up time for further innovation and leisure. — Página: 3017

So far at least, in economic terms, new technologies have not ultimately replaced labor; they have in the aggregate complemented it. — Página: 3022

These tools will only temporarily augment human intelligence. They will make us smarter and more efficient for a time, and will unlock enormous amounts of economic growth, but they are fundamentally labor replacing. — Página: 3027

With each additional robot per thousand workers there is a decline in the employment-to-population ratio, and consequently a fall in wages. Today algorithms perform the vast bulk of equity trades and increasingly act across financial institutions, and yet, even as Wall Street booms, it sheds jobs as technology encroaches on more and more tasks. — Página: 3036

believe this rosy vision is implausible over the next couple of decades; automation is unequivocally another fragility amplifier. — Página: 3045

The number of people who can get a PhD in machine learning will remain tiny in comparison to the scale of layoffs. And, sure, new demand will create new work, but that doesn’t mean it all gets done by human beings. — Página: 3053

If your sense of self is wedded to a particular kind of work, it’s little consolation if you feel your new job demeans your dignity. — Página: 3059

New jobs might be created in the long term, but for millions they won’t come quick enough or in the right places. — Página: 3064

Whichever side of the jobs debate you fall on, it’s hard to deny that the ramifications will be hugely destabilizing for hundreds of millions who will, at the very least, need to re-skill and transition to new types of work. — Página: 3072

The full amplification of fragility is missed because it often appears as if these impacts were happening incrementally and in convenient silos. They are not. — Página: 3083

This is not, then, confined to amplifying specific points of fragility; it is, in the slightly longer term, about a transformation of the very ground on which society is built. And in this great redistribution of power, the state, already fragile and growing more so, is shaken to its core, its grand bargain left tattered and precarious. — Página: 3093

new technologies help create new centers of power with new social infrastructures both enabling them and supporting them. — Página: 3119

Nota: Most of the imes we do not know what are those technologies.

While small changes in technology can fundamentally alter the balance of power, trying to predict exactly how, decades into the future, is incredibly difficult. — Página: 3122

Recall that growing access to power means everyone’s power will be amplified. — Página: 3126

The nation-state will be subject to massive centrifugal and centripetal forces, centralization and fragmentation. — Página: 3135

Nota: Tuebulence is on the way

But what this misses is that the most powerful forces in the world are actually groups of individuals coordinating to achieve shared goals. — Página: 3158

Nota: Sometimes intelligence is spread between individuals.

Big tech companies provide tools for everything from organizing a birthday to running multimillion-dollar businesses. — Página: 3173

Nota: And this is probably a problem

The world’s top fifty cities have the lion’s share of wealth and corporate power (45 percent of big company HQs; 21 percent of world GDP) despite having only 8 percent of the world’s population. The top 10 percent of global firms take 80 percent of the total profits. — Página: 3182

For Koreans today, Samsung is almost like a parallel government, a constant presence throughout people’s lives. — Página: 3192

All the big tech platforms either are mainly service businesses or have very large service businesses. — Página: 3207

Nota: Services are better than standalone products.

In this scenario there will be just a few mega-players whose scale and power will begin to rival traditional states. — Página: 3225

Nota: Companiss can challange governmens

Whatever the end point, we are heading to a place where unprecedented powers and abilities are out there, in the hands of already powerful actors who’ll no doubt use them to amplify their reach and further their own agenda. — Página: 3234

Nota: Power is tge key for most players

Almost every detail of life is logged, somewhere, by those with the sophistication to process and act on the data they collect. — Página: 3275

Nota: This is done today. I think maknly as a way to sell the data to advsertisers.

Societies of overweening surveillance and control are already here, and now all of this is set to escalate enormously into a next-level concentration of power at the center. — Página: 3309

It’s no secret that governments monitor and control their own populations, but these tendencies extend deep into Western firms, too. — Página: 3317

Consider that a combination of AI, cheap robotics, and advanced biotech coupled with clean energy sources might, for the first time in modernity, make living “off-grid” nearly equivalent to being plugged-in. — Página: 3353

The biggest AI models will cost hundreds of millions of dollars to train, and consequently few will have ownership. But paradoxically a countertrend will play out in parallel. AI breakthroughs already make their way into open-source code repositories within days of being published in open-access journals, making topflight models easy for anyone to access, experiment with, build, and modify in turn. Models down to the weights are published, leaked, and stolen. — Página: 3370

It’s already a remarkable fact that corporate titans spend most of their lives working on software, like Gmail or Excel, accessible to most people on the planet. Extend that, radically, with the democratization of empowerment, when everyone on the planet has unfettered access to the most powerful technologies ever built. — Página: 3392

Nota: At times it seams like everyone has access to fhs same tools. Hoever they do not have the same power.

Understanding the future means handling multiple conflicting trajectories at once. — Página: 3426

AI systems already make critical decisions with overt political implications: who receives a loan, a job, a place at college, parole; who gets seen by a senior physician. — Página: 3432

In all cases, the additional stress and volatility, the unpredictable amplification of power, the wrenching disruption of radical new centers of capability, will further stress the foundation of the liberal democratic nation-state system. — Página: 3443

Nota: Stress and problemss might be greater

Catastrophic events that once took place over years and decades could happen in minutes, at the push of a button. With the coming wave, we are poised to take yet another such leap, expanding both the upper bound of risk and the number of avenues available to those seeking to unleash catastrophic force. — Página: 3471

And yet here’s the dilemma: the most secure solutions for containment are equally unacceptable, leading humanity down an authoritarian and dystopian pathway. — Página: 3482

Nota: The technology bused for containment is a problem to ?

Once militaries are fully automated, the barriers to entry for conflict will be far lower. — Página: 3519

Nota: This might be true qnd a huge problem for the world since there is majod conflictsbhsppeninf today.

What if multiple such pathogens were released at once? This goes far beyond fragility amplification; it would be an unfathomable calamity. — Página: 3533

Nota: The next pandemic is on the way.

As AI gets more and more powerful, the most extreme scenarios will require serious consideration and mitigation. However, well before we get there, much could go wrong. — Página: 3547

Nota: Agibis a distant problem. We have a lot more to Worry now.

AI is both valuable and dangerous precisely because it’s an extension of our best and worst selves. And as a technology premised on learning, it can keep adapting, probing, producing novel strategies and ideas potentially far removed from anything before considered, even by other AIs. — Página: 3553

We don’t yet know the implications of AI for fields as diverse as agriculture, chemistry, surgery, and finance. That’s part of the problem; we don’t know what failure modes are being introduced and how deep they could extend. — Página: 3563

Nota: Should we not find what are the problems and risks?

Solving the question of AI alignment doesn’t mean doing so once; it means doing it every time a sufficiently powerful AI is built, wherever and whenever that happens. — Página: 3571

As the coming wave matures, however, the tools of destruction will, as we’ve seen, be democratized and commoditized. — Página: 3610

Nota: When technology spead sp does the dangers

Regardless of where we are with BSL-4 protocols or regulatory proposals or technical publications on the AI alignment problem, those incentives grind away, the technologies keep developing and diffusing. — Página: 3620

Nota: Safedy comss after the technology development.

Technology has penetrated our civilization so deeply that watching technology means watching everything. — Página: 3634

The door to dystopia is cracked open. Indeed, in the face of catastrophe, for some dystopia may feel like a relief. — Página: 3641

What level of societal control is appropriate to stopping an engineered pandemic? What level of interference in other countries is appropriate toward the same end? The consequences for liberty, sovereignty, and privacy have never been so potentially painful. — Página: 3653

Nota: Fhere is a lsvel where this thibgs are good?

The philosopher of technology Lewis Mumford talked about the “megamachine,” where social systems combine with technologies to form “a uniform, all-enveloping structure” that is “controlled for the benefit of depersonalized collective organizations.” — Página: 3669

Despite the weightlessness of the digital world, there’s a solidness and a profusion to the material world around us. It shapes our everyday expectations. — Página: 3682

Twenty-first-century civilization is a long way from the Maya, naturally, but the pressures of a huge and hungry superstructure, a large population, the hard limits of energy and civilizational capacity have not magically gone away; they’ve just been kept at bay. — Página: 3693

Nota: Limits exist. That can be good or bad.

Our entire edifice is premised on the idea of long-term economic growth. And long-term economic growth is ultimately premised on the introduction and diffusion of new technologies. — Página: 3697

it’s worth repeating: solving problems like climate change, or maintaining rising living and health-care standards, or improving education and opportunity is not going to happen without delivering — Página: 3705

Nota: Not sure that I agree with this. If looks to me that thdre is enought technolofy, awhat si lacking is distribuition. Both in technological and finantial terms.

This is not only about numbers but about expertise, tax base, and investment levels; retirees will be pulling money out of the system, not investing it for the long term. — Página: 3718

Even if it were possible, the idea of stopping the coming wave isn’t a comforting thought. Maintaining, let alone improving, standards of living needs technology. Forestalling a collapse needs technology. The costs of saying no are existential. And yet every path from here brings grave risks and downsides. This is the great dilemma. — Página: 3738

I’m still convinced that technology remains a primary driver for making improvements to our world and our lives. — Página: 3749

Nota: That is the main point of view from the author

My profound worry is that technology is demonstrating the real possibility to sharply move net negative, that we don’t have answers to arrest this shift, and that we’re locked in with no way out. — Página: 3754

Nota: Another pomit of view from the author.

Exponential change is coming. It is inevitable. That fact needs to be addressed. — Página: 3775

On paper regulation looks enticing, even obvious and straightforward; suggesting it lets people sound smart, concerned, and even relieved. — Página: 3788

Nota: It does sound easy. Buf necessary.

Talking about the ethics of machine learning systems is a world away from, say, the technical safety of synthetic bio. These discussions happen in isolated, echoey silos. They rarely break out. — Página: 3810

The central problem for humanity in the twenty-first century is how we can nurture sufficient legitimate political power and wisdom, adequate technical mastery, and robust norms to constrain technologies to ensure they continue to do far more good than harm. How, in other words, we can contain the seemingly uncontainable. — Página: 3821

Regulators regulate for things they can anticipate. This, meanwhile, is an age of surprises. — Página: 3835

Regulation doesn’t just rely on the passing of a new law. It is also about norms, structures of ownership, unwritten codes of compliance and honesty, arbitration procedures, contract enforcement, oversight mechanisms. All of this needs to be integrated and the public needs to buy in. — Página: 3859

Nota: Regulation needs people that comply to them.

Regulation is not enough, but at least it’s a start. Bold steps. A real understanding of the stakes involved in the coming wave. In a world where containment seems like it’s not possible, all of this gestures toward a future where it might be. — Página: 3894

Contained technology is technology whose modes of failure are known, managed, and mitigated, a situation where the means to shape and govern technology escalate in parallel with its capabilities. — Página: 3906

Before outlining a strategy it’s worth asking the following kinds of questions to prompt promising avenues: — Página: 3915

Is the technology omni-use and general-purpose or specific? — Página: 3916

Is the tech moving away from atoms toward bits? — Página: 3919

Are price and complexity coming down, and if so how fast? — Página: 3922

Are there viable alternatives ready to go? — Página: 3924

Does the technology enable asymmetric impact? — Página: 3926

Does it have autonomous characteristics? — Página: 3928

Does it confer outsized geopolitical strategic advantage? — Página: 3930

Does it favor offense or defense? — Página: 3932

Are there resource or engineering constraints on its invention, development, and deployment? — Página: 3934

The reality is, we have often not controlled or contained technologies in the past. And if we want to do so now, it would take something dramatically new, an all-encompassing program of safety, ethics, regulation, and control that doesn’t even really have a name and doesn’t seem possible in the first place. — Página: 3946

One challenge in even beginning to have this conversation is that technology, in the popular imagination, has become associated with a narrow band of often superfluous applications. “Technology” now mostly means social media platforms and wearable gadgets to measure our steps and heart rate. — Página: 3952

Nota: That is the problem of saying for example that someone works at a technology company. That are a lot of technology outside the so called technology compnay

Technology is not a niche; it is a hyper-object dominating human existence. — Página: 3956

“Technology” is not, on the face of it, a problem in the same sense as a heating planet. And yet it might be. — Página: 3973

Technical safety, up close, in the code, in the lab, is the first item on any containment agenda. — Página: 4024

Frontier AI safety research is still an undeveloped, nascent field focusing on keeping ever more autonomous systems from superseding our ability to understand or control them. — Página: 4041

And yet safety decisions made today will alter the future course of technology and humanity. — Página: 4051

Concretely, a good proposal for legislation would be to require that a fixed portion—say, a minimum of 20 percent—of frontier corporate research and development budgets should be directed toward safety efforts, with an obligation to publish material findings to a government working group so that progress can be tracked and shared. — Página: 4053

Rather than give an AI a set of fixed external objectives contained in what’s known as a written constitution, he recommends that systems gingerly infer our preferences and ends. They should carefully watch and learn. In theory, this should leave more room for doubt within systems and avoid perverse outcomes. — Página: 4085

The highest-level challenge, whether in synthetic biology, robotics, or AI, is building a bulletproof off switch, a means of closing down any technology threatening to run out of control. — Página: 4096

Actually having meaningful oversight and enforceable rules and reviewing technical implementations are vital. Technical safety advances and regulation will struggle to be effective if you can’t verify that they are working as intended. — Página: 4109

There has to be a well-defined, legal route to checking any new technology under the hood, in the code, in the lab, in the factory, or out in the wild. — Página: 4156

The wave can be slowed, at least for some period of time and in some areas. — Página: 4197

Buying time in an era of hyper-evolution is invaluable. Time to develop further containment strategies. Time to build in additional safety measures. Time to test that off switch. Time to build improved defensive technologies. Time to shore up the nation-state, regulate better, or even just get that bill passed. Time to knit together international alliances. — Página: 4198

So, as negative impacts become clear, we must use these choke points to create sensible rate-limiting factors, checks on the speed of development, to better ensure that good sense is implemented as fast as the science evolves. — Página: 4219

More than anyone else, those working on technology need to be actively working to solve the problems described in this book. The burden of proof and the burden of solutions rest on them, on us. — Página: 4230

The closer you are to a technology’s beating heart, the more you can affect outcomes, steer it in more positive directions, and block harmful applications. But this means also being part of what makes it a reality—for all the good and for all the harm it may do. — Página: 4252

Nota: This is ghe core os responsability.

In a world of entrenched incentives and failing regulation, technology needs critics not just on the outside but at its beating heart. — Página: 4268

I believe that figuring out ways to reconcile profit and social purpose in hybrid organizational structures is the best way to navigate the challenges that lie ahead, but making it work in practice is incredibly hard. — Página: 4276

shareholder capitalism works because it is simple and clear, and governance models too have a tendency to default to the simple and clear. In the shareholder model, lines of accountability and performance tracking are quantified and very transparent. It may be possible to design more modern structures in theory, but operating them in practice is another story. — Página: 4302

Technology companies that have strong containment mechanisms and goals written in as a fiduciary duty are a next step. — Página: 4332

Containment needs a new generation of corporations. It needs founders and those working in tech to contribute positively to society. It also needs something altogether more difficult. It needs politics. — Página: 4336

Accountability is enabled by deep understanding. Ownership gives control. Both require governments to get their hands dirty. — Página: 4349

in the twenty-first century it doesn’t make sense to have cabinet positions addressing matters like the economy, education, security, and defense without a similarly empowered and democratically accountable position in technology. — Página: 4365

The most sophisticated AI systems or synthesizers or quantum computers should be produced only by responsible certified developers. — Página: 4383

Nota: This is a different for of regulation ghat is needed.

Taxation also needs to be completely overhauled to fund security and welfare as we undergo the largest transition of value creation—from labor to capital—in history. — Página: 4390

Countries no more like giving up power than companies like missing out on profit, and yet these are precedents to learn from, shards of hope in a landscape riven with resurgent techno-competition. — Página: 4431

Best practices are hence not corporate secrets, an edge over rival airlines: they’re enthusiastically implemented by competitors in the common interests of collective industry trust and safety. — Página: 4502

While the tech industry talks a big game when it comes to “embracing failure,” it rarely does so when it comes to privacy or safety or technical breaches. — Página: 4506

Nota: Embracing does not mean been open about its failures.

ask not just what doing no harm means in an age of globe-spanning algorithms and edited genomes but how that can be enacted daily in what are often morally ambiguous circumstances. — Página: 4543

So, if the invocation of the grand “we” is at present meaningless, it prompts an obvious follow-up: let’s build one. Throughout history change came about because people self-consciously worked for it. — Página: 4566

Nota: We need to byild the we that we want.

The “we” that builds technology is scattered, subject to a mass of competing and different national, commercial, and research incentives. The more the “we” that is subject to it speaks clearly in one voice, a critical public mass agitating for change, demanding an alignment of approaches, the better chance of good outcomes. — Página: 4581

Kevin is determined to use every tool to stop them from happening. His program is one of the most holistic containment strategies around. It’s built around three pillars: delay, detect, and defend. — Página: 4597

Technical safety: Concrete technical measures to alleviate possible harms and maintain control. 2. Audits: A means of ensuring the transparency and accountability of technology. 3. Choke points: Levers to slow development and buy time for regulators and defensive technologies. 4. Makers: Ensuring responsible developers build appropriate controls into technology from the start. 5. Businesses: Aligning the incentives of the organizations behind technology with its containment. 6. Government: Supporting governments, allowing them to build technology, regulate technology, and implement mitigation measures. 7. Alliances: Creating a system of international cooperation to harmonize laws and programs. 8. Culture: A culture of sharing learning and failures to quickly disseminate means of addressing them. 9. Movements: All of this needs public input at every level, including to put pressure on each component and make it accountable. — Página: 4614

Safe, contained technology is, like liberal democracy, not a final end state; rather, it is an ongoing process, a delicate equilibrium that must be actively maintained, constantly fought for and protected. There’s no moment when we say, aha, we’ve solved the problem of proliferating technology! Instead, it’s about finding a way through, ensuring sufficient numbers of people are committed to keeping the unending balance between openness and closure. — Página: 4648

We should all get comfortable with living with contradictions in this era of exponential change and unfurling powers. Assume the worst, plan for it, give it everything. Stick doggedly to the narrow path. Get a world beyond the elites engaged and pushing. If enough people start building that elusive “we,” those glimmers of hope will become raging fires of change. — Página: 4689

Too many visions of the future start with what technology can or might do and work from there. That’s completely the wrong foundation. Technologists should focus not just on the engineering minutiae but on helping to imagine and realize a richer, social, human future in the broadest sense, a complex tapestry of which technology is just one strand. — Página: 4764

Rafael Alves ITM

Explorador

A próxima onda

📚 A Próxima Onda

🎯 Resumo da Obra

🧠 Minha Experiência

📝 Notas de Conteúdo

🖍️ Destaques e Citações

🔗 Conexões do Cofre

📚 Referências Externas

Visão de gráfico

Sumário

Backlinks