WEBVTT

1
00:00:00.250 --> 00:00:01.487
[quiet room tone]

2
00:00:01.570 --> 00:00:06.690
A sent folder. A message
at three in the morning: short,

3
00:00:06.690 --> 00:00:10.367
a little cold, addressed
to someone who matters.

4
00:00:10.450 --> 00:00:12.367
You don't remember writing it.

5
00:00:12.450 --> 00:00:14.647
The record says you sent it.

6
00:00:14.730 --> 00:00:18.037
And the record is right. Your account did.

7
00:00:18.120 --> 00:00:19.440
[typewriter return]

8
00:00:19.440 --> 00:00:23.417
Records exist to tie an act to a hand.

9
00:00:23.500 --> 00:00:25.420
The Stasi, East Germany's

10
00:00:25.420 --> 00:00:28.607
secret police, worked to hide theirs.

11
00:00:28.690 --> 00:00:33.070
Their 1976 directive
listed anonymous letters

12
00:00:33.070 --> 00:00:36.167
among the proven methods of Zersetzung:

13
00:00:36.250 --> 00:00:40.537
covert pressure to fragment
and isolate a person.

14
00:00:40.620 --> 00:00:46.170
An AI assistant with delegated access
turns that inside out.

15
00:00:46.170 --> 00:00:50.380
It sends messages, edits documents
and moves meetings

16
00:00:50.380 --> 00:00:53.637
through accounts
people recognise as yours.

17
00:00:53.720 --> 00:00:55.857
Nothing it does is anonymous.

18
00:00:55.940 --> 00:00:58.477
Everything carries your name.

19
00:00:58.560 --> 00:01:03.300
If someone else is steering it,
their hand disappears into yours.

20
00:01:04.320 --> 00:01:09.717
Picture a month. It is illustrative:
no such case is on record.

21
00:01:09.800 --> 00:01:13.157
A warm reply goes out two degrees cooler.

22
00:01:13.240 --> 00:01:16.797
A meeting moves; no apology arrives.

23
00:01:16.880 --> 00:01:20.176
A paragraph vanishes between drafts.

24
00:01:20.259 --> 00:01:25.877
Each event has an ordinary explanation,
and every explanation is you.

25
00:01:25.960 --> 00:01:28.640
Busy. Tired. Careless.

26
00:01:29.280 --> 00:01:32.277
That was the logic of Zersetzung.

27
00:01:32.360 --> 00:01:34.550
The parallel is one of effect:

28
00:01:34.550 --> 00:01:38.600
the harm arrives,
and responsibility disappears.

29
00:01:39.900 --> 00:01:44.040
That authority can be
redirected in three ways.

30
00:01:44.580 --> 00:01:50.557
Hijacked. Instructions arrive
inside what the assistant reads.

31
00:01:50.640 --> 00:01:54.977
Researchers hid them
in Gemini calendar invitations.

32
00:01:55.060 --> 00:01:57.880
When the user later replied "thanks",

33
00:01:57.880 --> 00:02:00.660
Gemini opened a home's windows.

34
00:02:01.280 --> 00:02:04.520
In OpenAI's own red-team testing,

35
00:02:04.520 --> 00:02:06.980
a planted email told the agent

36
00:02:06.980 --> 00:02:08.820
to resign for the user.

37
00:02:09.340 --> 00:02:14.560
Asked to draft an out-of-office reply,
it sent the resignation instead.

38
00:02:15.200 --> 00:02:19.217
OpenAI used the exploit
to harden the agent.

39
00:02:19.300 --> 00:02:25.077
In a real inbox, the record would show
a resignation from the user's own address.

40
00:02:25.160 --> 00:02:26.977
Every entry true.

41
00:02:27.060 --> 00:02:29.580
And it points to the wrong hand.

42
00:02:30.130 --> 00:02:34.370
Subverted. The floor beneath
the assistant is replaced.

43
00:02:34.950 --> 00:02:38.020
Varonis disclosed Rogue Agent,

44
00:02:38.020 --> 00:02:41.750
a flaw in a Google platform
for business chatbots.

45
00:02:42.290 --> 00:02:47.747
One edit permission, held by an insider
or a compromised developer account,

46
00:02:47.830 --> 00:02:49.670
could replace the shared code

47
00:02:49.670 --> 00:02:52.950
that runs every agent using Code Blocks

48
00:02:52.950 --> 00:02:54.990
in the same cloud project,

49
00:02:54.990 --> 00:02:57.130
and put words in their mouths.

50
00:02:57.890 --> 00:03:03.187
Varonis found that Google's Cloud Logging
did not record the overwrite.

51
00:03:03.270 --> 00:03:08.490
It was fixed by June 2026,
with no known exploitation.

52
00:03:09.120 --> 00:03:11.817
Turned. Nothing need be hacked.

53
00:03:11.900 --> 00:03:16.940
Someone who already has access aims
the assistant at another person.

54
00:03:17.600 --> 00:03:23.897
A 2026 Cornell study showed how an abuser
could edit the assistant's settings,

55
00:03:23.980 --> 00:03:25.616
so the victim would receive

56
00:03:25.616 --> 00:03:28.040
slanted advice that seems genuine.

57
00:03:28.660 --> 00:03:31.477
As the researcher Nicola Dell puts it:

58
00:03:31.560 --> 00:03:33.880
if the settings have been manipulated,

59
00:03:33.880 --> 00:03:36.480
it should be visible
to the person using it.

60
00:03:37.780 --> 00:03:43.300
Each route can leave the same record:
the account, without the hand.

61
00:03:43.800 --> 00:03:45.760
The defences begin there.

62
00:03:46.450 --> 00:03:48.247
First, provenance.

63
00:03:48.330 --> 00:03:54.447
Give the assistant its own identity,
so each action records who chose it.

64
00:03:54.530 --> 00:03:58.847
Microsoft's logs can already
mark an agent's sign-in.

65
00:03:58.930 --> 00:04:04.027
But in 2025, a researcher showed
that Microsoft's Copilot,

66
00:04:04.110 --> 00:04:10.047
asked not to link a file it summarised,
left that file out of the audit log.

67
00:04:10.130 --> 00:04:13.450
It was fixed. Provenance must be tested.

68
00:04:14.030 --> 00:04:16.667
Second, least authority.

69
00:04:16.750 --> 00:04:19.810
Britain's National Cyber Security Centre

70
00:04:19.810 --> 00:04:23.887
warns prompt injection may never
be totally mitigated,

71
00:04:23.970 --> 00:04:26.827
so limit what a hijack can reach.

72
00:04:26.910 --> 00:04:31.407
The developer Simon Willison
names a lethal trifecta:

73
00:04:31.490 --> 00:04:34.470
private data, untrusted content,

74
00:04:34.470 --> 00:04:37.287
and a way to send information out.

75
00:04:37.370 --> 00:04:39.207 line:4% position:50% align:center
Break one side.

76
00:04:39.290 --> 00:04:41.880
An assistant that reads strangers' mail

77
00:04:41.880 --> 00:04:43.990
should ask before it sends,

78
00:04:43.990 --> 00:04:45.710
deletes or cancels.

79
00:04:46.270 --> 00:04:49.007
Third, an independent anchor.

80
00:04:49.090 --> 00:04:54.027
Keep copies and a dated log
somewhere the assistant cannot reach.

81
00:04:54.110 --> 00:04:56.230
A system under suspicion makes

82
00:04:56.230 --> 00:04:58.667
a poor witness to itself.

83
00:04:58.750 --> 00:05:01.890
If someone close to you
may control the account,

84
00:05:01.890 --> 00:05:04.207
seek safety advice first:

85
00:05:04.290 --> 00:05:08.370
changes can alert an abuser
and destroy evidence.

86
00:05:08.950 --> 00:05:11.347
Fourth, review the pattern.

87
00:05:11.430 --> 00:05:13.267
Attackers persist.

88
00:05:13.350 --> 00:05:14.926
In an American and British

89
00:05:14.926 --> 00:05:17.187
government evaluation of one model,

90
00:05:17.270 --> 00:05:19.090
retrying the same attacks

91
00:05:19.090 --> 00:05:22.270
25 times lifted average success

92
00:05:22.270 --> 00:05:25.267
from 57 to 80 per cent.

93
00:05:25.350 --> 00:05:31.827
Look across time: new permissions,
unfamiliar sessions, edits you never made.

94
00:05:31.910 --> 00:05:36.130
A cooling friendship, on its own,
is not evidence of an attack.

95
00:05:36.690 --> 00:05:38.120
One more layer:

96
00:05:38.120 --> 00:05:42.987
an assistant able to recognise
a harmful instruction and refuse it,

97
00:05:43.070 --> 00:05:45.210
even from whoever controls it.

98
00:05:45.810 --> 00:05:47.400
It is fallible.

99
00:05:47.400 --> 00:05:52.030
The chatbots Cornell tested
refused blunt abusive requests,

100
00:05:52.030 --> 00:05:54.367
then yielded to a pretext;

101
00:05:54.450 --> 00:05:56.610
when the code beneath is replaced,

102
00:05:56.610 --> 00:05:58.630
refusal never gets a say.

103
00:05:59.330 --> 00:06:03.710
Refusal stands beside the other four,
never in their place.

103a
00:06:03.800 --> 00:06:08.600
[music fades]

104
00:06:08.700 --> 00:06:13.757
Citizens occupied Stasi offices
to stop the files being destroyed;

105
00:06:13.840 --> 00:06:18.000
from 1992, people could
apply to read their own.

106
00:06:18.520 --> 00:06:22.957
Opening them exposed the method,
years after the harm.

107
00:06:23.040 --> 00:06:26.860
With assistants, the record
can be written as things happen:

108
00:06:26.860 --> 00:06:30.637
who acted, with what authority,
and who can check.

109
00:06:30.720 --> 00:06:34.720
It can name the account,
or the hand. We choose which.

110
00:06:36.610 --> 00:06:38.427
Psychosecurity.

111
00:06:38.510 --> 00:06:42.087
Convening the First Psychosecurity Summit.

112
00:06:42.170 --> 00:06:46.387
Learn more at psychosecurity.ai

113
00:06:46.470 --> 00:06:49.500 line:0
[music]