Tuesday, 7:15 PM. A customer calls your bike shop. A friendly voice picks up, asks about the gear trouble and offers Thursday at 9.
The voice sounds human. It's an AI. That's exactly where the EU AI Act comes in: people who talk to an AI should know it.
This article walks through the obligations that apply to phone AI, with quotes from Regulation (EU) 2024/1689, so you can judge what they mean for your business and what to do about them.
Updated September 2026: the transparency obligations now apply, the European Commission has published its guidelines on them, and the so-called Digital Omnibus has changed some rules and deadlines.
What Is the EU AI Act?
Regulation (EU) 2024/1689, better known as the EU AI Act, is the EU's first comprehensive AI law. It entered into force on 1 August 2024 and applies in stages:
- 2 February 2025: Prohibited AI practices and AI literacy
- 2 August 2025: Rules for general-purpose AI models
- 2 August 2026: General date of application, including the transparency obligations in Article 50
- 2 December 2026: Deadline for machine-readable marking for systems already on the market before 2 August 2026
- 2 December 2027: High-risk rules for the areas listed in Annex III
- 2 August 2028: High-risk rules for AI in regulated products (Annex I)
The last three dates come from the Digital Omnibus, Regulation (EU) 2026/1744. It has applied since 27 July 2026 and pushed back the high-risk deadlines. The transparency obligations still apply from 2 August 2026 as planned. Only the marking duty gets a grace period for systems that were already on the market.
Put simply, the law has four tiers: prohibited, high-risk, transparency obligations and minimal risk. The tier decides which obligations apply.
Where Does Phone AI Fit?
An AI phone assistant that answers calls and books appointments is:
- Not prohibited. It doesn't do social scoring, manipulation or biometric identification.
- Not high-risk, unless it helps decide on something Annex III lists as high-risk, such as creditworthiness, job applications or access to education.
- Subject to transparency obligations, because it talks directly to people.
That sounds harmless. The details still matter.
The regulation has one exception: when the AI nature is "obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect, taking into account the circumstances and the context of use." The European Commission's guidelines say this exception "should be interpreted restrictively". A human-sounding voice counts against obviousness. If the general public can reach the system, including older people or people with disabilities, the exception "cannot be relied upon" (Guidelines, point 45). For a phone AI that takes customer calls, the disclosure is part of the job.
Recital (132) also has people in mind who need extra protection:
"In particular, natural persons should be notified that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect taking into account the circumstances and the context of use. When implementing that obligation, the characteristics of natural persons belonging to vulnerable groups due to their age or disability should be taken into account to the extent the AI system is intended to interact with those groups as well."
— Recital (132), Regulation (EU) 2024/1689
On a call, that means a clearly spoken disclosure, not one tucked into a side remark.
Obligation 1: The AI Disclosure Under Article 50(1)
The core of the transparency obligation is Article 50(1):
"Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect, taking into account the circumstances and the context of use."
— Art. 50(1), Regulation (EU) 2024/1689
The key words are "designed and developed." The obligation sits with the provider, the company that builds the phone AI. The provider has to build the disclosure into the system.
According to the guidelines, providers are responsible for ensuring that people are "effectively and consistently informed throughout the lifecycle of the AI system" (point 33). As the deployer, you should still listen to what your callers hear, especially if you customise the greeting.
When does the disclosure have to happen? Article 50(5) is clear:
"The information referred to in paragraphs 1 to 4 shall be provided to the natural persons concerned in a clear and distinguishable manner at the latest at the time of the first interaction or exposure. The information shall conform to the applicable accessibility requirements."
— Art. 50(5), Regulation (EU) 2024/1689
On a call, that's the greeting. Not the small print on your website. Not your terms and conditions.
What the Commission's Guidelines Add
On 20 July 2026, the European Commission approved and published the content of its guidelines on the transparency obligations. They are to be formally adopted once all language versions are available. The guidelines are non-binding (point 5). They are meant as practical guidance for authorities, providers and deployers (point 3).
What matters for phone AI:
- A spoken notice at the start. For voice-based or telephony contexts, the guidelines give the example of "explicit spoken statements at the beginning of the interaction (e.g. “This is an AI-powered assistant”)" (point 37). Audio cues such as tones are "not considered sufficient by themselves".
- One clear notice is usually enough. "A single, prominent notification before the first interaction of the AI system with a particular natural person is likely to suffice in most instances." In riskier contexts, periodic reminders are "likely to be necessary", for example when callers include older people or the call is about a complaint (point 40). The guidelines also mention reminders in longer interactions or when the AI's role changes, as appropriate (point 37).
- Vague wording doesn't work. On their own, "generic references to “assistant”" are insufficient, and so are disclosures that appear only in terms and conditions, URLs or documentation (point 38). People must be "explicitly informed about the artificial, non-human nature" of who they're talking to (point 35).
- Answer honestly when asked. Providers "must" ensure disclosure whenever the system is asked about its nature, or when the person is likely to be misled or confused about it (point 40).
- Say on whose behalf. AI agents that act for someone, "making bookings" among the examples, "must be designed and developed in such a way that they disclose both their artificial nature and the person on whose behalf they are acting" (point 31).
A greeting that covers this can be short: "Hi, this is the AI assistant for [your business name]. How can I help?"
Obligation 2: Marking the Synthetic Voice Under Article 50(2)
The second obligation covers what the AI produces:
"Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated."
— Art. 50(2), Regulation (EU) 2024/1689
A phone AI's voice is generated. Some systems turn text into speech, others produce the answer directly as speech. The guidelines explicitly count speech as audio (point 60), and Article 50(2) "may also apply to AI agents" whose outputs people can perceive (point 63). The obligation has two parts, machine-readable marking and a means of detection. One without the other isn't enough (points 69–70).
The obligation has applied since 2 August 2026. For systems that were already on the market before that date, the Digital Omnibus grants a grace period until 2 December 2026. The grace period covers marking only. The disclosure during the conversation has been required since 2 August 2026 for these systems too (point 153).
The technical solutions must be effective, interoperable, robust and reliable "as far as this is technically feasible", taking into account the "generally acknowledged state of the art". Providers aren't obliged to use a solution that isn't yet developed or available on the market (point 81). Recital (133) gives examples:
"Such techniques and methods should be sufficiently reliable, interoperable, effective and robust as far as this is technically feasible, taking into account available techniques or a combination of such techniques, such as watermarks, metadata identifications, cryptographic methods for proving provenance and authenticity of content, logging methods, fingerprints or other techniques, as may be appropriate."
— Recital (133), Regulation (EU) 2024/1689
In practice:
- Audio watermarks: inaudible signals in the generated voice that software can detect later.
- Provenance data in metadata: for example, following C2PA, the open standard from the Coalition for Content Provenance and Authenticity.
For real-time content, the guidelines describe a possible exemption. It covers content that is "ephemeral and consumed immediately, without being recorded, stored or disseminated further", where marking isn't technically feasible and people are made aware that the content is AI-generated (point 88). The examples given are video games and virtual reality. Phone calls aren't mentioned there.
There is also the Code of Practice on Transparency of AI-Generated Content. The guidelines call it "the only Union-wide recognised practical framework" for showing compliance (point 146). Signing and following a code assessed as adequate is described as a "straightforward, predictable, and legally certain way of demonstrating compliance" (point 147).
Obligation 3: AI Literacy Under Article 4
This obligation has applied since 2 February 2025. The Digital Omnibus rewrote it. Since 27 July 2026 it reads:
"Providers and deployers of AI systems shall take measures to support the development of AI literacy of their staff and other persons dealing with the operation and use of AI systems on their behalf, [...]. This obligation does not require providers or deployers to guarantee any specific level of AI literacy of any individual."
— Art. 4(1), Regulation (EU) 2024/1689 as amended by Regulation (EU) 2026/1744
The old wording asked providers and deployers to "ensure, to their best extent, a sufficient level of AI literacy". Now the point is to support it. Nobody has to guarantee a particular level.
Both providers and deployers are still covered. If you use a phone AI, your team should at least know:
- that an AI answers the calls,
- what it can and can't do,
- what to do when a caller complains or something goes wrong.
How Does This Relate to the GDPR?
The AI Act doesn't replace the GDPR. It adds to it. For phone AI, both apply side by side.
GDPR Art. 6: Lawful basis
Every processing of personal data needs a lawful basis. For booking appointments, that's usually:
- Art. 6(1)(b): steps before entering into a contract, or performing one. The caller wants an appointment, and that needs their data.
- Art. 6(1)(f): legitimate interest, for example in being reachable for bookings.
Only collect what the purpose actually requires. For a workshop appointment, name, phone number and the reason for the visit are usually proportionate.
GDPR Art. 13: Information obligation
Whoever collects data has to inform people "at the time when personal data are obtained": who the controller is, what the data is for, on what legal basis and how long it's kept. The controller is usually your business. The phone AI provider typically processes the data on your behalf (Art. 28 GDPR).
Reading a full privacy notice on the phone isn't practical. A pragmatic approach: the assistant can say on request who processes the data, why, and where the full information is. That requires this information to be stored in the system.
GDPR Art. 13(2)(f): Automated decision-making
The GDPR requires information about "the existence of automated decision-making, including profiling" under Art. 22. That means decisions with legal effects or a similarly significant impact. Whether a booked workshop appointment counts is doubtful. Being open about how the assistant works doesn't hurt either way.
How the rules fit together:
| Obligation | GDPR | AI Act |
|---|---|---|
| AI disclosure during the call | Not explicitly required | Art. 50(1): required |
| Timing of the information | "At the time when personal data are obtained" (Art. 13) | "At the latest at the time of the first interaction" (Art. 50(5)) |
| Mark the synthetic voice | Not required | Art. 50(2): required (grace period until 2 December 2026 for existing systems) |
| Privacy notice | Art. 13: required | Leaves the GDPR untouched |
| AI literacy of staff | Not required | Art. 4: since February 2025, reworded July 2026 |
| Recording the call | Needs a lawful basis | Not covered. Mind criminal law, see the excursus |
AI literacy under Art. 4 has applied since February 2025, and the transparency obligations under Art. 50 since 2 August 2026.
Excursus: Call Recording and Criminal Law
On the phone, a third law comes in that has nothing to do with AI at first. In Germany, Section 201 of the Criminal Code (StGB) protects the "privacy of spoken word". Among other things, it penalises anyone who,
"without being authorised to do so, makes an audio recording of the privately spoken words of another"
— Section 201(1) no. 1 StGB, official English translation
What's said on a phone call is usually private. If you want to record calls, you would normally ask for the caller's consent first. Austria (Section 120 of its Criminal Code) and Switzerland (Art. 179ter of its Criminal Code) have their own rules.
With phone AI, things are less clear-cut. To understand what's said, the AI has to process the voice during the call and turn it into text. How that technical processing is to be assessed under criminal law hasn't been conclusively settled.
What you can do:
- Ask what happens to the audio. Is it stored? For how long? Who can listen to it?
- If you record calls, get consent first.
- Say right at the start that an AI is speaking, and point to a page that explains what happens to the data.
The Fines
The AI Act has three tiers:
| Violation | Fine of up to |
|---|---|
| Prohibited AI practices (Art. 5) | EUR 35M or 7% of worldwide annual turnover |
| Transparency obligations (Art. 50) | EUR 15M or 3% of worldwide annual turnover |
| Incorrect, incomplete or misleading information to authorities or notified bodies | EUR 7.5M or 1% of worldwide annual turnover |
For small and medium-sized enterprises, Article 99(6) applies:
"In the case of SMEs, including start-ups, each fine referred to in this Article shall be up to the percentages or amount referred to in paragraphs 3, 4 and 5, whichever thereof is lower."
— Art. 99(6), Regulation (EU) 2024/1689
An example: with EUR 500,000 in annual turnover, the cap for a transparency violation is EUR 15,000 (3%), not EUR 15M. For a small business, that's still a lot of money.
The detailed penalty rules are set by the member states. When imposing penalties, they have to take into account the interests of SMEs and their economic viability (Art. 99(1)).
What to Check
The legal texts translate into concrete checkpoints, for your own business and when choosing a provider:
1. Does the greeting say clearly that an AI is speaking? Art. 50(1) requires the provider to build the disclosure into the system. Art. 50(5) requires it at the latest at the first interaction. Since August 2026, a greeting without an AI disclosure is a compliance risk. Listen to your greeting, especially if you wrote it yourself.
2. How is the synthetic voice marked? Since August 2026, AI-generated audio must be marked in a machine-readable way (Art. 50(2)), with existing systems given until 2 December 2026. Ask your provider how they handle it.
3. Where is call data processed? If caller data passes through servers outside the EU, a basis under Chapter V GDPR is needed, for example an adequacy decision (for the US: the Data Privacy Framework) or Standard Contractual Clauses. Reputable providers disclose this in their sub-processor list.
4. Does your team know what's going on? Art. 4 has applied since February 2025. Ask your provider what explanations and material they supply, and talk to your team about what the AI does on the phone.
5. Are retention periods and recordings clearly defined? The GDPR requires storage limitation (Art. 5(1)(e)). Check whether audio is stored, how long transcripts and appointment data are kept, and what you keep yourself. Some data has to be kept for tax or contractual reasons. The periods should be documented and justified.
What We Do at FlowCaptain
- AI disclosure in every greeting. Every call opens with the assistant saying that it is an AI. If you write your own greeting and it does not say so, that sentence is placed in front of your words — your greeting is not replaced. When your business name is on file and your own greeting does not already say it, the sentence names the business too.
- A page for callers. Since August 2026, our caller information page explains that an AI is on the line, which data is processed and how long it's kept.
- No audio stored. The voice is turned into text during the call. We don't store audio. There's no recording that anyone could listen to later.
- Clear retention periods. In our systems, the text of the conversation is anonymised after 90 days, and the appointment is deleted 90 days after it ends. Other data has its own periods, listed in our privacy policy. What you keep yourself, such as the email you get after every call or the entries in your calendar, is up to you.
- Core services in EU regions. Database, API and dashboard run in EU regions (Frankfurt/Amsterdam). Where individual providers process data in the US, this is listed in our sub-processor list and based on the Data Privacy Framework or standard contractual clauses.
We took part in the European Commission's consultation on the Article 50 guidelines.
Conclusion
The EU AI Act is no reason to panic. Its requirements make sense: if you use AI, say so. If you generate AI content, mark it. If you run AI systems, understand what they do.
For small businesses that let a phone AI take their calls, the obligations are manageable, but real. And they already apply: since 2 August 2026.
If you haven't done it yet: call your own number today and listen carefully. Does your greeting say clearly that an AI is speaking?
Further Resources
- Full text of Regulation (EU) 2024/1689 (AI Act)
- Digital Omnibus, Regulation (EU) 2026/1744: amendments to the AI Act, in force since 27 July 2026
- European Commission guidelines on the Article 50 transparency obligations
- Full text of Regulation (EU) 2016/679 (GDPR)
- German Criminal Code, official English translation (Section 201)
- AI Act Service Desk: the European Commission's official help desk with FAQ and guidelines
- EU AI Act Compliance Checker: find out which requirements apply
- German Federal Government on the AI Act (in German)
- Digital Austria: AI Act: information for Austrian businesses (in German)
Sources: Regulation (EU) 2024/1689 (AI Act) as amended by Regulation (EU) 2026/1744, Regulation (EU) 2016/679 (GDPR), Section 201 German Criminal Code. Quoted passages are from the official English versions in the Official Journal of the European Union and the official English translation of the German Criminal Code; omissions are marked [...]. As of September 2026. This article does not constitute legal advice.