CERT-In Advisory
CIAD-2025-0013
Best Practices against vulnerabilities while using Generative AI solutions
Original Issue Date: March 26, 2025
Description
Artificial Intelligence (AI) has become a hallmark of innovation, revolutionizing industries ranging from healthcare to communications. AI is increasingly used to handle activities traditionally undertaken by humans such as automating routine tasks, fostering creativity, and supporting business functions such as customer services, logistics, medical diagnosis, and cybersecurity.
As AI becomes increasingly advanced and become more ubiquitous, the associated risks also increase. Numerous attacks target AI applications, by taking advantage of flaws in data processing and machine learning models. These attacks pose significant threats to AI applications' security, reliability, and trustworthiness across a variety of fields.
Below are examples of how vulnerabilities in AI design, training, and interaction mechanisms could be exploited by threat actors:
Data poisoning:
Data poisoning involves manipulating an AI model's training data so that the model learns incorrect patterns and potentially misclassify data or generate inaccurate, biased or malicious outputs. An AI model's training data could be manipulated by inserting new, deceptive data or modifying existing data; or the training data could be taken from a compromised or maliciously tainted origin.
Adversarial attacks: Adversarial attacks change inputs to AI models to make them give wrong predictions. These changes appears unchanged to human observers but are very effective at tricking the AI.
Model inversion: Model inversion attacks are designed to extract sensitive information about a machine learning model's training data. By analyzing the model's outputs, attackers can deduce specifics about individual training samples or identify patterns within the data.
Model stealing: Model stealing attacks try to copy a machine learning model by repeatedly querying it and leveraging its outputs to construct a similar model. This strategy allows attackers to avoid the time and resources needed to train their own models.
For example, imagine an insurance company that has developed an AI model to generate insurance quotes for its clients. If a competitor were to extensively query this model and use the responses to create a replica of it, it could benefit from the investment that went into creating the model, without sharing in its development costs. Furthermore, certain inputs can lead generative AI models to inadvertently expose their training data. The exfiltration of training data can be a serious privacy concern, especially for models trained on sensitive or personally identifiable information.
Prompt injection is an input manipulation attack in which malicious instructions or hidden commands are introduced into an AI system. Such attacks enable malicious actors to hijack the AI model's output and jailbreak the AI system, effectively bypass its safeguards, such as content filters that restrict its functionality.
For instance, consider a music sharing platform that requires user submitted music to pass an AI powered copyright check before it is published. Through an adversarial example attack, a user could slightly alter the playback speed of a copyrighted song to bypass the AI's copyright check, while remaining recognisable to listeners.
Hallucination Exploitation: This refers to instances where malicious actors exploit the tendency of certain AI models to generate inaccurate, misleading, or entirely fabricated outputs (hallucinations). These hallucinated outputs can be leveraged to propagate false information, leading to confusion and diminishing trust in AI systems. Also, by taking advantage of such outputs, attackers may craft deceptive content to manipulate individuals or systems, potentially resulting in phishing attacks, fraudulent transactions, or security compromises.
Backdoor attacks: Backdoor attacks implant hidden triggers within an AI model during its training process. These triggers can be exploited later to make the model behave maliciously or incorrectly when specific inputs or conditions are met.
Best Practices
Here are some best practices for using generative AI solutions effectively and responsibly.
Choose AI Apps Carefully
Not all AI applications out there are safe. Threat actors can take advantage of the rising demand for AI apps to create fake apps designed to trick users into downloading them. If you download these fake AI apps on your device, it will maximize the opportunity to install malware designed to steal all of your data. To minimize AI cybersecurity risks, it is crucial to practice due diligence before downloading any AI app. The rule of thumb is only to use AI tools approved and verified by your organization.
Avoid Sharing Personal & Sensitive Information
It is important to recognize that with the majority of services, the data sent to the service is collected and used by the service provider to improve their models. It is advised to avoid utilizing generative AI tools available online for professional work involving sensitive information. As the client entity does not have control over these generative AI services, it is impossible to ensure that the confidentiality of data submitted for input meets the entity's security requirements.
If you wouldn't be comfortable with the information being made public, do not share it within the platform. This includes personal details, confidential client information, intellectual property, or any other sensitive data that could be compromised or misused, potentially leading to legal and ethical issues.
Carefully configure AI tools' access rights
Some third-party generative AI tools offer connections with office automation tools or common online business applications. It is essential to carefully configure these AI tools' access rights to the organization's business data, such as emails, documents storage, source code repositories, audio and video conferencing platforms etc.
Access rights configurations should be promptly reviewed when the tool is first activated to ensure default settings are neither too permissive nor insecure. Additionally, periodic reviews of these access rights are necessary to ensure that updates to the tool's functionality or security do not unintentionally affect user data access permissions.
Don't Rely on AI for Accuracy
Bad data or malicious hackers could fool AI tools to churn out inaccurate content (hallucinations). For AI to produce accurate outputs, algorithms must contain large, representative data sets. If certain groups of data sets are underrepresented, it will ultimately result in inaccurate outcomes and harmful decisions. The AI tool you are using is only as accurate as the data it uses. If the data it uses is old or incomplete, the content it churns out will be biased, inaccurate, or outright wrong.
Hence, you should not rely on AI alone to make crucial business decisions. It's essential to fact-check the responses provided and verify it against reliable sources.
Use the System for its Intended Purpose
AI tools are designed to assist with various tasks, such as answering questions, providing information, and generating content. It's essential to use these tools for their intended purposes and not rely on them to make critical decisions, especially in legal or medical contexts.
Use a Strong Password/Secure Access
As with any online service, it's imperative that you use a strong, unique password to secure your account. Enable two-factor authentication if available, and avoid sharing your login credentials with others. Be cautious when accessing the service from public or shared devices, and always log out when you're done. Regularly review and update access permissions, especially in professional settings, to ensure that only authorized individuals can use the tools on behalf of your organization.
Use an Anonymous Account
When signing up for AI services, consider using an anonymous account that is not linked to your personal or professional identity. This helps protect your privacy and prevents potential data breaches from being traced back to you. If you need to input sensitive information, anonymize it by removing or replacing identifying details.
Don't Use Plagiarized Content
It is entirely possible for an AI tool to match words or copy-paste words from other sources to create pieces of new content quickly. With many bloggers and businesses relying on AI writing programs for their websites, there is rising concern the work these tools produce could be plagiarized. When using AI tools to generate content for your website, you should be careful not to plagiarize content.
Stay Alert for Any Suspicious Activity
Threat actors can use these AI tools to generate fake content or deep fake videos to deceive or manipulate people. Generative AI can be used for malicious purposes, including generating convincing phishing emails and codes. Although some apps have some protections to prevent users from creating malicious codes, threat actors can leverage clever techniques to bypass them and create malware. For this reason, you should always stay alert for any suspicious activity when using AI tools and content.
References
CERT-In
Security implications of AI language based applications (CIAD-2023-0015)
https://www.cert-in.org.in/s2cMainServlet?pageid=PUBVLNOTES02&VLCODE=CIAD-2023-0015
Disclaimer
The information provided herein is on "as is" basis, without warranty of any kind.
Contact Information
Email: info@cert-in.org.in Phone: +91-11-22902657
Postal address
Indian Computer Emergency Response Team (CERT-In)
Ministry of Electronics and Information Technology
Government of India
Electronics Niketan
6, CGO Complex, Lodhi Road,
New Delhi - 110 003
India
|