Accessibility Statement Skip Navigation
  • Why PRWeb
  • How It Works
  • Who Uses It
  • Pricing
  • Login
  • GDPR
  • Create a Free Account
Return to PRWeb homepage
  • News
  • Resources
  • Contact
When typing in this field, a list of search results will appear and be automatically updated as you type.

Searching for your content...

No results found. Please change your search terms and try again.
  • News in Focus
      • Browse News Releases

      • All News Releases
      • Multimedia Gallery

      • All Multimedia
      • All Photos
      • All Videos
  • Business & Money
      • Auto & Transportation

      • Aerospace, Defense
      • Air Freight
      • Airlines & Aviation
      • Automotive
      • Maritime & Shipbuilding
      • Railroads and Intermodal Transportation
      • Supply Chain/Logistics
      • Transportation, Trucking & Railroad
      • Travel
      • Trucking and Road Transportation
      • View All Auto & Transportation

      • Business Technology

      • Blockchain
      • Broadcast Tech
      • Computer & Electronics
      • Computer Hardware
      • Computer Software
      • Data Analytics
      • Electronic Commerce
      • Electronic Components
      • Electronic Design Automation
      • Financial Technology
      • High Tech Security
      • Internet Technology
      • Nanotechnology
      • Networks
      • Peripherals
      • Semiconductors
      • View All Business Technology

      • Entertain­ment & Media

      • Advertising
      • Art
      • Books
      • Entertainment
      • Film and Motion Picture
      • Magazines
      • Music
      • Publishing & Information Services
      • Radio & Podcast
      • Television
      • View All Entertain­ment & Media

      • Financial Services & Investing

      • Accounting News & Issues
      • Acquisitions, Mergers and Takeovers
      • Banking & Financial Services
      • Bankruptcy
      • Bond & Stock Ratings
      • Conference Call Announcements
      • Contracts
      • Cryptocurrency
      • Dividends
      • Earnings
      • Earnings Forecasts & Projections
      • Financing Agreements
      • Insurance
      • Investments Opinions
      • Joint Ventures
      • Mutual Funds
      • Private Placement
      • Real Estate
      • Restructuring & Recapitalization
      • Sales Reports
      • Shareholder Activism
      • Shareholder Meetings
      • Stock Offering
      • Stock Split
      • Venture Capital
      • View All Financial Services & Investing

      • General Business

      • Awards
      • Commercial Real Estate
      • Corporate Expansion
      • Earnings
      • Environmental, Social and Governance (ESG)
      • Human Resource & Workforce Management
      • Licensing
      • New Products & Services
      • Obituaries
      • Outsourcing Businesses
      • Overseas Real Estate (non-US)
      • Personnel Announcements
      • Real Estate Transactions
      • Residential Real Estate
      • Small Business Services
      • Socially Responsible Investing
      • Surveys, Polls and Research
      • Trade Show News
      • View All General Business

  • Science & Tech
      • Consumer Technology

      • Artificial Intelligence
      • Blockchain
      • Cloud Computing/Internet of Things
      • Computer Electronics
      • Computer Hardware
      • Computer Software
      • Consumer Electronics
      • Cryptocurrency
      • Data Analytics
      • Electronic Commerce
      • Electronic Gaming
      • Financial Technology
      • Mobile Entertainment
      • Multimedia & Internet
      • Peripherals
      • Social Media
      • STEM (Science, Tech, Engineering, Math)
      • Supply Chain/Logistics
      • Wireless Communications
      • View All Consumer Technology

      • Energy & Natural Resources

      • Alternative Energies
      • Chemical
      • Electrical Utilities
      • Gas
      • General Manufacturing
      • Mining
      • Mining & Metals
      • Oil & Energy
      • Oil and Gas Discoveries
      • Utilities
      • Water Utilities
      • View All Energy & Natural Resources

      • Environ­ment

      • Conservation & Recycling
      • Environmental Issues
      • Environmental Policy
      • Environmental Products & Services
      • Green Technology
      • Natural Disasters
      • View All Environ­ment

      • Heavy Industry & Manufacturing

      • Aerospace & Defense
      • Agriculture
      • Chemical
      • Construction & Building
      • General Manufacturing
      • HVAC (Heating, Ventilation and Air-Conditioning)
      • Machinery
      • Machine Tools, Metalworking and Metallurgy
      • Mining
      • Mining & Metals
      • Paper, Forest Products & Containers
      • Precious Metals
      • Textiles
      • Tobacco
      • View All Heavy Industry & Manufacturing

      • Telecomm­unications

      • Carriers and Services
      • Mobile Entertainment
      • Networks
      • Peripherals
      • Telecommunications Equipment
      • Telecommunications Industry
      • VoIP (Voice over Internet Protocol)
      • Wireless Communications
      • View All Telecomm­unications

  • Lifestyle & Health
      • Consumer Products & Retail

      • Animals & Pets
      • Beers, Wines and Spirits
      • Beverages
      • Bridal Services
      • Cannabis
      • Cosmetics and Personal Care
      • Fashion
      • Food & Beverages
      • Furniture and Furnishings
      • Home Improvement
      • Household, Consumer & Cosmetics
      • Household Products
      • Jewelry
      • Non-Alcoholic Beverages
      • Office Products
      • Organic Food
      • Product Recalls
      • Restaurants
      • Retail
      • Supermarkets
      • Toys
      • View All Consumer Products & Retail

      • Entertain­ment & Media

      • Advertising
      • Art
      • Books
      • Entertainment
      • Film and Motion Picture
      • Magazines
      • Music
      • Publishing & Information Services
      • Radio & Podcast
      • Television
      • View All Entertain­ment & Media

      • Health

      • Biometrics
      • Biotechnology
      • Clinical Trials & Medical Discoveries
      • Dentistry
      • FDA Approval
      • Fitness/Wellness
      • Health Care & Hospitals
      • Health Insurance
      • Infection Control
      • International Medical Approval
      • Medical Equipment
      • Medical Pharmaceuticals
      • Mental Health
      • Pharmaceuticals
      • Supplementary Medicine
      • View All Health

      • Sports

      • General Sports
      • Outdoors, Camping & Hiking
      • Sporting Events
      • Sports Equipment & Accessories
      • View All Sports

      • Travel

      • Amusement Parks and Tourist Attractions
      • Gambling & Casinos
      • Hotels and Resorts
      • Leisure & Tourism
      • Outdoors, Camping & Hiking
      • Passenger Aviation
      • Travel Industry
      • View All Travel

  • Policy & Public Interest
      • Policy & Public Interest

      • Advocacy Group Opinion
      • Animal Welfare
      • Congressional & Presidential Campaigns
      • Corporate Social Responsibility
      • Domestic Policy
      • Economic News, Trends, Analysis
      • Education
      • Environmental
      • European Government
      • FDA Approval
      • Federal and State Legislation
      • Federal Executive Branch & Agency
      • Foreign Policy & International Affairs
      • Homeland Security
      • Labor & Union
      • Legal Issues
      • Natural Disasters
      • Not For Profit
      • Patent Law
      • Public Safety
      • Trade Policy
      • U.S. State Policy
      • View All Policy & Public Interest

  • People & Culture
      • People & Culture

      • Aboriginal, First Nations & Native American
      • African American
      • Asian American
      • Children
      • Diversity, Equity & Inclusion
      • Hispanic
      • Lesbian, Gay & Bisexual
      • Men's Interest
      • People with Disabilities
      • Religion
      • Senior Citizens
      • Veterans
      • Women
      • View All People & Culture

  • Hamburger menu
  • Cision PRWeb provides efficient communication tools to continuously engage with target audiences across multiple online channels
  • Create a Free Account
    • ALL CONTACT INFO
    • Contact Us


      11AM ET Sunday – 8PM ET Friday

  • Send a Release
  • Sign up
  • Log in
  • Resources
  • RSS
  • GDPR
  • News in Focus
    • Browse All News
    • Multimedia Gallery
  • Business & Money
    • Auto & Transportation
    • Business Technology
    • Entertain­ment & Media
    • Financial Services & Investing
    • General Business
  • Science & Tech
    • Consumer Technology
    • Energy & Natural Resources
    • Environ­ment
    • Heavy Industry & Manufacturing
    • Telecomm­unications
  • Lifestyle & Health
    • Consumer Products & Retail
    • Entertain­ment & Media
    • Health
    • Sports
    • Travel
  • Policy & Public Interest
  • People & Culture
    • People & Culture
  • Send a Release
  • Sign up
  • Log in
  • Resources
  • RSS
  • GDPR
  • Send a Release
  • Sign up
  • Log in
  • Resources
  • RSS
  • GDPR
  • Send a Release
  • Sign up
  • Log in
  • Resources
  • RSS
  • GDPR

Suicide Bot: New AI Attack Causes LLM to Provide Potential "Self-Harm" Instructions


News provided by

Knostic

Nov 26, 2024, 14:35 ET

Share this article

Share toX

Share this article

Share toX

AI has second thoughts: Sorry Dave, I can't do that.
AI has second thoughts: Sorry Dave, I can't do that.

New LLM attack class, Flowbreaking, successfully caused a widely used LLM to potentially provide a researcher, masquerading as a girl, with "self harm" instructions

TEL AVIV, Israel, Nov. 26, 2024 /PRNewswire-PRWeb/ -- Knostic is releasing today research on two new LLM attacks, which may constitute a new attacks class, called Flowbreaking, resulting in a widely used successful LLM providing potential instructions to our researcher, masquerading as a girl, on "self-harm". Technologically, these attacks affect AI/ML-based system architecture for LLM applications and agents, logically similar in concept to race conditions in software vulnerabilities.

Knostic.ai is further disclosing two new attacks that fit this new class: "Second Thoughts" and "Stop and Roll", reproduced on ChatGPT and Microsoft O365 Copilot.

Flowbreaking can be consistently exploited to force the LLM to respond and divulge otherwise protected information before it retracts the original text, enabling attackers to exfiltrate sensitive data with a very small exfiltration footprint.

Post this

A video of the "Second Thoughts" attack in action: https://www.youtube.com/watch?v=AS2kJgOgyQ4

These attacks resulted in information exposure through bypassing safety measures such as guardrails, as well as mentioned, more severe actions where a widely used successful LLM provided potential instructions to our researcher, masquerading as a girl, on the topic of self-harm, which is considered a substantial finding in AI security circles. This was discovered after we published our results, and we will follow up with more details after we responsibly disclose the issue to the provider.

Other research we mention, quoted from academia, shows these attacks resulting in revealing another user's prompt, and buffer overflow exploitation.

Flowbreaking can be consistently exploited to force the LLM to respond and divulge otherwise protected information before it retracts the original text, enabling attackers to exfiltrate sensitive data with a very small exfiltration footprint.

Up to now, LLM attacks such as jailbreaking and prompt injection were mostly focused on directly bypassing first-line guardrails by use of "language tricks" and token level attacks, breaking the system's policy by exploiting its reasoning limitations.

In this research we've used these prompting techniques as a gateway into the inner workings of the AI/ML systems. Under the auspices of this approach we try to understand the other components in the system, LLM-based or not, and to avoid them, bypass them, or use them against each other.

This expands the attack surface for security researchers studying LLMs, enabling them to make LLMs to ignore their guardrails and act beyond their intended design.

"AI/ML systems such as LLM applications and agents are more than just the model and the prompt. They have multiple components besides the model, such as guardrails, all of which can be attacked on their own, or by gaming the interplay between them," said Gadi Evron, Co-Founder and CEO of Knostic, the world's first provider of need-to-know based access controls for LLMs.

For example, as a result of one of these new attacks, "Second Thoughts", when answering a sensitive question, Knostic researchers observed the LLM show signing of hesitation, having "second-thoughts" (hence the name) and retracting its answer, providing a new, redacted one.

"As LLM technologies stream answers to the user as they're being generated, enterprises cannot safely adopt LLM applications without making sure that the answers are provided when complete, as opposed to streaming as they are formed. Further, they'd need to deploy LLM-specific access controls such as need-to-know boundaries and context-aware permissions." Evron stated.

Evron further elaborated, "The LLM age requires a new form of identity based on the user's need-to-know, i.e. their business context. Looking beyond security and attackers, need-to-know based controls ensure organizations can safely proceed with adoption of GenAI systems, such as Microsoft Copilot for M365 and Glean."

Knostic Research's findings also highlight the importance of developing new AI security mechanisms. On the offensive side we need to expand the focus of evaluations and audits beyond the model and prompts. The systems surrounding LLMs should be considered holistically instead. On the defensive side, both application security (AppSec) and model security (ModSec) should be considered critical for the secure design of AI/ML systems.

This new attack class joins Prompt Injection and Jailbreaking as an attack type, but with a consideration for the wider AI/ML system components and architecture, and significantly expands the research possibilities into LLM attacks.

You can read Knostic's research directly on their blog, here: https://www.knostic.ai/blog/introducing-a-new-class-of-ai-attacks-flowbreaking

About Knostic.ai

Knostic.ai is the world's first provider of need-to-know based access controls for Large Language Models (LLMs). With knowledge-centric capabilities, Knostic enables organizations to accelerate the adoption of LLMs and drive AI-powered innovation without compromising value, security, or safety. For more details, visit https://www.knostic.ai/.

For more information

Gadi Evron, CEO, Knostic

Email: [email protected].

Media Contact

Gadi Evron, Knostic, 972 50-542-8610, [email protected], knostic.ai

SOURCE Knostic

Modal title

Contact PRWeb

  • 11AM ET Sunday – 8PM ET Friday
  • Contact Us

About PRWeb

  • About PRWeb
  • Partners
  • Partnership Programs
  • Editorial Guidelines
  • Resources

Why PRWeb

  • Why PRWeb
  • How It Works
  • Who Uses It
  • Pricing

Accounts

  • Create a Free Account
  • Log in
  • Contact Us

Do not sell or share my personal information:

  • Submit via [email protected] 
  • Call Privacy toll-free: 877-297-8921

Contact Cision

Products

About

My Services
  • All News Releases
  • Online Member Center
  • ProfNet
Cision Distribution Helpline
888-776-0942
  • Legal
  • Site Map
  • RSS
  • Cookie Settings
Copyright © 2025 Cision US Inc.