Penetration Testing and Generative AI
Why Penetration Testers Benefit from AI
Using AI during Capture The Flag (CTF) competitions or customer engagements offers significant advantages. However, during customer engagements, technical teams should be cautious about the data they feed into AI services. Being general and avoiding sensitive information is crucial, as AI services may temporarily store user inputs. Despite these limitations, AI can significantly optimize workflows, such as by speeding up tool development. For instance, creating a Python tool to fuzz for subdirectories or subdomains is realistic with AI’s help, even though tools like WFUFF, FFUF, GOBUSTER, DIRB, and Burp Suite are already widely used.
In unique situations requiring team input, AI can effectively fill knowledge gaps and create more independent contributors. While escalation procedures remain necessary, AI can boost productivity across technical teams, fostering a collaborative and engaging experience in troubleshooting, tool development, and replication tasks. Time complexity is always a challenge in penetration testing assessments, where deadlines often make building original tools or exploring innovative approaches challenging. With AI as a supportive companion, penetration testers can stay creative, avoiding repetitive workflows that rely solely on past assessments' successful techniques.
The Value for Educating Technical Teams
AI tools like ChatGPT have become invaluable for discovering new ways to work with scripting and programming languages and tackling tough challenges. Conversing with AI can quickly fill knowledge gaps, enhancing both knowledge acquisition and retention. In penetration testing, AI serves as a sounding board, prompting testers to explore alternative paths and revisit skipped checks, avoiding the typical “rabbit holes” and directing focus toward viable solutions. This can help testers prioritize simple paths before diving into more complex, potentially non-existent vulnerabilities, ultimately saving time and increasing efficiency.
AI also simplifies learning programming languages for exploit and software development. An AI service’s ability to explain code line-by-line enhances understanding of libraries, optimization techniques, and helps generate new ideas. The ability to repeatedly create new concepts—such as alternative approaches, unconsidered functionalities, educational insights, and debugging support—adds immense value.
However, using AI for development isn’t without challenges. AI-generated solutions are not always sophisticated or bug-free, sometimes creating additional issues rather than solving them. These outputs may lack full developer intent, or require more time to refine than to create manually. For this reason, writing code manually is often more efficient, using AI to troubleshoot and improve non-functioning sections as needed.
Using Generative AI Against AI Services
A fascinating aspect of AI is using it to “outsmart” other AI systems. Prompt injection attacks, which exploit AI vulnerabilities, have demonstrated how specific phrasing can bypass LLM safeguards. At BSides Connecticut, a recent cybersecurity conference, there was even a CTF challenge centered on bypassing an LLM’s constraints to retrieve a flag. Although developers are continually mitigating these vulnerabilities, they remain a concern in AI applications.
Consulting AI for insights on targeting AI services in ethical hacking offers penetration testers a new, dynamic challenge. Few fields—like cloud computing—have so rapidly evolved to become integral to cybersecurity. Through AI documentation, skill-building, and tools like Hack The Box, penetration testers can practice these new skills in virtual environments, honing methods for attacking live AI targets without legal or ethical repercussions.
Exploring the Dark Side of AI Use-Cases
AI has also found a foothold in nefarious activities. While ethical safeguards are in place for free AI products like OpenAI’s models, the dark web has seen the rise of unrestricted LLMs such as "MalwareGPT" (formerly "CypherGPT Unrestricted AI"). These models promise to answer any prompt—regardless of ethical boundaries—and come preconfigured to generate malicious code. The demand for such unrestricted tools highlights their appeal to threat actors and others looking to bypass the ethical restrictions of legitimate platforms. For malicious users, these AI models open troubling opportunities to meet illicit or unethical goals without limitations.
Bridging Computer Science Fundamentals with AI Skills for Penetration Testing
Computer science programs cover core topics like programming, data structures, operating systems, algorithms, and networking. While these are essential for general computing, they often fall short on cybersecurity techniques specific to AI and penetration testing.
Penetration testers can benefit from specialized knowledge in machine learning, AI model training, and the security considerations surrounding LLMs (Large Language Models). Learning about transformers, natural language processing, and prompt engineering allows testers to understand—and potentially manipulate—AI functionality during assessments. Skills in API testing and adversarial AI techniques are also valuable for evaluating AI services.
In terms of depth, testers should aim for proficiency in AI-based tools, understanding enough to use, adapt, and assess them in engagements. Practicing with frameworks like TensorFlow or real-world scenarios on platforms like Hack The Box provides vital experience. This familiarity equips penetration testers for an AI-augmented threat landscape, enabling them to harness AI tools productively while upholding ethical standards.
