GenAI Analyst
About Arato
Arato is the reality check for GenAI products. We focus on one critical problem: ensuring GenAI-powered experiences deliver real business value – safely, reliably, and consistently – before they reach customers.
Our core product, Arato Simulate, is a comprehensive evaluation and simulation platform built for modern GenAI applications – from AI copilots, generative AI-powered workflows and customer-facing assistants. We simulate realistic, high-volume user behavior across a broad range of scenarios: expected use cases, edge cases, adversarial inputs, and domain-specific flows tailored to each product.
Arato enables teams to test at scale – running thousands of diverse, synthetic user journeys – while going deep on analysis. We don’t just surface failures; we quantify accuracy, consistency, regressions, security risks, and user experience breakdowns with structured, data-driven insights. The result is clear visibility into product readiness, risk exposure, and performance trade-offs.
Trusted by forward-thinking companies across industries, our mission is to help the world “Do AI Right” – by catching what matters before it impacts customers, revenue, and brand trust.
The Role
We’re looking for a GenAI Analyst to help ensure the insights we provide about GenAI products are accurate, meaningful, and useful for customers.
Our platform automatically tests GenAI-powered products by simulating large volumes of user interactions and generating structured analysis about how those products behave. Your role will be to review and validate these results – making sure the findings correctly reflect what actually happened during testing.
To do this well, you’ll need to understand each customer’s product, their goals, and the scenarios being tested. You’ll analyze AI-generated conversations and system responses to determine whether behavior is correct and whether issues are identified and explained clearly. As you review results, you’ll also identify opportunities to improve the analysis itself – suggesting refinements to the prompts and evaluation logic so the system produces more accurate and useful insights.
You’ll work closely with our product and engineering teams to improve the quality of analysis, refine testing scenarios, and help ensure the reports we deliver provide reliable insights for teams building GenAI products.
This role requires strong analytical thinking, attention to detail, and curiosity about how AI systems behave in real-world situations.
Qualifications
- Strong analytical thinking and attention to detail.
- Ability to interpret complex conversations and system behavior.
- Clear written communication skills.
- Curiosity about how AI systems behave in real-world scenarios.
- Ability to work independently and investigate ambiguous cases.
- Interest in learning about different industries and product use cases.
- Experience working with AI products, chatbots, or LLM-based systems.
- Familiarity with prompt engineering or AI evaluation concepts.
- Background in QA, product analysis, research, or data analysis – Advantage.
Why Arato.ai?
- Build the category-defining product in AI evaluation. AI evaluation and simulation is foundational to the future of GenAI – and it’s still largely unsolved. You’ll help define what “good” looks like.
- Extreme product leverage. Small team, big surface area, high ownership. Your decisions and execution will directly shape the product and the company’s trajectory.
- Real customer impact. You’ll work closely with product and engineering teams building GenAI products, shaping what we build based on real-world needs.
- High standards, high velocity. We move fast, focus on what matters, and cut what doesn’t. Quality and clarity matter.
- Strong culture, real flexibility. Collaborative team, direct communication, flexible working arrangements, and competitive compensation.