Generative AI Solutions Lead / LLM Application Developer / Prompt Evaluation Analyst

The Generative AI Solutions Lead / LLM Application Developer / Prompt Evaluation Analyst designs, develops, and evaluates applications powered by large language models (LLMs). This role translates business needs into practical AI solutions for knowledge retrieval, content generation, customer support, workflow assistance, and other use cases, balancing usefulness, reliability, cost, and user experience.

The position works closely with Product, Software Engineering, Data, Security, Legal, Operations, and business stakeholders. Responsibilities include assessing use cases, building application prototypes, integrating model APIs, implementing retrieval-augmented generation, and developing prompts and structured-output workflows. The role creates evaluation datasets, compares model and prompt configurations, analyses failure patterns, and tests whether applications meet defined requirements for accuracy, relevance, consistency, and appropriate behaviour.

At solutions-lead level, the role defines delivery priorities, guides architecture decisions, coordinates pilots, and establishes evaluation and release criteria. It oversees production monitoring, feedback collection, and continuous improvement while addressing issues such as unsupported answers, prompt injection, inappropriate data exposure, and unreliable tool execution. The role communicates performance limitations and trade-offs clearly, establishes suitable human review and escalation processes, and helps teams determine where generative AI delivers measurable business value.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related field, or equivalent practical experience.
• Experience developing generative AI applications, integrating LLMs, evaluating model outputs, or delivering applied AI solutions.
• For lead-level positions, demonstrated experience guiding technical delivery, prioritising use cases, and coordinating multidisciplinary teams.
• Understanding of LLM capabilities and limitations, including context constraints, output variability, unsupported responses, latency, and inference costs.
• Experience developing prompt templates, structured outputs, tool-calling workflows, and application-level validation.
• Practical knowledge of retrieval-augmented generation, document processing, embeddings, vector search, and retrieval-quality assessment.
• Ability to design representative evaluation datasets, scoring rubrics, reference answers, and repeatable testing workflows.
• Experience combining automated evaluations with human review, including awareness of bias and limitations in model-based scoring.
• Ability to analyse failures, compare model and prompt versions, conduct regression testing, and document evaluation findings.
• Familiarity with backend services, databases, cloud deployment, authentication, and integration with enterprise applications.
• Experience monitoring production applications for quality, errors, latency, usage, and cost, and using feedback to guide improvements.
• Understanding of privacy, access controls, sensitive-data handling, prompt-injection risks, and safeguards for tool-enabled applications.
• Ability to define business success measures, assess feasibility, and balance solution quality against delivery effort and operating costs.
• Excellent analytical, communication, stakeholder-management, and problem-solving skills, with the ability to explain AI behaviour and limitations clearly.