AI Operations Analyst

• Review a risk-weighted sample of AI chat conversations and Personalized Credit Insights summaries, and flag inaccurate, unsupported, off-topic, or unsafe responses.

• Categorize member feedback, guardrail blocks, and escalations into a failure-mode taxonomy, and track how it changes over time.

• Own the golden regression dataset by adding labeled examples from real production misses, keeping labels consistent, and documenting the rubric behind them.

• Label data to help measure and improve automated evaluators, including double-labeling a share of items to measure reviewer agreement.

• Publish a weekly quality report covering intent coverage, resolution and containment, refusal and guardrail rates, safety flags, top failure modes, and recommended fixes.

• Escalate flagged content to Legal, Compliance, and Trust & Safety with clear severity ratings, and follow each item through to resolution.

• Use SQL and dashboards to understand what members ask, where the experience falls short, and where quality is drifting.

• Partner with Product, the Applied AI Engineer, and QA engineers on review tooling, rubrics, and sampling methods, and verify fixes after each release.

• Bachelor's degree in a related field, or equivalent experience, and demonstrated experience in a quality, analytical, or operations role reviewing customer conversations, such as CX or QA, support operations, compliance QA, or conversation design.

• Strong SQL skills and comfort building and reading dashboards in a BI tool such as Tableau.

• Sound judgment and close attention to detail, with the ability to apply a written rubric consistently and explain your decisions.

• Strong written communication, including clear failure descriptions and concise summaries for leadership.

• Comfort working with sensitive consumer data in a regulated environment, and the discretion that requires.

• Experience working in Agile/Scrum teams.

• Experience in consumer finance, credit, banking, fintech, or other regulated environments (FCRA, ECOA, UDAAP), knowledge of credit reports and FICO Scores, and experience with conversational AI or LLM-based products, data labeling, rubric design, inter-rater agreement, failure-mode taxonomies, basic Python and Jupyter, or Legal and Compliance escalation processes is a plus.

• An inclusive culture strongly reflecting our core values: Act Like an Owner, Delight Our Customers and Earn the Respect of Others.

• The opportunity to make an impact and develop professionally by leveraging your unique strengths and participating in valuable learning experiences.

• Highly competitive compensation, benefits and rewards programs that encourage you to bring your best every day and be recognized for doing so.

• An engaging, people-first work environment offering work/life balance, employee resource groups, and social events to promote interaction and camaraderie.
• The targeted base pay range for this role is: $105,000 to $165,000 with this range reflecting differences in candidate knowledge, skills and experience.

FICO makes a real difference in the way businesses operate worldwide:

• Credit Scoring — FICO® Scores are used by 90 of the top 100 US lenders.
• Fraud Detection and Security — 4 billion payment cards globally are protected by FICO fraud systems.
• Lending — 3/4 of US mortgages are approved using the FICO Score.