Job Scout — Automated Job Search & CV Tailoring Pipeline

What Was Built

Job Scout is a serverless job-search automation pipeline built on AWS Lambda that continuously scans multiple job boards, filters listings for genuine relevance, and uses an LLM to score each job against a candidate's CV before automatically tailoring the CV for strong matches. The pipeline pulls listings from five sources — Remotive, RemoteOK, Arbeitnow, Jobicy, and Himalayas — normalizing them into a unified job schema and deduplicating entries across queries and runs using DynamoDB. Each candidate job is scored by Groq's LLM (currently openai/gpt-oss-120b, after migrating off Gemini's restrictive free-tier quota), which judges both seniority fit and location eligibility rather than relying on brittle keyword or threshold rules. Matches above the scoring threshold have their CV automatically tailored, stored in S3, and rolled up into a daily digest email sent via Amazon SES. The entire cycle runs on a daily schedule via Amazon EventBridge with no idle compute cost. Infrastructure is defined in Terraform with remote state (S3 backend + DynamoDB locking) to support safe, repeatable deploys, and the CI/CD pipeline builds and pushes a Docker image to Amazon ECR before applying infrastructure changes — all authenticated via GitHub Actions OIDC with no long-lived AWS credentials.

Why Was It Built

Manually searching job boards, judging fit against dozens of postings, and rewriting a CV for each strong match is slow and easy to fall behind on, especially when casting a wide net across multiple remote-friendly sources. Job Scout was built to automate that entire loop — continuous discovery, LLM-judged fit scoring, and CV tailoring — while staying on a free-tier-friendly, serverless architecture. It also served as a personal proving ground for pairing LLM-based judgment (rather than rigid filtering logic) with production CI/CD patterns: containerized Lambda deployment, Terraform-managed remote state, and secretless OIDC authentication.

How It Works

On a daily EventBridge trigger, the Lambda function queries all five job sources in parallel, applying per-source query logic (e.g., Jobicy's tag parameter, queried per search term) to maximize relevant results. Each new listing is checked against DynamoDB to skip previously seen or already-scored jobs, with an attempt-based retry counter (max 3 attempts) accounting for run-to-run variance in LLM scoring. Remaining jobs are passed to Groq's LLM along with the candidate's CV and location; the model returns a fit score along with reasoning that weighs experience level and, for remote roles without an explicit "worldwide" marker, whether the listed regional restriction plausibly covers the candidate's location. Jobs clearing the score threshold trigger CV tailoring, with the customized CV written to S3 and match details logged to DynamoDB. At the end of the run, all matches are compiled into a digest and sent via SES. On the deployment side, pushing to the repository triggers a GitHub Actions workflow that authenticates to AWS via OIDC, builds and pushes a Docker image to ECR, and runs terraform apply against remote state to update the Lambda and supporting infrastructure.

What Broke and How Did I Fix It

The CI/CD pipeline initially failed with "already exists" AWS resource errors because each run started from an empty Terraform state; this was fixed by migrating to a remote backend (S3 state bucket + DynamoDB lock table), consolidating the workflow to self-bootstrap the backend if missing, and adding a break-glass workflow to clear stale locks after a killed mid-apply run. The original LLM provider, Gemini, hit its 20-requests/day free-tier quota almost immediately; after a first attempted swap to a since-Enterprise-gated Llama model returned 404s, the pipeline was migrated to Groq's openai/gpt-oss-120b with JSON mode and retry/backoff for transient rate-limit errors. A batch of scoring failures was also crashing the entire run; this was fixed by wrapping each job's scoring/tailoring step in its own try/except so a single failure no longer takes down the batch. Location filtering was another blind spot: an early "open_worldwide" check only matched literal phrases like "worldwide" or "global," silently dropping genuinely remote-eligible listings with real region lists (e.g., "USA," "LATAM, Argentina, Brazil, Mexico"). This was resolved by letting Groq judge location eligibility directly — treating it as a hard disqualifier only when a listed restriction clearly excludes the candidate's region, while scoring ambiguous cases normally — which recovered a meaningful share of previously dropped jobs (24 passing filter locally, up from 15, once verified against `test_sources.py`).