Files
karaoke-gen/backend/workers/stale_review_processor.py
Andrew Beveridge ☄️andClaude Opus 5.5 2838d2dd1d fix(credits): keep credits_charged through job creation; one helper for confirm/refund/select (v0.278.2) (#1180)
## Summary
Fixes `credits_charged` being wiped from `state_data`, which made
duration confirm, refunds and the web cost dialog disagree. Mobile E2E
against the local stack found it: a 15-minute track showed "1 credit in
total (1 more)", and confirming the true total was rejected with 409.

**Root cause:** `create_job` debits the credit and records
`state_data.credits_charged`. But create-from-search, audio-search
create, made-for-you orders (which also lost `payment_bypassed`) and the
admin source override then replaced the whole `state_data` map, deleting
the field. Readers filled the gap with different defaults:
- reconcile: 1
- confirm-duration: 0
- 48h expiry: 0

**Fix**
- **Writers:** per-key `state_data.<key>` updates instead of whole-map
writes. Retry-from-scratch keeps the billing keys. Reconcile records
`credits_charged` when it pauses a job.
- **Readers:** one helper, `backend/services/job_credits.py`
(`get_credits_charged`, `is_payment_bypassed`). Every reader uses it:
reconcile, confirm-duration, `/select`, the bulk worker,
`_refund_credit_for_job` and the 48h expiry.
- For legacy jobs that lack the field, it returns what `create_job`
actually debited: 0 for bypassed, admin, tenant or no-email jobs,
otherwise 1.
- **`/select` and the bulk worker** never bill payment-bypassed jobs
(made-for-you, tenant, admin-created). This includes legacy ones whose
flag was wiped.
- **API responses:** `GET /api/jobs/{id}` and the job lists return the
resolved `credits_charged` when the stored field is missing. This is
response-only and never persisted, so the web and mobile clients
acknowledge exactly `credits_charged + pending`.
- **Confirm-duration** records the full total, so cancelling after a
confirm refunds it all. The 48h expiry refunds once:
`cancel_job(refund=False)` stops a second refund.
- **Admin source override** clears the old source's duration, selection
and confirm markers and keeps `credits_charged`. The next `/select` then
charges only the difference.
- **Web cost hook:** uses the server's resolved figure.
- Version 0.278.2.

## Testing
- New `test_credits_charged_consistency.py` covers:
- reconcile, then confirm with total 2, then cancel: 2 refunded, for new
and legacy jobs
  - 409 on the old total
  - refund amounts unchanged for jobs that have the field
  - `/select` and the bulk worker never deducting on bypassed jobs
  - the override followed by `/select` charging only the difference
- Stale-expiry legacy tests, and an emulator test for the dotted-path
merge (runs in CI).
- Backend unit tests: 6411 passed. Emulator integration tests run in CI;
locally they collided with a running local stack. Frontend jest: 1754
passed.
- An independent review found one Critical issue (`/select` billing
bypassed jobs) and one Important issue (web/server default mismatch).
Both are fixed and re-reviewed clean. Local CodeRabbit raised one minor,
which is fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011yLH4kWXMqjeKfZUG3ZiWZ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:35 -04:00

397 lines
18 KiB
Python

"""
Stale review processor.
Detects jobs stuck in review (awaiting_review / in_review) or awaiting duration
confirmation (awaiting_duration_confirm) and takes action:
- 24h: sends a gentle reminder email with expiry warning
- 48h: auto-cancels the job and refunds the user's credits
Called by Cloud Scheduler via an internal endpoint (hourly).
"""
import asyncio
import logging
from datetime import datetime, timezone
from typing import Any, Dict
from backend.config import get_settings
from backend.i18n import get_locale_prefix
from backend.models.job import JobStatus
from backend.services.email_service import get_email_service
from backend.services.firestore_service import FirestoreService
from backend.services.job_manager import JobManager
from backend.utils.test_data import is_test_email
logger = logging.getLogger(__name__)
# Configurable thresholds (hours since blocking_state_entered_at)
REVIEW_REMINDER_HOURS = 24
REVIEW_EXPIRY_HOURS = 48
# Final "N hours left" nudge for review states (REVIEW_LAST_CHANCE_ENABLED)
REVIEW_LAST_CHANCE_HOURS = 42
# Statuses treated as "review" states (1-credit refund via cancel_job)
_REVIEW_STATUSES = {JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW}
# Statuses treated as "duration confirm" states (credits_charged refund)
_DURATION_CONFIRM_STATUSES = {JobStatus.AWAITING_DURATION_CONFIRM}
async def process_stale_reviews() -> Dict[str, Any]:
"""Async entry point: run the (entirely synchronous) processor in a worker thread.
The body does up to ~1500 Firestore job reads, credit refunds and email
sends (Postmark API → SMTP fallback, up to ~15s each). It used to be an
``async def`` with no awaits, so the hourly cron froze every request on the
API instance for its whole duration (2026-10-03 analysis).
"""
return await asyncio.to_thread(process_stale_reviews_sync)
def process_stale_reviews_sync() -> Dict[str, Any]:
"""
Query for stale review / duration-confirm jobs and take action.
- Jobs in these states for >= 48h are auto-cancelled with credit refund
- Jobs in these states for >= 24h (but < 48h) get a reminder email
Review states (AWAITING_REVIEW, IN_REVIEW):
- 24h reminder: send_review_reminder
- 48h expiry: cancel_job (auto-refunds 1 credit) + send_review_expired
Duration-confirm state (AWAITING_DURATION_CONFIRM):
- 24h reminder: send_duration_confirm_reminder
- 48h expiry: add_credits(credits_charged) + cancel_job + send_duration_confirm_expired
The full credits_charged amount (not flat 1) is refunded.
Excludes made-for-you and tenant jobs. Jobs owned by automated-test accounts
(e.g. a stranded E2E job) are still expired, but never emailed.
Returns:
Summary dict with counts and any errors encountered.
"""
job_manager = JobManager()
firestore = FirestoreService()
email_service = get_email_service()
reminders_sent = 0
jobs_expired = 0
errors = []
# Query for jobs in all blocking states
stale_jobs = []
for status in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW, JobStatus.AWAITING_DURATION_CONFIRM]:
try:
jobs = firestore.list_jobs(status=status, limit=500)
stale_jobs.extend(jobs)
except Exception as e:
error_msg = f"Failed to query jobs with status {status.value}: {e}"
logger.error(error_msg)
errors.append(error_msg)
if not stale_jobs:
logger.info("No stale review jobs found")
return {
"status": "completed",
"reminders_sent": 0,
"jobs_expired": 0,
"errors": errors,
}
logger.info(f"Found {len(stale_jobs)} jobs in review states, checking for stale ones")
now = datetime.now(timezone.utc)
for job in stale_jobs:
try:
# Skip made-for-you jobs (admin-controlled)
if getattr(job, 'made_for_you', False):
continue
# Skip tenant/white-label jobs (B2B partners)
if getattr(job, 'tenant_id', '') and job.tenant_id:
continue
# Get blocking state entry time from state_data
state_data = job.state_data or {}
# Skip requests-board community picks — their ownership handoff worker
# (community_handoff) owns their review lifecycle; auto-cancel+refund
# here would kill a track voters are still waiting on.
if state_data.get('community_request_id'):
continue
blocking_entered_at_str = state_data.get('blocking_state_entered_at')
if not blocking_entered_at_str:
continue
# Parse the naive UTC ISO string and make it timezone-aware
try:
blocking_entered_at = datetime.fromisoformat(blocking_entered_at_str)
if blocking_entered_at.tzinfo is None:
blocking_entered_at = blocking_entered_at.replace(tzinfo=timezone.utc)
except (ValueError, TypeError):
logger.warning(
f"Job {job.job_id}: invalid blocking_state_entered_at: {blocking_entered_at_str}"
)
continue
hours_elapsed = (now - blocking_entered_at).total_seconds() / 3600
is_duration_confirm = job.status in _DURATION_CONFIRM_STATUSES
# A stranded E2E job must not email the test account (it forwards to
# a real inbox) — still expire it, just silently.
send_emails = bool(job.user_email) and not is_test_email(job.user_email)
if hours_elapsed >= REVIEW_EXPIRY_HOURS:
if is_duration_confirm:
# Duration-confirm expiry: refund full credits_charged, then cancel.
# Check idempotency flag first.
if state_data.get('duration_confirm_expiry_processed'):
continue
logger.info(
f"Job {job.job_id}: duration-confirm stale for {hours_elapsed:.1f}h "
f"(>={REVIEW_EXPIRY_HOURS}h), auto-expiring with credits_charged refund"
)
from backend.services.job_credits import get_credits_charged
credits_charged = get_credits_charged(job)
credits_refunded = 0
# Refund the full charged amount (not flat 1) BEFORE cancelling.
# We also mark credit_refunded=True on the job first so that
# cancel_job's built-in _refund_credit_for_job skips its 1-credit
# refund and we don't double-charge the user.
if credits_charged > 0 and job.user_email:
try:
from backend.services.user_service import get_user_service
user_service = get_user_service()
ok, _, _ = user_service.add_credits(
job.user_email,
amount=credits_charged,
reason="duration_confirm_expired",
job_id=job.job_id,
)
if ok:
credits_refunded = credits_charged
# Prevent any other path's refund (cancel_job is also
# told refund=False below).
try:
firestore.update_job(job.job_id, {'credit_refunded': True})
except Exception as flag_err:
logger.error(
f"Job {job.job_id}: refunded but failed to set credit_refunded: {flag_err}"
)
except Exception as refund_err:
logger.error(
f"Job {job.job_id}: failed to refund {credits_charged} credits: {refund_err}"
)
cancel_kwargs = {}
if credits_refunded > 0:
# Already refunded in full above — never let cancel_job
# refund again (even if the credit_refunded write failed).
cancel_kwargs["refund"] = False
cancelled = job_manager.cancel_job(
job.job_id,
reason="Duration confirmation not completed within 48 hours",
**cancel_kwargs,
)
if cancelled:
jobs_expired += 1
# Set idempotency flag AFTER a successful cancel so that a
# cancel_job failure leaves the job un-flagged and re-processable
# on the next run (the refund is idempotent via add_credits).
try:
firestore.update_job(job.job_id, {
'state_data.duration_confirm_expiry_processed': True,
'state_data.duration_confirm_expiry_processed_at': datetime.now(timezone.utc).isoformat(),
})
except Exception as flag_err:
logger.error(
f"Job {job.job_id}: failed to set duration_confirm_expiry_processed: {flag_err}"
)
if send_emails:
try:
email_service.send_duration_confirm_expired(
to_email=job.user_email,
artist=job.artist,
title=job.title,
credits_refunded=credits_refunded,
locale="en",
)
except Exception as email_err:
logger.error(
f"Job {job.job_id}: failed to send duration_confirm_expired email: {email_err}"
)
else:
logger.warning(f"Job {job.job_id}: cancel_job returned False")
else:
# Review-state expiry: cancel job (auto-refunds 1 credit via cancel_job)
logger.info(
f"Job {job.job_id}: review stale for {hours_elapsed:.1f}h (>={REVIEW_EXPIRY_HOURS}h), "
f"auto-expiring"
)
cancelled = job_manager.cancel_job(
job.job_id,
reason="Review not completed within 48 hours"
)
if cancelled:
jobs_expired += 1
# Send expiry notification email
if send_emails:
try:
from backend.services.user_service import get_user_service
user_service = get_user_service()
credits_balance = user_service.check_credits(job.user_email)
# Look up user locale for email
user_locale = "en"
try:
user = user_service.get_user(job.user_email)
if user and user.locale:
user_locale = user.locale
except Exception:
pass
email_service.send_review_expired(
to_email=job.user_email,
artist=job.artist,
title=job.title,
credits_balance=credits_balance,
locale=user_locale,
)
except Exception as email_err:
logger.error(
f"Job {job.job_id}: failed to send expiry email: {email_err}"
)
else:
logger.warning(f"Job {job.job_id}: cancel_job returned False")
elif (
hours_elapsed >= REVIEW_LAST_CHANCE_HOURS
and not is_duration_confirm
and get_settings().review_last_chance_enabled
and not state_data.get('last_chance_reminder_sent')
):
hours_left = max(1, round(REVIEW_EXPIRY_HOURS - hours_elapsed))
logger.info(f"Job {job.job_id}: {hours_left}h before expiry, sending last-chance reminder")
# Mark first (even if the send fails) so we never retry every hour
try:
firestore.update_job(job.job_id, {
'state_data.last_chance_reminder_sent': True,
'state_data.last_chance_reminder_sent_at': datetime.now(timezone.utc).isoformat(),
})
except Exception as update_err:
logger.error(f"Job {job.job_id}: failed to mark last_chance_reminder_sent: {update_err}")
continue
if send_emails:
try:
from backend.services.job_notification_service import review_login_url_for_job
from backend.services.user_service import get_user_service
user = get_user_service().get_user(job.user_email)
user_locale = (user.locale if user and user.locale else "en")
email_service.send_review_last_chance(
to_email=job.user_email,
artist=job.artist,
title=job.title,
review_url=(
review_login_url_for_job(job, locale=user_locale)
or f"{email_service.frontend_url}{get_locale_prefix(user_locale)}"
f"/app/jobs?utm_source=email&utm_medium=email&utm_campaign=review_last_chance"
f"#/{job.job_id}/review"
),
hours_left=hours_left,
locale=user_locale,
)
except Exception as email_err:
logger.error(f"Job {job.job_id}: failed to send last-chance reminder: {email_err}")
reminders_sent += 1
elif hours_elapsed >= REVIEW_REMINDER_HOURS:
# Check if expiry reminder was already sent
if state_data.get('expiry_reminder_sent'):
continue
logger.info(
f"Job {job.job_id}: stale for {hours_elapsed:.1f}h (>={REVIEW_REMINDER_HOURS}h), "
f"sending reminder"
)
# Send reminder email (different email per status)
if send_emails:
try:
# Look up user locale for email
user_locale = "en"
try:
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(job.user_email)
if user and user.locale:
user_locale = user.locale
except Exception:
pass
if is_duration_confirm:
email_service.send_duration_confirm_reminder(
to_email=job.user_email,
job_id=job.job_id,
artist=job.artist,
title=job.title,
locale=user_locale,
)
else:
from backend.services.job_notification_service import (
review_login_url_for_job,
)
email_service.send_review_reminder(
to_email=job.user_email,
artist=job.artist,
title=job.title,
job_id=job.job_id,
locale=user_locale,
# kjbox singers never signed in on gen → one-click sign-in link
review_url=review_login_url_for_job(job, locale=user_locale),
)
except Exception as email_err:
logger.error(
f"Job {job.job_id}: failed to send reminder email: {email_err}"
)
# Update state_data to mark reminder as sent (even if email failed,
# to avoid retrying every hour)
try:
firestore.update_job(job.job_id, {
'state_data.expiry_reminder_sent': True,
'state_data.expiry_reminder_sent_at': datetime.now(timezone.utc).isoformat(),
})
except Exception as update_err:
logger.error(
f"Job {job.job_id}: failed to update expiry_reminder_sent: {update_err}"
)
reminders_sent += 1
except Exception as e:
error_msg = f"Job {job.job_id}: unexpected error: {e}"
logger.exception(error_msg)
errors.append(error_msg)
logger.info(
f"Stale review processing complete: "
f"{reminders_sent} reminders sent, {jobs_expired} jobs expired, "
f"{len(errors)} errors"
)
return {
"status": "completed",
"reminders_sent": reminders_sent,
"jobs_expired": jobs_expired,
"errors": errors,
}