Files
karaoke-gen/docs/archive/2025-12-30-langchain-vertexai-to-genai-migration.md
Andrew Beveridge ☄️andClaude Opus 4.5 25be810442 feat(agentic): Migrate from ChatVertexAI to ChatGoogleGenerativeAI (#145)
## Summary

- Migrate from deprecated `langchain-google-vertexai` (gRPC) to
`langchain-google-genai` (REST) to fix silent hang issues during model
initialization
- Add initialization and warm-up timeouts using ThreadPoolExecutor for
cross-platform support
- Upgrade key logs from DEBUG to INFO with timing information for better
visibility

## Problem

Job `b1b2648c` hung indefinitely at "LangFuse prompt service
initialized" with no visible errors. Root cause: `ChatVertexAI` uses
gRPC which can hang during connection establishment with no timeout.

## Changes

| File | Changes |
|------|---------|
| `pyproject.toml` | Replace `langchain-google-vertexai` with
`langchain-google-genai` |
| `lyrics_transcriber_temp/pyproject.toml` | Same dependency change |
| `.../providers/config.py` | Add `initialization_timeout_seconds` (30s)
and `warmup_timeout_seconds` (15s) |
| `.../providers/model_factory.py` | Migrate to
`ChatGoogleGenerativeAI`, add timing logs |
| `.../providers/langchain_bridge.py` | Add ThreadPoolExecutor timeout
wrapper, upgrade logs to INFO |
| `.../agent.py` | Add timing logs for `classify_gap()` |

## Test plan

- [x] All 812 tests pass
- [x] E2E emulator tests pass
- [ ] Test with actual GOOGLE_API_KEY in staging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Configurable initialization and warmup timeouts.
* Support for Google AI Studio (new Google generative API) as a model
backend.

* **Improvements**
* Enhanced logging and timing visibility across model initialization,
warm‑up, invocation, and gap classification.
* Warm‑up timeouts treated as non‑fatal; initialization timeouts
surfaced clearly and reported.

* **Bug Fixes**
* Better handling/reporting for initialization timeouts and missing
Google API key.

* **Chores**
* Dependency key updated to new Google generative client; package
versions bumped.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-30 22:50:16 -05:00

3.2 KiB

LangChain Vertex AI to Google GenAI Migration

Date: 2025-12-30 PR: #145

Problem

Cloud job b1b2648c hung silently during agentic correction. No logs appeared after "LangFuse prompt service initialized". Investigation revealed ChatVertexAI (langchain-google-vertexai) was hanging during gRPC connection establishment with no timeout.

Solution

Migrated from langchain-google-vertexai to langchain-google-genai:

Before After
ChatVertexAI (gRPC-based) ChatGoogleGenerativeAI (REST-based)
Vertex AI service account auth GOOGLE_API_KEY (AI Studio)
No initialization timeout 30s init + 15s warm-up timeouts
DEBUG-level logs INFO-level logs with timing

Key Changes

Dependencies

# Before
langchain-google-vertexai = ">=3.1.1"

# After
langchain-google-genai = ">=2.0.0"

Model Creation

# Before (gRPC, can hang)
from langchain_google_vertexai import ChatVertexAI
model = ChatVertexAI(model=model_name, project=project, location=location)

# After (REST, explicit API key)
from langchain_google_genai import ChatGoogleGenerativeAI
model = ChatGoogleGenerativeAI(
    model=model_name,
    google_api_key=api_key,
    convert_system_message_to_human=True,  # Gemini doesn't support system messages
)

Timeout Implementation

Used ThreadPoolExecutor for cross-platform timeout (works on Windows, Linux, macOS):

from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(self._factory.create_chat_model, model, config)
    try:
        self._chat_model = future.result(timeout=30)
    except FuturesTimeoutError:
        raise InitializationTimeoutError("Model initialization timed out")

Warm-up Pattern

Added lightweight "warm-up" request after model creation to establish connection before real work:

warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
response = model.invoke([HumanMessage(content=warm_up_prompt)])

Files Modified

File Changes
pyproject.toml Replaced dependency, bumped version
lyrics_transcriber_temp/pyproject.toml Same
.../providers/config.py Added initialization_timeout_seconds, warmup_timeout_seconds
.../providers/model_factory.py Migrated to ChatGoogleGenerativeAI, added timing logs
.../providers/langchain_bridge.py Added ThreadPoolExecutor timeout wrapper, warm-up, INFO logs
.../agent.py Added timing logs for classify_gap

Why REST over gRPC

  1. Simpler debugging - HTTP requests are easier to trace than gRPC streams
  2. No connection pooling issues - REST is stateless
  3. Better timeout behavior - HTTP timeouts are well-understood
  4. Cloud-friendly - Works better through proxies and load balancers

Authentication

The new approach uses Google AI Studio API keys instead of Vertex AI service accounts:

Note: This uses Google AI Studio quotas, not GCP billing. For high-volume production use, consider keeping Vertex AI with proper timeout handling.