Financial AI / AI Agent Research and Development

Safely Switching AI Models — from 21-Item Isolation PASS to 24-Hour Monitoring

When Gemini and DeepSeek were compared across the same 60 patterns, DeepSeek had a cost advantage. However, we did not replace the production models solely on price. AI agents not only respond with text but also affect real data through recording, correction, billing, and safe shutdown. Therefore, switching itself must be validated as a single live production change.

This article is a research and validation log of the AI agent group, including indicators I developed and automated trading software (EA). In financial AI as well, we validate by separating judgment, execution, storage, and shutdown. The GOLD13 live account validation is publicly streamed as a 24-hour weekday trade drive.

Why the isolation environment was necessary

If we conduct comparative tests using the production port 4002, validation data may mix into the production DB, potentially affecting billing and records. Therefore, we launched a verification-only instance at 4102 and performed 9 rounds of real HTTP and real DB operations without touching the production connection.

Mock alone cannot verify real DB constraints, duplicate prevention, write-offs when balance is insufficient, or the order of tool invocations. In the isolation environment, we use the same save path as production, and after completion stop only the validation process. We also confirmed that production 4002 returns HTTP 200.

Scope verified in 21 items

• New records for at-bats, pitches, and game results
• Corrections of existing results and target identification
• Billing events and non-writes when balance is insufficient
• Prevention of duplicate charges for the same target
• No unnecessary Markdown symbols are saved
• The actual DB state changes as expected after model responses
• The production system remains operational after verification ends

This real HTTP / real DB verification PASSed all 21 items. However, only a limited verification path was proven here. It is not a full production quality assurance.

Quality gaps remaining after 60-pattern comparison

The end reasons were Gemini stop 45 and tool_calls 15, DeepSeek stop 43 and tool_calls 17, with length end and API errors at zero for both. Meanwhile, for DeepSeek, key technical questions required visual inspection were 2 cases, handoff to coaches across 4 of 5 languages, with zh-TW not detected.

Recording tool triggers: at-bat 3/5, pitch 4/5, game results 5/5, result corrections 5/5. There were cases where verification questions were returned to avoid misrecords, but the criterion of completing in one pass was not met. We evaluated cost and quality separately, noting inexpensive does not equal equivalent quality.

How to read the cost difference

In a conservative round-trip estimate, Gemini 2026 promotional price averages about 0.689 yen, 2027 formal price equivalent about 1.378 yen, DeepSeek peak band about 0.501 yen, off-peak about 0.251 yen. DeepSeek cached 307,065 of 318,893 input tokens, giving an actual hit rate of 96.3%.

Costs accumulate with high-volume processing, but unless we include the costs of correcting misrecords and misbilling, they do not reflect actual operating costs. Therefore, we do not decide live switches solely on unit price.

Phasewise production switch

1. Switch only a limited subset of agent families
2. Monitory 24 hours for response quality, tool invocations, errors, and save results
3. Predefine conditions to revert to the old model
4. If no issues, expand the scope

We keep the production provider's override settings empty and have not implemented proactive switching yet. We separate criteria for “cheap,” “passed tests,” and “safe to go production,” and only proceed to the next stage when evidence is gathered.

This article covers model migration and DB verification and does not embed unrelated trading videos, as it does not correspond to a specific GOLD transaction.

GOLD13 Sales Page
Weekday 24-Hour Trade Drive

This article is a research and validation log and does not guarantee profits or future performance.

Acceptance criteria for staged migration

After switching limited system chains, during 24 hours of operation we record API errors, average response time, tool invocation rate, confirmation question rate, DB write success rate, duplicate charges, and rollback occurrences. Merely not failing is not a pass.

If deviations occur from expectations, revert to the old model and compare inputs, model responses, tool arguments, and DB results item by item. We provide evidence that distinguishes whether the cause lies in the prompt, the model, the tools, or the storage layer.

Failures emphasized in real DB testing

The most dangerous cases are when the model appears to succeed but does not write to the DB, or when the same operation is resent causing duplicate charges or records. Therefore we do not decide pass/fail solely from the response body; we verify the actual stored state, balance, and target ID after saving.

We also confirm that writes do not occur when balance is insufficient, that re-executions on the same target do not incur additional charges, and that corrections do not update different players or different games. For AI agents with external effects, integrity takes precedence over response quality.

Rollback design

• Keep the old model settings in a state that can be reverted immediately
• Align old and new log formats to enable comparison
• Re-sending mid-process does not cause duplicate actions
• Limit the switching scope to narrow the fault domain
• In case of anomalies, prioritize safe shutdown over automatic switching

Changes that cannot be rolled back are not deployed in production even if small tests succeed. AI models can respond slightly differently to the same input, so we ensure re-execution safety on the data layer.

24-hour monitoring is not the end

Even if no issues are found on the first day, rare multilingual use, insufficient balance, ambiguous targets, timeouts, or provider outages may occur. We separate normal, boundary, and failure states and evaluate using real data over a fixed period.

We also track costs and latency as processing loads increase. If the cache hit rate drops, we assess how much DeepSeek’s cost advantage remains and whether peak-band response fits within business hours.

Common thinking with financial AI

Model switching is similar to adding new logic to an FX EA. Backtesting uses unit tests and replay; isolation DB is a demo environment; 24-hour monitoring of limited systems corresponds to a small live trial. A single-stage success is not sufficient for a final pass.

In financial AI as well, it is not enough that the decision sentence reads naturally to approve an order; only when signal, order, position, settlement, and logs all align do we deem an End-to-End pass. We will reflect this migration procedure not only in model selection but also in deploying EAs as a standard.

Evidence to be disclosed

If a production switch is implemented, we summarize the target systems, start and end times, number of processed items, errors, tool invocations, DB inconsistencies, and rollbacks. We disclose not only successes but also reasons for stoppages and untested items in the same article.

Conditions to suspend 24-hour monitoring

If any DB inconsistency, duplicate charge, saving to the wrong target, or write when balance is insufficient occurs even once, a rollback is executed immediately. API error rates and response delays do not expand the target scope beyond pre-set thresholds.

We do not weigh minor wording differences and potential data destruction equally; failures closer to external actions are judged more strictly. At monitoring end, we publish all counts for processed items, successes, pending, failures, retries, and rollbacks, selecting only the results with no issues.