feat(android): add IME, software keyboard, and stroke input - #1067
Open
DepengWang wants to merge 34 commits into
Open
feat(android): add IME, software keyboard, and stroke input#1067DepengWang wants to merge 34 commits into
DepengWang wants to merge 34 commits into
Conversation
Real-device crash-loop root cause: OpenLessBackendWarmupActivity's window Surface could be destroyed (not just backgrounded) via the default back-button behavior, racing HWUI's worker-pool teardown and aborting the whole process with a native destroyed-mutex signal. The in-memory retry throttle then reset on every crash-restart, so the IME immediately relaunched the same Activity and crashed again forever. - Persist the backend-warmup retry timestamp in SharedPreferences (30s cooldown) so it survives a process crash/restart. - Remove the "relaunch the focused app's launcher Activity" fallback, which could land the user on the wrong screen entirely. - Route settings through a single tracked warmup-Activity instance instead of ever starting a second, untracked MainActivity host. - Override onBackPressed() to always background (never finish) this Activity, since finishing it is what destroys the window Surface. Verified on-device: 3x cold-start + open-settings + back-button stress cycles with zero crashes across the crash log.
- Replace the Stroke/EN toggle text labels with fixed-size PNG icons (drawable-nodpi/toggle_stroke.png, toggle_en.png) so they no longer change size with the UI language, and enlarge them for legibility. - Fix header/toggle position drift: the four keyboard panels (voice, English, stroke, stroke-number) each used different root padding for their own body rows, which shifted the logo/toggle left-right and up-down when switching modes. Compensate with per-panel margins so all four land at the same canonical 16dp/8dp inset. - Enlarge stroke-panel labels (通配/分词/符号/繁/清除/123) and move the small per-key numbers down to match the "0" key's native TextView top-gravity position instead of sitting almost flush with the edge. - Remove the microphone key's "CHERRY◆" watermark text. - Long-pressing the stroke keyboard's mic/"0" key now also starts dictation immediately, instead of only switching to voice mode.
…, 3-state shift
Stroke keyboard:
- Implement real word segmentation (分词): pressing it marks the current
character as a segment in memory without committing it, shows a
leading "commit whole word" candidate, and folds any pending segments
into the next candidate tap so the last character never needs an
explicit segment press.
- Number/symbol panel: backspace/enter/voice/return/symbols labels are
now always Chinese regardless of UI language (also fixes enter having
shown "←" instead of "回车"); "符号" now switches the digit grid to a
paged symbol set (3 pages x 12, default page is common punctuation)
with ▲/▼ paging, and the toggle relabels to "数字" while in symbol mode.
Added a "0" key (phone dial-pad style, under "8").
- The main stroke panel's "符号" key now jumps straight into the number
panel's symbol mode instead of just typing "#".
- Left punctuation rail is swipeable: 5 groups of 5 symbols, one swipe
advances exactly one group.
- Redrew the 丿/丶/乙 (keys 3/4/5) stroke glyphs by pixel-tracing a
reference screenshot instead of eyeballing bezier curves.
Cross-panel swipe gestures:
- Left/right swipe on any panel's root steps InputMode toward
Voice/English (Voice-Stroke-English order, clamped at both ends),
matching the toggle switch.
- Fixed the vertical punctuation-rail swipe fighting Android's system
gesture navigation and visibly flickering the IME closed/open: defer
the view-rebuild until after the touch gesture unwinds (post{} instead
of calling inline mid-dispatch), add systemGestureExclusionRects, and
requestDisallowInterceptTouchEvent once a drag is claimed.
English keyboard:
- Shift is now 3-state (lowercase/shift-once/caps-lock) with 3 distinct
icons (hollow arrow / solid arrow / solid arrow + underline), cycled
by tapping. Letter key labels now show the case that will actually be
typed, including reverting to lowercase after a one-shot capital.
Voice panel:
- Removed the keyboard-switch ("⌨") button. Rearranged @ / return /
backspace to @-return-backspace, all 48dp tall, return centered.
Widened @ and backspace 1.5x and restyled all three as a flat pill
(matching the mic capsule's own fill, no elevation) instead of the
stroke panel's raised-keycap look. Backspace supports long-press
repeat.
- Recording waveform lengthened (17 bars) with a genuine left-flowing
traveling wave instead of a static shape with per-bar shimmer.
Thinking/processing indicator's orbit ring now breathes (radius
grows and shrinks) on top of its existing rotation, uniformly across
all dots rather than sizing each dot off its own angle.
…service Investigating a reproducible "host app visibly refreshes when you tap its text field" report (seen in Meituan, but not app-specific) showed the cause is structural, not a crash: whenever the Tauri/Rust backend needs a cold start, ensureBackendReady() launches OpenLessBackendWarmupActivity, which steals the foreground for a moment. Logs show the host app's window losing focus and coming back with HIDE_WINDOW_GAINED_FOCUS_WITHOUT_EDITOR — the host app sees its editor focus yanked and re-initializes its screen. This does not fix that architecture (the backend still needs an Activity to host Tauri's event loop), but widens the window for the warmup to happen before the user is looking at someone else's text field: - Move the warmup check out of OpenLessImeService into a shared OpenLessBackendWarmupActivity.ensureBackendReady(Context), so it can be called from any Context rather than only from the IME reacting to a focused editor. - Call it from OpenLessRuntimeService.onStartCommand() as well. That service is START_STICKY, so the system can restart it on its own (e.g. after reclaiming it under memory pressure) with no IME interaction involved — those restarts now get a chance to warm the backend early. The existing 30s persisted cooldown is shared by all callers, so the extra call site cannot increase how often the warmup Activity launches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ngBxFcjxx36GkNKt7UWHV
(cherry picked from commit aac9bef on origin/main, adapted for this branch) Pulls in the write-gate race-condition fix from upstream main: a local optimistic settings write could be overwritten by an older `prefs:changed` broadcast that arrives after the write started but before it resolves, making a toggle briefly "bounce back" to its old value. Fixes this by tracking per-field pending local changes and correlating incoming broadcasts against writes this webview itself issued (via an upgraded PreferencesWriteGate with value comparison and recent-write history, replacing the old counter-only version). Adapted for divergence from upstream main (this branch already contains all of beta, which has evolved well past main in this area): - android::commands::settings::set_settings (the #[cfg(mobile)] variant) now returns Result<UserPreferences, String> instead of Result<(), String>, matching desktop's set_settings and what ipc/settings.ts's setSettings() already expects. This also fixes a latent Android-only bug: the frontend's setSettings() has been typed to return Promise<UserPreferences> since it already matches desktop, but on Android it silently got `undefined` back at runtime. Kept this branch's existing coord.backend() accessor style rather than upstream's newer coord.prefs()/coord.style_packs() (this branch has never adopted that accessor split). Did not add upstream's inline app.emit("prefs:changed", ...) calls or an app: AppHandle parameter: this branch already broadcasts prefs changes through BackendEventKind::PreferencesChanged -> tauri_events.rs, which mobile_runtime.rs wires up for Android too, so an inline emit would just double-broadcast. - HotkeySettingsContext.tsx: ported upstream's write-gate integration (applyIncomingPrefs, pendingLocalChangesRef, prefsChangeVersionRef, waitForPersistence) but dropped its locale-output-preference sync effect — that feature (outputPrefsForLocale/SupportedLocale) no longer exists anywhere in this branch's i18n module, so re-adding it would resurrect removed functionality and fail to compile. Re-applied this branch's own applyStackedLayoutFromPrefs/applyConservativeLayout calls (upstream doesn't have these) in the three spots this branch already had them. - shared.tsx: merged cleanly on its own; manually added upstream's new SectionDesc no-op placeholder component that was missing here. - ipc/settings.ts, ExperimentalSectionTitle.test.ts, i18n/experimentalTitles.test.ts, i18n/{en,ja,ko,zh-CN,zh-TW}.ts, MultimodalPipelineSection.tsx, Casks/openless.rb: kept this branch's existing content — either already a superset of upstream's fix, or differed only in formatting with identical text/logic. - LocalModelSection.tsx: kept deleted (this branch already replaced it via beta's "2.0 model settings page" rework; upstream's change here only applied ExperimentalSectionTitle to the old file). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ngBxFcjxx36GkNKt7UWHV
…ection-rule bridge - Add global swipe-down-to-dismiss (120dp) and raise the panel-switch swipe threshold to 100dp, with a slide transition between panels. - Rework the stroke keyboard's encode+candidate strip to a fixed 24dp/36dp layout with a flat background and hairline divider; the selected/first candidate is now bold light-blue (reusing the encode text's own color) instead of red, with a "show more" overlay that doesn't resize the panel. - Tighten the 3x4 stroke grid's key spacing to ~2dp and align it exactly with the red action-key column and punctuation rail. - Add a full-screen native keyboard-settings activity (long-press the logo) with a vibration intensity/duration framework, replacing the old popup overlay. - Fix interface-language sync: OpenLessApplication watched MainActivity by exact class, which never matched OpenLessBackendWarmupActivity where settings actually run, so the mirrored locale pref went stale. - Add clipboard history persistence and restyle clipboard quick actions to match the stroke panel's button style. - Wire a correction-rule JNI bridge so hand-edited dictation results feed back into the shared correction dictionary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ngBxFcjxx36GkNKt7UWHV
…h check - Learn a per-code personal preference for the stroke keyboard's direct candidate list: picking a non-top candidate boosts it for that exact stroke code from then on, ranked ahead of (not blended with) the static corpus frequency. Shares StrokeUserFrequency's store with phrase association, now global instead of per-target-app, with the entry cap raised 512 -> 3000 to fit both. - Add a full light theme for every keyboard panel, sourced from the OpenLess app's own Settings > Appearance choice (mirrored via the existing WebView poll into openless_ime_ui, not the raw OS dark-mode setting), with a same-session OS-setting fallback before that's ever been resolved once. - Fix swipe-down-to-dismiss stealing the stroke panel's punctuation rail gesture: excludes the rail's screen region by touch-down position instead of relying on child-vs-parent intercept timing. - Add a mic silence watch to the voice panel: no meaningful input level within 3s of a recording actually starting shows a bold red warning below the button (tap to restart the app) that persists until either a fresh recording attempt or real audio is detected. Fixes the timer never actually resetting, since toggleDictation() flips `recording` true before onCapsuleStateChanged's own reset check could ever fire. - Keyboard settings: raise vibration duration's cap 100ms -> 500ms, fire a test pulse on slider release instead of a separate button, and show personal-preference usage against its cap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ngBxFcjxx36GkNKt7UWHV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a native Android IME input path to OpenLess, including:
Motivation
This gives Android users a complete system keyboard workflow instead of relying only on the floating overlay or clipboard-style text insertion. Voice remains the primary input method, while English and stroke modes provide practical fallback input directly inside the same IME.
Verification