vermilion10

Detecting Comic Lettering Under a False-Positive Budget: A Measurement-Driven Study of an In-Browser Comic Translator

Sat, 26 Sep 2026 00:00:00 GMT

vermilion10 · Personal project write-up · September 2026

::github{repo="vermilion10/comic-translator"}

Abstract

Automatic comic translation chains five stages: text detection, recognition, erasure, translation and typesetting. Each stage acts on every region the detector returns, so a false detection is not a harmless extra box. It erases part of the artwork and prints an invented translation on it. We built a comic translator that runs entirely inside a Chrome extension. It covers Japanese, Chinese and Korean sources, and every model runs client-side. We use it to study how text detectors should be evaluated when false positives carry this cost. Seen in that light, the usual evaluation is misleading. Our first fine-tuned detector tripled sound-effect recall at the default threshold, yet at the pipeline's tolerated false-positive rate of 0.9% it was worse than the model it was meant to replace. We therefore compare detectors at matched spurious-detection ceilings. We also correct a reference set whose ground truth had been pooled from the proposals of the baseline detector and one text finder, and we resample pages and training seeds before believing a ranking. Across six YOLO fine-tunes we found that fixing the labels and mining hard negatives moved the operating curve, while changing the starting checkpoint did not. We also found that seed-to-seed variance was as large as the difference between recipes. We then trained our own detector, TextSeg: a MobileNetV3 and FPN text-probability map with an ignore-aware DBNet-style loss. Combined with negatives mined against TextSeg itself, it beats the baseline detector at the 0.9% ceiling on all three seeds, with page-bootstrap probabilities of 0.92 to 0.97. Its sound-effect recall roughly triples (0.41 against 0.15). Around the detector we report measured choices for per-script OCR, balloon-aware erasing, contrast-guaranteed caption plates and free-tier cloud translation. We designed the evaluation protocol for comic-translation pipelines that erase what they detect, and we expect it to be useful for similar pipelines, though we have tested it only on this one.

Keywords: comic text detection, manga translation, operating-point evaluation, pooling bias, hard-negative mining, in-browser inference

1. Introduction

1.1 The problem

A reader who opens a Japanese manga, a Chinese manhua or a Korean webtoon in a browser sees text that most of them cannot read. Tools that translate these images already exist. zyddnys/manga-image-translator [50], comic-translate [49] and koharu [10] follow the same five-stage pattern: detect the lettering, recognise it, erase it, translate it and typeset the translation back into the art. Nearly all of them run as desktop applications or behind a server, so the reader has to save a page, open another program and come back. Hinami et al. [3] built a complete research system around the same pipeline and showed that translation quality depends on the image context as well as the text.

We wanted the same pipeline with no step between reading and translating. That meant a browser extension that works on the page the reader is already viewing. It had to keep the text on the device by default and send nothing to a server unless the reader chose a cloud translator. Every stage therefore runs client-side: ONNX models on onnxruntime-web, OpenCV.js in a sandboxed page, and a quantised language model on WebGPU through WebLLM [25].

1.2 Why detection has to be judged at an operating point

The detector decides what the rest of the pipeline touches. Each box it emits is cropped for OCR, erased by inpainting, translated and overwritten with English. A missed sound effect stays untranslated, which the reader barely notices. A false box on a face, a heart or a speed line erases part of the drawing and prints an invented translation on it, and the reader notices that at once. So the two error types cost very different amounts. The extension first shipped with a YOLO26n model fine-tuned for manga bubbles [48]. We call it the baseline throughout, because every candidate in this paper is compared against it; TextSeg (Section 6) later replaced it in the extension. On the original 525-region reference, the baseline produced 3 spurious boxes out of 346 detections (0.9%) at its default threshold. We take that rate as the pipeline's tolerance.

Detection papers for comics usually report average precision, F-measure or pixel-level scores over a fixed test set [4, 6, 53], and model comparisons are usually made at one threshold. That is the right summary when every error costs the same. It is the wrong one here, and our own work shows why. Our first fine-tune raised sound-effect recall at threshold 0.25 from 0.155 to 0.465. It also raised the spurious rate from 0.9% to 7.8%. At a matched spurious rate of 0.9% the fine-tune lost to the baseline on every slice. Most of the gain had come from emitting 462 boxes instead of 346.

Two more problems sit under any such comparison. First, a reference set whose regions were proposed by particular detectors favours those detectors. Information retrieval has known this about pooled judgements for decades [27, 28]. Real text found only by a new model counts as a false positive until someone adds it to the reference. Second, at the small data scale of a hobby project, training noise is large. Two runs of an identical recipe that differed only in seed disagreed as much as two different recipes did [29, 44].

1.3 Thesis and approach

Our thesis is that for a comic-translation pipeline that edits what it detects, detectors should be compared at matched false-positive ceilings, against a reference corrected for pooling bias, with page-level and seed-level uncertainty reported. Judged this way, several conclusions that look settled at a fixed threshold turn out to be wrong, and the changes that actually help become visible.

We support the thesis with a case study that ran from August 26 to September 26, 2026. It produced the extension, a 57-page hand-reviewed detection reference, a training set of about 3,000 pages, and fifteen detector training runs, each scored the same way. The method is empirical throughout. Every design choice described below was fixed by a measurement on real pages, and several of those measurements overturned the hypothesis they were meant to confirm.

1.4 Contributions

A fully client-side comic translation system for Manifest V3 browsers (Section 3). It runs five stages in one shared offscreen document. The extension-origin document is exempt from CORS for image fetches, and credential-free fetching closes the confused-deputy risk that exemption opens. OpenCV.js runs in a sandboxed page because its runtime needs eval-class code generation.
An evaluation protocol designed for destructive comic-translation pipelines (Section 4). It combines coverage-based recall, matched-spurious-ceiling sweeps, hand correction of a pooled reference, and page-bootstrap and multi-seed reporting.
An empirical account of adapting a detector with noisy automatic labels (Section 5). Six YOLO runs isolate four candidate causes of a precision gap: the labelling rule, the starting checkpoint, hard negatives and training-set expansion. A fifth effect, seed variance, turned out to be as large as the differences between recipes.
TextSeg (Section 6): a 12.8 MB text-probability-map detector trained with an ignore mask. It needs no Ultralytics code and is the first detector in this project to beat the baseline at the tight ceiling on every seed. We also describe how its map is reused to erase lettering without damaging balloon outlines.
Measured decisions for the downstream stages (Section 7): script-specific OCR, column chunking for Japanese, balloon fill, a caption plate whose ink never falls below 4.5:1 contrast, and a single batched request per page for free-tier cloud translation.

1.5 Roadmap

Section 2 places the work in the literature on comic understanding, scene-text detection and evaluation methodology. Section 3 describes the system. Section 4 defines the evaluation protocol, which the rest of the paper depends on. Section 5 follows the YOLO fine-tuning lineage and its diagnoses. Section 6 presents TextSeg, three training recipes and the analysis of what limits the tight ceiling. Section 7 covers OCR, erasing, rendering and translation. Section 8 discusses lessons, the development process, limitations and licensing. Section 9 concludes.

2. Related Work

2.1 Comic and manga understanding

Manga109 [1, 2] is the standard annotated manga corpus. Its bounding boxes for frames, faces, bodies and text supported early detection work [53]. A 2026 revision reports missing text regions, dialogue overlapping onomatopoeia, and under-segmented balloons in the original annotations [8]. These are the same defects we found in our own reference (Section 4.3). Magi [4] detects panels, text and characters jointly and orders dialogue for transcription. COO [5] targets comic onomatopoeia, which can be curved, truncated or split into parts, and treats their recognition and linking as open problems. Vivoli et al. [7] survey more than 300 comics papers and note that sound effects and text drawn on the art remain under-served. Del Gobbo and Herrera [6] argue that balloon detection alone cannot find manga text, since lettering often sits outside balloons, and they binarise text at the pixel level instead. Aramaki et al. [52] combined connected-component and region classifiers for manga text.

Hinami et al. [3] built the reference design for fully automatic manga translation. It has multimodal context-aware translation, a parallel corpus mined from published translations, and an end-to-end system with a server. Open-source tools [49, 50] use the same stage structure and depend on server-class models such as LaMa [24] for inpainting. Our work is complementary. We keep the stage structure but constrain the deployment to a browser, and we focus on how the detector should be evaluated under that pipeline's error costs.

2.2 Text detection architectures

Box regressors such as YOLO [16] and its NMS-free successors [17] suit browsers because their export is a fixed-shape tensor. The YOLO26 family used here is end-to-end, so no NMS has to be written in JavaScript. Transformer detectors such as RT-DETR [18, 19] are more accurate on comics. The model by ogkalu [47], trained on about 11,000 comic pages, reached sound-effect recall of 0.796 on our reference. At 168 MB and roughly 500 ms per page, though, we could use it only as a labelling teacher.

Segmentation-based scene-text detectors predict a per-pixel map and group it into regions afterwards. Examples include EAST [39], CRAFT [40] and DBNet [9]. DBNet shrinks each label polygon by an offset $D = A(1-r^2)/L$ and grows detected components back by the same offset, so touching instances stay apart. TextSeg adopts this label geometry and the combination of OHEM binary cross-entropy [13] and dice loss [14]. It sits on a MobileNetV3 backbone [11] with an FPN [12].

2.3 Recognition, inpainting and translation

manga-ocr [46] is a ViT encoder [22] with a two-layer BERT decoder, in the TrOCR style [21], trained for Japanese manga. PP-OCR [20] combines a DB text-line detector with CTC recognisers [35] for Chinese and Korean. Telea's fast-marching inpainting [23] is classical and cheap. LaMa [24] handles texture better but is too heavy for our target hardware. In-browser inference has moved from JavaScript tensor libraries such as TensorFlow.js [38] to WebAssembly and WebGPU runtimes. For on-device translation we use Gemma 2 [26] through WebLLM [25], which runs compiled WebGPU kernels at up to 80% of native speed.

2.4 Evaluation methodology

We borrow three ideas from outside computer vision. First, pooling bias. Test collections judged only on items that participating systems retrieved favour those systems [27, 28], and a detection reference proposed by particular detectors has the same flaw. Second, variance. Bouthillier et al. [29] show that data sampling, initialisation and hyperparameters shift benchmark results enough to reorder methods, and Picard [44] finds that the seed alone moves results by amounts often reported as improvements. Third, operating points. Comparing classifiers at a fixed false-positive rate, rather than at a fixed threshold, is standard in ROC analysis [36]. We apply it with the false-positive rate set by the pipeline's cost. For resampling we use the percentile bootstrap [30] over pages, the unit at which the reference was sampled.

Hard-negative mining has a long history in detection [13, 15]. Learning from partial or noisy labels is surveyed by Song et al. [34], and pseudo-labelling [33] is the usual way to scale up weak labels. Our automatic labeller is a pseudo-labeller over three independent detectors. Our ignore mask follows the partial-label idea: regions whose labels are too uncertain are left out of the loss rather than taught as background.

2.5 The gap

Four things are missing from this literature. No study we know of evaluates comic text detectors at the operating point imposed by a pipeline that erases and overwrites detections. None measures how references pooled from detector proposals bias comparisons between comic detectors. None reports seed variance for detectors fine-tuned on a few thousand comic pages. None describes a complete translation pipeline running in the browser under Manifest V3. This paper addresses all four with one system and one dataset. The trade-off is breadth: our reference is small, and we report its uncertainty rather than hide it.

3. System: A Five-Stage Pipeline Inside a Manifest V3 Extension

3.1 Where each stage can run

A Manifest V3 extension [51] has four kinds of execution context, and each one rules out some stages. We checked each constraint before building on it.

Content scripts share the host page's storage origin and are bound by its CORS rules. If the pipeline ran here, each site would get its own copy of the 1.4 GB translation model, and images from a CDN without CORS headers could not be read at all.
The service worker has no DOM and no WebGPU, which rules out on-device translation.
Extension pages run under script-src 'self' 'wasm-unsafe-eval'. This is enough for onnxruntime-web, but not for OpenCV.js. Its embind runtime builds every binding with new Function, which failed at initialisation under a simulated MV3 policy for both OpenCV 4.12 and 5.0. A control module loaded under the same guard succeeded, so the failure was OpenCV's and not the test's.
Sandboxed pages may use unsafe-eval but have no chrome.* APIs.

The design follows from these constraints (Figure 1). A content script finds the image and sends its URL. The whole pipeline runs in one offscreen document, created once per browser session, so every tab reuses the loaded models. Telea inpainting runs in a sandboxed child frame and exchanges pixel buffers with the offscreen document by transfer. An earlier version ran the pipeline in a hidden iframe per page. We verified that iframe design first: a Cache API marker written while it was framed by one origin could be read while it was framed by another, so the model cache was shared across sites. We later moved to the offscreen document so that models load once per browser session instead of once per tab.

flowchart TB
    IMG["img element on any site"] --> CS["Content script and floating control"]
    SW["Service worker: context menu, opens settings"] -.->|creates| OFF
    CS -->|image URL and settings| OFF["Offscreen document<br/>extension origin, cross-origin isolated<br/>fetches the image with credentials omitted"]
    OFF --> PIPE["Detect: TextSeg<br/>OCR: manga-ocr or PP-OCR<br/>Erase: balloon fill<br/>Translate: WebLLM or Gemini<br/>Render: Canvas 2D"]
    PIPE <-->|pixel buffers| SBX["Sandboxed page: OpenCV.js Telea"]
    PIPE -->|PNG data URL| CS

Figure 1. Runtime architecture. Only the content script touches the host page. The offscreen document owns every model.

3.2 Getting the pixels

Comic images are almost always served from a CDN on a different origin. Our first design had the content script fetch the image and transfer an ImageBitmap. That failed on real sites in two ways. A bitmap from a non-CORS-clean image cannot be transferred at all (DataCloneError). Some image hosts send no Access-Control-Allow-Origin header at all; two we tested were ||nhentai.net|| and ||e-hentai.org||. For those, no retry mode can help. Extension-origin documents with host permission are exempt from CORS, so the content script now sends only the URL, and the pipeline document fetches the bytes.

This required host_permissions: <all_urls>, which Chrome turns into its strongest install-time warning. It also created a security risk. The pipeline document is reachable from any page, so a credentialed fetch would let a hostile site use the extension to read a logged-in user's private images on another origin. Every fetch therefore uses credentials: 'omit', which limits the extension to publicly readable images.

3.3 Triggering without the page's cooperation

On webtoons.com every image carries oncontextmenu="return false;". manhuagui.com cancels the context menu for the whole document. On either site, a right-click menu entry can never appear. We therefore added a floating control that relies on no event the page can cancel. It ranks on-screen images by how much of the viewport they cover and offers the top one. On the three sites we measured, that image covered 58%, 38% and 32% of the viewport, and the runner-up at most 6%. We rejected a cursor-following design because it depends on the pointer events those same sites suppress. It also cannot see images inside shadow roots: on reddit.com a document-level listener reports the shadow host, never the image.

Two features cover cases detection handles badly. The user can draw a box over missed text, and that region enters the pipeline where a detected box would, so no second code path exists. Tiled webtoon strips are detected and reported, not stitched. One measured strip was 291 tiles of 800×1000 pixels, 285,374 pixels tall, and a tile boundary ran through a speech balloon. Only 2 of the 291 tiles held pixels at any moment, because the viewer lazy-loads. We accepted per-tile translation as sufficient for this release.

3.4 Stage summary

Table 1 lists the released configuration. All weights are pinned by revision and SHA-256 and downloaded at build time. The translation model is the exception: it downloads on first use and is cached in the Cache API.

Table 1. Released configuration (public release v0.0.1).

| Stage | Implementation | Size | Notes | |---|---|---|---| | Detect | TextSeg, MobileNetV3-Large + FPN text map (Section 6) | 12.8 MB | onnxruntime-web WASM, 4 threads | | OCR, Japanese | manga-ocr [46], int8 encoder and decoder, column chunking | 117 MB + 9.4 MB line detector | 10/12 exact on the reference crops | | OCR, Chinese | PP-OCRv6 small det + small rec [20] | 31 MB | CER 0.027 on horizontal text | | OCR, Korean | korean PP-OCRv5 mobile rec | 13.4 MB | shares the Chinese line detector | | Erase | Balloon fill, with Telea [23] in a sandbox as fallback | OpenCV.js 12.7 MB | text drawn on art is labelled instead (Section 7.3) | | Translate | Gemma 2 2B JPN, q4f16 [26] via WebLLM [25], or Gemini Flash-Lite | 1.4 GB (local) | cloud model is opt-in, one request per page | | Render | Canvas 2D typesetting, lobe-aware layout | – | caption plates with guaranteed contrast |

Cross-origin isolation (COOP/COEP headers) gives the offscreen document SharedArrayBuffer and therefore four onnxruntime threads. With them, detection takes about 0.7 s per page and OCR about 0.4 s per region. Single-threaded, detection took 1.9 to 2.7 s, and Japanese OCR took about 1.15 s to encode each bubble plus 1.5 s for every 30 characters decoded. A twelve-bubble page needed roughly 30 s. Telea inpainting takes about 235 ms for a round trip to the sandbox, and layout and drawing take 83 ms together.

4. Evaluation Methodology

4.1 The reference set

We built detect_reference.json from 57 real pages. Thirty-seven came from pages where earlier sessions had logged failures, plus the OCR reference sets. Twenty were sampled from Danbooru by tag across the three source languages, with a minimum short side of 600 pixels and restricted to general and sensitive ratings. Danbooru shows a licence for each image, so it can be checked at fetch time. The file stores post IDs and boxes rather than pixels, so the set can be rebuilt without redistributing comic pages.

The ground truth was first proposed by two finders: the baseline YOLO26n at its 0.05 floor, and PP-OCR's DB text detector. We inverted DB's unclip in closed form so that its boxes bound the ink rather than the training margin. A reviewer then judged the clustered proposals on contact sheets and added missed regions by hand. Of the original 525 regions, 311 are bubbles, 72 captions and 142 sound effects, and 86 have dark backgrounds. After the corrections in Section 4.4 the final reference has 540 regions: 314 bubbles, 74 captions and 152 sound effects. The reviewer added 58 regions by hand that neither finder had found. One page was left out entirely: it carried dozens of repeated sound effects along its edges and could not be labelled exhaustively. A half-labelled page would have flattered every detector.

4.2 Metrics

The pipeline crops each detected box for OCR. A box that is too loose costs a little inpainting, while a box that clips the lettering loses words. The primary measure is therefore coverage rather than IoU. For a ground-truth region $g$ and the set of detections $\mathcal{D}$:

$$ \mathrm{cov}(g) = \frac{\left| g \cap \bigcup_{d \in \mathcal{D}} d \right|}{|g|}, \qquad \text{found}(g) \iff \mathrm{cov}(g) \ge 0.5 . $$

A threshold of 0.5 marks the point where OCR starts returning fragments instead of short readings. A detection is spurious when less than 0.25 of its area lies inside any ground-truth region. The cut is generous on purpose, because detector boxes are looser than the lettering inside them. The spurious rate is

$$ S_m(t) = \frac{#{\text{spurious detections of model } m \text{ at threshold } t}}{#{\text{detections}}} . $$

Recall is reported by slice, because the known gaps show up in slices rather than in the mean. The slices are kind (bubble, sound effect, caption), background, and language. A region counts as dark when at least 25% of its pixels fall below luma 80. We also report IoU>0.5 as the conventional localisation measure [41], together with merge and fragmentation counts.

4.3 Matched-ceiling comparison

Detectors put their confidence on different scales, so a fixed threshold means something different for each model. We sweep thresholds from 0.05 to 0.95 on a 0.01 grid, reusing one inference pass per page. For each spurious ceiling $c$ we then take the threshold that maximises sound-effect recall without exceeding the ceiling:

$$ t^{*}m(c) = \arg\max{t ,:, S_m(t) \le c} R^{\text{sfx}}_m(t), $$

and we compare models on sound-effect, dark-background and overall recall at $t^{*}_m(c)$. The ceiling that decides shipping is $c = 0.9%$, the baseline's own rate at its default threshold. Looser ceilings (1.4% to 7%) show how the curves behave where the pipeline would tolerate more errors. The crossover is the lowest ceiling at which a candidate beats the baseline on both sound-effect and dark-background recall.

Two practical points matter here. First, spurious rate is not monotonic in threshold. Deduplication sees a different set of boxes at each threshold, so a lower threshold can have fewer spurious boxes. Second, the grid has to be fine where the operating points lie. An early grid had a 0.01 step only up to 0.65. Every TextSeg operating point sat above that, and one seed's apparent failure at 0.9% was an artefact of the coarse grid (Section 6.3).

[!IMPORTANT] At a fixed threshold, a detector that emits more boxes looks better on recall. At a matched spurious rate it has to earn its recall with the same false-positive budget. Every shipping decision in this paper uses the matched comparison. Fixed-threshold tables appear only for context.

4.4 Correcting the pooled reference

The reference was proposed by the baseline detector and DB. Anything real that the baseline finds was therefore proposed, reviewed and labelled. Real text that only a new model finds is missing, so it counts against the new model. This is the pooling bias known from information retrieval [27, 28]. We corrected it the only safe way: by hand, one box at a time. The reviewer drew each new box around the ink, not from the candidate detector's box, and never edited the JSON directly. The added regions were:

7 regions from reviewing v4's flagged boxes, 2 from v2's and 1 from v3's (525 to 535 regions);
3 sound effects from TextSeg seed 2's spurious boxes (538);
1 sound effect from seed 0's (539);
1 sound effect from the recipe 2 gate review (540).

Seven of v4's fifteen "spurious" boxes turned out to be unlabelled text. On the corrected reference, the genuine spurious rate at threshold 0.25 (535-region reference) is 0.6% for the baseline, 3.3% for v2 and 1.8% for v4. Once a comparison started, we never changed the reference in the middle of it. Every correction was followed by a re-sweep of all models.[^1]

4.5 Uncertainty

Two sources of noise dominate. Page sampling: the 0.9% gate rests on two or three boxes out of about 350. With about 150 sound-effect regions (147 at the 535-region stage, 152 in the final reference), the 95% interval on sound-effect recall is about ±0.075, which is as large as most differences between models. We therefore report a page bootstrap [30]: 1,000 resamples of the 57 pages, with the threshold re-chosen inside each resample, giving $P(\text{candidate beats the baseline on sfx and dark})$, where beating means strictly higher recall on both. Training seeds: we report every seed separately and compare distributions, never single runs [29, 44].

4.6 Leakage guards

The training, negative-mining and expansion sets are all sampled with the reference's post IDs excluded (53 IDs were refused on the first training draw). The dataset builder and the training notebook both re-assert that no reference page is present. We also tested the guards by planting a reference page into a synthetic dataset directory and confirming that both build scripts refused it. When a review found informative false positives on reference pages, those boxes did not become training negatives. Only their categories informed the next mining round on fresh pages.

5. Adapting an Off-the-Shelf Detector: Six YOLO Runs

5.1 Baseline and the two gaps

At the 0.05 floor, the baseline found 0.974 of bubbles but only 0.415 of sound effects and 0.477 of dark-background regions. At its default threshold of 0.25 those two figures fell to 0.155 and 0.384. Both gaps are ones readers notice: sound effects are left untranslated, and dialogue on night scenes or dark panels is skipped. Both gaps are also missing from a mean score, because bubbles make up most of the reference.

5.2 A training set built without exhaustive labelling

We sampled 2,925 Danbooru pages from outside the reference set, weighted toward the gaps. The weights were 1,170 sound-effect pages, 720 dark, 300 screentone, 540 Korean or Chinese, and 270 plain monochrome pages as a control, so that sound-effect recall could not be bought by forgetting bubbles. Hand-labelling at that scale was not possible, so labels came from three independent finders. The finders were proposers, and none of them was trusted as ground truth.

Two finders were not enough. On the original reference, the baseline and DB together proposed 90% of regions, but only 20% of the regions the reviewer had added by hand. Measured across all sound effects, 43 of 142 were found by neither. That mattered, because an unlabelled region is a negative in YOLO's loss, so a set built from those two finders would have taught the model to suppress the very sound effects we wanted it to find. Adding ogkalu's RT-DETR comic detector [47] as a third finder closed most of the hole. Its class for text outside balloons covers exactly the missing shape.

Clusters of proposals were then kept or dropped by a rule, autolabel.py, fitted against a hand-reviewed tier: 30 pages at first, 42 later, and 103 after the expansion described below. The first rule used finder confidence and box size. It labelled at precision 0.920 and recall 0.882 overall, and 0.766 / 0.735 on sound effects.

5.3 v1: recall solved, precision created

Fine-tuning Kiuyha/Manga-Bubble-YOLO with Ultralytics [45] on this set took 20 epochs on a free Colab T4 (imgsz 1280, batch 8, AdamW, learning rate 0.002, 1.06 h). Two engineering fixes came first. Pages as large as 79 megapixels made data loading take 96 s per iteration until the export capped the long side at 1,600 pixels. Unthrottled fetching triggered HTTP 429 after about 400 pages and silently shrank the dataset until we added a delay and backoff. We also confirmed that the export path changed nothing: re-exporting the base checkpoint reproduced the baseline's scores exactly, so any later change is due to training.

At threshold 0.25, v1 raised sound-effect recall from 0.155 to 0.465 and dark-background recall from 0.384 to 0.558. The spurious rate rose from 0.9% to 7.8%. The threshold sweep in Table 2 shows what that means.

Table 2. Fine-tune v1 against the baseline on the 525-region reference. Each row compares the two models at a similar spurious rate.

| Spurious rate | Baseline (threshold → sfx / dark / all) | v1 (threshold → sfx / dark / all) | |---|---|---| | ≈4.2% | 0.05 → 0.415 / 0.477 / 0.792 | 0.40 → 0.366 / 0.465 / 0.701 | | ≈3.4% | 0.10 → 0.296 / 0.453 / 0.741 | 0.425 → 0.345 / 0.430 / 0.688 | | ≈1.9% | 0.15 → 0.218 / 0.419 / 0.705 | 0.475 → 0.275 / 0.349 / 0.640 | | ≈0.8% | 0.25 → 0.155 / 0.384 / 0.659 | 0.65 → 0.113 / 0.244 / 0.512 |

At equal false-positive cost, v1 loses on overall recall, bubble recall and IoU at every point. It wins on sound effects only in a narrow band between 2% and 3.4%, and by about 0.05 rather than the 0.31 the fixed-threshold view suggested. The largest improvement available without training was simply to lower the baseline's own threshold. The baseline at 0.05, which users could already select, beat v1 at 0.40 on every slice at the same spurious rate. The model had learned the label noise as confident text. Because the noise did not sit at low confidence, no threshold could remove it.

5.4 v2: the label rule was reading the wrong variable

We broke the rule's false labels down by which finders had proposed each cluster, instead of tightening its thresholds. On the reviewed pages the base rates were far apart:

| Finders that proposed the cluster | Real text | |---|---| | Baseline YOLO, with or without others | 326 / 334 = 0.976 | | DB and RT-DETR only | 36 / 52 = 0.692 | | DB only | 10 / 28 = 0.357 | | RT-DETR only | 25 / 418 = 0.060 |

Fourteen of the old rule's 22 false labels came from lone RT-DETR proposals. Inside that group RT-DETR's confidence was inverted: the clusters the old rule kept scored 0.16 to 0.36 and were mostly art, while the real ones it dropped scored 0.05 to 0.14 and were large sound effects. DB proposals, for their part, arrive with no score at all, so the old score cut had silently thrown away every cluster DB found alone. The rebuilt rule looks at corroboration first and size second. Every YOLO-backed cluster is kept, DB-and-RT-DETR clusters need 2,500 px², DB-only clusters need 1,200 px², and anything else needs a lone finder at confidence 0.50 or higher. False labels fell from 22 to 6 while true positives rose from 254 to 256. A page-level bootstrap gave a precision gain of +0.057 (95% CI [+0.027, +0.088]). In split-half cross-validation, where the constants were fitted on 15 pages and scored on the other 15, the gain was +0.048 and positive in 99% of splits.

The reviewed set had included no screentone pages, and those were where the new rule changed the most. Twelve screentone pages were reviewed to cover them. The old rule's precision on them was 0.849, against 0.920 elsewhere, and the new rule raised it to 0.925. Over all 42 reviewed pages, precision went from 0.899 to 0.961 and sound-effect precision from 0.695 to 0.878.

The relabelled set had 2,926 pages and 23,480 regions. We retrained v2 with the same budget as v1 (1.055 h against 1.056 h), so only the labels differed between the two runs. At threshold 0.25 the spurious rate fell from 7.8% to 4.6%, and every recall slice rose. That combination is what a label-quality fix looks like, as opposed to trading recall for precision. At the 0.9% ceiling, however, v2 still lost: it reached sound-effect 0.085 and dark 0.233 at threshold 0.65, against the baseline's 0.155 and 0.384. Its crossover sat near 2.5%.

Fragmentation was a second symptom, not a second problem. The fine-tunes looked more fragmented: 13 fragmented regions against the baseline's 4. The metric counted a region as fragmented whenever two detections each covered more than 5% of it, which also fires when a neighbouring box clips its edge. Only 5 of v2's 13 were genuine splits. When the three models were matched on detection count rather than threshold (346, 346 and 344 boxes), the ordering reversed to 4, 3 and 2. The end-to-end head was intact, since duplicates before deduplication stayed at 0.047 to 0.058 per box, and removing the parent-drop rule changed nothing. One separable effect did appear: annotation granularity. The fine-tunes drew one box per vertical column where the baseline drew one per balloon, and each model reproduced the column-pair rate of its own training labels.

5.5 v3: the starting checkpoint is not the cause

v1 and v2 both started from the manga-tuned checkpoint the extension already shipped. To test whether that starting point was holding the operating point back, and to move toward a model the project owned end to end, we retrained from Ultralytics' generic COCO checkpoint on the same data. Two details show how easily a run's meaning can shift. First, optimizer='auto' would silently have switched from AdamW to SGD at 40 epochs, so we pinned the optimizer. Second, Ultralytics 8.4 changed the export default, so the NMS-free head is exported only when nms=False is passed. Our shape assertion caught the resulting [1, 5, 33600] output before any scoring ran.

v3 needed about 27 epochs to reach the validation fitness v2 reached in 20. It ended almost on top of v2: 5.1% spurious at 0.25, with the same shape of curve. Two very different starting points gave one answer, so the confident false positives come from the data and labels, not from the initial weights.

5.6 v4: hard negatives are the first lever that moves the curve

A later audit split the remaining label errors by page type. Screentone and hatching accounted for 7 of 15 false labels on the reviewed pages, marks backed by YOLO for 4, fragments for 2, and painterly colour pages for 2. We tested the hypothesis that dense, painterly art causes the false positives. Colourfulness [43] did not support it: precision was 0.940 on greyscale pages and 0.972 on colour pages.

So we mined the models' own confusions. We ran the baseline, v2 and v3 over 700 fresh pages, weighted toward textures, effect marks and rendered art, and kept every box at threshold 0.25. That gave 7,795 candidates. The two text finders were used only to order the review. Tier A (no finder agrees) turned out to be 64% negatives, tier B (only RT-DETR agrees) 27%, and tier C (DB agrees) 1%. A and B were reviewed exhaustively and C was sampled. Every kept crop was then checked again at full size, because two crops that looked clean on the contact sheet turned out to hold clipped lettering.

The result was 158 confirmed negative crops on 123 pages. Of these, 94 were effect and costume marks, 63 rendered art, 1 a fragment, and 0 screentone. Two of the diagnosed categories could not be reached this way. A fragment always has lettering inside any honest window around it. Screentone never appeared as a confident false positive on fresh pages. The crops were packed onto grey canvases as background images, making up 5% of the training set, and v2's recipe was otherwise left unchanged.

v4 was the first run to change the shape of the curve. At threshold 0.25 its spurious rate was 3.3% and every recall slice rose over v2. Genuine false positives at 0.25 fell from 15 to 8, and the ones that disappeared were exactly the marks and art the negatives targeted. Seven of v4's remaining fifteen "spurious" boxes were unlabelled text (Section 4.4).

5.7 v5 and v4b: the seed moves results as much as the recipe

A second mining round against v4 on 200 fresh pages found only 51 negatives. Most of what v4 now flagged was real text: in the DB-backed tier, 92 of 95 candidates were. The easy negatives were used up. We therefore expanded the true positives instead. Sixty-one fresh pages were reviewed with v4 as a fourth proposer: 587 regions were kept, 652 candidates rejected and 54 added by hand. The result was v5. v5 lost to v4 at every ceiling, and two cheap explanations failed. All 11 of v5's spurious boxes were genuine errors. The expansion regions that only v4 had proposed improved the least, which is the opposite of what circularity would predict.

We then reran v5's exact recipe with seed 1337, since v4 and v5 had both used seed 0 according to their published args.yaml. This replicate, v4b, settled the question (Table 3).

Table 3. Matched-ceiling recall (sfx / dark) on the 535-region reference. v5 and v4b are the same recipe with different seeds.

| Ceiling | Baseline | v2 | v3 | v4 | v5 | v4b | |---|---|---|---|---|---|---| | 0.9% | 0.150 / 0.384 | 0.136 / 0.279 | 0.265 / 0.384 | 0.265 / 0.372 | 0.184 / 0.326 | 0.327 / 0.442 | | 1.9% | 0.259 / 0.442 | 0.293 / 0.395 | 0.333 / 0.442 | 0.544 / 0.663 | 0.463 / 0.581 | 0.381 / 0.523 | | 2.8% | 0.265 / 0.442 | 0.449 / 0.547 | 0.435 / 0.547 | 0.612 / 0.709 | 0.524 / 0.674 | 0.503 / 0.628 | | Crossover | – | 1.39% | – | 0.96% | 1.32% | 0.81% |

The two runs of the identical recipe disagree by as much as v4 and v5 do: the same-recipe spread in sound-effect recall is 0.143 at 0.9% and 0.082 at 1.9%. The order also changes with the ceiling. v4b is the best of the three at 0.9% and the worst at 1.9% and 2.8%. No single 20-epoch run is evidence about its recipe. We did not promote v4b, because doing so would repeat the mistake the replicate had just exposed. The baseline stayed in the extension, since the pipeline's 0.9% tolerance sits inside this noise band.

gitGraph
    commit id: "baseline yolo26n"
    branch manga-yolo-lineage
    commit id: "v1 noisy labels"
    commit id: "v2 corroboration relabel"
    commit id: "v4 hard negatives"
    commit id: "v5 plus 61 pages"
    commit id: "v4b seed 1337"
    checkout main
    branch coco-start
    commit id: "v3 COCO checkpoint"
    checkout main
    branch textseg
    commit id: "R1 three seeds"
    commit id: "R2 fixed val, EMA"
    commit id: "R3 own negatives"
    checkout main
    merge textseg id: "ship TextSeg"

Figure 2. Detector lineage. Each commit is a trained and evaluated condition, and the three TextSeg commits are three seeds each.

[!NOTE] What the YOLO lineage established. The labels were a cause, and the corroboration rule fixed part of it. The starting checkpoint was not a cause. Hard negatives were the first lever to move the curve. At this data and compute scale, run-to-run variance was as large as the differences between recipes, so we compared distributions rather than single runs from then on.

6. TextSeg: A Text-Probability-Map Detector Trained With an Ignore Mask

6.1 Why write a detector at all

By this point the YOLO lineage had run into four limits. (i) The shipping test could not tell runs apart: at 0.9% on the 535-region reference, the baseline scored 2 false boxes of 347, v4 3 of 349, and v4b 3 of 372, so one box decided the outcome. (ii) 2,884 of the 2,987 training pages (96.6%) carried automatic labels, and the labeller finds only about 70% of sound effects. The rest were being taught as background, and Ultralytics offers no way to leave a region out of the loss. (iii) Every fine-tune scored about 0.61 on IoU>0.5 against 0.88 for the baseline, even as coverage rose, because the training boxes come from finders that draw a line, a text block or a whole balloon for the same lettering. (iv) Ultralytics states that its AGPL-3.0 licence covers models its training code produces, so every weight in the lineage inherited that provenance.

A per-pixel text map deals with (ii) to (iv) together. A column box and a block box paint mostly the same pixels. An ignore mask becomes one multiplication in a loss we write ourselves. The model, the loss and the training loop are this project's own code, on an Apache-2.0 ImageNet backbone.

6.2 Model, labels and loss

Model. A timm MobileNetV3-Large backbone [11] feeds features at strides 4, 8, 16 and 32 into an FPN [12] with 96 channels. Each level is smoothed to 24 channels, upsampled to stride 4, concatenated, and passed to a head that outputs one logit map. The head's bias starts at $-4$ ($\sigma(-4) \approx 0.02$), because nearly every pixel is background. Normalisation is built into the graph, so the ONNX file takes the same 0-to-1 input tensor, [1, 3, 1280, 1280], as the YOLO export did, and outputs [1, 1, 320, 320]. The file is 12.8 MB, and ONNX export drifts from PyTorch by less than $10^{-5}$.

Labels. Each text box is painted shrunk in the DBNet manner [9], with every side moved in by

$$ D = \frac{A,(1 - r^2)}{L}, \qquad r = 0.4, $$

where $A$ is the box area and $L$ its perimeter, so balloons that touch stay separate blobs. Every pixel of a training crop is one of four kinds. It is text if it lies inside a shrunk core. It is background if it lies elsewhere on the page. It is ignored if it lies inside a region the automatic labeller dropped even though DB had proposed it: such clusters are real text 36% to 69% of the time, so neither answer is safe to teach. Lone RT-DETR proposals and thin slivers are real only 6% and 1 in 29 of the time, so they stay background. The fourth kind is the ring between each core and its full box. At first we left the ring out of the loss. The smoke test showed that this let the model paint whole boxes, which the decoder then grew a second time, so the ring is now supervised as background, as DBNet does it. The full set holds 24,067 text regions and 1,848 ignore regions. Hard-negative pages enter whole, with everything except the confirmed windows ignored.

Loss. The loss is binary cross-entropy over masked pixels, keeping at most three hardest negatives per positive [13], plus a masked dice term [14]:

$$ \mathcal{L} = \frac{\sum_{p \in P} \ell_p + \sum_{n \in \mathrm{top}_{3|P|}(N)} \ell_n}{|P| + \min(|N|, 3|P|)} ;+; 1 - \frac{2\sum m,\hat{y},y + 1}{\sum m,\hat{y} + \sum m,y + 1}, $$

with $m$ the loss mask and $|P|$ floored at 1, so that a crop with no text still pushes its background down. Decoding. The map is thresholded at 0.3, and each 4-connected component of at least 3 cells becomes a region, scored by its mean probability. Its box is grown back by the unshrink offset, which is found by solving the same fixed-point equation as training. A TypeScript port of the decoder in the extension gave identical boxes to the Python version on all 57 reference maps (1,136 boxes). The parity check fails if a single constant is changed.

Training. We trained on Kaggle [42] because a Kaggle run continues after the browser closes, so no machine has to stay on during training. The GPU was a T4 ×2, with AdamW at learning rate $10^{-3}$, weight decay $10^{-4}$, 768-pixel crops, batch 8, one warm-up epoch and cosine decay to 2%. Pages are cached once at a 1,536-pixel long side (2,987 pages plus 160 negative pages, 1.2 GB, no failed downloads), so each run reads from the cache instead of re-downloading. A CPU smoke test first checked that the model learns at all: after 90 s of training on 10 pages, pixel F1 on 2 unseen reviewed pages rose from 0.64 to 0.72.

6.3 Three recipes, nine runs

Table 4. TextSeg matched-ceiling recall on the 540-region reference: range over seeds, with the mean. Every row, including the bootstrap row, is computed on the 540-region reference (1,000 page resamples; "beats" means strictly higher on both sound effects and dark backgrounds).

| Ceiling, slice | Baseline | R1: 80 ep, best-val epoch | R2: last-epoch EMA | R2: val-peak EMA | R3: R2 + own negatives | |---|---|---|---|---|---| | 0.9% sfx | 0.145 | 0.289–0.461 (0.390) | 0.289–0.414 (0.340) | 0.270–0.336 (0.307) | 0.375–0.441 (0.408) | | 0.9% dark | 0.379 | 0.414–0.598 (0.525) | 0.425–0.517 (0.460) | 0.414–0.471 (0.441) | 0.506–0.563 (0.533) | | 1.9% sfx | 0.250 | 0.487–0.553 (0.518) | 0.434–0.553 (0.480) | 0.572–0.592 (0.586) | 0.539–0.592 (0.568) | | 1.9% dark | 0.437 | 0.563–0.678 (0.613) | 0.529–0.598 (0.563) | 0.655–0.678 (0.670) | 0.655–0.667 (0.663) | | 4.3% sfx | 0.395 | 0.539–0.691 (0.629) | 0.605–0.658 (0.627) | 0.658–0.743 (0.700) | 0.717–0.743 (0.730) | | 4.3% dark | 0.471 | 0.632–0.793 (0.686) | 0.655–0.724 (0.693) | 0.713–0.793 (0.751) | 0.782–0.805 (0.793) | | $P$(beats baseline at 0.9%), seeds 0/1/2 | – | 0.65 / 0.97 / 1.00 | – | 0.82 / 0.71 / 0.69 | 0.97 / 0.92 / 0.97 |

Recipe 1 used seeds 0, 1 and 2 for 80 epochs each, at about 82 s per epoch and 1 h 53 min to 2 h per run, and exported the epoch with the best validation F1. On the 535-region reference, as first reported, seeds 1 and 2 cleared the 0.9% gate and seed 0 did not. An independent audit of the raw logs overturned three readings of that result:

The seeds were validated on different pages. The 25 validation pages were drawn with the training seed, so any two seeds shared only 4 to 8 of them. Validation F1 differed by 0.05 before training had even diverged. The apparent ordering in which "val F1 predicts the gate" (0.7085 < 0.7367 < 0.7544) could not be compared across seeds, and with three points a monotone order happens by chance one time in three.
Seed 0's failure was partly a grid artefact. Re-swept at 0.01 steps up to 0.95 on the 535-region reference, seed 0 does reach the 0.9% ceiling (threshold 0.81, 2 of 292 spurious). Its reported crossover of 1.14% could not be reproduced (0.97%).
What set seed 0 apart was unsupervised padding. The loss masked out the grey letterbox, so the model's output there was never trained. A component straddling the page edge survived decoding as a full-height sliver at $x = 0$. These slivers were 46 of seed 0's 83 spurious boxes at 0.25, against 15 for seed 1, 10 for seed 2 and none for the baseline. An unconstrained region is exactly where seeds should diverge.

The same audit tested a hypothesis that had been recorded as fact: that the training boxes were drawn larger than the reference boxes. The gap in size relative to the page is real. The median Japanese bubble box is 10.1 to 10.8 per mille of the page in training against 6.0 in the reference. But it reflects the pages, not the labelling. Boxes drawn by the same baseline detector at the same floor are 10.3 per mille on training pages and 5.9 on reference pages, because the training pages simply have larger balloons (median box side 194 px against 134 px). Most of TextSeg's low IoU instead came from its own decoder (Section 6.5). The ignore mask, for its part, has no clean evidence either way. Recall above the labeller's own sound-effect recall is not diagnostic, since v4 exceeds it too without any mask. An ablation would need many seeds per arm while the seed spread stays near 0.14.

Recipe 2 changed four things at once. It supervised the padding as background, fixed the validation pages across runs, cut training to 40 epochs (the best epochs had been 27 to 41), and kept an exponential moving average of the weights [31] with decay 0.999. All three seeds ran in a single 3 h 13 min Kaggle session. The page-edge slivers disappeared: zero at every threshold, against 46, 15 and 10 in recipe 1. The EMA's validation F1 peaked at epoch 22 or 23 in every seed, and the last-epoch export had already lost recall. Exporting the EMA at its validation peak gave the tightest condition yet from 1.9% upward, with a seed spread of 0.020 in sound-effect recall against 0.066 for recipe 1. At 0.9%, however, nothing could be resolved: bootstrap probabilities were only 0.69 to 0.82.

The gate was the same few boxes in every seed. Across the three seeds, the spurious boxes at the 0.9% and 1.9% operating points came to 15 distinct boxes, and all were reviewed by hand. Thirteen were genuine errors: 8 were art, such as a "100" badge, a doodled face, a chibi and a shirt logo, and 5 were marks, such as scribbles, a ♪ and a running-man icon. One was text not meant for translation, and one was a real sound effect missing from the reference. All six boxes that set the 0.9% ceilings were genuine, and four of them fired in all three seeds. Three of the six sat on the reference set's sketch pages, which carry only one or two regions each. The gate was therefore neither seed noise nor missing labels. It was a handful of glyph-like shapes that every run ranked above real text.

Recipe 3 targeted those shapes on fresh pages, never on the reference pages themselves. A Kaggle CPU notebook ran the recipe 2 models over 500 new pages and excluded 4,282 post IDs already used anywhere. The pages were weighted toward doodles, badges, scribbles, symbol marks and hatching. Punctuation-only bubbles were excluded from the weighting, because the reference counts "?" and "!" as text. Candidates scoring 0.5 or more went onto 17 review sheets (510 crops). The negatives were then checked again at full resolution, and this round kept 200 of them. Retraining with all three negative lists, from 3,263 training records including 301 negative pages, moved the 0.9% point on every seed. Each recipe 3 seed beats every recipe 2 seed on sound-effect and dark-background recall at the gate. The operating threshold fell from 0.74–0.76 to 0.70. The bootstrap probability of beating the baseline rose to 0.97, 0.92 and 0.97 for seeds 0, 1 and 2. The six gate boxes still fire, but at lower scores: the top one fell from 0.85 to 0.72.

[!WARNING] What this does not show. For seeds 0 and 2 the 95% interval of the dark-background improvement at 0.9% touches zero, and for seed 1 it crosses zero. The gate still rests on two or three boxes out of about 390. We do not rank TextSeg against v4 or v4b: they differ in architecture, in labels and in seed variance, and no controlled comparison separates those factors.

6.4 Seed variance, measured directly

Two recipes let us compare within-recipe spread to between-recipe differences. At 0.9%, recipe 1's seeds spanned 0.29 to 0.46 in sound-effect recall. That range is wider than any difference between recipes measured in this project. Fixing the validation split and exporting a smooth EMA narrowed the spread above 1.9% threefold. Only the targeted negatives moved the 0.9% point on every seed. The practical rule we took from this: a recipe gets judged on three seeds and on page-bootstrap intervals, never on one run and never at a fixed threshold. Seed 2 of recipe 1 illustrates why. Its sound-effect recall at 0.25 was exactly 109/147, the same as seed 0's, yet at matched ceilings they were the best and the worst of the three.

6.5 Box extent and the decoder

The score threshold only filters whole components, so box extent is set entirely by the binarisation cut and the unshrink offset. That is why IoU stays flat as the threshold changes. We ran a decoder-only grid over BINARIZE ∈ {0.2, 0.3, 0.4, 0.5} and unshrink scales from 1.25 down to 0.25, on all three recipe 3 seeds, with no retraining. The unshrink scale traded coverage against IoU and nothing else. IoU>0.5 rose from 0.36 to 0.53 and then 0.67 as the scale fell from 1.0 to 0.75 and then 0.5, and every step cost coverage and matched-ceiling recall. The oversized boxes had been compensating for cores that sit poorly on the lettering. Since coverage is the primary metric, the decoder stayed at 0.3 with the full offset. Setting 0.2 with ×0.75 kept coverage (0.909 against 0.906) and added 0.13 IoU, so we noted it for erasing, where box extent matters.

6.6 Erasing with the map, not the box

In the browser TextSeg found more text than YOLO, including ぬいっ, ガラ, スー, ビギッ and handwritten asides, but it erased worse. The erase stage relied on each box sitting inside its balloon so that Telea fills from clean interior, and TextSeg's looser boxes crossed balloon outlines, which Telea smeared inward. The raw map could not replace the box, because it paints a shrunk core per text block rather than individual strokes. Erasing only the map's cells left 13% to 36% of the lettering behind at every threshold and dilation we tried. Instead we grow each component's core by a disk of 0.75 times its own unshrink offset. This rebuilds the box minus its corners, and the corners are where a rectangle crosses a round balloon. Section 7.3 gives the measurements.

TextSeg ships as the extension's only detector at a default threshold of 0.60. This is looser than the 0.9% gate (threshold 0.70 for the deployed seed). We picked it by inspecting real pages in the development harness, where 0.5 to 0.6 added real lettering and almost no false boxes. It also sits near the model's matched 1.9% point on the reference (0.61). The default was not derived from the gate, and calibrating it per language against the matched sweep is listed as open work (Section 8.3).

7. Downstream Stages: Decisions Fixed by Measurement

The detector sets the operating point, and the downstream stages set what each region costs. This section reports the measurements behind each stage's design. We give them briefly, because each one follows the same pattern: a plausible default, then a check on real data that confirmed it or overturned it.

7.1 Recognition: the script chooses the engine

We sent Chinese and Korean text through manga-ocr to see how it would fail. It did not degrade gracefully: the output was unrelated to the input. Its vocabulary lacks 这, 种 and 门, and it contains only 191 of the 11,172 Hangul syllable blocks. Worse, it is confident when wrong: it assigned mean log-probabilities of −0.04 to −0.78 to garbled Chinese, mostly above the −0.5 cut our harness used to flag weak readings. That is why the source language is a user setting and not an automatic guess. A wrong guess produces fluent nonsense and gives no sign that it failed.

Table 5. OCR by script, measured on regions the detector found on real pages.

| Script, set | Engine | Exact | CER | |---|---|---|---| | Japanese, 12 reference crops [46] | manga-ocr, int8/int8 (111 MB) | 10/12 | 0.041 | | Japanese, same, without post-processing | manga-ocr | 6/12 | – | | Japanese, 24 multi-column balloons | whole region | 16/24 | 0.047 | | | chunks of ≤ 18 characters (released setting) | 21/24 | 0.004 | | Chinese, 16 horizontal regions | manga-ocr | 1/16 | 0.420 | | | PP-OCRv6 small det + small rec (31 MB) | 13/16 | 0.027 | | Chinese, 6 vertical columns | PP-OCR, fixed-pitch cells | 0/6 | 0.333 | | | PP-OCR, ink-projection cells (Otsu [32]) | 2/6 | 0.253 | | Korean, 67 regions on 11 pages | manga-ocr | 0/67 | 1.080 | | | PP-OCRv6 Chinese recogniser | 0/67 | 0.937 | | | korean PP-OCRv5 mobile (13.4 MB) | 21/67 | 0.153 |

Several details in this table came from measurement rather than design. Porting manga-ocr's post-processing, which folds half-width to full-width characters, doubled the number of exact readings. For Chinese, the choice of detector mattered more than the size of the recogniser: swapping in the older v5 line detector cost more than dropping from the medium recogniser to the small one. For Korean, two script-specific fixes mattered. Joining lines with a space lifted exact readings from 10 to 21. Line grouping had to compare box centres rather than overlap, because DB's unclip grows every line box by about 22 px, so neighbouring lines always overlap. Vertical Korean never appeared: all 83 Korean regions on 11 pages were horizontal. We therefore report vertical Korean as untested rather than handled. For Japanese, manga-ocr squashes each crop to 224×224 pixels, so a four-column balloon leaves each character only a few pixels. Reading long regions in groups of at most 18 characters, using PP-OCR's line detector to find the columns, fixed most multi-column misreads without changing the 12 standard crops.

7.2 Translation: the fallback that makes it reachable

The on-device model was chosen from WebLLM's 163 prebuilt models on the same 13 real OCR lines. On that set Qwen2.5-1.5B made a tenfold error in an amount of money and left a kanji untranslated in its English output. Gemma 2 2B JPN made neither error, at 1.49 GB against 0.88 GB. WebGPU is not universal, so we added a cloud fallback. We compared four providers on the constraints that decide the choice. CORS turned out not to matter, since the pipeline document is exempt anyway. The deciding factor was the free tier: only Gemini had one, and a fallback that needs a credit card would not help the users who lack WebGPU. DeepL also cannot be prompted, so it cannot follow the instruction to render sound effects as English sound effects.

Table 6. Meaning errors on 23 real OCR lines (12 Japanese, 5 Chinese, 6 Korean), same prompt and clean-up for both models.

| | Japanese | Chinese | Korean | of which SFX | All | ms/line | |---|---|---|---|---|---|---| | gemma-2-2b-jpn-it (local, WebGPU) | 6 | 3 | 4 | 2 of 2 | 12 | 2946* | | gemini-3.5-flash-lite (cloud) | 0 | 1 | 0 | 0 of 2 | 1 | 84 |

*Measured on a laptop GPU that was held at 210 MHz of its 2,100 MHz by a power policy (0.6 TFLOPS against an expected 15 to 18). The latency figure describes that machine, not the model.

The local model changed a place name, dropped a chapter heading, turned a scream and a doorbell into unrelated exclamations, and reversed grammatical person in Korean. These are errors of content, not the tone-and-idiom limitations the project had expected when it started. Korean is the weakest language, as expected for a Japanese-tuned model. The cloud path required one more change to be usable at all. Sending one request per bubble got 7 of 23 lines rejected with HTTP 429 on the free tier. Sending one numbered request per page got none rejected and was 17 times faster per line. It was also slightly better, because the model saw the whole page's dialogue. Three more practical facts about the cloud API: temperature: 0 did not make its output reproducible, a 96-token budget produced empty replies from models that spend tokens on reasoning, and the models-list endpoint was not a reliable sign that a model was available. When a selected target language is not supported by the local model, the extension stops and offers cloud translation. It never switches silently, because that would send the user's text to a third party they had not chosen.

7.3 Erasing: follow the balloon

We measured the erase stage on the reference pages with two quantities. Letters erased is the share of dark pixels (luma below 128) belonging to lettering that stays inside its reference box, and it should be high. Other ink is the number of dark pixels erased outside the lettering per page, which is damage to outlines and art, and it should be low.

Table 7. Erase masks on the 57 reference pages.

| Mask | Letters erased | Other ink erased (px/page) | |---|---|---| | YOLO box, padding 0 | 0.985 | 2,656 | | TextSeg box | 0.963 | 36,894 | | TextSeg shape, grow ×0.75 | 0.967 | 12,403 | | YOLO box + balloon fill | 0.985 | 892 | | TextSeg shape + balloon fill | 0.984 | 9,038 |

Box padding was a case where the obvious choice was wrong. We started with 3 px of padding for anti-aliasing halos. Measured residual ink rose steadily with padding: 7.3% at 0 px, 19.3% at 3 px and 38.4% at 8 px. Padding pushes the ring Telea samples from off the clean balloon interior and onto the dark outline.

Balloon fill handles a case from a benchmark page used since the start of the project: a single line of speech running across two joined balloons. Both detectors return one region spanning the join, and Telea smears the outline between the balloons into an X. The fill samples the ground colour at the region's text core and flood-fills pixels within 40 levels of it on every channel. The components that reach the core, with their holes filled, form the balloon. The letters are the holes, while the outline, including the one at the join, connects to the page outside and is left alone. The clipped region is then painted flat with the balloon's colour. Handing a clipped mask back to Telea failed, because once the mask stops at the outline, the ring Telea fills from is the outline. A guard sends a region back to Telea when more than 5% of its ink is letter-like shapes the fill would leave behind. This guard came from a failure: the first version, which checked for ink lost from the middle of the region, also switched off the fill on the joined balloons it was meant for.

Text drawn on art is labelled, not erased. When a region gets no balloon fill and its surroundings fail the flat-field test below, its lettering is left in place. The translation goes in a small solid label beside it, at most 16 px and three lines. This removed the star-burst smears Telea produced on sound effects over artwork. One case remains open. A sound effect with a white outline over grey art looks like a balloon to the fill, so it gets painted as a white silhouette. We measured five hand-built separators, and none of them split outlined lettering from real balloons. The principled fix is a second output channel on TextSeg that predicts whether text sits inside a balloon, trained from the region kinds the labels already carry (14,528 bubble and 8,515 sound-effect regions in the automatic tier alone).

7.4 Rendering: contrast by construction

Detector boxes around vertical Japanese are tall and narrow: 11 of 13 regions on a test page had aspect ratios between 0.15 and 0.86. In three cases the box was narrower than a single English word of its own translation at a 10 px font, so shrinking alone could never fit the text. The layout may therefore widen a line up to 2.5 times the box width, centre it, and flag the region as overflowed. It never truncates, because the translation is the only copy the reader gets. Growing a box into surrounding white space was rejected after a test in which the growth escaped a balloon into the page gutter, from 32 px to 203 px wide. A white balloon interior and a white page look the same to a brightness test. With balloon fill, text is now set inside the balloon's own largest interior rectangles. A translation is split across the lobes of a joined balloon only when no part would come out smaller than the whole text would in the main lobe.

A caption plate goes behind text that has no balloon. The trigger is the share of the region, plus a 6 px margin, covered by its two dominant tones on the source page. Lettering on any flat field is two-tone by construction, and artwork is not. On 133 regions from 17 pages, a cut at 0.73 caught 5 of the 9 text-on-art regions with no false alarms among 124 bubbles. Looser cuts added false alarms faster than hits (0.89 gave 8 of 9 hits but 47 false alarms). Two other candidates failed. Ring uniformity separated perfectly on the first page and collapsed on held-out pages, flagging 24 of 57 regions with only about 5 correct, because a box that hugs its balloon puts the outline inside the ring. Measuring on the erased image instead of the source raised false alarms from 0 to 5, because Telea had already replaced the evidence with its own guess.

The plate takes the panel's own tone, and the ink is black or white, whichever contrasts more. With relative luminance $L$, the contrast against white ink is $1.05/(L+0.05)$ and against black ink $(L+0.05)/0.05$. The two are equal at

$$ L^{*} = \sqrt{0.0525} - 0.05 \approx 0.1791, \qquad \text{contrast} = \frac{\sqrt{0.0525}}{0.05} \approx 4.58 , $$

so the better of the two choices never falls below the WCAG AA level of 4.5:1 [37] for any tone. Over all 136 regions the worst case is 8.57:1. Semi-transparency was the one gap in this guarantee. Blending with Telea's output underneath dropped the worst case to 3.28:1 for grounds with luma between 89 and 141, so plates in that band are drawn opaque.

8. Discussion

8.1 What generalises

Judge detectors where the pipeline operates. In this study the fixed-threshold comparison was misleading every time it disagreed with the matched one. v1's "+0.31 sound-effect recall" was mostly extra boxes. Seed 0 and seed 2 of recipe 1 looked identical on sound effects at 0.25 but were the worst and best at the gate. For a comic-translation pipeline that edits pixels, our experience suggests publishing the tolerated spurious rate and comparing models at that rate. We expect the same to hold for other pipelines that overwrite their detections, but this study does not test that.

A pooled reference is biased toward the detectors that proposed it. Half of v4's apparent false positives were real text that the baseline had never proposed. The fix is hand review of every candidate's flagged boxes, with the reference re-frozen before each comparison. As models improve, this matters more, because a better model finds more text that the reference does not yet contain. The 2026 revision of Manga109 [8] reports the same kinds of missing region.

Report variance before recipes. At our scale, the seed moved results as much as any recipe change did. Three seeds with a page bootstrap cost roughly three GPU-hours per recipe on a free T4, less than the Colab hours spent on single runs whose conclusions later reversed.

Diagnose labels by source, not by score. Which finders agreed on a region predicted real text far better than any finder's confidence, and one finder's confidence was inverted within its own group. Diagnostics grouped by source should come before any threshold tuning of an automatic labeller.

8.2 How this was built

This was a side project, built in spare time over about a month, not a formal academic study. We still tried to document the measurements carefully enough that the numbers can be checked. LLM coding agents (Claude Code) did a large share of the work: they implemented the extension and the training code, planned and ran experiments, analysed and audited results, and drafted this write-up. The author reviewed the reported measurements, did the hand annotation and review, made the technical and product decisions, and edited the final text. That review was mostly reading and spot-checking, not an independent re-run of every number, so errors of the kind the audits below caught may remain.

Two habits mattered for the results. First, later sessions audited earlier ones against the raw training logs. That audit found the per-seed validation split, the coarse-grid artefact behind seed 0's apparent gate failure, the unsupervised-padding slivers, and the evidence against the box-size hypothesis (Section 6.3). This was separate from an earlier tooling bug found during the v4b run itself: the sweep script's --out option overwrote the saved rows of models not named on the command line, and those rows were restored from version control before any comparison was made. Second, agent-reported numbers were treated as claims until reproduced. Trained models were re-scored locally from the files downloaded back from Hugging Face or Kaggle. Wherever a run had logged its own scores, the local numbers matched them.

8.3 Limitations

The reference has 57 pages, and the tight gate rests on two or three boxes, so we report bootstrap probabilities rather than claim significance. All pages come from one source, Danbooru, whose tag vocabulary shaped both sampling and slicing. The reviewing was done by one person without a second annotator, so labelling conventions (for example, excluding watermarks and printed cards from what needs translation) are one person's. TextSeg's ignore mask is untested by ablation, and its Japanese boxes remain about 1.5 times the reference area even with a tightened decoder. Browser detection differs slightly from the Python evaluation, because Chrome's canvas resize is not PIL's (per-region scores differ by up to 0.16). The translation comparison uses 23 lines and one annotator. On-device translation has not been re-verified since the pipeline moved to the offscreen document. Tiled webtoon strips are translated one tile at a time, not stitched.

8.4 Licensing and data ethics

Weights trained with Ultralytics' code are covered, in that company's stated view, by AGPL-3.0, whatever licence tag a derivative carries. That was one reason for writing TextSeg. The released extension ships only TextSeg (Apache-2.0 backbone), manga-ocr and PP-OCR (Apache-2.0), OpenCV.js (Apache-2.0) and onnxruntime-web (MIT). Gemma is downloaded by the user's browser under the Gemma Terms of Use. The repositories store post IDs and boxes, never pages. Every page is re-fetched from Danbooru, whose per-image licence can be checked at fetch time. We avoided sources such as scanned doujinshi, whose rights status is unclear. The public repository was assembled from scratch, with no shared git history, after a full-history secret scan of the private one.

9. Conclusion

We set out to put a comic translator into the browser and found that its hardest problem was measurement. A pipeline that erases and rewrites what it detects has a false-positive budget. For our baseline detector that budget was 0.9%, and at that budget most apparent improvements disappeared. Matched-ceiling sweeps, a reference corrected for pooling bias, and seed-and-page uncertainty turned a sequence of confusing reversals into a consistent account. Labels mattered and the starting checkpoint did not. Hard negatives were the first lever that moved the curve. Single runs could not rank recipes. With these methods in place, TextSeg, a 12.8 MB detector we wrote ourselves with an ignore-aware loss, beats the baseline at the tight ceiling on every seed, on one 57-page reference. It roughly triples sound-effect recall (0.41 against 0.15) and raises dark-background recall from 0.38 to 0.53.

Three directions follow from what remains open. A balloon-versus-sound-effect channel in TextSeg would let the pipeline fill real balloons and label outlined sound effects, removing the last major erase failure. A larger, independently reviewed reference, extended by pooling every candidate's proposals, would narrow the bootstrap intervals enough to test the ignore mask by ablation. Calibrating the default threshold per language, and moving detection and OCR to WebGPU, would make the matched-ceiling result the configuration users actually run. We think the protocol matters more than any single number in this paper. It was designed for, and tested on, one comic-translation pipeline; we expect it to be useful wherever a detector's errors end up drawn on the page, but showing that would take other datasets and pipelines.

References

[1] K. Aizawa, A. Fujimoto, A. Otsubo, T. Ogawa, Y. Matsui, K. Tsubota, H. Ikuta. "Building a Manga Dataset 'Manga109' with Annotations for Multimedia Applications." IEEE MultiMedia, 27(2), 2020.

[2] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, K. Aizawa. "Sketch-based Manga Retrieval using Manga109 Dataset." Multimedia Tools and Applications, 76, 2017.

[3] R. Hinami, S. Ishiwatari, K. Yasuda, Y. Matsui. "Towards Fully Automated Manga Translation." AAAI, 35(14):12998–13008, 2021. arXiv:2012.14271.

[4] R. Sachdeva, A. Zisserman. "The Manga Whisperer: Automatically Generating Transcriptions for Comics." CVPR, 2024. arXiv:2401.10224.

[5] J. Baek, Y. Matsui, K. Aizawa. "COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts." ECCV, 2022. arXiv:2207.04675.

[6] J. Del Gobbo, R. Matuk Herrera. "Unconstrained Text Detection in Manga: A New Dataset and Baseline." ECCV Workshops, 2020. arXiv:2009.04042.

[7] E. Vivoli et al. "One Missing Piece in Vision and Language: A Survey on Comics Understanding." arXiv:2409.09502, 2024.

[8] "Manga109-v2026: Revisiting Manga109 Annotations for Modern Manga Understanding." ICML Culture × AI Workshop, 2026. arXiv:2605.21182.

[9] M. Liao, Z. Wan, C. Yao, K. Chen, X. Bai. "Real-time Scene Text Detection with Differentiable Binarization." AAAI, 2020.

[10] mayocream. koharu (software). https://github.com/mayocream/koharu

[11] A. Howard et al. "Searching for MobileNetV3." ICCV, 2019.

[12] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie. "Feature Pyramid Networks for Object Detection." CVPR, 2017.

[13] A. Shrivastava, A. Gupta, R. Girshick. "Training Region-based Object Detectors with Online Hard Example Mining." CVPR, 2016.

[14] F. Milletari, N. Navab, S.-A. Ahmadi. "V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation." 3DV, 2016.

[15] P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan. "Object Detection with Discriminatively Trained Part-Based Models." IEEE TPAMI, 32(9), 2010.

[16] J. Redmon, S. Divvala, R. Girshick, A. Farhadi. "You Only Look Once: Unified, Real-Time Object Detection." CVPR, 2016.

[17] A. Wang et al. "YOLOv10: Real-Time End-to-End Object Detection." NeurIPS, 2024.

[18] Y. Zhao et al. "DETRs Beat YOLOs on Real-time Object Detection." CVPR, 2024.

[19] W. Lv, Y. Zhao, Q. Chang, K. Huang, G. Wang, Y. Liu. "RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer." arXiv:2407.17140, 2024.

[20] Y. Du et al. "PP-OCR: A Practical Ultra Lightweight OCR System." arXiv:2009.09941, 2020.

[21] M. Li et al. "TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models." AAAI, 2023.

[22] A. Dosovitskiy et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." ICLR, 2021.

[23] A. Telea. "An Image Inpainting Technique Based on the Fast Marching Method." Journal of Graphics Tools, 9(1), 2004.

[24] R. Suvorov et al. "Resolution-robust Large Mask Inpainting with Fourier Convolutions." WACV, 2022.

[25] C. F. Ruan et al. "WebLLM: A High-Performance In-Browser LLM Inference Engine." arXiv:2412.15803, 2024.

[26] Gemma Team. "Gemma 2: Improving Open Language Models at a Practical Size." arXiv:2408.00118, 2024.

[27] C. Buckley, D. Dimmick, I. Soboroff, E. Voorhees. "Bias and the Limits of Pooling for Large Collections." Information Retrieval, 10:491–508, 2007.

[28] J. Zobel. "How Reliable Are the Results of Large-Scale Information Retrieval Experiments?" SIGIR, 1998.

[29] X. Bouthillier et al. "Accounting for Variance in Machine Learning Benchmarks." MLSys, 2021.

[30] B. Efron. "Bootstrap Methods: Another Look at the Jackknife." Annals of Statistics, 7(1), 1979.

[31] B. T. Polyak, A. B. Juditsky. "Acceleration of Stochastic Approximation by Averaging." SIAM Journal on Control and Optimization, 30(4), 1992.

[32] N. Otsu. "A Threshold Selection Method from Gray-Level Histograms." IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 1979.

[33] D.-H. Lee. "Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks." ICML Workshop on Challenges in Representation Learning, 2013.

[34] H. Song, M. Kim, D. Park, Y. Shin, J.-G. Lee. "Learning from Noisy Labels with Deep Neural Networks: A Survey." IEEE TNNLS, 2023.

[35] A. Graves, S. Fernández, F. Gomez, J. Schmidhuber. "Connectionist Temporal Classification." ICML, 2006.

[36] T. Fawcett. "An Introduction to ROC Analysis." Pattern Recognition Letters, 27(8), 2006.

[37] W3C. "Web Content Accessibility Guidelines (WCAG) 2.1." W3C Recommendation, 2018.

[38] D. Smilkov et al. "TensorFlow.js: Machine Learning for the Web and Beyond." SysML, 2019.

[39] X. Zhou et al. "EAST: An Efficient and Accurate Scene Text Detector." CVPR, 2017.

[40] Y. Baek, B. Lee, D. Han, S. Yun, H. Lee. "Character Region Awareness for Text Detection." CVPR, 2019.

[41] M. Everingham, L. Van Gool, C. Williams, J. Winn, A. Zisserman. "The PASCAL Visual Object Classes (VOC) Challenge." IJCV, 88(2), 2010.

[42] Kaggle. "Notebooks" documentation (background "Save & Run All" execution, GPU quotas). https://www.kaggle.com/docs/notebooks

[43] D. Hasler, S. Süsstrunk. "Measuring Colourfulness in Natural Images." Proc. SPIE Human Vision and Electronic Imaging, 2003.

[44] D. Picard. "torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision." arXiv:2109.08203, 2021.

[45] G. Jocher et al. Ultralytics YOLO (software, AGPL-3.0). https://github.com/ultralytics/ultralytics

[46] kha-white. manga-ocr (software and model, Apache-2.0). https://github.com/kha-white/manga-ocr

[47] ogkalu. comic-text-and-bubble-detector (RT-DETR-v2 model, Apache-2.0). Hugging Face.

[48] Kiuyha. Manga-Bubble-YOLO (YOLO26n model). Hugging Face.

[49] ogkalu. comic-translate (software). https://github.com/ogkalu2/comic-translate

[50] zyddnys. manga-image-translator (software). https://github.com/zyddnys/manga-image-translator

[51] Google. "Manifest V3" and "chrome.offscreen" developer documentation. https://developer.chrome.com/docs/extensions

[52] Y. Aramaki, Y. Matsui, T. Yamasaki, K. Aizawa. "Text Detection in Manga by Combining Connected-Component-Based and Region-Based Classifications." ICIP, 2016.

[53] T. Ogawa, A. Otsubo, R. Narita, Y. Matsui, T. Yamasaki, K. Aizawa. "Object Detection for Comics using Manga109 Annotations." arXiv:1803.08670, 2018.

Appendix A. Artefacts and Reproducibility

:::details[Released artefacts] | Artefact | Location | |---|---| | Extension source, MIT, TextSeg version | vermilion10/comic-translator (GitHub), release v0.0.1 | | TextSeg weights | Hugging Face vermilion10/manga-textseg | | YOLO fine-tunes v1, v2, v4, v5, v4b | Hugging Face vermilion10/manga-bubble-yolo-finetune (tags) | | YOLO v3 (COCO start) | Hugging Face vermilion10/manga-text-yolo26, tag v3 | | Detection reference (57 pages, 540 regions) | ml/eval/detect_reference.json (post IDs and boxes) | | OCR references | ml/eval/{zh,ko,ja_page}_reference.json | | Sweep and scoring | ml/scripts/sweep_detect.py, ml/eval/score_detect.py | | Parity checks (Python ↔ TypeScript) | extension/scripts/check-text-map.mjs, check-balloon.mjs | :::

:::details[Training budgets] | Run | Hardware | Epochs | Wall time | |---|---|---|---| | YOLO v1, v2, v4, v5, v4b | Colab T4 | 20 | 1.06–1.12 h each | | YOLO v3 | Colab T4 | 40 | 2.19 h | | TextSeg recipe 1, per seed | Kaggle T4 ×2 | 80 | 1.88–2.00 h | | TextSeg recipe 2, three seeds | Kaggle T4 ×2 | 40 | 3.22 h total | | TextSeg recipe 3, three seeds | Kaggle T4 ×2 | 40 | 3.04 h total | :::

:::details[Timeline (2026)]

Aug 26–28. Proposal, extension scaffold, then detect, OCR, erase, translate and render wired stage by stage.
Aug 28–31. Page integration, host permissions, options, Chinese and Korean OCR, deduplication, floating control, manual boxes, caption plates, and the 57-page detection reference.
Sep 2–3. Training set, v1, the threshold sweep, the corroboration relabel, v2, the fragmentation diagnosis.
Sep 10–12. Cloud fallback, vertical Chinese OCR, release audit, v3, false-positive taxonomy, hard-negative mining.
Sep 13–22. v4, reference correction, second mining round, expansion (v5), seed replicate (v4b).
Sep 23–26. TextSeg design and three recipes on Kaggle, map-shaped erasing, balloon fill, Japanese chunking, offscreen pipeline, public release. :::

[^1]: The reference grew from 525 to 540 regions as corrections were added, and every comparison was made on the reference as it stood at the time. Each detection table names its reference (Table 2: 525, Table 3: 535, Table 4: 540), and all rows in a table share it. Numbers quoted in the text say which reference they come from when it is not the final 540-region one.

Building My Own Blog Editor with Tauri

Mon, 28 Sep 2026 00:00:00 GMT

Writing a post for this blog used to go like this on my phone: open Termux, cd all the way into the repo, git pull, switch to a text editor, write, switch back to Termux, then git add, git commit, git push. Images were their own adventure: edit them in Image Toolbox, upload them with RSAF, then type the R2 key into the markdown by hand and hope I didn't make a typo.

That's a lot of steps just to write a few paragraphs. So I thought, why not make an app that does only this one job? This post was written in that app.

:::carousel{aspect="16/10" fit="cover"} :::

What it does

Three views: a markdown code editor, a visual (WYSIWYG) editor with a formatting bar, and a live preview that looks exactly like the real site.
Straight to the repo: it lists the posts from GitHub, keeps drafts on the device, and publishing is one commit through the GitHub API. No git and no terminal.
Frontmatter form: title, dates, tags (with suggestions from other posts), language and password protection, so I don't have to hand-edit YAML.
Syntax cheat sheet in a sidebar. Tap an item to insert it at the cursor.
Image uploads to R2 with conversion settings in the style of Image Toolbox: format, lossy or lossless, quality, or a target file size.
A change gutter like VS Code's, marking the lines I've added or changed since the last publish.
Runs on Windows and Android, with a phone layout (bottom navigation bar) and a tablet layout (rail plus side pane).

The stack

| Part | Choice | |---|---| | App shell | Tauri 2 (Rust, with WebView2 on Windows and the system WebView on Android) | | UI | Vue 3, Vite, Tailwind 4, Material Design 3 | | Code editor | CodeMirror 6 | | Visual editor | ProseMirror with prosemirror-markdown | | Preview | the blog's own markdown-it renderer (Shiki, Mermaid, KaTeX) | | Image encoding | jSquash (WebP, AVIF, JPEG, PNG, OxiPNG) in a Web Worker | | Storage | Cloudflare R2, signed on the Rust side with rusty-s3 | | Secrets | Windows Credential Manager, Android Keystore |

I picked Tauri over Electron mostly for size. The Windows installer is about 10 MB and the Android release APK is 23 MB. The same Vue frontend runs on both, and the Rust side handles whatever the web view shouldn't: secrets, signing requests, and a few Android-only things.

flowchart LR
    UI["Vue frontend<br/>(CodeMirror, ProseMirror, preview)"] -->|posts, commits| GH[GitHub API]
    UI -->|invoke| RS[Rust core]
    RS -->|presigned requests| R2[Cloudflare R2]
    RS --> KS["Credential Manager /<br/>Android Keystore"]
    UI -->|encode| W[Image worker]

The preview is the real renderer

A preview that's only close to the site is useless, because the thing I want to catch is exactly where they differ. So the app doesn't have its own renderer. A small script copies the blog's MarkdownRenderer.vue into the app before every dev run and build, so the preview runs the same markdown-it pipeline, with the same plugins and the same CSS, as the live site.

There's a catch: the blog repo is private, and the editor repo is public. So the copied renderer is git-ignored and never lands in the editor repo; it's pulled from my local checkout at build time.

The hard part: a visual editor that doesn't wreck my markdown

This was the part that took the longest. The usual way a WYSIWYG markdown editor works is: parse the markdown into a document, let you edit it, then serialize the whole thing back to markdown. The problem is that the serializer writes markdown its way. *italic* becomes _italic_, list markers change, blank lines move, and my custom ::: blocks come back mangled. Open a post, fix one typo, and the commit diff touches every line.

The fix was to remember where every block came from. When a post is loaded, each top-level block in the ProseMirror document is mapped back to the exact slice of the source it was parsed from. When saving, a block that hasn't been touched is written back byte for byte from the original. Only the blocks I actually edited go through the serializer.

To make sure it holds, I ran all my existing posts through an open-and-save round trip with no edits. All 13 came back byte-identical, across 1147 blocks.

The blog's custom blocks (Mermaid diagrams, GitHub cards, carousels, embeds) show up in the visual editor as cards that render the real thing, with an edit button that opens the raw source.

Making the diagrams fast

The first version froze. Opening the visual view on the Mermaid examples post blocked the UI for about 15 seconds, because every diagram was rendered up front, and the preview re-rendered every diagram on every keystroke. Two changes fixed it:

a Mermaid cache keyed by the diagram source, so a diagram that didn't change is never rendered again, and
lazy cards that render only when they scroll into view, with a skeleton in their place until then.

Opening the visual view went from 15.6 s blocked to 0.6 s, and a preview re-render from 4.9 s to 0.14 s.

Images: R2 without the storage manager

Uploading is drag and drop (or a file picker on Android). The image dialog lets me pick the format, lossy or lossless, quality, max width, or a target size. The encoding runs in a Web Worker with jSquash, so the UI doesn't stall. For a target size it does a binary search over the quality: encode, check the size, go up or down. When I set 100 KB, the file in the bucket came out at 97 KB.

The R2 secret never touches the web view. The frontend only asks the Rust side to "put these bytes at this key", and Rust signs the request with rusty-s3 and sends it with reqwest. The key follows the same layout my old posts already use, posts/<year>/<slug>/, and uploads never overwrite. If cover.webp already exists, the new one becomes cover.v2.webp.

:::note On Android, Tauri can't hand raw bytes from the web view to Rust, so there the image goes over as base64 inside JSON. Desktop keeps the raw path. I only found this when the first upload on the tablet failed with "expected file as raw body". :::

The change gutter

I wanted the little colored bars you get in VS Code: green for added lines, blue for changed, a red marker where lines were deleted. CodeMirror has a merge extension, but its chunks are character-based and too coarse, so one small edit could paint half a paragraph. I replaced it with a plain line-level Myers diff against the published version, and fuzz-tested it with 3000 random edits.

Keeping the token safe

Signing in takes a fine-grained GitHub token limited to the blog repo with Contents set to read and write. Nothing more.

On Windows the token and the R2 keys go into the Windows Credential Manager through the keyring crate.
On Android there's a small Kotlin plugin that encrypts each value with AES-256-GCM, under a key that lives in the Android Keystore. Only the ciphertext is stored in the app's private storage.

The R2 secret is also marked Rust-only: the frontend can save it, but can never read it back.

Android was its own adventure

The desktop app worked early. Android took a few more rounds:

Edge to edge. The first build drew under the status bar and the gesture bar. The WebView does report their size as safe-area insets, so every screen now pads itself with env(safe-area-inset-*), and the status bar icons switch between light and dark with the theme.
The keyboard. Newer Android no longer resizes edge-to-edge apps when the keyboard opens, so it covered the editor. The app now sizes itself to visualViewport, hides the bottom bar while typing, and scrolls the cursor back into view.
The back gesture closes the open dialog first, then goes back to the post list, and only then leaves the app.
A red herring. At one point R2 and even GitHub looked unreachable. It was Android's background network restriction, which kicked in because the app wasn't in the foreground while I was testing it over USB.

The UI

The UI follows Material Design 3. The whole palette is generated from one seed color with material-color-utilities: orange by default, changeable in the settings, or taken from the wallpaper on Android 12 and up. It's dark by default with a light theme toggle. The layout changes with the width: bottom navigation on phones, a rail with a single pane on small tablets, and a rail plus a side pane (posts, details or cheat sheet) next to the editor on big screens like the Legion Y700.

Where it's at

It works, I'm using it, and this post went through it from the first line to the publish button. There are ideas left, like slash commands in the visual editor and scroll sync for the split view, but for now I'm parking it here.

The code is on GitHub (the editor only; the blog itself stays private):

::github{repo="vermilion10/vermilion10-blog-editor"}

Reverse Engineering the Shiny Colors Asset Pipeline

Sun, 27 Sep 2026 00:00:00 GMT

Reverse Engineering the Shiny Colors Asset Pipeline

This document records the reverse-engineering work carried out while developing shinymas-dl, a .NET downloader and extractor for THE IDOLM@STER SHINY COLORS. It is an internal research record rather than a usage guide. Values that identify or authenticate an account are intentionally omitted.

The investigation began with a simple question: given that the game runs entirely in the browser, how much of its asset pipeline can be reconstructed from the client itself? The answer was considerably more than expected. The browser client exposed a complete asset catalogue, the naming and hashing rules for ordinary CDN objects, the local format used for text resources, and, after authenticated API work, the metadata needed to resolve most of the card assets that initially appeared to be unavailable.

The final result was a C# implementation with separate catalogue, download, naming, extraction, and API layers. The full album crawl recorded 1,440 card hashes for 1,461 card IDs present in asset-map v442, with no unowned card among the covered characters left without a hash.

::github{repo="vermilion10/shinymas-dl"}

1. Scope and initial reconnaissance

The starting point was an existing family of game asset tools: kc-dl, exilium-dl, and pgrasset. The goal for shinymas-dl was not to reproduce the game client as a whole, but to establish a reliable archive pipeline with three properties:

the upstream catalogue should be recoverable without a game installation;
the raw mirror should be resumable and independent from the final extraction layout; and
naming and organization should come from data shipped by the game wherever possible.

The first pass therefore avoided writing application code. The site, client bundle, CDN behaviour, catalogue format, and resource transformations were investigated first.

The game entry page was reachable without a geographic restriction. The HTML referenced a small env.js followed by the main application bundle. The client identified the API root, asset root, game identifier, and a flag indicating that its resource-processing layer was enabled. The main bundle was approximately 3.7 MB, and later inspection identified 1,109 lazy-loaded chunks.

The useful distinction at this stage was between information that could be obtained from the public client and information that depended on an authenticated game session. The former became the basis of the downloader. The latter was investigated separately and only integrated once its behaviour had been demonstrated with a test account.

2. Reconstructing the asset catalogue

The most important early discovery was that the client did not hard-code the complete list of resources. Instead, it loaded an asset-map manifest and a set of versioned chunks.

The live manifest used 129 chunks. On the build investigated in this session, the result was:

| Category | Files | |---|---:| | sounds/ | 273,121 | | images/ | 76,921 | | spine/ | 17,163 | | json/ | 10,804 | | movies/ | 3,352 | | ae/ | 2,524 | | particles/ | 2,213 | | fonts/ | 4 | | Total | 386,102 |

The manifest reported 35.19 GiB in total.

Each chunk maps a logical asset path to a version value. Those paths are the names used by the application before the CDN-specific filename transformation is applied. That distinction mattered because the visible game resource path and the actual URL requested from the CDN are not the same thing.

The refresh implementation keeps the manifest and chunks in data/. A subsequent refresh compares chunk versions and downloads only chunks whose version changed. The manifest is written last, so an interrupted update does not leave a new manifest pointing at a partially updated chunk set.

The first successful run reproduced all 386,102 entries locally. A second refresh against the unchanged manifest performed no unnecessary chunk downloads.

3. CDN naming and resource transformations

3.1 Filename hashing

The browser does not request a resource by its logical path. The client-side hashing function takes the filename stem, its first and last characters, and the full path under /assets/, then hashes the result with SHA-256.

Conceptually, for an asset path p with filename stem s, the input is:

first(s) + last(s) + "/assets/" + p

The digest is represented as lowercase hexadecimal. Media files keep their .mp3, .mp4, or .m4a extension on the CDN; the other resource classes use the bare hexadecimal name.

This was not inferred from a single sample. The same rule reproduced hashes for images, Spine files, JSON, animation data, audio and video. A representative ordinary path is:

images/bg/001.jpg

which becomes a 64-character hexadecimal object name derived from the rule above.

The asset map therefore contains enough information to construct ordinary CDN URLs without consulting any external database.

3.2 Text resources

The next obstacle was that not all resources were served as their final plaintext representation. JSON and Spine .atlas resources were transformed before being written to the network.

Static inspection of the client revealed picosha2, zlib, an embedded string used as a key, and an inflate path. Several hypotheses were tested against known resource files. The useful signal was not a printable-text score but the compressed data itself: once the XOR transformation was correct, the resulting stream had a valid gzip/deflate structure and decompressed to the same plaintext used by the game.

The implemented transformation is a repeating XOR followed by raw-deflate decompression. The embedded ASCII key is 54 bytes long. After XOR, the resource contains the gzip header and a deflate body; the client-side format omits the normal gzip trailer, so the C# port inflates the raw deflate portion directly.

The important implementation decision was to keep this transformation in the raw-to-plain extraction boundary. The raw mirror remains a faithful copy of the game resources, while extract is responsible for producing readable JSON, scripts and other derived outputs.

For a concrete validation, a downloaded json/produce_events/...json resource was decrypted and parsed successfully, and a large Spine .json resource expanded to readable JSON. Images, audio, video and fonts did not require this transformation.

4. The CDN's most misleading property

Once the ordinary path transformation worked, the next problem appeared during bulk testing: a missing resource did not necessarily produce an HTTP 404.

Unknown paths were served as the game's single-page application shell with status 200 and an HTML content type. A downloader that considered only the HTTP status would therefore save copies of index.html under resource filenames.

The downloader treats an HTML response as the CDN's SPA fallback. Missing entries are recorded in the download index so that a normal resumed run does not repeatedly probe the same unavailable object. A --retry-missing option removes those records when a later investigation shows that the object may have become available.

The distinction is useful beyond this project. The server's HTTP status was technically successful while the requested resource was absent. Content validation was therefore part of the downloader's correctness, not merely an error-handling detail.

5. Building the first downloader

The first implementation mirrored the asset map directly into output/raw/. The logical path was preserved, which made the raw tree independent from any future naming convention.

The download index records the upstream version and local size for each completed resource. A file is skipped when the version is unchanged and its on-disk size still matches the recorded value. A version change forces a refetch.

Index writes are performed atomically. The implementation serializes a sorted snapshot into a temporary file and then moves it into place. Progress is stored periodically rather than only at the end, because a full mirror is too large for an interrupted run to be treated as a single transaction.

Downloads use bounded parallelism through Parallel.ForEachAsync. Large files are written to a temporary .part file and renamed only after the transfer completes. Network failures and transient request failures are retried with bounded exponential backoff.

One practical test downloaded the complete story-script category: 10,804 files totaling about 21 MiB, using a concurrency of 16. The run took roughly four minutes. The limiting factor at that size was request latency rather than aggregate bandwidth.

The resulting CLI was split into five operations:

| Command | Function | |---|---| | refresh | Fetch and update the asset map | | list | Search the cached map and produce statistics | | download | Mirror resources into the raw tree | | names | Build character names from story data | | extract | Decrypt and organize the mirror offline |

This separation proved useful later when the resource-availability assumptions changed. The downloader did not need to be rewritten simply because the extraction rules grew more complicated.

6. Naming the data without an external database

6.1 The initial problem

The catalogue identifies characters and cards primarily by numeric IDs. A raw mirror such as

images/content/characters/icon_circle/001.png
images/content/idols/card/1040010010.jpg

is machine-readable but not particularly useful as an archive.

The project therefore set a constraint: use the game's own data to build names instead of depending on a community-maintained character table.

6.2 Story scripts as a name source

The story resources were particularly useful because their records contain information about the character visible in a scene, a romanized character label, and the Japanese speaker name.

After the text resources were decrypted, the implementation scanned the mirrored scripts and built a name index from repeated observations. The resulting cache contains 33 characters: all 28 idols, Hazuki, and four collaboration characters.

The naming process is intentionally statistical rather than based on a single line. A scene may identify one character while another character is speaking, so the visible character label and speaker name are collected as separate observations and combined by frequency.

The complete script collection was sufficient to build the table locally in about 1.2 seconds. No external character metadata was needed.

6.3 Card ID structure

The card IDs themselves also contained useful structure. A representative produce card ID can be separated as:

1 | 04 | 001 | 0010

where the fields represent card type, rarity, character ID, and sequence within that character's card set. Support cards use the corresponding support-card type.

This meant that most card paths could be organized directly from their own IDs. For example, character 001 can be resolved to the mano name through names.json, and a card beginning 104001... can be routed without a separate card database.

The structure also explained several apparently unusual groups beginning with 193, 194, 293, and 294. At first those prefixes did not match the normal rarity constants found in the client. They were later shown to be present in the album data as well, so they were not malformed IDs.

7. Extraction layout

The raw mirror deliberately preserves upstream naming. extract is the layer that turns it into an archive organized for human use.

The principal routes are:

| Source | Extracted form | |---|---| | Character image | characters/<id>_<name>/... | | Idol card | idols/<id>_<name>/<cardId>/... | | Support card | support_idols/<id>_<name>/<cardId>/... | | Story JSON | scenarios/<type>/<id>/script.json | | Story voice | scenarios/<type>/<id>/voice/<file>.m4a | | Everything else | other/<original game path> |

A readable script.txt is generated alongside story JSON. The text representation includes the speaker, line, choices where applicable, and a matching voice path when one exists.

The extractor also handles Spine directories and the game's other structured resource types. Text assets are decrypted during extraction, so the raw mirror remains suitable for later reprocessing if the archive layout changes.

8. The first coverage estimate was wrong

At this point the downloader appeared mostly complete, but a random reachability test suggested that only about ten percent of resources were missing. That estimate was misleading because the sample was dominated by resource classes that remained publicly reachable, especially voice and Spine assets.

A full audit of card-related paths gave a different result:

| Resource group | Reachable without additional metadata | |---|---:| | Produce card images | 68 / 546 | | Support card images | 76 / 915 | | SSR produce card images | 19 / 348 | | Produce card icons | same per-card reachability as card art | | Whisper voice files | 0 / 42 |

The exact equivalence between idol card art and idol icons was significant. The same cards failed in the same pattern, including the same rarity distribution. That strongly suggested that the problem was attached to a card-level attribute rather than to individual files.

The investigation then shifted from the CDN to the client code that constructs resource paths.

9. Discovering the protected asset convention

Static analysis of the main bundle and its lazy-loaded chunks identified a second URL form used by card resources:

<hash>_<id>.<extension>

The hash was not derived from the individual file path. It belonged to the card record itself.

The client uses the same card hash for several related resources, including:

card icons;
card art;
card thumbnails;
some card movies;
awake and memorial-related artwork;
certain fes-related images.

The code also exposed several hash-bearing fields with slightly different roles:

| Field | Observed use | |---|---| | hash | Main card resource paths | | idolHash | Awake and memorial-related paths | | supportIdolHash | Support-card resources | | centerIdolHash | Fes icon and state-specific resources | | voiceHash | Individual whisper voice files |

Other game systems have their own independent hashes, such as gasha resources and some event-specific content. Those were outside the card-asset scope of this investigation.

The asset map was therefore not actually missing the card objects. It was missing one piece of information needed to construct their current CDN object names.

10. Where the card hashes came from

The character album turned out to be the key data source.

Static analysis found a character album request of the form:

POST characterAlbums/characters/{characterId}

The album entries contain card IDs and their hash values. The client also tracks ownership separately through fields such as hasIdol and hasSupportIdol.

A particularly important observation came from the album UI code. The album constructs an icon for every entry, not only the cards owned by the account. Unowned cards are rendered as grey and are made non-interactive, but they are still part of the list. The card-art screen is only entered for owned cards.

That resolved a confusion from the original network captures. The apparent "availability after a condition" was not evidence that the server omitted every piece of metadata until the condition was met. The more precise description was that the UI withheld the art request behind an ownership condition while the album data already contained the card record and its hash.

The list of characters displayed by the album comes from GET album/top.

11. Understanding the authenticated API transport

The game's API uses a different transport from ordinary CDN resources. The browser serializes the logical request as a raw HTTP request, encrypts that representation, and sends it as a single application/octet-stream POST.

The relevant client module is named request_hash. Static analysis showed an asm.js build and a WebAssembly variant. The surrounding code identifies the following behaviour:

Construct the raw request text, including method, path, headers, and body.
Pass the text through encodeRequest.
Send the resulting bytes to the game API as application/octet-stream.
Receive an encrypted response body as an array buffer.
Decode the response with the x-sessionid returned by the server.

The API client uses a rotating session identifier. Requests are queued and sent serially because the response to one request supplies the session identifier used by the next request.

The browser build investigated in this session used x-version: 410 and an x-em-version value embedded in the client bundle. The API client also recognised retryable statuses 1012 and 1090, and it stops when the response contains x-is-banned.

The request codec was first exercised offline. A request encoded with the game's own module could be decoded back to its original text. This demonstrated that the format itself was understood before any authenticated API experiment was attempted.

11.1 A misleading transport failure

Early attempts from Node produced HTTP 403 responses. The initial assumption was that the environment might be geographically restricted or that the API required a special browser fingerprint.

A controlled test disproved both explanations. The same encoded no-token login was accepted by the game server when sent with curl, producing HTTP 401 and an encrypted error response. .NET's HttpClient, both with its default User-Agent and with a browser-style User-Agent, also reached the server and produced the same 401.

The resulting decoded error was:

{"title":"Unauthorized","status":1010,"instance":"/login"}

The 403 therefore belonged to the earlier transport path rather than to a game-side geographic block. This mattered because it prevented the API implementation from being built around a false region-lock theory.

12. The request codec implementation

One could fully reimplement the game's request codec in native C#, but there was no practical reason to do that for the downloader. The game already supplies the codec module on its public client side, and the client changes whenever the game changes.

The implementation instead resolves the live client at runtime. It reads the current index.html, env.js, and application bundle, extracts the API root, game ID, client version headers, and the current request_hash chunk name, then caches those client files under data/client/.

The codec is executed inside .NET using Jint. The implementation loads the game's own JavaScript module and calls its exported encodeRequest and decodeResponse functions. No copy of the game's codec source is bundled into the repository.

This decision has two advantages. First, the tool remains aligned with the current client bundle. Second, it avoids maintaining a native implementation of a format whose details are internal to the game.

The separate reverse-engineering experiments did, however, reveal a useful property of the request format. An encoded request has an eight-byte header, and the output changes with the current Unix second. Holding the input constant and delaying the call changed the encoded output. Static inspection tied the timestamp source to Date.now() and the native implementation to an SFMT-based generator. That observation was enough to explain why repeated identical requests do not necessarily produce identical ciphertext.

13. Establishing the login chain

The remaining problem was obtaining a valid game session for the authenticated album calls.

The browser's Enza integration first supplies a short-lived game token, which is then used by the game login endpoint. A successful login returns the initial game X-Sessionid; subsequent API calls rotate this value.

The test sequence that finally succeeded was:

| Step | Request | Result | |---|---|---| | 1 | Enza platform token exchange using the browser session | Short-lived game token | | 2 | Game POST /login with the game token | Game login succeeded; initial session established | | 3 | GET gameTop | Session headers returned | | 4 | GET album/top | 28 album characters | | 5 | POST characterAlbums/characters/1 | Complete character album |

No account credential is stored in the project. The session value is supplied to the process environment for the duration of a run and is not written to the data directory, logs, source tree, generated JSON, or git history.

One false start came from using a different Enza token type. The platform treats those credentials differently. The game accepted the correct browser session-derived game token but treated the other token as a guest context. This was eventually resolved by inspecting the actual browser session flow rather than assuming that every Enza token represented the same credential layer.

14. The first authenticated album test

Character 001 was used as the controlled test case before the full crawl.

The album contained 64 cards: 22 produce cards and 42 support cards. The test account owned only a small subset of them. Nevertheless, every one of the 64 album entries carried a non-empty 32-character hash, including the cards the account did not own.

The album also contained 23 costume entries with their own hash fields.

The important end-to-end test used the unowned card 1940010010. The ordinary path continued to return the SPA fallback, while the hashed CDN form returned a real JPEG with a size of 76,235 bytes. This established the entire chain:

asset map card ID
        |
        v
character album
        |
        v
card hash
        |
        v
<hash>_<cardId>.jpg
        |
        v
real CDN image

This was the point at which the original reachability problem stopped being a catalogue problem. The downloader had already known that the card existed. It now had a reproducible mapping from the logical card ID to the current protected CDN object name.

15. Whisper voices and client-side release state

Whisper voices required a separate check because they use voiceHash rather than the main card hash.

The support-card data contains a list of whisper voice records together with their release conditions. The client constructs the voice path from idolId, voice ID, and voiceHash, then evaluates whether the voice is released from the card's evolution stage.

Character 014 provided the decisive test case. The test account owned none of the support cards carrying the relevant whisper set. Six voice entries were nevertheless present, all marked as locked by the client and all carrying voiceHash values.

All six downloaded through their hashed paths. Together they accounted for 64.6 MiB.

The observed model is therefore slightly different from the initial assumption that a locked voice simply did not exist to the client. The client receives both the resource metadata and the release condition, then decides locally which tracks should be exposed in the current UI state.

Six whisper files in the asset map remained unresolved. They correspond to the ...000.m4a track in each set and are not referenced by an album entry in the data examined during this session.

16. Full album crawl

After the controlled tests passed, the same process was applied to all 28 characters returned by album/top.

The crawl completed in 59.4 seconds on the test environment.

| Result | Count | |---|---:| | Album characters processed | 28 | | Card hashes recorded | 1,440 | | Card IDs in asset map v442 | 1,461 | | Card IDs without a hash | 21 | | Unowned cards without a hash among album characters | 0 | | Whisper voices with voiceHash | 36 | | Whisper files in asset map | 42 |

The 21 cards without a recorded hash were concentrated in characters not represented by the album list used by the crawler: the four collaboration characters and Hazuki. The six unresolved whisper files were the ...000 member of each whisper set rather than normal album-referenced voice entries.

The crawl was also made resumable. A completed character is recorded in the local album-hash store, so a second invocation skipped all 28 characters rather than repeating the API calls.

This changed the downloader's semantics from "try every known CDN path" to "build the best-known current CDN path from game metadata, then fall back to the ordinary path when no hash is known."

17. Integrating the album data into `shinymas-dl`

The resulting feature was implemented as an albums command.

The command performs the following operations:

resolve the current client bundle and API transport;
obtain an authenticated game session from the supplied environment credential;
read the album character list;
fetch each character album serially;
store card and costume IDs with their corresponding hashes; and
make those hashes available to the download layer.

The persistent album file contains only resource IDs and hashes. It does not contain authentication state.

The downloader's asset representation was extended with an optional CdnPath. If a card hash is known, the hashed CDN form is attempted first. If no hash is available, the existing ordinary path logic remains unchanged. This preserves the original behaviour for resources outside the album-derived set.

The API-side client also follows the browser's sequential session behaviour. It refreshes the current session ID after each response, applies the observed retry semantics, and treats the ban header as a hard stop rather than trying to continue.

18. Validation and security checks

Because the album feature is the only part of the project that handles an account session, the implementation was checked separately from the functional tests.

The following checks were performed after the completed crawl:

the session value did not appear in git history;
no JWT-looking token strings appeared in git history;
the data/ and output/ trees were not tracked by git;
the card-hash JSON contained only IDs and hashes;
the session value, game token, and per-request session IDs were not printed or persisted by the API client.

The browser session was treated as a credential and was not copied into the research document. The test account should be logged out of Enza after experimentation rather than leaving the session active indefinitely.

19. Bugs and corrections

The investigation also produced several failures unrelated to the asset protection itself. They were important because each one could have produced a plausible but incorrect archive.

19.1 The name-cache regression

The original extract command rebuilt names.json from whatever story scripts happened to be present in the current raw mirror. On a deliberately small test mirror, that reduced the source set to two scripts and replaced the previously complete 33-entry cache with a one-entry table.

The failure was silent because the extraction still completed successfully.

The fix was to merge a newly built name table into the cached table. Newly observed values replace stale entries, while names not represented in the current sample remain in the cache.

This is a general issue with derived caches: a smaller input set is not evidence that previously known data should be deleted.

19.2 Awake images initially routed as generic output

Awake-step images were initially falling under other/ because their resource paths did not match the first set of extraction rules. The underlying card association was later shown by the client to use the card hash, but the extraction routing itself remained separate from the CDN lookup. This is an organizational issue rather than a download failure.

19.3 The progress renderer

The terminal progress renderer used carriage returns to update a single line. When stdout was redirected, those updates became permanent log lines. The fix was to suppress high-frequency redraws when output is redirected.

19.4 The apparent region lock

The 403 responses from the early Node transport caused the API to look geographically restricted. Controlled requests using curl and .NET disproved that assumption. The actual cause was the earlier HTTP path rather than a regional requirement.

This was a useful reminder to isolate transport failures before attributing them to application-level restrictions.

20. Final architecture

The final repository has four logically distinct layers:

flowchart TD
    A[Live game client] --> B[Asset map]
    A --> C[Client bundle and API metadata]
    B --> D[Catalog]
    D --> E[Downloader]
    C --> F[Authenticated album API]
    F --> G[Card hash store]
    G --> E
    E --> H[Raw mirror]
    H --> I[Text decryption and extraction]
    I --> J[Organized archive]
    I --> K[Readable story scripts]

The important boundary is between acquisition and interpretation. The raw mirror remains close to the upstream layout. All naming, grouping, decryption, and presentation decisions happen later.

This means a change in extraction rules does not require a new download, and a change in card-hash resolution does not invalidate the rest of the archive.

21. What remains unresolved

The investigation did not establish a complete source for every hash-bearing resource family in the game. In particular:

21 card IDs in the v442 asset map were not represented in the 28-character album crawl. They belong to four collaboration characters and Hazuki.
Six whisper ...000.m4a files were present in the asset map but had no corresponding album entry in the examined data.
Some specialised hashed systems outside the card and whisper paths were not integrated because they have their own records and were outside the scope of shinymas-dl.
The exact server-side consequences, if any, of calling the album POST repeatedly were not established. The downloader therefore treats the calls as read-oriented but does not assume that the endpoint is formally documented as side-effect free.
The request codec's internal SFMT construction was understood far enough to characterize its time-dependent behaviour, but the finished tool deliberately executes the game's codec instead of maintaining a separate native cryptographic implementation.

These are boundaries of the investigation rather than claims that the corresponding resources are permanently unavailable.

22. Conclusion

The difficult part of shinymas-dl was not writing a parallel downloader. It was identifying the layers between a logical game resource and the object ultimately requested by the browser.

The first layer was public and straightforward once the client was read carefully: the asset map described 386,102 resources, the filename transformation could be reconstructed from the resource-hashing code, and the text format could be reversed through known compressed output.

The second layer was more deceptive. The CDN returned 200 for nonexistent paths, making the initial coverage estimate look much better than the actual card-art coverage. Once card requests were compared with their client-side path builders, the missing information was identifiable as a card-level hash.

The final step was to determine where that metadata entered the browser. The character album exposed hashes for all of its card entries, including entries not owned by the test account. The distinction between metadata availability and UI interaction explained why the network trace appeared to show a condition-gated resource.

The final implementation reflects that understanding. shinymas-dl can maintain an ordinary raw mirror, augment it with authenticated card metadata when available, download the corresponding protected resources, and organize the result offline without coupling the archive layout to the game's current UI.

As of asset-map v442, the completed album crawl recorded 1,440 hashes for 1,461 card IDs and established a complete hash mapping for every unowned card represented by the 28-character album set. The remaining gaps are specific and identifiable rather than an undifferentiated collection of HTTP 404s.

Appendix A. Selected resource-path examples

# Logical resource path
images/content/idols/card/1040010010.jpg

# Ordinary CDN lookup
/assets/<sha256-derived-name>?v=<version>

# Card-hash lookup after album resolution
images/content/idols/card/<card-hash>_1040010010.jpg

For media resources, the hashed CDN object keeps its media extension, for example .mp4 or .m4a.

Appendix B. Relevant client-side fields and modules

The exact webpack module IDs are build-specific and will change when the client is rebuilt. The following names were useful during this investigation:

| Client element | Role | |---|---| | env.js | API root, game ID and environment values | | asset-map loader | Manifest and versioned asset chunks | | resource_hash | Ordinary CDN filename hashing and text resource processing | | request_hash | API request and response transport encoding | | character album module | characterAlbums/characters/{characterId} | | album/top | Character list for the album UI | | idolWhisperVoices | Whisper-voice records and release conditions |

Appendix C. Reproducibility notes

All numerical measurements in this document refer to the 27 September 2026 investigation and, where applicable, asset-map v442. The authenticated tests used a dedicated test account. Authentication material itself is intentionally absent from this document.

The raw session transcript used to prepare this report is archived separately. The report is written from the observed sequence of experiments, corrections, implementation changes, and validation results in that transcript, rather than from assumptions about how the game ought to work.

Reverse Engineering RcloneView Before Trusting It With My S3 Keys

Sun, 06 Sep 2026 00:00:00 GMT

[!NOTE] Everything here is against apps I installed on my own device (Galaxy S24 FE, Android 16), for the purpose of deciding whether to trust them with my own credentials. No servers were touched that weren't mine or the vendor's own public endpoints. The fake AWS keys used in the traffic test are the well-known documentation examples.

In the last post, on cloning apps into Samsung's dual-app profile, I mentioned almost in passing that I'd gone looking for a trustworthy rclone/S3 client for Android and that "closed-source options were an easy no." That was a lazy heuristic and I knew it while writing it. "Closed source" is not the same as "untrustworthy," it just means the cost of finding out is higher. So I decided to actually pay that cost.

The app is RcloneView Mobile, a proprietary rclone GUI that had just landed on Google Play. It looked genuinely good. It also wanted the one thing I'm least willing to hand to a black box: my S3 access key and secret.

The claims on the record

Before touching a disassembler, it's worth knowing what you're testing. Someone had already asked the obvious question on the vendor's own forum. In a thread bluntly titled "Safety, closed source", a user named Stan_Manioc asked:

Why should we feel safe since you are closed source and you could be leaking credentials and private cloud files?

After a week of silence he pushed again, saying he couldn't understand how to safely hand "config password, cloud api info, to a black box." On 4 January 2026, the RcloneView account replied with four specific, falsifiable claims:

"RcloneView itself is only a GUI layer."
"All actual operations, authentication, encryption, data transfer, and file handling, are performed entirely by the rclone binary."
"Credentials and config data are stored locally using the OS's secure storage mechanisms (such as Keychain on macOS or equivalent)."
"We do not transmit your credentials or file data to any external servers, nor do we have visibility into your cloud contents."

Their marketing says the same thing in softer words: the security checklist post describes a "local-first" design where "data flows directly between your machine and your clouds, no intermediary" and "no third-party server touches your data."

That's a good set of claims because every one of them is checkable. And crucially, that forum thread is about the desktop app. The mobile app is new, version 1.0.0, and nobody had checked it at all.

Getting the app apart

The app was already installed, so no sketchy APK mirrors were needed:

$ adb shell pm path com.bdrive.rcloneviewmobile
package:/data/app/~~yOrvTTLcKbYUtCx9JNxnsQ==/...base.apk
package:/data/app/~~.../split_config.arm64_v8a.apk
package:/data/app/~~.../split_config.en.apk

Six splits, pulled with adb pull. The arm64 split is 99 MB, which is the first interesting fact. Nothing about a file browser needs 99 MB of native code unless it's carrying an entire Go program.

$ unzip -l split_config.arm64_v8a.apk | sort -rn
 70058008   lib/arm64-v8a/libgojni.so
 11316480   lib/arm64-v8a/libflutter.so
 10879920   lib/arm64-v8a/libapp.so
  1717000   lib/arm64-v8a/libsqlite3.so

So it's three layers:

| Layer | What it is | |---|---| | classes.dex | Thin Flutter plugin glue, wrapped in PairIP (Google Play's anti-tamper) | | libapp.so | Dart AOT snapshot, all the app logic | | libgojni.so | A real rclone, compiled to a shared library via gomobile |

The dex is nearly worthless for analysis, it's plugin registration and PairIP scaffolding. The interesting parts are the two blobs that decompilers don't touch.

[!TIP] Flutter release builds are AOT-compiled Dart, so jadx gives you almost nothing. But the AOT snapshot retains library and symbol names. A plain strings pass over libapp.so is far more productive than any decompiler.

That turned out to be the whole ballgame:

$ python strings.py libapp.so 6 | grep -oE "package:rcloneview_mobile/[a-z_/]+\.dart" | sort -u | wc -l
246

246 source file paths, giving me the app's entire architecture for free:

api/secure_storage_service.dart
api/drive_credentials_extractor.dart
api/remote_config_injector.dart
api/license_client.dart
api/device_guid.dart
pairing/api/qr_crypto.dart
util/diag_log.dart
...

Where the credentials actually live

Two findings settle claim 1 and claim 2 together.

First, the JNI bridge. The Go layer exports 113 symbols under a package called ndwrap, and the config-related ones are exactly what you'd hope:

ndwrap.SetRemoteConfig
ndwrap.SetRemoteConfigObscured
ndwrap.GetRemoteConfig
ndwrap.inMemoryStorage

That last symbol is the important one. Combined with RemoteConfigInjector and a rclone-inject tag on the Dart side, it says the rclone configuration is injected into the engine in memory. There is no rclone.conf written to disk at all. I confirmed the negative case by searching the device:

$ adb shell "find /sdcard -iname '*rclone*' -o -iname '*bdrive*'"
(nothing)

Second, persistence. The app stores state through flutter_secure_storage under these keys, all recovered from the snapshot:

rcloneview.remotes.ini
rcloneview.license.info
bdrive.auth.token
bdrive.session.key
bdrive.device.guid

And both of that plugin's Android backends are present in the dex: Tink-backed EncryptedSharedPreferences (AES-256-GCM), plus the legacy keystore cipher:

public final Cipher p() {
    return Cipher.getInstance("RSA/ECB/OAEPPadding", "AndroidKeyStoreBCWorkaround");
}

Either path wraps the data with a key held in the AndroidKeyStore, which on an S24 FE is hardware-backed and non-exportable. So claim 3 holds: credentials are ciphertext at rest, unlocked by a key that can't leave the device.

[!WARNING] One caveat worth understanding. SetRemoteConfigObscured maps to rclone's obscure function, and rclone's "obscure" is reversible AES with a hardcoded key. It is obfuscation, not encryption. That's fine here, because the obscured blob is itself inside Keystore-backed storage, but the obscuring layer contributes exactly zero real protection on its own.

What it sends home

Static analysis found only two first-party domains in the entire Dart snapshot:

https://rcloneview.com
https://license.rcloneview.com

Everything else was a cloud provider you'd expect (graph.microsoft.com, api.dropboxapi.com, s3.amazonaws.com). More importantly, I went looking for telemetry SDKs and found none: no Firebase Analytics, no Crashlytics, no AppsFlyer, Adjust, Sentry, Amplitude, or Mixpanel. The firebase-*.properties files in the APK are transitive ML Kit dependencies pulled in by the QR scanner, not analytics.

That's a good static result. But static analysis can only prove what code exists, never what it does at runtime. Claim 4 needed a real test.

Actually watching the traffic

Here's where I hit the wall I expected. The app's network security config declares no user CA trust anchors, so a normal MITM proxy can't decrypt anything, and PairIP exists specifically to make patching the APK unpleasant. I initially wrote this off as unverifiable.

That was wrong, and the fix is that I didn't need decryption. I only needed to know who it talks to, and TLS SNI is sent in the clear. PCAPdroid captures per-app traffic through a local VPN with no root at all:

$ adb shell am start -n com.emanuelef.remote_capture/.activities.CaptureCtrl \
    -e action start -e app_filter com.bdrive.rcloneviewmobile
$ adb shell ip addr show tun0
87: tun0: <POINTOPOINT,UP,LOWER_UP> mtu 10000 ...

With the capture live and filtered to just that one package, I force-stopped RcloneView, cold-started it, let it idle, then added an S3 remote using the canonical fake AWS documentation keys and hit save and verify. It returned "Access denied," which is precisely right, that's a genuine HTTP 403 coming back from AWS.

Every connection the app made, for the entire session:

| Time | Destination | SNI | Sent | Rcvd | |---|---|---|---|---| | 15:04:08 | 52.219.37.6 | s3.ap-southeast-1.amazonaws.com | 2,735 B | 7,230 B | | 15:04:14 | 52.219.37.6 | s3.ap-southeast-1.amazonaws.com | 2,815 B | 7,270 B | | 15:05:43 | 3.5.147.222 | s3.ap-southeast-1.amazonaws.com | 2,775 B | 7,310 B | | 15:05:53 | 3.5.147.222 | s3.ap-southeast-1.amazonaws.com | 2,695 B | 7,230 B |

Four connections. All to AWS S3. Zero contact with license.rcloneview.com or rcloneview.com across a cold start, three minutes of idle UI use, credential entry, and browsing.

The byte counts are worth a glance too: roughly 2.7 KB outbound is about right for a TLS handshake plus a SigV4-signed S3 request. There's no room in there for bulk exfiltration, and nowhere for it to go regardless, since AWS was the only peer.

The trap I nearly fell into

I run AdGuard as a Private DNS resolver, and I very nearly published the above as a clean bill of health without thinking it through. The problem: if AdGuard had been blocking license.rcloneview.com, the app would never have opened a TCP connection to it, and my capture would look identical to a clean one. I'd be reporting "no telemetry" when the real finding was "my DNS filter hid the telemetry."

So I tested the resolver directly rather than trusting the absence:

$ curl -s -H 'accept: application/dns-json' \
    "https://dns.adguard.com/resolve?name=license.rcloneview.com&type=A"
{"Answer":[{"data":"lb-bdweb-230694326.us-east-1.elb.amazonaws.com."},
           {"data":"100.59.159.229"},{"data":"32.194.61.192"}],"Status":0}

AdGuard resolves it, identically to Cloudflare. Nothing was being filtered into invisibility. The silence in the capture was real silence.

[!IMPORTANT] This is the part I'd emphasise to anyone doing this kind of testing. An absence of evidence in a packet capture is only meaningful if you've independently proven that the evidence could have appeared. DNS filtering, VPN routing, and app filters all silently manufacture clean results.

The verdict on RcloneView

All four of the developer's claims held up under testing. Credentials go straight to AWS with no vendor server in the path, they're stored under a hardware-backed key, rclone does the actual work, and the app did not phone home once.

That's a better result than I expected, and I want to state it plainly because my starting position was prejudicial. Still, the findings weren't uniformly good:

allowBackup="true" with no dataExtractionRules and no backup agent. App data is eligible for cloud and device-to-device backup. The practical impact is limited, since Keystore keys never leave the device so restored ciphertext is undecryptable, but for an app holding credentials this should be off.
MANAGE_EXTERNAL_STORAGE, all-files access. Defensible for a file manager, but it's the broadest permission in the manifest.
No ACCESS_LOCAL_NETWORK. On Android 17+ that permission gates local network access, so SMB to a NAS will break on upgrade until they ship a fix.
rclone 1.73.4, a few releases behind.
Developer paths left in the binary: /Users/tsjeong/go, /Users/tsjeong/workspace. Cosmetic, but it means no -trimpath.
Version 1.0.0, and the forum's Android bug thread from July 2026 has a user reporting 250 of 1,300 uploads silently failing, no bulk retry, and no transfer history. Staff acknowledged all of it. The security posture is fine; the maturity isn't there yet.

And the structural point that no single test can fix: today's clean capture says nothing about version 1.1.0. With a closed-source app you're not verifying once, you're re-verifying forever.

The open-source comparison

For contrast I did the same review on RSAF, where "review" means actually reading the code.

::github{repo="chenxiaolong/RSAF"}

Its credential handling is a straightforwardly stronger design. rclone.conf lives in app-private storage and is encrypted with rclone's own native config encryption, not merely obscured. The password for that is a randomly generated 128-byte value, which is itself wrapped by Tink AES-256-GCM under an AndroidKeyStore master key:

private val hardwareWrappedPassword: Password
    get() {
        ...
        passwordStore.password = RandomUtils.generatePassword(128)

Two real layers instead of one. There are also small signs of a careful author throughout, like the password type refusing to render itself into a log line:

data class Password(val value: String) {
    override fun toString(): String = "<password>"
}

The backup handling is the standout, though. Rather than a bare manifest flag, RSAF ships a BackupAgentHelper that refuses to back up at all unless the transport is verifiably encrypted:

if (data.transportFlags and FLAG_CLIENT_SIDE_ENCRYPTION_ENABLED != 0) {
    Log.i(TAG, "Client-side encrypted backup")
} else if (data.transportFlags and FLAG_DEVICE_TO_DEVICE_TRANSFER != 0) {
    Log.i(TAG, "Device-to-device transfer")
} else {
    Log.e(TAG, "Plain-text backup is not allowed")
    return
}

It's also gated behind a preference that's off by default. The README is honest about why: if you do enable it, the config leaves the device in plaintext and you're trusting the backup transport, and the author notes that "some older OEM Android builds have backup systems that lie about end-to-end encryption."

Telemetry: none. The only URLs anywhere in the app source are documentation links in comments.

I verified the shipped 4.12 build matches its claims. It carries rclone 1.75.1, it leaks no developer paths (so -trimpath is working), it embeds an assets/archive.tar snapshot of its own source, and the signature checks out against the digest published in the README:

$ apksigner verify --print-certs RSAF-4.12-arm64-v8a-release.apk
V2 Signer: certificate DN: CN=Andrew Gunnerson, OU=RSAF
V2 Signer: certificate SHA-256 digest:
  b2506499bea1c5a6e658f07be6773fe486999dc124204c6522af7407503ac9f9

Why my build failed

I'd tried to build RSAF from source and couldn't, which is what sent me down the "just sideload something" path in the first place. The answer is unglamorous: upstream only builds on Linux. The CI is runs-on: ubuntu-latest, full stop.

The Windows failure is specific. RSAF puts a shim go.exe first on PATH (rcbridge/gowrapper/go.go) so it can strip absolute paths out of go list -json output for reproducible builds. gomobile's newer tool-directive check doesn't survive that shim, and you get a misleading error:

gomobile bind requires golang.org/x/mobile in the current module,
but it is not in the module dependency graph.

That message sends you straight to editing go.mod, which is the wrong move. The module is fine, as running the query directly proves:

$ go list -m golang.org/x/mobile
golang.org/x/mobile v0.0.0-20260821190718-4776eadac327

Build it in a clean Linux clone and the problem evaporates. Don't copy a Windows working tree across either, that trips Gradle's committed dependency verification metadata.

Side by side

| | RcloneView 1.0.0 | RSAF 4.12 | |---|---|---| | Source | Proprietary, PairIP-hardened | GPL-3.0, readable | | Config at rest | Obscured INI in Keystore storage | Encrypted rclone.conf + 128-byte Keystore-wrapped password | | Backup | allowBackup=true, no rules | Agent refuses plaintext, off by default | | Telemetry | License server exists, never fired in testing | None | | rclone | 1.73.4 | 1.75.1 | | Android 17 ready | Missing ACCESS_LOCAL_NETWORK | Declared | | Build hygiene | Dev paths present | -trimpath, embedded source | | Verifiable | Only by re-testing each release | Signature + reproducible build |

One honest asymmetry: these aren't drop-in equivalents. RSAF is a Storage Access Framework provider, it exposes remotes to other apps rather than giving you a browser UI. RcloneView is a full file manager with photo backup and a rather clever encrypted QR pairing flow for importing remotes from desktop (PBKDF2 plus AES-GCM, chunked across multiple QR codes, PIN-authenticated, entirely offline). If you want the app itself to be the interface, they don't compete.

What I'd actually tell you

I set out to catch a closed-source app doing something wrong and it didn't. The developer's public claims were accurate, at least for the version I tested, and the "closed source is an easy no" instinct I started with was intellectually lazy. Verification beats vibes.

But I'm still putting my keys in RSAF, for a reason that has nothing to do with RcloneView's behaviour: the cost of staying confident is completely different. Auditing RcloneView means repeating this entire exercise on every release. Reading RSAF means reading a diff.

[!CAUTION] None of this replaces bounding your blast radius. Use a dedicated IAM user scoped to a single bucket, never account-wide or root keys, and rotate them. Then the worst case is bounded no matter which app you pick, and no amount of reverse engineering is a substitute for that.

What I could not verify

To be explicit about the limits, since a security writeup without them is marketing:

Payload contents. No CA trust means no decryption. I know who the app talked to, not what it said. The byte counts constrain the possibilities but don't eliminate them.
That the license server is never contacted. LicenseClient and DeviceGuidGenerator exist in the binary. A six-minute window proves they don't fire at startup or during ordinary S3 use, nothing more.
On-disk ciphertext. The device isn't rooted and the app isn't debuggable, so I inferred the storage format from the code rather than reading shared_prefs directly.

Tooling

Everything here used free tools and an unrooted phone:

jadx for the dex and manifest
A 20-line Python strings extractor, since Git Bash ships without strings
adb for pulling splits, permission state and dumpsys flags
PCAPdroid for rootless per-app capture
apksigner from the Android SDK build-tools
DNS-over-HTTPS queries to sanity-check the resolver

The highest-leverage trick by far was realising that Flutter AOT snapshots keep their symbol names. That one grep turned an opaque 10 MB blob into a file listing of the entire application.

Sources: Safety, closed source · RcloneView Android App issues · Encrypt config file? · Cloud Storage Security Checklist · RSAF · Google Play listing

Cloning Any App Into Samsung's Dual App Profile

Sat, 05 Sep 2026 00:00:00 GMT

[!NOTE] Package names, device model, and Android version in this post are real (Galaxy S24 FE, One UI 8.5 on Android 16). I've redacted local network addresses since they aren't relevant to the technique

at first this had nothing to do with dual apps at all. It started with trying to share a couple of Windows folders with my phone over SMB, tightened down to that one device instead of the whole Wi-Fi network. That went fine, but it left me wanting proper cloud storage access from Android instead of ad-hoc file shares, so I went looking for a trustworthy rclone/S3 client. Closed-source options were an easy no, so I picked two open-source ones, RSAF and a maintained fork of Round-Sync, and decided to build both from source rather than sideload someone else's binary.

RSAF never built, its toolchain is pinned to an Android SDK version so new that even the maintainer's own CI has to pin the emulator to a lower API "until the action supports it." Round-Sync did build, after fixing a missing GOOS=android in its dependency resolution and a 16KB page-alignment issue in its native library. None of that is this post. This post is about a side effect I noticed while reinstalling Round-Sync over and over during that debugging: two identical app icons

Two icons, one APK

I'd run adb install -r app-debug.apk, and the launcher would show a duplicate, same name, same icon, one with a small badge overlay. My first guess was Secure App, Samsung's Knox-isolated profile. Nah, it was wrong, the badge didn't look like Secure Folder's lock icon, and a permission error from pm list packages alone doesn't distinguish between Samsung's various profile types anyway. Also, the Secure App only apear in Secure Folder, not on home screen. So I checked properly:

$ adb shell pm list users
Users:
	UserInfo{0:null:4c13} running
	UserInfo{95:DUAL_APP:20001010} running
	UserInfo{150:Secure Folder:61010} running

Three profiles. 150 is Secure Folder, confirmed separately. 95 is named DUAL_APP, and a package check confirmed it:

$ adb shell pm list packages --user 95 | grep felixnuesse
package:de.felixnuesse.extract.debug

the app showed up on both the main profile and this dual-app profile, though I'd never opened Samsung's dual-app settings for it

[!IMPORTANT] Settings > Advanced features > Dual Messenger on this device lists exactly two apps: WhatsApp and Facebook. There is no toggle for anything else. And yet an unrelated file-manager app was sitting in that profile.

Ruling out the obvious explanation

My first guess was that Dual Messenger secretly supports more than it lets on, or maybe some Good Lock module got switched on somewhere along the way. Neither one held up though:

The Dual Messenger app list has exactly those two apps, no hidden "add more" screen.
Apps installed the normal way, tapping a downloaded APK through the system installer, never get cloned. A local test with an unrelated open-source app confirmed this: install by tapping the file, no duplicate appears.

The only apps that ended up duplicated were the ones I had installed via adb install in the same session. That's the actual variable. The dual-app setting had nothing to do with it.

$ adb shell dumpsys user
  UserInfo{95:xxx:20001010} serialNo=95 isPrimary=false parentId=0
    Type: android.os.usertype.profile.CLONE
    Flags: 536875024 (DUALAPP_PROFILE|INITIALIZED|PROFILE)
    Created: +537d20h20m49s210ms ago

android.os.usertype.profile.CLONE is AOSP's generic clone-profile primitive. Samsung didn't invent it. Dual Messenger is one front-end that writes into it, with a curated app allowlist baked into its own UI. The profile itself has no such allowlist, it's a plain Android user, and pm doesn't care which front-end you'd normally use to populate it.

The actual technique

If the profile is generic, installing into it directly should work without going anywhere near Dual Messenger's UI at all. Android's package manager has a command built for exactly this: cloning an app that's already installed into another user, without touching its APK.

$ adb shell pm list packages --user 0 | grep discord
package:com.discord

$ adb shell pm install-existing --user 95 com.discord
Package com.discord installed for user: 95

$ adb shell pm list packages --user 95 | grep discord
package:com.discord

That's the whole trick: no APK re-download, no re-signing. It's the same installed package already verified and running on your main profile, registered against a second user, so there's no risk of ending up with a different build. The new icon appears in the launcher, with its own isolated /data/user/95/ storage: a second, independent login.

flowchart LR
    subgraph Device
        M[Main profile<br/>user 0]
        D[Dual App profile<br/>user 95, CLONE type]
        S[Secure Folder<br/>user 150]
    end
    UI[Dual Messenger UI] -->|writes only<br/>WhatsApp/Facebook| D
    ADB["pm install-existing --user 95 &lt;pkg&gt;"] -->|writes any package| D
    style ADB fill:#22c55e,color:#000
    style UI fill:#6b7280,color:#fff

The Dual Messenger UI and the raw pm command both write into the same profile. One gates it behind an allowlist, the other doesn't gate it at all.

[!TIP] If the app isn't installed on the main profile yet, pm install-existing has nothing to clone. Install it normally first (Play Store or a verified APK), then run the command above.

Then it disappeared

I wrote the first draft of this post, logged into the cloned Discord, and moved on. Some time later the second icon was gone. The app and its data were gone too:

$ adb shell dumpsys package com.discord | grep "User 95"
    User 95: ceDataInode=-1 deDataInode=-1 installed=false ...

ceDataInode=-1 means the profile's copy and its data were deleted. Same for the two other apps I'd cloned. Profile 95 was back to exactly one third-party package: WhatsApp.

Android's log buffer only holds a few hours, so whatever did it had scrolled away. I re-cloned Discord and started watching. Nothing happened for fourteen minutes of normal use. Then I rebooted the phone, and two minutes after boot:

18:13:19.385 E DualAppManagerService: getAllWhitelistedPackages : empty
18:13:19.385 E SemDualAppManager: getAllWhitelistedPackages : null returned. Return default
18:13:19.395 I PackageManager: START INSTALL-EXISTING PACKAGE: userId{95} pkg{com.samsung.android.permissioncontroller}
18:13:19.395 I PackageManager:           Request from pId{12519 / com.samsung.android.da.daagent}
18:13:19.447 I PackageManager: START DELETE PACKAGE: observer{138464497}
18:13:19.447 I PackageManager: pkg{com.discord}, user{95}, caller{1000} flags{0}
18:13:19.451 I ActivityManager: Force stopping com.discord appid=10378 user=95: deletePackageX

That's the whole mechanism in seven lines. com.samsung.android.da.daagent, the Dual Messenger agent, runs a reconciliation pass on every boot:

Ask the system service for the allowlist. It comes back empty, so it falls back to the built-in default (WhatsApp and Facebook).
install-existing its own helper packages into the profile, using the exact same command I'd used, which is a nice confirmation that this is the intended way to populate it.
Delete every package in the profile that isn't on the list. Data included.

The profile is generic, the restriction lives in a separate app, and that separate app is also a janitor that shows up after every reboot. Nothing in the fourteen minutes of runtime triggered it, only the boot did.

sequenceDiagram
    participant Boot
    participant DA as daagent
    participant PM as PackageManager
    participant P95 as Profile 95
    Boot->>DA: start
    DA->>PM: getAllWhitelistedPackages()
    PM-->>DA: empty → default [WhatsApp, Facebook]
    DA->>PM: install-existing helpers → user 95
    DA->>PM: for pkg in profile 95 not in list: deletePackage(pkg, 95)
    PM->>P95: remove com.discord + data

I looked for somewhere to write to that allowlist. dumpsys dual_app prints nothing, there are no settings keys for it, and daagent's data directory is off-limits without root. Samsung either baked the list into the APK or feeds it from their servers. Either way, it isn't ours to edit.

Making it stick

If the allowlist can't be changed, the other lever is the thing that enforces it. daagent is an ordinary app, and ordinary apps can be disabled per user:

$ adb shell pm install-existing --user 95 com.discord
Package com.discord installed for user: 95

$ adb shell pm disable-user --user 0 com.samsung.android.da.daagent
Package com.samsung.android.da.daagent new state: disabled-user

Then reboot. Two minutes after boot, no START DELETE PACKAGE, no daagent process, and both clones still there and working, including the WhatsApp one that Samsung set up itself.

[!IMPORTANT] Disabling daagent doesn't uninstall anything or touch the profile's data, it only stops the app that would delete things. In the "will my WhatsApp clone get wiped" sense, this makes the profile safer, not riskier. The one thing that can still wipe it is turning Dual Messenger off in Settings, which removes the whole profile, and that's true with or without this trick.

Disabling daagent costs you a few things (from what i researched on the internet): the Dual Messenger settings page probably won't work, and clone notifications might be less reliable since daagent handles some of the profile's plumbing. The badge overlay on clone icons may go away too. All of it comes back with the mirror command:

$ adb shell pm enable com.samsung.android.da.daagent

which also means the next reboot will sweep Discord out again.

[!TIP] Back up the WhatsApp clone (Settings → Chats → Chat backup, from inside the clone) before doing any of this. The commands above don't threaten it. A clone profile with no backup is one accidental Settings toggle away from gone regardless.

If you'd rather not touch a system app at all, the fallback is to re-run install-existing after each reboot and log in again. Safe, but annoying.

What's still a guess

One thing I never nailed down is why plain adb install (no --user flag) put apps into the dual-app profile in the first place, which is what started all of this. The working theory is that adb install on this OEM build targets all users while the tap-to-install path targets only the current one, but I don't have a controlled test to call that proven. Everything in the two sections above, the reboot sweep and the fix for it, is verified from the logs and reproducible on demand.

Caveats before you do this

Permission prompts don't stick. The cloned Discord asked for photo access every time and never got it. dumpsys package showed why: every runtime permission for user 95 was granted=false with no USER_SET flag, so my "Allow" taps were never recorded, and POST_NOTIFICATIONS was denied the same way. Grant them from the shell instead:
```
$ adb shell pm grant --user 95 com.discord android.permission.READ_MEDIA_IMAGES
$ adb shell pm grant --user 95 com.discord android.permission.POST_NOTIFICATIONS
```
Same for READ_MEDIA_VIDEO, CAMERA, RECORD_AUDIO, whatever the app asks for. Force-close the clone afterwards.
Notifications can still be less reliable on a clone-profile instance of an app that was never designed with Samsung's dual-app framework in mind, even once the permission is granted. Log in and send yourself a test message before relying on it.
This isn't an officially supported configuration for arbitrary apps, only Dual Messenger's own allowlisted apps get Samsung's testing and support.
Check the app's terms of service if the clone is for running a second account. Some services restrict multi-accounting, and this post is about the Android mechanism, not a statement on any particular app's policy.
Removing a clone is the mirror command: adb shell pm uninstall --user 95 <package>.

Takeaways

A restrictive front-end UI doesn't mean the underlying platform mechanism is restrictive too. Samsung's Dual Messenger allowlist lives in its own app. The clone-profile primitive it writes to has none.
pm install-existing --user <id> <package> clones an already-installed app into any user profile without touching the APK. Samsung's own agent uses the same command.
If something you set up reverts on its own, suspect a boot-time job before suspecting a bug. logcat right after a reboot, filtered on the package name, found the culprit in seconds.
pm disable-user on the enforcing app is a legitimate, reversible fix when the policy it enforces isn't editable. Know what else that app does before you switch it off.

Building a Local-First Gacha Tracker: Reverse-Engineering Wuthering Waves' Convene Log

Sun, 23 Aug 2026 00:00:00 GMT

::github{repo="vermilion10/wuwa-pulltrack"}

Building a Local-First Gacha Tracker

Every gacha tracker asks you to paste a URL into someone else's website. That URL contains your account identifier and a signed token. The site fetches your pull history, stores it, and shows you a dashboard.

Nothing about that pipeline requires a third party. The game already writes everything needed onto your own disk. This post is the record of removing the middleman, including the two places where my first assumption was wrong, and the bug that would have silently destroyed the very data the tool exists to preserve.

[!NOTE] All account identifiers and tokens in this post are redacted. The record_id in particular is a short-lived credential, treat it like a password.

The premise

The hypothesis was simple:

The game writes a gacha-history URL somewhere local.
Tracker sites just read that URL and call the game publisher's own API.
Therefore the whole loop can run locally.

All three turned out to be true. Getting to them took longer than expected.

Method: three dead ends

Dead end 1, grep finds nothing

The community scripts all describe the same approach: search Client.log for a URL matching aki-gm-resources.../aki/gacha/index.html#/record. So I searched every log in the game's log directory.

Zero matches. Not even the bare substring gacha.

The file was clearly alive, modified seconds earlier, 3.5 MB. So the data was there and my reader was wrong. Dumping the first bytes explained why:

EF BB BF B4 97 95 97 93 8B 95 9D 8B 97 DE C2 DE DA ...

A UTF-8 BOM (EF BB BF), then bytes that are not valid UTF-8. The log is obfuscated. A rotated backup, closed, not being written, showed the same shape, so this wasn't an artifact of reading a file mid-write.

Dead end 2, the webview's storage

The convene history screen is a webview, so its storage seemed promising. Client/Saved/LocalStorage/LocalStorage.db is a plain SQLite file:

CREATE TABLE LocalStorage(key text primary key not null, value text not null);

It holds ~250 keys, graphics settings, red-dot state, RecentlyLoginUID, and teasingly GachaPoolOpenRecord_<uid>. But that last one is just a list of banner IDs you've viewed. No URL, no token.

Dead end 3, the SDK's debug log

Older scripts mention a second path:

Client/Binaries/Win64/ThirdParty/KrPcSdk_Global/KRSDKRes/KRSDKWebView/debug.log

This one is plaintext, a Chromium debug log. It still exists, but contains only GPU errors and spdy_session warnings. Whatever used to log the record URL there no longer does.

[!IMPORTANT] Three plausible sources, all dry. At this point the only remaining lead was the obfuscated log itself.

Finding: the cipher is one line

A single-byte XOR brute force scored no key above ~0.83 printable ratio, so not a trivial cipher. But there was a much better lever available than brute force: known plaintext.

UE4 logs open with their own timestamp, and the rotated backup file was named Client-backup-2026.08.21-15.21.53-.... So the plaintext almost certainly began:

[2026.08.21-15.21.53:...

Aligning that against the cipher bytes:

| Position | Cipher | Plain | cipher ^ plain | |---|---|---|---| | 0 | B4 | [ (5B) | EF | | 1 | 97 | 2 (32) | A5 | | 2 | 95 | 0 (30) | A5 | | 3 | 97 | 2 (32) | A5 | | 4 | 93 | 6 (36) | A5 | | 5 | 8B | . (2E) | A5 | | 10 | DE | 1 (31) | EF | | 11 | C2 | - (2D) | EF |

Two keys, 0xA5 and 0xEF, but alternating on no positional schedule. Sorting by which key applied reveals the rule instantly:

Key 0xA5 applies to: 32 30 36 2E 38 3A, all even
Key 0xEF applies to: 5B 31 2D 35 33 5D, all odd

The key is selected by the plaintext byte's parity. That sounds circular, you need the plaintext to pick the key, except both keys have bit 0 set, so XOR always flips the low bit. The cipher byte's own low bit tells you which key was used.

def decode(raw: bytes) -> str:
    body = raw[3:] if raw[:3] == b"\xef\xbb\xbf" else raw   # BOM is not obfuscated
    return bytes((b ^ 0xA5) if (b & 1) else (b ^ 0xEF)
                 for b in body).decode("utf-8")

Result: valid UTF-8, 84,002 lines, readable UE4 log, including Chinese-language entries, which is why the naive "printable ASCII ratio" heuristic capped at 0.86 and made the decode look wrong at first glance.

[!TIP] When a brute force stalls, look for known plaintext. A log file that embeds its own timestamp in its filename is a gift.

And there it was:

https://aki-gm-resources-oversea.aki-game.net/aki/gacha/index.html#/record
  ?player_id=<REDACTED>
  &record_id=<REDACTED>
  &svr_id=<REDACTED>
  &resources_id=<REDACTED>
  &gacha_type=1&lang=en&platform=PC

The API

Those four parameters are everything the record endpoint needs:

POST https://gmserver-api.aki-game2.net/gacha/record/query
{
  "playerId":     "<player_id>",
  "cardPoolId":   "<resources_id>",
  "cardPoolType": 1,               # 1..8, the actual banner selector
  "languageCode": "en",
  "recordId":     "<record_id>",
  "serverId":     "<svr_id>"
}

Probing all eight pool types showed cardPoolType is the real selector, cardPoolId doesn't need to vary with it. Each call returns that banner's complete window, newest first:

{
  "cardPoolType": "Resonators Accurate Modulation",
  "resourceId": 21030013,
  "qualityLevel": 3,
  "resourceType": "Weapon",
  "name": "Pistols of Night",
  "count": 1,
  "time": "2026-08-20 10:23:17"
}

The one irreducible constraint

record_id is minted when you open Convene History in-game, and expires in roughly a day. No tool avoids this, not this one, not the hosted trackers. It is a property of the API, not of running locally.

sequenceDiagram
    participant P as Player
    participant G as Game client
    participant L as Client.log
    participant T as Tracker (local)
    participant K as Kuro API

    P->>G: Open Convene History
    G->>L: write signed URL (obfuscated)
    P->>T: run `sync`
    T->>L: read + XOR-decode
    L-->>T: player_id, record_id, svr_id
    loop pool types 1..8
        T->>K: POST /gacha/record/query
        K-->>T: full window for that banner
    end
    T->>T: archive into SQLite

Architecture

No server, no client, no backend. One CLI that produces a report.

flowchart LR
    A[Client.log] -->|XOR decode| B[convene URL]
    B -->|parse params| C[Kuro record API]
    C -->|~6 month window| D[(pulls.db<br/>append-only)]
    D --> E[report.html]
    style A fill:#4a3aa7,color:#fff
    style D fill:#eda100,color:#000
    style E fill:#1baf7a,color:#fff

A static-file pipeline beats a local web server here: nothing to keep running, nothing listening on a port, and the output is a file you can archive or email.

The bug that mattered

Kuro's API serves a rolling window of roughly six months. Older pulls stop coming back. That single fact is the tool's entire reason to exist, and it broke my first schema.

The API gives no pull ID. So identity has to be derived. My first attempt keyed each row on its position from the oldest record:

pull_index = total - i   # WRONG

The reasoning: history only grows at the newest end, so position-from-oldest is stable. That reasoning holds only while the window doesn't slide.

When the oldest records age out, every index shifts by the number dropped. The upsert then writes new content onto old keys. Replaying that rule against a simulated slide on a real 550-row banner:

v1 keys written on full window : 550
v1 keys written on slid window : 350
keys whose CONTENT changed     : 350   <-- silent overwrites
  e.g. pull_index 350: was 'Originite: Type I' @ 2026-06-13
                       now 'Pistols of Night'  @ 2026-08-20

350 of 550 rows silently corrupted, precisely the old history the archive exists to protect. No error, no constraint violation. Just wrong data.

The fix: content-addressed identity

Anchor identity to absolute time, never to position:

PRIMARY KEY (player_id, pool_type, time, seq)

seq is needed because timestamps alone don't identify a pull:

A 10-pull shares one timestamp across all ten records.
The same resource_id can appear three times within that single second.

So within each timestamp, pulls are numbered 1..n oldest-first. That key survives any window movement, because a timestamp never changes.

stateDiagram-v2
    [*] --> Fetched: API returns window
    Fetched --> Existing: key already in archive
    Fetched --> New: key unseen
    Existing --> Kept: refresh last_seen only
    New --> Stored: insert with first_seen
    Kept --> [*]
    Stored --> [*]
    note right of Kept
        Rows the API stopped
        returning are never
        touched. That is the
        archive.
    end note

Verified against the real database:

| Scenario | Expected | Result | |---|---|---| | Drop oldest 200 from response | archive unchanged | 550 rows, byte-identical | | Slid window + 3 new pulls | old kept, new added | 553 rows, old intact | | Re-sync identical data | no-op | 0 new rows |

[!WARNING] If you build one of these: never key gacha rows by position in the API response. It works perfectly until the day it destroys your history, and it fails silently when it does.

Findings from 755 pulls

With the archive in place, the data is worth a look. Across three banners:

| Banner | Pulls | 5★ | Rate | Avg pity | |---|---|---|---|---| | Featured Resonator | 550 | 10 | 1.82% | 54.1 | | Featured Weapon | 84 | 1 | 1.19% | 63.0 | | Standard | 121 | 2 | 1.65% | 45.5 | | All | 755 | 13 | 1.72% | 53.5 |

The observed rate $\hat{p} = 13/755 = 1.72%$ sits close to the advertised 0.8% base rate once pity is accounted for. The Wilson 95% interval for this sample is wide:

$$ \hat{p} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = 0.0172 \pm 1.96\sqrt{\frac{0.0172 \times 0.9828}{755}} $$

which gives roughly $[0.86%,\ 2.65%]$, with 13 successes, this sample cannot distinguish much. Thirteen five-stars is an anecdote, not a measurement.

The pity distribution is more interesting than the rate:

| Pity bin | 11–20 | 31–40 | 51–60 | 61–70 | 71–80 | |---|---|---|---|---|---| | Count | 3 | 1 | 1 | 5 | 3 |

Eight of thirteen landed at pity 61 or higher, and the 21–30, 41–50 bins are empty. That bimodal shape, a few early, a cluster late, a gap between, is the visible fingerprint of a soft pity mechanic: the rate stays low, then ramps sharply somewhere in the 60s. You can see the mechanic in the shape even when the sample is far too small to pin down the rate.

What it looks like

The output is a single self-contained report.html, no CDN, no server, no network. Per banner: current pity against the 80-pull hard cap, 5★ rate, average pity, and every 5★ as a bar whose length is the pity it landed at. Plus an all-banners summary with monthly volume and the histogram above.

Dark mode is a selected palette rather than an inverted one, and the two data colors (gold for 5★, violet for 4★) were validated for colorblind separation rather than eyeballed, worst-pair ΔE 41.0 in light, 27.3 in dark.

Takeaways

Obfuscation is not encryption. A parity-selected two-key XOR is one line to undo once you have known plaintext. The filename was the known plaintext.
Prefer the boring architecture. A CLI that writes a static HTML file needs no server, no port, no process manager, and no deploy.
Interrogate your primary key against time. "History only grows at the end" was true on the day I wrote it and false six months later. Any key derived from position in a paginated response is a bug waiting for a rolling window.
Local-first means the archive outlives the API. The upstream keeps six months. The SQLite file keeps everything.

The whole thing is ~600 lines of Python with no dependencies outside the standard library. The interesting part was never the code, it was the twenty minutes of staring at hex.

Automated Daily Check-in System for Arknights Endfield: Architecture and Implementation

Sat, 15 Aug 2026 00:00:00 GMT

::github{repo="vermilion10/akef-checkin"}

Abstract

This article presents the design and implementation of akef-checkin, an automated daily check-in system for Arknights Endfield (SKport platform). The system employs a full OAuth 2.0 authorization flow with HMAC-SHA256 request signing to authenticate against Hypergryph and SKport APIs, eliminating the need for manual token refresh. Deployed as a serverless workflow via GitHub Actions, the system executes daily at 16:00 UTC with multi-account support and Discord webhook notifications. We detail the cryptographic signing mechanism, the four-step authentication pipeline, and the deployment architecture.

1 Introduction

Daily check-in systems for live-service games typically require authenticated API requests to claim rewards. The Arknights Endfield platform (operated by SKport/Hypergryph)[^1][^2] implements a multi-layer authentication scheme involving:

Long-lived account token (ACCOUNT_TOKEN) — stored by user, valid for weeks to months
Short-lived OAuth code — obtained via POST /user/oauth2/v2/grant
Session credential (cred) — generated via POST /user/auth/generate_cred_by_code
Signing token (signToken) — retrieved via GET /auth/refresh
Game role identifier — resolved via GET /game/player/binding

Traditional approaches store short-lived tokens, requiring frequent manual updates. Our system regenerates all transient credentials on every execution using only the long-lived ACCOUNT_TOKEN, achieving zero-maintenance operation[^3].

2 System Architecture

2.1 High-Level Overview

The system comprises four modular components:

| Component | File | Responsibility | |-----------|------|----------------| | Entry Point | src/index.ts | Orchestration, multi-account iteration, result aggregation | | Authentication | src/auth.ts | Four-step OAuth flow, cryptographic signing | | Notification | src/discord.ts | Discord webhook delivery | | Configuration | src/constants.ts | API endpoints, headers, platform constants |

2.2 Execution Pipeline

flowchart TD
    A[Start: Read SKPORT_TOKENS] --> B{For each account}
    B --> C[Step 1: Get OAuth Code]
    C --> D[Step 2: Generate Cred]
    D --> E[Step 3: Get Sign Token]
    E --> F[Step 4: Get Game Role]
    F --> G[Sign Attendance Request]
    G --> H[POST /game/endfield/attendance]
    H --> I{Response Code}
    I -->|0| J[Success: Parse Rewards]
    I -->|1001, 10001| K[Already Signed In]
    I -->|10002| L[Token Expired]
    I -->|Other| M[Failure: Log Error]
    J --> N[Aggregate Result]
    K --> N
    L --> N
    M --> N
    N --> O{More Accounts?}
    O -->|Yes| B
    O -->|No| P[Send Discord Webhook]
    P --> Q[Exit]
    
    style A fill:#f97316,color:#fff
    style Q fill:#22c55e,color:#fff
    style J fill:#3b82f6,color:#fff
    style K fill:#3b82f6,color:#fff
    style L fill:#f97316,color:#fff
    style M fill:#ef4444,color:#fff

3 Authentication Flow

3.1 Sequence Diagram

The complete authentication sequence involves two identity providers: Hypergryph (as.gryphline.com) for OAuth grant and SKport (zonai.skport.com) for game-specific credentials.

sequenceDiagram
    participant Script
    participant Gryphline as as.gryphline.com
    participant SKport as zonai.skport.com

    Script->>Gryphline: POST /user/oauth2/v2/grant<br/>{token: ACCOUNT_TOKEN, appCode, type: 0}
    Gryphline-->>Script: {code: "oauth_code"} (one-time use)

    Script->>SKport: POST /web/v1/user/auth/generate_cred_by_code<br/>{code: oauth_code, kind: 1}
    SKport-->>Script: {cred: "session_credential", token: "..."}

    Script->>SKport: GET /web/v1/auth/refresh<br/>(cred, timestamp, platform, vName)
    SKport-->>Script: {token: "sign_token"}

    Script->>SKport: GET /api/v1/game/player/binding<br/>(cred + signed request)
    SKport-->>Script: {list: [{appCode: "endfield", bindingList: [...]}]}

    Script->>SKport: POST /web/v1/game/endfield/attendance<br/>(cred + gameRole + signed request)
    SKport-->>Script: {code: 0, data: {signInCount, awardIds, resourceInfoMap}}

3.2 Cryptographic Signing (V2)

All signed requests use the V2 signature algorithm: HMAC-SHA256 followed by MD5.

export function generateSignV2(
    path: string,
    timestamp: string,
    platform: string,
    vName: string,
    salt: string
): string {
    const headerJson = JSON.stringify({ platform, timestamp, dId: "", vName });
    const s = `${path}${timestamp}${headerJson}`;
    const hmac = crypto.createHmac("sha256", salt).update(s).digest("hex");
    return crypto.createHash("md5").update(hmac).digest("hex");
}

Mathematical formulation:

$$ \text{sign} = \text{MD5}\big(\text{HMAC-SHA256}(\text{salt}, \text{path} \parallel \text{timestamp} \parallel \text{JSON}({\text{platform}, \text{timestamp}, \text{dId}, \text{vName}}))\big) $$

Where:

$\parallel$ denotes string concatenation
$\text{salt} = \text{signToken}$ (per-session secret)
$\text{path}$ = API endpoint path (e.g., /web/v1/game/endfield/attendance)
$\text{timestamp}$ = Unix epoch seconds as string

[!NOTE] The double-hash construction (HMAC-SHA256 → MD5) provides defense-in-depth: even if MD5 collision resistance is compromised, the HMAC-SHA256 layer preserves integrity assuming the salt remains secret.

3.3 Request Structure

Each signed request includes the following headers:

| Header | Value | |--------|-------| | cred | Session credential from Step 2 | | platform | "3" (constant) | | vName | "1.2.0" (client version) | | timestamp | Current Unix timestamp (seconds) | | sign | V2 signature (hex) | | sk-game-role | Game role identifier (format: 3_{roleId}_{serverId}) | | sk-language | Locale code (en, ja, zh_Hans, etc.) | | dId | Empty string (device ID placeholder) |

4 Implementation Details

4.1 Multi-Account Processing

The entry point processes accounts sequentially with a 2-second delay between requests to avoid rate limiting:

for (let i = 0; i < accounts.length; i++) {
    const account = accounts[i];
    const result = await performCheckin(account);
    results.push(result);

    if (i < accounts.length - 1) {
        await new Promise(resolve => setTimeout(resolve, 2000));
    }
}

4.2 Response Parsing

On successful check-in (HTTP 200, code === 0), the response contains reward information:

interface AttendanceResponse {
    code: number;
    message?: string;
    data?: {
        signInCount?: number;
        awardIds?: Array<{ id: string }>;
        resourceInfoMap?: Record<string, { name: string; count: number }>;
    };
}

Rewards are mapped by awardIds → resourceInfoMap and formatted as ItemName xCount.

4.3 Error Handling

The system categorizes responses into four classes:

| Code / Condition | Category | Action | |------------------|----------|--------| | code === 0 | Success | Parse rewards, report day count | | code ∈ {1001, 10001} or "already" in message | Already Signed In | Report success (idempotent) | | code === 10002 | Token Expired | Alert user to update ACCOUNT_TOKEN | | Other / Network Error | Failure | Log error, continue to next account |

5 Deployment Architecture

5.1 GitHub Actions Workflow

The system runs serverlessly via GitHub Actions[^5] on ubuntu-latest with Bun runtime:

name: Endfield Auto Check-in

on:
  schedule:
    - cron: '0 16 * * *'  # Daily at 16:00 UTC (00:00 WIB)
  workflow_dispatch:

jobs:
  checkin:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v1
        with:
          bun-version: latest
      - run: bun run src/index.ts
        env:
          SKPORT_TOKENS: ${{ secrets.SKPORT_TOKENS }}
          DISCORD_WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}

Schedule interpretation:

0 16 * * * → Minute 0, Hour 16 (UTC) → Midnight WIB (UTC+7)
workflow_dispatch enables manual triggering via GitHub UI

5.2 Secrets Management

Two repository secrets are required:

| Secret | Required | Format | |--------|----------|--------| | SKPORT_TOKENS | Yes | JSON array of account objects | | DISCORD_WEBHOOK_URL | No | Discord webhook URL |

Account object schema:

{
  "accountToken": "string (from Network tab)",
  "accountName": "string (display name)",
  "language": "string (optional, default: en)"
}

6 Token Acquisition Procedure

The ACCOUNT_TOKEN is not visible in browser cookies. It must be extracted from the OAuth grant request:

Navigate to https://game.skport.com/endfield/sign-in while logged in
Open DevTools (F12) → Network tab
Filter for grant
Reload page (F5)
Select POST https://as.gryphline.com/user/oauth2/v2/grant
Open Payload tab (not Response)

Copy token value from JSON body:

{
  "token": "ACCOUNT_TOKEN_HERE",
  "appCode": "6eb76d4e13aa36e6",
  "type": 0
}

[!IMPORTANT] The ACCOUNT_TOKEN is long-lived (weeks to months). The script regenerates all short-lived credentials (cred, signToken, gameRole) on every run, so token rotation is rarely needed.

7 Local Development

7.1 Prerequisites

Bun v1.0+

7.2 Configuration

cp .env.example .env
# Edit .env with your ACCOUNT_TOKEN

7.3 Execution

bun run start

Expected output:

Starting Arknights Endfield Auto Check-in...

Processing check-in for account: MyAccount...
  [1/4] Getting OAuth code...
  [2/4] Generating cred...
  [3/4] Getting sign token...
  [4/4] Getting player binding (game role)...
  Auth complete! Game role: 3_12345678_9

**Endfield:** Check-in for `MyAccount`
✅ Success!
 Day: 15
 Rewards: Credit x1000, Synthetic Fuel x5

Sending webhook to Discord...
Done!

8 Security Analysis

8.1 Threat Model

| Asset | Threat | Mitigation | |-------|--------|------------| | ACCOUNT_TOKEN | Exfiltration via logs | Never logged; only used in memory | | signToken | Replay attacks | Bound to timestamp; expires quickly | | cred | Session hijacking | Short-lived; regenerated per run | | Discord webhook | Spam/abuse | Optional; rate-limited by Discord |

8.2 Cryptographic Properties

The V2 signature[^4] provides:

Integrity: Any modification to path, timestamp, or headers invalidates signature
Replay resistance: Timestamp inclusion limits window to ~1 second (server-side validation)
Key separation: signToken derived per-session, not reusable across accounts

8.3 Operational Security

No persistent storage of credentials
All secrets injected via GitHub Actions encrypted secrets
Network requests use HTTPS with certificate validation (default in fetch)
User-Agent mimics legitimate browser (Firefox 147)

9 Related Work

| Project | Platform | Approach | |---------|----------|----------| | akef-checkin (this work) | Arknights Endfield (SKport) | Full OAuth flow regeneration | | hoyolab-auto-checkin | Genshin Impact / Star Rail | Cookie-based with manual refresh | | arknights-auto-sign | Arknights (Global/CN) | Token caching with TTL |

Our approach differs by eliminating token caching entirely — every execution performs a fresh OAuth grant, trading ~3 extra HTTP requests for zero-maintenance operation.

10 Conclusion

The akef-checkin system demonstrates a practical application of OAuth 2.0 and request signing for game automation. Key contributions:

Zero-maintenance authentication — single long-lived token generates all ephemeral credentials
Cryptographically sound signing — HMAC-SHA256 + MD5 V2 algorithm with per-session keys
Serverless deployment — GitHub Actions cron with Bun runtime, no infrastructure management
Multi-account support — sequential processing with rate-limit awareness
Observability — structured logging and Discord notifications

Future work includes: exponential backoff for transient failures, Prometheus metrics export, and support for additional SKport game titles.

Appendix A: API Endpoint Reference

| Constant | URL | |----------|-----| | GRANT | https://as.gryphline.com/user/oauth2/v2/grant | | GENERATE_CRED | https://zonai.skport.com/web/v1/user/auth/generate_cred_by_code | | REFRESH_TOKEN | https://zonai.skport.com/web/v1/auth/refresh | | BINDING | https://zonai.skport.com/api/v1/game/player/binding | | ATTENDANCE | https://zonai.skport.com/web/v1/game/endfield/attendance |

Appendix B: Type Definitions

interface AccountConfig {
    accountToken: string;
    accountName: string;
    language?: string;
}

interface AuthResult {
    cred: string;
    signToken: string;
    gameRole: string;
}

interface ResourceInfo {
    name: string;
    count: number;
}

interface AttendanceData {
    signInCount?: number;
    awardIds?: Array<{ id: string }>;
    resourceInfoMap?: Record<string, ResourceInfo>;
}

References

[^1]: Hypergryph OAuth 2.0 Implementation, as.gryphline.com (reverse-engineered) [^2]: SKport Web API v1, zonai.skport.com (reverse-engineered) [^3]: RFC 6749 — The OAuth 2.0 Authorization Framework [^4]: RFC 2104 — HMAC: Keyed-Hashing for Message Authentication [^5]: GitHub Actions Documentation — Scheduled Events (on.schedule)

This article documents the akef-checkin repository as of version 1.0.0. Source code available at the project repository.