Reverse Engineering the Shiny Colors Asset Pipeline#

This document records the reverse-engineering work carried out while developing shinymas-dl, a .NET downloader and extractor for THE IDOLM@STER SHINY COLORS. It is an internal research record rather than a usage guide. Values that identify or authenticate an account are intentionally omitted.

The investigation began with a simple question: given that the game runs entirely in the browser, how much of its asset pipeline can be reconstructed from the client itself? The answer was considerably more than expected. The browser client exposed a complete asset catalogue, the naming and hashing rules for ordinary CDN objects, the local format used for text resources, and, after authenticated API work, the metadata needed to resolve most of the card assets that initially appeared to be unavailable.

The final result was a C# implementation with separate catalogue, download, naming, extraction, and API layers. The full album crawl recorded 1,440 card hashes for 1,461 card IDs present in asset-map v442, with no unowned card among the covered characters left without a hash.

1. Scope and initial reconnaissance#

The starting point was an existing family of game asset tools: kc-dl, exilium-dl, and pgrasset. The goal for shinymas-dl was not to reproduce the game client as a whole, but to establish a reliable archive pipeline with three properties:

1. the upstream catalogue should be recoverable without a game installation;
2. the raw mirror should be resumable and independent from the final extraction layout; and
3. naming and organization should come from data shipped by the game wherever possible.

The first pass therefore avoided writing application code. The site, client bundle, CDN behaviour, catalogue format, and resource transformations were investigated first.

The game entry page was reachable without a geographic restriction. The HTML referenced a small env.js followed by the main application bundle. The client identified the API root, asset root, game identifier, and a flag indicating that its resource-processing layer was enabled. The main bundle was approximately 3.7 MB, and later inspection identified 1,109 lazy-loaded chunks.

The useful distinction at this stage was between information that could be obtained from the public client and information that depended on an authenticated game session. The former became the basis of the downloader. The latter was investigated separately and only integrated once its behaviour had been demonstrated with a test account.

2. Reconstructing the asset catalogue#

The most important early discovery was that the client did not hard-code the complete list of resources. Instead, it loaded an asset-map manifest and a set of versioned chunks.

The live manifest used 129 chunks. On the build investigated in this session, the result was:

The manifest reported 35.19 GiB in total.

Each chunk maps a logical asset path to a version value. Those paths are the names used by the application before the CDN-specific filename transformation is applied. That distinction mattered because the visible game resource path and the actual URL requested from the CDN are not the same thing.

The refresh implementation keeps the manifest and chunks in data/. A subsequent refresh compares chunk versions and downloads only chunks whose version changed. The manifest is written last, so an interrupted update does not leave a new manifest pointing at a partially updated chunk set.

The first successful run reproduced all 386,102 entries locally. A second refresh against the unchanged manifest performed no unnecessary chunk downloads.

3. CDN naming and resource transformations#

3.1 Filename hashing#

The browser does not request a resource by its logical path. The client-side hashing function takes the filename stem, its first and last characters, and the full path under /assets/, then hashes the result with SHA-256.

Conceptually, for an asset path p with filename stem s, the input is:

first(s) + last(s) + "/assets/" + p

The digest is represented as lowercase hexadecimal. Media files keep their .mp3, .mp4, or .m4a extension on the CDN; the other resource classes use the bare hexadecimal name.

This was not inferred from a single sample. The same rule reproduced hashes for images, Spine files, JSON, animation data, audio and video. A representative ordinary path is:

images/bg/001.jpg

which becomes a 64-character hexadecimal object name derived from the rule above.

The asset map therefore contains enough information to construct ordinary CDN URLs without consulting any external database.

3.2 Text resources#

The next obstacle was that not all resources were served as their final plaintext representation. JSON and Spine .atlas resources were transformed before being written to the network.

Static inspection of the client revealed picosha2, zlib, an embedded string used as a key, and an inflate path. Several hypotheses were tested against known resource files. The useful signal was not a printable-text score but the compressed data itself: once the XOR transformation was correct, the resulting stream had a valid gzip/deflate structure and decompressed to the same plaintext used by the game.

The implemented transformation is a repeating XOR followed by raw-deflate decompression. The embedded ASCII key is 54 bytes long. After XOR, the resource contains the gzip header and a deflate body; the client-side format omits the normal gzip trailer, so the C# port inflates the raw deflate portion directly.

The important implementation decision was to keep this transformation in the raw-to-plain extraction boundary. The raw mirror remains a faithful copy of the game resources, while extract is responsible for producing readable JSON, scripts and other derived outputs.

For a concrete validation, a downloaded json/produce_events/...json resource was decrypted and parsed successfully, and a large Spine .json resource expanded to readable JSON. Images, audio, video and fonts did not require this transformation.

4. The CDN’s most misleading property#

Once the ordinary path transformation worked, the next problem appeared during bulk testing: a missing resource did not necessarily produce an HTTP 404.

Unknown paths were served as the game’s single-page application shell with status 200 and an HTML content type. A downloader that considered only the HTTP status would therefore save copies of index.html under resource filenames.

The downloader treats an HTML response as the CDN’s SPA fallback. Missing entries are recorded in the download index so that a normal resumed run does not repeatedly probe the same unavailable object. A --retry-missing option removes those records when a later investigation shows that the object may have become available.

The distinction is useful beyond this project. The server’s HTTP status was technically successful while the requested resource was absent. Content validation was therefore part of the downloader’s correctness, not merely an error-handling detail.

5. Building the first downloader#

The first implementation mirrored the asset map directly into output/raw/. The logical path was preserved, which made the raw tree independent from any future naming convention.

The download index records the upstream version and local size for each completed resource. A file is skipped when the version is unchanged and its on-disk size still matches the recorded value. A version change forces a refetch.

Index writes are performed atomically. The implementation serializes a sorted snapshot into a temporary file and then moves it into place. Progress is stored periodically rather than only at the end, because a full mirror is too large for an interrupted run to be treated as a single transaction.

Downloads use bounded parallelism through Parallel.ForEachAsync. Large files are written to a temporary .part file and renamed only after the transfer completes. Network failures and transient request failures are retried with bounded exponential backoff.

One practical test downloaded the complete story-script category: 10,804 files totaling about 21 MiB, using a concurrency of 16. The run took roughly four minutes. The limiting factor at that size was request latency rather than aggregate bandwidth.

The resulting CLI was split into five operations:

This separation proved useful later when the resource-availability assumptions changed. The downloader did not need to be rewritten simply because the extraction rules grew more complicated.

6. Naming the data without an external database#

6.1 The initial problem#

The catalogue identifies characters and cards primarily by numeric IDs. A raw mirror such as

images/content/characters/icon_circle/001.png
images/content/idols/card/1040010010.jpg

is machine-readable but not particularly useful as an archive.

The project therefore set a constraint: use the game’s own data to build names instead of depending on a community-maintained character table.

6.2 Story scripts as a name source#

The story resources were particularly useful because their records contain information about the character visible in a scene, a romanized character label, and the Japanese speaker name.

After the text resources were decrypted, the implementation scanned the mirrored scripts and built a name index from repeated observations. The resulting cache contains 33 characters: all 28 idols, Hazuki, and four collaboration characters.

The naming process is intentionally statistical rather than based on a single line. A scene may identify one character while another character is speaking, so the visible character label and speaker name are collected as separate observations and combined by frequency.

The complete script collection was sufficient to build the table locally in about 1.2 seconds. No external character metadata was needed.

6.3 Card ID structure#

The card IDs themselves also contained useful structure. A representative produce card ID can be separated as:

1 | 04 | 001 | 0010

where the fields represent card type, rarity, character ID, and sequence within that character’s card set. Support cards use the corresponding support-card type.

This meant that most card paths could be organized directly from their own IDs. For example, character 001 can be resolved to the mano name through names.json, and a card beginning 104001... can be routed without a separate card database.

The structure also explained several apparently unusual groups beginning with 193, 194, 293, and 294. At first those prefixes did not match the normal rarity constants found in the client. They were later shown to be present in the album data as well, so they were not malformed IDs.

7. Extraction layout#

The raw mirror deliberately preserves upstream naming. extract is the layer that turns it into an archive organized for human use.

The principal routes are:

A readable script.txt is generated alongside story JSON. The text representation includes the speaker, line, choices where applicable, and a matching voice path when one exists.

The extractor also handles Spine directories and the game’s other structured resource types. Text assets are decrypted during extraction, so the raw mirror remains suitable for later reprocessing if the archive layout changes.

8. The first coverage estimate was wrong#

At this point the downloader appeared mostly complete, but a random reachability test suggested that only about ten percent of resources were missing. That estimate was misleading because the sample was dominated by resource classes that remained publicly reachable, especially voice and Spine assets.

A full audit of card-related paths gave a different result:

The exact equivalence between idol card art and idol icons was significant. The same cards failed in the same pattern, including the same rarity distribution. That strongly suggested that the problem was attached to a card-level attribute rather than to individual files.

The investigation then shifted from the CDN to the client code that constructs resource paths.

9. Discovering the protected asset convention#

Static analysis of the main bundle and its lazy-loaded chunks identified a second URL form used by card resources:

<hash>_<id>.<extension>

The hash was not derived from the individual file path. It belonged to the card record itself.

The client uses the same card hash for several related resources, including:

• card icons;
• card art;
• card thumbnails;
• some card movies;
• awake and memorial-related artwork;
• certain fes-related images.

The code also exposed several hash-bearing fields with slightly different roles:

Other game systems have their own independent hashes, such as gasha resources and some event-specific content. Those were outside the card-asset scope of this investigation.

The asset map was therefore not actually missing the card objects. It was missing one piece of information needed to construct their current CDN object names.

10. Where the card hashes came from#

The character album turned out to be the key data source.

Static analysis found a character album request of the form:

POST characterAlbums/characters/{characterId}

The album entries contain card IDs and their hash values. The client also tracks ownership separately through fields such as hasIdol and hasSupportIdol.

A particularly important observation came from the album UI code. The album constructs an icon for every entry, not only the cards owned by the account. Unowned cards are rendered as grey and are made non-interactive, but they are still part of the list. The card-art screen is only entered for owned cards.

That resolved a confusion from the original network captures. The apparent “availability after a condition” was not evidence that the server omitted every piece of metadata until the condition was met. The more precise description was that the UI withheld the art request behind an ownership condition while the album data already contained the card record and its hash.

The list of characters displayed by the album comes from GET album/top.

11. Understanding the authenticated API transport#

The game’s API uses a different transport from ordinary CDN resources. The browser serializes the logical request as a raw HTTP request, encrypts that representation, and sends it as a single application/octet-stream POST.

The relevant client module is named request_hash. Static analysis showed an asm.js build and a WebAssembly variant. The surrounding code identifies the following behaviour:

1. Construct the raw request text, including method, path, headers, and body.
2. Pass the text through encodeRequest.
3. Send the resulting bytes to the game API as application/octet-stream.
4. Receive an encrypted response body as an array buffer.
5. Decode the response with the x-sessionid returned by the server.

The API client uses a rotating session identifier. Requests are queued and sent serially because the response to one request supplies the session identifier used by the next request.

The browser build investigated in this session used x-version: 410 and an x-em-version value embedded in the client bundle. The API client also recognised retryable statuses 1012 and 1090, and it stops when the response contains x-is-banned.

The request codec was first exercised offline. A request encoded with the game’s own module could be decoded back to its original text. This demonstrated that the format itself was understood before any authenticated API experiment was attempted.

11.1 A misleading transport failure#

Early attempts from Node produced HTTP 403 responses. The initial assumption was that the environment might be geographically restricted or that the API required a special browser fingerprint.

A controlled test disproved both explanations. The same encoded no-token login was accepted by the game server when sent with curl, producing HTTP 401 and an encrypted error response. .NET’s HttpClient, both with its default User-Agent and with a browser-style User-Agent, also reached the server and produced the same 401.

The resulting decoded error was:

{"title":"Unauthorized","status":1010,"instance":"/login"}

The 403 therefore belonged to the earlier transport path rather than to a game-side geographic block. This mattered because it prevented the API implementation from being built around a false region-lock theory.

12. The request codec implementation#

One could fully reimplement the game’s request codec in native C#, but there was no practical reason to do that for the downloader. The game already supplies the codec module on its public client side, and the client changes whenever the game changes.

The implementation instead resolves the live client at runtime. It reads the current index.html, env.js, and application bundle, extracts the API root, game ID, client version headers, and the current request_hash chunk name, then caches those client files under data/client/.

The codec is executed inside .NET using Jint. The implementation loads the game’s own JavaScript module and calls its exported encodeRequest and decodeResponse functions. No copy of the game’s codec source is bundled into the repository.

This decision has two advantages. First, the tool remains aligned with the current client bundle. Second, it avoids maintaining a native implementation of a format whose details are internal to the game.

The separate reverse-engineering experiments did, however, reveal a useful property of the request format. An encoded request has an eight-byte header, and the output changes with the current Unix second. Holding the input constant and delaying the call changed the encoded output. Static inspection tied the timestamp source to Date.now() and the native implementation to an SFMT-based generator. That observation was enough to explain why repeated identical requests do not necessarily produce identical ciphertext.

13. Establishing the login chain#

The remaining problem was obtaining a valid game session for the authenticated album calls.

The browser’s Enza integration first supplies a short-lived game token, which is then used by the game login endpoint. A successful login returns the initial game X-Sessionid; subsequent API calls rotate this value.

The test sequence that finally succeeded was:

No account credential is stored in the project. The session value is supplied to the process environment for the duration of a run and is not written to the data directory, logs, source tree, generated JSON, or git history.

One false start came from using a different Enza token type. The platform treats those credentials differently. The game accepted the correct browser session-derived game token but treated the other token as a guest context. This was eventually resolved by inspecting the actual browser session flow rather than assuming that every Enza token represented the same credential layer.

14. The first authenticated album test#

Character 001 was used as the controlled test case before the full crawl.

The album contained 64 cards: 22 produce cards and 42 support cards. The test account owned only a small subset of them. Nevertheless, every one of the 64 album entries carried a non-empty 32-character hash, including the cards the account did not own.

The album also contained 23 costume entries with their own hash fields.

The important end-to-end test used the unowned card 1940010010. The ordinary path continued to return the SPA fallback, while the hashed CDN form returned a real JPEG with a size of 76,235 bytes. This established the entire chain:

asset map card ID
        |
        v
character album
        |
        v
card hash
        |
        v
<hash>_<cardId>.jpg
        |
        v
real CDN image

This was the point at which the original reachability problem stopped being a catalogue problem. The downloader had already known that the card existed. It now had a reproducible mapping from the logical card ID to the current protected CDN object name.

15. Whisper voices and client-side release state#

Whisper voices required a separate check because they use voiceHash rather than the main card hash.

The support-card data contains a list of whisper voice records together with their release conditions. The client constructs the voice path from idolId, voice ID, and voiceHash, then evaluates whether the voice is released from the card’s evolution stage.

Character 014 provided the decisive test case. The test account owned none of the support cards carrying the relevant whisper set. Six voice entries were nevertheless present, all marked as locked by the client and all carrying voiceHash values.

All six downloaded through their hashed paths. Together they accounted for 64.6 MiB.

The observed model is therefore slightly different from the initial assumption that a locked voice simply did not exist to the client. The client receives both the resource metadata and the release condition, then decides locally which tracks should be exposed in the current UI state.

Six whisper files in the asset map remained unresolved. They correspond to the ...000.m4a track in each set and are not referenced by an album entry in the data examined during this session.

16. Full album crawl#

After the controlled tests passed, the same process was applied to all 28 characters returned by album/top.

The crawl completed in 59.4 seconds on the test environment.

The 21 cards without a recorded hash were concentrated in characters not represented by the album list used by the crawler: the four collaboration characters and Hazuki. The six unresolved whisper files were the ...000 member of each whisper set rather than normal album-referenced voice entries.

The crawl was also made resumable. A completed character is recorded in the local album-hash store, so a second invocation skipped all 28 characters rather than repeating the API calls.

This changed the downloader’s semantics from “try every known CDN path” to “build the best-known current CDN path from game metadata, then fall back to the ordinary path when no hash is known.”

17. Integrating the album data into shinymas-dl#

The resulting feature was implemented as an albums command.

The command performs the following operations:

1. resolve the current client bundle and API transport;
2. obtain an authenticated game session from the supplied environment credential;
3. read the album character list;
4. fetch each character album serially;
5. store card and costume IDs with their corresponding hashes; and
6. make those hashes available to the download layer.

The persistent album file contains only resource IDs and hashes. It does not contain authentication state.

The downloader’s asset representation was extended with an optional CdnPath. If a card hash is known, the hashed CDN form is attempted first. If no hash is available, the existing ordinary path logic remains unchanged. This preserves the original behaviour for resources outside the album-derived set.

The API-side client also follows the browser’s sequential session behaviour. It refreshes the current session ID after each response, applies the observed retry semantics, and treats the ban header as a hard stop rather than trying to continue.

18. Validation and security checks#

Because the album feature is the only part of the project that handles an account session, the implementation was checked separately from the functional tests.

The following checks were performed after the completed crawl:

• the session value did not appear in git history;
• no JWT-looking token strings appeared in git history;
• the data/ and output/ trees were not tracked by git;
• the card-hash JSON contained only IDs and hashes;
• the session value, game token, and per-request session IDs were not printed or persisted by the API client.

The browser session was treated as a credential and was not copied into the research document. The test account should be logged out of Enza after experimentation rather than leaving the session active indefinitely.

19. Bugs and corrections#

The investigation also produced several failures unrelated to the asset protection itself. They were important because each one could have produced a plausible but incorrect archive.

19.1 The name-cache regression#

The original extract command rebuilt names.json from whatever story scripts happened to be present in the current raw mirror. On a deliberately small test mirror, that reduced the source set to two scripts and replaced the previously complete 33-entry cache with a one-entry table.

The failure was silent because the extraction still completed successfully.

The fix was to merge a newly built name table into the cached table. Newly observed values replace stale entries, while names not represented in the current sample remain in the cache.

This is a general issue with derived caches: a smaller input set is not evidence that previously known data should be deleted.

19.2 Awake images initially routed as generic output#

Awake-step images were initially falling under other/ because their resource paths did not match the first set of extraction rules. The underlying card association was later shown by the client to use the card hash, but the extraction routing itself remained separate from the CDN lookup. This is an organizational issue rather than a download failure.

19.3 The progress renderer#

The terminal progress renderer used carriage returns to update a single line. When stdout was redirected, those updates became permanent log lines. The fix was to suppress high-frequency redraws when output is redirected.

19.4 The apparent region lock#

The 403 responses from the early Node transport caused the API to look geographically restricted. Controlled requests using curl and .NET disproved that assumption. The actual cause was the earlier HTTP path rather than a regional requirement.

This was a useful reminder to isolate transport failures before attributing them to application-level restrictions.

20. Final architecture#

The final repository has four logically distinct layers:

flowchart TD
    A[Live game client] --> B[Asset map]
    A --> C[Client bundle and API metadata]
    B --> D[Catalog]
    D --> E[Downloader]
    C --> F[Authenticated album API]
    F --> G[Card hash store]
    G --> E
    E --> H[Raw mirror]
    H --> I[Text decryption and extraction]
    I --> J[Organized archive]
    I --> K[Readable story scripts]

The important boundary is between acquisition and interpretation. The raw mirror remains close to the upstream layout. All naming, grouping, decryption, and presentation decisions happen later.

This means a change in extraction rules does not require a new download, and a change in card-hash resolution does not invalidate the rest of the archive.

21. What remains unresolved#

The investigation did not establish a complete source for every hash-bearing resource family in the game. In particular:

• 21 card IDs in the v442 asset map were not represented in the 28-character album crawl. They belong to four collaboration characters and Hazuki.
• Six whisper ...000.m4a files were present in the asset map but had no corresponding album entry in the examined data.
• Some specialised hashed systems outside the card and whisper paths were not integrated because they have their own records and were outside the scope of shinymas-dl.
• The exact server-side consequences, if any, of calling the album POST repeatedly were not established. The downloader therefore treats the calls as read-oriented but does not assume that the endpoint is formally documented as side-effect free.
• The request codec’s internal SFMT construction was understood far enough to characterize its time-dependent behaviour, but the finished tool deliberately executes the game’s codec instead of maintaining a separate native cryptographic implementation.

These are boundaries of the investigation rather than claims that the corresponding resources are permanently unavailable.

22. Conclusion#

The difficult part of shinymas-dl was not writing a parallel downloader. It was identifying the layers between a logical game resource and the object ultimately requested by the browser.

The first layer was public and straightforward once the client was read carefully: the asset map described 386,102 resources, the filename transformation could be reconstructed from the resource-hashing code, and the text format could be reversed through known compressed output.

The second layer was more deceptive. The CDN returned 200 for nonexistent paths, making the initial coverage estimate look much better than the actual card-art coverage. Once card requests were compared with their client-side path builders, the missing information was identifiable as a card-level hash.

The final step was to determine where that metadata entered the browser. The character album exposed hashes for all of its card entries, including entries not owned by the test account. The distinction between metadata availability and UI interaction explained why the network trace appeared to show a condition-gated resource.

The final implementation reflects that understanding. shinymas-dl can maintain an ordinary raw mirror, augment it with authenticated card metadata when available, download the corresponding protected resources, and organize the result offline without coupling the archive layout to the game’s current UI.

As of asset-map v442, the completed album crawl recorded 1,440 hashes for 1,461 card IDs and established a complete hash mapping for every unowned card represented by the 28-character album set. The remaining gaps are specific and identifiable rather than an undifferentiated collection of HTTP 404s.

Appendix A. Selected resource-path examples#

# Logical resource path
images/content/idols/card/1040010010.jpg

# Ordinary CDN lookup
/assets/<sha256-derived-name>?v=<version>

# Card-hash lookup after album resolution
images/content/idols/card/<card-hash>_1040010010.jpg

For media resources, the hashed CDN object keeps its media extension, for example .mp4 or .m4a.

Appendix B. Relevant client-side fields and modules#

The exact webpack module IDs are build-specific and will change when the client is rebuilt. The following names were useful during this investigation:

Appendix C. Reproducibility notes#

All numerical measurements in this document refer to the 27 September 2026 investigation and, where applicable, asset-map v442. The authenticated tests used a dedicated test account. Authentication material itself is intentionally absent from this document.

The raw session transcript used to prepare this report is archived separately. The report is written from the observed sequence of experiments, corrections, implementation changes, and validation results in that transcript, rather than from assumptions about how the game ought to work.