feat(frontend): add HuggingFace media output rendering in result panel#5675
Open
juliethecao wants to merge 59 commits into
Open
feat(frontend): add HuggingFace media output rendering in result panel#5675juliethecao wants to merge 59 commits into
juliethecao wants to merge 59 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5675 +/- ##
============================================
+ Coverage 56.89% 57.13% +0.23%
- Complexity 3025 3026 +1
============================================
Files 1127 1128 +1
Lines 43756 43987 +231
Branches 4736 4795 +59
============================================
+ Hits 24894 25130 +236
+ Misses 17389 17380 -9
- Partials 1473 1477 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 393 | 0.24 | 23,528/37,554/37,554 us | 🔴 +9.6% / 🔴 +149.2% |
| 🔴 | bs=100 sw=10 sl=64 | 803 | 0.49 | 123,262/150,213/150,213 us | 🔴 +8.3% / 🔴 +39.6% |
| ⚪ | bs=1000 sw=10 sl=64 | 911 | 0.556 | 1,097,863/1,164,793/1,164,793 us | ⚪ within ±5% / 🔴 +13.3% |
Baseline details
Latest main 2442e8f from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 393 tuples/sec | 413 tuples/sec | 777.62 tuples/sec | -4.8% | -49.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.24 MB/s | 0.252 MB/s | 0.475 MB/s | -4.8% | -49.4% |
| bs=10 sw=10 sl=64 | p50 | 23,528 us | 24,826 us | 12,612 us | -5.2% | +86.5% |
| bs=10 sw=10 sl=64 | p95 | 37,554 us | 34,272 us | 15,070 us | +9.6% | +149.2% |
| bs=10 sw=10 sl=64 | p99 | 37,554 us | 34,272 us | 18,360 us | +9.6% | +104.5% |
| bs=100 sw=10 sl=64 | throughput | 803 tuples/sec | 839 tuples/sec | 988.31 tuples/sec | -4.3% | -18.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.49 MB/s | 0.512 MB/s | 0.603 MB/s | -4.3% | -18.8% |
| bs=100 sw=10 sl=64 | p50 | 123,262 us | 118,937 us | 101,066 us | +3.6% | +22.0% |
| bs=100 sw=10 sl=64 | p95 | 150,213 us | 138,713 us | 107,594 us | +8.3% | +39.6% |
| bs=100 sw=10 sl=64 | p99 | 150,213 us | 138,713 us | 115,830 us | +8.3% | +29.7% |
| bs=1000 sw=10 sl=64 | throughput | 911 tuples/sec | 910 tuples/sec | 1,019 tuples/sec | +0.1% | -10.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.556 MB/s | 0.555 MB/s | 0.622 MB/s | +0.2% | -10.6% |
| bs=1000 sw=10 sl=64 | p50 | 1,097,863 us | 1,095,991 us | 986,982 us | +0.2% | +11.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,164,793 us | 1,171,267 us | 1,028,491 us | -0.6% | +13.3% |
| bs=1000 sw=10 sl=64 | p99 | 1,164,793 us | 1,171,267 us | 1,058,493 us | -0.6% | +10.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,509.44,200,128000,393,0.240,23527.65,37554.15,37554.15
1,100,10,64,20,2490.84,2000,1280000,803,0.490,123261.91,150212.59,150212.59
2,1000,10,64,20,21962.13,20000,12800000,911,0.556,1097862.58,1164793.17,1164793.17
juliethecao
force-pushed
the
hf/08-result-media
branch
from
1d05e38 to
339130b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Texera’s HuggingFace operator experience by (1) detecting/rendering media-like outputs in the workflow result UI and (2) adding/expanding HuggingFace property-editor support (task/model selection plus image/audio inputs). It also includes backend HuggingFace codegen additions and a small schema/output-column handling adjustment.
Changes:
- Add frontend media-type detection helpers (+ unit tests) and use them in the result table and row-detail modal.
- Add HuggingFace Formly field components (model selector, image upload, audio upload) and task preview/visibility logic in the property editor.
- Add backend HuggingFace task codegen objects (image/audio/media/QA) and adjust output-schema/result-column handling.
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| frontend/src/app/workspace/service/workflow-graph/model/mock-workflow-data.ts | Adds a mock HuggingFace operator predicate for frontend tests/mocks. |
| frontend/src/app/workspace/service/operator-metadata/mock-operator-metadata.data.ts | Adds a mock HuggingFace operator schema and registers it in the mock schema list. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.ts | Adds is*Cell helpers delegating to media-type util. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.spec.ts | Adds tests around media URL detection for result cells and util consistency. |
| frontend/src/app/workspace/component/result-panel/result-table-frame/result-table-frame.component.html | Updates table cell rendering to show media placeholders/icons based on detected type. |
| frontend/src/app/workspace/component/result-panel/result-panel-model.component.scss | Adds modal/row-detail styling for media display and copy actions. |
| frontend/src/app/workspace/component/result-panel/result-panel-modal.component.ts | Builds per-field row entries with media metadata + copy-to-clipboard + media proxy URL resolution. |
| frontend/src/app/workspace/component/result-panel/result-panel-modal.component.html | Renders video/audio/image inline in the modal with copy fallback for text. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.ts | Adds HuggingFace task preview samples, task-based field visibility/validators, and Formly mapping adjustments. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.spec.ts | Adds tests for HuggingFace task preview behavior (known/unknown/empty/non-HF). |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.scss | Adds styling for the HuggingFace task preview card. |
| frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.html | Renders the HuggingFace task preview card above the Formly form. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.ts | Introduces a HuggingFace model/task selector Formly field with caching, pagination, and search. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.spec.ts | Adds basic unit tests for exported constants/cache invalidation. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.scss | Adds styles for the HuggingFace model select UI. |
| frontend/src/app/workspace/component/hugging-face/hugging-face.component.html | Adds the HuggingFace task dropdown, search, model list, and pagination UI. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.ts | Adds client-side image compression + data-URL storage for HF image inputs. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.spec.ts | Adds unit tests for image upload component state/error paths. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.scss | Styles the image upload and preview UI. |
| frontend/src/app/workspace/component/hugging-face-image-upload/hugging-face-image-upload.component.html | Adds template for selecting/previewing/clearing uploaded images. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.ts | Adds backend-stream upload + preview for HF audio inputs. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.spec.ts | Adds a minimal spec scaffolding for the audio upload component. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.scss | Styles the audio upload and preview UI. |
| frontend/src/app/workspace/component/hugging-face-audio-upload/hugging-face-audio-upload.component.html | Adds template for selecting/uploading/previewing/clearing audio. |
| frontend/src/app/common/util/media-type.util.ts | Adds isImageUrl / isAudioUrl / isVideoUrl helpers. |
| frontend/src/app/common/util/media-type.util.spec.ts | Adds unit tests covering detection across prefixes/extensions/query strings/CDN URLs. |
| frontend/src/app/common/formly/formly-config.ts | Registers new HuggingFace Formly field types. |
| frontend/src/app/app.module.ts | Adds HuggingFace components to the app module declarations. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceInferenceOpDesc.scala | Adjusts result-column fallback logic in codegen and output schema. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/QaRankingCodegen.scala | Adds codegen for QA/zero-shot/similarity/ranking task family. |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/MediaGenCodegen.scala | Adds codegen for prompt-driven media generation tasks (text-to-image/video). |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/ImageTaskCodegen.scala | Adds codegen for image task family (image-only and image+prompt). |
| common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/codegen/AudioTaskCodegen.scala | Adds codegen for audio task family (ASR/audio-classification/TTS). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Automated Reviewer SuggestionsBased on the
|
juliethecao
force-pushed
the
hf/08-result-media
branch
from
b83d948 to
1eeba34
Compare
juliethecao
force-pushed
the
hf/08-result-media
branch
from
de3f53d to
fa6cfd0
Compare
xuang7
reviewed
Jun 28, 2026
xuang7
left a comment
xuang7
left a comment
Contributor
There was a problem hiding this comment.
Left one comment. Could you rebase onto the latest main and also increase the test coverage, since this change is relatively large?
…nent Register the huggingface formly field type and declare HuggingFaceComponent in AppModule. Provides a task dropdown, paginated model list with client-side search, and per-task field state preservation when switching tasks.
The rxjs/no-implicit-any-catch ESLint rule requires explicit type annotations on error callbacks in .subscribe() calls. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mponent Satisfies the rxjs-angular/prefer-takeuntil lint rule. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mponent Fix setInterval/setTimeout leaks by tracking timer IDs and clearing them in ngOnDestroy. Remove takeUntil(destroy$) from shared module-level task fetch to prevent cache poisoning when the initiating component is destroyed. Remove unused TASK_TAG_MAP and TASK_NAMES exports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…FaceComponent inFlightByTag was written but never read, so multiple component instances mounting for the same uncached task would each fire a full backend request. Add an in-flight guard that polls for the existing request's completion, matching the pattern already used for task fetches. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… in HuggingFaceComponent
Read X-Texera-Truncated header via { observe: "response" } to detect
when the backend's model list is incomplete. Show a notice prompting
users to search. When truncated, search queries are sent to the backend
search endpoint with debounce; otherwise local filtering is used.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r lint rule Add back destroy$ and takeUntil(this.destroy$) on all subscribe calls. For the shared task fetch, add finalize() to reset tasksFetchSubscription when takeUntil fires before next/error, preventing the stale-guard bug the reviewer originally flagged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…anup Mirror the defensive pattern from the tasks fetch so that when takeUntil(destroy$) cancels mid-flight, inFlightByTag is cleared and subsequent component instances can re-fetch instead of polling forever. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Elliot Lin <36275109+ELin2025@users.noreply.github.com>
- Fix model polling indefinite loop by detecting cancelled in-flight fetch - Fix CSS class mismatch: hf-tasks-error → hf-error to match SCSS - Remove unused imports in spec file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Catch HTTP errors inside switchMap so search stream survives failures - Use empty string instead of null when clearing model selection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 38 integration tests covering task loading, model loading, pagination, search, task selection, and model selection logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Register the huggingface-audio-upload formly field type and declare HuggingFaceAudioUploadComponent in AppModule. Handles server-side audio storage via the /huggingface/upload-audio endpoint with local preview. Co-Authored-By: Anish Shivamurthy <anish@uci.edu>
…iew tests Add tests for sentencesColumn, inputImageColumn, inputAudioColumn visibility, validator edge cases with column fallbacks, and additional task preview kinds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Plugs the 9-task image family into the dispatcher pattern established
in PR 2:
image-only image-classification, object-detection,
image-segmentation, image-to-text
image + prompt visual-question-answering, document-question-answering,
zero-shot-image-classification, image-text-to-text,
image-to-image
- ImageTaskCodegen supplies payload + parse Python for all 9 tasks
- TaskCodegen trait gains a `tasks: Set[String]` default method so a
single codegen can register under multiple task strings; the
dispatcher map in HuggingFaceInferenceOpDesc is built from
registeredCodegens.tasks.flatMap(...)
- CodegenContext extended with imageInput + inputImageColumn
(EncodableString)
- HuggingFaceInferenceOpDesc gains 2 new @JsonProperty fields and
registers ImageTaskCodegen
PythonCodegenBase grows to host the shared image infrastructure:
- image_only_tasks / image_prompt_tasks / image_tasks tuples and
image_headers in process_table
- per-row image bytes resolution from upload (self._read_image_input)
or input column (self._read_binary_value + self._compress_image_bytes)
- use_raw_binary_body / raw_binary_headers state threaded through
_post_with_fallback (signature extended)
- _post_with_fallback adds the image-text-to-text chat-completions
branch and the model-author vision branch
- _call_provider adds branches for zai-org's custom API, Replicate
predictions + polling, Fal-ai, Wavespeed submit+poll, and image
embedding in OpenAI-compatible / unknown-provider fallbacks
- image-content-type response handling returns data:image URLs
- image helpers added: _read_image_input, _compress_image_bytes,
_image_input_as_base64, _read_binary_value, _looks_like_html,
_html_to_image_bytes, _extract_json_arg, _url_to_data_url
User-input strings continue to flow through pyb"..." + EncodableString
so they reach Python as self.decode_python_template('<base64>') rather
than raw literals. PythonCodeRawInvalidTextSpec still passes
(117/117 descriptors py_compile cleanly).
Frontend integration adds only the HF lines (no agent / dataset
noise from the source branch):
- HuggingFaceImageUploadComponent declared in app.module.ts
- huggingface-image-upload formly type registered in formly-config.ts
- Image upload component .ts/.html/.scss cherry-picked from huggingFace
- HuggingFace.png + sample-image.png assets
PR 3 of a stacked 9-PR series. Stacks on hf/02-operator-textgen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d media proxy Introduces a new Jersey REST resource exposing endpoints used by the upcoming HuggingFace operator UI: - GET /api/huggingface/models — browse / search models per task - GET /api/huggingface/tasks — list HF pipeline tags with hosted inference - POST /api/huggingface/upload-audio — upload audio for HF audio tasks - GET /api/huggingface/audio-preview — stream uploaded audio (path-validated) - GET /api/huggingface/media-proxy — proxy remote media URLs to bypass CORS This is the first PR in a stacked series landing the HF operator end-to-end. No operator code yet; this resource is independently useful and lets the frontend integrate with HF before the operator class lands.
Addresses xuang7's review on PR apache#5124 — both endpoints previously buffered the full payload into a heap-resident byte[] with no upper bound, leaving the JVM open to OOM on a hostile or buggy upstream response (/media-proxy) or out-of-band write into the audio temp dir (/audio-preview). - /media-proxy: switch from Unirest.asBytes() to asObject(Function<RawResponse, T>), streaming the upstream body in 8 KiB chunks with a running byte counter. Aborts with 413 if the declared Content-Length exceeds the cap (pre-check) or if the body crosses the cap mid-read (defends against missing/lying Content-Length). New MAX_MEDIA_PROXY_BYTES = 50 MiB, sized for HF inference media (text-to-image ~5 MiB, text-to-video ~30 MiB) with headroom. - /audio-preview: add Files.size() defense-in-depth check before readAllBytes. /upload-audio already enforces MAX_AUDIO_BYTES on ingest; this catches the case where a bug or out-of-band write puts an oversized file in the temp dir. Adds a spec covering the audio-preview cap using a sparse-file fixture so the test stays fast (87/87 spec passes). The media-proxy cap path is exercised via the existing input-validation suite plus the new streamMediaWithCap helper - a follow-up can add a fake-RawResponse unit test if reviewers want explicit coverage of the chunked-read cap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review on apache#5124 (xuang7, Ma77Ball): mark the resource with @RolesAllowed(Array("REGULAR", "ADMIN")) to document that all five endpoints require an authenticated user. The annotation isn't enforced yet — that's coming with the auth-enforcement PR @Yicong-Huang and @Ma77Ball are working on — but adding it now means no follow-up change is needed when enforcement lands, and it matches the convention used by UserConfigResource / AdminSettingsResource. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… panel Add isImageUrl/isAudioUrl/isVideoUrl detection helpers and wire them into the result table and row detail modal so image, audio, and video outputs from HuggingFace tasks render inline instead of as raw URLs. Gate backend string truncation on output mode so HF data URLs are never cut off.
…media-type spec Vitest does not support Jasmine-style toBeTrue() and toBeFalse() matchers.
… headers and body Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Julie Cao <116243642+juliethecao@users.noreply.github.com>
…sult panel row mapping
…y undefined, and use ogv for video extension
…gingFace component
juliethecao
force-pushed
the
hf/08-result-media
branch
from
7cecdc0 to
5634a94
Compare
…nel components Add 55 new tests across 5 spec files to cover previously untested branches: - media-type.util: non-string type guard (3 partials) - result-panel-modal: prettyRowJson, copyText, trackByEntryKey, non-string row values, null-tuple guard, rowData constructor path - hugging-face-audio-upload: localPreviewUrl in previewSrc, stale upload response/error guards, ngOnDestroy blob URL revocation - hugging-face: cached task/model error paths, in-flight poll paths (inFlightByTag and tasksFetchSubscription branches), destroy cleanup - result-table-frame: template rendering for video, audio, image, and plain-text cells with real row data
Signed-off-by: Julie Cao <116243642+juliethecao@users.noreply.github.com>
Contributor
Author
|
@xuang7 Done. I have addressed the review comment, added test cases, and rebased/resolved conflicts. I will check back here again once the Codecov report gets officially updated to see if I need to make more test cases as needed. |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… into hf/08-result-media
Signed-off-by: Julie Cao <116243642+juliethecao@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Render image, audio, and video outputs from HuggingFace tasks inline in the workflow result panel instead of displaying raw data URLs as text.
New file —
media-type.util.ts:isImageUrl— detectsdata:image/data URLs and common image extensions (.png,.jpg,.jpeg,.gif,.webp)isAudioUrl— detectsdata:audio/data URLs and common audio extensions (.mp3,.wav,.ogg,.m4a,.flac)isVideoUrl— detectsdata:video/data URLs, common video extensions (.mp4,.webm,.ogg), and thefal.mediaCDN host used by fal.ai text-to-video outputsChanges to
result-table-frame.component.{ts,html}:isImageCell/isAudioCell/isVideoCellmethods that delegate to the detection helpers<img>,<audio controls>,<video controls>elements render inline in the row detail modalChanges to
result-panel-modal.component.{ts,html}:rowEntrieswith per-field media metadata on modal openChanges to
result-panel-model.component.scss:New file —
media-type.util.spec.ts:fal.mediaCDN URL, cross-type rejection, and empty/plain-string inputsAny related issues, documentation, discussions?
How was this PR tested?
Unit tests in
media-type.util.spec.ts(19 cases) andresult-table-frame.component.spec.ts. Run withng test.Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Sonnet 4.6 in compliance with ASF guidelines