Bug: Fix audio file with cover image treated as video file by Rushi1109 · Pull Request #91 · omeryusufyagci/fast-music-remover

Rushi1109 · 2024-12-23T19:09:02Z

fixes the bug raised in #90

The ffprobe tool is used to check if any stream contains codec_name as mjpeg.

If mjpeg is present, the media is treated as Audio, other media is treated as Video

… audio

omeryusufyagci · 2024-12-24T09:49:44Z

Hey @Rushi1109, thanks for the PR. Frankly, I wasn't familiar with the full output set of ffprobe, but from what I see they provide disposition.attached_pic as a field in itself. Could you please get the json output which should give you a list of streams, with each stream providing you this field. From what I briefly checked, that should literally be what we're looking for.

Thanks for checking!

Rushi1109 · 2024-12-24T14:43:25Z

Hey @omeryusufyagci,

When looking at the raw data of the media I see DISPOSITION:attached_pic = 1 for stream index 2 which is the video stream.

But, When I tried to get the information about disposition.attached_pic from ffprobe tool, I am not able to get it's value. I tried extracting from json but still it's not showing attached_pic for that file. Do you have any idea about why it's getting filtered out, by ffmpeg?

Rushi1109 · 2024-12-24T14:46:57Z

Moreover @omeryusufyagci,

I have researched a bit and got to know that If the video stream contains a static image, the average frame rate for that stream will be 0/0. My current implementation is using the avg_frame_rate from video stream and decides the media type.

Do you see any issue with this approach?

omeryusufyagci · 2024-12-25T13:25:20Z

Hey @omeryusufyagci,

When looking at the raw data of the media I see DISPOSITION:attached_pic = 1 for stream index 2 which is the video stream.

But, When I tried to get the information about disposition.attached_pic from ffprobe tool, I am not able to get it's value. I tried extracting from json but still it's not showing attached_pic for that file. Do you have any idea about why it's getting filtered out, by ffmpeg?

Please make sure you include disposition after the -show_entries flag, e.g. -show_entries stream=index,<other_stuff>,disposition.

Could you try like that, and if there's still an issue please provide the command you used and the output you received.

I have researched a bit and got to know that If the video stream contains a static image, the average frame rate for that stream will be 0/0. My current implementation is using the avg_frame_rate from video stream and decides the media type.

Do you see any issue with this approach?

Thanks for figuring this out! Good to know that'd be a valid fix. The disposition.attached_image would be a more robust solution, as it explicitly identifies cover images and aligns better with the intent of filtering out such streams.

Could you please verify this and let me know if it resolves the issue? Thank you!

Rushi1109 · 2024-12-25T16:57:03Z

Please make sure you include disposition after the -show_entries flag, e.g. -show_entries stream=index,<other_stuff>,disposition

Yes I included disposition. I am using the same command of ffprobe but instead of avg_frame_rate, I added disposition and removed that awk command. But, I am not seeing any output for disposition.

Can you check it on your side? Do you see disposition in output?

omeryusufyagci · 2024-12-29T12:11:09Z

Please make sure you include disposition after the -show_entries flag, e.g. -show_entries stream=index,<other_stuff>,disposition

Yes I included disposition. I am using the same command of ffprobe but instead of avg_frame_rate, I added disposition and removed that awk command. But, I am not seeing any output for disposition.

Can you check it on your side? Do you see disposition in output?

HI @Rushi1109, you're right. I was able to reproduce the same behavior. It depends on what's bundled with the metadata, and we won't be able to count on the existence of that field. That means we'll have to check for this ourselves.

You're also correct, that a cover image is reported as a video stream that has 0/0 avg frame rate. It may also be reported as 'Cover' in a comment field, but that again is dependent on the metadata, whereas the 0/0 avg frame rate seems to be always available.

In this case, I propose to get the same ffprobe output as json (fyi, we're already using nlohmann/json in the project), parse the avg frame rate into a vector, and check if a given file has only 1 video stream and that video stream has 0/0 avg frame rate. Meaning, as soon as we iterate through a second video stream, we should conclude our check to avoid wasting time there.

Could you update this PR to implement this? Thanks

Rushi1109 · 2024-12-30T02:18:28Z

HI @Rushi1109, you're right. I was able to reproduce the same behavior. It depends on what's bundled with the metadata, and we won't be able to count on the existence of that field. That means we'll have to check for this ourselves.

You're also correct, that a cover image is reported as a video stream that has 0/0 avg frame rate. It may also be reported as 'Cover' in a comment field, but that again is dependent on the metadata, whereas the 0/0 avg frame rate seems to be always available.

In this case, I propose to get the same ffprobe output as json (fyi, we're already using nlohmann/json in the project), parse the avg frame rate into a vector, and check if a given file has only 1 video stream and that video stream has 0/0 avg frame rate. Meaning, as soon as we iterate through a second video stream, we should conclude our check to avoid wasting time there.

Could you update this PR to implement this? Thanks

Sure. I'll look into that today.

Rushi1109 · 2024-12-31T06:06:18Z

Hello @omeryusufyagci,

I have made required changes. I have tested it manually and it is working as expected.

Can you test it on your side and let me know if any change is required?

omeryusufyagci

Thanks for the changes @Rushi1109 !

Here's some comments for you:

About the parseFrameRate() addition:
The MediaProcessor::Utils NS is normally meant for generic utility. In this case, the frame rate helper is intrinsically coupled with getMediaType(). So, we should move this under the Engine class as a private member function. Also, for expressiveness, let's rename it to hasZeroFrameRate().

Regarding the implementation of this helper, I believe we can make it simpler. We want to know if an otherwise audio file has a video stream that has exactly "0/0" for the avg frame rate. So, we can simply and do a full string comparison, instead of parsing each char in the field. Pass a const ref of the stream object, and it can be as simple as a 1-line return statement that checks if it matches the expected case. We should however, make sure to cover the cases where the field is missing or invalid.

Updates on getMediaType():
Good start on this with the json parsing, but we can also simplify this similar to above. We should also encapsulate the logic where we're determining if we have a valid video stream. Otherwise, it makes it needlessly difficult understand what this function does at a glance.

Here's roughly what I have in mind:

for (const auto& stream : streams) {
    if (hasValidVideoStream(stream)) {
        return MediaType::Video;
    }
    if (hasValidAudioStream(stream)) {
        hasAudioStream = true; // define this flag false before the loop
    }
}

// if we haven't returned for Video and have found a valid audio stream
if (hasAudioStream) {
    return MediaType::Audio;
}

// otherwise
return MediaType::Unsupported;
}

So please encapsulate both checks (hasValid...) into individual functions. I realize the one for audio will likely be a 1-line return statement, but I'd still prefer to have it for the sake of consistency.
Within the hasValidVideoStream(), you'd call hasZeroFrameRate() with a const ref of the json stream for the avg frame rate field.
The other 2 helpers should also be called with const refs of the stream and all 3 new functions should be const themselves too.

This approach should make it easier to understand what getMediaType() does, and in general simplify the deduction logic to see if we have a cover image. Could you please update this PR accordingly?

Please let me know if you have any questions along the way. Many thanks!

Rushi1109 · 2025-01-02T16:29:00Z

Updated the PR.

omeryusufyagci

Hi @Rushi1109, thanks for the updates. Definitely moving in the right direction! Could you please take a look and update the PR? Thanks!

omeryusufyagci · 2025-01-05T17:04:09Z

+    /**
+     * @brief Checks if the provided stream is valid video stream
+     * 
+     * @param stream json data of stream
+     *
+     * @return True if the stream is valid. False otherwise.
+     */
+    bool hasValidVideoStream(const nlohmann::json& stream) const;


What does valid mean here? We should describe what exactly that means here

omeryusufyagci · 2025-01-14T11:23:40Z

+    nlohmann::json streamData = nlohmann::json::parse(*output);
+
+    bool containsAudioStream;
+    for (const auto& stream : streamData["streams"]) {
+        if (hasValidVideoStream(stream)) {
+            return MediaType::Video;
+        } 
+        if (hasValidAudioStream(stream)) {
+            containsAudioStream = true;
+        }
+    }
+
+    if (containsAudioStream) {
        return MediaType::Audio;
-    } else {
-        throw std::runtime_error("Unsupported media type detected.");
    }
+
+    return MediaType::Unsupported;
+}


Since the helpers are called hasValid...Stream it could be nicer to pass the entire stream and handle the iterations within the helpers. In doing so, I feel there won't be a need for the audio helper flag containsAudioStream.
You can check if we have a valid video stream and return Video, otherwise check for audio and return Audio, otherwise return Unsupported.

This would make getMediaType very easy to reason about.

Since the helpers are called hasValid...Stream it could be nicer to pass the entire stream and handle the iterations within the helpers. In doing so, I feel there won't be a need for the audio helper flag containsAudioStream.
You can check if we have a valid video stream and return Video, otherwise check for audio and return Audio, otherwise return Unsupported.

Done. I have made the changes. Can you please review it?

omeryusufyagci · 2025-01-15T17:27:36Z

+        if (stream["codec_type"] != "video") {
+            continue;
+        }
+        if(!stream.contains("avg_frame_rate")) {
+            continue;
+        }


It would be better to check for the inverse condition and return false, instead of continuing for the expected case. The earlier implementation in getMediaType was like that already.

omeryusufyagci · 2025-01-15T17:33:58Z

+    /**
+     * @brief Checks if the streams contains valid video stream and not static media (i.e. cover image)
+     * 
+     * @param streamData json data of all the streams
+     *
+     * @return True if the stream is valid video stream. False otherwise.
+     */
+    bool hasValidVideoStream(const nlohmann::json& streamData) const;
+
+    /**
+     * @brief Checks if the streams contains valid audio stream
+     * 
+     * @param streamData json data of all the streams
+     *
+     * @return True if the stream is valid. False otherwise.
+     */
+    bool hasValidAudioStream(const nlohmann::json& streamData) const;
+
+    /**
+     * @brief Checks if the stream's avg_frame_rate is 0/0
+     * 
+     * @param frameRate the avg_frame_rate field from stream
+     *
+     * @return True if avg_frame_rate equals to 0/0. False otherwise.
+     */
+    bool hasZeroFrameRate(const std::string& frameRate) const;


So far we have not included [at]params for function parameters in the project. Let's keep it that way, and eventually we'll likely do a review of our commenting practices in the public APIs. Anyway, for private member functions, it's not necessary to provide docs in the header file.

Also, my previous remark on what valid stands for, for the 2 helpers remain open. I would add a line in the description, i.e. after the brief to define what valid means in our context.

Rushi1109 added 2 commits December 24, 2024 00:27

check if the stream contains mjpeg as codec_name, then treat media as…

8fd08bd

… audio

fix typo in codec_name

2408253

Rushi1109 mentioned this pull request Dec 23, 2024

Bug: Audio file treated as a video file #90

Open

used avg_frame_rate of video stream to detect if it is a static image

0fcceb5

omeryusufyagci added bug Something isn't working core Media processing component, the core C++ binary. iteration Requires further iteration for approval labels Dec 24, 2024

Use avg_frame_rate from json to validate media type

3997b5c

Rushi1109 changed the title ~~check if the stream contains mjpeg as codec_name, then treat media as audio~~ Bug: Fix audio file with cover image treated as video file Dec 31, 2024

omeryusufyagci requested changes Jan 1, 2025

View reviewed changes

Rushi1109 changed the base branch from main to 90-bug-audio-file-treated-as-a-video-file January 2, 2025 15:43

Rushi1109 added 2 commits January 2, 2025 21:26

Refactor getMediaType function

ae5db3f

remove throw call

5462260

Rushi1109 requested a review from omeryusufyagci January 2, 2025 16:28

omeryusufyagci requested changes Jan 5, 2025

View reviewed changes

Addressed review changes

1d3199d

Rushi1109 requested a review from omeryusufyagci January 6, 2025 05:40

Renamed function

a48a184

omeryusufyagci reviewed Jan 14, 2025

View reviewed changes

Pass whole stream data to helper functions

7fa49ca

Rushi1109 requested a review from omeryusufyagci January 14, 2025 19:45

omeryusufyagci reviewed Jan 15, 2025

View reviewed changes

removed continue statements

10bea94

omeryusufyagci approved these changes Jan 15, 2025

View reviewed changes

omeryusufyagci merged commit 2d6f448 into omeryusufyagci:90-bug-audio-file-treated-as-a-video-file Jan 15, 2025

Conversation

Rushi1109 commented Dec 23, 2024

Uh oh!

omeryusufyagci commented Dec 24, 2024

Uh oh!

Rushi1109 commented Dec 24, 2024

Uh oh!

Rushi1109 commented Dec 24, 2024

Uh oh!

omeryusufyagci commented Dec 25, 2024

Uh oh!

Rushi1109 commented Dec 25, 2024

Uh oh!

omeryusufyagci commented Dec 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rushi1109 commented Dec 30, 2024

Uh oh!

Rushi1109 commented Dec 31, 2024

Uh oh!

omeryusufyagci left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rushi1109 commented Jan 2, 2025

Uh oh!

omeryusufyagci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

omeryusufyagci Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

omeryusufyagci Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

Rushi1109 Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

omeryusufyagci Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

omeryusufyagci Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

omeryusufyagci commented Dec 29, 2024 •

edited

Loading

omeryusufyagci left a comment •

edited

Loading