These all look interesting:
{{{
W/GeckoLinker(30125): /data/app/org.torproject.torbrowser_alpha-1/base.apk!/assets/armeabi-v7a/libmozavutil.so: Relocation to NULL @0x000293d0 for symbol "__cxa_type_match"
W/GeckoLinker(30125): /data/app/org.torproject.torbrowser_alpha-1/base.apk!/assets/armeabi-v7a/libmozavcodec.so: Relocation to NULL @0x00023360 for symbol "__cxa_type_match"
W/VideoCapabilities(30125): Unrecognized profile/level 32768/2 for video/mp4v-es
I/VideoCapabilities(30125): Unsupported profile 4 for video/mp4v-es
I/Gecko (30125): [30125, MediaPlayback #2 (closed)] WARNING: Decoder=91a31250 state=DECODING_METADATA Decode metadata failed, shutting down decoder: file /home/user/tor-browser/dom/media/MediaDecoderStateMachine.cpp, line 340
I/Gecko (30125): [30125, MediaPlayback #2 (closed)] WARNING: Decoder=91a31250 Decode error: NS_ERROR_DOM_MEDIA_CANCELED (0x806e000a): file /home/user/tor-browser/dom/media/MediaDecoderStateMachine.cpp, line 3118
}}}
Yes. I get the latter two, though, on my Linux box as well once I close the playing video. Thus, I suspect at the moment that is not related to this bug at least.
After some bisecting it turns out this is caused by our patching of proxy-bypass issues mentioned in #28125 (moved). In particular, the problem at hand stems from the exoplayer related patch.
03-25 15:50:40.313 527 1326 I ExoPlayerImpl: Init ExoPlayerLib/2.4.0 [kminilte, SM-G800F, samsung, 23]03-25 15:50:40.368 527 1333 I DefaultHttpDataSource: This is Tor Browser. Skipping.03-25 15:50:40.373 527 1333 I DefaultHttpDataSource: This is Tor Browser. Skipping.03-25 15:50:41.378 527 1333 I DefaultHttpDataSource: This is Tor Browser. Skipping.03-25 15:50:43.378 527 1333 I DefaultHttpDataSource: This is Tor Browser. Skipping.03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: Source error.03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: com.google.android.exoplayer2.upstream.HttpDataSource$HttpDataSourceException: Unable to connect to https://video.twimg.com/ext_tw_video/1108828787624411136/pu/pl/EagyvHO-kYyNBwEE.m3u8?tag=803-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:197)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DefaultDataSource.open(DefaultDataSource.java:123)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DataSourceInputStream.checkOpened(DataSourceInputStream.java:102)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DataSourceInputStream.open(DataSourceInputStream.java:65)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.ParsingLoadable.load(ParsingLoadable.java:114)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:295)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at java.lang.Thread.run(Thread.java:818)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: Caused by: java.io.IOException03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:400)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:361)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:195)03-25 15:50:43.398 527 1327 E ExoPlayerImplInternal: ... 8 more
Okay, I dug into this a little today. This is coming from Fennec's support for HLS (HTTP Live Streaming) via the vendored exoplayer2 library.
Three easy solutions I see:
Disable MOZ_ANDROID_HLS_SUPPORT (we can change the default value in mobile/android/moz.configure)
Change the returned value in dom/media/DecoderTraits.cpp::CanHandleMediaType() for HLSDecoder from CANPLAY_MAYBE -> CANPLAY_NO.
Disable the media.hls.enabled pref on Android.
These changes are not tested. I worry only disabling MOZ_ANDROID_HLS_SUPPORT will not solve the problem if one of the other conditional blocks in that function returns CANPLAY_MAYBE. Maybe one of the other changes will be enough (or a combination of these).
These don't solve the root cause where throwing an Exception within exoplayer2 (Java code) results in native Gecko freezing/crashing/etc. I suspect this is because Gecko is multi-threaded but one of the threads is "blocking" on a callback from exoplayer - however, this callback never occurs because the exception is thrown and exoplayer doesn't inform Gecko about this failure. exoplayer catches the exception internally and resets its own state. It's possible/likely the "blocking" behavior is more complicated, but I'm not familiar with the this area of the code or MediaDecoder{,StateMachine}.
Okay, I dug into this a little today. This is coming from Fennec's support for HLS (HTTP Live Streaming) via the vendored exoplayer2 library.
Three easy solutions I see:
Disable MOZ_ANDROID_HLS_SUPPORT (we can change the default value in mobile/android/moz.configure)
Change the returned value in dom/media/DecoderTraits.cpp::CanHandleMediaType() for HLSDecoder from CANPLAY_MAYBE -> CANPLAY_NO.
Disable the media.hls.enabled pref on Android.
I think there is no big difference between 1) and 3) given that HLSDecoder::IsSupportedType(aType) is doing return IsEnabled() && DecoderTraits::IsHttpLiveStreamingType(aContainerType); and IsEnabled() is checking the pref. The advantage of 1) is that we don't bundle the whole exoplayer stuff in the first place which would would give us an .apk which is slightly smaller. Actually, looking at: https://hg.mozilla.org/mozilla-central/rev/9545b9df0c6d it seems both was used to treat HLS support appropriately. We should do the same I guess.
Somewhat orthogonal: https://bugzilla.mozilla.org/show_bug.cgi?id=1371247 makes me itch + that the exoplayer shipped is rather old. I wonder whether there were some security vulnerabilities fixes meanwhile (I almost bet that there were).
That said: I am not sure I understand your concerns regarding 1) (be it with or without 3) ). Are you concerned the exoplayer library is used for something other than HLS as well and we would therefore run into the same problems? Or that we just discover other hangs/freezes because we run into similar issues when trying to use a different media handler?
Okay, I dug into this a little today. This is coming from Fennec's support for HLS (HTTP Live Streaming) via the vendored exoplayer2 library.
Three easy solutions I see:
Disable MOZ_ANDROID_HLS_SUPPORT (we can change the default value in mobile/android/moz.configure)
Change the returned value in dom/media/DecoderTraits.cpp::CanHandleMediaType() for HLSDecoder from CANPLAY_MAYBE -> CANPLAY_NO.
Disable the media.hls.enabled pref on Android.
I think there is no big difference between 1) and 3) given that HLSDecoder::IsSupportedType(aType) is doing return IsEnabled() && DecoderTraits::IsHttpLiveStreamingType(aContainerType); and IsEnabled() is checking the pref. The advantage of 1) is that we don't bundle the whole exoplayer stuff in the first place which would would give us an .apk which is slightly smaller. Actually, looking at: https://hg.mozilla.org/mozilla-central/rev/9545b9df0c6d it seems both was used to treat HLS support appropriately. We should do the same I guess.
Somewhat orthogonal: https://bugzilla.mozilla.org/show_bug.cgi?id=1371247 makes me itch + that the exoplayer shipped is rather old. I wonder whether there were some security vulnerabilities fixes meanwhile (I almost bet that there were).
That said: I am not sure I understand your concerns regarding 1) (be it with or without 3) ). Are you concerned the exoplayer library is used for something other than HLS as well and we would therefore run into the same problems? Or that we just discover other hangs/freezes because we run into similar issues when trying to use a different media handler?
My concern is the latter - that fennec tries handling the content with another handler. I don't know if this would happen, but when I skimmed the code it seems possible.
Somewhat orthogonal: https://bugzilla.mozilla.org/show_bug.cgi?id=1371247 makes me itch + that the exoplayer shipped is rather old. I wonder whether there were some security vulnerabilities fixes meanwhile (I almost bet that there were).
Also, so it's documented. It looks like the current shipped version is r2.4.0. Exoplayer is now at r2.9.6.
Replying to sysrqb:
That said: I am not sure I understand your concerns regarding 1) (be it with or without 3) ). Are you concerned the exoplayer library is used for something other than HLS as well and we would therefore run into the same problems? Or that we just discover other hangs/freezes because we run into similar issues when trying to use a different media handler?
My concern is the latter - that fennec tries handling the content with another handler. I don't know if this would happen, but when I skimmed the code it seems possible.
Yes. The videos I tested play without HLS on my mobile device (while freezing the browser with HLS support). Given that we have blocked HLS for a while now and not heard much back about frozen browsers I guess there are fallbacks active that might kick in or it's not as widespread an issue one would assume. Either way I think we are better of if we disable HLS completely for now (and don't ship a broken version) until we can properly support it.
Okay, so, if we really can't handle HLS, be it on desktop or mobile we get the options to download the m3u8 file and/or open it with an external app. One can test that behavior with
On mobile with 8.5a10 the symptoms of this bug are visible but with my patch one gets the desktop behavior of the external helper app dialog showing up. Now, why can videos on Twitter still be played?
If there is no HSL player available you'll see something like
[2386:Main Thread]: D/nsMediaElement 0x61a00026a080 CanPlayType(application/x-mpegURL) = ""[04-01 15:23:14] Torbutton INFO: tor SOCKS: https://abs.twimg.com/k/en/15.hls11.en.7d01861d350d623607c9.js via twitter.com:f6ce0dc9bc9687a8480179a1e5b8b7b7
in your logs. The first line is interesting because it shows that the browser is not falling back to something else via DecoderTraits::CanHandleMIMEType, because there is the empty string after the "=", which boils down to CANPLAY_NO (via HTMLMediaElement::GetCanPlay() either because !containerType or because DecoderTraits::CanHandleConteinerType gave it back (which itself called
CanHandleMediaType for the check)).
The second line is interesting because it shows that the HLS support is provided by Twitter via some JS file which explains why the content (both on desktop and mobile) is playing at all. How frequent that fallback mechanism is on the web I don't know but either way, I think we can live with the patch for now.
Once this is fixed I'll open a follow-up bug for properly enabling HLS support again.
Okay, so, if we really can't handle HLS, be it on desktop or mobile we get the options to download the m3u8 file and/or open it with an external app. One can test that behavior with
On mobile with 8.5a10 the symptoms of this bug are visible but with my patch one gets the desktop behavior of the external helper app dialog showing up.
The second line is interesting because it shows that the HLS support is provided by Twitter via some JS file which explains why the content (both on desktop and mobile) is playing at all. How frequent that fallback mechanism is on the web I don't know but either way, I think we can live with the patch for now.
I've heard this is popular with some of the popular video-streaming sites, but I don't know how popular it is in general. I suspect not-very-popular.
I get a few errors in the logcat while testing on an emulator running API level 19, but the end result is the video element displaying a message saying "The media could not be played" - so this is better than crashing. This is testing the twitter video in the description.
W/dalvikvm( 1989): dvmFindClassByName rejecting 'org/mozilla/gecko/media/CodecProxy$NativeCallbacks' W/dalvikvm( 1989): dvmFindClassByName rejecting 'org/mozilla/gecko/media/CodecProxy' I/ActivityManager( 1592): Start proc org.torproject.torbrowser_alpha:media for service org.torproject.torbrowser_alpha/org.mozilla.gecko.media.MediaManager: pid=2336 uid=10065 gids={50065, 3003, 1028, 1015}...E/OMXNodeInstance( 2336): OMX_GetExtensionIndex OMX.google.android.index.storeMetaDataInBuffers failed W/OMXNodeInstance( 2336): OMX_GetExtensionIndex OMX.google.android.index.prepareForAdaptivePlayback failed E/OMXNodeInstance( 2336): OMX_GetExtensionIndex OMX.google.android.index.storeMetaDataInBuffers failed E/ACodec ( 2336): [OMX.google.h264.decoder] storeMetaDataInBuffers failed w/ err -2147483648 I/SoftAAC2( 2336): Reconfiguring decoder: 0->48000 Hz, 0->2 channels E/SoftAVC ( 2336): Decoder failed: -2E/ACodec ( 2336): [OMX.google.h264.decoder] ERROR(0x80001001)E/MediaCodec( 2336): Codec reported an error. (omx error 0x80001001, internalError -2147483648)W/System.err( 2336): java.lang.IllegalStateExceptionW/System.err( 2336): java.lang.IllegalStateExceptionW/System.err( 2336): at android.media.MediaCodec.dequeueInputBuffer(Native Method)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec$BufferPoller.pollInputBuffer(JellyBeanAsyncCodec.java:216)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec$BufferPoller.handleMessageLocked(JellyBeanAsyncCodec.java:199)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec$CancelableHandler.handleMessage(JellyBeanAsyncCodec.java:58)W/System.err( 2336): at android.os.Handler.dispatchMessage(Handler.java:102)W/System.err( 2336): at android.os.Looper.loop(Looper.java:136)W/System.err( 2336): at android.os.HandlerThread.run(HandlerThread.java:61)E/GeckoAsyncCodecAPIv16( 2336): codec error:-10000W/System.err( 2336): at android.media.MediaCodec.queueInputBuffer(Native Method)W/System.err( 2336): at java.lang.Exception: codec error:-10000W/System.err( 2336): at org.mozilla.gecko.media.Codec$Callbacks.onError(Codec.java:48)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec.queueInputBuffer(JellyBeanAsyncCodec.java:371)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec$CallbackSender.handleMessageLocked(JellyBeanAsyncCodec.java:144)W/System.err( 2336): org.mozilla.gecko.media.Codec$InputProcessor.feedSampleToBuffer(Codec.java:167)W/System.err( 2336): at org.mozilla.gecko.media.Codec$InputProcessor.queueSample(Codec.java:107)W/System.err( 2336): at org.mozilla.gecko.media.Codec$InputProcessor.onSample(Codec.java:95)W/System.err( 2336): at org.mozilla.gecko.media.Codec$InputProcessor.access$1700(Codec.java:66)W/System.err( 2336): at org.mozilla.gecko.media.Codec.queueInput(Codec.java:553)W/System.err( 2336): at org.mozilla.gecko.media.ICodec$Stub.onTransact(ICodec.java:150)W/System.err( 2336): at at android.os.Binder.execTransact(Binder.java:404)W/System.err( 2336): at dalvik.system.NativeStart.run(Native Method)I/Gecko ( 1989): [1989, MediaPlayback #2] WARNING: Decoder=91bb2940 Decode error: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005) - void mozilla::JavaCallbacksSupport::OnError(bool): file /home/android/tor-browser/dom/media/MediaDecoderStateMachine.cpp, line 3118
With the Azure reference, when I go to the Azure HTML5 player (DASH) and press play the script basically hangs. I don't know what it's doing, but the browser shows the unresponsive-script dialog but the video player never plays the video, and I see a lot of:
I/Choreographer( 2503): Skipped 61 frames! The application may be doing too much work on its main thread.I/Choreographer( 1989): Skipped 79 frames! The application may be doing too much work on its main thread.I/Choreographer( 2503): Skipped 113 frames! The application may be doing too much work on its main thread.
04-10 20:31:23.570 2886 2910 W VideoCapabilities: Unrecognized profile 4 for video/hevc04-10 20:31:23.593 2886 2910 I VideoCapabilities: Unsupported profile 4 for video/mp4v-es...04-10 20:31:33.895 1848 1865 I ActivityManager: Start proc 3115:org.torproject.torbrowser_alpha:media/u0a376 for service org.torproject.torbrowser_alpha/org.mozilla.gecko.media.MediaManager...04-10 20:31:34.457 3115 3130 W VideoCapabilities: Unrecognized profile 4 for video/hevc 04-10 20:31:34.496 3115 3130 I VideoCapabilities: Unsupported profile 4 for video/mp4v-es 04-10 20:31:34.529 3115 3134 I OMXClient: IOmx service obtained 04-10 20:31:34.531 1718 2111 I OMXMaster: makeComponentInstance(OMX.google.aac.decoder) in omx@1.0-service process 04-10 20:31:34.552 3115 3134 I MediaCodec: MediaCodec will operate in async mode...04-10 20:31:34.902 1718 1718 W OMXNodeInstance: [0xedaa6c60:google.h264.decoder] component does not support metadata mode; using fallback...04-10 20:31:35.671 3115 3140 D SoftwareRenderer: setting dataspace on output surface to #103 04-10 20:31:36.096 1996 2241 I OpenGLRenderer: Davey! duration=737ms; Flags=0, IntendedVsync=247180479682, Vsync=247180479682, OldestInputEvent=9223372036854775807, NewestInputEvent=0, HandleInputStart=247189000051, AnimationStart=247189012334, PerformTraversalsStart=247189012745, DrawStart=247189869221, SyncQueued=247189906569, SyncStart=247245032677, IssueDrawCommandsStart=247245109814, SwapBuffers=247949524636, FrameCompleted=247973058364, DequeueBufferDuration=8072000, QueueBufferDuration=8073000, 04-10 20:31:36.280 2886 2886 D GeckoToolbar: onTabChanged: MEDIA_PLAYING_CHANGE 04-10 20:31:36.280 2886 2886 D GeckoBrowserApp: BrowserApp.onTabChanged: 2: MEDIA_PLAYING_CHANGE 04-10 20:31:36.280 2886 2886 D MediaControlService: onStateChanged, state = PLAYING
Okay, on older Android versions (Jellybean and earlier) Fennec uses a custom async wrapper around Android built-in MediaCodec system for handling media files/streams. On newer versions, Android provides an async MediaCodec library, and Fennec uses that directly. Therefore, it seems like this should work on all supported versions, but the error log above indicates this isn't the reality:
E/MediaCodec( 2336): Codec reported an error. (omx error 0x80001001, internalError -2147483648)W/System.err( 2336): java.lang.IllegalStateExceptionW/System.err( 2336): java.lang.IllegalStateExceptionW/System.err( 2336): at android.media.MediaCodec.dequeueInputBuffer(Native Method)W/System.err( 2336): at org.mozilla.gecko.media.JellyBeanAsyncCodec$BufferPoller.pollInputBuffer(JellyBeanAsyncCodec.java:216)
I'll dig into this a little more (and GeKo suggested testing a vanilla Fennec build, so I'll try that, too).
I opened #30139 (moved) for reenabling HLS support (with child tickets for the issues we have found so far (that is proxy bypass of DNS requests and old exoplayer library)).
Trac: Status: merge_ready to closed Resolution: N/Ato fixed