Detailed explanation of mobile HTML5 audio and video problems and solutions

Recently, we are studying the use of video to replace animation, and the use of video to replace sprite animation. We call this kind of video interactive video.

1. Large disk space, slow downloading, especially online playback, will be even slower
2. There are too many files. When playing online, too many http requests will lead to slow response or abnormal behavior.

Therefore, there is an urgent need to develop a set of technologies to replace sprite animations with videos. We call this kind of video interactive video

1. Traditional videos can only be played in a square area
2. Traditional videos are played in a window on the iPad, but on the iPhone they can only be played in full screen.
3. Traditional videos will definitely appear at the front when playing.

1. Under iPhone, there is no need to play in full screen, you can play in a region
2. Interactive videos can appear underneath ordinary graphics objects
3. Interactive videos can be masked, which removes the background of the video and blends the video with ordinary graphic objects.

The research has initially yielded results. By the way, I will summarize the practical problems encountered in audio and video during the development of mobile H5 in the past few years and give my own solutions.

Take a look at the final actual effect: compatible with PC (>IE9), iPhone, iPad, Android 5.0

It solves the problems of manual, automatic, windowing, etc. on iPhone, and can basically be used in actual production.

On the right is the original video mp4 file

The video on the left replaces the animation and supports the background mask effect, which can reveal the base image and support a series of interactive operations.

Every time an audio object is passed through new Audio, you can see that a new thread will be generated on IOS. This is disgusting.

Solution: Use a new Audio object to achieve the purpose of not opening more threads by replacing different audio addresses.

Solution: The problem on lower versions of Android is not solved. Generally, in hybrid development, the underlying interface can be adjusted, such as phonegap.

Solution: Automatic playback on iPhone is a process done when IOS was designed. It seems to be to prevent automatic theft of traffic.

To put it simply, you need to simulate the user to trigger it manually, so we need to call this piece of code at the beginning:

This is from my project, I just deducted it.

 //Fix the problem that the ios browser cannot automatically play audio. Create a new audio when loading and replace the src when using it. Xut.fix = Xut.fix||{};if (Xut.plat.isBrowser && Xut.plat. isIOS) { var isAudio = false var fixaudio = function() { if (!isAudio) { isAudio = true; Xut.fix.audio = new Audio(); document.removeEventListener('touchstart', fixaudio, false); } }; document.addEventListener('touchstart', fixaudio, false);}

If you bind such a code to the body: create an audio object through manual triggering, and then save it in the global

When using it, it is as follows

 //If you use Xut.fix.audio to specify src initialization for ios browser, see app.jsif (Xut.fix.audio) { audio = Xut.fix.audio; audio.src = url;} else { audio = new Audio(url );}audio.autoplay = true;audio.play();

Just replace the audio object directly. To put it simply, if it is to be automatically created, it must be an object triggered by the user to play.

The video tag may be rarely used on mobile devices. Android support is terrible. It only improved in 5.0.

An old problem on the iPhone is that it cannot play automatically (save data, save your sister!!!), and the default is full-screen control playback

For a long time, I ignored this video processing. Android uses the bottom layer, and iPhone uses VideoJS directly. It has built-in flash and h5 switching. Flash also has support issues.

A while ago, my boss had a request. We have too many application animations, all of which are combination animations of sprite routes. One app can range from hundreds of megabytes to hundreds of megabytes.

So there is an urgent need for a solution to compress images

The final solution is to use video instead of animation, because video compression technology has been developed for many years and has become very mature. Current video compression technology can easily compress 720P high-definition movies to 10M/minute, or 160K/second. It is at least dozens of times smaller than the file size of the image sequence. At the same time, most devices support hardware decompression of videos. In this way, the CPU consumption of video playback is very low, the battery consumption is also very low, and the playback speed is still fast. Even full-screen playback at 25 frames can be easily achieved.

After the plan is finalized, a few problems need to be solved.

1. The entire video, including certain objects in the video, can respond to user clicks, slides and other operations.

2. Under iPhone, you can play in a window

3. Ability to filter out the background so it can be used like a PNG image

The final actual effect is also shown in the starting gif animation:

Video replaces animation and supports background masking effect, which can reveal the base image.

At the same time, it also solved the problem of manual, automatic and not full screen.

Solution:

Processed through canvas + video tag combination

Principle: Get the original frame of the video and draw it to the page through canavs

Here I directly attach the source code. The code is generally written, but several key points are highlighted.

http://stackoverflow.com/questions/3699552/html5-inline-video-on-iphone-vs-ipad-browser

This is a bit troublesome. You need to interact and drag the canvas to control the image. I haven't finished writing it all yet. General company needs will not have this. Here is a simple description. It is also processed by canvas + video, but it needs There is a cached canvas container for preprocessing. Through preprocessing, the pixels of each image are obtained. By changing the RBG value of each pixel, the background can be filtered out, so that it can be used like a PNG image. In the future It’s written, let’s publish it~~

The above is the entire content of this article. I hope it will be helpful to everyone’s study. I also hope everyone will support VeVb Wulin Network.