CN103888840A

CN103888840A - Method and device for dragging and zooming video mobile terminal in real time

Info

Publication number: CN103888840A
Application number: CN201410119164.6A
Authority: CN
Inventors: 田玲; 周益民; 罗光春; 闫科
Original assignee: University of Electronic Science and Technology of China
Current assignee: University of Electronic Science and Technology of China
Priority date: 2014-03-27
Filing date: 2014-03-27
Publication date: 2014-06-25
Anticipated expiration: 2034-03-27
Also published as: CN103888840B

Abstract

The invention discloses a method and device for dragging and zooming a video mobile terminal in real time. The device mainly comprises a decoding module, a drawing module and an interaction module. The decoding module comprises a decoding submodule, a decoding control submodule, a conversion submodule and double registers. The drawing module comprises an extraction submodule, a filling submodule and a drawing control submodule. The interaction module comprises a monitoring submodule, a computing submodule, a request diverter, a zooming adapter and a dragging adapter. The method and device solve the problems existing in video playing of a mobile terminal in the prior art, the implementation process is simple, and efficiency is high. Meanwhile, a method for detecting the range of a video source going beyond a display area after dragging or zooming is provided at the same time, and operation to video playing at the mobile terminal is effectively facilitated.

Description

Method and device for dragging and zooming video mobile terminal in real time

Technical Field

The invention belongs to the technical field of video processing, and particularly relates to a method and a device for dragging and zooming a video mobile terminal in real time.

Background

The development of information technology is gradually changing along with the wide popularization of internet, the era of mobile internet has come, 3G networks are now widely applied to the lives of people, at present, more than 20 countries in the world deploy 4G networks, and more terminal devices support the 4G networks. Video related services are also transferred from traditional television terminals, PC terminals and the like to mobile terminals, and the mobile terminals have high client alignment viscosity and dependence due to portability, so that the internet is moved in the future. Users have become accustomed to using portable devices to access the internet for enjoying services anytime and anywhere, and the continuous innovation of high compression ratio coding standards, such as MPEG4, h.264/AVC and HEVC, makes low bit rate and high quality video playback no longer difficult, which accelerates the arrival of the "dynamic exhibition" era. The traditional internet video has the characteristics of high resolution, large display size and high resolution, and the mobile terminal has portability, is generally small in size and limited in size of a display screen, so that a user can expect to obtain complete information and detailed information of the video. The mobile terminal device has the characteristics that the desktop device does not have, so that the technology of dragging and zooming the video on the mobile terminal is particularly important and indispensable.

The mobile terminal has a limited display data amount due to the small size of the display screen, and displayed information is incomplete when the current video resources on the internet are directly transferred to the mobile terminal. For real-time video playing, the same video resources are played on various devices at the same time, characteristics of playing ends are different, particularly diversity of the mobile terminal is large, differences of display size and resolution are large, and users have strong requirements for real-time performance, so that urgent user requirements and great social application value are provided for researching real-time processing of videos of the mobile terminal. The most common operations of the mobile terminal are dragging a selection display area and zooming display detail information.

Disclosure of Invention

The invention aims to provide a method and a device for dragging and zooming a video mobile terminal in real time in order to solve the problem that videos are played on the mobile terminal in the prior art.

The technical scheme of the invention is as follows: a method for dragging and zooming a video mobile terminal in real time specifically comprises the following steps: the method comprises the following steps of decoding a thread, a drawing thread and an interaction thread, wherein the decoding thread, the drawing thread and the interaction thread run in parallel;

the thread decoding method comprises the following specific steps:

s11, after the video stream reaches the mobile terminal, decoding the video stream to obtain an original data frame;

s12, collecting the scaling parameters generated by the interactive threads and controlling the scaling ratio of the video;

s13, converting the video format, and finally converting the video format into an RGB intermediate frame of an ARGB8888 standard in an RGB format;

s14, storing the converted RGB intermediate frame into a double buffer;

s15, waking up the drawing thread after frame synchronization, and simultaneously entering the decoding of the next frame of video;

the thread drawing method comprises the following specific steps:

s21, starting a drawing thread after receiving a wake-up signal of the decoding thread;

s22, collecting the positioning parameters generated by the interactive thread, and controlling the extraction area of the video image;

s23, extracting data of a part to be displayed from the double buffers;

s24, writing the data to be displayed extracted in the step S23 into the area to be displayed in the area to be displayed;

s25, entering a dormant state after the data filling is finished until a wake-up signal of the decoding thread is received again, and entering the drawing of the next frame;

the specific steps of the interaction thread are as follows:

s31, starting an interaction thread, and monitoring an interaction event in a display window, wherein the interaction event comprises: single-finger dragging and double-finger zooming;

s32, calculating and analyzing the interaction events in the display window, and judging whether the requirements of the user change; if yes, the step S33 is entered, otherwise, the step S31 is returned to continue monitoring the interaction events in the display window;

s33, calculating a zooming component and a dragging component;

s34, performing coordinate conversion calculation according to the zooming components to obtain zooming ratio, generating zooming parameters, and transmitting the zooming parameters to the decoding thread;

and S35, performing coordinate conversion calculation according to the dragging component to obtain the dragging displacement, generating a positioning parameter, and transmitting the positioning parameter to the drawing thread.

Further, the specific process of generating the scaling parameter in step S34 is as follows:

before zooming, the finger touch points are respectively Et and Ft, and after zooming, the finger touch points are respectively E_t+1And F_t+1，S_tIn order to extract the start of the data area,

is E_tRelative to S_tThe coordinates of the points are such that,is F_tRelative to S_tThe coordinates of the points are such that,

is E_t+1Relative to S_tThe coordinates of the points are such that,

is F_t+1Relative to S_tCoordinates of points, W_v、H_vWidth and height, W, respectively, of the extracted data area_t、H_tRespectively the width and height of RGB intermediate frame at t moment before scaling operation, and W after scaling operation_t、H_tWill become smaller in geometric proportion, W_v、H_vThe change is not changed;

selecting

Calculating a zoom ratio Z in the extracted data region as a ratio of the extracted data region to the RGB intermediate frame size_v，

Z_{v} = \frac{| | E_{t} F_{t} | |}{| | E_{t + 1} F_{t + 1} | |},

Wherein,

| | E_{t} F_{t} | | = \sqrt{{(x^{S_{t}} |_{E_{t}} - x^{S_{t}} |_{F_{t}})}^{2} + {(y^{S_{t}} |_{E_{t}} - y^{S_{t}} |_{F_{t}})}^{2}},

| | E_{t + 1} F_{t + 1} | | = \sqrt{{(x^{S_{t}} |_{E_{t + 1}} - x^{S_{t}} |_{F_{t + 1}})}^{2} + {(y^{S_{t}} |_{E_{t} + 1} - y^{S_{t}} |_{F_{t} + 1})}^{2}},

scaling ratio of RGB inter frame size

Z_{R} = Z_{v} / (\frac{H_{v}}{H_{t}});

W after performing a zoom operation_t+1、H_t+1Respectively as follows:

further, the specific process of generating the positioning parameter in step S35 is as follows:

the finger touch point before the drag operation is set to A_tThe finger touch point after the drag operation is A_t+1Point D is the starting point of the RGB intermediate frame, point S_tAnd S_t+1Respectively as a starting point for extracting data before and after the drag operation,

is marked as A_tDot relative to S_tThe coordinates of the points are determined by the coordinates of the points,

is marked as A_t+1Dot relative to S_tThe coordinates of the points are determined by the coordinates of the points,is marked as S_tThe coordinates of the point with respect to the D point,is S_t+1Relative to the coordinates of point D, said

The formula for calculating the coordinates is:

\{\begin{matrix} x^{D} |_{S_{t + 1}} = x^{D} |_{S_{t}} + x^{S_{t}} |_{A_{t + 1}} - x^{S_{t}} |_{A_{t}} \\ y^{D} |_{S_{t + 1}} = y^{D} |_{S_{t}} + y^{S_{t}} |_{A_{t + 1}} - y^{S_{t}} |_{A_{t}} \end{matrix} .

further, when the positioning parameter value after the zoom operation or the drag operation exceeds the RGB intermediate frame range, the zoom parameter and the positioning parameter are subjected to border crossing detection, which specifically comprises the following steps: let W_vAnd H_vRespectively representing the width and height of the display area, W_SAnd H_sRepresenting the width and height of the video source, (x)^D,y^D)|_SCoordinates representing an origin S of the extracted data region with respect to an origin D of the RGB intermediate frame;

when W is_S<W_VAnd W_S-W_V<H_S-H_VWhen it is set to W_S=W_V，H_S=W_S′H_V/W_V；

When H is present_S<H_VAnd H_S-H_V<W_S-W_VThen set H_S=H_V，W_S=H_S′W_v/H_v;

When x is^D|_S<0 or W_S<W_VThen x is set^D|_S=0；

When y is^D|_S<0 or H_S<H_VThen, set y^D|_S=0；

When x is^D|_S>W_S-W_VThen x is set^D|_S=W_S-W_V；

When y is^D|_S>H_S-H_VThen, set y^D|_S=H_S-H_V。

Meanwhile, in order to solve the technical problem, the invention also provides a device for dragging and zooming the video mobile terminal in real time, which specifically comprises the following steps: a decoding module, a drawing module and an interaction module,

the decoding module includes: the decoding submodule, the decoding control submodule, the conversion submodule and the double buffer; the decoding submodule decodes the video stream file into video frame data; the decoding control sub-module is used for receiving the zooming parameters transmitted by the interactive module and controlling the conversion sub-module to carry out format conversion; the conversion submodule is used for converting the video frame data obtained by the decoding submodule into RGB intermediate frames of an ARGB8888 standard in an RGB format; the double buffers are used for storing the converted RGB intermediate frames;

the drawing module includes: an extraction submodule, a filling submodule and a drawing control submodule; the extraction submodule is used for extracting data of a part to be displayed from the double buffers; the filling submodule writes the extracted data to be displayed into an area to be displayed; the drawing control submodule is used for receiving the positioning parameters transmitted by the interaction module and controlling the extraction submodule to extract the position of the data;

the interaction module comprises: the system comprises a monitoring submodule, a calculation submodule, a request diverter, a zooming adapter and a dragging adapter; the monitoring submodule is used for monitoring user interaction requests in a display area in real time; the computing submodule is used for computing the interaction request of the user and transmitting the computing result to the request diverter; the request diverter is used for diverting the interactive request, and separating out whether the request is a zooming request or a dragging request or both the requests; the zooming adapter receives a zooming request separated by the request diverter and calculates zooming coordinates to obtain zooming parameters; the dragging adapter is used for receiving dragging requests separated by the request diverter and carrying out dragging coordinate calculation to obtain positioning parameters.

Furthermore, the interaction module further comprises a border crossing sub-module, and the border crossing sub-module is used for detecting whether the interaction request crosses the border and performing border crossing correction.

The invention has the beneficial effects that: the method and the device for dragging and zooming the video mobile terminal in real time solve the problems of video playing of the mobile terminal in the prior art by dragging and zooming the video image in real time at the mobile terminal, have simple and convenient realization process and high efficiency, provide an out-of-range detection and correction method for detecting and correcting the video source range exceeding the display area after dragging or zooming, and more effectively facilitate the operation of video playing at the mobile terminal.

Drawings

Fig. 1 is a flowchart illustrating a method for dragging and zooming a video mobile terminal in real time according to an embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating scaling coordinates in a scaling operation according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a conversion of dragging coordinates in a dragging operation according to an embodiment of the present invention;

fig. 4 is a block diagram illustrating a real-time dragging and zooming apparatus of a video mobile terminal according to an embodiment of the present invention.

Detailed Description

The invention is further illustrated with reference to the figures and the specific examples.

The invention discloses a method for dragging and zooming a video mobile terminal in real time, which specifically comprises the following steps: the method comprises the following steps of decoding a thread, a drawing thread and an interaction thread, wherein the decoding thread, the drawing thread and the interaction thread run in parallel;

the thread decoding method comprises the following specific steps:

s14, storing the converted RGB intermediate frame into a double buffer;

the thread drawing method comprises the following specific steps:

s23, extracting data of a part to be displayed from the double buffers;

the specific steps of the interaction thread are as follows:

s33, calculating a zooming component and a dragging component;

In order that those skilled in the art can understand and implement the technical solution of the present invention, the method of the present invention will be explained with reference to the flow chart shown in fig. 1:

step 1, opening a file to complete initialization: after the decoding thread is started, finding and opening a video stream file according to the position of the file, and completing corresponding parameter setting initialization;

and step 2, decoding: decoding the video stream file according to a specified decoder to obtain an original data frame;

step 3, collecting scaling parameters to control scaling ratio: collecting control scaling ratios transmitted by the interactive threads;

step 4, system conversion: carrying out format conversion on the original data frame, and converting the original data frame into an RGB format which can be processed by the mobile terminal;

and 5, storing the RGB intermediate frame to a double buffer: storing the RGB intermediate frame after system conversion in an idle one of the double buffers;

and 6, awakening a drawing thread after frame synchronization: storing RGB intermediate frames and synchronizing double buffer areas, then going to step 16, waking up a drawing thread to draw, and simultaneously going to step 2 to perform the decoding process of the next frame;

step 7, monitoring window events: monitoring events in a display window after a monitoring thread is started, wherein main window events comprise single click, double click, multi-point touch and the like, and only single-finger drag and drop and double-finger zoom events are concerned;

step 8, calculating and analyzing the interaction behavior: calculating and analyzing the interaction behavior, and judging whether the user requirement is changed;

step 9, changing user requirements: judging whether the tolerance is exceeded or not according to the calculated interactive behavior result, and if so, entering step 10 to further separate the user requirements; otherwise, entering step 7, and continuing to monitor the window event;

step 10, component separation: under the condition that the user requirement is changed, further calculating dragging and zooming components, separating the dragging and zooming components, separating corresponding vectors, and entering the steps 11 and 12;

step 11, generating a scaling parameter: separating to obtain a scaling component, carrying out coordinate conversion calculation to obtain a scaling ratio, generating a scaling parameter, transmitting the scaling parameter to a collection scaling parameter of a decoding thread to control the scaling ratio, and entering the step 3;

step 12, generating positioning parameters: separating to obtain dragging components, performing coordinate conversion calculation to obtain dragging displacement, generating dragging positioning parameters, transmitting the dragging positioning parameters to a collection positioning parameter control extraction area of the drawing thread, and entering step 13;

step 13, collecting positioning parameters to control an extraction area: collecting positioning parameters transmitted by the interactive threads, determining an extraction initial position, and performing data extraction in step 14;

step 14, extraction: extracting data to be displayed from the double caches, extracting only partial data to be displayed in order to improve efficiency without extracting complete RGB intermediate frames, and entering step 15 to fill data after extraction is completed;

step 15, filling: and opening the display area, writing the extracted data into the display area in a matching way, finishing the drawing of one frame of image, and then entering step 16, wherein the drawing thread enters a dormant state.

Step 16, sleep/wake-up trigger: and after the data is filled, the drawing thread enters a dormant state, waits for the arrival of the next frame data to wake up the drawing thread, and enters the step 13 to draw the next frame.

Firstly, in a decoding thread, after a video stream reaches a mobile terminal, a video needs to be decoded, and a common decoding processor comprises MPEG, h.261, h.263, WMV and the like, wherein in the embodiment of the present invention, an FFmpeg is taken as an example, a data frame obtained by decoding is in a YUV420P format, and is finally converted into an RGB intermediate frame in an ARGB8888 standard in an RGB format, and since the FFmpeg cannot be directly used in an Android platform, the Android platform of the mobile terminal in the embodiment of the present invention needs to be moved to the Android platform, and a specific transplanting process is as follows:

1. acquiring an FFmpeg source code and an NDK development kit;

2. configuring an NDK development environment;

3. adding an mk file according to the NDK project specification, and modifying the original compiling configuration file to enable the original compiling configuration file to be in line with the NDK compiling specification;

4. compiling the FFmpeg by using an NDK-built command of the NDK development kit;

5. the compiled so shared library is deployed.

After the platform of the decoder is transplanted, video decoding can be performed, a data frame obtained by decoding is in a YUV420P format, the YUV420P format is divided into three components, Y represents brightness, namely a gray value, and U and V represent chroma, and the functions of describing image color and saturation are used for specifying the color of a pixel. Since the mobile terminal cannot directly process the data in YUV420P format, it needs to be converted into RGB format.

According to the requirement, considering that the present invention adopts parallel processing of decoding and rendering, the bottleneck influencing the system operation should be in decoding rather than rendering, and the decoding time consumption should be reduced as much as possible, so the embodiment of the present invention converts the YUV420P format into the RGB intermediate frame of the ARGB8888 standard of the RGB format. According to the API provided by FFmpeg, the API can be directly called to carry out format conversion, the conversion can be completed with the subsequent scaling operation, and only the target frame format member variable in the SwsContext structure variable needs to be set as ARGB8888 in the conversion stage.

Secondly, in the drawing thread, the drawing thread adopts a passive triggering type message mechanism, after the decoding thread completes the decoding of one frame of data, the drawing thread is informed to be woken up to draw one frame of image, and the drawing thread enters a dormant state after the drawing is completed to wait for being woken up next time.

The data to be displayed is extracted from the RGB intermediate frame firstly when the image is drawn, in order to simplify calculation, the extraction operation only needs to acquire the coordinate of the video visual area relative to the RGB intermediate frame origin, the coordinate is controlled by the interaction thread and is stored in the drawing thread as a global variable, if a dragging request occurs, the coordinate value is updated, otherwise, the coordinate is repeatedly used all the time.

In practical application, a drawing thread needs to extract data from an RGB intermediate frame, a decoding thread needs to write data into the RGB intermediate frame, and the RGB intermediate frame needs to be subjected to synchronous mutual exclusion processing in order to maintain data consistency and normal operation of a program, which reduces system operation efficiency. The invention adopts the bidirectional RGB intermediate frame to simulate the double buffer, detects and takes out the idle RGB intermediate frame for storing the newly obtained frame after decoding and conversion, manages a flag bit in the drawing thread to represent the occupation state, provides a uniform outlet in the decoding thread, shields the characteristic of double buffer for the drawing thread, and simplifies the operation of the drawing thread.

In the interaction thread, the mobile terminal mainly monitors gesture operation on a display component SurfaceView for gesture recognition, and mainly monitors single-finger touch dragging and double-finger touch zooming events for the embodiment of the invention.

And when zooming operation is carried out, because the video display accords with the visual habit of human, when zooming is carried out to the maximum and then zooming is carried out, the image edge is blackened to influence the visual effect, and the zooming at the moment is meaningless zooming, the processing request can be rejected. The maximum value of zooming is set to display the whole intermediate frame data, zooming is performed according to linear equal proportion in the embodiment of the invention, because the display screen of the mobile terminal is smaller, if the zooming is performed by adopting an exponential growth mode in the prior art, the zooming speed is too fast, and a user is difficult to finely control, so that the zooming can be performed by adopting the linear equal proportion, and secondary zooming can be performed when the zooming is not expected to a certain degree, so that the zooming is more fine and the requirements of the user are better met although the zooming times are increased.

The real-time dragging of the video makes the drawing thread respond immediately after the dragging operation occurs, the position of extracting data from the RGB intermediate frame is changed and then the data is filled in the display area, but the real-time zooming of the video does not wake up the drawing thread immediately after the zooming occurs, but sends the zooming parameter to the decoding thread to control the decoding of the next frame, because the currently available RGB intermediate frame is generated and can not be changed, the size of the video can not be changed immediately according to the zooming request; and the origin coordinates of the extracted data area are relative to the new scale ratio, if rendered immediately, the rendering thread will render the old frame with the origin coordinates of the new extracted data area, which can cause video jitter. Therefore, the embodiment of the invention adopts the awakening trigger of the decoding thread as the synchronous command of the drawing thread, so that the drawing of the new visible area and the new decoded frame are kept synchronous, and the occurrence of video jitter is avoided.

Fig. 2 is a schematic diagram illustrating scaling coordinates in a scaling operation according to an embodiment of the present invention, where a specific process of generating scaling parameters includes:

the finger touch points before zooming are respectively E_tAnd F_tAfter the zoom operation, the finger touch points are respectively E_t+1And F_t+1，S_tIn order to extract the start of the data area,

is E_tRelative to S_tThe coordinates of the points are such that,

is F_tRelative to S_tThe coordinates of the points are such that,

is E_t+1Relative to S_tThe coordinates of the points are such that,

due to the fact that

Therefore, the user can not zoom in and out according to the zoom ratio at the same time, and the user can choose the zoom method in consideration of the visual habits of the user and pay more attention to the longitudinal zooming effect

Z_{v} = \frac{| | E_{t} F_{t} | |}{| | E_{t + 1} F_{t + 1} | |},

Wherein,

| | E_{t} F_{t} | | = \sqrt{{(x^{S_{t}} |_{E_{t}} - x^{S_{t}} |_{F_{t}})}^{2} + {(y^{S_{t}} |_{E_{t}} - y^{S_{t}} |_{F_{t}})}^{2}},

| | E_{t + 1} F_{t + 1} | | = \sqrt{{(x^{S_{t}} |_{E_{t + 1}} - x^{S_{t}} |_{F_{t + 1}})}^{2} + {(y^{S_{t}} |_{E_{t} + 1} - y^{S_{t}} |_{F_{t} + 1})}^{2}},

scaling ratio of RGB inter frame size

Z_{R} = Z_{v} / (\frac{H_{v}}{H_{t}});

W after performing a zoom operation_t+1、H_t+1Respectively as follows:

the presence of positioning parameters is not taken into account in the above-described scaling operationHowever, the calculation of the positioning parameters is the same as the calculation of the positioning parameters described below, and the embodiments of the present invention will not be described in detail.

Fig. 3 is a schematic diagram illustrating conversion of a dragging coordinate in a dragging operation according to an embodiment of the present invention, where a specific process of generating a positioning parameter includes:

is marked as A_t+1Dot relative to S_tThe coordinates of the points are determined by the coordinates of the points,

is marked as S_tThe coordinates of the point with respect to the D point,

is S_t+1Relative to the coordinates of point D, said

The formula for calculating the coordinates is:

\{\begin{matrix} x^{D} |_{S_{t + 1}} = x^{D} |_{S_{t}} + x^{S_{t}} |_{A_{t + 1}} - x^{S_{t}} |_{A_{t}} \\ y^{D} |_{S_{t + 1}} = y^{D} |_{S_{t}} + y^{S_{t}} |_{A_{t + 1}} - y^{S_{t}} |_{A_{t}} \end{matrix} .

however, no matter single-finger dragging or double-finger zooming operation is performed, after the positioning coordinate and the zooming ratio are calculated, the calculation is updated to the drawing and decoding thread, and border-crossing detection and correction are also needed to ensure that the value of the transformation parameter falls within a reasonable range.

Correspondingly, after the scaling operation is performed, the size of the target video needs to be: 1. the situation that the size of a target video is smaller than that of a display area, so that the periphery of the displayed video is kept black, which is not in line with the visual habit of a user, is avoided; 2. overflow caused by unregulated amplification is avoided, the size of a target video has a maximum plane which can be reached, the value is specified during initialization, and the limit cannot be exceeded during amplification; 3. and reasonably processing the problem that the proportion of the target display area is not matched with the proportion of the video source.

After the dragging operation is carried out, the positioning parameters cannot take values beyond the range of the RGB intermediate frame, otherwise, errors can occur in reading data; and when the positioning parameters are in the RGB intermediate frame range, the values are reasonable, so that the upper, lower, left and right boundaries of the target display area do not exceed the upper, lower, left and right boundaries of the video when the video is dragged.

Therefore, it is summarized that when performing single-finger dragging or two-finger zooming, the problems needing attention include: 1. the minimum form of the video must be displayed completely in the target area, and the irregular area is filled with black; 2. the minimal form cannot leave a black edge in both the x and y directions, which is not consistent with the user's visual habits; 3. for convenient processing, when a black edge appears, the upper left corner of the video and the target display area are preferably aligned; 4. when the video size is entirely larger than the target display area, a black border cannot be displayed. In response to the above problem, the solution of the embodiment of the present invention is that, when the value of the positioning parameter after the zoom operation or the drag operation exceeds the RGB intermediate frame range, the zoom parameter and the positioning parameter are subjected to border crossing detection, and the specific process is as follows: let Wv and Hv represent the width and height of the display area, respectively, and WS and Hs represent the width and height of the video source, (xD, yD) | S represents the origin coordinates of the extracted data area;

When H is present_S<H_VAnd H_S-H_V<W_S-W_VThen set H_S=H_V，W_S=H_S′W_v/H_v；

When x is^D|_S<0 or W_S<W_VThen x is set^D|_S=0；

When y is^D|_S<0 or H_S<H_VThen, set y^D|_S=0；

When x is^D|_S>W_S-W_VThen x is set^D|_S=W_S-W_V；

When y is^D|_S>H_S-H_VThen, set y^D|_S=H_S-H_V。

Meanwhile, aiming at the method, the invention also provides a device for dragging and zooming the video mobile terminal in real time, which specifically comprises the following steps: a decoding module, a drawing module and an interaction module,

the interaction module comprises: the system comprises a monitoring submodule, a calculation submodule, a request diverter, a zooming adapter and a dragging adapter; the monitoring submodule is used for monitoring user interaction requests in a display area in real time; the computing submodule is used for computing the interaction request of the user and transmitting the computing result to the request diverter; the zooming adapter is used for shunting the interactive request, and separating the zooming request or the dragging request or both the zooming request and the dragging request; the zooming adapter receives a zooming request separated by the request diverter and calculates zooming coordinates to obtain zooming parameters; the dragging adapter is used for receiving dragging requests separated by the request diverter and carrying out dragging coordinate calculation to obtain positioning parameters.

The interaction module further comprises a border crossing processing submodule, and the border crossing processing submodule is used for detecting whether the interaction request crosses the border and performing border crossing correction.

The following will describe each sub-module of the apparatus in detail with reference to the block diagram of the apparatus for real-time dragging and zooming of the video mobile terminal shown in fig. 4:

decoding the sub-module: decoding of a video stream file is referred to as video frame data, which is the original frame. The decoding standards of video are many, MPEG, h.261, h.263, WMV and the like are common, and the present invention takes FFmpeg as an example, and the data frame obtained by decoding is in YUV420P format.

The decoding control sub-module: the module is a control sub-module under the decoding module, receives the zoom ratio control parameter transmitted by the interactive module, and controls the conversion sub-module to carry out format conversion.

A conversion submodule: and converting the original frame in YUV420P format obtained by the coding submodule into an RGB intermediate frame in RGB format of ARGB8888 standard. According to the requirement, the YUV420P format can be converted into RGB565 or RGB8888 standard, under FFmpeg, the YUV420P format is converted into RGB8888 standard faster than RGB565 standard, although RGB format of ARGB8888 standard increases the burden of drawing picture, the speed of drawing one frame is much faster than that of decoding one frame, the invention adopts parallel processing of decoding and drawing, the bottleneck influencing the system operation is decoding rather than drawing, therefore, the invention needs to shorten the decoding time consumption, therefore, the invention adopts the scheme of converting YUV420P format into RGB format of ARGB8888 standard for format conversion.

Double buffer: and detecting the free one in the double buffer, and storing the converted RGB intermediate frame therein.

Extracting a submodule: and extracting data to be displayed from a double-buffer area in a decoding module, only extracting data for a display part, and not extracting the whole RGB intermediate frame. The extracted data serves as source data for populating the sub-modules.

Filling a submodule: and opening the target display area, and writing the data obtained by the extraction sub-module in a matching way.

A drawing control submodule: the module is a control submodule under the drawing module, receives the positioning parameters generated by dragging transmitted by the interaction module, and controls the position of the extraction submodule for extracting data.

A monitoring submodule: and monitoring the user interaction request in the display area in real time, wherein the user interaction request is mainly represented as a touch display area.

A calculation submodule: the triggered request is calculated, and if the request generation exceeding the tolerance is detected, the request is processed by a request splitter.

Requesting a diverter: the interactive request is shunted, and whether the request is a zooming request or a dragging request or both are separated.

Scaling the adapter: and receiving a zooming request for requesting the splitter to separate, and calculating zooming coordinates to obtain zooming parameters.

Dragging the adapter: and receiving a dragging request for requesting the splitter to separate, and calculating dragging coordinates to obtain positioning parameters.

And a border crossing processing submodule: and detecting whether the interaction request is out of range and performing out-of-range correction.

It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Those skilled in the art can make various other specific changes and combinations based on the teachings of the present invention without departing from the spirit of the invention, and these changes and combinations are within the scope of the invention.

Claims

1. A method for dragging and zooming a video mobile terminal in real time is characterized by specifically comprising the following steps: the method comprises the following steps of decoding a thread, a drawing thread and an interaction thread, wherein the decoding thread, the drawing thread and the interaction thread run in parallel;

the thread decoding method comprises the following specific steps:

s14, storing the converted RGB intermediate frame into a double buffer;

the thread drawing method comprises the following specific steps:

s23, extracting data of a part to be displayed from the double buffers;

the specific steps of the interaction thread are as follows:

s33, calculating a zooming component and a dragging component;

2. The method for dragging and zooming in and out of a video mobile terminal in real time according to claim 1, wherein the specific process of generating the zooming parameter in the step S34 is as follows:

is E_tRelative to S_tThe coordinates of the points are such that,

is F_tRelative to S_tThe coordinates of the points are such that,

is E_t+1Relative to S_tThe coordinates of the points are such that,

selectingCalculating a zoom ratio Z in the extracted data region as a ratio of the extracted data region to the RGB intermediate frame size_v，

Z_{v} = \frac{| | E_{t} F_{t} | |}{| | E_{t + 1} F_{t + 1} | |},

Wherein,

| | E_{t} F_{t} | | = \sqrt{{(x^{S_{t}} |_{E_{t}} - x^{S_{t}} |_{F_{t}})}^{2} + {(y^{S_{t}} |_{E_{t}} - y^{S_{t}} |_{F_{t}})}^{2}},

| | E_{t + 1} F_{t + 1} | | = \sqrt{{(x^{S_{t}} |_{E_{t + 1}} - x^{S_{t}} |_{F_{t + 1}})}^{2} + {(y^{S_{t}} |_{E_{t} + 1} - y^{S_{t}} |_{F_{t} + 1})}^{2}},

scaling ratio of RGB inter frame size

Z_{R} = Z_{v} / (\frac{H_{v}}{H_{t}});

W after performing a zoom operation_t+1、H_t+1Respectively as follows:

3. the method for dragging and zooming in and out of a video mobile terminal in real time as claimed in claim 1, wherein the specific process of generating the positioning parameter in step S35 is as follows:

is marked as A_t+1Dot relative to S_tThe coordinates of the points are determined by the coordinates of the points,is marked as S_tThe coordinates of the point with respect to the D point,

is S_t+1Relative to the coordinates of point D, said

The formula for calculating the coordinates is:

\{\begin{matrix} x^{D} |_{S_{t + 1}} = x^{D} |_{S_{t}} + x^{S_{t}} |_{A_{t + 1}} - x^{S_{t}} |_{A_{t}} \\ y^{D} |_{S_{t + 1}} = y^{D} |_{S_{t}} + y^{S_{t}} |_{A_{t + 1}} - y^{S_{t}} |_{A_{t}} \end{matrix} .

4. the method as claimed in any one of claims 1 to 3, wherein when the value of the zoom operation or the positioning parameter after the drag operation exceeds the RGB mid-frame range, the zoom parameter and the positioning parameter are detected to be out of range, which comprises the following steps: let W_vAnd H_vRespectively representing the width and height of the display area, W_SAnd H_sIndicating the width and height of the video source,

coordinates representing an origin S of the extracted data region with respect to an origin D of the RGB intermediate frame;

When x is^D|_S<0 or W_S<W_VThen x is set^D|_S=0；

When y is^D|_S<0 or H_S<H_VThen, set y^D|_S=0；

When x is^D|_S>W_S-W_VThen x is set^D|_S=W_S-W_V；

When y is^D|_S>H_S-H_VThen, set y^D|_S=H_S-H_V。

5. A device for dragging and zooming a video mobile terminal in real time is characterized by specifically comprising: a decoding module, a drawing module and an interaction module,

6. The apparatus for real-time dragging and zooming of a video mobile terminal according to claim 5, wherein the interactive module further comprises an out-of-range processing sub-module, and the out-of-range processing sub-module is configured to detect whether the interactive request is out-of-range and perform out-of-range correction.