WO2023015404A1

WO2023015404A1 - Method for playing audio, apparatus, electronic device, and storage medium

Info

Publication number: WO2023015404A1
Application number: PCT/CN2021/111435
Authority: WO
Inventors: 张金梁
Original assignee: 深圳Tcl新技术有限公司
Priority date: 2021-08-09
Filing date: 2021-08-09
Publication date: 2023-02-16
Also published as: JP2024529083A

Abstract

Disclosed in embodiments of the present application are a method for playing audio, an apparatus, an electronic device, and a storage medium. An embodiment comprises: decoding an audio dataset to be played on a terminal device, the audio dataset comprising at least one frame of audio data; recording a start decoding reference time and actual decoding time for decoding each frame of audio data; obtaining an expected play time and a benchmark offset time corresponding to audio data; adjusting the start decoding reference time for each frame of audio data according to the benchmark offset time, and obtaining a start decoding adjusted time; determining a reference play time of audio data according to the start decoding adjusted time and the actual decoding time; adjusting the reference play time according to a time change; and playing audio data according to the terminal device once the adjusted reference play time reaches the expected play time. The adjusted audio play time of the audio data is made to be less than or equal to the expected play time, and audio data is made to be able to play synchronously.

Description

Audio playback method, device, electronic device and storage medium

technical field

The present application relates to the technical field of display control, in particular to an audio playback method, device, electronic equipment and storage medium.

Background technique

With the rapid development of audio playback technology, there are more and more types of audio players, and audio synchronization is an important basis for audio playback. In order to ensure audio synchronization playback, the audio server needs to calibrate the audio player connected to it before sending audio data. The system reference time in the device;

However, after the audio player receives the audio data, it needs to perform back-end data processing such as decoding on the audio data. Therefore, a certain decoding time needs to be reserved for the processing of the audio data before playing the audio data.

technical problem

It is difficult to unify and determine the back-end decoding processing time of different audio data, and the data volume of some audio data is huge, and the decoding processing time is relatively long, and the decoding processing rate has high requirements on the product hardware. Therefore, when the hardware performance reaches After the bottleneck, the expected playback time of the audio cannot be accurately synchronized with the reference time of the device terminal where the audio player is located, and audio playback is prone to out-of-sync audio and picture, frame loss, or freezing.

technical solution

Embodiments of the present application provide an audio playback method, device, electronic equipment, and storage medium, which can increase the time reserved for back-end decoding processing and realize synchronization of audio playback.

The embodiment of the present application provides an audio playback method, including:

Decoding the audio data set to be played on the terminal device, the audio data set includes at least one frame of audio data;

Record the start decoding reference time and the actual decoding processing time of the audio data decoding processing of each frame;

Acquiring the reference offset time and expected playing time corresponding to the audio data;

For each frame of audio data, adjust the decoding start reference time according to the reference offset time to obtain the decoding start adjustment time;

determining a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

updating the reference playing time according to the time change;

When the adjusted reference playing time reaches a desired playing time, the audio data is played according to the terminal device.

Correspondingly, the embodiment of the present application also provides an audio playback device, including:

A decoding module, configured to decode an audio data set to be played on the terminal device, where the audio data set includes at least one frame of audio data;

A recording module, configured to record the start decoding reference time and the actual decoding processing time of the audio data decoding process of each frame;

An acquisition module, configured to acquire a reference offset time and an expected playback time corresponding to the audio data;

An adjustment module, configured to adjust the decoding start reference time according to the reference offset time for each frame of audio data, to obtain the decoding start adjustment time;

A determination module, configured to determine a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

A timing module, configured to update the reference playing time according to time changes;

A playing module, configured to play the audio data according to the terminal device when the adjusted reference playing time reaches a desired playing time.

Optionally, in some embodiments of the present application, the audio data includes current audio data, and the obtaining module includes:

a sampling unit, configured to sample at least one frame of historical audio data, where the historical audio data is of the same data type as the current audio data;

The first acquisition unit is used to respectively acquire the historical reference playback time and the historical expected playback time after decoding the historical audio data of each frame;

A first determining unit, configured to respectively determine the offset time of each frame of the historical audio data according to the historical reference playing time and the historical expected playing time;

The second determining unit is configured to determine a reference offset time of the current audio data according to at least one of the offset times.

Optionally, in some embodiments of the present application, the second determining unit includes:

A first determining subunit, configured to determine an average value of at least one offset time to obtain an average offset time;

The second determining subunit is configured to determine the reference offset time of the current audio data according to the average offset time.

Optionally, in some embodiments of the present application, the second determination unit is specifically further configured to:

For determining an average value of at least one of the offset times to obtain an average offset time;

It is used to round the average offset time to obtain the average integer offset time corresponding to the historical audio data;

and determining the average integer offset time as a reference offset time of the current audio data.

for rounding the at least one offset time respectively to obtain at least one integer offset time;

For determining the integer average offset time of each frame of audio data according to the at least one integer offset time;

used to determine the integer average offset time as a reference offset time of the current audio data.

Optionally, in some embodiments of the present application, the sampling unit is also specifically used for:

Used to acquire historical audio data according to a preset frequency, the historical audio data including at least one frame;

It is used for sampling the at least one frame of historical audio data.

Optionally, in some embodiments of the present application, the reference offset time includes an actual offset time, and the adjustment module includes:

a second acquiring unit, configured to acquire the reserved processing time of the audio data;

A third determining unit, configured to determine the actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

The third obtaining unit is configured to adjust the decoding start reference time according to the actual offset time to obtain the decoding start adjustment time.

Optionally, in some embodiments of the present application, the second acquiring unit includes:

The third determining subunit is configured to determine the reserved processing time of the audio data according to the reference time of starting decoding of the audio data and the expected playing time.

Optionally, in some embodiments of the present application, the second acquiring unit is specifically further configured to:

A reserved processing time for obtaining said audio data;

For determining the actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

for determining an error offset time of the audio data according to the reference offset time and the actual offset time;

It is used to adjust the decoding start reference time according to the reference offset time and the error offset time to obtain the decoding start adjustment time.

Optionally, in some embodiments of the present application, the recording module is specifically used to:

It is used to do timing feedback according to the beating of the crystal oscillator, and obtain the start decoding time and the end decoding time of the audio data respectively;

It is used for determining the actual decoding processing time of the audio data according to the decoding start time and the decoding end time.

Optionally, in some embodiments of the present application, the determination module is specifically used to:

used to obtain the start decoding time and the end decoding time of the audio data decoding process;

It is used to determine the basic reference playback time of the audio data according to the decoding start adjustment time and the decoding start time;

It is used for determining the reference playing time of the audio data according to the basic reference playing time and the end decoding time.

Optionally, in some embodiments of the present application, the acquisition module is specifically used to:

for extracting the desired play time of the audio data directly from the audio data.

Optionally, in some embodiments of the present application, the obtaining module is also specifically used to:

an audio data type for identifying said audio data;

It is used to obtain a mapping relationship set, the mapping relationship set includes a mapping relationship between a preset audio data type and a preset reference offset time;

The method is used to acquire the reference offset time corresponding to the audio data according to the mapping relationship set and the audio data type.

Beneficial effect

In the embodiment of the present application, by reducing the reference playback time of audio data, the reference playback time changes with time, and after the change reaches the desired playback time, the audio data is played, that is, when the reference playback time of the audio data meets the desired playback time After the time, the audio data is played synchronously; the reference playback time is reduced according to the reduction of the reference time for decoding, and the actual decoding processing time of the audio data is not affected or limited. Therefore, this kind of audio data The corresponding reference playback time reduction method provides enough decoding time for the decoding process of audio data, and avoids audio and picture asynchrony, frame loss, or freezes in audio data playback due to too long decoding processing time. Phenomenon.

Description of drawings

The technical solutions and beneficial effects of the present application will be apparent through the detailed description of the specific embodiments of the present application below in conjunction with the accompanying drawings.

FIG. 1 is a schematic diagram of a scene of an audio playback method provided by the present application;

Fig. 2 is a schematic flow chart of the audio playing method provided by the present application;

Fig. 3 is another schematic flow chart of the audio playing method provided by the present application;

Fig. 4 is another schematic flow chart of the audio playback method provided by the present application;

FIG. 5 is a schematic structural diagram of an audio playback device provided by the present application.

FIG. 6 is a schematic structural diagram of an electronic device provided by the present application.

The realization, functional features and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Embodiments of the present invention

It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

Embodiments of the present application provide an audio playback method, device, electronic equipment, and storage medium. Specifically, the audio playback method provided by the embodiment of the present application can be performed by an electronic device, where the electronic device includes a terminal device or a server; where the terminal device can be a TV, a mobile phone, a notebook, a desktop or a tablet computer, etc.; the server can be It is an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, and it can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, and middleware services , domain name service, security service, content delivery network (CDN, ContentDelivery Network), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, etc. The servers can be directly or indirectly connected through wired or wireless communication.

Specifically, in this embodiment of the present application, the terminal device and the server jointly execute the audio playback method as an example, wherein data transmission can be performed between the terminal device and the server through a wired network, a wireless network, or broadcasting, wherein, between the terminal device and the server Basically, other devices can also be added to assist in completing the audio playback method, and the types of other devices are not limited here; wherein, the specific process of the terminal device and the server jointly executing the audio playback method is as follows:

The terminal device receives the audio data set (audio data packet) to be played from the server, and then the processor in the terminal device decodes the received audio data set, and records each frame of audio data to start the decoding process The decoding reference time and the actual decoding processing time of each frame of audio data. In addition, the terminal device also obtains the reference offset time and expected playback time of each frame of audio data from the audio data. Then, for each frame of audio data, according to the reference The offset time adjusts the start decoding reference time to obtain the start decoding adjustment time, and then, according to the start decoding adjustment time and the actual decoding processing time, determine the reference playback time after the audio data decoding process is completed, and finally, at the reference playback time. After the timing changes, and when the reference playing time is equal to the expected playing time, the terminal device plays the audio data; wherein, if the terminal device is a TV, the audio data can be played directly without using an additional audio player.

Among them, the reference time of the audio data (such as the reference time of starting decoding) is synchronized with the reference clock on the terminal device, and the reference clock is coordinated and synchronized with the accurate time according to the Network Time Protocol (NTP, Network Time Protocol). Among them, The source of accurate time is Coordinated Universal Time (UTC, Universal Time Coordinated), for example, the Beijing time displayed by the reference clock on the terminal device, when the terminal device starts decoding audio data at 6:00 Beijing time, the reference time to start decoding is also 6:00;

The reference playback time is the time when the audio data can be played after decoding. Theoretically, the reference playback time is also synchronized with the Beijing time, that is, the Beijing time after the audio data is decoded is the reference playback time of the audio data. However, the reference clock The timing of the clock is updated according to the data transmission, and there is time consumption in the transmission of data, so there is a time error in the timing of the reference clock. Therefore, in the embodiment of the present application, the reference clock is determined based on the reference time of starting decoding and the actual decoding processing time of decoding. Playing time, wherein, the actual decoding processing time is timed according to the feedback of the crystal vibration, so the time record is very accurate, therefore, you can get an actual reference playing time after the audio data decoding processing, at this time, get The timestamp of the reference playback time is no longer synchronized with the timestamp of the reference clock, but the reference playback time still increases according to the time (which can be Beijing time). For example, when the audio data is decoded and processed, the reference playback time is At 8 o'clock, the Beijing time can be 8:01 at this time, that is, the time stamps of the reference broadcast time and Beijing time are no longer the same;

Among them, when the reference playback time of the audio data is equal to the expected playback time, the audio data can be played, but the decoding process of the audio data takes a certain amount of time, and the reference playback time of the audio data after decoding often exceeds the corresponding expectation of the audio data. Playing time, therefore, it is necessary to call back the reference playing time after audio data decoding to make it synchronized with the expected playing time or synchronize with the expected playing time after the reference playing time increases with time, and then play the audio data.

Among them, Network Time Protocol (NTP, Network Time Protocol) is a protocol used to synchronize the computer time, it can make the computer synchronize its server or clock source (such as quartz clock, GPS, etc.), it can provide high-precision time correction (on LAN with The difference between the standard is less than 1 millisecond, tens of milliseconds on the WAN). Among them, NTP can obtain UTC time from atomic clocks, observatories, satellites, or from the Internet.

Among them, the expected playback time is the time when the audio data is expected to be displayed, such as the display time tag (PTS, Presentation Time Stamp). When the expected playback time of the audio data is synchronized with the reference playback time of the terminal device, the audio data can be played, that is, the audio The expected playback time of the data is synchronized with the international standard time. When the expected playback time of the audio data is later than the reference playback time of the terminal device, the audio data cannot be played accurately. Dun and so on.

Each will be described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the order of priority of the embodiments.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of an audio playback method provided by an embodiment of the present application. The specific process of the audio playback method can be as follows:

101. Perform decoding processing on an audio data set to be played on a terminal device, where the audio data set includes at least one frame of audio data.

Among them, the audio data received by the terminal device to be played is often coded and compressed data. For example, in digital TV, the transmitted data packet TLV is the ISDBS3 standard data packet, and the TLV packet corresponding to the 4K ultra-high-definition program is generally The data volume of the program content is very large. Therefore, in order to improve the transmission efficiency, it is generally necessary to encode and compress the audio data. The audio data received by the terminal device is compressed data that needs to be decoded. The terminal device can only decode the audio data after decoding the audio data. Play audio data.

102. Record the decoding reference time and actual decoding processing time of the audio data decoding processing of each frame.

Wherein, the decoding start reference time is synchronized with the reference clock on the terminal device, that is, when the audio data starts to be decoded, the time of the reference clock is the audio data decoding start reference time;

Wherein, the actual decoding processing time of the audio data is the actual time used in the audio data decoding process.

Wherein, the actual decoding processing time of the audio data can be accurately timed according to the hardware timing device, that is, optionally, in some embodiments, the step "recording the actual decoding processing time of the audio data decoding processing of each frame" may specifically include :

Timing feedback is performed according to the beating of the crystal oscillator, and the start decoding time and the end decoding time of the audio data are obtained respectively;

The actual decoding processing time of the audio data is determined according to the decoding start time and the decoding end time.

For example, timing can be performed by crystal oscillator feedback, such as an oscillator, when the terminal device is turned on, it starts timing from 0:00:00 seconds, and the timing process is fed back by the 27M crystal oscillator beating. Among them, using an oscillator for timing is more accurate.

Among them, timing is performed when the audio data starts to be decoded, and the start decoding time is obtained. When the audio data decoding is completed, the end decoding time of the audio data is obtained, and the audio data can be obtained according to the difference between the end decoding time and the start decoding time. Actual decoding processing time for decoding.

103. Acquire a reference offset time and an expected playback time corresponding to the audio data.

Among them, when the reference playback time of the audio data is greater than the expected playback time, the audio data cannot be played accurately, and there will be phenomena such as out-of-sync, lost, or stuck audio and video. Therefore, it is necessary to adjust the reference playback time of the audio data. Adjustment, that is, adjusting the data of the reference playback time referenced by the audio data playback, rather than adjusting the data corresponding to the reference clock, that is, not adjusting the display time on the terminal device;

Wherein, the reference offset time can be determined according to the difference between the reference play time and the expected play time of the audio data, and the reference play time is rolled back to the above difference, so that the adjusted reference play time is less than or equal to the expected play time.

Among them, since the decoding time of audio data is at the millisecond level, the adjustment of the reference time is also a millisecond difference that is difficult for the human body to perceive. For example, when the original reference playback time of audio data is 7:0500ms, that is, the terminal device reference clock The display time of the clock is 7:00, and the reference offset time is 200ms. Therefore, the adjusted reference playback time is 7:0300ms. At this time, the display time of the reference clock is still 7:00, and the millisecond level of 200ms Adjustment is difficult for users to perceive, but the time of 200ms is difficult to ignore for audio decoding. Since, when the original reference playback time exceeds the expected playback time, the audio data cannot be played accurately, so you can adjust that part of the reference The method of shifting time adjusts the reference playback time so that the audio data can be played, and provides enough time for audio data decoding, reducing the performance requirements for hardware products, and it is also difficult for users to perceive this part of the time difference.

In this embodiment of the present application, the reference offset time of the current audio data can be determined according to the offset of the historical data in the past, that is, optionally, in some embodiments, the step "acquire the reference offset time corresponding to the audio data Offset time", which can specifically include:

Sampling at least one frame of historical audio data, where the historical audio data is of the same data type as the current audio data;

Respectively obtain the historical reference playback time and the historical expected playback time after the decoding of the historical audio data of each frame;

According to the historical reference playing time and the historical expected playing time, respectively determine the offset time of each frame of the historical audio data;

A reference offset time of the current audio data is determined according to at least one of the offset times.

Among them, according to the acquisition of the offset time of the historical audio data of the same data type, the reference offset time that the reference playback time of the current audio data should be adjusted can be inferred, because the sampling is the same historical audio data type as the current audio data Therefore, the acquisition of the reference offset time is more accurate.

Wherein, in order to ensure the accuracy of the reference offset time, the historical audio data to be sampled can be replaced according to a certain frequency, that is, optionally, in some embodiments, the step "sampling at least one frame of historical audio data", Specifically can include:

Acquiring historical audio data according to a preset frequency, the historical audio data including at least one frame;

Sampling the at least one frame of historical audio data.

Wherein, according to different and new historical audio data as a reference for obtaining the reference offset time, it has certain timeliness and accuracy.

Wherein, the reference offset of the current audio data may be determined by averaging the offset times corresponding to the historical audio data, that is, optionally, in some embodiments, the step "according to at least one of the offset times , to determine the reference offset time of the current audio data", which may specifically include:

determining an average value of at least one of said offset times to obtain an average offset time;

A reference offset time of the current audio data is determined according to the average offset time.

Among them, the average value indicates the overall offset of the historical audio data of the same data type as the current audio data, and the overall offset can reflect the range to be adjusted for the reference playback time of the current audio data. Therefore, the historical audio data The average offset time is used as the reference offset time of the current audio data, which has certain accuracy and reference value.

Wherein, since the decoding time is at the level of milliseconds, in order to improve the calculation efficiency, the time in units of milliseconds can be ignored, so the offset time can be adjusted by an integer-level offset value, that is, optional, in some In an embodiment, the step of "determining the reference offset time of the current audio data according to at least one of the offset times" may specifically include:

Rounding the average offset time to obtain the average integer offset time corresponding to the historical audio data;

The average integer offset time is determined as a reference offset time of the current audio data.

Wherein, the method of rounding can be a method of converting zeros into wholes, for example, adjusting the value after the decimal point to 1, so that the adjusted reference playback time is less than the expected playback time, so as to ensure that the audio data can be played smoothly.

Wherein, the reference offset time of an integer can intuitively know the interval to be adjusted of the reference playback time of the audio data.

Among them, since the decoding time is at the level of milliseconds, in order to improve the calculation efficiency, the time below the unit of milliseconds can be ignored, that is, only the integer part of each offset time can be used as a consideration for the selection of the reference offset time, and each The offset time is rounded first, and the average value of these offset times is calculated, that is, optionally, in some embodiments, the step "determine the reference offset of the current audio data according to at least one of the offset times shift time", which may specifically include:

respectively rounding the at least one offset time to obtain at least one integer offset time;

determining an integer average offset time of each frame of audio data based on the at least one integer offset time;

The integer average offset time is determined as a reference offset time of the current audio data.

Wherein, in order to ensure that the adjusted reference playback time is less than or equal to the expected playback time, each offset time can be rounded up to zero, for example, the data after the decimal point of each offset data is adjusted to 1, that is, the integer part is increased by 1 ms.

Among them, the reference offset time indicates the basic offset data or the overall offset of the historical audio data of the same data type as the current audio data. Therefore, the adjustment of the reference playback time by the reference offset time can basically meet the requirements of the reference The playing time is adjusted to be less than or equal to the expected playing time.

Among them, the above-mentioned reference offset time is obtained on the basis of taking the average value, and the average value of each audio data offset time represents the overall offset situation of the audio data set, therefore, the obtained by the above method The reference offset time can satisfy the adjustment requirement of the reference playback time to a certain extent.

However, for audio data that requires high precision, it is also possible to determine the error offset that still exists in the audio data based on the adjustment of the reference offset time based on the actual offset time corresponding to the audio data on the basis of the reference offset time. shift time, adjust the reference play time of the audio data simultaneously according to the reference offset time and the error offset time, that is, optionally, in some embodiments, the step "according to the reference offset time to start decoding The reference time is adjusted to obtain the start decoding adjustment time", which may include:

Acquiring the reserved processing time of the audio data;

determining an actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

determining an error offset time of the audio data according to the reference offset time and the actual offset time;

The decoding start reference time is adjusted according to the reference offset time and the error offset time to obtain an adjustment start decoding time.

Among them, since the reference offset time is calculated by the average value of the offset time of historical audio data, there is still a certain error after adjusting the reference playback time according to the reference offset time, so, according to the actual offset time and the reference offset time to determine the error offset time, and then, after the reference playback time is adjusted according to the reference offset time, further adjustments are made according to the error offset time, which can improve the accuracy of the reference playback time adjustment of the audio data , that is, ensure that the adjusted reference playback time is less than or equal to the expected playback time.

For example, if the reference offset time is an integer, certain precision will be lost. Therefore, in order to meet the accuracy requirements, the actual audio data corresponding to each frame of audio can be determined, and the error offset can be determined according to the actual audio data and the reference offset time Time, then, on the basis of adjusting the reference playback time according to the reference offset time, further adjustment is made according to the error offset time.

Wherein, for audio data requiring high precision, it can be adjusted according to the actual offset time after decoding the audio data, that is, the reference offset time also includes the actual offset time, then, optionally, in some embodiments, The step of "adjusting the decoding start reference time according to the reference offset time to obtain the decoding start adjustment time" may specifically include:

Acquiring the reserved processing time of the audio data;

The decoding start reference time is adjusted according to the actual offset time to obtain the decoding start adjustment time.

Wherein, after the audio data is decoded and before the audio data is output, the audio data decoding start reference time is adjusted according to the actual offset time of the audio data.

Wherein, the reserved processing time is the processing time reserved for the decoding of audio data, wherein, the acquisition of the reserved processing time can be obtained according to the decoding start reference time and the expected playback time, that is, optional, in some embodiments , the step of "obtaining the reserved processing time of the audio data" may specifically include:

The reserved processing time of the audio data is determined according to the decoding start reference time of the audio data and the expected playing time.

Among them, the reserved processing time of audio data can be obtained according to the difference between the expected playback time and the reference time of starting decoding, and the actual deviation of audio data decoding can be obtained by comparing the reserved processing time with the actual decoding processing time. shift time.

Wherein, the expected playing time of the audio data may be directly extracted from the data information of the audio data, that is, optionally, in some embodiments, the step "obtaining the expected playing time corresponding to the audio data" may specifically include:

The expected playing time of the audio data is directly extracted from the audio data.

Wherein, the reference offset time of the audio data may also be determined according to the type of the audio data, that is, optionally, in some embodiments, the step of "acquiring the reference offset time corresponding to the audio data" may specifically include :

identifying an audio data type of the audio data;

Obtain a mapping relationship set, the mapping relationship set includes a mapping relationship between a preset audio data type and a preset reference offset time;

Acquire a reference offset time corresponding to the audio data according to the set of mapping relationships and the audio data type.

Wherein, the mapping relationship set can be determined according to the relationship between the type of historical audio data and the offset time in the past, for example, according to the statistics of the type of historical audio data and the offset time, comprehensively determine the reference offset corresponding to the data type of the audio data shift time.

104. For each frame of audio data, adjust the decoding start reference time according to the reference offset time to obtain a decoding start adjustment time.

Wherein, since the decoding reference time before the audio data decoding process and the reference playback time after the decoding process are on the same timeline, that is, the change of the reference playback time will bring about the change of the reference playback time, so the decoding of the start The adjustment of the decoding reference time at the start of processing is equivalent to the adjustment of the reference playback time after decoding processing.

Wherein, please refer to FIG. 3. FIG. 3 is a schematic flow diagram of reference playback time adjustment in the embodiment of the present application. The reference time for starting decoding when the audio data starts to be processed can be adjusted. For example, after calling back the reference time for starting decoding, use The final reference playback time after audio data processing is also changed and adjusted accordingly, which is equivalent to updating and adjusting the final reference playback time after starting to decode the reference time adjustment, as follows:

111. Obtain the audio data set, that is, the audio data packet (TLV stream);

112. When decoding the audio data in the audio data packet, record the real-time time of the reference clock of the terminal device when the audio data starts to be decoded (that is, the reference time for starting decoding of the audio data) and the actual decoding processing time;

113. Obtain the expected playing time corresponding to the audio data;

114. Adjust the decoding start reference time according to the reference offset time and the error offset time, so that the sum of the adjusted start decoding adjustment time and the actual decoding processing time (that is, the reference playback time) is less than or equal to the expectation corresponding to the audio data play time;

115. Continue timing and updating the adjusted reference play time (that is, after the audio data is decoded), so that the reference play time is changed to the expected play time corresponding to the audio data, and the audio data is played at this time.

Wherein, by adjusting the reference time to start decoding when the audio data starts to decode, the hardware device playing audio data can obtain a reference play time that is less than or equal to the expected play time, that is, after the reference time to start decoding is adjusted, in The reference playback time obtained by the hardware device after decoding is the adjusted reference playback time.

105. Determine a reference playback time of the audio data according to the decoding start adjustment time and the actual decoding processing time, and the reference playback time is less than or equal to the expected playback time.

Wherein, the sum of the start decoding reference time of the audio data and the actual decoding processing time is the reference time after the decoding of the audio data. Due to referring to the offset time of the historical audio data, the start reference time of the current audio data is adjusted to make the audio data The reference playing time after the data is decoded is less than or equal to the expected playing time of the audio data.

Wherein, the reference playing time of the audio data may be determined according to the time information before the audio data decoding processing and the time information after the decoding processing, that is, optionally, in some embodiments, the step "according to the start decoding and adjusting the time and the The actual decoding processing time determines the reference playing time of the audio data", which may specifically include:

Obtain the start decoding time and end decoding time of the audio data decoding process;

determining a basic reference playback time of the audio data according to the start decoding adjustment time and the start decoding time;

A reference playback time of the audio data is determined according to the basic reference playback time and the end decoding time.

Among them, the difference between the start decoding adjustment time and the start decoding time can obtain the basic reference playback time of the audio data, and the sum of the basic reference playback time and the end decoding time can reach the reference playback time of the audio data, that is, the reference playback time and the start time The difference in decoding adjustment time is the difference between the end decoding time and the start decoding time.

Wherein, the reference playing time is acquired by the basic reference playing time, so that the adjustment of the reference offset time can be adjusted when the decoding of the audio data starts, that is, the reference time of starting decoding of the audio data is adjusted.

Wherein, by adjusting the basic reference playback time before audio data decoding starts, the adjusted decoding start adjustment time can synchronously act on the final reference playback time.

106. Update the reference playing time according to the time change.

Wherein, the reference playback time is the reference playback time of the audio data, and when the reference playback time reaches the desired playback time, the audio data can be played;

Among them, since the reference offset time of the current audio data can be inferred according to the offset time of the historical audio data, the reference playback time is called back according to the reference offset time, so that the reference playback time is not kept within the time of the reference clock Sync, but adjust to a time that is less than or equal to the desired playback time.

107. When the adjusted reference playing time reaches a desired playing time, play the audio data according to the terminal device.

Wherein, when the reference playback time continues to count, when the timestamp corresponding to the reference playback time is the same as the timestamp of the expected playback time, the audio data can be played, wherein, since the audio data is decoded at the millisecond level, so The time difference between the adjusted reference playback time and the expected playback time should also be at the millisecond level.

Wherein, because the time stamp difference corresponding to the expected playing time of each audio data in the audio data set is relatively fixed, therefore, the difference between the reference playing time corresponding to each audio data after adjustment is also relatively fixed, and can still be Realize continuous playback of audio.

Wherein, please refer to FIG. 4, in the embodiment of the present application, the specific process of adjusting the reference playback time of audio data according to the reference offset time is as follows:

121. Obtain the audio data set, that is, the audio data packet (TLV stream), obtain the time stamp of the start decoding reference time of the audio data from the audio data set, and verify the time stamp of the start decoding reference time, and judge the time stamp data , if valid, enter step 122, otherwise, proceed to step 121;

122. Obtain the timestamp of the expected playback time of each frame of audio data from the audio data set, and check the timestamp of the expected playback time to judge the validity of the timestamp data. If it is valid, enter step 123; otherwise, continue to step 122. 122;

123. Record the start decoding time of audio data decoding, and check the timestamp of the start decoding time to judge the validity of the timestamp data. If it is valid, enter step 124, otherwise, continue to step 123;

124. Obtain the basic reference playing time;

125. Adjust the basic reference playback time according to the reference offset time (in essence, adjust the reference time to start decoding);

126. Obtain the actual offset time;

127. Determine the error offset time of the reference playback time according to the reference offset time and the actual offset time, and continue to adjust the basic reference playback time according to the error offset time on the basis of adjusting the reference playback time (essentially or adjust the reference time for starting decoding), obtain the basic reference playback adjustment time, and check the validity of the basic reference playback adjustment time, if invalid, proceed to step 127, and if valid, proceed to step 128;

128. Update the basic reference playback adjustment time to the audio playback hardware device, and determine the adjusted reference playback time along with the actual decoding processing time.

Wherein, in the adjustment process, first adjust the reference time for starting decoding according to the reference offset time, and then determine the adjusted error offset time according to the actual decoding processing time; then, continue to adjust according to the error offset time, The accuracy of adjusting the reference time to start decoding can be improved, that is, the accuracy of the final reference playback time of audio data can be improved.

In order to better implement the audio playing method of the present application, the present application also provides an audio playing device based on the above audio playing method. The meanings of the nouns are the same as those in the above audio playing method, and for specific implementation details, please refer to the description in the method embodiments.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of an audio playback device provided by the present application, wherein the audio playback device may include a decoding module 201, a recording module 202, an acquisition module 203, an adjustment module 204, a determination module 205, a timing module 206 and Play module 207, specifically can be as follows:

The decoding module 201 is configured to decode an audio data set to be played on a terminal device, where the audio data set includes at least one frame of audio data.

The recording module 202 is configured to record the decoding reference time and the actual decoding processing time of the audio data decoding processing of each frame.

Optionally, in some embodiments of the present application, the recording module 202 is specifically used to:

Among them, since the reference time is updated after receiving the time of the reference clock, and time consumption is also required in the process of data transmission and update, it is not accurate to obtain the actual decoding time of audio data according to the reference time. Therefore, this A timing device based on the beating of the crystal oscillator is applied to realize the timing of the actual decoding time of the audio data, and the timing processing using the beating feedback of the crystal oscillator is more accurate.

An acquisition module 203, configured to acquire a reference offset time and an expected playback time corresponding to the audio data;

Among them, since the decoding time of audio data is at the level of milliseconds, the adjustment of the reference time is also a millisecond difference that is difficult for the human body to perceive. For example, when the original reference playback time of audio data is 7:0500ms, that is, the reference clock The display time of the clock is 7:00, and the reference offset time is 200ms. Therefore, the adjusted reference playback time is 7:0300ms. At this time, the display time of the reference clock is still 7:00, and the millisecond level of 200ms Adjustment is difficult for users to perceive, but the time of 200ms is difficult to ignore for audio decoding. Since, when the original reference playback time exceeds the expected playback time, the audio data cannot be played accurately, so you can adjust that part of the reference The method of shifting time adjusts the reference playback time so that the audio data can be played, and provides enough time for audio data decoding, reducing the performance requirements for hardware products, and it is also difficult for users to perceive this part of the time difference.

Optionally, in some embodiments of the present application, the audio data includes current audio data, and the obtaining module 203 includes:

It is used for sampling the at least one frame of historical audio data.

Optionally, in some embodiments of the present application, the acquiring module 203 is specifically used to:

Optionally, in some embodiments of the present application, the acquiring module 203 is specifically further configured to:

an audio data type for identifying said audio data;

The adjustment module 204 is configured to adjust the decoding start reference time according to the reference offset time for each frame of audio data to obtain the decoding start adjustment time.

Among them, the adjustment of the start decoding reference time can be adjusted according to the reference offset time, and can also be adjusted according to the actual offset time, or on the basis of the adjustment of the reference offset time, the error offset time can be adjusted to ensure the reference playback Accuracy of time adjustments.

Optionally, in some embodiments of the present application, the reference offset time includes an actual offset time, and the adjustment module 204 includes:

The third determining subunit is configured to determine the reserved processing time of the audio data according to the reference time of starting decoding of the audio data and the expected playback time.

A reserved processing time for obtaining said audio data;

A determining module 205, configured to determine a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

Optionally, in some embodiments of the present application, the determination module 205 is specifically used to:

A timing module 206, configured to update the reference playing time according to time changes;

The playing module 207 is configured to play the audio data according to the terminal device when the adjusted reference playing time reaches a desired playing time.

In the embodiment of the present application, the audio data acquired by the terminal device is firstly decoded by the decoding module 201. At the same time, the recording module 202 records the decoding reference playback time before the audio data decoding process and the actual decoding processing time of the decoding process. Secondly, The expected playing time of each frame of audio data and the corresponding reference offset time of each frame of audio data are extracted from the audio data by the acquisition module 203, and then the adjustment module 204 adjusts the reference playback time for starting decoding according to the reference offset time ( turn down), so that the start decoding reference play time is no longer synchronized with the time of the reference clock, and then, the determination module 204 determines the reference play time after the decoding process of the audio data according to the start decoding adjustment time and the actual decoding processing time (that is, the reference The playback time is determined according to the start decoding adjustment time and the actual decoding processing time, rather than being synchronized with the time of the reference clock), at the same time, the timing module 206 re-times the adjusted reference playback time, when the re-timed reference playback After the current time corresponding to the time is equal to the expected playback time of the audio data, the corresponding audio data is played by the playback module 207; in the embodiment of the present application, the reference playback time is changed with time by reducing the reference playback time of the audio data , and after the change reaches the expected playback time, the audio data is played, that is, when the reference playback time of the audio data meets the expected playback time, the audio data is played synchronously; wherein, the reference playback time is based on the start of decoding reference time. Small and small, and the actual decoding processing time of the audio data is not affected or limited, therefore, this method of reducing the reference playback time corresponding to the audio data provides sufficient decoding time for the decoding processing of the audio data , to avoid audio and video out-of-sync, frame loss, or stuttering in the playback of audio data due to the long decoding processing time.

In addition, the present application also provides an electronic device, as shown in FIG. 6 , which shows a schematic structural diagram of the electronic device involved in the present application, specifically:

The electronic device may include a processor 401 of one or more processing cores, a memory 402 of one or more computer-readable storage media, a power supply 403, an input unit 404 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 6 does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange different components. in:

The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device, by running or executing software programs and/or modules stored in the memory 402, and calling the Data, perform various functions of electronic equipment and process data, so as to monitor electronic equipment as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 401 .

The memory 402 can be used to store software programs and modules, and the processor 401 executes various functional applications and decoding processing by running the software programs and modules stored in the memory 402 . The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required by at least one function; Data created by the use of electronic devices, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 402 may further include a memory controller to provide the processor 401 with access to the memory 402 .

The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

The electronic device can also include an input unit 404, which can be used to receive input numbers or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may also include a display unit, etc., which will not be repeated here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the executable file stored in the The application program in memory 402, thereby realizes various functions, as follows:

Perform decoding processing on the audio data set to be played on the terminal device, the audio data set includes at least one frame of audio data; record the start decoding reference time and actual decoding processing time of each frame of the audio data decoding process; obtain the audio The reference offset time and expected playback time corresponding to the data; for each frame of audio data, adjust the reference time to start decoding according to the reference offset time to obtain the adjustment time to start decoding; adjust the time according to the start decoding and The actual decoding processing time determines the reference playback time of the audio data; the reference playback time is updated according to time changes; when the adjusted reference playback time reaches the expected playback time, according to the terminal device Play the above audio data.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not repeated here.

By reducing the reference playback time of the audio data, the reference playback time changes with time, and after the change reaches the expected playback time, the audio data is played, that is, when the reference playback time of the audio data meets the expected playback time, the The audio data is played synchronously; the reference playback time is reduced according to the reduction of the decoding reference time, and the actual decoding processing time of the audio data is not affected or limited. Therefore, this kind of audio data corresponds to the reference The method of reducing the playback time provides enough decoding time for the decoding process of the audio data, and avoids audio and picture asynchrony, frame loss or freezes in the playback of audio data due to too long decoding processing time.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, and the instructions can be stored in a computer-readable storage medium, and is loaded and executed by the processor.

To this end, the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any audio playback method provided in the present application. For example, the command can perform the following steps:

Wherein, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or CD, etc.

Due to the instructions stored in the storage medium, the steps in any audio playback method provided in this application can be executed, therefore, the beneficial effects that can be achieved by any audio playback method provided in this application can be realized. See the previous embodiments, and details will not be repeated here.

A kind of audio playing method, device, electronic equipment and storage medium provided by this application have been introduced in detail above. In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for To help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application range. In summary, the content of this specification is not It should be understood as a limitation of the present invention.

The above is only the preferred embodiment of the application, and does not limit the patent scope of the application. Any equivalent structure or process transformation made by using the specification and drawings of the application, or directly or indirectly used in other related technical fields , are all included in the patent protection scope of the present application in the same way.

Claims

An audio playback method, characterized in that, comprising:

Decoding the audio data set to be played on the terminal device, the audio data set includes at least one frame of audio data;

Record the start decoding reference time and the actual decoding processing time of the audio data decoding processing of each frame;

Acquiring the reference offset time and expected playing time corresponding to the audio data;

For each frame of audio data, adjust the decoding start reference time according to the reference offset time to obtain the decoding start adjustment time;

determining a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time, and the reference playback time is less than or equal to the expected playback time;

updating the reference playing time according to the time change;

When the adjusted reference playing time reaches a desired playing time, the audio data is played according to the terminal device.
The method according to claim 1, wherein the audio data includes current audio data, and the acquiring the reference offset time corresponding to the audio data includes:

Sampling at least one frame of historical audio data, where the historical audio data is of the same data type as the current audio data;

Respectively obtain the historical reference playback time and the historical expected playback time after the decoding of the historical audio data of each frame;

According to the historical reference playing time and the historical expected playing time, respectively determine the offset time of each frame of the historical audio data;

A reference offset time of the current audio data is determined according to at least one of the offset times.
The method according to claim 2, wherein the determining the reference offset time of the current audio data according to at least one of the offset times comprises:

determining an average value of at least one of said offset times to obtain an average offset time;

A reference offset time of the current audio data is determined according to the average offset time.
The method according to claim 2, wherein the determining the reference offset time of the current audio data according to at least one of the offset times comprises:

determining an average value of at least one of said offset times to obtain an average offset time;

Rounding the average offset time to obtain the average integer offset time corresponding to the historical audio data;

The average integer offset time is determined as a reference offset time of the current audio data.
The method according to claim 2, wherein the determining the reference offset time of the current audio data according to at least one of the offset times comprises:

respectively rounding the at least one offset time to obtain at least one integer offset time;

determining an integer average offset time of each frame of audio data based on the at least one integer offset time;

The integer average offset time is determined as a reference offset time of the current audio data.
The method according to claim 2, wherein said sampling at least one frame of historical audio data comprises:

Acquiring historical audio data according to a preset frequency, the historical audio data including at least one frame;

Sampling the at least one frame of historical audio data.
The method according to claim 1, wherein the reference offset time includes an actual offset time, and the reference time for starting decoding is adjusted according to the reference offset time to obtain an adjustment time for starting decoding, include:

Acquiring the reserved processing time of the audio data;

determining an actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

The decoding start reference time is adjusted according to the actual offset time to obtain the decoding start adjustment time.
The method according to claim 7, wherein said obtaining the reserved processing time of said audio data comprises:

The reserved processing time of the audio data is determined according to the decoding start reference time of the audio data and the expected playing time.
The method according to claim 1, wherein the adjusting the reference time for starting decoding according to the reference offset time to obtain the adjustment time for starting decoding comprises:

Acquiring the reserved processing time of the audio data;

determining an actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

determining an error offset time of the audio data according to the reference offset time and the actual offset time;

The decoding start reference time is adjusted according to the reference offset time and the error offset time to obtain an adjustment start decoding time.
The method according to claim 1, wherein the recording the actual decoding processing time of the audio data decoding processing of each frame comprises:

Timing feedback is performed according to the beating of the crystal oscillator, and the start decoding time and the end decoding time of the audio data are obtained respectively;

The actual decoding processing time of the audio data is determined according to the decoding start time and the decoding end time.
The method according to claim 1, wherein the determining the reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time comprises:

Obtain the start decoding time and end decoding time of the audio data decoding process;

determining a basic reference playback time of the audio data according to the start decoding adjustment time and the start decoding time;

A reference playback time of the audio data is determined according to the basic reference playback time and the end decoding time.
The method according to claim 1, wherein the obtaining the expected playback time corresponding to the audio data comprises:

The expected playing time of the audio data is directly extracted from the audio data.
The method according to claim 1, wherein the acquiring the reference offset time corresponding to the audio data comprises:

identifying an audio data type of the audio data;

Obtain a mapping relationship set, the mapping relationship set includes a mapping relationship between a preset audio data type and a preset reference offset time;

Acquire a reference offset time corresponding to the audio data according to the set of mapping relationships and the audio data type.
An audio playback device, characterized in that it comprises:

A decoding module, configured to decode an audio data set to be played on the terminal device, where the audio data set includes at least one frame of audio data;

A recording module, configured to record the start decoding reference time and the actual decoding processing time of the audio data decoding process of each frame;

An acquisition module, configured to acquire a reference offset time and an expected playback time corresponding to the audio data;

An adjustment module, configured to adjust the decoding start reference time according to the reference offset time for each frame of audio data, to obtain the decoding start adjustment time;

A determination module, configured to determine a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

A timing module, configured to update the reference playing time according to time changes;

A playing module, configured to play the audio data according to the terminal device when the adjusted reference playing time reaches a desired playing time.
The device according to claim 14, wherein the audio data includes current audio data, and the acquiring module includes:

a sampling unit, configured to sample at least one frame of historical audio data, where the historical audio data is of the same data type as the current audio data;

The first acquisition unit is used to respectively acquire the historical reference playback time and the expected playback time after decoding the historical audio data of each frame;

A first determining unit, configured to respectively determine the offset time of each frame of the historical audio data according to the historical reference playing time and the expected playing time;

The second determining unit is configured to determine a reference offset time of the current audio data according to at least one of the offset times.
The device according to claim 15, wherein the second determining unit comprises:

A first determining subunit, configured to determine an average value of at least one offset time to obtain an average offset time;

The second determining subunit is configured to determine the reference offset time of the current audio data according to the average offset time.
The device according to claim 14, wherein the reference offset time comprises an actual offset time, and the adjustment module comprises:

a second acquiring unit, configured to acquire the reserved processing time of the audio data;

A third determining unit, configured to determine the actual offset time of the audio data according to the reserved processing time and the actual decoding processing time;

The third obtaining unit is configured to adjust the decoding start reference time according to the actual offset time to obtain the decoding start adjustment time.
The device according to claim 17, wherein the second acquiring unit comprises:

The third determining subunit is configured to determine the reserved processing time of the audio data according to the audio data decoding start reference time and the expected playing time.
An electronic device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein, when the processor executes the program, it realizes:

Decoding the audio data set to be played on the terminal device, the audio data set includes at least one frame of audio data;

Record the start decoding reference time and the actual decoding processing time of the audio data decoding processing of each frame;

Acquiring the reference offset time and expected playing time corresponding to the audio data;

For each frame of audio data, adjust the decoding start reference time according to the reference offset time to obtain the decoding start adjustment time;

determining a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

updating the reference playing time according to the time change;

When the adjusted reference playing time reaches a desired playing time, the audio data is played according to the terminal device.
A computer-readable storage medium, characterized in that a computer program is stored thereon, wherein when the computer program is executed by a processor, it realizes:

Decoding the audio data set to be played on the terminal device, the audio data set includes at least one frame of audio data;

Record the start decoding reference time and the actual decoding processing time of the audio data decoding processing of each frame;

Acquiring the reference offset time and expected playing time corresponding to the audio data;

For each frame of audio data, adjust the decoding start reference time according to the reference offset time to obtain the decoding start adjustment time;

determining a reference playback time of the audio data according to the start decoding adjustment time and the actual decoding processing time;

updating the reference playing time according to the time change;

When the adjusted reference playing time reaches a desired playing time, the audio data is played according to the terminal device.