JP4497885B2 - Signal processing device - Google Patents

Description

  The present invention relates to a signal processing device, and more particularly to a signal processing device that receives and processes an image signal and an audio signal from a server in real time as in a surveillance camera system, for example.

An example of a conventional signal processing apparatus of this type is disclosed in Patent Document 1. In this prior art, the server decomposes the image and sound into a plurality of objects, and sequentially transmits these objects to the terminal. The terminal sequentially receives the sent objects, and reconstructs images and sounds from the plurality of received objects. When the bandwidth of the transmission path is narrow, an object requiring high real-time property is preferentially transmitted. Thereby, the transmission quality fall at the time of a narrow band is suppressed, As a result, a user's discomfort is eased.
JP 2001-359067 [H04N 7/15, H04J 3/00, H04L 29/06, H04N 7/24, H04N 7/18]

    However, in the prior art, since the server controls the transmission order of objects to the terminals, particularly when the number of terminals is large, a heavy burden is placed on the server.

  Therefore, a main object of the present invention is to provide a signal processing apparatus that can acquire an image signal and an audio signal from a server without imposing a burden on the server, and can perform reproduction with less sense of incongruity.

The present invention relates to an image requesting means for periodically outputting an image transfer request to a capturing device in a signal processing device connected through a communication line to a capturing device that captures an image signal and an audio signal in real time, and an audio transfer request Audio request means for periodically outputting to the capture device, an image signal transferred from the capture device in response to an image transfer request, and an audio signal transferred from the capture device in response to an audio transfer request e Bei control means for controlling an output period of the image transfer request on the basis of the amount of signal processing means, and is transferred from the capture device and the audio signal not yet subjected to a predetermined process for performing predetermined processing, the processing means Is an image reproducing means for reproducing the image signal transferred from the capturing device, an audio reproducing means for reproducing the audio signal transferred from the capturing device, and an audio signal reproduced by the audio reproducing means. Audio output means for outputting, and image output means for outputting the image signal reproduced by the image reproduction means in synchronization with the audio signal output from the audio output means, and the control means has a signal amount based on the audio signal The signal processing apparatus is characterized in that the output cycle is lengthened when a threshold set based on the quality of an image based on audio and image signals is not satisfied .

In this invention, it is connected via a communication line to a capture device that captures image signals and audio signals in real time, the image request means periodically outputs an image transfer request to the capture device, and the audio request means requests the audio transfer. Output periodically to the capture device. The processing means performs predetermined processing on the image signal transferred from the capture device in response to the image transfer request and the audio signal transferred from the capture device in response to the audio transfer request. The control means controls the output cycle of the image transfer request based on the signal amount of the audio signal transferred from the capturing device and not yet subjected to the predetermined processing.

According to the present invention, in the signal processing device, the control means has a feature of extending the output cycle when the signal amount is less than a threshold value.

The present invention, in the signal processing device, the processing means, the image reproducing means for reproducing the image signal transferred from the capture device, and further comprising a sound reproducing means for reproducing a sound signal transferred from the capture device.

In the present invention, in the signal processing apparatus, the image reproduction means reproduces the transfer image signal, and the sound reproduction means reproduces the transfer sound signal.

The present invention further includes registration means for registering, in the order of transfer, an identifier of an image signal transferred from the capture device and an identifier of an audio signal transferred from the capture device in the signal processing device.

In the present invention, in the signal processing apparatus, the registration means registers the identifier of the transfer image signal and the identifier of the transfer audio signal in the order of transfer. The reproduction means selects a signal to be reproduced next by referring to the identifier registered in the registration means.

According to the present invention, in the signal processing apparatus, the processing unit outputs an audio signal reproduced by the audio reproducing unit, and an audio output from the audio output unit outputs the image signal reproduced by the image reproducing unit. Image output means for outputting in synchronization with the signal is further included.

According to the present invention, in the signal processing device, the audio signal has an arbitrary time length, and time information based on a cumulative time length of the audio signal output before the image signal is added to the image signal, and the image output means Detects the output timing of the image signal based on the time information.

According to the present invention, in the signal processing apparatus, when the audio signal has an arbitrary time length, the accumulated time length of the audio signal output before the image signal is added to the image signal as time information. Since the image output means detects the output timing of the image signal based on this time information, the image signal can be output in synchronization with the audio signal.

  According to the present invention, the output period of the image transfer request is controlled based on the signal amount of the audio signal transferred from the capture device and not yet subjected to the predetermined processing. Since the image request cycle is lengthened when less than the above, it is possible to reduce the possibility that the sound is interrupted when the bandwidth of the communication line is narrowed.

  In this case, although the quality of the image is reduced, the user feels less discomfort than when the sound is interrupted. Moreover, since the capture device only has to transfer the requested signal, the burden on the capture device does not increase.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  With reference to FIG. 1, the monitoring camera system 10 of this embodiment includes a server 16 and a terminal 60. A camera 12 and a monitor 14 are connected to the server 16. The camera 12 captures the scene and outputs an image signal to the server 16. The monitor 14 receives the image signal from the server 16 and displays the scene captured by the camera 12 on the screen.

  The server 16 is further connected to a microphone 52 via an amplifier 50 and a speaker 56 via an amplifier 54. The microphone 52 captures sound waves generated in the object scene and outputs an audio signal to the amplifier 50. The amplifier 50 amplifies the audio signal from the microphone 52 and outputs the amplified audio signal to the server 16. The amplifier 54 receives the audio signal from the server 16 and drives the speaker 56 with this signal. Thereby, the sound wave captured by the microphone 52 is output from the speaker 56.

  Server 16 and terminal 60 are connected via network 58. The image signal input from the camera 12 to the server 16 and the audio signal input from the microphone 52 to the server 16 through the amplifier 50 are also transmitted to the terminal 60 through the network 58.

  A monitor 90 is connected to the terminal 60. The monitor 90 receives an image signal from the terminal 60 and displays the scene captured by the camera 12 on the screen. A speaker 94 is further connected to the terminal 60 via an amplifier 92. The amplifier 92 receives the audio signal from the server 16 and drives the speaker 94 with this signal. As a result, sound waves captured by the microphone 52 are output from the speaker 94.

  Next, the configuration of the server 16 will be described. Server 16 includes two data buses 46 and 48. The JPEG codec 18 is connected to the data bus 46, and the CPU 42 and the flash memory 32 are connected to the data bus 48.

  The data bus 46 further includes an HDD 26 via an IDE-I / F circuit 24, a camera 12 via a D1-I / F (IN) circuit 20, and a D1-I / F (OUT) circuit 22. The monitor 14 is connected to the SDRAM 30 via the SDRAM controller 28, and the audio A / D converter 36 and the audio D / A converter 38 via the audio I / F circuit 40.

  An SDRAM 30 is further connected to the data bus 48 via the SDRAM controller 28. Further, a network controller 44 is connected to the data bus 48 via the expansion bus I / F circuit 34.

  In the server 16, the JPEG codec 18, the audio A / D converter 36, and the audio D / A converter 38 are connected to a data bus 46 that is different from the data bus 48 to which the CPU 42 is connected. We are trying to improve the speed. The hardware on the data bus 46 side and the hardware on the data bus 48 side can exchange data with each other through the SDRAM controller 28.

  Next, the configuration of the terminal 60 will be described. Terminal 60 includes a data bus 88. A JPEG codec 62, a network controller 64, a flash memory 66, an operation panel 84, and a CPU 86 are connected to the data bus 88.

  The data bus 88 further includes an HDD 78 via an IDE-I / F circuit 76, a monitor 90 via a D1-I / F circuit 70, an SDRAM 82 via an SDRAM controller 80, and an audio I / F circuit 72. Audio D / A converters 74 are connected to each other.

  Next, each element constituting the server 16 will be described. The D1-I / F (IN) circuit 20 converts the image signal from the camera 12 into a signal suitable for the data bus 46. The D1-I / F (OUT) circuit 22 converts the image signal from the data bus 46 into a signal suitable for the monitor 14.

  The IDE-I / F circuit 24 receives an instruction from the CPU 42 and controls the HDD 26. The HDD 26 records an image signal (JPEG file) and an audio signal (WAVE file) from the data bus 46 on the HD 26a according to control of the IDE-I / F circuit 24, and reads the image signal and audio signal from the HD 26a. The data is output to the data bus 46.

  The audio A / D converter 36 A / D converts the audio signal from the amplifier 50. The audio I / F circuit 40 converts the audio signal from the audio A / D converter 36 into a signal suitable for the data bus 46. The audio I / F circuit 40 also converts the audio signal from the data bus 46 into a signal suitable for the audio D / A converter 38. The audio D / A converter 38 D / A converts the audio signal from the audio I / F circuit 40.

  The SDRAM 30 includes a plurality of storage areas (banks), and audio signals, image signals, and the like are held in the plurality of storage areas. Details of the SDRAM 30 will be described later.

  In response to an instruction from the CPU 42, the SDRAM controller 28 writes the image signal and audio signal from the data bus 46 to the SDRAM 30, reads the image signal and audio signal from the SDRAM 30, and outputs them to the data bus 46 or the data bus 48.

  The JPEG codec 18 encodes the image signal input from the data bus 46 in accordance with the JPEG method, and outputs the image signal obtained by encoding to the data bus. The encoded image signal input from the data bus 46 is decoded according to the JPEG method, and the image signal obtained by decoding is output to the data bus 46.

  Here, the encoding process and the decoding process will be briefly described. The image signal is encoded by the following procedure. The JPEG codec 18 first blocks the image signal. Next, each of the blocks is subjected to DCT transformation, and each of the DCT coefficients obtained thereby is quantized. Next, each quantized DCT coefficient is entropy encoded. Then, the JPEG file is formed by arranging the entropy-encoded data in a predetermined order and attaching additional information such as a header. One JPEG file formed in this way corresponds to one image frame. When decoding the encoded image signal, the JPEG codec 18 performs a process reverse to the above encoding.

  In this embodiment, the JPEG method is adopted, but the JPEG 2000 method or other encoding methods may be used. In either case, the recording time 56a is embedded in the header or additional information area of the encoded image signal.

  Returning to the description of the components of the server 16 again. The expansion bus I / F circuit 34 mediates data exchange between the network controller 44 and the data bus 48. The network controller 44 receives a request signal issued by the terminal 60 from the network 58. In response to an instruction from the CPU 42, the image signal and audio signal held in the SDRAM 30 are sent to the network 58.

  The flash memory 32 stores a program describing the processing procedure of the CPU 42. The CPU 42 controls each of the above components according to the program in the flash memory 32. As a result, various processes such as recording and playback, and transmission of a JPEG file and a WAVE file are realized in the server 16.

  Next, each element constituting the terminal 60 will be described. The D1-I / F circuit 70 converts the image signal from the data bus 88 into a signal suitable for the monitor 90. The IDE-I / F circuit 76 receives an instruction from the CPU 86 and controls the HDD 78. The HDD 78 records the image signal (JPEG file) and the audio signal (WAVE file) from the data bus 88 on the HD 78a according to the control of the IDE-I / F circuit 76, and reads the image signal and the audio signal from the HD 78a. The data is output to the data bus 88.

  The audio I / F circuit 72 converts the audio signal from the data bus 88 into a signal suitable for the audio D / A converter 74. The audio D / A converter 74 D / A converts the audio signal from the audio I / F circuit 72 and outputs the converted audio signal to the amplifier 92.

  The SDRAM 82 includes a plurality of storage areas (banks), and audio signals, image signals, and the like are held in the plurality of storage areas. Details of the SDRAM 82 will be described later.

  In response to an instruction from the CPU 86, the SDRAM controller 80 writes the image signal and audio signal from the data bus 88 to the SDRAM 82, reads the image signal and audio signal from the SDRAM 82, and outputs them to the data bus 88.

  The JPEG codec 62 encodes the image signal input from the data bus 88 according to the JPEG method, and outputs the encoded image signal to the data bus 88. Also, the encoded image signal input from the data bus 88 is decoded according to the JPEG method, and the image signal obtained by decoding is output to the data bus 88. The encoding process and decoding process here are the same as those by the JPEG codec 18.

  The network controller 64 transmits a request signal to the server 16 via the network 58 in response to an instruction from the CPU 86. Then, an image signal and an audio signal are received from the server 16 via the network 58, and the received signal is output to the data bus 88.

  The operation panel 84 includes a Play key 84a, a Stop key 84b, and a numeric keypad 84c. The numeric keypad 48c includes ten numeric keys and several symbol keys. When any key on the operation panel 84 is pressed, a signal corresponding to the pressed key is transmitted from the operation panel 84 to the CPU 86.

  The flash memory 66 stores a reproduction list (LIST) 68. The reproduction list 68 will be described later.

  The flash memory 66 further stores a program describing the processing procedure of the CPU 86. The CPU 86 controls each of the above components in accordance with a signal from the operation panel 84 and in accordance with a program in the flash memory 66. Thereby, in the terminal 60, various processes such as acquisition of a JPEG file and a WAVE file from the server 16 and recording and reproduction of the acquired file are realized.

  Next, the SDRAM 30 in the server 16 will be described. Referring to FIG. 2, SDRAM 30 includes an audio buffer 30a and an image buffer 30b. Audio signals and image signals from the D1-I / F (IN) circuit 20 and the audio A / D converter 36 are stored in the audio buffer 30a and the image buffer 30b, respectively. Audio signals and image signals in the audio buffer 30a and the image buffer 30b are read out to the D1-I / F (OUT) circuit 22 and the audio D / A converter 38, respectively, in accordance with instructions from the CPU 42. As a result, a live image is displayed on the monitor 14, and live sound is output from the speaker 56.

  If there is a WAVE request from the terminal 60, all audio signals stored in the audio buffer 30a at that time are read to the network controller 44, and a WAVE file including the audio signals is generated. The generated WAVE file is sent to the network 58 by the network controller 44.

  However, a first threshold is set in the audio buffer 30a. If the accumulated amount of the audio buffer 30a does not exceed the first threshold, reading from the audio buffer 30a is not performed, and the accumulated amount exceeds the first threshold. This is done at some point. The details of the audio signal reading process from the audio buffer 30a to the network controller 44 will be described later.

  Furthermore, if there is a JPEG request from the terminal 60, the image signal corresponding to the frame currently being output to the D1-I / F (OUT) circuit 22 is read from the image buffer 30b to the JPEG codec 18. The JPEG codec 18 encodes the image signal, and further generates a JPEG file including the encoded image signal. The generated JPEG file is sent to the network 58 through the network controller 44.

  Next, the SDRAM 82 in the terminal 60 will be described. Referring to FIG. 3, SDRAM 82 includes an audio buffer 82a and an image buffer 82b. The audio buffer 82a and the image buffer 82b store the image signal of the JPEG file received from the network 58 by the network controller 64 and the audio signal of the WAVE file similarly received.

  The audio signal and the image signal accumulated in the audio buffer 82a and the image buffer 82b are sequentially read and reproduced according to a reproduction list 68 (described later) in the flash memory 66.

  However, a second threshold value and a third threshold value are set in the audio buffer 82a. If the accumulated amount of the audio buffer 82a does not exceed the second threshold value, reading from the audio buffer 82a is not performed, and this is executed when the second threshold value is exceeded. When the accumulated amount of the audio buffer 82a is below the third threshold at the transmission timing of the JPEG request, the JPEG request is skipped and a WAVE request is transmitted instead.

  Note that the greater the third threshold value, the less likely the sound is to be interrupted, and the smaller the third threshold value, the less likely the image dropout occurs. In this embodiment, the third threshold is set to, for example, 0.8 times the second threshold in consideration of the quality balance between sound and image.

  Next, an audio signal reading process from the audio buffer 30a to the network controller 44 side will be described. Referring to FIG. 4, as the audio signal is written to audio buffer 30a, the write pointer (dotted arrow) moves on audio buffer 30a. Further, as the audio signal is read from the audio buffer 30a, the read pointer (audio_adr; solid arrow) moves after the write pointer. The accumulated amount D of the audio buffer 30a at that time is given by the difference between the write pointer and the read pointer.

  More specifically, at time t = 2, the write pointer is at the “2” position and the read pointer is at the “0” position. Therefore, the accumulation amount D is “2”, and immediately after this, an audio signal for 2 seconds is read from the audio buffer 30a.

  At t = 3, as a result of writing an audio signal for one second in one second at t = 2 to 3, the write pointer has moved to the position “3”. On the other hand, the read pointer has moved to the position “2” as a result of reading out the audio signal for 2 seconds in 1 second from t = 2 to 3. Accordingly, the accumulation amount D is “1”, and immediately after this, an audio signal for one second is read from the audio buffer 30a.

  At t = 4, as a result of writing an audio signal for one second in one second at t = 3 to 4, the write pointer has moved to the position “4”. On the other hand, the read pointer has moved to the position “3” as a result of reading the audio signal for one second in one second at t = 3 to 4. Accordingly, the accumulation amount D is “1”, and immediately after this, an audio signal for one second is read from the audio buffer 30a.

  Next, the reproduction list 68 will be described. Referring to FIG. 5, the reproduction list 68 includes a JPEG column 68a, a WAVE column 68b, and a list number column 68c. List numbers list [1], list [2],... Are assigned to the list number column 68c. When a JPEG file or a WAVE file is stored in the SDRAM 82, the identifier of the file is assigned to the list numbers list [1], list [2],.

  For example, a JPEG file (identifier: 0001.jpg), a WAVE file (identifier: 0001.wav), a WAVE file (identifier: 0002.wav), a WAVE file (identifier: 0003.wav), etc. are received in this order. Assume that the data is stored in the corresponding buffer in the SDRAM 82. In this case, first, the file identifier 0001. jpg is registered, and then the file identifier 0001. wav is registered, and the file identifier 0002. wav is registered, and the file identifier 0003. wav is registered.

  A reproduction pointer list [n] indicated by an arrow in the figure indicates any one of list numbers list [1], list [2],. The file assigned to the list number pointed to by the playback pointer list [n] becomes the next file to be played.

  The operation of the surveillance camera system 10 configured as described above is the same as that of the overall configuration diagram of FIG. 1, the processing flowchart of the server CPU 42 shown in FIGS. 6 to 8, and the operation of the terminal CPU 86 shown in FIGS. This will be described below based on the processing flowchart and the timing chart of FIG.

  With reference to FIG. 1, this surveillance camera system 10 is used in, for example, an office building or an apartment house. The camera 12 is installed in, for example, a front door or a passage, and the server 16 and the monitor 14 are installed in, for example, a management room. Note that the number of cameras 12 is not limited to one. When a plurality of cameras 12 are installed, each camera 12, 12,... Is connected to the D1-I / F (IN) circuit 20 via a multiplexer (not shown).

  The network 58 is, for example, an intranet or the Internet, and the terminal 60 is installed at any place that can be connected to the network 58. Therefore, the user can view the monitoring image not only in the management room but also at an arbitrary place through the terminal 60.

  In the server 16, a plurality of operation modes such as a real-time playback mode and a recording / playback mode are prepared, and one of the modes is selected. When the real-time playback mode is selected, the monitoring image captured by the camera 12 and the sound captured by the microphone 52 are temporarily stored in the buffer and then played back, and are displayed and output in real time through the monitor 14 and the speaker 56. The

  In the recording / reproducing mode, the monitoring image and audio are displayed and output in real time as described above, and at the same time, subjected to compression and / or encoding processing as necessary and recorded on the HD 26a. The monitoring image and sound recorded on the HD 26a are read and reproduced when necessary. In the following description, it is assumed that the server 16 is operating in the real-time playback mode.

  The terminal 60 issues a browsing request for the currently monitored monitoring image and audio to the server 16 that is reproducing the monitoring image and audio in real time. The browsing request is composed of a plurality of file requests transmitted at intervals of 1 second. The server 16 transmits the requested file to the terminal 60.

  There are two types of file requests, JPEG requests and WAVE requests, and the terminal 60 basically transmits JPEG requests and WAVE requests alternately. However, if the transmission bandwidth of the network 58 becomes narrow and only a part of the requested file can be transmitted, the scheduled JPEG request is skipped and a WAVE request is sent instead. As a result, under a narrow band situation, the WAVE file is preferentially transmitted from the server 16 over the JPEG file. As a result, the terminal 60 can continue to reproduce the sound without interruption even though the frames of the monitoring image are dropped.

  Specifically, the CPU 42 of the server 16 performs the following processing. Referring to FIG. 6, when server 16 is activated, CPU 42 executes an initial process in step S1. The initial processing includes processing such as instructing the SDRAM controller 28 to initialize the SDRAM 30, instructing the network controller 44 to establish connection with the network 58, and selecting a reproduction mode. In this embodiment, the real-time playback mode is selected.

  In step S3, the CPU 42 determines whether an image signal is input from the D1-I / F (IN) circuit 20. If the determination result is affirmative, the process proceeds to step S5, and if negative, the process proceeds to step S7. In step S <b> 5, the CPU 42 stores the image signal in the image buffer 30 b in the SDRAM 30 through the SDRAM controller 28. Then, it progresses to step S7.

  In step S <b> 7, the CPU 42 determines whether audio data is input from the audio I / F circuit 40. If the determination result is affirmative, the process proceeds to step S9, and if negative, the process proceeds to step S11. In step S <b> 9, the CPU 42 stores the audio data in the audio buffer 30 a in the SDRAM 30 through the SDRAM controller 28. Then, it progresses to step S11.

  In step S11, the CPU 42 instructs the SDRAM controller 28 to send the image signal stored in the image buffer 30b from the D1-I / F (OUT) circuit 22 to the monitor 14, and the audio data stored in the audio buffer 30a. The audio signal is output to the audio D / A converter 38 through the audio I / F circuit 40. In this way, the monitoring image is displayed on the monitor 14 and at the same time, sound is output from the speaker 56.

  In step S <b> 13, the CPU 42 determines whether the network controller 44 has received a JPEG request from the terminal 60. If the determination result is affirmative, the process proceeds to step S15, and if negative, the process proceeds to step S17. In step S15, the JPEG file is transmitted to the terminal 60. Thereafter, the process proceeds to step S17. Details of the JPEG file transmission process will be described later.

  In step S <b> 17, the CPU 42 determines whether the network controller 44 has received a WAVE request from the terminal 60. If the determination result is affirmative, the process proceeds to step S19, and if negative, the process proceeds to step S21. In step S19, the WAVE file is transmitted to the terminal 60. Then, it progresses to step S21. The details of the WAVE file transmission process will be described later.

  In step S21, the CPU 42 determines whether or not to continue the process. If the determination result is affirmative, the process returns to step S3 to repeat the same processing as above, and if negative, the processing is terminated.

  Next, the JPEG file transmission process in step S15 will be described. Referring to FIG. 7, in step S41, CPU 42 instructs JPEG codec 18 to execute encoding of an image signal corresponding to a frame currently being output to monitor 14 side. In response, the JPEG codec 18 reads the corresponding image signal from the SDRAM 30 through the SDRAM controller 28, and encodes the read image signal according to the JPEG method. The encoded image signal is temporarily stored in the SDRAM 30.

  In step S43, the CPU 42 generates a JPEG file based on the encoded image signal. At that time, a time stamp (wave_time) is attached to the end of the encoded image signal.

  Here, the time stamp is used to synchronize the image and the sound when reproduction processing is performed on the terminal 60 side. Specifically, the terminal 60 is provided with a plurality of modes such as a mode for reproducing images and sounds in real time, a mode for recording images and sounds while reproducing them in real time, and a mode for reproducing recorded images and recorded sounds later. Has been. In any mode of reproduction, the terminal 60 detects the output timing of the image signal based on the time stamp attached to the end of the image signal, and outputs the image signal in synchronization with the audio signal.

  In step S <b> 45, the CPU 42 transmits the JPEG file to the terminal 60 through the network 58 through the network controller 44. Then, the process returns to the upper layer routine.

  Next, the WAVE file transmission process in step S19 will be described. Referring to FIG. 8, in step S61, CPU 42 determines whether or not the WAVE request received by network controller 44 is an initial request. Note that a flag is attached to the first WAVE request, and the determination is made by detecting this flag. If the determination result in step S61 is affirmative, the process proceeds to step S63, and if negative, the process proceeds to step S65.

  In step S63, the CPU 42 sets “0” to each of the variable wav_time and the variable audio_adr. Note that the values of variables such as wav_time and audio_adr are held in a register in the CPU 42, for example. Thereafter, the process proceeds to step S65.

  In step S65, the CPU 42 calculates the accumulation amount D of the audio buffer 30a. The accumulation amount D is given by the difference between the write pointer and the read pointer (audio_adr) (see FIG. 4).

  In step S67, the CPU 42 determines whether or not the calculated accumulation amount D is larger than the first threshold value. If the determination result is affirmative, the process proceeds to step S69, and if negative, the process returns to step S65.

  In step S69, the CPU 42 sets {(audio_adr) + D} to the variable audio_adr. In step S71, all audio data is read from the audio buffer 30a through the SDRAM controller 28. In step S73, the time length T of the read audio data is calculated. The time length T is obtained by dividing the read audio data amount D by the bit rate of the audio data.

  In step S75, the CPU 42 sets {(wav_time) + T} to the variable wav_time. In step S77, a WAVE file is generated by attaching additional information such as a time stamp (wav_time) to the audio data read in step S71. In step S79, the WAVE file is transmitted to the terminal 60 through the network 58 through the network controller 44. Thereafter, the process returns to the upper layer routine.

  On the other hand, the CPU 86 of the terminal 60 performs the following processing. The terminal 60 is also prepared with several playback modes. In this embodiment, only the case of performing real-time playback will be described.

  Referring to FIG. 9, when terminal 60 is activated, CPU 86 executes an initial process in step S91. The initial processing includes processing such as instructing the SDRAM controller 80 to initialize the SDRAM 82 and instructing the network controller 64 to establish connection with the network 58.

  In step S93, the CPU 86 determines whether or not the Play key 84a has been pressed. If the determination result is affirmative, the process proceeds to step S95, and if negative, the process proceeds to step S99. In step S95, a file acquisition task is activated. In step S97, a reproduction task is activated. Thereafter, the process proceeds to step S99. The file acquisition task and the reproduction task will be described later.

  In step S99, the CPU 86 executes other processing. In step S101, it is determined whether the file acquisition task and the reproduction task are being executed. If the determination result is negative, that is, if at least one task has been completed, the process returns to step S93. If the determination result is affirmative, that is, if both tasks are still being executed, the process returns to step S99.

  Next, the file acquisition task will be described. Referring to FIG. 10, in step S111, CPU 86 sets “1” to each of variable num, variable wav_n, and variable jpg_n. In step S113, a JPEG file corresponding to the variable jpg_n is requested. In step S115, {(jpg_n) +1} is set to the variable jpg_n. In step S117, it is determined whether the file acquisition by the network controller 64 is completed. If the determination result is negative, the process proceeds to step S119. If the determination result is affirmative, the process proceeds to step S121.

  In step S119, the CPU 86 determines whether or not the stop key 84b has been pressed. If the determination result is affirmative, this task ends. If negative, the process returns to step S117.

  In step S121, the CPU 86 extracts an image signal from the JPEG file acquired by the network controller 64, and stores the extracted image signal in the image buffer 82b. In step S123, {num + 1} is set to the variable num. In step S125, the same image signal as that stored in the image buffer 82b is stored in the HD 78a.

  In step S127, the CPU 86 assigns the identifier {(jpg_n) .n of the corresponding file to the list number list [num] of the reproduction list 68 (see FIG. 5). jpg} is registered. In step S129, it is determined whether or not the Stop key 84b has been pressed. If the determination result is affirmative, this task ends. If negative, the process proceeds to step S131.

  Referring to FIG. 11, in step S131, CPU 86 determines whether or not the value of variable wav_n is “1”. If the determination result is affirmative, the process proceeds to step S133, and if negative, the process proceeds to step S135. In step S133, an initial WAVE request is transmitted, and then the process proceeds to step S135. A flag indicating that this is the first WAVE request is attached to the initial WAVE request.

  In step S135, it is determined whether one second has elapsed since the previous request transmission. If the determination result is affirmative, the process proceeds to step S137, and if negative, the process waits.

  In step S137, the WAVE file corresponding to the variable wav_n, that is, {(wav_n). request wav}. Thereafter, the process proceeds to step S139. In step S139, {(wav_n) +1} is set to the variable wav_n. In step S141, it is determined whether the file acquisition by the network controller 64 is completed. If the determination result is negative, the process proceeds to step S143. If the determination result is affirmative, the process proceeds to step S145.

  In step S143, the CPU 86 determines whether or not the stop key 84b has been pressed. If the determination result is affirmative, this task ends. If negative, the process returns to step S141.

  In step S145, the CPU 86 extracts audio data from the WAVE file acquired by the network controller 64, and stores the extracted audio data in the audio buffer 82a. In step S147, {num + 1} is set to the variable num. In step S149, the same audio data as that stored in the audio buffer 82a is stored in the HD 78a.

  In step S151, the CPU 86 assigns the identifier {(wav_n). register wav}. In step S153, it is determined whether or not the accumulated amount of the audio buffer 82a has exceeded a third threshold value (see FIG. 3). If the determination result is affirmative, the process returns to step S113, and if negative, the process proceeds to step S155.

  In step S155, the CPU 86 determines whether or not the stop key 84b has been pressed. If the determination result is affirmative, this task ends. If the determination result is negative, the process returns to step S131.

  Next, the above reproduction task will be described. Referring to FIG. 12, in step S181, CPU 86 sets 1 to variable n of playback pointer list [n]. The reproduction pointer list [n] indicates one of a plurality of list numbers 68c (list [1], list [2],...) In the reproduction table 68 (see FIG. 5).

  In step S183, the CPU 86 determines whether or not the accumulated amount of the audio buffer 82a exceeds the second threshold (see FIG. 3). If the determination result is affirmative, the process proceeds to step S185, and if negative, the process waits.

  In step S185, it is determined whether the identifier of the JPEG file is registered in the list number 68c pointed to by the reproduction pointer list [n]. If the determination result is affirmative, the process proceeds to step S187, and if negative, the process proceeds to step S191.

  In step S187, the CPU 86 reads out the JPEG file corresponding to the registration identifier from the image buffer 82b. In step S189, (n + 1) is set to the variable n. Thereafter, the process proceeds to step S191.

  In step S191, the CPU 86 determines whether or not the list number 68c pointed to by the reproduction pointer list [n] is blank. If the determination result is affirmative, this task ends. If the determination result is negative, the process proceeds to step S193.

  In step S193, the CPU 86 reads from the audio buffer 82a the WAVE file corresponding to the registration identifier of the list number 68c pointed to by the reproduction pointer list [n]. In step S195, (n + 1) is set to the variable n. In step S197, it is determined whether or not the Stop key 84b has been pressed. If the determination result is affirmative, this task ends. If the determination result is negative, the process proceeds to step S199.

  In step S199, it is determined whether audio reproduction has been completed. If the determination result is affirmative, the process proceeds to step S201, and if negative, the process returns to step S197. In step S201, it is determined whether or not the list number 68c pointed to by the playback pointer list [n] is blank. If the determination result is affirmative, this task ends. If the determination result is negative, the process returns to step S185.

  As is apparent from the description with reference to the flowchart, as a basic operation, the terminal 60 first transmits a JPEG request when a user gives a reproduction instruction, that is, presses the Play key 84a, and then transmits a WAVE request after one second. Thereafter, the JPEG request and the WAVE request are continuously transmitted alternately at intervals of 1 second until the reproduction target file is obtained.

  The server 16 transmits the JPEG file to the terminal 60 via the network 58 when receiving the JPEG request and the WAVE file when receiving the WAVE request. The JPEG file to be transmitted stores the image signal being reproduced at that time. The transmitted WAVE file stores all the audio data stored in the audio buffer 30a at that time.

  When receiving the JPEG file, the terminal 60 extracts an image signal from the received JPEG file and stores the extracted image signal in the image buffer 82b. When the WAVE file is received, the audio data is extracted from the received WAVE file, and the extracted audio data is stored in the audio buffer 82a.

  When the accumulation amount of the audio buffer 82a reaches the second threshold value, the terminal 60 starts a process of reading and reproducing the image signal and audio data stored in the image buffer 82b and the audio buffer 82a. In the playback process, the terminal 60 synchronizes the output timing of the playback image with that of the playback audio based on the time stamp (wave_time) added to the end of the image signal.

  As a characteristic operation, when the terminal 60 performs the file acquisition process and the reproduction process as described above, the accumulation amount of the audio buffer 82a is set to the third threshold value (this is the second threshold value) at the transmission timing of the second and subsequent JPEG requests. If the threshold value is not reached (80%), this JPEG request is skipped. Therefore, during the period from when the first JPEG request is transmitted until the amount of accumulation in the audio buffer 82a reaches the third threshold, a plurality of WAVE requests are transmitted in succession.

  The timing chart of FIG. 13 shows the transmission timing of the JPEG request and the WAVE request immediately after the reproduction instruction, the accumulation status of the audio buffer 82a, and the audio output status. The second threshold value is set to an amount corresponding to an audio signal for 5 seconds. Specifically, if the WAVE audio rate is 7867 bytes / sec, {7867 × 5} bytes is the second threshold. For the sake of simplicity, the accumulation amount in the audio buffer 82a is described as “5 seconds” below. The third threshold value is 4 seconds (= 5 × 0.8).

  Referring to FIG. 13, terminal 60 transmits the first JPEG request when time t = 0. When t = 2, the reception of the first JPEG file is completed, and at the same time, the first WAVE request is transmitted.

  At t = 3, reception of the first WAVE file is completed. The first WAVE file includes an audio signal for 2 seconds. Therefore, the accumulated amount of the audio buffer 82a at this time is 2 seconds. Since this is less than the third threshold, that is, 4 seconds, the second JPEG request is skipped, and the second WAVE request is transmitted instead.

  At t = 3.5, the reception of the second WAVE file is completed. The second WAVE file includes an audio signal for one second. At this point, since only 0.5 seconds have elapsed since the previous request transmission, the third WAVE request is not yet transmitted.

  At t = 4, the accumulated amount of the audio buffer 82a is 3 seconds, which is still less than the third threshold value. Therefore, the second JPEG request is skipped, and the third WAVE request is transmitted instead.

  At t = 4.5, the reception of the third WAVE file is completed. The third WAVE file includes an audio signal for 1 second, and the accumulated amount of the audio buffer 82a is 4 seconds at this point. That is, since the accumulated amount has reached the third threshold value, the second JPEG request is transmitted here.

  At t = 6.5, the reception of the second JPEG file is completed, and at the same time, the fourth WAVE request is transmitted.

  At t = 8, reception of the fourth WAVE file is completed. The fourth WAVE file includes an audio signal for 2.5 seconds, and the accumulated amount of the audio buffer 82a at this time is 6.5 seconds. However, when the accumulated amount exceeds 5 seconds, the audio signal for 2 seconds corresponding to the first WAVE file is read from the audio buffer 82a. As a result, the accumulation amount is 3.5 seconds. This is below the third threshold, so the third JPEG request is skipped and a fifth WAVE request is sent instead.

  At t = 8.5, reception of the fifth WAVE file is completed. The fifth WAVE file includes an audio signal for 1.5 seconds, and the accumulated amount of the audio buffer 82a is 5 seconds at this point. However, since the audio signal for one second corresponding to the second WAVE file is read from the audio buffer 82a, the accumulation amount is 4 immediately after. This has reached the third threshold, so the third JPEG request is sent.

  In the subsequent period of t = 8.5 to 12.5, the same processing as that performed in the period of t = 4.5 to 8.5 is repeated. Thereafter, the same processing is repeated as long as the accumulated amount of the audio buffer 82a changes in the vicinity of the second threshold value.

  However, when a situation occurs in which the accumulated amount of the audio buffer 82a is continuously lower than the second threshold due to a decrease in the bandwidth of the network 58, etc., it was performed in the period of t = 0 to 4.5. The same processing as that described above, that is, processing for transmitting WAVE requests in succession while skipping JPEG requests is performed. As a result, the accumulated amount of the audio buffer 82a is quickly pushed up to the vicinity of the second threshold value, and as a result, interruption of the reproduced audio is avoided.

  As is apparent from the above description, in this embodiment, the terminal CPU 86 periodically acquires an image signal and an audio signal from the server 16 via the network 58 via the network controller 64. The acquired image signal and audio signal are accumulated in the image buffer 82b and the audio buffer 82a of the SDRAM 82, respectively. When the accumulated amount of the audio buffer 82a is less than the third threshold, the CPU 86 lengthens the image signal acquisition cycle by the network controller 64.

  Thereby, when the bandwidth of the network 58 becomes narrow, the possibility that the playback audio is interrupted in the terminal 60 is reduced. In this case, although the quality of the image is reduced, the user's discomfort is less than when the sound is interrupted. Further, since the server 16 simply transmits the requested signal to the terminal 60, the burden on the server 16 does not increase.

  More specifically, the terminal 60 performs real-time playback of moving images and sounds by periodically acquiring JPEG files and WAVE files from the server 16 via the network 58. At this time, the terminal 60 controls the JPEG file acquisition cycle, thereby avoiding audio interruption. In this case, since the client side performs control processing for avoiding voice interruption, the server 16 does not need to perform special control processing, and a simple HTTP server can be used as the server 16.

  In other words, in a general system including an HTTP server and a client, a JPEG file and a WAVE file are periodically transmitted from the HTTP server to the client, and the client side controls the JPEG transmission cycle according to the network bandwidth. In addition, it is possible to perform real-time playback of moving images and audio while avoiding audio interruptions that occur in a narrow band.

It is a block diagram which shows the structure of the surveillance camera system of this Example. It is a memory map figure which shows the structural example of server SDRAM. It is a memory map figure which shows the structural example of terminal SDRAM. It is an illustration figure for demonstrating the writing process with respect to an audio | voice buffer, and a reading process. It is an illustration figure which shows a play list. It is a flowchart which shows a part of process by server CPU. It is a flowchart which shows a part of other process by server CPU. It is a flowchart which shows the other part of the process by server CPU. It is a flowchart which shows a part of process by terminal CPU. It is a flowchart which shows a part of other process by terminal CPU. It is a flowchart which shows the other part of the process by CPU of a terminal. It is a flowchart which shows a part further process by CPU of a terminal. FIG. 4 is a timing chart showing request transmission timing, audio buffer accumulation status, and audio output status at a terminal.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Surveillance camera system 12 ... Camera 14, 90 ... Monitor 16 ... Server 52 ... Microphone 56, 94 ... Speaker 60 ... Terminal

Claims (3)

In a signal processing device connected through a communication line with a capture device that captures image signals and audio signals in real time,
Image request means for periodically outputting an image transfer request to the capture device;
A voice request means for periodically outputting a voice transfer request to the capture device;
Processing means for performing predetermined processing on an image signal transferred from the capture device in response to the image transfer request and an audio signal transferred from the capture device in response to the audio transfer request; and the capture e Bei control means for controlling an output period of the image transfer request on the basis of the signal of the audio signal is and not yet subjected to the predetermined process transfer from the device,
The processing means includes
Image reproduction means for reproducing the image signal transferred from the capture device;
Audio reproduction means for reproducing the audio signal transferred from the capture device;
Audio output means for outputting an audio signal reproduced by the audio reproduction means; and
Image output means for outputting the image signal reproduced by the image reproduction means in synchronization with the audio signal output from the audio output means,
The signal processing apparatus is characterized in that the control means lengthens the output cycle when the signal amount is less than a threshold set based on the quality of the sound based on the sound signal and the image based on the image signal. . Registration means for registering the identifier of the image signal transferred from the capture device and the identifier of the audio signal transferred from the capture device in the order of transfer; and
The signal processing apparatus according to claim 1, further comprising: a reproduction control unit that controls the image reproduction unit and the audio reproduction unit to reproduce the image signal and the audio signal in the order of transfer registered by the registration unit. The audio signal has an arbitrary time length,
Time information based on the cumulative time length of the audio signal output before this image signal is added to the image signal,
The image output means detects the output timing of the image signal based on the time information, the signal processing apparatus according to claim 1 or claim 2, wherein.

Images