Record ogg files continuously

Designing hardware that use VLSI Solution's devices as the system controller for the entire design.
Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Record ogg files continuously

Post by Davides » Mon 2016-10-10 15:12

Hi,

I'm working on a project about a remote live audio streaming system, and I wonder if is it posssible with the encoder (vs1063a) to record continuously audio files of a fixed (approximately) duration (for example 1 second) without losing any audio between a file and another?

I really appreciate any help you can provide.

Davide

Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Re: Record ogg files continuously

Post by Davides » Mon 2016-10-17 11:25

Hi,

I'm using a vs1063a to stream out an Ogg stream via UART to another system (linux based). In this other system the Ogg stream is received correctly, in fact i can save it and listen it without any problems (in addition the command "ogginfo" says that the stream OGG is good and doesn't have any errors or warning). The received stream OGG is composed by 3 initial OGG pages (page 0,1, 2) that contain the Vorbis headers (all with "granule" equal to zero); starting from the page number 3, the OGG pages contain the audio data and in fact the "granule" (audio samples) field is greater than zero.

If I stop the streaming (using the bit SM_CANCEL of the SCI_MODE register) and I split the stream received in different ogg audio files (of about 1 second of duration), the single files are valid (ogginfo and other tools say so), but when i play them (sox, ogg123, ffmpeg) continuously one after the other there is a click noise between one file and another.

I used different ways to split the received ogg stream:

- ffmpeg
- oggcut
- mp3splt
- with an embedded c program that creates ogg files composed by the 3 initial OGG pages (page 0,1, 2) that contain the Vorbis headers (all with "granule" equal to zero) and then one ogg page (from the ones of the original received stream ) that contain audio data (in fact the "granule" field is greater than zero); in this case I had to modified the page sequence number, the granule and the checksum fields in order to make these ogg files valid.

If I open with audacity two consecutive ogg files where there is a click noise i can see that at the end of first there is a different magnitude level compared to the magnitude level of the beginning of the next.

If i merge together the ogg files in one stream using ffmpeg (that doesn't re-encode the files) , this merged ogg audio stream doesn't have any click noises; if instead of using ffmpeg i use sox (sox re-encode) the final merged ogg stream has the click noises.

Hope I explain myself in a clear way.

Could you please help me to remove (if is possible) these click noises in order to obtain a Seamless/Gapless Audio Playback of these single tracks?

Thank you very much.

Davide

User avatar
Henrik
VLSI Staff
Posts: 1144
Joined: Tue 2010-06-22 14:10

Re: Record ogg files continuously

Post by Henrik » Tue 2016-10-18 9:51

Hello Davide!

You have a very interesting, and, frankly, complex question there!

First, you have already noticed that stopping and restarting the VS1063a Ogg Vorbis encoder will not lead in a seamless stream, and that is something that cannot be changed. So, basically, your question comes down to: "How to split an Ogg Vorbis file so that the split portions could be played back-to-back sample-accurately and without glitches". This is quite non-trivial, so let me think about this for a while and get back to you.

One further question: is it all right if you can decode / play back the files seamlessly with e.g. Sox or Ogg123, or do they need to be played back seamlessly with e.g. VS1063a?

Kind regards,
- Henrik
Good signatures never die. They just fade away.

User avatar
Henrik
VLSI Staff
Posts: 1144
Joined: Tue 2010-06-22 14:10

Re: Record ogg files continuously

Post by Henrik » Tue 2016-10-18 10:23

Hello again,

there are several programs that would do this for you. I have just tested "vcut" from the vorbis-tools Linux package, and it will make a perfect split, like this:

Code: Select all

% vcut Robots.ogg R1.ogg R2.ogg +2
Processing: Cutting at 2.000000 seconds
%
R1.ogg and R2.ogg are now perfect splits of the original.

Another option is oggsplt which is part of the mp3splt package:

Code: Select all

% oggsplt Robots.ogg 0.00 0.02 EOF
mp3splt 2.4.2 (13/05/12) - using libmp3splt 0.7.2
        Matteo Trotta <mtrotta AT users.sourceforge.net>
        Alexandru Munteanu <io_fx AT yahoo.fr>
THIS SOFTWARE COMES WITH ABSOLUTELY NO WARRANTY! USE AT YOUR OWN RISK!
 Processing file 'Robots.ogg' ...
 info: file matches the plugin 'ogg vorbis (libvorbis)'
 info: Ogg Vorbis Stream - 44100 - 125 Kb/s - 2 channels - Total time: 0m.10s
 info: starting normal split
   File "Robots_00m_00s__00m_02s.ogg" created                    
   File "Robots_00m_02s__00m_10s_37h.ogg" created                    
 file split (EOF)
%
Are these options satisfactory? At least they are both easily-accessible Linux packages and, according to my tests, they work sample-accurately.

Kind regards,
- Henrik
Good signatures never die. They just fade away.

Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Re: Record ogg files continuously

Post by Davides » Tue 2016-10-18 10:55

Hi,

first of all thank you so much for the answer and for the further help you could provide.

The goal of our project is to use the encoder vs1063a to send an ogg stream to another device (linux based), in this linux-based system we would like to split the received ogg stream, simultaneously with the reception of the same ogg stream (although with a small delay), into multiple ogg audio files, and then send these ogg audio files to a remote server where there will be a web interface with an audio player to listen seamless the tracks (this should work like a sort of "real-time" (with some delays) remote audio stream player).
So we don't need to play the tracks with the vs1063a; in addition we would like (if it will be possible) to not decode and re-encode again the ogg files (obtained from the splitting procedure) for reasons of CPU consumption.

I have already tried lots of tools as I wrote in the previous post (like mp3split or oggcut) and the split is perfect, the problem is that when you try to listen these tracks (for example with "play" or "ogg123") seamless there are click noises between one track and another.

Kind regards
Davide

Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Re: Record ogg files continuously

Post by Davides » Tue 2016-10-18 11:22

Could you please try this and check if there are click noises between one track and another:

oggsplt audioStream.ogg 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 EOF

then try to listen the obtained tracks with:

play audioStream_*

I did the same and there are click noises (in order to hear it you have to be very careful because the noises are very low) between a track and another.

Davide

User avatar
Henrik
VLSI Staff
Posts: 1144
Joined: Tue 2010-06-22 14:10

Re: Record ogg files continuously

Post by Henrik » Tue 2016-10-18 13:16

Hello Davides,

this is quite interesting... With one test file I could get oggsplt to exactly split the file with the parameters you provided, but with another, it created two glitches. So it seems it is not a perfect solution - probably a bug or two lurking there. As I said, seamless cutting is non-trivial. :-)

What I did was I took two short audio samples, cut them into pieces, then put them back together to see if there was any difference. I created the following script called splitjoin.sh for testing:

Code: Select all

#!/bin/sh
rm -f audio_*
oggsplt audio.ogg 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 EOF
oggdec -o audio.wav audio.ogg
for F in audio_*.ogg
do
  oggdec -o `basename $F .ogg`.wav $F
  echo $F
done
sox audio_*.wav audioConcatenated.wav
cmp audio.wav audioConcatenated.wav
Then tested with two audio files as follows. This one succeeds:

Code: Select all

% cp Robots.ogg audio.ogg ; ./splitjoin.sh
[...]
-rw-rw-r-- 1 leopold users 1830244 Oct 18 14:10 audioConcatenated.wav
-rw-rw-r-- 1 leopold users 1830244 Oct 18 14:10 audio.wav
This one doesn't:

Code: Select all

% cp twosins.ogg audio.ogg ; ./splitjoin.sh
[...]
-rw-rw-r-- 1 leopold users 2118536 Oct 18 14:11 audioConcatenated.wav
-rw-rw-r-- 1 leopold users 2116844 Oct 18 14:11 audio.wav
audio.wav audioConcatenated.wav differ: byte 5, line 1
Perhaps contacting the authors of the software with the failing example would persuade them to try to fix it?


On another note: how accurate are you splitting requirements? I am asking this, because if it is ok that splitting is only done where it is "easy", in other words, if it doesn't matter if the cut point varies randomly around the point you asked for, or that the software chooses the first "easy" cut point after the point you have requested, then in that case cutting an Ogg Vorbis file losslessly becomes much easier.

E.g. is it OK if you specify a cut point of one file each minute, and what you get is what you get: usually the actual cut point will be between 60-60.5 seconds, but it may sometimes be, semi-randomly, longer?

Kind regards,
- Henrik
Attachments
Robots.ogg
Works with this file
(166.99 KiB) Downloaded 64 times
twosins.ogg
Fails with this file
(90.36 KiB) Downloaded 62 times
Good signatures never die. They just fade away.

Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Re: Record ogg files continuously

Post by Davides » Tue 2016-10-18 15:36

Hi Henrik,

thank you very much for your answer, this discussion is getting more and more interesting.

As regards your question, the answer is no, we don't need to be too strict about the cut points, but we need to have ogg files as small as possible in order to make faster the transfer to the server; so for example we could have the first cut point at 1 second, the next one at 2.5 seconds, then after 4.5 second, then after 7 seconds.

Could you please tell me about a tool or a method to cut an Ogg Vorbis file losslessly if the cut point varies randomly around the point we asked for?

I have another interesting thing i want to tell you:

I developed a C program that take in input the ogg stream from the encoder vs1063 and splits this stream in multiple audio ogg files; all the ogg files created have the same first 3 OGG pages (i stored it in a local directory), because these 3 pages contain the vorbis headers; after the first 3 ogg pages, i used every successive ogg page that the c program receives from the encoder (delimited from the "OggS" capture pattern field) to create an ogg audio file.
I developed this method after I studied how a valid ogg vorbis file should be.

C-Pseudocode:

while(1)
{
if((received == "OggS")&&(PageSequenceNumber >= 3))
{
createNewOggfile;
insertLocalVorbisHeaders; // first 3 ogg pages stored locally
receiveOggPage; // completes the Ogg page reception
modifyHeaderType; // write EOS in the 4th ogg page of the new file
modifyGranule; // in order to have time/sample information valid for each file
modifyPageSequenceNumber; // in order to have page number starting from 0
modifyChecksum; // in order to have valid ogg files after the modifications applied
insertReceivedSegments; // fill the ogg file with the vorbis audio payload received from the encoder
saveOggFile;
}
}

I think that from a theorical point of view, the program is correct, in fact i obtained valid ogg audio files, and all the tools i used to validate them say that the ogg files are fine.
Unfortunatelly, these files have the click noise problem, and when i listen them with an audio player like "ogg123" or "play" there are click noises between one and another.

What do you think about this?

Kind regards
Davide

User avatar
Henrik
VLSI Staff
Posts: 1144
Joined: Tue 2010-06-22 14:10

Re: Record ogg files continuously

Post by Henrik » Tue 2016-10-18 16:09

Hello!

Your method is correct in that way that it creates valid Ogg files - at least most of the time part. I'm not quite sure if it is valid to have a first audio frame with the "continued packet flag" (bit 0 of byte 5 of each Ogg container frame), which would sometimes happen with this approeach.

Nevertheless, the issue is that the Ogg Vorbis decoder needs to be primed for correct decode to happen, in other words some Vorbis data needs to be repeated for the decoder to be able to fully decode the first frame. Add to this some alignment issues between the Ogg container and Vorbis audio format, and you will lose from at least one, probably two, or possibly even three Vorbis audio frames each time you make a cut in the way you've done.

Let me think about this overnight; if the less accurate cut point is ok, I think I could write a shortish C program that could do it properly. No promises, though!

Kind regards,
- Henrik
Good signatures never die. They just fade away.

Davides
Senior User
Posts: 34
Joined: Thu 2016-05-26 17:37

Re: Record ogg files continuously

Post by Davides » Tue 2016-10-18 17:04

Ok, great! Thank you so much.

You pointed out very interesting points, but as regards the "continued packet flag", in all the tests i did i never found it enabled; so i think that is more likely that the issue is because of the second point you indicated.

I'm going to look for what's the best way to choose the cut points, hope to find some documentation online.

Kind regards

Davide

Post Reply