FAT SD-Card is nice to use, but (very) slow
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
FAT SD-Card is nice to use, but (very) slow
Hi Ben
At first, the Fat from FC is realy nice to use and is working fine. It's create file by file on a card.
I have PIC 18LF6722. It's working on 32Mhz. There is a SD Card on the board. The card is working with clock/4
I used the Blockwrite (write file sector). The goal is to sample 2Channel 16Bit with 2kHz. For this I used two buffers.
If I sample data in buffer_A and I write buffer_B to the disk. If the buffer is full I change the buffers.
That's working fine. To write 512Byte (one Block) to of the SD-card, needs 25mSek. It's ok....For the "move to the next sector" needs 5mSek. It's ok. But every 64times this command needs about 58mSek. this is to long....
We are talking about the speed I need.....
2Khz x 4Byte =8kByte per second or
2Khz x 4Byte *60= 0.5Megabyte per minute
I used a Kinston class 10 Card with 45MB/s
I need I good idea to speed up the writing to disk.
At first, the Fat from FC is realy nice to use and is working fine. It's create file by file on a card.
I have PIC 18LF6722. It's working on 32Mhz. There is a SD Card on the board. The card is working with clock/4
I used the Blockwrite (write file sector). The goal is to sample 2Channel 16Bit with 2kHz. For this I used two buffers.
If I sample data in buffer_A and I write buffer_B to the disk. If the buffer is full I change the buffers.
That's working fine. To write 512Byte (one Block) to of the SD-card, needs 25mSek. It's ok....For the "move to the next sector" needs 5mSek. It's ok. But every 64times this command needs about 58mSek. this is to long....
We are talking about the speed I need.....
2Khz x 4Byte =8kByte per second or
2Khz x 4Byte *60= 0.5Megabyte per minute
I used a Kinston class 10 Card with 45MB/s
I need I good idea to speed up the writing to disk.
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hello,
It sounds like the 64 sector is the cluster boundary and at which point the FAT routine needs to jump to the File Allocation tables and hunt for and assign a new unused sector address.
If you didn't need the file system FAT support and just wanted to use the card as a data storage then this problem would go away.
I'll have another look at the code to see if I can make any further improvement on the FAT housekeeping but I can't promise anything as I wrote the code originally to be as efficient as possible.
It sounds like the 64 sector is the cluster boundary and at which point the FAT routine needs to jump to the File Allocation tables and hunt for and assign a new unused sector address.
If you didn't need the file system FAT support and just wanted to use the card as a data storage then this problem would go away.
I'll have another look at the code to see if I can make any further improvement on the FAT housekeeping but I can't promise anything as I wrote the code originally to be as efficient as possible.
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Maybe also you could use a circular buffer component to store some of the data allowing you to maintain and catch up from the new cluster delay.
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Yes it's every 64times that is going long.
Unfortunately I need the FAT. A ringbuffer is not so practical, the 18LF6722 has "only" 3936 Bytes RAM.
I hope it's possible to do some improvement of the speed. I think we are close that to the 2Khz goes ....
Anyway the FAT System from you is very comfortable to use and easy to understand.
Unfortunately I need the FAT. A ringbuffer is not so practical, the 18LF6722 has "only" 3936 Bytes RAM.
I hope it's possible to do some improvement of the speed. I think we are close that to the 2Khz goes ....
Anyway the FAT System from you is very comfortable to use and easy to understand.
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
I've taken out a double check routine from the SendBuffer function.
I've also got rid of a buffer read when calling the WriteFileSector macro so the buffer will no longer already contain the sector data after a write.
Both of these should give you a little speed boost for your application but otherwise are not recommended for general use.
Let me know how you get on.
Regarding the circular buffer, I thought maybe you could just use this as a stop gap to overcome the additional 64th sector delay, after the delay you stream your data to your non writing buffer, when the buffer is full you store to the circular buffer until the write is complete, then stream to the card from the buffer and from the circular buffer before resume your alternate buffer functionality. Just a thought, maybe complicated to implement.
I've also got rid of a buffer read when calling the WriteFileSector macro so the buffer will no longer already contain the sector data after a write.
Both of these should give you a little speed boost for your application but otherwise are not recommended for general use.
Let me know how you get on.
Regarding the circular buffer, I thought maybe you could just use this as a stop gap to overcome the additional 64th sector delay, after the delay you stream your data to your non writing buffer, when the buffer is full you store to the circular buffer until the write is complete, then stream to the card from the buffer and from the circular buffer before resume your alternate buffer functionality. Just a thought, maybe complicated to implement.
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
Thank for the FAT. I'll test that. And I hope for the 2channel 2kHz recorder with this older PIC will work.
Soon I start to go on with a 10channel 2Khz recorder. I saw microchip has now ARM® Cortex® from Atmel
As if there are faster (300Mega) and need less power... They have a HSMCI port (High Speed MultiMedia Card Interface).
What do you think I should change?
http://www.microchip.com/design-centers ... am-4e-mcus
Thank for the FAT. I'll test that. And I hope for the 2channel 2kHz recorder with this older PIC will work.
Soon I start to go on with a 10channel 2Khz recorder. I saw microchip has now ARM® Cortex® from Atmel
As if there are faster (300Mega) and need less power... They have a HSMCI port (High Speed MultiMedia Card Interface).
What do you think I should change?
http://www.microchip.com/design-centers ... am-4e-mcus
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hello,
I would suggest a ST ARM over an Atmel Arm, We have a new version of Flowcode which will be available shortly which will add these devices.
The 16-bit PICs are also very good especially the EP range of devices (70 x 16-bit MIPs vs your current device at 8 x 8-bit MIPs), the PIC32 I am yet to play more with but again these are pretty fast devices.
A standard 8-bit AVR clocked at 16MHz will run at pretty much double the speed of your current 18F PIC device at 32MHz.
I would suggest a ST ARM over an Atmel Arm, We have a new version of Flowcode which will be available shortly which will add these devices.
The 16-bit PICs are also very good especially the EP range of devices (70 x 16-bit MIPs vs your current device at 8 x 8-bit MIPs), the PIC32 I am yet to play more with but again these are pretty fast devices.
A standard 8-bit AVR clocked at 16MHz will run at pretty much double the speed of your current 18F PIC device at 32MHz.
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
"We have a new version of Flowcode"
That nice to hear. Is this FC8 or FC7.2 ? And when can I have it?
Is there a newsletter about FC or general?
"We have a new version of Flowcode"
That nice to hear. Is this FC8 or FC7.2 ? And when can I have it?
Is there a newsletter about FC or general?
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hello,
http://www.matrixtsl.com/mailing_list.php
It's 7.2 and it's being beta tested at the moment. We have uncovered some problems which we are currently working on. So coming soon, hopefully the next couple of weeks.That nice to hear. Is this FC8 or FC7.2 ? And when can I have it?
There certainly is you can sign up for it here.Is there a newsletter about FC or general?
http://www.matrixtsl.com/mailing_list.php
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
Thank You for the special FAT.It's working faster and I can use my 2channel recorder.
I have now a another small problem to write in a corect way the data to the wave file.
before with CCS I dit it like this to convert the 10Bit uint from the AD-converter to little Endian signed16Bit. But how can i do it in FC7?
ADCIn2=(ADCIn2<<3);
ADCIn2=(ADCIn2-0x1000);
highByte1 = make8(ADCIn2,1);
lowByte1 = make8(ADCIn2,0);
lowByte1 |=0x07;
Thank You for the special FAT.It's working faster and I can use my 2channel recorder.
I have now a another small problem to write in a corect way the data to the wave file.
before with CCS I dit it like this to convert the 10Bit uint from the AD-converter to little Endian signed16Bit. But how can i do it in FC7?
ADCIn2=(ADCIn2<<3);
ADCIn2=(ADCIn2-0x1000);
highByte1 = make8(ADCIn2,1);
lowByte1 = make8(ADCIn2,0);
lowByte1 |=0x07;
- Benj
- Matrix Staff
- Posts: 15312
- Joined: Mon Oct 16, 2006 10:48 am
- Location: Matrix TS Ltd
- Has thanked: 4803 times
- Been thanked: 4314 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hello,
You can do it like this.
ADCIn2 = ADC::ReadAsInt()
ADCIn2=(ADCIn2 << 3)
ADCIn2=(ADCIn2 - 0x1000)
highByte1 = ADCIn2 >> 8
lowByte1 = ADCIn2 | 0x07
You can do it like this.
ADCIn2 = ADC::ReadAsInt()
ADCIn2=(ADCIn2 << 3)
ADCIn2=(ADCIn2 - 0x1000)
highByte1 = ADCIn2 >> 8
lowByte1 = ADCIn2 | 0x07
Regards Ben Rowland - MatrixTSL
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
Flowcode Product Page - Flowcode Help Wiki - Flowcode Examples - Flowcode Blog - Flowcode Course - My YouTube Channel
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
perfect! Thank You. And here a sinus 200Hz Signal sampled with 2kHz and saved as data like a wavefile on a micro-sdcard and open with Audacity (import raw).
perfect! Thank You. And here a sinus 200Hz Signal sampled with 2kHz and saved as data like a wavefile on a micro-sdcard and open with Audacity (import raw).
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
Now I have to begin with the 10channel recorder. For this project I try to use the chipkit wi-fire with the PIC32MZ2048.
The board include a micro-SDcard connector. I tested the card but it's not fast enough. Is there a possibility to to increase the speed.
This Pic has much more ram. Maybe we can write a cluster 64x512Byte=32768 in one go?
Now I have to begin with the 10channel recorder. For this project I try to use the chipkit wi-fire with the PIC32MZ2048.
The board include a micro-SDcard connector. I tested the card but it's not fast enough. Is there a possibility to to increase the speed.
This Pic has much more ram. Maybe we can write a cluster 64x512Byte=32768 in one go?
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
A long time ago you changed the fat for me.
Can you make the "normal" FAT faster for all users?
or another idea would be to have 2 buffers for the FAT.
So I could fill buffer A while buffer B is written to disk and then fill buffer B while buffer A is written to the disk. At the moment I do it like this but I have alway to copy to the writebuffer from the FAT.
Or maybe you some bether ideas.....
A long time ago you changed the fat for me.
But I do not know if I have a new FAT again and again with the FC8 update... I always want the latest update. That would be goodI've taken out a double check routine from the SendBuffer function.
I've also got rid of a buffer read when calling the WriteFileSector macro so the buffer will no longer already contain the sector data after a write.
Both of these should give you a little speed boost for your application but otherwise are not recommended for general use.
Can you make the "normal" FAT faster for all users?
or another idea would be to have 2 buffers for the FAT.
So I could fill buffer A while buffer B is written to disk and then fill buffer B while buffer A is written to the disk. At the moment I do it like this but I have alway to copy to the writebuffer from the FAT.
Or maybe you some bether ideas.....
-
- Valued Contributor
- Posts: 654
- Joined: Fri Aug 19, 2016 2:09 pm
- Location: switzerland
- Has thanked: 182 times
- Been thanked: 179 times
- Contact:
Re: FAT SD-Card is nice to use, but (very) slow
Hi Ben
There is a small problem after 64 sector. The card is like making a break for the cluster.
I have no more errors in the file and I can write in 1 minute a 3MB file (10hours for 1.8GB) My next step is test this 10hours.....
There is a small problem after 64 sector. The card is like making a break for the cluster.
So I made 2 big buffer (64512 Byte), while the card make his break I store the date and write it after the break.....It sounds like the 64 sector is the cluster boundary and at which point the FAT routine needs to jump to the File Allocation tables and hunt for and assign a new unused sector address
I have no more errors in the file and I can write in 1 minute a 3MB file (10hours for 1.8GB) My next step is test this 10hours.....