FAT: faster way to write data

For Flowcode users to discuss projects, flowcharts, and any other issues related to Flowcode 6.

Moderator: Benj

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

FAT: faster way to write data

Post by mikn »

which way is faster: use WriteFileSector or AppendStringToFile?

I have pic24 operating at 8mhz
ADC is reading analogue data at 8khz frequency (using TMR0).
FAT prescaler is Fosc/4.
Every 512bytes I call writefilesector to write data to sdcard. While chip operates with sdcard I miss new data from analogue port. Write operation takes around 100ms.

Are there any way to avoid the gap? Sdcard is set to maximum speed now, are there any tricks to do fast data writting?
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

The sector method is likely to be slightly faster though I did work in some new optimisations for the AppendString function so it is now not as slow as it was in Flowcode v5.

To maintain samples while your talking to the card can you use a timer interrupt and a circular buffer component to collect the ADC readings and store them away. You can then pull out the readings as you need them.

There is a basic example on circular buffering techniques here.

http://www.matrixmultimedia.com/mmforum ... fer#p33484

And additional help on the circular buffer component available from here if you don't want to manage the circular buffer yourself.

http://www.matrixmultimedia.com/wiki/in ... f2ad670a8a

Let us know how your getting on.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

Benj wrote: And additional help on the circular buffer component available from here if you don't want to manage the circular buffer yourself.
http://www.matrixmultimedia.com/wiki/in ... f2ad670a8a
I don't have it in my Advanced panel. How to install it?

UPD: Sorry, found it in /components folder and just copy-pasted fcpx file into the project.
Where are they in visual component list? I see tons of components in that folder but only few in FC upper panel.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

Some of the lesser user or advanced components have been hidden away into the search component toolbar in 6.0.4.

If you do a blank search then you should get a list of all installed components.
Search.jpg
(86.28 KiB) Downloaded 2666 times

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

Finally , I've made my own cycled array of 1024 bytes. While it records first half (512bytes) I transfer second half to sd card.
I put copy of array to fat byffer inside the interrupt because it was slow to do copy in the main loop (while it was copying 512bytes and writting it to sdcard, Interrupt was filling the second part and began overwritting first part).

So there's gap in a raw analogue data while it is copying the array to FAT buffer.
Any ideas?
pic24 is running 8mhz. tmr1 is 8khz.
Original firmware for this devcie can record data upto 44khz. How can they do it?

My main flowchart and interrupt flowchart are included below. Any advices?

PS: when I use GetAverageByte(2,1000) instead of GetByte - chip totally freezes. Same for RawAverage.. What am I doing wrong?
Attachments
int
int
adcread.GIF (8.72 KiB) Viewed 15071 times
main
main
adcmain.GIF (5.6 KiB) Viewed 15071 times
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

What PIC24 device are you using? My guess is they use the on-board PLL to multiply the 8MHz input frequency from the crystal up to a much higher frequency.

The Raw Analogue CAL is known to have lockup's on some devices and is already on the list as something which needs looking into.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

Benj wrote:What PIC24 device are you using?
i'm using fj64ga. maybe you can advice me some alternative settings?
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

This page has some good info on using PLLs.

http://www.matrixmultimedia.com/wiki/in ... _%28PLL%29

If your struggling to work out what the C code needs to be to setup the PLL for your hardware then I will see if I can dig out any reference code.

Let me know how your getting on

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

I'm not very good with PLLs and oscillator settings but will try to do some experiments. Any advices will be appreciated. Thanks.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

Right you have a input oscillator source of 8MHz and want to convert this to max speed of 32MHz.

Looking at the PIC24FJ64GA104 family the PLL is very simple and is a fixed 4x multiplier. Should just be a case of flicking the "initial oscillator select" config setting to "Primary Osc with PLL".

Change the clock speed to 32000000 in the project options and then to test out the PLL use a 1 second flasher approach.

http://www.matrixmultimedia.com/wiki/in ... ED_flasher

Let me know how you get on.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

I'm using internal oscillator. So I put FastRC with PLL.
Made test with 1 second LED and it blinks with 2 seconds pause.

_config1 = 0x3f7f
_config2 = 0x391f
FC 6.1.3.2 (18.02.2016)

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

I did other tests:
with FRCDIV led blinks 8 times slower
with FRC it blinks 4 times slower
with FRCPLL it blinks 2 times slower than it must be.
it's all with 32MHz clock speed.
If i make clock speed to 4MHz it become blinking faster. The lower is clock speed the faster it blinks (delay time decreases).

Didn't find any other settings for oscillator and I am confused what it may be. Isn't it a bug? I mean that if it was my project then it could have some code error, but it's just LED blinking, simple as 1-2-3. I am disappointed. Any advices?
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

Looks like you need to start up with the FRC then flick this to 8MHz instead of 4MHz and then switch on the PLL to obtain the 32MHz operation.

I will try and dig out some C code tomorrow that will get you up and running.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

I am not familiar with switching clock speed and PLL on the fly. If you can provide me with some example code to include into flowchart, that will be very appreciated.
Will also wait for news from you about FC PLL settings, because I am at dead end..

ps: please review that problem with saving _config parameters. Every time I start the FC and load my project it loads default config (WDT enabled, JTAG enabled, etc...), however I changed them and saved the project including new config.
FC 6.1.3.2 (18.02.2016)

User avatar
Benj
Matrix Staff
Posts: 15312
Joined: Mon Oct 16, 2006 10:48 am
Location: Matrix TS Ltd
Has thanked: 4803 times
Been thanked: 4314 times
Contact:

Re: FAT: faster way to write data

Post by Benj »

Hello,

Try adding this code in a C icon at the top of main and set the config setting to the FRC option without PLL.

Code: Select all

    unsigned int pllCounter;
    OSCCONBITS OSCCONbitsCopy;
 
    // Copy the current Clock Setup
    OSCCONbitsCopy = OSCCONbits;
    // Slow output clock down to 4Mhz
    CLKDIVbits.CPDIV = 3;
    // Enable the PLL - Note: Fuse bits don't do this
    CLKDIVbits.PLLEN = 1;
    // Wait for the PLL to stabalise
    for (pllCounter = 0; pllCounter < 600; pllCounter++);
 
        // Setup the uC to use the internal FRCPLL mode
        OSCCONbitsCopy.NOSC = 1;
        OSCCONbitsCopy.OSWEN = 1;
 
    // Switch over to the new clock setup
    __builtin_write_OSCCONH( BITS2BYTEH( OSCCONbitsCopy ) );
    __builtin_write_OSCCONL( BITS2BYTEL( OSCCONbitsCopy ) );
    // Wait for this transfer to take place
    while (OSCCONbits.COSC != OSCCONbits.NOSC);
    // Setup the DIV bits for the FRC, this values means the config word needs to be: PLLDIV_DIV2
    CLKDIVbits.RCDIV0 = 0;
 
    // Setup the PLL divider for the correct clock frequency
        CLKDIVbits.CPDIV = 0;
 
    // Check that the PLL is enabled again and locked properly to the new setup
    CLKDIVbits.PLLEN = 1;
    // Note - don't want to do this check if we are running in the MPLAB X simulator as it won't work
        while(_LOCK != 1);
Let me know how you get on.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

could not compile it. below is the log:
D:\flowcode\PIC24T~1>pic30-gcc -c -mcpu="24FJ64GA002" -funsigned-char -fno-short-double -Os -I"C:\PROGRA~1\FLOWCO~2\COMPIL~1\pic16\BATCHF~1\..\support\h" -I"C:\PROGRA~1\FLOWCO~2\COMPIL~1\pic16\BATCHF~1\..\MX_support" -Wall -std=gnu99 "Flowcode1sma".c -o "Flowcode1sma".o
Flowcode1sma.c: In function 'FCD_08f41_adc_base1__GetString':
Flowcode1sma.c:392: warning: pointer targets in passing argument 3 of 'FCI_FLOAT_TO_STRING' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__ReadStringFromFile':
Flowcode1sma.c:651: warning: pointer targets in passing argument 1 of 'FCI_SCOPY' differ in signedness
Flowcode1sma.c:651: warning: pointer targets in passing argument 3 of 'FCI_SCOPY' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__ReadByteFromFile':
Flowcode1sma.c:832: warning: unused variable 'FCL_IDX'
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__DeleteFile':
Flowcode1sma.c:952: warning: passing argument 1 of 'FCI_COMPARE' discards qualifiers from pointer target type
Flowcode1sma.c:952: warning: pointer targets in passing argument 3 of 'FCI_COMPARE' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__CreateFile':
Flowcode1sma.c:1306: warning: large integer implicitly truncated to unsigned type
Flowcode1sma.c:1314: warning: large integer implicitly truncated to unsigned type
Flowcode1sma.c:1322: warning: large integer implicitly truncated to unsigned type
Flowcode1sma.c:1338: warning: large integer implicitly truncated to unsigned type
Flowcode1sma.c:1346: warning: large integer implicitly truncated to unsigned type
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__OpenFolder':
Flowcode1sma.c:2476: warning: passing argument 1 of 'FCI_COMPARE' discards qualifiers from pointer target type
Flowcode1sma.c:2476: warning: pointer targets in passing argument 3 of 'FCI_COMPARE' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__OpenFile':
Flowcode1sma.c:2690: warning: passing argument 1 of 'FCI_COMPARE' discards qualifiers from pointer target type
Flowcode1sma.c:2690: warning: pointer targets in passing argument 3 of 'FCI_COMPARE' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__AppendStringToFile':
Flowcode1sma.c:3003: warning: pointer targets in passing argument 1 of 'FCI_GETLENGTH' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__Format_File_String':
Flowcode1sma.c:3169: warning: pointer targets in passing argument 1 of 'FCI_TOUPPER' differ in signedness
Flowcode1sma.c:3169: warning: pointer targets in passing argument 3 of 'FCI_TOUPPER' differ in signedness
Flowcode1sma.c: In function 'FCD_07fa1_FAT1__Initialise':
Flowcode1sma.c:3252: warning: unused variable 'FCL_RETVAL'
Flowcode1sma.c: In function 'main':
Flowcode1sma.c:3693: error: 'CLKDIVBITS' has no member named 'CPDIV'
Flowcode1sma.c:3695: error: 'CLKDIVBITS' has no member named 'PLLEN'
Flowcode1sma.c:3704: warning: implicit declaration of function 'BITS2BYTEH'
Flowcode1sma.c:3705: warning: implicit declaration of function 'BITS2BYTEL'
Flowcode1sma.c:3712: error: 'CLKDIVBITS' has no member named 'CPDIV'
Flowcode1sma.c:3715: error: 'CLKDIVBITS' has no member named 'PLLEN'
Flowcode1sma.c:3777: warning: passing argument 1 of 'FCD_07fa1_FAT1__DeleteFile' discards qualifiers from pointer target type
Flowcode1sma.c:3785: warning: passing argument 1 of 'FCD_07fa1_FAT1__CreateFile' discards qualifiers from pointer target type
Flowcode1sma.c:3789: warning: passing argument 1 of 'FCD_07fa1_FAT1__OpenFile' discards qualifiers from pointer target type
Flowcode1sma.c:3897: warning: label 'FCC_Main_A' defined but not used

Error returned from [pic30-gcc.exe]
Completed compilation, return = 1

C:\Program Files\Flowcode 6\compilers\pic16\batchfiles\pic16_C30_comp.bat reported error code 1


FINISHED
FC 6.1.3.2 (18.02.2016)

Mathy
Posts: 333
Joined: Mon Oct 05, 2009 2:39 pm
Has thanked: 30 times
Been thanked: 33 times
Contact:

Re: FAT: faster way to write data

Post by Mathy »

Hello,

What do you want ?
running at 32 Mhz without crystal ?

Try this :

supplementary code :

_CONFIG1(0x3f7f) //LASTWORD
_CONFIG2(0xf19f) //LASTWORDONE
_CONFIG3(0xFFFF) //LASTWORDTWO

c code :

OSCCON = 0x11C0;
CLKDIV = 0x0000;

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

Mathy wrote:What do you want ?
I want to capture analog signal 10.000 times in a second and write the data into SD card.
Now the problem is that card writes it much slower than signal capture and I have losses in signal raw.
FC 6.1.3.2 (18.02.2016)

Mathy
Posts: 333
Joined: Mon Oct 05, 2009 2:39 pm
Has thanked: 30 times
Been thanked: 33 times
Contact:

Re: FAT: faster way to write data

Post by Mathy »

Have you tried my piece of code and tried a blink test to see if you are running at 32 Mhz ?

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

Yes, just tried. It blinks normal with 1sec delay. Tried to run my code with same settings, but have losses.
Below is my project.
Attachments
Flowcode1sma.rar
(3.03 KiB) Downloaded 184 times
FC 6.1.3.2 (18.02.2016)

Mathy
Posts: 333
Joined: Mon Oct 05, 2009 2:39 pm
Has thanked: 30 times
Been thanked: 33 times
Contact:

Re: FAT: faster way to write data

Post by Mathy »

You will have best result using hardware peripheral instead of software.

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

It could be best decision, but...
it's a finished device, i have original firmware which works perfect and captures analogue input with frequency upto 30khz. It works with internal clock. I dont have sources and wonder how they do it?
FC 6.1.3.2 (18.02.2016)

Mathy
Posts: 333
Joined: Mon Oct 05, 2009 2:39 pm
Has thanked: 30 times
Been thanked: 33 times
Contact:

Re: FAT: faster way to write data

Post by Mathy »

I don't understand your post.
Maybe the original product code source use hardware SPI instead of software as I said in my previous post.


Sending one char in software mode with 18F67J60 MCU running at 25 Mhz takes around 38.40us with FOSC/S
Sending one char in hardware mode with the same parameters takes around ....................................1us :!:

User avatar
mikn
Posts: 209
Joined: Mon Mar 03, 2014 10:11 pm
Has thanked: 54 times
Been thanked: 41 times
Contact:

Re: FAT: faster way to write data

Post by mikn »

hardware SPI instead of software as I said in my previous post.
Sorry, I thought you are talking about external clock.
Should I use SPI instead of SDcard module in FC? Is there big difference?
FC 6.1.3.2 (18.02.2016)

User avatar
JonnyW
Posts: 1230
Joined: Fri Oct 29, 2010 9:13 am
Location: Matrix Multimedia Ltd
Has thanked: 63 times
Been thanked: 290 times
Contact:

Re: FAT: faster way to write data

Post by JonnyW »

Hi.

I don't know how your programs work but there may also be blocking calls which take too much time.

As you are using a circular buffer are you sure this is correctly implemented and the interrupt is firing constantly. I would suggest to ensure your SW is working 100% in hardware as expected, get hold of an LED array and have:

- 2 LEDs that flash every time you read data to each half of your 512-byte buffer (at 8kHz)
- 2 LEDs that turns on at the start of FAT write for each half of your 512-byte buffer

This helps make sure the data is constantly being read reliably and that the FAT is never reading the same block of memory that you are buffering to.

In order to do this add some code like:

Code: Select all

 ---------- Interrupt ----------
StatusBits = StatusBits & ~((BufferPosition >> 9) << 2)

---------- Read ADC to a buffer ---------
BufferPosition = (BufferPosition + 1) & 0x3FF // Increment scrolling buffer
if (BufferPosition & 0x1FF) == 0 then SDWriteSection = (BufferPosition ^ 0x200)

StatusBits = StatusBits ^ ((BufferPosition >> 9) << 2)
$PORTA = StatusBits // Update
And in your main polling routine:

Code: Select all

if SDWriteSection  != -1 then
  StatusBits = (StatusBits & ~0x3) | (SDWriteSection >> 9)
  $PORTA = StatusBits // Update
  ---------- Write to FAT ---------
  StatusBits = (StatusBits & ~0x3)
  $PORTA = StatusBits // Update
  SDWriteSection = -1
endif
(I hope this pseudo-code makes sense!)

The other thing you can try if the write is just too long is run-length encoding your data or doing differential compression (which may work better), so you have less overhead unless there are lots of large fluctuations on the ADC.

I hope any of this helps / makes a jot of sense!

EDIT: Another thing to note (don't know if this has been covered) is that 8Kz reads into 512 bytes takes 64ms. If the FAT write is 100ms you will always get a loss unless you increase your buffer size to 3 sections, so 0x600 bytes in your buffer.

Jonny

Post Reply