Embeded Database

Go To Last Post
22 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I know that we can use SQL lite as an embedded database, But I want to know if you happen to know something that needs even lesser memory!

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Exactly what kind of operations do you need to perform and how much data? Clearly a linked list (maybe doubly linked for fast deletion?) of structs might work.

I almost ended up writing one in this previous thread:

https://www.avrfreaks.net/index.p...

(esp. see my second last post in that thread).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I need to be able to save 50K records, need to search them and retrieve data, add ,delete, sort etc... I think structs and linked lists won’t help me here.
In my previous project I have used structs, but now I think I should go with a database!

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I assume you use a (FatFs) file system ? Than its probably best to work with several files, tables with fixed length.

Patrick

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I think structs and linked lists won’t help me here.

What makes you think that? What do you think is at the heart of a database if not list of records? The code I presented does everything on your wishlist but sort. That can actually be done quite easily with a linked list as when "shuffling" records the data does not move - just the links between items.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

http://www.contiki-os.org/ - look for "Antelope database".
Found via Google Contiki database.

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Cliff, please don't shatter the illusion that magic happens in databases!

Ali, you might want to do some Googling for terms like 'b-tree, binary tree, red-black trees, hashing' etc as well as 'bubble sort, shuffle sort, sorting' etc. Donald Knuth wrote some classic references for all the classic algorithms.
http://www-cs-faculty.stanford.e...

Many embedded guys come from an engineering background and miss computer algorithms. It helps to have a comp sci background as well.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

And database design is a whole field by itself; stuff like normalization, referential keys, indexing etc.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
What do you think is at the heart of a database [...]?

A database management system caters at least this, IMO:

A "logical storage model" (my made-up term, my memory is failing intermittently). The dominating one for the last twenty or so years is the relational model, where data is stored in tables (or "relations").

A "physical storage model" offering one or several data structures with/in which you can implement the logical model. This includes blocked and clustered storage of "records" ("rows in tables", "tuples") as well as structures for ease of access and upholding constraints (such as index structures, the B-trees mentioned in a post above being one dominant such.

Mechanisms for describing the structure of an instance of a database, often referred to as "the schema" (including descriptions of constraints and other demands on data, e.g. uniqueness), and mechanisms to assure that data actually adheres to the schema.

A central concept for DBMSs (IMO) is the transaction concept. Think of it as "atomic blocks for sequences of DB operations" on steroids. Not only do they help solve the problem of concurrent access (although in a different way than our ATOMIC_BLOCKs) - they also give you an "all-or-nothing" guarantee. This means that if the sysmen breaks down, or you decide to back out, after 99% of the operations in a transaction is executed then nothing will actually be stored.

While it is not magic, but just algorithms and data structures at work, it still is a both advanced and useful thing. If any of the above was trying to imply that "databases are just about sorting and searching" then I object strongly. Also, if a data storing system does not support what I described above (and possibly a few things that I left out) then it should not be allowed to be titulated "a database system". This term, and the acronym "DBMS" has been raped heavily for the last decade or so. The sad thing is that theres a lot of "nothing special" going about. Us old database people go just as mad over this as embedded persons twist over "can I control my nuclear reactor with the Arduino?".

If it's just about storing, sorting and searching then its a "data structure". Not bad, but not a DBMS. (No, it is not enough that you manipulate the data structure with the SQL language for it to be called "a database (system)".

-----

Ali!

Why cant you sort, search etc a linked list of 50K records? What are the timing constraints.

Remember: Anything mechanism that is introduced to fasten up some operation slows other operations down, or costs more memory, or makes the software harder to maintain, or possibly all of those (a.k.a. "there is no such thing as a free lunch"). Tell us something about the size of each record, how many fields there are in a record etc.

A binary search through 50 K records requires 2log(50K) lookups in the worst case, i.e. looking at 16 entries. Assuming looking at one entry takes one ms (clearly on the safe side, right?) will still locate a unique record in less time than the blink of an eye.

Really bad sorting algorithms like "bubble sort" can still sort 50K records. How fast? Well .. here is my guesstimate: loop over all 50 K records, for each such loop we do an inner loop over the records from the one we're at in the outer loop until the end of records (i.e. on average 25K records). So we will ultimately do 1,250,000 compares. Assume each test takes 1 ms. Assume half of the tests will involve a swap and that a swap takes 5 ms. Total time to sort (guessing rather wildly) would be 1250000*1 + 1250000/2*5 = 4,375,000 us or about 4.5 seconds. Acceptable?

Do some real tests on an AVR. Then do the same, but using e.g. Antelope. It just might be that for a single "flat file" an array ot a linked list might be the winner.

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for your help, But I think by using a database system I will not lose that much, actually I will gain more!
Here you can see a previsions project from me,
https://www.avrfreaks.net/index.p...

Now I’m just using segger GUI library, And it's not that slow! And my program is easier to maintain.
I think in my next project I will use LPC43XX and they have enough power to be able to use a more sophisticated database system than using linked list and structs....and actually 4.5 second is a huge time for my system, because my system needs frequent updates.in my previous design each record used a 128 bye storage.

Attachment(s): 

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I think I should either buy a camera or throw my HTC cell phone to trash, Because its pictures sucks! I will assure you that the graphics looks really great!

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

A Database in Every Sensor by Nicolas Tsiftes and Adam Dunkels (Swedish Institute of Computer Science, ACM SenSys*11, 2011-Nov).
Antelope DBMS
Timing - Figure 12 (page 11) (MSP430 3.9MHz).
Sizing - Table 3 and Figure 7 (page 9) (MSPGCC 3.2.3).
Power consumption, wireless, no database versus database - Figure 13 (page 12).
Wear-leveling filesystems - ELF, Coffee (in Contiki).
Antelope DBMS is dependent on a filesystem.

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Is an embedded DBMS necessary?
Alternatives:
1. Web servers (serve data on MCU, serve information from a DBMS on a PC) (ref. 1).
2. DDS (Data Distribution Service). "Like a database, DDS naturally decouples the data from the application logic (hence "data-centric")." (ref. 2)
Systems design (like where to place the database) will aid implementation (ref. 3).
References:
Embedded Systems Design, 2011-Sep.
1. Web services puts M2M in The Cloud by Joel K. Young.
2. The embedded cloud: IT at the edge by Rajive Joshi.
3. Using Monte Carlo methods to design domain-driven device clouds by Mark Benson.

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If you're so hung up on 'embedded databases' why not just use Linux? Use a LPC31xx device although price wise (according to Digikey) the Freescale iMX23 is $8 vs around $11 for the LPC3154. Just add a DDR ram chip and serial flash and that's your embedded linux system. No problems with graphics libraries, touch screens, etc. All been done.

According to your specs, you have 50,000 records of 128 bytes, theat equals 6.4MB of data - you'll need external storage for that be it serial flash or sdcard. If you add ram to cope with your graphics, then you've got the same amount of chips compared to a Linux system.

I was working on an access control system many years ago that had to store a number (1000's) of user ids. I used a hashing technique to get into the ballpark, then did a brute force search from there. There was a significant overall speed increase from just a simple brute force search.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If you use linux, you can use it's own integrated (and very secure if configured properly) access control. And there are kernel drivers for smartcard readers and biometrics sensors. If you need to store more data just cross-compile MySQL.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Sqlite seems to be the choice of embedded databases at the moment.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

If you're so hung up on 'embedded databases' why not just use Linux? Use a LPC31xx device although price wise (according to Digikey) the Freescale iMX23 is $8 vs around $11 for the LPC3154. Just add a DDR ram chip and serial flash and that's your embedded linux system. No problems with graphics libraries, touch screens, etc. All been done.

According to your specs, you have 50,000 records of 128 bytes, theat equals 6.4MB of data - you'll need external storage for that be it serial flash or sdcard. If you add ram to cope with your graphics, then you've got the same amount of chips compared to a Linux system.

I was working on an access control system many years ago that had to store a number (1000's) of user ids. I used a hashing technique to get into the ballpark, then did a brute force search from there. There was a significant overall speed increase from just a simple brute force search.


You are right, but I do not know why I still fear Linux....
I just need a good starting point! I have always used windows....so where should someone like me start learning Linux the easier way?

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Google "micro244" it's a cheap and easy way to get started with embedded Linux and very easy to use. (they already have kernel support for all the SoC systems). That's a Samsun ARM9 doing about 400MHz.(*)

To learn about writing kernel drivers search out the (free to download) book by Rubini. That is http://lwn.net/Kernel/LDD3/ As K&R is the "bible" for C programmers this is the "bible" for anyone wanting to interface directly with hardware in Linux.

(*) actually one of the very cheapest ways to get started with Linux is actually a motherboard from Intel wit an Atom processor already soldered down. The boards start from about $50 (retail), $40 (in bulk) and deliver a 1.5GHz processor including GPU that can simply run something like Ubuntu off the shelf and has access to Ubuntu's software repository with 50,000+ install packages available so if you want an Apache webserver or an FTPd FTP deamon or OHCI/EHCI/UHCI USB host support (or just about anything you can think of) it's all prebuilt and ready to be installed. This includes SQLite.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I just need a good starting point! I have always used windows....so where should someone like me start learning Linux the easier way?

This is the absolutely first step you should take: Install a "friendly" desktop distribution, e.g. Ubuntu, and familiarize yourself with working on the command prompt /in a terminal window. If you have an old PC lying around you could use that one. Otherwize, install the free VirtualBox software, and create a virtual machine and install GNU/Linux there.

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I dabbled with linux and unix for many years but only got more serious when i started hacking wireless routers with openwrt. I'm not an expert by any means but i know enough to build kernels and write apps. Shell scripts and make files are still a bit of a mystery. Elektor have a aeries starting this month on a lpc31xx based board.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for your replies.

I love Digital
and you who involved in it!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

One more interesting way to dabble cheaply in embedded Linux are wireless routers. Most of them run Linux. Any wireless router compatible with openwrt will do. There are plenty of Linux packages already available for it and also a lot of information/resources etc.

The big advantage is that you can pick them up cheaply from local sources. Ebay, Craigslist, etc have plenty of them for cheap.

Markus