Newton 2.x Q&A Category: Data Storage (Soups)

Copyright © 1997 Newton, Inc. All Rights Reserved. Newton, Newton Technology, Newton Works, the Newton, Inc. logo, the Newton Technology logo, the Light Bulb logo and MessagePad are trademarks of Newton, Inc. and may be registered in the U.S.A. and other countries. Windows is a registered trademark of Microsoft Corp. All other trademarks and company names are the intellectual property of their respective owners.


For the most recent version of the Q&As on the World Wide Web, check the URL: http://www.newton-inc.com/dev/techinfo/qa/qa.htm
If you've copied this file locally, click here to go to the main Newton Q&A page.
This document was exported on 7/23/97.

Data Storage (Soups)

FrameDirty is Deep, But Can Be Fooled (8/19/94)

Q: Does the global function FrameDirty see changes to nested frames?

A: Yes. However, FrameDirty is fooled by changes to bytes within binary objects. Since strings are implemented as binary objects, this means that FrameDirty will not see changes to individual characters in a string. Since clParagraphViews try (as much as possible) to work by manipulating the characters in the string rather than by creating a new string, this means that FrameDirty can be easily fooled by normal editing of string data.

Here is an NTK Inspector-based example of the problem:

s := GetStores()[0]:CreateSoup("Test:DTS", []);
e := s:Add({slot: 'value, string: "A test entry", nested: {slot: 'notherValue}})
#4410B69  {slot: value, 
           String: "A test entry", 
           nested: {slot: notherValue}, 
           _uniqueID: 0}
FrameDirty(e)
#2        NIL

e.string[0] := $a; // modify the string w/out changing its reference
FrameDirty(e)
#2        NIL

EntryChange(e);
e.string := "A new string";    // change the string reference
FrameDirty(e)
#1A       TRUE

EntryChange(e);
e.nested.slot := 'newValue;    // nested change, FrameDirty is deep.
FrameDirty(e)
#1A       TRUE

s:RemoveFromStore()    // cleanup.

Limits on Soup Entry Size (2/12/96)

Q: How big can I make my soup entries?

A: In practice, entries larger than about 16K will significantly impact performance, and 8K should be considered a working limit for average entry size. No more than 32K of text (total of all strings, keeping in mind that one character is 2 bytes) can go in any soup entry.

There is no size limit built into the NewtonScript language; however, another practical limit is that there must be space in the NewtonScript heap to hold the entire soup entry.

There is a hard upper limit of 64K on Store object sizes for any store type. With SRAM-based stores there is a further block size limit of 32K. Trying to create an entry larger than this will result in evt.ex.fr.store exceptions. These limits are for the encoded form that the data takes when written to a soup, which varies from the object's size in the NS heap.

Newton Backup Utility and Newton Connection Utility cannot handle entries larger than 32K.

Note that Virtual Binary Objects (VBOs) in Newton 2.0 are no subject to the same restrictions. If you can store large objects as VBOs, you can store more information in your soup entries by referencing those VBOs.

Choosing EntryFlushXMit and EntryChangeXMit (4/17/96)

Q: What is the difference between the functions EntryFlushXMit and EntryChangeXMit?

A: The most important criterion when choosing between EntryFlushXMit and EntryChangeXMit is what will be done with the entry after the flush or change.

When an entry is added or changed, the system ensures that a cached entry frame exists in the NewtonScript heap. The system then writes the data in the frame to the store, skipping _proto slots. The result is that the data will be written to the store, and a cached frame will exist. Often, this is exactly what is desired because the entry is still needed since it will soon be accessed or modified.

In some cases, the data will be written to the soup with no immediate access afterwards. In other words, the data will not be used after being written to the soup. In these cases creating or keeping a cached entry frame in the NewtonScript heap is unnecessary and just wastes space and time. In these situations, EntryFlushXMit is a better option; it writes the data to the soup without creating the cached entry.

If any code accesses an entry that was just flushed, a new cached frame will be read in from the soup, just like when an existing entry is read for the first time.

The rule of thumb is: if an entry will be used soon after saving to the soup, then use AddXMit or EntryChangeXMit. If the entry will not soon be used again (so it doesn't need to take up heap space with the cached frame), then use AddFlushedXmit or EntryFlushXMit.

Some examples of good usage:
while entry do
begin
  entry.fooCount := entry.fooCount + 1;
  // nil appSymbol passed so don't broadcast
  EntryFlushXMit(entry, nil);   
  entry := cursor:Next();
end;                           // Could broadcast now 


foreach x in kInitialData do     // if new, may not need broadcast
    soup:AddFlushedXmit(Clone(x), nil);

Limit on Soup Name Length (12/10/96)

Q: What is the maximum number of characters I can use for a soup name?

A: With the Newton OS 2.0 release, soup names, like index data, are limited to 39 Unicode characters. If you attempt to create a soup with a longer name, the OS will create a soup with only the first 39 characters of the longer name. We recommend you avoid this truncation, because typically the truncation removes some or all of your registered signature, and the resulting name will not be guaranteed unique.

You can still provide longer/prettier names for users, by using the soupDef mechanism and putting the long name (typically without appended signature) in the 'userName slot of that data structure.

How to Use Begin and End Symbols with WhichEnd (1/8/97)

Q: The WhichEnd cursor method returns the symbols 'begin or 'end, depending on where the cursor is in a soup. Why does NTK complain when I try to check for these symbols?

A: Unfortunately, these are reserved words so NTK won't let you type them into normal code. The work-around is to enclose the symbol in vertical bars.

For instance, you can use code like:
if myCursor:WhichEnd() = '|begin| then
    :WeAreAtBeginning();

EntryChange on Modified Tags Array Throws -48022 (1/15/97)

Q: I added a tag to the array of tags in an entry. When I call EntryChange on the modified entry I get a -48022 error. What is wrong?

A: There is a known bug in Newton OS 2.0 (which is fixed in Newton OS 2.1) that can cause this problem. It can occur when you initially create an entry with an empty array as the value of the tag index slot. The workaround is not to use the empty array. Use NIL instead. If you need to add an array of tags later, you can do so.

How to Avoid Resets When Using VBOs (2/27/97)

Q: When writing large amounts of information to virtual binary objects (VBOs), my Newton device sometimes resets. What is going wrong?

A: The problem happens because of how the Newton OS manages the memory for VBOs. Writing to VBOs in low memory conditions can sometimes cause the device to reset because no free pages are available for other OS operations.

To work around this problem, you can periodically call the global function ClearVBOCache while modifying VBOs. You can also work around the problem by putting the VBO in a soup entry and using EntryChangeXmit or EntryFlushXmit.

In all versions of the Newton 2.x OS released to date, VBOs (including packages) are managed in 1K pages. When you write to a VBO, the "dirty" pages can remain in the system heap, taking up space. ClearVBOCache takes a reference to a VBO as an argument, and moves the dirty pages for a given VBO to the store, freeing up the system memory. Note that this function does not commit the changes to the VBO, while EntryChangeXmit and EntryFlushXmit do commit the changes.

The likelihood of the problem depends on the amount of system memory currently available and how many pages of VBOs are modified. We recommend that you modify no more than 32 pages of VBOs before committing the changes or calling ClearVBOCache. For example, modifying 32K of contiguous data, or a single byte in 32 different pages of one VBO, or even a single byte in 32 different VBOs all modify 32 total pages of VBO data. Don't do this too often, though. Calling ClearVBOCache repeatedly for modifications to the same page of a VBO or when there are only a few modified pages will needlessly slow the machine.

If you are experiencing this problem, you should consider redesigning your application to minimize the amount of uncommited VBO data. When finished with a VBO, commit it to a soup entry as soon as possible or let it become unreferenced.

CHANGED: Why Xmit Functions Seem to Leak Memory (5/5/97)

Q: I've noticed that the system seems to leak a little bit of memory every time I call an Xmit soup function, but not if I call the non-xmitting versions of the functions. For example, executing this code shows a little less free memory each time Stats() is called:
    gc(); stats();
    soup:AddToDefaultStoreXmit({ foo : "a test string"}, '|bar:SIG|);
    gc(); stats();
    soup:AddToDefaultStoreXmit({ foo : "another test string"}, '|bar:SIG|);
    gc(); stats();



A: There is no leak. What's actually going on is that the Xmit versions of the soup methods do their broadcasting in deferred actions. That's good, because it means that broadcast handlers that might throw or have other side effects won't break your code. A little bit of heap memory is used to keep track of the deferred action and its arguments. This memory is released after the deferred action executes, which is typically immediately after control returns to the top level.

If you select all the test code in the NTK Inspector and press the Enter key, NTK compiles the entire selection and executes it as a single operations, so control doesn't return to the top level (thus allowing the deferred actions to execute) until after the last operation. The deferred action created by each call to the Xmitting function will still be pending, so the space won't have been released yet, and stats reflect this. If the example is executed one line at a time, you'll see that no memory is actually leaked.

If you pass nil for the changeSym argument to the Xmit functions, no notification occurs and the deferred action is not created. Normally, this is a bad idea, since you want other applications to know about your soup changes. However, Xmit notification may not be necessary for specialized applications that use only their own application soups and do not publish information about reading/writing soup data for extensibility.

If you are modifying a very large number of entries (for instance, creating a new soup of thousands of entries), you might pass nil for the changeSym to void immediate notification. Afterwards, use the XmitSoupChange global function with the 'whatThe symbol. (See the documentation for XmitSoupChange for more information.)