gosuki/browsers/firefox

Links

• Mozilla places database
• places explorations
• Firefox nsNavBookmarks.cpp
• Running firefox in test mode: MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote \ -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank
• https://shallowsky.com/blog/tech/web/modifying-omni.ja.html

File locking & VFS

• Sqlite allows file locking of the database using the local file system VFS. Previous versions of FF allowed external processes to access the file.

• Since firefox v(?) this has changed, when initializing the database FF checks the preference option storage.multiProcessAccess.enabled which is not documented officially.

Source code links:

• https://dxr.mozilla.org/mozilla-central/source/storage/TelemetryVFS.cpp#884
• https://dxr.mozilla.org/mozilla-central/source/storage/mozStorageService.cpp#377
• the change on github

UPDATE (2020-09-01)

• Mozilla keeps changing this option, a new change in vfs locks happenend in

• We need to come up with a strategy to avoid being at the mercy of mozilla changes, the following strategy is proposed:

  1. Main strategy try somehow to make Firefox use non exclusive lock
  2. If non exclusive lock not possible use the copy method as follows:
  • Copy the places.sqlite* database files to a temporary destination
  • Parse bookmarks from the temporary destination

Queries

See inside ./queries.sql

Find duplicate links

select metadata,url from bookmarks join (select url,metadata, count(url) as x from bookmarks group by url having x > 1) using (metadata,url) order by url limit 10

Find all places for a tag

SELECT moz_places.id, moz_places.url, moz_places.title, moz_bookmarks.parent    
    FROM moz_places    
    LEFT OUTER JOIN moz_bookmarks    
    ON moz_places.id = moz_bookmarks.fk    
    WHERE moz_bookmarks.parent = N(tag id ?)

Mozilla places.sqlite

Updates Events

When firefox updates places.sqlite it emits multiple WRITE events. We need to implement a debounce system to avoid reparsing everything.

Table moz_bookmarks

• keyword_id is not used

• folder_type is not used

  columns:

  id type fk parent position title

Column Types

type

1: URL 2: Tag or Folder 3:

parent (main ids)

-- Root FOLDERS

1. Root
2. Bookmarks Menu
3. Bookmarks Toolbar
4. Tags
5. Other Bookmarks
6. Mobile Bookmarks -- Mozilla Folders
7. Moz Firefox Folder
8. Help & Tutorials Folder 9.. .. 12 Last mozilla reserved id -- User bookmarks
9. User Bookmark XX

user bookmarks start from id=13 and more

NOTES:

• Mozilla reserves ids (1-12 for their own use, user defined bookmarks start at id=13 ?

for ex to find all tags select * from moz_bookmarks where parent = 4

• !Regarding mobile bookmarks:

By default firefox hides the mobile bookmarks folder. It can be activated using the prefs.js or about:config boolean option browser.bookmarks.showMobileBookmarks to true

How are bookmarks represented

1. Bookmarks with tags

|----|------|----------|----------------|-------------|

id	type	fk	title	parent
42	2	NULL	Tag Name	4 (Tags)
xx	1	place_id	Bookmark Title	5 (unfiled)
yy	1	place_id	NULL	42
----	------	----------	----------------	-------------

A. An entry for the tag itself B. The bookmark entry (type=1) which points also to fk=moz_places.id with as parent the dummy unfiled folder. C. A link between Tag(A) and Bookmark (B)

2. Bookmarks with folders

|----|------|----------|----------------|------------------|

id	type	fk	title	parent
xx	1	place_id	Bookmark Title	parent_folder_id
----	------	----------	----------------	------------------

A. An entry for the bookmark with a link to places and the folder id as parent. note: the parent folder can itself be child of other folders recursively until reaching any of the root folders: menu(2), toolbar(3) and mobile(6)

3. Bookmarks with tags AND within folders

|----|------|----------|----------------|------------------|

id	type	fk	title	parent
42	2	NULL	Tag Name	4 (Tags)
xx	1	place_id	Bookmark Title	parent_folder_id
yy	1	place_id	NULL	42
----	------	----------	----------------	------------------

A. An entry for the tag itself B. The bookmark entry (type=1) which points also to fk=moz_places.id note that compared to bookmarks with tags only, this links' parent is a folder and not the unfiled dummy folder C. A link between Tag(A) and Bookmark (B)

Parsing Algorithm

• Create a main Root node
• Recursive parsing from the Selected Root Folders (id = 1) in the same way as chrome
• To compare between a URL and folder use the type (2=folder, 1=URL)

Root folders to parse from

• Bookmarks Toolbar: id = 3 (where parent = 3)
• Tags: id = 4 (where parent = 4)
• Mobile Bookmarks: id = 6 (where parent = 6)

Sqlite WAL

• WAL automatic checkpoints may prevent us from seeing bookmark changes
• we may need ways to force vacuuming of the WAL file or reverse engineer it with a tool like this one ?

moz_bookmarks timestamps

• firefox stores timestamps in milliseconds as integer
• sqlite3 strftime('%s', ...) returns seconds