6.0 KiB
Links
- Mozilla places database
- places explorations
- Firefox nsNavBookmarks.cpp
- Running firefox in test mode:
MOZILLA_DISABLE_PLUGINS=1 ./firefox -safe-mode -no-remote \ -profile $(mktemp -d tmp-firefox-profile-XXXXXXXXXX) -offline about:blank
- https://shallowsky.com/blog/tech/web/modifying-omni.ja.html
File locking & VFS
-
Sqlite allows file locking of the database using the local file system VFS. Previous versions of FF allowed external processes to access the file.
-
Since firefox v(?) this has changed, when initializing the database FF checks the preference option
storage.multiProcessAccess.enabled
which is not documented officially.
Source code links:
- https://dxr.mozilla.org/mozilla-central/source/storage/TelemetryVFS.cpp#884
- https://dxr.mozilla.org/mozilla-central/source/storage/mozStorageService.cpp#377
- the change on github
UPDATE (2020-09-01)
-
Mozilla keeps changing this option, a new change in vfs locks happenend in
-
We need to come up with a strategy to avoid being at the mercy of mozilla changes, the following strategy is proposed:
- Main strategy try somehow to make Firefox use non exclusive lock
- If non exclusive lock not possible use the copy method as follows:
- Copy the
places.sqlite*
database files to a temporary destination - Parse bookmarks from the temporary destination
Queries
See inside ./queries.sql
Find duplicate links
select metadata,url from bookmarks join (select url,metadata, count(url) as x from bookmarks group by url having x > 1) using (metadata,url) order by url limit 10
Find all places for a tag
SELECT moz_places.id, moz_places.url, moz_places.title, moz_bookmarks.parent
FROM moz_places
LEFT OUTER JOIN moz_bookmarks
ON moz_places.id = moz_bookmarks.fk
WHERE moz_bookmarks.parent = N(tag id ?)
Mozilla places.sqlite
Updates Events
When firefox updates places.sqlite
it emits multiple WRITE
events. We need
to implement a debounce system to avoid reparsing everything.
Table moz_bookmarks
-
keyword_id is not used
-
folder_type is not used
columns:
id type fk parent position title
Column Types
type
1: URL 2: Tag or Folder 3:
parent
(main ids)
-- Root FOLDERS
- Root
- Bookmarks Menu
- Bookmarks Toolbar
- Tags
- Other Bookmarks
- Mobile Bookmarks -- Mozilla Folders
- Moz Firefox Folder
- Help & Tutorials Folder 9.. .. 12 Last mozilla reserved id -- User bookmarks
- User Bookmark XX
user bookmarks start from id=13 and more
NOTES:
- Mozilla reserves ids (1-12 for their own use, user defined bookmarks start at id=13 ?
for ex to find all tags
select * from moz_bookmarks where parent = 4
- !Regarding mobile bookmarks:
By default firefox hides the mobile bookmarks folder.
It can be activated using the prefs.js or about:config boolean option
browser.bookmarks.showMobileBookmarks
to true
How are bookmarks represented
1. Bookmarks with tags
|----|------|----------|----------------|-------------|
id | type | fk | title | parent |
---|---|---|---|---|
42 | 2 | NULL | Tag Name | 4 (Tags) |
xx | 1 | place_id | Bookmark Title | 5 (unfiled) |
yy | 1 | place_id | NULL | 42 |
---- | ------ | ---------- | ---------------- | ------------- |
A. An entry for the tag itself
B. The bookmark entry (type=1) which points also to fk=moz_places.id with as
parent the dummy unfiled
folder.
C. A link between Tag(A) and Bookmark (B)
2. Bookmarks with folders
|----|------|----------|----------------|------------------|
id | type | fk | title | parent |
---|---|---|---|---|
xx | 1 | place_id | Bookmark Title | parent_folder_id |
---- | ------ | ---------- | ---------------- | ------------------ |
A. An entry for the bookmark with a link to places and the folder id as parent. note: the parent folder can itself be child of other folders recursively until reaching any of the root folders: menu(2), toolbar(3) and mobile(6)
3. Bookmarks with tags AND within folders
|----|------|----------|----------------|------------------|
id | type | fk | title | parent |
---|---|---|---|---|
42 | 2 | NULL | Tag Name | 4 (Tags) |
xx | 1 | place_id | Bookmark Title | parent_folder_id |
yy | 1 | place_id | NULL | 42 |
---- | ------ | ---------- | ---------------- | ------------------ |
A. An entry for the tag itself B. The bookmark entry (type=1) which points also to fk=moz_places.id note that compared to bookmarks with tags only, this links' parent is a folder and not the unfiled dummy folder C. A link between Tag(A) and Bookmark (B)
Parsing Algorithm
- Create a main Root node
- Recursive parsing from the Selected Root Folders (id = 1) in the same way as chrome
- To compare between a URL and folder use the type (2=folder, 1=URL)
Root folders to parse from
Bookmarks Toolbar
: id = 3 (where parent = 3)Tags
: id = 4 (where parent = 4)Mobile Bookmarks
: id = 6 (where parent = 6)
Sqlite WAL
- WAL automatic checkpoints may prevent us from seeing bookmark changes
- we may need ways to force vacuuming of the WAL file or reverse engineer it with a tool like this one ?
moz_bookmarks timestamps
- firefox stores timestamps in milliseconds as integer
- sqlite3 strftime('%s', ...) returns seconds