[GUI Plugin] Highlights to Obsidian

[jm289765]

Version 1.3.3

• Clarified "filepath too long" error
• Fixed the problem of every highlight being sent twice
• Added option to choose whether to send local user's or web user's highlights

Originally Posted by petzi

Number 3 (Not replacing YAML variables) worked as before: Worked for smaller files (38,000 characters) but failed with longer files (238,687 characters).

Can you send me your full header format? I'm probably going to give up on this issue, but maybe I'll figure something out if I see what formatting you're using.

[petzi]

Here is my Obsidian header format using the Obsidian Templater plugin.

---
UID: <% tp.file.creation_date("YYYYMMDDHHmm") %>
creation: <% tp.file.creation_date("YYYY-MM-DD HH:mm") %>
updated: 
note: fleeting
status: to_process
language: english
source: calibre
category: books
subject: 
alias: FN {title}
---

# <% tp.file.creation_date("YYYYMMDDHHmm") %> - 👉 FN - {title}

Could you please explain to me what is the difference between user's web highlights and local user's highlights mentioned in your new option"Send web user's highlights (instead of local user's highlights)"? I know this solved the problem of exporting all highlights twice, but I do not understand what this option means.

[jm289765]

Originally Posted by petzi

Could you please explain to me what is the difference between user's web highlights and local user's highlights

Calibre stores highlights under two different user types: web and local. Local is for highlights made in the desktop app. Web is for highlights made from a web browser using calibre's content server.

You can set up calibre to make copies of web user highlights as local user highlights. Since H2O was previously sending both local and web highlights, this caused the duplicated notes bug:

Finally, if you use the calibre Content server’s in browser viewer, you can have the viewer sync its annotations with the browser viewer by going to Preferences → Miscellaneous in the viewer preferences and entering the username of the Content server viewer to sync with. Use the special value * to sync with anonymous users.

Originally Posted by petzi

Here is my Obsidian header format using the Obsidian Templater plugin.

[...]

From a Templater discussion, "it appears there may be some kind of race condition or other issue interfering with the processing of new notes in either Obsidian or Templater." The attached zip file adds a 100 ms delay between each note, which should fix the problem.

[musstank]

Hi! It's a great idea to make an option to send local or web highlights, but unfortunately the latter doesn't work for me. It says "There are no highlights to send".

In View -> Browser Annotations page there are highlights with my login (which I input with the password) as an author of annotations.

It's problematic because, as I see, these annotations duplicate only when I open desktop book-viewer, which I don't really use. So I need exactly web annotations

[jm289765]

Version 1.4.0:

• Added config for specifying the username of a web user
• Added config to wait for a specified time between each highlight sent (fixes YAML header bug)

Originally Posted by musstank

[...] option to send local or web highlights, but unfortunately the latter doesn't work for me. It says "There are no highlights to send".

Update to this version and go to H2O config -> Other Options -> Web user's username. Enter the username you use to log in to the calibre web viewer.

[musstank]

Now it works, thanks!

The only inconvenience for me now is the splitting of the files. But I've managed it somehow.
As I understood, the problems with too long path arise only when trying to add a lot of NEW highlights. So, for now, I have manually combined separated notes into one, and new highlights are being automatically added to that note at the bottom.

[MSBack]

Thanks for a great and much needed plugin.

I have a feature request related to headers:

Originally Posted by jm289765

With the way I've implemented headers, it'll append another header to the note each time you send highlights. To have a single header at the start of the file, you'll need to send highlights once with headers enabled, and then disable them.

Considering that an obvious use case for the headers is to add yaml metadata to the literature note, I think a more seamless way to only add the header once is necessary. Going into the formatting options every time you send new highlights to an existing note, and then again before you create a new one, is not really viable.

Here are some ideas for a better workflow:

1. Only create a header when using the "Send All Highlights" option. Or in other words, don't create a header when only sending new annotations, as this implies that an existing note is likely being updated.
   - However, this neglects the original use case for the header, which the standard header template represents, namely to add contextual information about the number of annotations sent, and when they were sent.
2. This leads to proposal number two: There needs to be an additional template element, namely a true note header created only once at the top of the document. Combined with point 1., this header would be added when sending all highlights. The current "header" template is still useful for adding information about each import, but it doesn't really serve the purpose of a document header, when highlights are sent multiple times.

In my view, "Send All Highlights..." and "Send New Highlights..." could be called "Create Literature Note in Obsidian" and "Update Literature Note in Obsidian" or something similar. However, I'm sure there are other workflows and situations where this might not be so simple. But I think it is worth discussing.

Beyond that, it would be great it there were more templating options like:

• More metadata about the books, such as ISBN, publisher and tags or even the comments block.
• Information about the color of the highlights so these could be either grouped together or somehow indicated by each highlight.

[jm289765]

Originally Posted by MSBack

I have a feature request related to headers

I've lost interest in H2O and probably won't do any further development on it, except maybe for important bug fixes. I originally made it so that I could easily review and update notes and important points from books, but I've found that incremental learning is a much more efficient way to do this.

If anyone else wants to add these features, my thoughts here might be useful to you:

Originally Posted by MSBack

a more seamless way to only add the header once

Since there isn't a way to ask Obsidian whether a file already exists, you would need to store a list of file names that highlights have been sent to. But this is a bad idea. It would cause problems if you delete a file and resend that book's highlights with different formatting, which I think is a relatively common use case.

So there isn't a way to fully automate once-per-file headers. But here's a way to make it easier for the user: add a second header format in the config and add a menu button to send only that header, and nothing else, for the currently selected books. (also put the header format options in their own config submenu, since the formatting menu is already pretty big).

Originally Posted by MSBack

More metadata about the books, such as ISBN, publisher and tags or even the comments block.

You would need to add a new input to make_book_format_dict(...) in highlight_sender.py. The "data" arg only has information that's stored in calibre annotations, which doesn't include most of the book's metadata. The book's metadata would be accessed from button_actions.py, send_highlights(...) function, "db" object.

Originally Posted by MSBack

Information about the color of the highlights so these could be either grouped together or somehow indicated by each highlight.

Unlike book metadata, highlight color is included in calibre annotation data and would be pretty easy to add to make_highlight_format_dict(...). They can be grouped together by using color as the sort key in H2O's config. It might be useful to add a way to use more than one sort key, e.g. sort by color and then location.

Originally Posted by MSBack

In my view, "Send All Highlights..." and "Send New Highlights..." could be called "Create Literature Note in Obsidian" and "Update Literature Note in Obsidian" or something similar. However, I'm sure there are other workflows and situations where this might not be so simple. But I think it is worth discussing.

Changing them from "All" and "New" to "Create" and "Update" would be misleading, since both of them will create and/or update notes, depending on whether the note already exists. And changing their functionality so that they can only create or update notes would have the problem mentioned above for once-per-note headers.

Regarding the term "Literature Note": Since H2O is used from calibre, it makes sense to label its functionality in terms of calibre ("Highlights") instead of Obsidian ("Note"). And it's unclear what the difference is between a "literature" note and a normal note.

I will continue to receive notifications from this thread.

[hadezb]

Originally Posted by jm289765

Highlights to Obsidian is a plugin that formats and sends highlights to the Obsidian.md markdown editor.

Main Features:

• Send all highlights to Obsidian
• Only send new highlights or highlights of selected books
• Customizable formatting
• Optional header to be attached to the beginning of each batch of highlights
• When multiple highlights are sent to the same file, user can specify how they will be sorted

Formatting options:

Spoiler:

Book Data:

• {title}: Title of the book the highlight is in.
• {authors}: Authors of the book the highlight is in.
• {bookid}: The book's ID in calibre.

Highlight Data:

• {highlight}: The highlighted text.
• {blockquote}: The highlighted text, formatted as a blockquote. An arrow and a space "> " are added to the beginning of each line.
• {notes}: The user's notes on this highlight, if any notes exist. There is a config option that allows you to set different formatting depending on whether a highlight includes notes.
• {url}: A calibre url to open the ebook viewer to this highlight. Note that this may not work if your library's name contains unsafe URL characters. Numbers, letters, spaces, underscores, and hyphens are all safe.
• {location}: The highlight's EPUB CFI location in the book. For example, "/2/8/6/5:192". As a sort key, this will order highlights by their position in the book.
• {timestamp}: The highlight's Unix timestamp. As a sort key, this will order highlights by when they were made.
• {uuid}: The highlight's unique ID in calibre. For example, "TlNlh8_I5VGKUtqdfbOxDw".

Time Data:

• {date}: Date the highlight was made, formatted as YYYY-MM-DD.
• {time}: Time the highlight was made, formatted as HH:MM:SS.
• {datetime}: Date and time highlight was made, formatted as YYYY-MM-DD HH:MM:SS.
• {day}: Day of the month the highlight was made, as in 03 or 17.
• {month}: Month the highlight was made, as in 04 for April or 10 for October.
• {year}: Full year the highlight was made, as in 2022.
• {hour}: Hour the highlight was made, based on a 24-hour (not 12-hour) system.
• {minute}: Minute the highlight was made.
• {second}: Second the highlight was made.
• {utcnow}: current time, formatted same as {datetime}.
• {datenow}: Current date, formatted same as {date}.
• {timenow}: Current time, formatted same as {time}.
• {timezone}: The timezone that your computer is currently set to. Note that this may not always match the timezone the highlight was made in. Also note that this might use the full name "Coordinated Universal Time" instead of the abbreviation "UTC".
• {utcoffset}: The UTC offset of your computer's current time zone. For example, UTC time gives +0:00. EST time can be -4:00 or -5:00, depending on daylight savings time.
• All time options use UTC by default. To use your computer's local time zone instead, add "local" to the beginning: {localdate}, {localtime}, {localdatetime}, {localday}, {localmonth}, {localyear}, {localhour}, {localminute}, {localsecond}, {localnow}, {localdatenow}, {localtimenow}.

H2O Data:

• {totalsent}: The total number of highlights being sent.
• {booksent}: The total number of highlights being sent to this Obsidian note. If a large note is split into multiple smaller notes, {booksent} will give the total being sent to all of those smaller notes.
• {highlightsent}: This highlight's position in the highlights being sent to this note. For example, "{highlightsent} out of {booksent}" might result in "3 out of 5".

Note: For an example of how to use these, see the default format settings in the plugin's config.

Special Notes:

• After installing, go to Preferences -> Toolbars & menus -> The main toolbar. The Highlights to Obsidian menu button is listed as H2O.
• You can set keyboard shortcuts in Preferences -> Shortcuts -> H2O.
• Source code
• Version history

Is there a formatting option for the publication date of my document? i.e. {published}

Appreciate the help.

[jm289765]

Originally Posted by hadezb

Is there a formatting option for the publication date of my document? i.e. {published}

No. For data about the book or document itself, the only implemented formatting options are {title}, {authors}, and {bookid}.

[JLC42]

How exactly do you set the location of the vault? All I can see is the option to set my vault name.

[jm289765]

Originally Posted by JLC42

How exactly do you set the location of the vault? All I can see is the option to set my vault name.

H2O doesn't currently have an option to set the full path of the vault. It uses the Obsidian URI new action with the parameters vault, file, and content.

If you want to modify H2O to let you specify the full path, you could probably just change line 626 of highlight_sender.py from

"file": note_file if len(note_file) < 180 else note_file[:172] + "... " + note_file[-4:],

to

"path": note_file if len(note_file) < 220 else note_file[:212] + "... " + note_file[-4:],

and then, in the config, write the full path where you would normally put the file name. The vault name option will be overridden.

Note: Replacing the "file" parameter instead of the "vault" parameter on line 624 lets you use formatting options in the path.

[JLC42]

Just tried to use this for the first time & it's saying "There are no highlights to send." despite just highlighting content on a book and making notes.

[jm289765]

Originally Posted by JLC42

Just tried to use this for the first time & it's saying "There are no highlights to send." despite just highlighting content on a book and making notes.

Here's two things you could try:
- In H2O Config -> Other Options, make sure the "Last time highlights were sent" option is set to a time before you made a highlight in the book
- If you use the calibre content server to read ebooks in a web browser, adjust the web user settings in H2O Config -> Other Options

If neither of these work, then I don't know how to fix this problem.

[JLC42]

Originally Posted by jm289765

Here's two things you could try:
- In H2O Config -> Other Options, make sure the "Last time highlights were sent" option is set to a time before you made a highlight in the book
- If you use the calibre content server to read ebooks in a web browser, adjust the web user settings in H2O Config -> Other Options

If neither of these work, then I don't know how to fix this problem.

"Last time highlights were sent" is still set to the default date of 1970.