Zetteldeft

This document contains all the source code to a set of functions for emacs which aim to extend the deft package and turn it into a (very very) basic Zettelkasten note-taking system.

Check out the Github repository to get the source. Read on for an introduction and some documentation.

Latest additions:

1 What?

1.1 A Zettelkasten system for emacs based on deft

This is my feeble attempt at recreating a Zettelkasten environment by extending the excellent deft package in emacs.1 I call it zetteldeft.

It is inspired by the The Archive app. For this and more on the Zettelkasten way of taking notes, see zettelkasten.de. They have a forum for discussion on both software and the specifics of the Zettelkasten philosophy.

The code that follows is created and maintained for my personal use, shared here in hope that it can benefit others as well. I’d be happy to learn how you use it and expand upon it.

It is very much WIP and I’m fairly new to elisp, so it might contain some stupid code.

Anyway, here we go.

1.2 How to use this source

This package requires:

  • deft, obviously
  • avy to jump & search

From the Github repository, either

  • download the zetteldeft.el file,
  • download the org-file and org-tangle it yourself. It should contain everything.

Whichever way you go, load up the package by adding the package to your load path and requiring:

(add-to-list 'load-path "~/path/to/folder/"))
(require 'zetteldeft)

Or, if you use use-package, do something like:

(use-package zetteldeft
  :load-path "~/git/zetteldeft/"
  :after deft)

and you’re good to go! Well, not quite. First you must read on about the basics of zetteldeft. You’ll also want some keybindings. Check out the suggested setup below.

1.3 Basic concepts

Notes reside in the deft-directory. Notes are written in org-mode syntax (although most functions should work in markdown as well).

The filename of a note starts with a unique id based on the time and the date, for example: 2018-07-09-2115 This is a note.org.

This unique id can be used to link notes together. A link consists of the § character followed by the id. For example: §2018-07-09-2115 should link to the file above. A link can appear anywhere in the text. See below for advanced information about IDs and links.

When searching deft with the id as a filter, you’ll find both the original note (with the id in its name) and all the notes that link to this note (with the id in its body). Do so with zd-search-current-id and zd-avy-link-search respectively

Notes can contain tags in plain text: words prepended with a #. This is a tag: #tag. Tags make it easy to retrieve notes. They can appear anywhere in the note, but I’d suggest putting them somewhere at the top.

1.4 Basic actions

Create a note with zd-new-file and provide a name.

To insert links to other notes, either

  • enter their links manually,
  • use zd-find-file-id-insert and select a file from the list,

With zd-find-file-full-title-insert, you guessed it, the note’s title is included as well.

To easily branch out from the current note (i.e. create a new one and link to it in one go), use zd-new-file-and-link.

To search for a tag or anything else under cursor, use zd-search-at-point. Combined with the power of avy to jump to any character on screen, use these to jump and search in one go: zd-avy-link-search and zd-avy-tag-search.

To open the note behind a link, use zd-follow-link.

Want more functionality? How about showing a list of tags or gathering notes with a certain search string? Or maybe a graph visualizing how notes are linked?

Still hungry? I’m welcoming both contributions and suggestions. Feel free to submit comments or pull requests on Github.

1.5 An overview

While there are many, these should be enough to get you started. Default keybindings are suggested at the end of this document. Here is an overview.

Function Use Keybinding
zd-new-file Create new note and open SPC d n
zd-new-file-and-link Create new note and insert link SPC d N
zd-find-file-id-insert Pick a note and insert a link SPC d i
zd-follow-link Follow a link SPC d f
zd-avy-link-search Select and search a link’s ID SPC d l
zd-avy-tag-search Select a tag and search for it SPC d t
zd-search-at-point Search for thing at point SPC d s
zd-search-current-id Search for id of current file SPC d c

Read on, dear reader, for all of this and much more.

2 The zetteldeft package

2.1 Package preparation

The required preamble and some other initial settings. To know how this package works, please skip right past this to the next section.

2.1.1 Preamble

Some declaration.

;;; zetteldeft.el --- a simple package                     -*- lexical-binding: t; -*-

;; Copyright (C) 2018-2019  EFLS

;; Author: EFLS <Elias Storms>
;; Website: https://efls.github.io/zetteldeft/
;; Keywords: deft zettelkasten zetteldeft
;; Version: 0.0.1

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; Zetteldeft: turn deft into a zettelkasten writing system to create linked notes.

;;; Code:

2.1.2 Requirements

deft is required, obviously, and avy is needed for some utility functions.

(require 'deft)

(unless (require 'avy nil 'no-error)
  (user-error "Avy not installed, required for zd-avy-* functions"))

Since february 2019, the avy API changed and avy--generic-jump is replaced by avy-jump. Unfortunately, this change doesn’t seem to be indicated with a specific version number. So let’s check whether that function is available, and show a message when it’s not.

(unless (fboundp 'avy-jump)
  (display-warning 'zetteldeft
    "Function `avy-jump' not available. Please update `avy'"))

2.1.3 Customization

For easy but minor customization options.

(defgroup zetteldeft nil
  "A zettelkasten on top of deft.")

2.2 Basic zetteldeft functions

In this section:

2.2.1 Search functions

2.2.1.1 zd-get-thing-at-point returns string

Returns the thing at point as string.

Tries to get, in the following order:

  • links between [[
  • hashtags: §, # or @
  • words

Based on snippet suggested by saf-dmitry on deft’s Github.

(defun zd-get-thing-at-point ()
  "Return the thing at point, which can be a link, tag or word."
  (require 'thingatpt)
  (let* ((link-re "\\[\\[\\([^]]+\\)\\]\\]")
         (htag-re "\\([§#@][[:alnum:]_-]+\\)"))
   (cond
    ((thing-at-point-looking-at link-re)
      (match-string-no-properties 1))
     ((thing-at-point-looking-at htag-re)
      (match-string-no-properties 1))
     (t (thing-at-point 'word t)))))
2.2.1.2 zd-search-at-point thing at point

Search the thing at point.

Based on snippet suggested by saf-dmitry on deft’s Github.

(defun zd-search-at-point ()
  "Search deft with `thing-at-point' as filter.
Thing can be a double-bracketed link, a hashtag, or a word."
  (interactive)
  (let ((string (zd-get-thing-at-point)))
   (if string
       (zd-search-global string t)
     (user-error "No search term at point"))))
2.2.1.3 zd-search-global for string

Search with deft for given string. If there is only one result, that file is opened, unless additional argument is true.

Based on snippet suggested by saf-dmitry on deft’s Github.

(defun zd-search-global (str &optional dntOpn)
  "Search deft with STR as filter.
If there is only one result, open that file (unless DNTOPN is true)."
  ;; Sanitize the filter string
  (setq str (replace-regexp-in-string "[[:space:]\n]+" " " str))
  ;; Call deft search on the filter string
  (let ((deft-incremental-search t))
   (deft)
   (deft-filter str t))
  ;; If there is a single match, open the file
  (unless dntOpn
   (when (eq (length deft-current-files) 1)
     (deft-open-file (car deft-current-files)))))
2.2.1.4 zd-search-filename for string

Deft search on filename. If there is only one result, open that file.

Incremental search is turned off, and the filter is set to filenames only.

(defun zd-search-filename (thisStr &optional otherWindow)
  "Search for deft files with string THISSTR in filename.
Open if there is only one result (in another window if OTHERWINDOW is non-nill)."
  ;; Sanitize the filter string
  (setq thisStr (replace-regexp-in-string "[[:space:]\n]+" " " thisStr))
  ;; Call deft search on the filter string
  (let ((deft-filter-only-filenames t))
   (deft-filter thisStr t))
  ;; If there is a single match, open the file
  (when (eq (length deft-current-files) 1)
    (deft-open-file (car deft-current-files) otherWindow)))
2.2.1.5 zd-search-current-id searches current id

Deft search on the id of the current file.

This function is useful to easily see which notes link to the current file.

Result is not opened automaticaly.

Steps:

  1. Get the filename from the current buffer.
  2. Lift the ID from it.
  3. Search with resulting string.
(defun zd-search-current-id ()
  "Search deft with the id of the current file as filter.
Open if there is only one result."
  (interactive)
  (zd--check)
  (zd-search-global (zd-lift-id (file-name-base (buffer-file-name))) t))
2.2.1.6 zd-get-file-list returns file list with search term

Get a list of the files with given search string.

To fix: sorting of results.

The code searches for the given string and returns deft-current-files.

(defun zd-get-file-list (srch)
  "Return a list of files with the search item SRCH."
  (let ((deft-current-sort-method 'title))
    (deft-filter srch t)
    deft-current-files))

2.2.2 IDs, links and tags

2.2.2.1 A note on nomenclature, IDs and links

In zetteldeft, the concepts of “links” and “IDs” have related meanings, but in the documentation they are not synonyms.

An ID refers to the unique numeric string identifying each note. For example: 2019-01-20-1433. These are generated by zd-generate-id.

A link is an ID prepended by a character to easily identify it as a link. For example: §2019-01-20-1433.

This identifying character can be changed by setting the zd-link-indicator variable and is § by default. Note that if this variable is set to nil, the zd-avy-link-search and zd-avy-file-search functions will not work.

2.2.2.2 zd-id-format for generating ID strings

Format used to generate ids.

See documentation of format-time-string for more info on possible placeholders.

If you customize this value, make sure to edit the zd-id-regex as well, so that the IDs can be found by other functions.

(defcustom zd-id-format "%Y-%m-%d-%H%M"
  "Format used when generating zetteldeft IDs.

Be warned: the regexp to find IDs is set separately.
If you change this value, set `zd-id-regex' so that
the IDs can be found.

Check the documentation of the `format-time-string'
function to see which placeholders can be used."
  :type 'string
  :group 'zetteldeft)

Another popular option would be to set this value to "%Y%m%d%H%M", so that a similar ID is generated without any dashes. See below for a corresponding regular expression.

While we’re at it, lets tell deft to use this format when creating new files. For good measure: I advise creating new notes in the zetteldeft system with zd-new-file or zd-new-file-and-link as defined below, rather than through deft itself.

(setq deft-new-file-format zd-id-format)

Next up, a function to generate an ID string in the above format.

(defun zd-generate-id ()
  "Generate an ID in the format of `zd-id-format'."
  (format-time-string zd-id-format))
2.2.2.3 zd-id-regex for finding IDs

The regular expression used to search for zetteldeft IDs as set in zd-id-format.

The default regex dictates that a zetteldeft ID should consist of:

  1. a series of exactly 4 numbers
  2. followed by exactly 3 sets of a dash and two or more numbers
(defcustom zd-id-regex "[0-9]\\{4\\}\\(-[0-9]\\{2,\\}\\)\\{3\\}"
  "The regular expression used to search for zetteldeft IDs.
Set it so that it matches strings generated with
`zd-id-format'."
  :type 'string
  :group 'zetteldeft)

If you use a "%Y%m%d%H%M" format for note naming, you might want to set the regular expression to "20[0-9]\\{10\\}" so that it matches any string starting with 20 followed by 10 other digits.

2.2.2.4 zd-link-indicator prepends ID links

To make it easier to distinguish links to zetteldeft notes, the ID can be prepended with a symbol. By default, this is set to §, but it can be changed (or nil).

(defcustom zd-link-indicator "§"
  "String to indicate zetteldeft links.
String prepended to IDs to easily identify them as links to zetteldeft notes.
This variable should be a string containing only one character."
  :type 'string
  :group 'zetteldeft)
2.2.2.5 zd-lift-id filters the ID from a string

Return the zetteldeft ID from any string.

Searches with a temporary buffer, from the end of the string backwards (hence the -1 argument), which implies that the last zetteldeft string is returned.

(defun zd-lift-id (str)
  "Extract the zetteldeft ID from STR with the regular expression stored in `zd-id-regex'."
  (with-temp-buffer
    (insert str)
    (when (re-search-forward zd-id-regex nil t -1)
      (match-string 0))))

Or are there better ways than working with-temp-buffer?

Here is a little test.

(zd-lift-id "2018-11-09-1934-12 Some text (1989) - testing (2000 p. 12-25)")
2018-11-09-1934
2.2.2.6 zd-tag-regex for finding tags

This regular expression indicates what tags can look like.

By default, tags start with a # or @ and contain least one or more lower case letters. Dashes are allowed.

(defcustom zd-tag-regex "[#@][a-z-]+"
  "Regular expression for zetteldeft tags."
  :type 'string
  :group 'zetteldeft)

Note that this regular expression does not handle hashtags used in, for example, urls.

2.2.3 Finding & linking files from minibuffer

2.2.3.1 zd-find-file opens file from minibuffer

Select file from the deft folder from the minibuffer.

Based on deft-find-file.

(defun zd-find-file (file)
  "Open deft file FILE."
  (interactive
    (list (completing-read "Deft find file: "
           (deft-find-all-files-no-prefix))))
  (deft-find-file file))
2.2.3.2 zd-find-file-id-insert inserts file id

Select file from minibuffer and insert its link, prepended by § (or zd-link-indicator to be precise).

Based on deft-find-file.

(defun zd-find-file-id-insert (file)
  "Find deft file FILE and insert a link."
  (interactive (list
    (completing-read "File to insert id from: "
      (deft-find-all-files-no-prefix))))
  (insert (concat zd-link-indicator (zd-lift-id file))))
2.2.3.3 zd-find-file-full-title-insert inserts id and title

Select file from minibuffer and insert its link, prepended by § (or zd-link-indicator).

Based on deft-find-file.

(defun zd-find-file-full-title-insert (file)
  "Find deft file FILE and insert a link with title."
  (interactive (list
    (completing-read "File to insert full title from: "
      (deft-find-all-files-no-prefix))))
  (insert (concat zd-link-indicator (file-name-base file))))

2.2.4 New file creation

2.2.4.1 zd-new-file creates new file

Create new file with filename as zd-id-format and a string.

Either provide a name as argument, or (when called interactively) enter one in the mini-buffer. Unless an additional parameter is provided, the title (prepended by zd-title-prefix and appended by zd-title-suffix) is automatically added.

When evil is used, enter the insert state as well. The full name is added to the kill ring.

Note that the file is only actually created upon save.

(defun zd-new-file (str &optional empty)
  "Create a new deft file.
Filename is `zd-id-format' appended by STR.
No file extension needed.

The title is inserted in `org-mode' format (unless EMPTY is true)
and the file name (without extension) is added to the kill ring.
When `evil' is loaded, enter instert state."
  (interactive (list (read-string "name: ")))
  (let* ((zdId (zd-generate-id))
         (zdName (concat zdId " " str)))
  (deft-new-file-named zdName)
  (kill-new zdName)
  (unless empty (zd-insert-title))
  (save-buffer)
  (when (featurep 'evil) (evil-insert-state))))
2.2.4.2 zd-new-file-and-link inserts generated id

Generate an id, append a name, and generate a new file based on id and link.

Either provide a name as argument, or enter one in the mini-buffer.

(defun zd-new-file-and-link (str)
  "Insert generated id with `zd-id-format' appended with STR.
Creates new deft file with id and STR as name."
  (interactive (list (read-string "name: ")))
  (insert zd-link-indicator (zd-generate-id) " " str)
  (zd-new-file str))

2.2.5 Moving around with avy

2.2.5.1 Following links with zd-follow-link

This is a wrapper function to follow links to a file. When point is in a link, open the note it links to. When point is not in a link, avy to jump to and open a selected link.

(defun zd-follow-link ()
  "Follows zetteldeft link to a file if point is on a link.
Prompts for a link to follow with `zd-avy-file-search' if it isn't."
  (interactive)
  (if (thing-at-point-looking-at (concat zd-link-indicator zd-id-regex))
      (zd-search-filename (zd-lift-id (zd-get-thing-at-point)))
    (zd-avy-file-search)))
2.2.5.2 zd-avy-tag-search

Use avy to jump to a tag and search for it.

The search term should include the # as tag identifier, so it’s as easy as jumping to the # and running zd-search-at-point.

Avy uses zd-tag-regex as a regular expression.

(defun zd-avy-tag-search ()
  "Call on avy to jump to a tag.
Tags are filtered with `zd-tag-regex'."
  (interactive)
  (save-excursion
    (avy-jump zd-tag-regex)
    (zd-search-at-point)))
2.2.5.3 zd-avy-link-search

Use avy to jump to a link search for its ID in deft.

Jumps to a link identifier (i.e. zd-link-indicator, § by default) and searches for the ID found. Note that this will not work when zd-link-indicator is set to nil.

(defun zd-avy-link-search ()
  "Use `avy' to perform a deft search on a zetteldeft link.
Links are found via `zd-link-indicator'.
Opens immediately if there is only one result."
  (interactive)
  (unless zd-link-indicator
    (user-error "Zetteldeft avy functions won't work when `zd-link-indicator' is nil"))
  (save-excursion
    (avy-goto-char (string-to-char zd-link-indicator))
    (zd-search-global (zd-lift-id (zd-get-thing-at-point)))))
2.2.5.4 zd-avy-file-search

Use avy to jump to a link and find the corresponding file. Since the ID should be unique, there should be only one result. That file is then opend (in another window if requested).

(defun zd-avy-file-search (&optional otherWindow)
 "Use `avy' to follow a zetteldeft link.
Links are found via `zd-link-indicator'
Open that file (in another window if OTHERWINDOW)."
  (interactive)
  (unless zd-link-indicator
    (user-error "Zetteldeft avy functions won't work when `zd-link-indicator' is nil"))
  (save-excursion
    (avy-goto-char (string-to-char zd-link-indicator))
    (zd-search-filename (zd-lift-id (zd-get-thing-at-point)) otherWindow)))

Let’s also define a function to open a file in another window. Selection of the window occurs via ace-window.

(defun zd-avy-file-search-ace-window ()
  "Use `avy' to follow a zetteldeft link in another window.
When there is only one search result, as there should be,
open that file in a window selected through `ace-window'."
  (interactive)
  (unless zd-link-indicator
    (user-error "Zetteldeft avy functions won't work when `zd-link-indicator' is nil"))
  (require 'ace-window)
  (save-excursion
    (avy-goto-char (string-to-char zd-link-indicator))
    (let ((ID (zd-lift-id (zd-get-thing-at-point))))
      (select-window (aw-select "Select window..."))
      (zd-search-filename ID))))

2.2.6 Utility functions

2.2.6.1 Deft new search

The following function launches deft, clears the filter and enters evil-insert-state (when evil is used).

(defun zd-deft-new-search ()
  "Launch deft, clear filter and enter insert state."
  (interactive)
  (deft)
  (deft-filter-clear)
  (when (featurep 'evil) (evil-insert-state)))
2.2.6.2 zd--check checks if file is part of zetteldeft

A quick but necessary check to see whether the provided file is part of the deft directory.

To achieve this, first check whether the buffer is visiting a file. When that is the case, take the path of the file the buffer is currently visiting, and check whether the deft-directory is part of that. Signal a user error if it is not.

The file-truename is there to make sure that deft-directory is first expanded to an absolute path before comparing it to the file name of the current buffer (which is already an absolute path).

(defun zd--check ()
  "Check if the currently visited file is in `zetteldeft' territory:
whether it has `deft-directory' somewhere in its path."
  (unless (buffer-file-name)
    (user-error "Buffer not visiting a file"))
  (unless (string-match-p
            (regexp-quote (file-truename deft-directory))
            (buffer-file-name))
    (user-error "Not in zetteldeft territory")))
2.2.6.3 zd-file-rename renames visited file

Rename the current file. Based on the function deft-rename-file with only minor changes in the way old-filename is set: from current buffer rather than from deft search buffer.

Probably requires some more testing. Anyway, best to use this only when visiting a file in the deft directory.

The function also updates the #+title: at the top of the buffer, if any is present.

(defun zd-file-rename ()
  "Rename the current file via the deft function. Use this on files in the deft-directory."
  (interactive)
  (zd--check)
    (let ((old-filename (buffer-file-name))
          (deft-dir (file-name-as-directory deft-directory))
          new-filename old-name new-name)
      (when old-filename
        (setq old-name (deft-base-filename old-filename))
        (setq new-name (read-string
                        (concat "Rename " old-name " to (without extension): ")
                        old-name))
        (setq new-filename
              (concat deft-dir new-name "." deft-default-extension))
        (rename-file old-filename new-filename)
        (deft-update-visiting-buffers old-filename new-filename)
        (zd-update-title-in-file)
        (deft-refresh))))

To update the title of the currently visited file, the following function is used. It simply looks for the zd-title-prefix, deletes that line, and replaced it with a new title line.

A limitation of this workflow is that it will not work when the zd-title-prefix has a new line in it.

(defun zd-update-title-in-file ()
  "Update the title of the current file, if present.
Does so by looking for `zd-title-prefix'."
  (save-excursion
    (let ((zd-title-suffix ""))
      (goto-char (point-min))
      (when (re-search-forward (regexp-quote zd-title-prefix) nil t)
        (delete-region (line-beginning-position) (line-end-position))
        (zd-insert-title)))))
2.2.6.4 zd-lift-file-title returns file title from path

Returns only the file title from a file, removing path, extension, and link ID.

(defun zd-lift-file-title (zdFile)
  "Return the title of a zetteldeft note.
ZDFILE should be a full path to a note."
  (let ((baseName (file-name-base zdFile)))
    (replace-regexp-in-string
     "[0-9]\\{2,\\}-[0-9-]+[[:space:]]"
     "" baseName)))
2.2.6.5 zd-insert-title inserts file title in org-mode

Easily insert the title of the current file.

The code below gets the base of the buffer file name, takes from it the file title (i.e. strips the link id at the beginning), and inserts the remaining string.

Below the title, an additional template string is inserted automatically. This string, variable zd-title-suffix, can be customized and is empty by default.

(defun zd-insert-title ()
  "Insert filename of current zd note, stripped from its ID.
Prepended by `zd-title-prefix' and appended by `zd-title-suffix'."
  (interactive)
  (zd--check)
  (insert
    zd-title-prefix
    (zd-lift-file-title (file-name-base (buffer-file-name)))
    zd-title-suffix))

Prefix to include before the title.

(defcustom zd-title-prefix "#+TITLE: "
  "Prefix string included when `zd-insert-title' is called.
Formatted for `org-mode' by default.
Don't forget to include a space."
  :type 'string
  :group 'zetteldeft)

Customize the string to be inserted below the title. Used when generating a new file.

(defcustom zd-title-suffix ""
  "String inserted below title when `zd-insert-title' is called.
Empty by default.
Don't forget to add `\\n' at the beginning to start a new line."
  :type 'string
  :group 'zetteldeft)
2.2.6.6 zd-count-words counts total number of words

To count the total number of words, lets loop over all the files and count words in each. The total is printed in the minibuffer.

(defun zd-count-words ()
  "Prints total number of words and notes in the minibuffer."
  (interactive)
  (let ((numWords 0))
    (dolist (deftFile deft-all-files)
      (with-temp-buffer
        (insert-file-contents deftFile)
        (setq numWords (+ numWords (count-words (point-min) (point-max))))))
    (message "Your zettelkasten contains %s notes with %s words in total." (length deft-all-files) numWords)))
2.2.6.7 zd-copy-id-current-file copies id in filename

Add the ID from the current file to the kill ring.

Steps:

  1. Get the filename from the buffer
  2. Strip the ID from it.
  3. Prepend the ID with zd-link-indicator to create a full link.
  4. Result can be empty string when no id is detected in the filename.
(defun zd-copy-id-current-file ()
  "Add the id from the filename the buffer is currently visiting to the kill ring."
  (interactive)
  (zd--check)
  (let ((ID (concat zd-link-indicator
                    (zd-lift-id (file-name-base (buffer-file-name))))))
    (kill-new ID)
    (message "%s" ID)))
2.2.6.8 zd-id-to-full-title returns title from ID

Convert a zetteldeft ID into its full title.

The ID should lead to only one title, obviously, so an error is thrown when this is not the case.

(defun zd-id-to-full-title (zdID)
  "Return full title from given zetteldeft ID ZDID.
Throws an error when either none or multiple files are found."
  (let ((deft-filter-only-filenames t))
    (deft-filter zdID t))
  (unless (eq (length deft-current-files) 1)
    (user-error "ID Error. Either no or multiple zetteldeft files found with ID %s" zdID))
  (file-name-base (car deft-current-files)))

2.3 Listing all tags

zd-all-tags puts all tags in zd-tag-list and returns them.

Use zd-tag-buffer to create a buffer with all tags.

2.3.1 zd-all-tags returns them all

Extracting tags with zd-extract-tags.

(defun zd-all-tags ()
  "Return a list of all the tags found in zetteldeft files."
  (setq zd-tag-list (list))
  (dolist (deftFile deft-all-files)
    (zd-extract-tags deftFile))
  zd-tag-list)

2.3.2 zd-tag-buffer puts all tags in a buffer

The name of the buffer we’ll be using:

(setq zd-tag-buffer-name "*zd-tag-buffer*")

And some code to create that buffer.

Move to the zd-tag-buffer-name

(defun zd-tag-buffer ()
  "Switch to the *zd-tag-buffer* and list tags."
  (interactive)
  (switch-to-buffer zd-tag-buffer-name)
  (erase-buffer)
  (dolist (zdTag (zd-all-tags))
    (insert (format "%s \n" zdTag)))
  (unless (eq major-mode 'org-mode) (org-mode))
  (sort-lines nil (point-min) (point-max)))

2.3.3 Tag extracting functions

Some utility functions to achieve all of this.

2.3.3.1 zd-tag-format adjusts the tag finding regex

The regular expression used to filter out tags, zd-tag-regex works, but doesn’t filter strictly enough. Hashtags used in URLs are also found, for example.

That’s why we can make the existing regex more precise by stating that tags should be positioned either be at the beginning of a new line, or preceded by a space.

(setq zd-tag-format (concat "\\(^\\|\s\\)" zd-tag-regex))
2.3.3.2 zd-extract-tags from a file

Open a given file in a temporary buffer. Loop a search for the tag regexp. When a tag is found, remove any whitespace from it and add it to the zd-tag-list if it isn’t there already. Delete the found tag and search again.

(defun zd-extract-tags (deftFile)
  "Find all tags in DEFTFILE and add them to `zd-tag-list'."
  (with-temp-buffer
    (insert-file-contents deftFile)
    (while (re-search-forward zd-tag-format nil t)
      (let ((foundTag (replace-regexp-in-string " " "" (match-string 0))))
        ;; Add found tag to zd-tag-list if it isn't there already
        (unless (member foundTag zd-tag-list)
          (push foundTag zd-tag-list)))
      ;; Remove found tag from buffer
      (delete-region (point) (re-search-backward zd-tag-format)))))

2.4 Gathering notes

2.4.1 On “gathering notes”

Sometimes you want to easily gather all notes with a certain tag or search term. Say you want to quickly generate a list of links to all files including the tag #zetteldeft.

The following functions do that for you. There are three of them, each either taking a search term as argument or prompting for one:

  1. zd-insert-list-links inserts a simple list of notes which contain the search term, spelling out the full filename for each note (including ID).
  2. zd-org-search-include generates org-mode syntax to #+INCLUDE the files below a header with their title.
  3. zd-org-search-insert inserts the contents of all of these notes below their respective titles.

More documentation can be found below.

2.4.2 List of links

2.4.2.1 zd-insert-list-links generates list with tagged files

Creates and inserts a list with links to all files with selected search term.

The code gets a list of files that contain the search string, runs through said list and inserts a link for each entry.

When called from a note within zetteldeft, exclude the note itself from the generated list. This is necessary so that when called from an org code block within a note, the note itself is not included (since it will be found by deft, as the search string will be part of that note). To achieve this, get the full file name of the current buffer, and remove it from the search results if its found there. The when part is there so that this deletion is not attempted if the current buffer is not visiting a file.

(defun zd-insert-list-links (zdSrch)
  "Search for ZDSRCH and insert a list of zetteldeft links to all results."
  (interactive (list (read-string "search string: ")))
  (let ((zdResults (zd-get-file-list zdSrch))
        (zdThisNote (buffer-file-name)))
    (when zdThisNote (setq zdResults (delete zdThisNote zdResults)))
    (dolist (zdFile zdResults)
      (zd-list-entry-file-link zdFile))))
2.4.2.2 zd-insert-list-links-missing generates list with new links

Does the same as the above function, but only inserts IDs that aren’t already present in the current file. In contrast with zd-insert-list-links, this function can only be used from within a zetteldeft note.

This is especially handy when you want to check wheter all notes with a certain tag are linked to, or simply to list notes with a specific tag that are not linked to yet. Similar to the function above, filter out ID of the current note. In contrast to the function above, this one works with IDs rather than full paths.

Another fundamental shortcoming of this piece of code, is that after it is executed, the note now includes the previously missing ID links, which in turn means that on the next run no links will be included… The most immediate solution is for the user to be wary of this, remove any previously inserted links, save the buffer and refresh the deft cache with deft-refresh before calling this function.

(defun zd-insert-list-links-missing (zdSrch)
  "Insert a list of links to all deft files with a search string ZDSRCH.
In contrast to `zd-insert-list-links' only include links not yet present
in the current file.
Can only be called from a file in the zetteldeft directory."
  (interactive (list (read-string "search string: ")))
  (zd--check)
  (let (zdThisID zdCurrentIDs zdFoundIDs zdFinalIDs)
    (setq zdCurrentIDs (zd-extract-links (buffer-file-name)))
    ; filter IDs from search results
    (dolist (zdFile (zd-get-file-list zdSrch))
      (push (zd-lift-id zdFile) zdFoundIDs))
    ; create new list with unique ids
    (dolist (zdID zdFoundIDs)
      (unless (member zdID zdCurrentIDs)
        (push zdID zdFinalIDs)))
    ; remove the ID of the current buffer from said list
    (setq zdThisID (zd-lift-id (file-name-base (buffer-file-name))))
    (setq zdFinalIDs (delete zdThisID zdFinalIDs))
    ; finally find full title for each ID and insert it
    (if zdFinalIDs
        (dolist (zdID zdFinalIDs)
          (setq zdID (zd-id-to-full-title zdID))
          (insert " - " (concat zd-link-indicator zdID "\n")))
      ; unless the list is empty, then insert a message
      (insert (format zd-list-links-missing-message zdSrch)))))

When no missing links are found, i.e. all the notes with the provided strings are already linked to in the current note, a message is printed instead.

To be able to customize this message, include a defcustom.

(defcustom zd-list-links-missing-message
  "   No missing links with search term =%s= found\n"
  "Message to insert when no missing links are found
by `zd-insert-list-links-missing'.
%s will be replaced by the search term provided to
that function."
  :type 'string
  :group 'zetteldeft)
2.4.2.3 zd-list-entry-file-link includes a file link as list entry

Inserts for given file a link id and title as a list entry.

(defun zd-list-entry-file-link (zdFile)
  "Insert ZDFILE as list entry."
  (insert " - " (concat zd-link-indicator (file-name-base zdFile)) "\n"))

2.4.3 Compiling a single org

2.4.3.1 Idea and example
  1. Including notes with given search term

    The following explains what zd-org-search-include does, but the concept is more or less the same for zd-org-search-insert.

    For each of the notes with the provided search term, it inserts a heading, a line with #+INCLUDE and the full path to the relevant notes. This results in a single file that can be easily exported.

    The only function meant for use on the users end, is zd-org-search-include.

    For example,

    (zd-org-search-include "#export")
    

    inserts necessary code to include all files containing the tag #export. The results would look like the following:

    \* First file title
    #+INCLUDE: "/path/to/2018-07-13-2210 First file title.org"
    
    \* File two
    #+INCLUDE: "/path/to/2018-07-13-2223 File two.org"
    

    All functions are documented below.

  2. Semi-automated example

    You could, for example, add the following code to a document and execute (or evaluate) it from within org-mode. Add it under a “comment” type heading to prevent it from being exported itself, like so: * COMMENT Code.

    (let (frst)
      (save-excursion
        ;; Move to next heading
        (outline-next-heading)
        (setq frst (point))
        ;; Delete everything after
        (delete-region frst (point-max))
        ;; Include the files
        (zd-org-search-include "#tag")
        ; Sort these entries alphabetically (set mark to use a region)
    ;   (goto-char frst) (set-mark (point-max))
    ;   (org-sort-entries nil ?a)
      ))
    

    The code deletes everything after the current header and inserts all notes with #tag in them.

    In order to also sort the entries alphabetically, uncomment the last two lines.

    A final caveat: don’t put the file with the above code in you deft folder, or it will attempt to include itself (since it has #tag in it).

  3. Issues & things to note

    Before we look at the functions, a note on limitations of the current implementation.

    1. Over-enthousiastic inclusion Sometimes, a tag appears in a file without the need for it to be included. For example, a file with a list of all tags will also include the tag one wants. In the future, this might be resolved by filtering, for example with http://ergoemacs.org/emacs/elisp_filter_list.html.
    2. Inclusion from second line onwards Currently, the #+INCLUDE lines only include from the second line onwards. This is a work-around to prevent #+TITLE lines from being included (and messing up the title on org-export. To change this, edit the inserted strings in the zd-org-include-file function.
    3. Sorting The files included are unsorted, or rather: sorted as deft provides the results. Attempts at sorting by title are included in zd-get-file-list, but not working properly. As a solution, use org-sort manually after running zd-org-search-include.
2.4.3.2 zd-org-search-include generates #+INCLUDE syntax

Asks user for a search string and inserts headers and #+INCLUDE code for all files with said tag. When used on #tag, make sure to include the # manually.

(defun zd-org-search-include (zdSrch)
  "Insert `org-mode' syntax to include all files containing ZDSRCH.
Prompt for search string when called interactively."
  (interactive (list (read-string "tag (include the #): ")))
  (dolist (zdFile (zd-get-file-list zdSrch))
    (zd-org-include-file zdFile)))
2.4.3.3 zd-org-search-insert generates titles & file content

Very similar to the previous function, but rather than writing syntax to include files, insert their contents directly.

(defun zd-org-search-insert (zdSrch)
  "Insert the contents of all files containing ZDSRCH.
Files are separated by `org-mode' headers with corresponding titles.
Prompt for search string when called interactively."
  (interactive (list (read-string "Search term: ")))
  (dolist (zdFile (zd-get-file-list zdSrch))
    (zd-org-insert-file zdFile)))

2.4.4 Helper functions

2.4.4.1 zd-insert-file-contents returns the contents of a file

Returns the contents of a file.

(defun zd-file-contents (zdFile &optional removeLines)
  "Insert file contents of a zetteldeft note.
ZDFILE should be a full path to a note.

Optional: leave out first REMOVELINES lines."
  (with-temp-buffer
    (insert-file-contents zdFile)
    (when removeLines
      (kill-whole-line removeLines))
    (buffer-string)))
2.4.4.2 zd-org-include-file includes a file in org format

Inserts the title as a new header, with the #+INCLUDE line below. Includes only from the second line onward, so that any #+TITLE lines are omitted.

(defun zd-org-include-file (zdFile)
  "Insert code to include org file ZDFILE."
  (insert
    ;; Insert org-mode title
    "* " (zd-lift-file-title zdFile) "\n"
    ;; Insert #+INCLUDE: "file.org" :lines 2-
    "#+INCLUDE: \"" zdFile "\" :lines \"2-\"\n\n"))
2.4.4.3 zd-org-insert-file inserts a files content

For a file, insert its title and contents (without first 3 lines).

Even better would be: without any of the lines starting with # at the beginning of the file.

(defun zd-org-insert-file (zdFile)
  "Insert title and contents of ZDFILE."
  (insert
    ;; Insert org-mode title
    "\n* " (zd-lift-file-title zdFile) "\n\n"
    ;; Insert file contents (without the first 3 lines)
    (zd-file-contents zdFile 3)))

2.5 Creating visuals

2.5.1 Introducing graphs

Linking notes together in plain text is fun, but sometimes you want to visualize which notes are connected.

The following functions attempt to provide said functionallity, but are in a very early stage of development. They generate an org source block for graphviz, which can then be executed to generate a pdf.

A brief introduction:

  • zd-org-graph-search creates a graph with all the notes containing a provided string.
  • zd-org-graph-note creates a graph that starts at a note, connects all notes linked to it, and all notes linked to those. In other words, it looks two levels deep.

The resulting graph looks something like this:

zd-graph.jpg

It’s worth noting, again, that this is very provisional.

2.5.2 Graph functions

2.5.2.1 zd-org-graph-search creates graph from search string

An org code block with graphviz code for a graph.pdf.

Find all notes with the provided search term. Loop over this list, and insert title and links for each one.

(defun zd-org-graph-search (str)
  "Insert org source block for graph with zd search results.
STR should be the search the resulting notes of which should be included in the graph."
  (interactive (list (read-string "search string: ")))
  (setq zd-graph--links (list))
  (let ((zdList (zd-get-file-list str)))
    (insert zd-graph-syntax-begin)
    (insert "\n  // links\n")
    (dolist (oneFile zdList)
      (insert "\n")
      (zd-graph-insert-links oneFile))
    (zd-graph-insert-all-titles))
  (insert zd-graph-syntax-end))
2.5.2.2 zd-org-graph-note creates graph from note

Insert an org source code block for a graphviz presentation of a note and its connections.

When links are added, they are also stored in zd-graph--links which is later used to insert titles.

When called interactively, select a file from the completion interface.

(defun zd-org-graph-note (deftFile)
  "Create a graph starting from note DEFTFILE."
  (interactive (list
    (completing-read "Note to start graph from: "
      (deft-find-all-files))))
  (setq zd-graph--links (list))
  (insert zd-graph-syntax-begin)
  (insert "\n  // base note and links \n")
  (zd-graph-insert-links deftFile)
  (zd-graph-insert-additional-links)
  (zd-graph-insert-all-titles)
  (insert zd-graph-syntax-end))

2.5.3 Building blocks

2.5.3.1 zd-graph-syntax-begin provides opening syntax

Within graphviz, I advise to use fdp, twopi (which overlaps more) or circo as layouts.

(defcustom zd-graph-syntax-begin
  "#+BEGIN_SRC dot :file ./graph.pdf :cmdline -Kfdp -Tpdf
  \n graph {\n"
  "Syntax to be included at the start of the zetteldeft graph.")
2.5.3.2 zd-graph-syntax-end provides closing syntax
(defcustom zd-graph-syntax-end
  "} \n#+END_SRC\n"
  "Syntax to be included at the end of the zetteldeft graph.")
2.5.3.3 zd-extract-links pulls links from a file

Very similar to the zd-extract-tags function, but returns links instead of storing them.

(defun zd-extract-links (deftFile)
  "Find all links in DEFTFILE and return a list."
  (let ((zdLinks (list)))
    (with-temp-buffer
      (insert-file-contents deftFile)
      (while (re-search-forward zd-id-regex nil t)
        (let ((foundTag (replace-regexp-in-string " " "" (match-string 0))))
          ;; Add found tag to zdLinks if it isn't there already
          (unless (member foundTag zdLinks)
            (push foundTag zdLinks)))
        ;; Remove found tag from buffer
        (delete-region (point) (re-search-backward zd-id-regex))))
   zdLinks))
2.5.3.4 zd-graph-insert-links inserts graphviz links

Insert the sanitized ID from the file, followed by an arrow and all of the links.

Store both the deft file provided and any found files in zd-graph--links.

(defun zd-graph-insert-links (deftFile)
  "Insert links in DEFTFILE in dot graph syntax on a single line.
Any inserted ID is also stored in `zd-graph--links'."
  (insert "  \""
          (zd-lift-id deftFile)
          "\" -- {")
  (dolist (oneLink (zd-extract-links deftFile))
    (zd-graph-store-link oneLink t)
    (insert "\"" oneLink "\" "))
  (insert "}\n")
  (zd-graph-store-link deftFile))
2.5.3.5 zd-graph-insert-title inserts graphviz title line

Titles have to be inserted in the correct graphviz format, like so:

B [label = "Node B"]

The following function should achieve that.

(defun zd-graph-insert-title (deftFile)
  "Insert the DEFTFILE title definition in a one line dot graph format."
  (let ((zdTitle (replace-regexp-in-string "\"" "" (zd-lift-file-title deftFile)))
        (zdId    (zd-lift-id deftFile)))
    (insert "  \"" zdId "\""
            " [label = \"" zdTitle " (" zd-link-indicator zdId ")\"")
    (insert "]" "\n"))
  (zd-graph-store-link deftFile))

The title is taken from the file string and any additional quotes removed.

2.5.3.6 zd-graph-store-link stores provided notes

For future reference, linked files are stored in zd-graph--links. This function facilitates that process.

Provide a link to a file to store it. Simply providing an ID works too, if you provide the second argument as true.

(defun zd-graph-store-link (deftFile &optional idToFile)
  "Push DEFTFILE to zd-graph--links unless it's already there.
When IDTOFILE is non-nil, DEFTFILE is considered an id
and the the function first looks for the corresponding file."
  (when idToFile
    (let ((deft-filter-only-filenames t))
      (progn
        (deft-filter deftFile t)
        (setq deftFile (car deft-current-files)))))
  (unless (member deftFile zd-graph--links)
    (push deftFile zd-graph--links)))
2.5.3.7 zd-graph-insert-additional-links inserts stored links

Insert links stored in the zd-graph--links list. Except the first list item, as this is considered the base file already included.

(defun zd-graph-insert-additional-links ()
  "Insert rest of `zd-graph--links'."
  (setq zd-graph--links (cdr zd-graph--links))
  (dolist (oneFile zd-graph--links)
    (zd-graph-insert-links oneFile)))
2.5.3.8 zd-graph-insert-all-titles inserts all stored titles

Insert all titles stored in zd-graph--links.

(defun zd-graph-insert-all-titles ()
  "Insert all graphviz title lines for all links stored in `zd-graph--links'."
  (insert "\n  // titles \n")
  (dolist (oneLink zd-graph--links)
    ;; Sometimes, a 'nil' list item is present. Ignore those.
    (when oneLink
      (zd-graph-insert-title oneLink))))

2.6 Aesthetics

2.6.1 Highlighting zetteldeft links

To highlight zetteldeft links, let’s add a font-lock keyword to org-mode.

The regex used for highlighting is the zd-id-regex prepended by zd-link-indicator.

(font-lock-add-keywords 'org-mode
  `((,(concat zd-link-indicator zd-id-regex) . font-lock-warning-face)))

For some reason, highlighting is not working in org comments.

2.7 End of package

That’s all folks!

(provide 'zetteldeft)
;;; zetteldeft.el ends here

3 Suggested setup

The following assumes deft is loaded manually in your dotfile, it merely configures the package.

None of these code blocks are tangled into the .el file, they are here merely as a guide.

3.1 Suggested zeteldeft keybindings

3.1.1 Keybindings with general

To call zetteldeft functions from anywhere behind a leader key such as SPC, I recommend general.el with the following setup.

(general-define-key
  :prefix "SPC"
  :non-normal-prefix "C-SPC"
  :states '(normal visual motion emacs)
  :keymaps 'override
  "d"  '(nil :wk "deft")
  "dd" '(deft :wk "deft")
  "dD" '(zd-deft-new-search :wk "new search")
  "dR" '(deft-refresh :wk "refresh")
  "ds" '(zd-search-at-point :wk "search at point")
  "dc" '(zd-search-current-id :wk "search current id")
  "df" '(zd-follow-link :wk "follow link")
  "dF" '(zd-avy-file-search-ace-window :wk "avy file other window")
  "dl" '(zd-avy-link-search :wk "avy link search")
  "dt" '(zd-avy-tag-search :wk "avy tag search")
  "dT" '(zd-tag-buffer :wk "tag list")
  "di" '(zd-find-file-id-insert :wk "insert id")
  "dI" '(zd-find-file-full-title-insert :wk "insert full title")
  "do" '(zd-find-file :wk "find file")
  "dn" '(zd-new-file :wk "new file")
  "dN" '(zd-new-file-and-link :wk "new file & link")
  "dr" '(zd-file-rename :wk "rename"))

3.1.2 Spacemacs keybindings

Suggested keybindings for spacemacs.

These can be called from anywhere behind SPC d.

;; Prefix
(spacemacs/declare-prefix "d" "deft")
;; Launch deft
(spacemacs/set-leader-keys "dd" 'deft)
(spacemacs/set-leader-keys "dD" 'zd-deft-new-search)
;; SEARCH
 ; Search thing at point
   (spacemacs/set-leader-keys "ds" 'zd-search-at-point)
 ; Search current file id
   (spacemacs/set-leader-keys "dc" 'zd-search-current-id)
 ; Jump & search with avy 
 ;  search link as filename
    (spacemacs/set-leader-keys "df" 'zd-follow-link)
    (spacemacs/set-leader-keys "dF" 'zd-avy-file-search-ace-window)
 ;  search link as contents
    (spacemacs/set-leader-keys "dl" 'zd-avy-link-search)
 ;  search tag as contents
    (spacemacs/set-leader-keys "dt" 'zd-avy-tag-search)
 ;  find all tags
    (spacemacs/set-leader-keys "dT" 'zd-tag-buffer)
;; LINKS
 ; Insert link from filename
   (spacemacs/set-leader-keys "di" 'zd-find-file-id-insert)
 ; Insert link with full filename
   (spacemacs/set-leader-keys "dI" 'zd-find-file-full-title-insert)
;; FILES
 ; Open file
   (spacemacs/set-leader-keys "do" 'zd-find-file)
 ; Create new file
   (spacemacs/set-leader-keys "dn" 'zd-new-file)
   (spacemacs/set-leader-keys "dN" 'zd-new-file-and-link)
 ; Rename file
   (spacemacs/set-leader-keys "dr" 'zd-file-rename)
;; UTILITIES
(spacemacs/set-leader-keys "dR" 'deft-refresh)

3.2 Suggested deft setup

3.2.1 Notes & extensions

Note extensions are md, txt and org. First of this list is the default for new notes.

(setq deft-extensions '("org" "md" "txt"))

3.2.2 Set deft-directory

Search the deft directory recursively, to include subdirectories.

(setq deft-directory (concat org-directory "/notes/zetteldeft"))
(setq deft-recursive t)

3.2.3 Additional deft functions

Some personal additions. Note that these are functional suggestions, and not included in the zetteldeft package.

A small function to open a file in the other window and shifting focus to it. That final part is what the t argument does.

(defun efls/deft-open-other ()
 (interactive)
 (deft-open-file-other-window t))

Let’s add another function, to simply preview in the other window, i.e. not switch focus to it.

(defun efls/deft-open-preview ()
 (interactive)
 (deft-open-file-other-window))

To select results from the item list without leaving the insert state, I add the following keys.

(with-eval-after-load 'deft
  (define-key deft-mode-map
    (kbd "<tab>") 'efls/deft-open-preview)
  (define-key deft-mode-map
    (kbd "<s-return>") 'efls/deft-open-other)
  (define-key deft-mode-map
    (kbd "s-j") 'evil-next-line)
  (define-key deft-mode-map (kbd "s-k") 'evil-previous-line))

3.2.4 Ignore more org-mode metadata

I tend to write org-mode titles with #+title: (i.e., uncapitalized). Also other org-mode code at the beginning is written in lower case.

In order to filter these from the deft summary, let’s alter the regular expression:

(setq deft-strip-summary-regexp
 (concat "\\("
         "[\n\t]" ;; blank
         "\\|^#\\+[a-zA-Z_]+:.*$" ;;org-mode metadata
         "\\)"))

Its original value was \\([\n ]\\|^#\\+[[:upper:]_]+:.*$\\).

4 Brainstorm

Some ideas for the future.

  • Export HTML with clickeable links between notes.
  • Export markdown through pandoc.
  • Create a minor mode to higlight links & tags and display clickeable links.
  • Saved searches, somewhere easily accessible.

Footnotes:

1

For those not yet familiar: deft is a note manager within emacs, for easily searching and retrieving plain text notes. It is inspired by the popular Notational Velocity. Check out jblevins.org/projects/deft/ and notational.net.

Author: EFLS

Modified: 2019-05-30 Thu 19:49

Emacs 26.1 (Org mode 9.2.3)