Zetteldeft

This document contains all the source code to a set of functions for emacs which aim to extend the deft package and turn it into a (very very) basic Zettelkasten note-taking system.

A concice presentation about the package can be found in Introducing Zetteldeft.

Check out the Github repository to get the source. Read on for an introduction and some documentation.

Latest additions:

Older changes are mentioned in the changelog below.

1 What?

1.1 A Zettelkasten system for emacs based on deft

This is my feeble attempt at recreating a Zettelkasten environment by extending the excellent deft package in emacs.1 I call it zetteldeft.

It is inspired by the The Archive app. For this and more on the Zettelkasten way of taking notes, see zettelkasten.de. They have a forum for discussion on both software and the specifics of the Zettelkasten philosophy.

The code that follows is created and maintained for my personal use, shared here in hope that it can benefit others as well. I’d be happy to learn how you use it and expand upon it.

It is very much WIP and I’m fairly new to elisp, so it might contain some stupid code.

Anyway, here we go.

1.2 Installing zetteldeft

1.2.1 Installing with MELPA and use-package

If you use MELPA & use-package, installing is easy. Simply add 

(use-package zetteldeft
  :after deft
  :config
    (zetteldeft-set-classic-keybindings))

to your init.el.

The zetteldeft-set-classic-keybindings sets up some defaults as per section 3.1. If you prefer evil style bindings, leave out this function call and take a look at 3.1.2.

For the best experience, configure deft as suggested in section 3.2.

An installation of avy is required, as it is used for some functions that make following links easier.

1.2.2 spacemacs installation

Installing and enabling zetteldeft with the default configuration is straightforward with spacemacs:

  1. Add the deft layer to your .spacemacs file and enable zetteldeft support. Locate dotspacemacs-configuration-layers in your .spacemacs and add the variables as suggested below. This should install both deft and zetteldeft and load them in correct order, and set default keybindings as shared in section 3.1.3. 

    (setq-default dotspacemacs-configuration-layers
  '((deft :variables deft-zetteldeft t)))
  2. You can further configure deft according to section 3.2. Do this the spacemacs way, i.e. in dotspacemacs/user-config. Simply add (setq ...) there.

1.2.3 Installing directly from source

This package requires:

  • deft, obviously
  • avy to jump & search
  • ace-window to follow a link in a window of choice

From the Github repository, either

  • download the zetteldeft.el file,
  • download the org-file and org-tangle it yourself. It should contain everything.

Whichever way you go, load up the package by adding the package to your load path and requiring: 

(add-to-list 'load-path "~/path/to/folder/")
(require 'zetteldeft)

and you’re good to go!

Well, not quite. First you must read on about the basics of zetteldeft. You’ll also want some keybindings. Check out the suggested setup below.

1.3 Basic concepts

Notes reside in the deft-directory. Notes are written in org-mode syntax (although most functions should work in markdown as well).

The filename of a note starts with a unique id based on the time and the date, for example: 2018-07-09-2115 This is a note.org.

This unique id can be used to link notes together. A link consists of the § character followed by the id. For example: §2018-07-09-2115 should link to the file above. A link can appear anywhere in the text. See below for advanced information about IDs and links.

When searching deft with the id as a filter, you’ll find both the original note (with the id in its name) and all the notes that link to this note (with the id in its body). Do so with zetteldeft-search-current-id and zetteldeft-avy-link-search respectively

Notes can contain tags in plain text: words prepended with a #. This is a tag: #tag. Tags make it easy to retrieve notes. They can appear anywhere in the note, but I’d suggest putting them somewhere at the top.

1.4 Basic actions

Create a note with zetteldeft-new-file and provide a name.

To insert links to other notes, either

  • enter their links manually,
  • use zetteldeft-find-file-id-insert and select a file from the list,

With zetteldeft-find-file-full-title-insert, you guessed it, the note’s title is included as well.

To easily branch out from the current note (i.e. create a new one and link to it in one go), use zetteldeft-new-file-and-link.

To search for a tag or anything else under cursor, use zetteldeft-search-at-point. Combined with the power of avy to jump to any character on screen, use these to jump and search in one go: zetteldeft-avy-link-search and zetteldeft-avy-tag-search.

To open the note behind a link, use zetteldeft-follow-link.

Want more functionality? How about showing a list of tags or gathering notes with a certain search string? Or maybe a graph visualizing how notes are linked?

Still hungry? I’m welcoming both contributions and suggestions. Feel free to submit comments or pull requests on Github.

1.5 An overview

While there are many, these should be enough to get you started. Here is an overview.

Note that the package itself does not define any keybindings. You’ll need to set up those yourself, but defaults are suggested at the end of this document. Keybindings in the overview below are preceded by C-c d or SPC d, depending on the setup.

Function Description Keybinding
zetteldeft-new-file Create new note and open d n
zetteldeft-new-file-and-link Create new note and insert link d N
zetteldeft-find-file-id-insert Pick a note and insert a link d i
zetteldeft-follow-link Follow a link d f
zetteldeft-avy-link-search Select and search a link’s ID d l
zetteldeft-avy-tag-search Select a tag and search for it d t
zetteldeft-search-at-point Search for thing at point d s
zetteldeft-search-current-id Search for id of current file d c

Read on, dear reader, for all of this and much more.

2 The zetteldeft package

2.1 Package preparation

The required preamble and some other initial settings. To know how this package works, please skip right past this to the next section.

2.1.1 Preamble

Some declaration. 

;;; zetteldeft.el --- Turn deft into a zettelkasten system -*- lexical-binding: t -*-

;; Copyright (C) 2018-2020  EFLS

;; Author: EFLS <Elias Storms>
;; URL: https://efls.github.io/zetteldeft/
;; Keywords: deft zettelkasten zetteldeft wp files
;; Version: 0.3
;; Package-Requires: ((emacs "25.1") (deft "0.8") (ace-window "0.7.0"))

;; This file is not part of Emacs

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;; Zetteldeft is an extension of the deft package for Emacs.
;; It generates unique IDs to create stable links between notes, which
;; allows the user to make an interconnected system of notes.
;; Zetteldeft uses deft to find and follow links to notes.
;; For more information, see zetteldeft.org
;; or https://efls.github.io/zetteldeft

;; Note: this file is tangled from zetteldeft.org.
;; The .org contains documentation and notes on usage of the package.

;;; Code:

2.1.2 Requirements

deft is required, obviously, and avy is needed for some utility functions. thingatpt is needed to easily search & jump. ace-window is used to follow links in another window. 

(require 'deft)

(unless (require 'avy nil 'no-error)
  (user-error "Avy not installed, required for zetteldeft-avy-* functions"))

(require 'thingatpt)

(require 'ace-window)

Since February 2019, the avy API changed and avy--generic-jump is replaced by avy-jump. Unfortunately, this change doesn’t seem to be indicated with a specific version number. So let’s check whether that function is available, and show a message when it’s not. 

(declare-function avy-jump "avy")
(unless (fboundp 'avy-jump)
  (display-warning 'zetteldeft
    "Function `avy-jump' not available. Please update `avy'"))

2.1.3 Customization

For easy but minor customization options. 

(defgroup zetteldeft nil
  "A zettelkasten on top of deft."
  :group 'deft
  :link '(url-link "https://efls.github.io/zetteldeft"))

2.2 Basic zetteldeft functions

In this section:

2.2.1 Search functions

2.2.1.1 zetteldeft-search-at-point thing at point

Search the thing at point.

Based on snippet suggested by saf-dmitry on deft’s Github. 

;;;###autoload
(defun zetteldeft-search-at-point ()
  "Search via `deft' with `thing-at-point' as filter.
Thing can be a double-bracketed link, a hashtag, or a word."
  (interactive)
  (let ((string (zetteldeft--get-thing-at-point)))
   (if string
       (zetteldeft--search-global string t)
     (user-error "No search term at point"))))
2.2.1.2 zetteldeft-search-current-id searches current id

Deft search on the id of the current file.

This function is useful to easily see which notes link to the current file.

Result is not opened automatically.

Steps:

  1. Get the filename from the current buffer.
  2. Lift the ID from it.
  3. Search with resulting string.
;;;###autoload
(defun zetteldeft-search-current-id ()
  "Search deft with the id of the current file as filter.
Open if there is only one result."
  (interactive)
  (zetteldeft--search-global
    (zetteldeft--current-id) t))
2.2.1.3 zetteldeft--get-thing-at-point returns string

Returns the thing at point as string.

Tries to get, in the following order:

  • links between [[
  • hashtags according to zetteldeft-tag-regex
  • links according to the zetteldeft-link-indicator and zetteldeft-id-regex
  • words

Based on snippet suggested by saf-dmitry on deft’s Github. 

(defun zetteldeft--get-thing-at-point ()
  "Return the thing at point.
This can be
 - a link: a string between [[ brackets ]],
 - a tag matching `zetteldeft-tag-regex',
 - a link matching `zetteldeft-link-indicator',
    `zettteldeft-id-regex' and `zetteldeft-link-suffix',
 - or a word."
 (let* ((link-brackets-re "\\[\\[\\([^]]+\\)\\]\\]")
        (link-id-re (concat zetteldeft-link-indicator
                            zetteldeft-id-regex
                            zetteldeft-link-suffix))
        (htag-re zetteldeft-tag-regex))
   (cond
    ((thing-at-point-looking-at link-brackets-re)
      (match-string-no-properties 1))
    ((thing-at-point-looking-at link-id-re)
      (match-string-no-properties 0))
    ((thing-at-point-looking-at htag-re)
      (match-string-no-properties 0))
    (t (thing-at-point 'word t)))))
2.2.1.4 zetteldeft--search-global for string

Search with deft for given string. If there is only one result, that file is opened, unless additional argument is true.

Based on snippet suggested by saf-dmitry on deft’s Github. 

(defun zetteldeft--search-global (str &optional dntOpn)
  "Search deft with STR as filter.
If there is only one result, open that file (unless DNTOPN is true)."
  ;; Sanitize the filter string
  (setq str (replace-regexp-in-string "[[:space:]\n]+" " " str))
  ;; Call deft search on the filter string
  (let ((deft-incremental-search t))
   (deft)
   (deft-filter str t))
  ;; If there is a single match, open the file
  (unless dntOpn
   (when (eq (length deft-current-files) 1)
     (deft-open-file (car deft-current-files)))))
2.2.1.5 zetteldeft--search-filename for string

Deft search on filename. If there is only one result, open that file.

Incremental search is turned off, and the filter is set to filenames only. 

(defun zetteldeft--search-filename (thisStr &optional otherWindow)
  "Search for deft files with string THISSTR in filename.
Open if there is only one result (in another window if OTHERWINDOW is non-nil)."
  ;; Sanitize the filter string
  (setq thisStr (replace-regexp-in-string "[[:space:]\n]+" " " thisStr))
  ;; Call deft search on the filter string
  (let ((deft-filter-only-filenames t))
   (deft-filter thisStr t))
  ;; If there is a single match, open the file
  (when (eq (length deft-current-files) 1)
    (deft-open-file (car deft-current-files) otherWindow)))
2.2.1.6 zetteldeft--get-file-list returns file list with search term

Get a list of the files with given search string.

To fix: sorting of results.

The code searches for the given string and returns deft-current-files. 

(defun zetteldeft--get-file-list (srch)
  "Return a list of files with the search item SRCH."
  (let ((deft-current-sort-method 'title))
    (deft-filter srch t)
    deft-current-files))

2.2.2 IDs, links and tags

In zetteldeft, the concepts of ‘links’ and ‘IDs’ have related meanings, but in the documentation they are not synonyms.

An ID refers to the unique, generated string included in the filename, identifying each note. By default, IDs are numeric and time-based. For example: 2019-01-20-1433. These are generated by zetteldeft-generate-id. (To customize how IDs are generated, take a look at the variable zetteldeft-custom-id-function.)

A link is an ID prepended by a character to easily identify it as a link. For example: §2019-01-20-1433.

This identifying character can be changed by setting the zetteldeft-link-indicator variable and is § by default.

Set it to an empty string (i.e. "") to disable the indicator. See also zetteldeft-link-suffix (which is empty by default) if you prefer a closure to the links.

2.2.2.2 Helper function zetteldeft--id-font-lock-setup

Before the customization, let’s declare a helper function that adds a font-lock keyword for org-mode specifically. The function below – courtesy of bymoz089 – is called when zetteldeft-link-indicator or zetteldeft-id-regex are customized.

The function

  1. Removes existing keywords from the font-lock setup
  2. Sets the variable and value
  3. Adds them as font-lock keywords

To highlight links, zetteldeft-id-regex prepended by zetteldeft-link-indicator. 

(defun zetteldeft--id-font-lock-setup (var val)
  "Add font-lock highlighting for zetteldeft links.
Called when `zetteldeft-link-indicator' or
`zetteldeft-id-regex' are customized."
  (when (and (boundp 'zetteldeft-link-indicator)
             (boundp 'zetteldeft-id-regex)
             (boundp 'zetteldeft-link-suffix))
     (font-lock-remove-keywords 'org-mode
        `((,(concat zetteldeft-link-indicator
                    zetteldeft-id-regex
                    zetteldeft-link-suffix)
           . font-lock-warning-face))))
  (set-default var val)
  (when (and (boundp 'zetteldeft-id-regex)
             (boundp 'zetteldeft-link-indicator)
             (boundp 'zetteldeft-link-suffix))
     (font-lock-add-keywords 'org-mode
        `((,(concat zetteldeft-link-indicator
                    zetteldeft-id-regex
                    zetteldeft-link-suffix)
           . font-lock-warning-face)))))

Note that highlighting is not working in org comments.

2.2.2.3 zetteldeft-id-format for generating time-based ID strings

Format used to generate time-based IDs.

See documentation of format-time-string for more info on possible placeholders.

If you customize this value, make sure to edit the zetteldeft-id-regex as well, so that the IDs can be found by other functions. 

(defcustom zetteldeft-id-format "%Y-%m-%d-%H%M"
  "Format used when generating time-based zetteldeft IDs.

Be warned: the regexp to find IDs is set separately.
If you change this value, set `zetteldeft-id-regex' so that
the IDs can be found.

Check the documentation of the `format-time-string'
function to see which placeholders can be used."
  :type 'string
  :group 'zetteldeft)

Another popular option would be to set this value to "%Y%m%d%H%M", so that a similar ID is generated without any dashes. See below for a corresponding regular expression.

Note that the default setup means that IDs are not unique when you make multiple notes per minute. If you plan to do so, extend the zetteldeft-id-format and add %S to include seconds. These IDs should still be recognized by the default zetteldeft-id-regex below.

While we’re at it, lets tell deft to use this format when creating new files. For good measure: I advise creating new notes in the zetteldeft system with zetteldeft-new-file or zetteldeft-new-file-and-link as defined below, rather than through deft itself. 

(setq deft-new-file-format zetteldeft-id-format)
2.2.2.4 zetteldeft-generate-id generates an ID stirng

Next up, a function to generate an ID string. By default, the above time-based format will be used, unless a custom ID generator function is specified: to override the way an ID is generated, point zetteldeft-custom-id-function to a custom function.

The above function performs a rudimentory check to ensure that the generated ID is indeed available (i.e., not yet used in a filename). The current response to such a duplicate is to throw an error. With the default, time-based IDs, this happens when notes are created in quick succession. 

(defun zetteldeft-generate-id (title &optional filename)
  "Generate and return a Zetteldeft ID.
The ID is created using `zetteldeft-id-format', unless
`zetteldeft-custom-id-function' is bound to a function, in which case
that function is used and TITLE and FILENAME are passed to it."
  (let ((id
          (if-let ((f zetteldeft-custom-id-function))
              (funcall f title filename)
            (format-time-string zetteldeft-id-format))))
    (if (zetteldeft--id-available-p id)
        id
      (error "Generated ID %s is not unique." id))))

The check whether an ID is available is performed by a separate function. 

(defun zetteldeft--id-available-p (str)
  "Return t only if provided string STR is unique among Zetteldeft filenames."
  (let ((deft-filter-only-filenames t))
    (deft-filter str t))
  (eq 0 (length deft-current-files)))
2.2.2.5 zetteldeft-custom-id-function for fine-grained control over ID formatting

For full control over the output of zetteldeft-generate-id, you can set the variable zetteldeft-custom-id-function to a function of your choice. When left nil, default time-based IDs are generated. 

(defcustom zetteldeft-custom-id-function nil
  "User-defined function to generate an ID.
The specified function must accept arguments for note `TITLE'
and &optional `FILENAME'. The returned ID must be a string."
  :type 'function
  :group 'zetteldeft)

The function indicated by zetteldeft-custom-id-function must accept arguments for title and optional filename. The second argument will be passed only when a new ID is being generated for an existing note that does not already have an ID (for example, one created outside of zetteldeft). The function may use or ignore its argument values as desired.

Let’s look at an example. Below is a custom ID function that generates an ID based on the current time for new notes, and based on the file’s last-updated time for existing notes not yet having an ID: 

;; Hypothetical custom function defined in init.el
(defun my-time-based-id-example (title &optional filename)
  "Generate an ID using a note's last-updated time if available"
  (if filename
      (let ((last-updated (file-attribute-modification-time
                           (file-attributes filename))))
        (format-time-string zetteldeft-id-format last-updated))
    (format-time-string zetteldeft-id-format)))

;; Tell zetteldeft to use the custom ID function
(setq zetteldeft-custom-id-function #'my-time-based-id-example)

Custom ID functions can produce IDs in any format, not just time-based ones. The following (non-working) example code illustrates this: 

;; Hypothetical custom function defined in init.el
(defun my-arbitrary-id-example (title &optional filename)
  "Generate an ID exactly the way I like it"
  (if filename
      (cryptographic-hash filename)
    (random-anagram title)))

;; Tell zetteldeft to use the custom ID function
(setq zetteldeft-custom-id-function #'my-arbitrary-id-example)

If you set zetteldeft-custom-id-function, be sure that the values your function generates are matched by zetteldeft-id-regex, so that the IDs can be found by Zetteldeft.

2.2.2.6 zetteldeft-id-regex for finding IDs

The regular expression used to search for zetteldeft IDs as set in zetteldeft-id-format.

The default regex dictates that a zetteldeft ID should consist of:

  1. a series of exactly 4 numbers
  2. followed by exactly 3 sets of a dash and two or more numbers

When customized, the function zetteldeft--id-font-lock-setup is so that the zetteldeft id is fontified correctly. 

(defcustom zetteldeft-id-regex "[0-9]\\{4\\}\\(-[0-9]\\{2,\\}\\)\\{3\\}"
  "The regular expression used to search for zetteldeft IDs.
Set it so that it matches strings generated with
`zetteldeft-id-format'."
  :type 'string
  :group 'zetteldeft
  :set 'zetteldeft--id-font-lock-setup)

If you use a "%Y%m%d%H%M" format for note naming, you might want to set the regular expression to "20[0-9]\\{10\\}" so that it matches any string starting with 20 followed by 10 other digits.

2.2.2.7 zetteldeft-link-indicator prepends ID links

To make it easier to distinguish links to zetteldeft notes, the ID can be prepended with a symbol. By default, this is set to §, but it can be changed (or nil). 

(defcustom zetteldeft-link-indicator "§"
  "String to indicate zetteldeft links.
String prepended to IDs to easily identify them as links to zetteldeft notes.
This variable should be a string containing only one character."
  :type 'string
  :group 'zetteldeft
  :set 'zetteldeft--id-font-lock-setup)
2.2.2.8 zetteldeft-link-suffix is appended to ID links

For further customizability, let’s also introduce a suffix. This will be appended to links.

By default, this is set to an empty string. 

(defcustom zetteldeft-link-suffix ""
  "String to append to zetteldeft links.
To disable, set to empty string rather than to nil."
  :type 'string
  :group 'zetteldeft
  :set 'zetteldeft--id-font-lock-setup)

Users who prefer Markdown notes and interoperability with The Archive, might want to set this to ]], and the zetteldeft-link-indicator to [[.

2.2.2.9 zetteldeft-tag-regex for finding tags

This regular expression indicates what tags can look like.

By default, tags start with a # or @ and contain least one or more lower case letters. Dashes are allowed. 

(defcustom zetteldeft-tag-regex "[#@][[:alnum:]_-]+"
  "Regular expression for zetteldeft tags."
  :type 'string
  :group 'zetteldeft)

Note that this regular may trip over # symbols used in URLS.

2.2.2.10 zetteldeft--lift-id filters the ID from a string

Return the zetteldeft ID from any string.

Searches with a temporary buffer, from the end of the string backwards (hence the -1 argument), which implies that the last zetteldeft string is returned. 

(defun zetteldeft--lift-id (str)
  "Extract zetteldeft ID from STR.
This is done with the regular expression stored in
`zetteldeft-id-regex'."
  (with-temp-buffer
    (insert str)
    (when (re-search-forward zetteldeft-id-regex nil t -1)
      (match-string 0))))

Or are there better ways than working with-temp-buffer?

Here is a little test. 

(zetteldeft--lift-id "2018-11-09-1934-12 Some text (1989) - testing (2000 p. 12-25)")

2018-11-09-1934

2.2.3 Finding & linking files from minibuffer

2.2.3.1 zetteldeft-find-file opens file from minibuffer

Select file from the deft folder from the minibuffer.

Based on deft-find-file. 

;;;###autoload
(defun zetteldeft-find-file (file)
  "Open deft file FILE."
  (interactive
    (list (completing-read "Deft find file: "
            (deft-find-all-files-no-prefix))))
  (deft-find-file file))
2.2.3.2 zetteldeft-go-home to open a base note

A Zettelkasten system has no fixed single hierarchy, but it is often convenient to maintain a base note, or a home note. Such a note can link to other notes that contain links related to a specific subtopic.

Opening up the home note should be quick and easy.

Let’s first store the availability to store an ID. 

(defvar zetteldeft-home-id nil
  "String with ID of home note, used by `zetteldeft-go-home'.")

A function to open up this note is trivial. 

(defun zetteldeft-go-home ()
  "Move to a designated home note.
Set `zetteldeft-home-id' to an ID string of your home note."
  (interactive)
  (if (stringp zetteldeft-home-id)
      (zetteldeft-find-file
        (zetteldeft--id-to-full-path zetteldeft-home-id))
    (message "No home set. Provide a string to zetteldeft-home-id.")))
2.2.3.3 zetteldeft-find-file-id-insert inserts file id

Select file from minibuffer and insert its link, prepended by § (or zetteldeft-link-indicator to be precise).

Based on deft-find-file. 

;;;###autoload
(defun zetteldeft-find-file-id-insert (file)
  "Find deft file FILE and insert a link."
  (interactive (list
    (completing-read "File to insert id from: "
      (deft-find-all-files-no-prefix))))
  (insert (concat zetteldeft-link-indicator
                  (zetteldeft--lift-id file)
                  zetteldeft-link-suffix)))
2.2.3.4 zetteldeft-backlink-add adds a backlink to note

Adding a backlink to another note is a good way to create structure.

For such a backlink, we need a prefix. 

(defcustom zetteldeft-backlink-prefix "# Backlink: "
  "Prefix string included before a back link.
Formatted as `org-mode' comment by default."
  :type 'string
  :group 'zetteldeft)

Include the prefix on a new line after the title, and add ID and title. 

;;;###autoload
(defun zetteldeft-backlink-add (file)
  "Find deft file FILE and insert a backlink to it.
Finds the title line, and adds `backlink-prefix' with
ID and title on a new line."
  (interactive (list
    (completing-read "File to add backlink to: "
      (deft-find-all-files-no-prefix))))
  (save-excursion
    (goto-char (point-min))
    (when (re-search-forward
            (regexp-quote zetteldeft-title-prefix) nil t))
    (forward-line)
    (insert zetteldeft-backlink-prefix
            (concat zetteldeft-link-indicator
                    (zetteldeft--lift-id file)
                    zetteldeft-link-suffix
                    " "
                    (zetteldeft--lift-file-title
                      (concat deft-directory file)))
            "\n"))
  (message "Backlink added."))

Potential enhancements include:

  • Check whether a backlink is already present and update it if so
  • More control over placement of backlink (rather than directly below title)
2.2.3.5 zetteldeft-find-file-full-title-insert inserts id and title

Select file from minibuffer and insert its link, prepended by § (or zetteldeft-link-indicator).

Based on deft-find-file. 

;;;###autoload
(defun zetteldeft-find-file-full-title-insert (file)
  "Find deft file FILE and insert a link with title."
  (interactive (list
    (completing-read "File to insert full title from: "
      (deft-find-all-files-no-prefix))))
  (insert zetteldeft-link-indicator
          (zetteldeft--lift-id file)
          zetteldeft-link-suffix
          " "
          (zetteldeft--lift-file-title (concat deft-directory file))))

2.2.4 New file creation

2.2.4.1 zetteldeft-id-filename-separator separates id from rest of filename

The string inserted between an ID and the rest of the filename. By default, this is set to a single space, but it can be changed. Use this when you use deft-file-naming-rules to remove spaces from filenames. 

(defcustom zetteldeft-id-filename-separator " "
  "String to separate zetteldeft ID from filename."
  :type 'string
  :group 'zetteldeft)
2.2.4.2 zetteldeft-new-file creates new file

Create new file with filename as zetteldeft-id-format and a string.

Either provide a title as argument, or (when called interactively) enter one in the mini-buffer.

Additionally:

  • generate an ID (unless one is provided as optional argument)
  • insert a title in the newly created file, wrapped in zetteldeft-title-prefix and zetteldeft-title-suffix
  • when evil is loaded, enter the insert state
  • add a full link to the kill ring

First, let’s make sure emacs knows where to find evil-insert-state. 

(declare-function evil-insert-state "evil")

Now for the function itself. 

;;;###autoload
(defun zetteldeft-new-file (str &optional id)
  "Create a new deft file.

The filename is a Zetteldeft ID, appended by STR. The ID will be
generated, unles ID is provided.
A file title will be inserted in the newly created file wrapped in
`zetteldeft-title-prefix' and `zetteldeft-title-suffix'. Filename
(without extension) is added to the kill ring. When `evil' is loaded,
change to insert state."
  (interactive (list (read-string "Note title: ")))
  (let* ((zdId (or id
                   (zetteldeft-generate-id str)))
         (zdName (concat zdId zetteldeft-id-filename-separator str)))
  (deft-new-file-named zdName)
  (kill-new zdName)
  (zetteldeft--insert-title str)
  (save-buffer)
  (when (featurep 'evil) (evil-insert-state))))

Note that the file is only actually created when save-buffer is called.

2.2.4.3 zetteldeft-new-file-and-link inserts generated id

Similar to the previous function, but also insert a link to the newly created note.

The generated ID is passed to zetteldeft-new-file. 

;;;###autoload
(defun zetteldeft-new-file-and-link (str)
  "Create a new note and insert a link to it.
Similar to `zetteldeft-new-file', but insert a link to the new file."
  (interactive (list (read-string "Note title: ")))
  (let ((zdId (zetteldeft-generate-id str)))
    (insert zetteldeft-link-indicator
            zdId
            zetteldeft-link-suffix
            " " str)
    (zetteldeft-new-file str zdId)))
2.2.4.4 zetteldeft-new-file-and-backlink links and backlinks

A further extension includes both a link and a backlink:

  • a link in the original note
  • a backlink in the new note

This obviously only works when the new note is branched out from within a Zetteldeft note (which is verified by zetteldeft--check).

Store a link, generate a new one, and we’re done. 

;;;###autoload
(defun zetteldeft-new-file-and-backlink (str)
  "Create a new note and insert link and backlink."
  (interactive (list (read-string "Note title: ")))
  (let ((ogId (zetteldeft--current-id))
        (zdId (zetteldeft-generate-id str)))
    (insert zetteldeft-link-indicator
            zdId
            zetteldeft-link-suffix
            " " str)
    (zetteldeft-new-file str zdId)
    (newline)
    (insert zetteldeft-backlink-prefix
            zetteldeft-link-indicator
            ogId
            " "
            (zetteldeft--lift-file-title
              (zetteldeft--id-to-full-path ogId)))))

2.2.5 Moving around with avy

2.2.5.1 Following links with zetteldeft-follow-link

This is a wrapper function to follow links to a file. When point is in a link, open the note it links to. When point is not in a link, call avy to jump to and open a selected link. 

;;;###autoload
(defun zetteldeft-follow-link ()
  "Follows zetteldeft link to a file if point is on a link.
Prompts for a link to follow with `zetteldeft-avy-file-search' if it isn't."
  (interactive)
  (if (thing-at-point-looking-at
        (concat zetteldeft-link-indicator
                zetteldeft-id-regex
                zetteldeft-link-suffix))
      (zetteldeft--search-filename
        (zetteldeft--lift-id (zetteldeft--get-thing-at-point)))
    (zetteldeft-avy-file-search)))
2.2.5.2 zetteldeft-avy-tag-search selects and searches tags with avy

Use avy to jump to a tag and search for it.

The search term should include the # as tag identifier, so it’s as easy as jumping to the # and running zetteldeft-search-at-point.

Avy uses zetteldeft-tag-regex as a regular expression. 

;;;###autoload
(defun zetteldeft-avy-tag-search ()
  "Call on avy to jump to a tag.
Tags are filtered with `zetteldeft-tag-regex'."
  (interactive)
  (save-excursion
    (let ((avy-all-windows nil))
    (when (consp (avy-jump zetteldeft-tag-regex))
      (zetteldeft-search-at-point)))))
2.2.5.3 zetteldeft-avy-file-search selects and follows links with avy

Use avy to jump to a link and find the corresponding file. Since the ID should be unique, there should be only one result. That file is then opened (in another window if requested).

Links are found by concatenating zetteldeft-link-indicator, zetteldeftd-id-regex and zetteldeft-link-suffix. 

;;;###autoload
(defun zetteldeft-avy-file-search (&optional otherWindow)
 "Use `avy' to follow a zetteldeft link.
Links are found via `zetteldeft-link-indicator' and `zetteldeft-id-regex'.
Open that file (in another window if OTHERWINDOW)."
  (interactive)
  (save-excursion
    (when (consp (avy-jump (concat zetteldeft-link-indicator
                                   zetteldeft-id-regex
                                   zetteldeft-link-suffix)))
      (zetteldeft--search-filename
        (zetteldeft--lift-id (zetteldeft--get-thing-at-point)) otherWindow))))

Some notes:

  • Function avy-jump returns a cons cell if the character is found and otherwise returns t. That’s why the (when (consp is needed.
  • The optional otherWindow is passed to zetteldeft--search-filename, and from there to deft-open-file.

Let’s also define a function to open a file in another window of choice. Selection of the window occurs via ace-window. 

(declare-function aw-select "ace-window")

When only one window is open, split it first. ace-window will select the other window automatically when only two are available. 

;;;###autoload
(defun zetteldeft-avy-file-search-ace-window ()
  "Use `avy' to follow a zetteldeft link in another window.
Similar to `zetteldeft-avy-file-search', but with window selection.
When only one window is active, split it first.
When more windows are active, select one via `ace-window'."
  (interactive)
  (save-excursion
    (when (consp (avy-jump (concat zetteldeft-link-indicator
                                   zetteldeft-id-regex
                                   zetteldeft-link-suffix)))
      (let ((ID (zetteldeft--lift-id (zetteldeft--get-thing-at-point))))
        (when (eq 1 (length (window-list))) (split-window))
        (select-window (aw-select "Select window..."))
        (zetteldeft--search-filename ID)))))
2.2.5.4 zetteldeft-avy-link-search selects and searches links with avy

Use avy to jump to a link and search for its ID in deft. This means that each note containing this ID is found. If you want to open the note with the ID as its name (i.e., follow a link), use zetteldeft-avy-file-search. 

;;;###autoload
(defun zetteldeft-avy-link-search ()
  "Use `avy' to perform a deft search on a zetteldeft link.
Similar to `zetteldeft-avy-file-search' but performs a full
text search for the link ID instead of filenames only.
Opens immediately if there is only one result."
  (interactive)
  (save-excursion
    (when (consp (avy-jump (concat zetteldeft-link-indicator
                                   zetteldeft-id-regex
                                   zetteldeft-link-suffix)))
      (zetteldeft--search-global
        (zetteldeft--lift-id (zetteldeft--get-thing-at-point))))))

2.2.6 Utility functions

2.2.6.1 zetteldeft-deft-new-search starts a new deft search

The following function launches deft, clears the filter and enters evil-insert-state (when evil is used). 

;;;###autoload
(defun zetteldeft-deft-new-search ()
  "Launch deft, clear filter and enter insert state."
  (interactive)
  (deft)
  (deft-filter-clear)
  (when (featurep 'evil) (evil-insert-state)))
2.2.6.2 zetteldeft--check checks if file is part of zetteldeft

A quick but necessary check to see whether the provided file is part of the deft directory.

To achieve this, first check whether the buffer is visiting a file. When that is the case, take the path of the file the buffer is currently visiting, and check whether the deft-directory is part of that. Signal a user error if it is not.

The file-truename is there to make sure that deft-directory is first expanded to an absolute path before comparing it to the file name of the current buffer (which is already an absolute path). 

(defun zetteldeft--check ()
  "Check if the currently visited file is in `zetteldeft' territory:
whether it has `deft-directory' somewhere in its path."
  (unless (buffer-file-name)
    (user-error "Buffer not visiting a file"))
  (unless (string-match-p
            (regexp-quote (file-truename deft-directory))
            (file-truename (buffer-file-name)))
    (user-error "Not in zetteldeft territory")))
2.2.6.3 zetteldeft--current-id retrieves currentn id

Easy enough. 

(defun zetteldeft--current-id ()
  "Retrieve ID from current file."
  (zetteldeft--check)
  (zetteldeft--lift-id
    (file-name-base (buffer-file-name))))
2.2.6.4 zetteldeft--insert-title inserts file title

Easily insert the title of the current file. Used for generating a new file and renaming a file.

First, a customizable prefix to include before the title. 

(defcustom zetteldeft-title-prefix "#+TITLE: "
  "Prefix string included when `zetteldeft--insert-title' is called.
Formatted for `org-mode' by default.
Don't forget to include a space."
  :type 'string
  :group 'zetteldeft)

Second, a custom string inserted after the title. Below the title, an additional template string is inserted automatically. This string, variable zetteldeft-title-suffix, can be customized and is empty by default. 

(defcustom zetteldeft-title-suffix ""
  "String inserted below title when `zetteldeft--insert-title' is called.
Empty by default.
Don't forget to add `\\n' at the beginning to start a new line."
  :type 'string
  :group 'zetteldeft)

Now the function itself. It gets the base of the buffer file name, takes from it the file title (i.e. strips the link id at the beginning), and inserts the remaining string. 

(defun zetteldeft--insert-title (title)
  "Insert TITLE as title in file.
Prepended by `zetteldeft-title-prefix' and appended by `zetteldeft-title-suffix'."
  (zetteldeft--check)
  (insert
    zetteldeft-title-prefix
    title
    zetteldeft-title-suffix))
2.2.6.5 zetteldeft--lift-file-title returns file title from path

Returns the file title from a file, relying on Deft to do so, thus respecting Deft title rules. 

(defun zetteldeft--lift-file-title (zdFile)
  "Return the title of a zetteldeft note.
ZDFILE should be a full path to a note."
  (let ((deft-use-filename-as-title nil))
    (deft-parse-title
      zdFile
      (with-temp-buffer
        (insert-file-contents zdFile)
        (buffer-string)))))

For this to work, we need to make sure deft-use-filename-as-title is nil, otherwise deft-parse-title will just return the filename.

2.2.6.6 zetteldeft-file-rename updates title and renames visited file

Rename the current note. Prompt for a new title, use the new title to also change the filename (relying on deft name rules), and update the title in the note.

When the current file has no Zetteldeft ID, one is generated. This means that zetteldeft-file-rename can be used to easily generate IDs for notes that have none. 

;;;###autoload
(defun zetteldeft-file-rename ()
  "Change current file's title, and use the new title to rename the file.
Use this on files in the `deft-directory'.
When the file has no Zetteldeft ID, one is generated and included in the new name."
  (interactive)
  (zetteldeft--check)
  (let ((old-filename (buffer-file-name)))
    (when old-filename
      (let* ((old-title (zetteldeft--lift-file-title old-filename))
             (prompt-text (concat "Change " old-title " to: "))
             (new-title (read-string prompt-text old-title))
             (id (or (zetteldeft--lift-id (file-name-base old-filename))
                     (zetteldeft-generate-id new-title old-filename)))
             (new-filename
               (deft-absolute-filename
                 (concat id zetteldeft-id-filename-separator new-title))))
        (rename-file old-filename new-filename)
        (deft-update-visiting-buffers old-filename new-filename)
        (zetteldeft-update-title-in-file new-title)
        (deft-refresh)))))

To update the title of the currently visited file, the following function is used. It simply looks for the zetteldeft-title-prefix, deletes that line, and replaced it with a new title line.

A limitation of this workflow is that it will not work when the zetteldeft-title-prefix has a new line in it. 

(defun zetteldeft-update-title-in-file (title)
  "Update the title of the current file, if present.
Does so by looking for `zetteldeft-title-prefix'.
Replaces current title with TITLE."
  (save-excursion
    (let ((zetteldeft-title-suffix ""))
      (goto-char (point-min))
      (when (re-search-forward (regexp-quote zetteldeft-title-prefix) nil t)
        (delete-region (line-beginning-position) (line-end-position))
        (zetteldeft--insert-title title)))))
2.2.6.7 zetteldeft-count-words counts total number of words

To count the total number of words, lets loop over all the files and count words in each. The total is printed in the minibuffer. 

;;;###autoload
(defun zetteldeft-count-words ()
  "Prints total number of words and notes in the minibuffer."
  (interactive)
  (let ((numWords 0))
    (dolist (deftFile deft-all-files)
      (with-temp-buffer
        (insert-file-contents deftFile)
        (setq numWords (+ numWords (count-words (point-min) (point-max))))))
    (message
      "Your zettelkasten contains %s notes with %s words in total."
      (length deft-all-files) numWords)))
2.2.6.8 zetteldeft-copy-id-current-file copies id in filename

Add the ID from the current file to the kill ring.

Steps:

  1. Get the filename from the buffer
  2. Strip the ID from it.
  3. Wrap the ID with zetteldeft-link-indicator and zetteldeft-link-suffix to create a full link.
  4. Result can be empty string when no id is detected in the filename.
;;;###autoload
(defun zetteldeft-copy-id-current-file ()
  "Copy current ID.
Add the id from the filename the buffer is currently visiting to the
kill ring."
  (interactive)
  (zetteldeft--check)
  (let ((ID (concat zetteldeft-link-indicator
                    (zetteldeft--lift-id (file-name-base (buffer-file-name)))
                    zetteldeft-link-suffix)))
    (kill-new ID)
    (message "%s" ID)))
2.2.6.9 zetteldeft--id-to-full-path returns title from ID

Convert a zetteldeft ID into its full path.

The ID should lead to only one file, obviously, so an error is thrown when this is not the case. 

(defun zetteldeft--id-to-full-path (zdID)
  "Return full path from given zetteldeft ID ZDID.
Throws an error when either none or multiple files are found."
  (let ((deft-filter-only-filenames t))
    (deft-filter zdID t))
  (unless (eq (length deft-current-files) 1)
    (user-error "ID Error. Either no or multiple zetteldeft files found with ID %s" zdID))
  (car deft-current-files))

2.3 Listing all tags

Let’s display all tags in Zetteldeft.

With zetteldeft-tag-buffer they are all gathered in a buffer, including how often they appear.

2.3.1 zetteldeft-tag-buffer puts all tags in a buffer

The name of the buffer we’ll be using: 

(defconst zetteldeft--tag-buffer-name "*zetteldeft-tag-buffer*")

And some code to create that buffer.

It works like so:

  1. Move to the zetteldeft--tag-buffer-name and clear it.
  2. Gather all tags in a property list, as strings with an integer counting how often they appear. This is what zetteldeft--get-all-tags achieves.
  3. Loop through that list and print strings and counts.
  4. Sort results alphabetically.
;;;###autoload
(defun zetteldeft-tag-buffer ()
  "Switch to the `zetteldeft-tag-buffer' and list tags."
  (interactive)
  (switch-to-buffer zetteldeft--tag-buffer-name)
  (erase-buffer)
  (let ((tagList (zetteldeft--get-all-tags)))
    (dolist (zdTag tagList)
      (when (stringp zdTag)
        (insert (format "%s (%d) \n"
                        zdTag
                        (lax-plist-get tagList zdTag)))))
    (unless (eq major-mode 'org-mode) (org-mode))
    (sort-lines nil (point-min) (point-max))))

Note: the zetteldeft--get-all-tags returns a property list, with alternating tags (i.e., a string) and how often they appear (i.e., a number). To only act on the strings, we use a when (stringp zdTag).

2.3.2 zetteldeft--get-all-tags lists all tags

First, we need a variable to store the tags in. 

(defvar zetteldeft--tag-list nil
  "A temporary property list to store all tags.")

Extracting tags with zetteldeft--extract-tags.

  1. Loop through all files in Zetteldeft and extract tags from them
  2. Return this list
(defun zetteldeft--get-all-tags ()
  "Return a plist of all the tags found in zetteldeft files."
  (setq zetteldeft--tag-list (list))
  (dolist (deftFile deft-all-files)
    (zetteldeft--extract-tags deftFile))
  zetteldeft--tag-list)

2.3.3 Tag extracting functions

Some utility functions to achieve all of this.

2.3.3.1 zetteldeft--tag-format adjusts the tag finding regex

The regular expression used to filter out tags, zetteldeft-tag-regex works, but doesn’t filter strictly enough. Hashtags used in URLs are also found, for example.

That’s why we can make the existing regex more precise by stating that tags should be positioned either be at the beginning of a new line, or preceded by a space. 

(defun zetteldeft--tag-format ()
  "Adjust `zetteldeft-tag-regex' for more accurate results."
  (concat "\\(^\\|\s\\)" zetteldeft-tag-regex))

This means that search results now (could) include a space in front of the tag. These spaces are removed in the zetteldeft--extract-tags function, when adding the found tags to the tag buffer.

2.3.3.2 zetteldeft--extract-tags from a file

To extract tags from a single file:

  1. Open a given file in a temporary buffer.
  2. Loop a search for the tag regexp.
  3. When a tag is found, remove any white space from it and add it to the list and increase its count with zetteldeft--tag-count.
  4. Delete the found tag and search again.
(defun zetteldeft--extract-tags (deftFile)
  "Find all tags in DEFTFILE and add them to `zetteldeft--tag-list'.
Increase counters as we go."
  (with-temp-buffer
    (insert-file-contents deftFile)
    (while (re-search-forward (zetteldeft--tag-format) nil t)
      (let ((foundTag (replace-regexp-in-string " " "" (match-string 0))))
        ;; Add found tag to zetteldeft--tag-list if it isn't there already
        (zetteldeft--tag-count foundTag))
      ;; Remove found tag from buffer
      (delete-region (point) (re-search-backward (zetteldeft--tag-format))))))
2.3.3.3 zetteldeft--tag-count increases count of a tag in plist

Check if a provided tag is already in the tag property list. If it isn’t add it and set counter to 1. If it is, increase its counter. 

(defun zetteldeft--tag-count (zdTag)
  (let ((tagCount (lax-plist-get zetteldeft--tag-list zdTag)))
    (if tagCount
        ; if the tag was there already, inc by 1
        (setq zetteldeft--tag-list
          (lax-plist-put zetteldeft--tag-list zdTag (1+ tagCount)))
      ; if tag was not there yet, add & set to 1
      (setq zetteldeft--tag-list
        (lax-plist-put zetteldeft--tag-list zdTag 1)))))

2.4 Gathering notes

In this section:

2.4.1 On ‘gathering notes’

Sometimes you want to easily gather all notes with a certain tag or search term. Say you want to quickly generate a list of links to all files including the tag #zetteldeft.

The following functions do that for you. There are three of them, each either taking a search term as argument or prompting for one:

  1. zetteldeft-insert-list-links inserts a simple list of notes which contain the search term, spelling out the full filename for each note (including ID).
  2. zetteldeft-org-search-include generates org-mode syntax to #+INCLUDE the files below a header with their title.
  3. zetteldeft-org-search-insert inserts the contents of all of these notes below their respective titles.

More documentation can be found below.

2.4.2 List of links

2.4.2.1 zetteldeft-insert-list-links generates list with tagged files

Creates and inserts a list with links to all files with selected search term.

The code gets a list of files that contain the search string, runs through said list and inserts a link for each entry.

When called from a note within zetteldeft, exclude the note itself from the generated list. This is necessary so that when called from an org code block within a note, the note itself is not included (since it will be found by deft, as the search string will be part of that note). To achieve this, get the full file name of the current buffer, and remove it from the search results if its found there. The when part is there so that this deletion is not attempted if the current buffer is not visiting a file. 

;;;###autoload
(defun zetteldeft-insert-list-links (zdSrch)
  "Search for ZDSRCH and insert a list of zetteldeft links to all results."
  (interactive (list (read-string "search string: ")))
  (let ((zdResults (zetteldeft--get-file-list zdSrch))
        (zdThisNote (buffer-file-name)))
    (when zdThisNote (setq zdResults (delete zdThisNote zdResults)))
    (dolist (zdFile zdResults)
      (zetteldeft--list-entry-file-link zdFile))))
2.4.2.2 zetteldeft-insert-list-links-missing generates list with new links

Does the same as the above function, but only inserts IDs that aren’t already present in the current file. In contrast with zetteldeft-insert-list-links, this function can only be used from within a zetteldeft note.

When no missing links are found, i.e. all the notes with the provided strings are already linked to in the current note, a message is printed instead. To be able to customize this message, include a defcustom. 

(defcustom zetteldeft-list-links-missing-message
  "   No missing links with search term =%s= found\n"
  "Message to insert when no missing links are found.
This is used by `zetteldeft-insert-list-links-missing'.
%s will be replaced by the search term provided to
this function."
  :type 'string
  :group 'zetteldeft)

Including missing links is especially handy when you want to check whether all notes with a certain tag are mentioned in a note, or simply to list notes with a specific tag that are not linked to yet (in the current note). Similar to the function above, filter out ID of the current note. In contrast to the function above, this one works with IDs rather than full paths.

A fundamental shortcoming of this piece of code, is that after it is executed, the note now includes the previously missing ID links, which in turn means that on the next run no links will be included… The most immediate solution is for the user to be wary of this, remove any previously inserted links, save the buffer and refresh the deft cache with deft-refresh before calling this function. 

;;;###autoload
(defun zetteldeft-insert-list-links-missing (zdSrch)
  "Insert a list of links to all deft files with a search string ZDSRCH.
In contrast to `zetteldeft-insert-list-links' only include links not
yet present in the current file. Can only be called from a file in the
zetteldeft directory."
  (interactive (list (read-string "search string: ")))
  (zetteldeft--check)
  (let (zdThisID zdCurrentIDs zdFoundIDs zdFinalIDs)
    (setq zdCurrentIDs (zetteldeft--extract-links (buffer-file-name)))
    ;; filter IDs from search results
    (dolist (zdFile (zetteldeft--get-file-list zdSrch))
      (push (zetteldeft--lift-id zdFile) zdFoundIDs))
    ;; create new list with unique ids
    (dolist (zdID zdFoundIDs)
      (unless (member zdID zdCurrentIDs)
        (push zdID zdFinalIDs)))
    ;; remove the ID of the current buffer from said list
    (setq zdThisID (zetteldeft--lift-id (file-name-base (buffer-file-name))))
    (setq zdFinalIDs (delete zdThisID zdFinalIDs))
    ;; finally find full title for each ID and insert it
    (if zdFinalIDs
        (dolist (zdID zdFinalIDs)
          (insert " - "
                  zetteldeft-link-indicator
                  zdID
                  zetteldeft-link-suffix
                  " "
                  (zetteldeft--lift-file-title
                    (zetteldeft--id-to-full-path zdID))
                  "\n"))
      ;; unless the list is empty, then insert a message
      (insert (format zetteldeft-list-links-missing-message zdSrch)))))

The final dolist contains some ugliness to make things work with zetteldeft-link-suffix, in order to get the note’s title without the link in it. Might need to be fixed sometime.

2.4.2.3 zetteldeft--list-entry-file-link includes a file link as list entry

Inserts for given file a link id and title as a list entry. 

(defun zetteldeft--list-entry-file-link (zdFile)
  "Insert ZDFILE as list entry."
  (insert " - "
          zetteldeft-link-indicator
          (zetteldeft--lift-id (file-name-base zdFile))
          zetteldeft-link-suffix
          " "
          (zetteldeft--lift-file-title zdFile)
          "\n"))

2.4.3 Compiling a single org

2.4.3.1 Idea and example
  1. Including notes with given search term

    The following explains what zetteldeft-org-search-include does, but the concept is more or less the same for zetteldeft-org-search-insert.

    For each of the notes with the provided search term, it inserts a heading, a line with #+INCLUDE and the full path to the relevant notes. This results in a single file that can be easily exported.

    The only function meant for use on the users end, is zetteldeft-org-search-include.

    For example, 

    (zetteldeft-org-search-include "#export")

    inserts necessary code to include all files containing the tag #export. The results would look like the following: 

    \* First file title
#+INCLUDE: "/path/to/2018-07-13-2210 First file title.org"

\* File two
#+INCLUDE: "/path/to/2018-07-13-2223 File two.org"

    All functions are documented below.

  2. Semi-automated example

    You could, for example, add the following code to a document and execute (or evaluate) it from within org-mode. Add it under a ‘comment’ type heading to prevent it from being exported itself, like so: * COMMENT Code. 

    (let (frst)
  (save-excursion
    ;; Move to next heading
    (outline-next-heading)
    (setq frst (point))
    ;; Delete everything after
    (delete-region frst (point-max))
    ;; Include the files
    (zetteldeft-org-search-include "#tag")
    ; Sort these entries alphabetically (set mark to use a region)
;   (goto-char frst) (set-mark (point-max))
;   (org-sort-entries nil ?a)
  ))

    The code deletes everything after the current header and inserts all notes with #tag in them.

    In order to also sort the entries alphabetically, uncomment the last two lines.

    A final caveat: don’t put the file with the above code in you deft folder, or it will attempt to include itself (since it has #tag in it).

  3. Issues & things to note

    Before we look at the functions, a note on limitations of the current implementation.

    1. Over-enthusiastic inclusion Sometimes, a tag appears in a file without the need for it to be included. For example, a file with a list of all tags will also include the tag one wants. In the future, this might be resolved by filtering, for example with http://ergoemacs.org/emacs/elisp_filter_list.html.
    2. Inclusion from second line onwards Currently, the #+INCLUDE lines only include from the second line onwards. This is a work-around to prevent #+TITLE lines from being included (and messing up the title on org-export. To change this, edit the inserted strings in the zetteldeft--org-include-file function.
    3. Sorting The files included are unsorted, or rather: sorted as deft provides the results. Attempts at sorting by title are included in zetteldeft--get-file-list, but not working properly. As a solution, use org-sort manually after running zetteldeft-org-search-include.
2.4.3.2 zetteldeft-org-search-include generates #+INCLUDE syntax

Asks user for a search string and inserts headers and #+INCLUDE code for all files with said tag. When used on #tag, make sure to include the # manually. 

;;;###autoload
(defun zetteldeft-org-search-include (zdSrch)
  "Insert `org-mode' syntax to include all files containing ZDSRCH.
Prompt for search string when called interactively."
  (interactive (list (read-string "tag (include the #): ")))
  (dolist (zdFile (zetteldeft--get-file-list zdSrch))
    (zetteldeft--org-include-file zdFile)))
2.4.3.3 zetteldeft-org-search-insert generates titles & file content

Very similar to the previous function, but rather than writing syntax to include files, insert their contents directly. 

;;;###autoload
(defun zetteldeft-org-search-insert (zdSrch)
  "Insert the contents of all files containing ZDSRCH.
Files are separated by `org-mode' headers with corresponding titles.
Prompt for search string when called interactively."
  (interactive (list (read-string "Search term: ")))
  (dolist (zdFile (zetteldeft--get-file-list zdSrch))
    (zetteldeft--org-insert-file zdFile)))

2.4.4 Helper functions

2.4.4.1 zetteldeft-insert-file-contents returns the contents of a file

Returns the contents of a file. 

(defun zetteldeft--file-contents (zdFile &optional removeLines)
  "Insert file contents of a zetteldeft note.
ZDFILE should be a full path to a note.

Optional: leave out first REMOVELINES lines."
  (with-temp-buffer
    (insert-file-contents zdFile)
    (when removeLines
      (kill-whole-line removeLines))
    (buffer-string)))
2.4.4.2 zetteldeft--org-include-file includes a file in org format

Inserts the title as a new header, with the #+INCLUDE line below. Includes only from the second line onward, so that any #+TITLE lines are omitted. 

(defun zetteldeft--org-include-file (zdFile)
  "Insert code to include org file ZDFILE."
  (insert
    ;; Insert org-mode title
    "* " (zetteldeft--lift-file-title zdFile) "\n"
    ;; Insert #+INCLUDE: "file.org" :lines 2-
    "#+INCLUDE: \"" zdFile "\" :lines \"2-\"\n\n"))
2.4.4.3 zetteldeft--org-insert-file inserts a files content

For a file, insert its title and contents (without first 3 lines).

Even better would be: without any of the lines starting with # at the beginning of the file. 

(defun zetteldeft--org-insert-file (zdFile)
  "Insert title and contents of ZDFILE."
  (insert
    ;; Insert org-mode title
    "\n* " (zetteldeft--lift-file-title zdFile) "\n\n"
    ;; Insert file contents (without the first 3 lines)
    (zetteldeft--file-contents zdFile 3)))

2.5 Creating visuals

In this section:

2.5.1 Introducing graphs

Linking notes together in plain text is fun, but sometimes you want to visualize which notes are connected.

The following functions attempt to provide said functionality, but are in a very early stage of development. They generate an org source block for graphviz, which can then be executed to generate a PDF.

A brief introduction:

  • zetteldeft-org-graph-search creates a graph with all the notes containing a provided string.
  • zetteldeft-org-graph-note creates a graph that starts at a note, connects all notes linked to it, and all notes linked to those. In other words, it looks two levels deep.

The resulting graph looks something like this:

zetteldeft-graph.jpg

It’s worth noting, again, that this is very provisional.

2.5.2 Customizable elements

We begin by setting up some customizable parts: syntax that should go at the start and the end of the org-mode source blocks that will be generated.

2.5.2.1 zetteldeft-graph-syntax-begin provides opening syntax

Within graphviz, I advise to use fdp, twopi (which overlaps more) or circo as layouts. 

(defcustom zetteldeft-graph-syntax-begin
  "#+BEGIN_SRC dot :file ./graph.pdf :cmdline -Kfdp -Tpdf
  \n graph {\n"
  "Syntax to be included at the start of the zetteldeft graph."
  :type 'string
  :group 'zetteldeft)
2.5.2.2 zetteldeft-graph-syntax-end provides closing syntax

This merely closes the source block. 

(defcustom zetteldeft-graph-syntax-end
  "} \n#+END_SRC\n"
  "Syntax to be included at the end of the zetteldeft graph."
  :type 'string
  :group 'zetteldeft)

2.5.3 Graph functions

2.5.3.1 zetteldeft-org-graph-search creates graph from search string

An org code block with graphviz code for a graph.pdf.

Find all notes with the provided search term. Loop over this list, and insert title and links for each one.

The links are temporarily stored in zetteldeft--graph-links. 

(defvar zetteldeft--graph-links)

Now for the function itself. 

;;;###autoload
(defun zetteldeft-org-graph-search (str)
  "Insert org source block for graph with zd search results.
STR should be the search the resulting notes of which should be included in the graph."
  (interactive (list (read-string "search string: ")))
  (setq zetteldeft--graph-links (list))
  (let ((zdList (zetteldeft--get-file-list str)))
    (insert zetteldeft-graph-syntax-begin)
    (insert "\n  // links\n")
    (dolist (oneFile zdList)
      (insert "\n")
      (zetteldeft--graph-insert-links oneFile))
    (zetteldeft--graph-insert-all-titles))
  (insert zetteldeft-graph-syntax-end))
2.5.3.2 zetteldeft-org-graph-note creates graph from note

Insert an org source code block for a graphviz presentation of a note and its connections.

When links are added, they are also stored in zetteldeft--graph-links which is later used to insert titles.

When called interactively, select a file from the completion interface. 

;;;###autoload
(defun zetteldeft-org-graph-note (deftFile)
  "Create a graph starting from note DEFTFILE."
  (interactive (list
    (completing-read "Note to start graph from: "
      (deft-find-all-files))))
  (setq zetteldeft--graph-links (list))
  (insert zetteldeft-graph-syntax-begin)
  (insert "\n  // base note and links \n")
  (zetteldeft--graph-insert-links deftFile)
  (zetteldeft--graph-insert-additional-links)
  (zetteldeft--graph-insert-all-titles)
  (insert zetteldeft-graph-syntax-end))

2.5.4 Building blocks

2.5.4.1 zetteldeft--extract-links pulls links from a file

Very similar to the zetteldeft--extract-tags function, but returns links instead of storing them. 

(defun zetteldeft--extract-links (deftFile)
  "Find all links in DEFTFILE and return a list."
  (let ((zdLinks (list)))
    (with-temp-buffer
      (insert-file-contents deftFile)
      (while (re-search-forward zetteldeft-id-regex nil t)
        (let ((foundTag (replace-regexp-in-string " " "" (match-string 0))))
          ;; Add found tag to zdLinks if it isn't there already
          (unless (member foundTag zdLinks)
            (push foundTag zdLinks)))
        ;; Remove found tag from buffer
        (delete-region (point) (re-search-backward zetteldeft-id-regex))))
   zdLinks))
2.5.4.2 zetteldeft--graph-insert-links inserts graphviz links

Insert the sanitized ID from the file, followed by an arrow and all of the links.

Store both the deft file provided and any found files in zetteldeft--graph-links. 

(defun zetteldeft--graph-insert-links (deftFile)
  "Insert links in DEFTFILE in dot graph syntax on a single line.
Any inserted ID is also stored in `zetteldeft--graph-links'."
  (let ((zdId (zetteldeft--lift-id deftFile)))
    (when zdId
      (insert "  \"" zdId "\" -- {")
      (dolist (oneLink (zetteldeft--extract-links deftFile))
        (zetteldeft--graph-store-link oneLink t)
        (insert "\"" oneLink "\" "))
      (insert "}\n")
      (zetteldeft--graph-store-link deftFile))))

Only do all of these things if a Zetteldeft ID is found in the filename, or else insert won’t work (as zdID will be nil).

2.5.4.3 zetteldeft--graph-insert-title inserts graphviz title line

Titles have to be inserted in the correct graphviz format, like so: 

B [label = "Node B"]

The following function should achieve that. 

(defun zetteldeft--graph-insert-title (deftFile)
  "Insert the DEFTFILE title definition in a one line dot graph format."
  (let ((zdTitle
          (replace-regexp-in-string "\"" ""
            (zetteldeft--lift-file-title deftFile)))
        (zdId    (zetteldeft--lift-id deftFile)))
    (when zdId
      (insert "  \"" zdId "\""
              " [label = \"" zdTitle " ("
              zetteldeft-link-indicator zdId zetteldeft-link-suffix ")\"")
      (insert "]" "\n"))
    (zetteldeft--graph-store-link deftFile)))

The title is taken from the file string and any additional quotes removed.

2.5.4.4 zetteldeft--graph-store-link stores provided notes

For future reference, linked files are stored in zetteldeft--graph-links. This function facilitates that process.

Provide a link to a file to store it. Simply providing an ID works too, if you provide the second argument as true. 

(defun zetteldeft--graph-store-link (deftFile &optional idToFile)
  "Push DEFTFILE to zetteldeft--graph-links unless it's already there.
When IDTOFILE is non-nil, DEFTFILE is considered an id
and the the function first looks for the corresponding file."
  (when idToFile
    (let ((deft-filter-only-filenames t))
      (progn
        (deft-filter deftFile t)
        (setq deftFile (car deft-current-files)))))
  (unless (member deftFile zetteldeft--graph-links)
    (push deftFile zetteldeft--graph-links)))
2.5.4.5 zetteldeft--graph-insert-additional-links inserts stored links

Insert links stored in the zetteldeft--graph-links list. Except the first list item, as this is considered the base file already included. 

(defun zetteldeft--graph-insert-additional-links ()
  "Insert rest of `zetteldeft--graph-links'."
  (setq zetteldeft--graph-links (cdr zetteldeft--graph-links))
  (dolist (oneFile zetteldeft--graph-links)
    (zetteldeft--graph-insert-links oneFile)))
2.5.4.6 zetteldeft--graph-insert-all-titles inserts all stored titles

Insert all titles stored in zetteldeft--graph-links. 

(defun zetteldeft--graph-insert-all-titles ()
  "Insert graphviz title lines.
Does this for all links stored in `zetteldeft--graph-links'."
  (insert "\n  // titles \n")
  (dolist (oneLink zetteldeft--graph-links)
    ;; Sometimes, a 'nil' list item is present. Ignore those.
    (when oneLink
      (zetteldeft--graph-insert-title oneLink))))

2.6 Helper function to set keybindings

Since zetteldeft doesn’t provide its own minor mode, keybindings should be global.

The function below sets up some defaults. While I’d suggest people integrate these with their personal setup, this function lowers the bar for entry.

Other setups, for evil or spacemacs, are suggested in section 3.1. 

;;;###autoload
(defun zetteldeft-set-classic-keybindings ()
  "Sets global keybindings for `zetteldeft'."
  (interactive)
  (define-prefix-command 'zetteldeft-prefix)
  (global-set-key (kbd "C-c d") 'zetteldeft-prefix)
  (global-set-key (kbd "C-c d d") 'deft)
  (global-set-key (kbd "C-c d D") 'zetteldeft-deft-new-search)
  (global-set-key (kbd "C-c d R") 'deft-refresh)
  (global-set-key (kbd "C-c d s") 'zetteldeft-search-at-point)
  (global-set-key (kbd "C-c d c") 'zetteldeft-search-current-id)
  (global-set-key (kbd "C-c d f") 'zetteldeft-follow-link)
  (global-set-key (kbd "C-c d F") 'zetteldeft-avy-file-search-ace-window)
  (global-set-key (kbd "C-c d h") 'zetteldeft-go-home)
  (global-set-key (kbd "C-c d l") 'zetteldeft-avy-link-search)
  (global-set-key (kbd "C-c d t") 'zetteldeft-avy-tag-search)
  (global-set-key (kbd "C-c d T") 'zetteldeft-tag-buffer)
  (global-set-key (kbd "C-c d i") 'zetteldeft-find-file-id-insert)
  (global-set-key (kbd "C-c d I") 'zetteldeft-find-file-full-title-insert)
  (global-set-key (kbd "C-c d o") 'zetteldeft-find-file)
  (global-set-key (kbd "C-c d n") 'zetteldeft-new-file)
  (global-set-key (kbd "C-c d N") 'zetteldeft-new-file-and-link)
  (global-set-key (kbd "C-c d B") 'zetteldeft-new-file-and-backlink)
  (global-set-key (kbd "C-c d b") 'zetteldeft-backlink-add)
  (global-set-key (kbd "C-c d r") 'zetteldeft-file-rename)
  (global-set-key (kbd "C-c d x") 'zetteldeft-count-words))

2.7 End of package

That’s all folks! 

(provide 'zetteldeft)
;;; zetteldeft.el ends here

3 Suggested setup

The following assumes deft is loaded manually in your dotfile, it merely configures the package.

None of these code blocks are tangled into the .el file, they are here merely as a guide.

3.1 Suggested zetteldeft keybindings

Since it doesn’t provide a minor mode, zetteldeft package doesn’t have it’s own keymap and doesn’t set any keys by default. It does, however, provide a function to do that for you: zetteldeft-set-classic-keybindings.

Calling this will bind some keys behind C-c d. To set something else, simply copy this code and do your own thing.

Later paragraphs in this section suggest different setups, namely bindings for evil and integration with spacemacs.

3.1.1 Classical keybindings behind C-c d

Calling zetteldeft-set-classic-keybindings will set the following (see 2.6):

Table 1: Classic keys
Key binding Function
C-c d d deft
C-c d D zetteldeft-deft-new-search
C-c d R deft-refresh
C-c d s zetteldeft-search-at-point
C-c d c zetteldeft-search-current-id
C-c d f zetteldeft-follow-link
C-c d F zetteldeft-avy-file-search-ace-window
C-c d h zetteldeft-go-home
C-c d l zetteldeft-avy-link-search
C-c d t zetteldeft-avy-tag-search
C-c d T zetteldeft-tag-buffer
C-c d i zetteldeft-find-file-id-insert
C-c d I zetteldeft-find-file-full-title-insert
C-c d o zetteldeft-find-file
C-c d n zetteldeft-new-file
C-c d N zetteldeft-new-file-and-link
C-c d B zetteldeft-new-file-and-backlink
C-c d b zetteldeft-backlink-add
C-c d r zetteldeft-file-rename
C-c d x zetteldeft-count-words

3.1.2 Keybindings for evil with general

The setup below is the one I use personally. It relies on general.el for modal editing with evil and configures deft and zetteldeft functions behind leader key SPC d. 

(general-define-key
  :prefix "SPC"
  :non-normal-prefix "C-SPC"
  :states '(normal visual motion emacs)
  :keymaps 'override
  "d"  '(nil :wk "deft")
  "dd" '(deft :wk "deft")
  "dD" '(zetteldeft-deft-new-search :wk "new search")
  "dR" '(deft-refresh :wk "refresh")
  "ds" '(zetteldeft-search-at-point :wk "search at point")
  "dc" '(zetteldeft-search-current-id :wk "search current id")
  "df" '(zetteldeft-follow-link :wk "follow link")
  "dF" '(zetteldeft-avy-file-search-ace-window :wk "avy file other window")
  "dh" '(zetteldeft-go-home :wk "go home")
  "dl" '(zetteldeft-avy-link-search :wk "avy link search")
  "dt" '(zetteldeft-avy-tag-search :wk "avy tag search")
  "dT" '(zetteldeft-tag-buffer :wk "tag list")
  "di" '(zetteldeft-find-file-id-insert :wk "insert id")
  "dI" '(zetteldeft-find-file-full-title-insert :wk "insert full title")
  "do" '(zetteldeft-find-file :wk "find file")
  "dn" '(zetteldeft-new-file :wk "new file")
  "dN" '(zetteldeft-new-file-and-link :wk "new file & link")
  "dB" '(zetteldeft-new-file-and-backlink :wk "new file & backlink")
  "db" '(zetteldeft-backlink-add :wk "add backlink")
  "dr" '(zetteldeft-file-rename :wk "rename")
  "dx" '(zetteldeft-count-words :wk "count words"))

3.1.3 Keybindings for spacemacs

For spacemacs, keybindings are defined in the deft layer and activated when deft-zetteldeft is t (see instructions above). These keybindings are reproduced below for reference. Some keys are available globally and some only when in an org-mode buffer.

Table 2: Global keys for spacemacs
Key binding Function
SPC a n z n zetteldeft-new-file
SPC a n z T zetteldeft-tag-buffer
SPC a n z s zetteldeft-search-at-point
SPC a n z o zetteldeft-find-file
Table 3: Org-mode keys for spacemacs
Key binding Function
SPC m z c zetteldeft-search-current-id
SPC m z f zetteldeft-follow-link
SPC m z t zetteldeft-avy-tag-search
SPC m z N zetteldeft-new-file-and-link
SPC m z r zetteldeft-file-rename
SPC m z i zetteldeft-find-file-id-insert
SPC m z I zetteldeft-find-file-full-title-insert
SPC m z s zetteldeft-search-at-point
SPC m z l zetteldeft-avy-link-search
SPC m z F zetteldeft-avy-file-search-ace-window
SPC m z o zetteldeft-find-file

If these don’t suit you, or you would like to add different keys, doing so is trivial. For example: 

(spacemacs/declare-prefix "d" "deft")
(spacemacs/set-leader-keys "df" 'zetteldeft-follow-link)

3.2 Suggested deft setup

3.2.1 Notes & extensions

Note extensions are md, txt and org. First of this list is the default for new notes. 

(setq deft-extensions '("org" "md" "txt"))

3.2.2 Set deft-directory

Search the deft directory recursively, to include subdirectories. 

(setq deft-directory (concat org-directory "/notes/zetteldeft"))
(setq deft-recursive t)

3.2.3 Additional deft functions

Some personal additions. Note that these are functional suggestions, and not included in the zetteldeft package.

A small function to open a file in the other window and shifting focus to it. That final part is what the t argument does. 

(defun efls/deft-open-other ()
 (interactive)
 (deft-open-file-other-window t))

Let’s add another function, to simply preview in the other window, i.e. not switch focus to it. 

(defun efls/deft-open-preview ()
 (interactive)
 (deft-open-file-other-window))

To select results from the item list without leaving the insert state, I add the following keys. 

(with-eval-after-load 'deft
  (define-key deft-mode-map
    (kbd "<tab>") 'efls/deft-open-preview)
  (define-key deft-mode-map
    (kbd "<s-return>") 'efls/deft-open-other)
  (define-key deft-mode-map
    (kbd "s-j") 'evil-next-line)
  (define-key deft-mode-map (kbd "s-k") 'evil-previous-line))

3.2.4 Ignore more org-mode metadata

I tend to write org-mode titles with #+title: (i.e., uncapitalized). Also other org-mode code at the beginning is written in lower case.

In order to filter these from the deft summary, let’s alter the regular expression: 

(setq deft-strip-summary-regexp
 (concat "\\("
         "[\n\t]" ;; blank
         "\\|^#\\+[a-zA-Z_]+:.*$" ;;org-mode metadata
         "\\)"))

Its original value was \\([\n ]\\|^#\\+[[:upper:]_]+:.*$\\).

4 Changelog

An overview of changes, in addition to those mentioned at the top of this document:

  • 09 May: Introduced zetteldeft-link-suffix, which is "" by default.
  • 15 Apr: Replace avy-goto-char with avy-jump. avy functions now work even when zetteldeft-link-indicator is "".
  • 05 Apr: Add zetteldeft-set-classic-keybindings for convenience.
  • 14 Mar: Simplified zetteldeft--get-thing-at-point to better work with customized tags.
  • 14 Mar: Fixed font-lock when customizing the id indicator (thanks to bymoz089).
  • 14 Feb 2020: Documentation updates for usage with spacemacs (thanks to brunosmmm).
  • 30 Dec 2019: Minor fixes for MELPA.
  • 23 Nov: Update suggested keybindings.
  • 10 Nov: Rename namespace prefix zd- to zetteldeft- for MELPA compliance. This requires users to update their keybinding setups.
  • 9 Nov: Rename internal functions & prepare for MELPA.
  • 30 May: Rename zetteldeft-string-after-title to zetteldeft-title-suffix and add customizable zetteldeft-title-prefix.
  • 6 Apr: Add zetteldeft-follow-link for convenience.
  • 28 Feb: Update zetteldeft-avy-tag-search to use avy-jump since avy--generic-jump is obsolete.
  • 16 Feb: include customizable message zetteldeft-list-links-missing for when no missing links are found.
  • 14 Feb: fix zetteldeft-insert-list-links and its brother zetteldeft-insert-list-links-missing.
  • 21 Jan: introduction of customizable zetteldeft-tag-regex.
  • 20 Jan 2019: replaced § with customizable zetteldeft-link-indicator.
  • 18 Dec: insert not yet included links with zetteldeft-insert-list-links-missing.
  • 15 Nov: open link in other window with zetteldeft-avy-file-search-ace-window.
  • 13 Nov: functions to create graphviz added.
  • 4 Nov: add zetteldeft-tag-buffer to find all tags
  • 21 Oct: zetteldeft is now a package.
  • 10 Oct: Insert contents of files with given search term with zetteldeft-org-search-insert.
  • 24 Sep: Count the total number of words in your zetteldeft with zetteldeft-count-words.
  • 18 July 2018: Include a list of links with zetteldeft-insert-list-links, or a list of files with zetteldeft-org-search-include.

Footnotes:

1

For those not yet familiar: deft is a note manager within emacs, for easily searching and retrieving plain text notes. It is inspired by the popular Notational Velocity. Check out jblevins.org/projects/deft/ and notational.net.