Commit 34004d9d authored by Michael Murtaugh's avatar Michael Murtaugh
Browse files

big shift in directory listing to use docroot as cwd

parent d6b9a7ed
......@@ -2,6 +2,27 @@
title: LOG
---
Design designs log
=======================
Extended discussion / notes of design decisions
Single makefile
----------------
After encountering a subtle bug with the "cascading makefile" system, making decision to try a simple single makefile (either explicitly specified, or defaulting to ./makefile).
Problem: "index_.html" exists in the subdirectory 2016-10-05. There's also a (local) makefile in 2016-10-05 which is being detected by makeserver. When this makefile is --question'ed, make returns 0 (instead of 2) because make --question of a file which exists will always be 0 when the file in questions is NOT a target of the makefile.
QUESTION the usefulness of multiple levels of makefiles ... it's creating confusion! ... A SIMPLE (but maybe too drastic) solution: Assume a single makefile, default is any one in the root of the webserver. Eventually other processes can be run for sub directories.
From Indexical back to dynamic index
-------------------------------------
Static index documents are (1) annoying to leave litered around in each visited directory, (2) inherantly tricky to manage the make rules to regenerate them as needed (on file deletion for instance).... ie now fighting with bad indexes. (3) Git integration will only add to this complexity. That said indexical is still interesting as a separate project for generating / merging data + documents.
By Date
=================
24 nov 2016
-------------
Woke up this morning with the thought: this project must be *promiscuous*.
......@@ -103,5 +124,82 @@ March 2017
* 12 April 2017: Observation: auto "edit" linking functionality would be nice -- but from the index page (ie automagically adding the ?edit, which can then be manually removed) (ie original decision to do this auto editing INSIDE the server was too rigid)
August 2017
----------------
Tackling the rather bad problem of path errors with the directory listing.
So, what's subtle is that because make is run in each particular (sub) directory, that is when viewing some PATH make is run with cwd=PATH, implicit rules report paths that are in PATH but rules with path components are displayed with a path relative to the makefile. For example:
cd /var/www/2017/08/; make -f /var/www/makefile -n --debug=v
with makefile implicit rule:
%.html: %.md
Make -n (verbose output) reports:
Prerequisite '2017-08-03.md' is older than target '2017-08-03.html'.
No need to remake target '2017-08-03.html'.
The current code deals with this situation well because there's no handling of slashes in filenames (assumes all file refs are local to cwd).
The problem is then with:
cd /var/www/2017/08/ccframe/; make -f /var/www/2017/08/ccframe/makefile -n --debug=v
and rule:
dist/ccframe.js: src/ccframe.js
Reports:
No need to remake target 'src/ccframe.js'.
Finished prerequisites of target file 'dist/ccframe.js'.
And when viewing the "dist" folder:
cd /var/www/2017/08/ccframe/dist/; make -f /var/www/2017/08/ccframe/makefile -n --debug=v
The reports show the same style of path:
Finished prerequisites of target file 'src/ccframe.js'.
Must remake target 'src/ccframe.js'.
Doesn't work. In fact looking more carefully at the debug output:
cd /var/www/2017/08/ccframe/dist/; make -n -f ../makefile ccframe.js
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Reading makefile '/var/www/2017/08/ccframe/makefile'...
Updating goal targets....
Considering target file 'dist/ccframe.js'.
File 'dist/ccframe.js' does not exist.
Considering target file 'src/ccframe.js'.
File 'src/ccframe.js' does not exist.
Finished prerequisites of target file 'src/ccframe.js'.
Must remake target 'src/ccframe.js'.
Exposes a fundamental problem as in fact the way cwd changes to the subfolder breaks the makefile as it interprets "src/ccframe.js" as relative to "dist" thus "dist/src/ccframe.js" which indeed doesn't exist.
Conclusion:
* the makefile should also run with cwd == docroot ?! Does this fuck other things up?
* reported filepaths should also be processed as being relative to docroot (and thus filtered by the directory listing to check if files are actually within the folder)
However, the implications of this change are that in the makefile I currently use for my "blog", you need to change the "casual" directory specific reference to something that actually finds ALL markdown sources, ie:
# mdsrc=$(shell ls *.md)
mdsrc=$(shell find . -iname "*.md")
And now a make dryrun produces the somewhat alarming result that it wants to convert ALL 'md' files to 'html' (which is of course what the rule says to do). But also in "unexpected" places like node modules that have markdown sources. As far as makeserver is concerned this isn't a problem per se (as it never runs make all, just per file).
A quite bad consequence of this means that every request gets quite slow as it must run the rule for checking for all markdown sources. Another solution (and a better one anyway) would be if the make parser would know about implicit rules and then (always) check implicit rules to generate a more (from itself pro-active) list of potential files (rather than depending on the makefile's all rule itself producing all possible interesting targets). This change perhaps follows a more "promiscuous pipeline" approach? (in that the server is more active in suggesting (more) possibilities).
......@@ -92,25 +92,6 @@ TODO
* Backend to monitor make progress / log (via a websocket?!)
Design designs log
=======================
Extended discussion / notes of design decisions
Single makefile
----------------
After encountering a subtle bug with the "cascading makefile" system, making decision to try a simple single makefile (either explicitly specified, or defaulting to ./makefile).
Problem: "index_.html" exists in the subdirectory 2016-10-05. There's also a (local) makefile in 2016-10-05 which is being detected by makeserver. When this makefile is --question'ed, make returns 0 (instead of 2) because make --question of a file which exists will always be 0 when the file in questions is NOT a target of the makefile.
QUESTION the usefulness of multiple levels of makefiles ... it's creating confusion! ... A SIMPLE (but maybe too drastic) solution: Assume a single makefile, default is any one in the root of the webserver. Eventually other processes can be run for sub directories.
From Indexical back to dynamic index
-------------------------------------
Static index documents are (1) annoying to leave litered around in each visited directory, (2) inherantly tricky to manage the make rules to regenerate them as needed (on file deletion for instance).... ie now fighting with bad indexes. (3) Git integration will only add to this complexity. That said indexical is still interesting as a separate project for generating / merging data + documents.
Changelog
================
[See LOG](LOG.html)
(function () {
/*
# Description
Creates a widget that gives clickable link access to the href's
of *link* elements in (typically *head* of) the document.
For more about the link element, see:
<https://developer.mozilla.org/en-US/docs/Web/HTML/Link_types>
*/
var ELT_ID = "aalinks",
STYLES = {
"#aalinks" : {
position: "fixed",
fontSize: "12px",
fontFamily: "monospace",
"font-size": "10px",
"font-family": "monospace",
top: "10px",
right: "10px",
background: "#BBF",
padding: "10px"
},
"#aalinks p.aalinkshead": {
"text-transform" : "uppercase"
},
"#aalinks p": {
margin: '0'
},
"#aalinks ul.aalinks": {
margin: "0",
listStyle: "square",
paddingLeft: "0"
"list-style" : "square",
"padding-left": "0"
},
"#aalinks div.aalinkscontents": {
marginTop: "0.5em"
}
"margin-top": "1em",
display: 'none'
},
"#aalinks.active div.aalinkscontents" : {
display: 'block'
},
"#aalinks .aalinks-stylesheet": { display: "none" }
};
// https://developer.mozilla.org/en-US/docs/Web/HTML/Link_types
//https://davidwalsh.name/add-rules-stylesheets
function inject_styles(styles) {
......@@ -83,7 +104,7 @@ function linkspanel (elt_id) {
var p1 = _make("p", {
parent: elt,
html: "Links",
class: "aalinksp"
class: "aalinkshead"
}),
contents = _make("div", {
parent: elt,
......@@ -118,15 +139,14 @@ function linkspanel (elt_id) {
}
/* Rollover */
contents.style.display = "none";
elt.addEventListener("touchstart", function () {
contents.style.display = "block";
elt.classList.add("active");
})
elt.addEventListener("mouseenter", function () {
contents.style.display = "block";
elt.classList.add("active");
});
elt.addEventListener("mouseleave", function () {
contents.style.display = "none";
elt.classList.remove("active");
});
/* Process <link> elements */
......
#!/usr/bin/env python
from __future__ import division
from __future__ import division, print_function
import os, urlparse, urllib, subprocess, re, datetime
import jinja2
from jinja2 import Environment, FileSystemLoader
import sys # debug output
from makeserver.makefile import Makefile, ls
MAKE = "make"
def strftime (x, template="%Y-%m-%d %H:%M:%S"):
return x.strftime(template)
def uri_to_path (uri, env):
# Apache CONTEXT_DOCUMENT_ROOT, DOCUMENT_ROOT, CONTEXT_PREFIX
# Twisted PWD
cdr = env.get("CONTEXT_DOCUMENT_ROOT", env.get("DOCUMENT_ROOT", env.get("PWD")))
# eg /home/murtaugh/Music ... path to current docroot
# Strip CONTEXT_PREFIX from uri
# eg /music... the URI component corresponding to the CDR
cp = env.get("CONTEXT_PREFIX", "")
if cp and uri.startswith(cp):
uri = uri[len(cp)+1:]
path = urllib.unquote(uri.lstrip("/"))
return os.path.join(cdr, path)
def is_filename (f):
base, ext = os.path.splitext(f)
return len(ext) > 1 or f.endswith("/")
def humanize_bytes(bytes, precision=0):
"""Return a humanized string representation of a number of bytes.
......@@ -67,79 +49,13 @@ def humanize_bytes(bytes, precision=0):
break
return '%.*f %s' % (precision, bytes / factor, suffix)
# http://stackoverflow.com/questions/898669/how-can-i-detect-if-a-file-is-binary-non-text-in-python
textchars = bytearray({7,8,9,10,12,13,27} | set(range(0x20, 0x100)) - {0x7f})
is_binary_string = lambda bytes: bool(bytes.translate(None, textchars))
def is_binary_file (p):
""" returns none on ioerror """
try:
return not os.path.isdir(p) and is_binary_string(open(p, 'rb').read(1024))
except IOError:
return None
def is_text_file (p):
""" returns none on ioerror """
if os.path.isfile(p):
try:
return not is_binary_string(open(p, 'rb').read(1024))
except IOError:
return None
def targets (makefile):
with open(makefile) as f:
text = f.read().decode("utf-8")
return re.findall(r"^(\w+):", text, flags=re.M)
def make_n (path, makefile=None):
output = ""
if makefile:
args = [MAKE]
args.append("-f")
args.append(makefile)
args.append("-n")
args.append("--debug=v")
try:
output = subprocess.check_output(args, cwd=path)
except subprocess.CalledProcessError, e:
output = ""
makeable = set()
prereqpat = re.compile(r"^\s*Prerequisite '(.+?)' is older than target '(.+?)'.\s*$", re.M)
for a, b in prereqpat.findall(output):
if is_filename(b):
makeable.add(b)
remakepat = re.compile(r"^\s*Must\ remake\ target\ [\`'](.+?)\'\.\s*$", re.M)
remake = set()
for x in remakepat.findall(output):
if is_filename(x):
remake.add(x)
files = os.listdir(path)
def name(x):
fp = os.path.join(path, x)
if os.path.isdir(fp):
return x+"/"
return x
files = [name(x) for x in files]
missingpat = re.compile(r"^\s*File\ [\`'](.+?)\'\ does\ not\ exist\.\s*$", re.M)
missing = set()
for x in missingpat.findall(output):
if is_filename(x):
missing.add(x)
files.append(x)
files.sort()
def wrap (x):
fp = os.path.join(path, x)
is_text = is_text_file(fp)
return (x, os.path.isdir(fp), x in makeable, x in remake, x in missing, is_text)
return [wrap(x) for x in files]
class MakeDirectoryListing (object):
def __init__ (self, makefile, template, docroot, path):
self.makefile = makefile
if makefile:
self.makefile = Makefile(makefile, cwd=docroot)
else:
self.makefile = None
self.template = template
self.docroot = docroot
self.path = path
......@@ -153,8 +69,6 @@ class MakeDirectoryListing (object):
else:
parent_link, _ = os.path.split(rpath)
parent_link = urllib.quote("/"+parent_link)
files = make_n(self.path, self.makefile)
request.setHeader(b"content-type", b"text/html; charset=utf-8")
tvars = {
'request_label': rpath,
......@@ -164,55 +78,12 @@ class MakeDirectoryListing (object):
if not rurl.startswith("/"):
rurl = "/" + rurl
tvars['rurl'] = rurl
items = []
for file, is_dir, makeable, remake, missing, is_text in files:
classes = []
if is_dir:
classes.append("dir")
if is_text:
classes.append("text")
if makeable:
classes.append("makeable")
if remake:
classes.append("remake")
if missing:
classes.append("missing")
classes = " ".join(classes)
link = urllib.quote(file)
label = file
fp = os.path.join(self.path, file)
try:
stat = os.stat(fp)
lastmod = datetime.datetime.fromtimestamp(stat.st_mtime)
fsize = stat.st_size
except OSError:
lastmod = None # "&mdash;"
fsize = 0
buttons = ' '
_, ext = os.path.splitext(file)
ext = ext[1:].lower()
items.append({
'is_dir': is_dir,
'is_text': is_text,
'remake': remake,
'missing': missing,
'makeable': makeable,
'link': link,
'label': label.decode("utf-8"),
'lastmod': lastmod,
'size': fsize,
'buttons': buttons,
'classes': classes,
'ext': ext
})
tvars['items'] = items
# files = make_n(self.path, self.makefile)
if self.makefile:
tvars['targets'] = targets(self.makefile)
tvars['items'] = self.makefile.ls(self.path)
tvars['targets'] = self.makefile.targets()
else:
tvars['items'] = ls(self.path)
tvars['targets'] = []
return self.template.render(tvars).encode("utf-8")
......
import os, subprocess
# def makefile_candidates (path, docroot):
# """
# Return a list of (makefiles, cwd)
# Considers paths from parent(path) up to and including docroot.
# And cwd's of 1. makefile, and 2. parent(path) (if different)
# Example, given files:
# test/makefile
# test/foo/makefile
# test/foo/bar/baz/makefile
# make_candidates(test/foo/bar/baz/index.html)
# =>
# [('test/foo/bar/baz/makefile', 'test/foo/bar/baz'),
# ('test/foo/makefile', 'test/foo'),
# ('test/foo/makefile', 'test/foo/bar/baz'),
# ('test/makefile', 'test'),
# ('test/makefile', 'test/foo/bar/baz')]
# """
# docroot = docroot.rstrip("/")
# filepath, _ = os.path.split(path)
# # 1. Consider all directories from that of path up to (and including) docroot
# allpaths = []
# p = path
# while True:
# p, _ = os.path.split(p)
# allpaths.append(p)
# if p == docroot:
# break
# # 2. Where to makefiles exist?
# makefiles = []
# for p in allpaths:
# mp = os.path.join(p, "makefile")
# if os.path.exists(mp):
# makefiles.append(mp)
# mp = os.path.join(p, "Makefile")
# if os.path.exists(mp):
# makefiles.append(mp)
# # 3. For each makefile consider performing it in both the path of makefile (first), and in dir of path itself (second)
# candidates = []
# for mp in makefiles:
# p, _ = os.path.split(mp)
# candidates.append((mp, p))
# if (p != filepath):
# candidates.append((mp, filepath))
# return candidates
def makefile_candidates (path, docroot):
"""
Returns (makefile, docroot)
"""
docroot = docroot.rstrip("/")
mp = os.path.join(docroot, "makefile")
if os.path.exists(mp):
return ((mp, docroot), )
mp = os.path.join(docroot, "Makefile")
if os.path.exists(mp):
return ((mp, docroot), )
def make_question (path, docroot):
# print ("make_question {0} {1}".format(path, docroot))
# make --question return codes: 0 : file is up to date, 1 : file needs remaking, 2 : file is not makeable
for mp, cwd in makefile_candidates(path, docroot):
# try:
# print ("make?", path, mp, cwd)
rpath = os.path.relpath(path, cwd)
r = subprocess.call(["make", "--question", "-f", mp, rpath], cwd = cwd)
# print ("*make_question {0} {1} {2}".format(path, cwd, r))
if (r == 0 or r == 1):
return (r, mp, cwd)
# except subprocss.CalledProcessError as e:
# pass
return (2, None, None)
from __future__ import print_function
import os, subprocess, re, sys, urllib, datetime
# old interface (del ?)
def make_question_simple (path, makefile, cwd):
# print ("make_question {0} {1}".format(path, docroot))
# make --question return codes: 0 : file is up to date, 1 : file needs remaking, 2 : file is not makeable
rpath = os.path.relpath(path, cwd)
# print ("*make_question {0} {1} {2}".format(path, cwd, r))
return subprocess.call(["make", "--question", "-f", makefile, rpath], cwd = cwd)
textchars = bytearray({7,8,9,10,12,13,27} | set(range(0x20, 0x100)) - {0x7f})
is_binary_string = lambda bytes: bool(bytes.translate(None, textchars))
def is_text_file (p):
""" returns none on ioerror """
if os.path.isfile(p):
try:
return not is_binary_string(open(p, 'rb').read(1024))
except IOError:
return None
class FileInfo (object):
def __init__ (self, path, cwd):
self.cwd = cwd
self.path = path
self.parent, self.filename = os.path.split(self.path)
self.base, self.ext = os.path.splitext(self.filename)
self.ext = self.ext[1:].lower()
self.fullpath = os.path.join(self.cwd, path)
self.makeable = False
self.remake = False
self.prerequisites = []
self.targets = []
self.exists = os.path.exists(self.fullpath)
self.missing = not self.exists
self.is_dir = os.path.isdir(self.fullpath)
if self.is_dir and not self.path.endswith("/"):
self.path += "/"
self.is_text = is_text_file(self.fullpath)
self.link = "/" + urllib.quote(path)
self.label = self.filename.decode("utf-8")
try:
stat = os.stat(self.fullpath)
self.lastmod = datetime.datetime.fromtimestamp(stat.st_mtime)
self.size = stat.st_size
except OSError:
self.lastmod = None # "&mdash;"
self.size = 0
def code (self):
ret = " "
if not self.exists:
ret = "?"
elif self.remake:
ret = "R"
return ret
@property
def classes (self):
classes = []
if self.is_dir:
classes.append("dir")
if self.is_text:
classes.append("text")
if self.makeable:
classes.append("makeable")
if self.remake:
classes.append("remake")
if self.missing:
classes.append("missing")
return " ".join(classes)
def __repr__ (self):
return "{0} {1}".format(self.code(), self.path)
def ls (path, cwd):
ret = [FileInfo(x, cwd) for x in os.listdir(path)]
ret.sort(lambda x: x.path)
return ret
class Makefile (object):
"""
Wraps a makefile and provides methods for getting out useful information
such as:
* lists of makeable files (that may or may not yet exist)
* prerequisite/target relationship
* names of static targets
"""
def __init__ (self, path, make_command = "make", cwd=None):
self.path = path
self.make_command = make_command
if cwd == None:
self.cwd = os.path.abspath(os.path.split(path)[0])
else:
self.cwd = cwd
self._targets = None
def question (self, path):
""" From the make manual:
-q, --question
``Question mode''. Do not run any commands, or print anything; just return an exit status that is zero
if the specified targets are already up to date, nonzero otherwise.
"""
rpath = os.path.relpath(path, self.cwd)
return subprocess.call(["make", "--question", "-f", self.makefile, rpath], cwd = self.cwd)
def is_static_target (self, f):
""" used to filter out static target names from make's debug output """
# base, ext = os.path.splitext(f)
# return not (len(ext) > 1 or f.endswith("/"))
return f in self.targets()
def targets (self):
""" Return a list of static targets """
if self._targets != None:
return self._targets
with open(self.path) as f:
text = f.read().decode("utf-8")
self._targets = re.findall(r"^(\w+):", text, flags=re.M)
return self._targets
def dryrun (self):
output = ""
if makefile:
args = [self.make_command]
args.append("-f")
args.append(self.path)
args.append("-n")
try:
output = subprocess.check_output(args, cwd=self.cwd)
except subprocess.CalledProcessError, e:
output = ""
return output