LOG.md 23 KB
Newer Older
Michael Murtaugh's avatar
Michael Murtaugh committed
1
---
Michael Murtaugh's avatar
Michael Murtaugh committed
2
title: "Makeserver: Change log"
Michael Murtaugh's avatar
Michael Murtaugh committed
3
4
---

Michael Murtaugh's avatar
Michael Murtaugh committed
5
6
2016
=================
7

Michael Murtaugh's avatar
Michael Murtaugh committed
8
It seemed very strong how when generalizing the data join, I suddenly (finally) started to care about the form of the incoming data. In other words, the specific labels of the data started to have an effect as they connected (or didn't) with the names of properties in the rdfa "template" element. Thus the document becomes instrumentalized as a sort of filter / query into the incoming data stream. Now, finally, the benefits of data rewriting via a mechanism like json-ld would be most useful (and appreciated) to complete the loop without a need for ad hoc reframing / tweaking of the data. The hit or miss effect of the selection, filtering and ignoring unnecessary data, is reminiscent (in an encouraging way) of json-ld's contexts as filters.
9

Michael Murtaugh's avatar
Michael Murtaugh committed
10
A surprising discovery has been the re-finding of writing "template" code in the form of a sample table row with rdfa properties. It seems it might be useful to revisit early web publishing flows that must have used this kind of mechanism (vague memories of Internet Explorer specific html element properties like data-source).
11

Michael Murtaugh's avatar
Michael Murtaugh committed
12
Next step? Data value filters (based on a type? ... or simply connected to a specific value) to do date formating and file size niceness!
13

Michael Murtaugh's avatar
Michael Murtaugh committed
14
(later: I realize that this data join stuff was best separate from the main line of the makefile based server and can better be split off and developed on it raises quite a few questions on it's own)
15

Michael Murtaugh's avatar
Michael Murtaugh committed
16
24 nov
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
-------------
Woke up this morning with the thought: this project must be *promiscuous*.

* Are widgets just different kinds of "server flavors" like ?edit and ?serve ?! ... Would be nice to roll the media specific code back into a "media player" HTML ... then make the framing code just deal with hashchange events?!
* Simply drawing on the background could trigger widgets.
* Widgets should "receive" links and have other input/output *nodes* to route messages (instead of the current mess of relaying ALL messages).
* Different kinds of widgets should exist:
    * Viewer (as in iframe, but then better -- ie pdf.js)
* Think how link clicking can map into a TEXT document.
* Widget: Close box
* Editor: hide + extend the options!
* Editor: status: temporary overlay
* Widget: Need font with case (can't check the filenames!)
* Widget: Z-Indexing! (as they overlap!)
* File selection (via icons) + actions (such as delete)... rename with search / replace & patterns (regexp!)
* CRUCIAL: refresh listing without reloading page (and losing widgets!)
* etherpad widget/viewer/editor (or could a similar experience occur somehow directly ?!?!)

Maybe need to make a choice:

* Widgets are all about producing rich editing experiences with interacting components (view + text edit)
* Widgets are about publishing static documents, new ways of writing.
* Or do both of these things work together (rather than need to be split)?

SHOUT OUT TO WEBSTALKER!

WRITING WITH CODE
(promiscuous is a performative writing environment)



Michael Murtaugh's avatar
Michael Murtaugh committed
48
23 nov
49
50
51
52
53
54
-----------
* Enabled operation without makefile
* iframe widget
* move fragment code from player to "iframe"(frame) widget


Michael Murtaugh's avatar
Michael Murtaugh committed
55
22 nov
56
57
58
59
60
61
62
63
64
65
---------------

* <s>Rehookup index.cgi as directory listing</s> -- actually this became a proper class (no CGI!)
* <s>Add a frame creator widget (via mini tool palette!)</s> POC complete, now for the real deal
* make sure drag and drop works with this (to enable play & edit!)
* ensure frame messages working
* Eventually remove the cgi-bin!? ... need to transition to HTTP VERBS? (POST for save?)



Michael Murtaugh's avatar
Michael Murtaugh committed
66
67
68
69
70
71
* 4 Nov 2016: Switch from hard coded directory listing to one that's makefile driven (indexical!)
* 4 Nov 2016: REMOVAL of jinja2 dependency!
* 14 Nov 2016: Change to simplified single makefile
* 18 Nov 2016: Added ?edit syntax and "direct editor" view.


Michael Murtaugh's avatar
Michael Murtaugh committed
72
1 Dec
73
74
75
76
77
----------------
In explaining the project yesterday, I naturally described the server as the "make server" ... and given that make is now by far the focus of it's operation, this seems indeed the best / most descriptive and honest naming. So I made the change. One interesting consequent is that in typing makeserver to start it, one is reminded, nicedly of the make command. Also: Folded in the submodules including editor (leaving ace-builds a submodule). Need to decide what to do with draggable / player.

TODO: Clean the editor code to not require the (rather) unholy mess of both jquery + d3.

Michael Murtaugh's avatar
Michael Murtaugh committed
78
79
80
81
* Enable making files in subdirectories that don't (yet) exist. (make_server.py:FileMaker)


2017
Michael Murtaugh's avatar
Michael Murtaugh committed
82
=======
Michael Murtaugh's avatar
Michael Murtaugh committed
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

Request.process
	res = site.getResourceFor()
	res.render

NEED FileSaver Resource that gets returned for a file.

TODO

* Separate editor options / cleanup editor
* Use favicon for editor / file type specific ?!
* Wrapping options (without having to resize the browser window)
* Font + font sizes!
* "Sticky" preferences (like theme/fonts/etc)


Michael Murtaugh's avatar
Michael Murtaugh committed
99
March
Michael Murtaugh's avatar
Michael Murtaugh committed
100
101
102
103
104
105
-------------
* add ?touch query option + functionality in main code (remove cgi)

* 12 April 2017: Observation: auto "edit" linking functionality would be nice -- but from the index page (ie automagically adding the ?edit, which can then be manually removed) (ie original decision to do this auto editing INSIDE the server was too rigid)


Michael Murtaugh's avatar
Michael Murtaugh committed
106
August
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
----------------
Tackling the rather bad problem of path errors with the directory listing.

So, what's subtle is that because make is run in each particular (sub) directory, that is when viewing some PATH make is run with cwd=PATH, implicit rules report paths that are in PATH but rules with path components are displayed with a path relative to the makefile. For example:

    cd /var/www/2017/08/; make -f /var/www/makefile -n --debug=v

with makefile implicit rule:

    %.html: %.md

Make -n (verbose output) reports:

       Prerequisite '2017-08-03.md' is older than target '2017-08-03.html'.
      No need to remake target '2017-08-03.html'.

The current code deals with this situation well because there's no handling of slashes in filenames (assumes all file refs are local to cwd).

The problem is then with:

    cd /var/www/2017/08/ccframe/; make -f /var/www/2017/08/ccframe/makefile -n --debug=v

and rule:

    dist/ccframe.js: src/ccframe.js

Reports:

     No need to remake target 'src/ccframe.js'.
    Finished prerequisites of target file 'dist/ccframe.js'.

And when viewing the "dist" folder:

    cd /var/www/2017/08/ccframe/dist/; make -f /var/www/2017/08/ccframe/makefile -n --debug=v

The reports show the same style of path:

     Finished prerequisites of target file 'src/ccframe.js'.
    Must remake target 'src/ccframe.js'.

Doesn't work. In fact looking more carefully at the debug output:

    cd /var/www/2017/08/ccframe/dist/; make -n -f ../makefile ccframe.js

    GNU Make 4.1
    Built for x86_64-pc-linux-gnu
    Copyright (C) 1988-2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    Reading makefiles...
    Reading makefile '/var/www/2017/08/ccframe/makefile'...
    Updating goal targets....
    Considering target file 'dist/ccframe.js'.
     File 'dist/ccframe.js' does not exist.
      Considering target file 'src/ccframe.js'.
       File 'src/ccframe.js' does not exist.
       Finished prerequisites of target file 'src/ccframe.js'.
      Must remake target 'src/ccframe.js'.

Exposes a fundamental problem as in fact the way cwd changes to the subfolder breaks the makefile as it interprets "src/ccframe.js" as relative to "dist" thus "dist/src/ccframe.js" which indeed doesn't exist.

Conclusion:

* the makefile should also run with cwd == docroot ?! Does this fuck other things up?
* reported filepaths should also be processed as being relative to docroot (and thus filtered by the directory listing to check if files are actually within the folder)

However, the implications of this change are that in the makefile I currently use for my "blog", you need to change the "casual" directory specific reference to something that actually finds ALL markdown sources, ie:

    # mdsrc=$(shell ls *.md)
    mdsrc=$(shell find . -iname "*.md")

And now a make dryrun produces the somewhat alarming result that it wants to convert ALL 'md' files to 'html' (which is of course what the rule says to do). But also in "unexpected" places like node modules that have markdown sources. As far as makeserver is concerned this isn't a problem per se (as it never runs make all, just per file).

A quite bad consequence of this means that every request gets quite slow as it must run the rule for checking for all markdown sources. Another solution (and a better one anyway) would be if the make parser would know about implicit rules and then (always) check implicit rules to generate a more (from itself pro-active) list of potential files (rather than depending on the makefile's all rule itself producing all possible interesting targets). This change perhaps follows a more "promiscuous pipeline" approach? (in that the server is more active in suggesting (more) possibilities).

Michael Murtaugh's avatar
Michael Murtaugh committed
183
Next Step: Should further integrate the new "Makefile" class into the server (currently it separately implements the actual calls to make) -- in this way force a synchronisation of behaviour between the directory listing and the actual make process.
Michael Murtaugh's avatar
Michael Murtaugh committed
184

Michael Murtaugh's avatar
laptop  
Michael Murtaugh committed
185
2018
Michael Murtaugh's avatar
Michael Murtaugh committed
186
==========
Michael Murtaugh's avatar
laptop  
Michael Murtaugh committed
187
188
189
190
191
192
193
194
195

Thinking to move away from the ?edit way of editing...
1. keep ?edit but map to a shadowable HTML/JS

What in a make file corresponds to the application of a file as filter.
(define a local variable to define an editor!)


Editor via plugin
Michael Murtaugh's avatar
Michael Murtaugh committed
196

Michael Murtaugh's avatar
laptop  
Michael Murtaugh committed
197
198
199
* Shadowable __lib__ folder... or else __server__, __app__ hierarchy... Do shadowing dynamically.
* Editors to offer for files in index.
* Editors as link modifiers? (javascript means of dynamically processing links?)
Michael Murtaugh's avatar
Michael Murtaugh committed
200
201

2020
Michael Murtaugh's avatar
Michael Murtaugh committed
202
========
Michael Murtaugh's avatar
Michael Murtaugh committed
203

Michael Murtaugh's avatar
Michael Murtaugh committed
204
Bumped the version to "3" for an updated implementation based on python3 + aiohttp (replacing twisted).
Michael Murtaugh's avatar
Michael Murtaugh committed
205
206
207
208
209
210
211
212
213
214
215

Makeserver is an inside-out web server.

Makeserver is a scaffolding server that allows dynamic websites to be developed using a makefile.

To do

* (websockets-based) view to monitor output of make command
* add --js option to mixin custom javascript plugins (a la greasemonkey)
* add --saveable option to mixin saveable javascript

Michael Murtaugh's avatar
Michael Murtaugh committed
216
Cookbook as a curated set of deployable makefiles: for instance to convert AVI to webm
Michael Murtaugh's avatar
Michael Murtaugh committed
217
218
219


2021
Michael Murtaugh's avatar
Michael Murtaugh committed
220
========
Michael Murtaugh's avatar
Michael Murtaugh committed
221
222
Putting "code folding" into practice, merging many strands into the original project.

Michael Murtaugh's avatar
Michael Murtaugh committed
223
224
Finally I encounter and start to understand/appreciate [Scons](http://scons.org). Thanks in large part to [Vijya Kumar's From Make to SCons](http://www.bravegnu.org/blog/make-to-scons.html).

Michael Murtaugh's avatar
Michael Murtaugh committed
225
226
227
228
229
230
231
232
233
234
235
236
backend:
python / aiohttp server to serve files, run scons / terminal

(terminado / tornado?)

Bootstrap

Minimal starting point is makeserver like functionality with:
* scons
* terminal output
* (interactive terminal?!)
* (auth)
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
* ?edit => local editor frame (based on code mirror ?)
*       ... if terminado is in place could also be say nano ... but would that be interesting / useful INSIDE editor
*       ... linking to etherpad WOULD be useful, as indeed an OUTSIDE shared editor

... for "sponge" like editing ...
a javascript "session" with access to metadata ?
But it's a server... so if anything (temporary) let that be on the backend .. eventually (as in sponge) connected to a db / persistent index

So what if ... server "sniffs" files producing it's own (ephemeral) rdf graph.

RDF as a light weight (i'm serious ;) abstraction for understanding / working with a graph of files (and eventually related external resources)

Both make and scons define (indirectly) a dependency graph of files .. ie files that can be "sources" for other files. A light way graph structure for representing these relationships is useful as an abstraction between these different tools and the core makeserver. In addition, the idea of a graph is one open to intervention from "outside" via "sponge" like interventions -- ie tools that leave linked data traces such as indexalist, etc.

Challenge: support ephemeral indexing (no backend, or in memory) and also long term indexing (db backend).

INDEX listing using graph <!> inspired by:

  scons -Q -n --tree=status

cheat sheet:

  -Q: Suppress "Reading/Building" progress messages.
  -n: dry run
  --tree=status Print a dependency tree in various formats...
Michael Murtaugh's avatar
Michael Murtaugh committed
262
263
264
265
266
267
268
269
270
271
272

The Long term process question

Some processes take a long time to run (like a video edit, a web crawl, etc). Desire should be to initiate these processes interactively, then "detach" (in tmux style) and later rejoin, evt debug when things go awry. Detaching processes from their interface would be useful (maybe literally making implicit/automatic use of tmux/screen?). (OR maybe it's about performing the function of tmux -- ie deciding how much to buffer / or not (overcome rather than repeat the frustration of "lost" terminal output with tmux -- in ability to scrollback)...) In general, it's about really engaging with the underlying system (processes, etc) while still managing the complexity (designing) for users focussed on aspects other than the system (not "novice";).


* Adapt ptyprocess/xterm (terminado) to aiohttp


[scons use guide on tree](https://scons.org/doc/production/HTML/scons-user.html#idp140637540709104)

273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385

compate output of

  make -n --debug=v

```
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'all'.
 File 'all' does not exist.
  Considering target file 'LOG.html'.
    Considering target file 'LOG.md'.
     Finished prerequisites of target file 'LOG.md'.
    No need to remake target 'LOG.md'.
   Finished prerequisites of target file 'LOG.html'.
   Prerequisite 'LOG.md' is newer than target 'LOG.html'.
  Must remake target 'LOG.html'.
pandoc --toc --css style.css --standalone LOG.md -o LOG.html
  Successfully remade target file 'LOG.html'.
  Considering target file 'README.html'.
    Considering target file 'README.md'.
     Finished prerequisites of target file 'README.md'.
    No need to remake target 'README.md'.
   Finished prerequisites of target file 'README.html'.
   Prerequisite 'README.md' is older than target 'README.html'.
  No need to remake target 'README.html'.
 Finished prerequisites of target file 'all'.
Must remake target 'all'.
Successfully remade target file 'all'.
```

  scons -Q -n --tree=status

```
scons: `.' is up to date.
 E         = exists
  R        = exists in repository only
   b       = implicit builder
   B       = explicit builder
    S      = side effect
     P     = precious
      A    = always build
       C   = current
        N  = no clean
         H = no cache

[E b   C  ]+-.
[E B   C  ]  +-LOG.html
[E     C  ]  | +-LOG.md
[E     C  ]  | +-style.css
[E     C  ]  | +-/usr/bin/pandoc
[E     C  ]  +-LOG.md
[E B   C  ]  +-README.html
[E     C  ]  | +-README.md
[E     C  ]  | +-style.css
[E     C  ]  | +-/usr/bin/pandoc
[E     C  ]  +-README.md
[E     C  ]  +-SConstruct.py
[E     C  ]  +-style.css
```

04 Feb 2021: Code folding + Framing!
-----------------
Putting "collage code" into the project feels very right, suddenly playing youtube videos through the player, but in the context of an annotation with full CC writability! This feels like a very satisfying confirmation that the idea of sponge + collage code + makeserver fitting together is indeed justified. It's a joy to (not have to)  paste a youtube URL directly in the browser bar but rather in a text file where I can contextualize it with writing ... see it as a basis for something useful -- so "Sex crimes" from the Eurthmics becomes the starting point of a playlist, (eventually to be transformed into something, sharable, more durable and *critically* under my own conditions rather than those of the site in question.

Annotation? It's a VIEW of a (future) file...
File menu ipv Annotations makes sense with standard opts (save etc).
<!> need to differentiate local files + external URLs. But this division is indeed a key distinction to the whole inside out basis of the makeserver (and of course collage code, video wiki, etc).


"Frame" tabs are linked to a "target" (initially there is a single "default"). This is a simple extension of the standard behaviour of the browser. In a way then browser tabs correspond to additional targets (which indeed are then shown as separate (cc) tabs).

There's something then special about the relationship of the Annotation & Frame with an eventual Source & Target relationship (from make/scons)... ie the annotation is linked in part to the target ... 

When editing a markdown source (say README.md) while README.html is open, there's never a need to open README.md in the frame (it's not HTML or media for one).

When opening a document, it's a useful view to see a "map" with the recognized IDs within the document.
AHA: DUAL view: frame shows directly while other views show "filtered" view... scanning the src, for instance ... (still there's a need /use for a filtered view that localizes links ...)

sponge models...
* part-whole
* original-format (versions)
* tags
* media

AHA original->format is like source->target

The preferred referent is not always the original? (but in any case the resolution of links in a weft to an "original" is interesting).
How does an therpad view come to be?

(another) powerful idea:
etherpad is (just) another frame... what "adapter"(s) (sponge, etherpad plugin?) need to be in place to integrate as a space for writing annotations (or just to functionally use as a source for derived targets) -- the case of wefts is a good one thus.

CODEMIRROR is a view for a (local) file (could also be used for remote file, just not editable then)...

post shower aha: the mising link with etherpad (etherdump) scons/make ... is an abstraction of version / last modified / content md5 ... a key function of the system would be to (be able to) bridge between (local) targets and (partially) online sources via an abstraction of version. etherdump, in this sense, is *suggestive* of a solution (rather than directly a solution), ie making an abstraction to api/scraping to (efficiently determine when resources have changed, and integrating their eventual download and integration into a local make workflow is key.

ultimately the inside out ness of the system doesn't dictate what external tools are used, but it does favor those tools with porousness / e.g. API access, that allows adaptation an integration (recurring theme, see for instance zulip/integrations).

The confusion of documents/data that Daniel Punday writes about is reflecting in a confusion of work practices and standards for representation (RDF). A line that stands out now for me in Punday is the observation of the signficance (not expanded upon) of Licklider's desire for destroying barriers of disciplinarity (is Punday thus saying that such an erasure isn't possible/desirable!).

Inspiration: Things I hate: playing music in Rhythm box and not being able to edit the title + artist data (presumably because the right extensions aren't installed to actually edit the metadata for whatever format the music file is) ... but also ... what if the file is the whole album (not broken into tracks... also!)...

rdf is like binary... an extreme reduction that is on one hand revolutionary and a foundation for making computing general purpose, and on the other hand not in and of itself an important "encoding" to consider / look at. In actual use, digital data need interpretive/algorithmic frames to reconstitute and represent it. I think in a similar way, RDF as a data structure is a kind of extreme data normalization, permitting a general purpose way for relational data to be expressed. But in terms of actually using/reading/writing the data, its more appropriate to consider the framing as an essential part of making the process meaningful. (There's nothing pure about a bit, nor is there about a triple of "data").... zooming out and framing is necessary to ascribe meaning and the consider the quality / recombine documents. When a document / article / book is "rediscovered", read again years after publication (or an initial reading) -- and one "discovers" something new -- it's because the value derived from documents and from reading depends on a changing (personal / culutural) frame.

Michael Murtaugh's avatar
Michael Murtaugh committed
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
Feb 18
--------------------
* Look into integrating with: https://www.npmjs.com/package/pdfjs-dist for #page= support ? ... [using in website](https://github.com/mozilla/pdf.js/wiki/Setup-pdf.js-in-a-website) ... also example with [webpack](https://github.com/mozilla/pdf.js/tree/master/examples/webpack)


Feb 21
------------

For embedding scons, it's useful to look at the Script/Main.py code, in my install that's currently:

~/.local/lib/python3.7/site-packages/SCons/Script/Main.py

Interesting bits: in the QuestionTask (line 415) you see:

```python
	if self.targets[0].get_state() != SCons.Node.up_to_date or \
           (self.top and not self.targets[0].exists()):
            global exit_status
            global this_build_status
            exit_status = 1
            this_build_status = 1
            self.tm.stop()
```
line 313

```python
            t = self.targets[0]
            for tp in self.options.tree_printers:
                tp.display(t)
            if self.options.debug_includes:
                tree = t.render_include_tree()
                if tree:
                    print()
                    print(tree)
```
Michael Murtaugh's avatar
Michael Murtaugh committed
421

Michael Murtaugh's avatar
Michael Murtaugh committed
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439

Design decisions
=======================

Why were certain key design decisions made?

Single makefile
----------------
After encountering a subtle bug with the "cascading makefile" system, making decision to try a simple single makefile (either explicitly specified, or defaulting to ./makefile).

Problem: "index_.html" exists in the subdirectory 2016-10-05. There's also a (local) makefile in 2016-10-05 which is being detected by makeserver. When this makefile is --question'ed, make returns 0 (instead of 2) because make --question of a file which exists will always be 0 when the file in questions is NOT a target of the makefile.

QUESTION the usefulness of multiple levels of makefiles ... it's creating confusion! ... A SIMPLE (but maybe too drastic) solution: Assume a single makefile, default is any one in the root of the webserver. Eventually other processes can be run for sub directories.

From Indexical back to dynamic index
-------------------------------------
Static index documents are (1) annoying to leave litered around in each visited directory, (2) inherantly tricky to manage the make rules to regenerate them as needed (on file deletion for instance).... ie now fighting with bad indexes. (3) Git integration will only add to this complexity. That said indexical is still interesting as a separate project for generating / merging data + documents.

440
441
442
... (feb 2021) it's not an either/or, a "live" view showing the dependency graph from make/scons is clearly central to the idea of the make server, the relationship with (statically) generated index documents is related and useful, and indeed rightfully external.